Develop Reference

Best way to batch create if not exists in firestore

I am working with a project where we create a bunch of entries in firestore based on results from an API endpoint we do not control, using a firestore cloud function. The API endpoint returns ids which we use for the document ids, but it does not include any timestamp information. Since we want to include a createdDate in our documents, we are using admin.firestore.Timestamp.now() to set the timestamp of the document.
On subsequent runs of the function, some of the documents will already exist so if we use batch.commit with create, it will fail since some of the documents exist. However, if we use batch.commit with update, we will either not be able to include a timestamp, or the current timestamp will be overwritten. As a final requirement, we do update these documents from a web application and set some properties like a state, so we can't limit the permissions on the documents to disallow update completely.
What would be the best way to achieve this?
I am currently using .create and have removed the batch, but I feel like this is less performant, and I occasionally do get the error Error: 4 DEADLINE_EXCEEDED on the firestore function.
First prize would be a batch that can create or update the documents, but does not edit the createdDate field. I'm also hoping to avoid reading the documents first to save a read, but I'd be happy to add it in if it's the best solution.
Thanks!
Current code is something like this:
const createDocPromise = docRef
.create(newDoc)
.then(() => {
// success, do nothing
})
.catch(err => {
if (
err.details &&
err.details.includes('Document already exists')
) {
// doc already exists, ignore error
} else {
console.error(`Error creating doc`, err);
}
});

This might not be possible with batched writes as set() will overwrite the existing document, update() will update the timestamp and create() will throw an error as you've mentioned. One workaround would be to use create() for each document with Promise.allSettled() that won't run catch() if any of the promise fails.
const results = [] // results from the API
const promises = results.map((r) => db.doc(`col/${r.id}`).create(r));
const newDocs = await Promise.allSettled(promises)
// either "fulfilled" or "rejected"
newDocs.forEach((result) => console.log(result.status))
If any documents exists already, create() will throw an error and status for that should be rejected. This way you won't have to read the document at first place.
Alternatively, you could store all the IDs in a single document or RTDB and filter out duplicates (this should only cost 1 read per invocation) and then add the data.

Since you prefer to keep the batch and you want to avoid reading the documents, a possible solution would be to store the timestamps in a field of type Array. So, you don't overwrite the createdDate field but save all the values corresponding to the different writes.
This way, when you read one of the documents you sort this array and take the oldest value: it is the very first timestamp that was saved and corresponds to the document creation.
This way you don't need any extra writes or extra reads.

Using a variable in a WHERE clause MySQL/Node

I'm trying to make a MySQL query to filter data from a table. Effectively what I want to do is:
SELECT data FROM table WHERE column IN ?
The filter is coming from checkboxes in a form on a webpage, so I can pass an array or object fairly easily, but it'll be a varying number of parameters for the IN each time, so I can't us multiple ?. I tried making a for loop to make multiple queries concatenate the arrays that the queries returned, but I ran into scope issues with that. I also tried passing an array directly to the query, but that throws a syntax error. I'm sure there's a straightforward answer to this but I'm not sure how to do it.
Edit: source code added:
Here's where I'm at:
const filterShowQuery = `SELECT sl_Show.showId, sl_Band.BandName,
sl_Show.date, sl_Venue.venueName,
sl_Show.length, sl_Show.attendance, sl_Show.encore FROM sl_Show
JOIN sl_Band on sl_Show.BandID = sl_Band.BandId
JOIN sl_Venue on sl_Show.VenueId = sl_Venue.VenueId
WHERE sl_Band.BandName IN (?)
ORDER BY sl_Band.BandName;`;
Trying to get an array into the ? in WHERE sl_Band.BandName IN
const getShows = (req, res,next) =>{
var {bands, venues} = req.body;
var i = 0; //left over from previous attempt
var data = [];
for (b in bands){
mysql.pool.query(filterShowQuery, bands[b], (err, result) => {
if(err){
console.log('filter band error');
next(err);
return;
}
data = data.concat(result);
console.log(data); //data concatenates property and increases through for loop
})
// same action to be performed with venues once solved
// for (v in venues){
// conditions[i] = venues[v];
// i++;
console.log(data); //data is empty when logging from here or using in res
res.json({rows:data});
}
}

SECURITY WARNING!
I must to say: NEVER, NEVER PASS DATA DIRECTLY TO YOUR SQL!
If you don't know why, just google for SQL Injection. There are lots of examples on how it is done, how easily it can be done, and how to protect your application from this sort of attack.
You should always parametrize your queries. But in the very rare case which you really need to insert data concatenating a string into your sql, validate it before.
(E.g.) If it's a number, than use a Regex or some helper method to check if the value you are inserting into your SQL String is really and only a number and nothing else.
But aside that, you did not provide any source code, so it's really hard to give any help before you do that.

Dialogflow to firestore inline editor javascript: sorting and filtering on different parameters of collection

I have a database in Google cloud firestore with a collection of documents that each includes the following parameters:
country, headline, URL, date.
I need to return two separate things depending on the input in Dialogflow. The first thing is the headlines of the 3 latest news and the second thing is the headlines of the 3 latest news in France.
I am to do it in Dialogflows inline editor, which is javascript, but it doesn't seem entirely the same as javascript e.g. in Node.
I have already solved the first problem with the following code:
function TopNews_Global(agent) {
agent.add('Here are the latest Global news');
let documents = db.orderBy('Date', 'desc').limit(3);
return documents.get()
.then(snapshot => {
snapshot.forEach(doc => {
agent.add('-' + doc.data().Headline + '.');
agent.add('Link:(' + doc.data().URL + ')');
});
return Promise.resolve('Read complete');
}).catch(() => {
agent.add('Error reading entry from the Firestore database.');
});
}
Where db is my collection in firebase.
The following code is my solution to the second thing I want to return, and this is where I am stuck. It is not possible to filter and order over two different fields as I do. But there MUST be a way around this - it's a pretty simple thing I wanna do.
https://firebase.google.com/docs/firestore/manage-data/export-import
function TopNews_France(agent) {
agent.add('Here are the latest French news');
let documents = db.orderBy('Date', 'desc').where('Country', '==', 'France').limit(3);
return documents.get()
.then(snapshot => {
snapshot.forEach(doc => {
agent.add('-' + doc.data().Headline + '.');
agent.add('Link:(' + doc.data().URL + ')');
});
return Promise.resolve('Read complete');
}).catch(() => {
agent.add('Error reading entry from the Firestore database.');
});
}

Assuming that as you state, db is a collection object, your query is a valid one for Firestore.
db.orderBy('Date', 'desc').where('Country', '==', 'France').limit(3);
Firestore does allow queries across multiple fields, assuming that the only orderBy or range queries are on the same field. Thus, your query is valid (one orderBy and one non-range where on different fields).
You didn't mention what error you are getting, but I suspect it is this (nominally visible in a Javascript console):
The query requires an index. You can create it here: (url)
That is because this is a query which requires a composite index -- it sorts on one field and filters on another.
Thus, to fix this, you just need to create the index. In an ideal case, you can just click on the URL in that error message and be brought to the exact point in the Firebase console to create the exact index you need. If for some reason you need to create it manually, you can also do that through the console. After it completes building, your query should work.
In this case, you will need to create an index for whatever collection db is pointing at, with the fields Date (in descending order) and Country (in either descending or ascending order -- the equality won't care), and with collection scope.

trying to fix the problems arising from asynchronous code in javascript

I am new in database systems and what I am trying to do is to check whether the e-mail entered by the user during login exists in the database or not. I use Firebase Databse. So, the code I have written is this:
function login(){
var e_mail = document.getElementById("e-mail").value;
rootRef = firebase.database().ref();
rootRef.orderByChild("E_mail").on("child_added", function(snapshot){
lst.push(snapshot.val().E_mail);
//console.log(lst);
})
console.log(lst);
}
let lst = [];
login_btn.onclick = function() {login()};
I want to fetch all e-mails from the database, add them in the list and then loop through that list. Maybe this is not the best way, but that's what I'm working on. I could also just say if (snapshot.val().E_mail == e_mail){alert("there is such a user");}but the problem I have encountered and want to deal with is not that, it's the "callback" function inside login function. When I console the list in the outer function it shows an empty list as it does not run the inner function until it is done with the outer one. I understand this. But how can I avoid or fix this. I want to get the full list of e-mails to be able to loop through it then. Also, I don't know how to end the "loop" in Firebase, because it is sort of looping when it gets the e-mails. So I would like to stop at the moment when it finds a matching e-mail.

You're downloading all users to see if one name exists already. That is a waste of bandwidth.
Instead you should use a query to match the email you're looking for, and only read that node:
rootRef.orderByChild("E_mail").equalTo(e_mail).once("value", function(snapshot){
if (snapshot.exists()) {
// A node with the requested email already exists
}
})
In general, if you need to process all nodes, you'll want to use a value event, which executes for all matching nodes at once. So to get all users from the database, add them to a list, and then do something with that list:
rootRef.orderByChild("E_mail").once("value", function(snapshot){
var list = [];
snapshot.forEach(function(childSnapshot) {
list.push(childSnapshot.val());
});
console.log(list); // this will display the populated array
})
Note that you won't be able to access the list outside of the callback. Even if you declare the variable outside of the callback, it will only be properly populated inside the callback. See Xufox' comment for a link explaining why that is.

How to query orchestrate.io

I was searching for an easy and simple database for a little highscore system for a some games I'm developing in javascript.
I saw Orchestrate.io in github's student developer pack. I found a suitable drivermodule nodejs orchestrate and have integrated them.
The problem comes with querying orchestrate for my data. I have managed saving scores and querying them with db.list('collection'), but this seems to not responding with all data. It appered to me that some values are not returned.
I read about the db.search('collection','query') function. But I don't really understand how I could return all data because I don't want to query in a specific way.
My objects are as simple as follows:
{"name":"Jack","score":1337}
As I understand, one has to send a key, when putting such values to an orchestrate-collection. But I'd like to query the whole collection and get the values in a descendant order in regard to the score.
As for now I end up sorting the result on the client-side.
I hope you guys can give me some hints for a query that can sort for specific values!

You have the option to use a SearchBuilder
db.newSearchBuilder() //Build a search object
.collection('collection') //Set the collection to be searched
.sort(score, 'desc') //Set the order of the results
.query("*") //Empty search
.then(function (res) { //Callback function for results
//Do something with the results
})
Source

By default, .list uses a pagination limit of 10. You can either increase that, e.g.:
db.list('collection', { limit: 100 })
Or use .links, .links.next (from the docs):
db.list('collection', { limit: 10 })
.then(function (page1) {
// Got First Page
if (page1.links && page1.links.next) {
page1.links.next.get().then(function (page2) {
// Got Second Page
})
}
})