Develop Reference

My firebase function update document takes MINUTES to execute the first time

I am developping an app to order food online. As backend service I am using firestore to store the data and files. The user can order dishes and there are limited stocks. So every time a user order a dish and create a basket I update the stock of the corresponding ordered dishes. I am using a firebase function in order to perform this action. To be honest it is the first I am creating firebase function.
Into the Basket object, there is a list of ordered Dishes with the corresponding database DishID. When the basket is created, I go through the DishID list and I update the Quantity in the firestore database. On my local emulator it works perfectly and very fast. But online it takes minutes to perform the first update. I can deal with some seconds. Even if it takes a few seconds (like for cold restart) it's okay. But sometimes it can take 3 minutes and someone else can order a dish during this time.
Here is my code:
//Update the dishes quantities when a basket is created
exports.updateDishesQuantity = functions.firestore.document('/Baskets/{documentId}').onCreate(async (snap, context) => {
try{
//Get the created basket
const originalBasket = snap.data();
originalBasket.OrderedDishes.forEach(async dish => {
const doc = await db.collection('Dishes').doc(dish.DishID);
console.log('Doc created');
return docRef = doc.get()
.then((result) =>{
console.log('DocRef created');
if(result.exists){
console.log('Result exists');
const dishAvailableOnDataBase = result.data().Available;
console.log('Data created');
const newQuantity = { Available: Math.max(dishAvailableOnDataBase - dish.Quantity, 0)};
console.log('Online doc updated');
return result.ref.set(newQuantity, { merge: true });
}else{
console.log("doc doesnt exist");
}
})
.catch(error =>{
console.log(error);
return null;
});
});
}catch(error){
console.log(error);
}
});
I have a couple of logs output to debug the outputs on the server. It's the doc.get() function that takes 2 minutes to execute as you can see on the logger below:
Firebase logger
Thanks for your help,

Thansk for your help. I just edited a little bit your code to make it work. I post my edited code. Thanks a lot, now it takes just 4 seconds to update the quantities.
Kid regards
//Update the dishes quantities when a basket is created
exports.updateDishesQuantity = functions.firestore.document('/Baskets/{documentId}').onCreate(async (snap, context) => {
try {
//Get the created basket
const originalBasket = snap.data();
const promises = [];
const quantities = [];
originalBasket.OrderedDishes.forEach(dish => {
promises.push(db.collection('Dishes').doc(dish.DishID).get());
quantities.push(dish.Quantity);
});
const docSnapshotsArray = await Promise.all(promises);
console.log("Promises", promises);
const promises1 = [];
var i = 0;
docSnapshotsArray.forEach(result => {
if (result.exists) {
const dishAvailableOnDataBase = result.data().Available;
const newQuantity = { Available: Math.max(dishAvailableOnDataBase - quantities[i], 0) };
promises1.push(result.ref.set(newQuantity, { merge: true }));
}
i++;
})
return Promise.all(promises1)
} catch (error) {
console.log(error);
return null;
}
});

You should not use async/await within a forEach() loop, see "JavaScript: async/await with forEach()" and "Using async/await with a forEach loop".
And since your code executes, in parallel, a variable number of calls to the asynchronous Firebase get() and set() methods, you should use Promise.all().
You should refactor your Cloud Function along the following lines:
//Update the dishes quantities when a basket is created
exports.updateDishesQuantity = functions.firestore.document('/Baskets/{documentId}').onCreate(async (snap, context) => {
try {
//Get the created basket
const originalBasket = snap.data();
const promises = [];
originalBasket.OrderedDishes.forEach(dish => {
promises.push(db.collection('Dishes').doc(dish.DishID).get());
});
const docSnapshotsArray = await Promise.all(promises);
const promises1 = [];
docSnapshotsArray.forEach(snap => {
if (result.exists) {
const dishAvailableOnDataBase = result.data().Available;
const newQuantity = { Available: Math.max(dishAvailableOnDataBase - dish.Quantity, 0) };
promises1.push(result.ref.set(newQuantity, { merge: true }));
}
})
return Promise.all(promises1)
} catch (error) {
console.log(error);
return null;
}
});
Note that instead of looping and calling push() you could use the map() method for a much concise code. However, for SO answers, I like the clarity brought by creating an empty array, populating it with a forEach() loop and passing it to Promise.all()...
Also note that since you are updating quantities in a basket you may need to use a Transaction.

how to remove an array item from firestore using JavaScript?

I'm trying to add a delete button to my page. the event listener callback is working properly except for the updateDoc function.
const deleteBook = document.getElementsByClassName('deleteBook');
for (let i = 0; i < deleteBook.length; i++) {
deleteBook[i].addEventListener('click', async () => {
//book to delete
const bookToDelete = deleteBook[i].parentElement.firstElementChild.textContent
// collection title to delete the book from
const bookCol = deleteBook[i].parentElement.parentElement.parentElement.firstElementChild.textContent
// get a snap of the database
const docRef = doc(dataBase, 'users', `${auth.currentUser.uid}`)
const docSnap = (await getDoc(docRef)).data();
// loop over the collections and get a match with the bookCol
for (const col in docSnap) {
if (docSnap[col].title === bookCol) {
console.log('col to delete from found')
console.log(`book to delete ${bookToDelete}`)
await updateDoc(doc(dataBase, 'users', `${auth.currentUser.uid}`), {
[`${col}.books`]: arrayRemove(`${bookToDelete}`)
}).then(()=>{
// fullfiled
console.log('book deleted')
}, ()=>{
// rejected
console.log('promis rejected')
})
}
}
})
}
Col is the object that contains the books array. In the console it always prints book deleted, but in the firestore console, nothing changes. this is a screenshot of the database.
I would really appreciate any help and thank you.

I have replicated the behavior that you're experiencing. I tried changing the content of ${bookToDelete} to any word or even ID. It always returns book deleted even if its deleted or not. The line of code below should be changed in order to get the correct output.
.then(()=>{
// fullfiled
console.log('book deleted')
}, ()=>{
// rejected
console.log('promis rejected')
})
I have created a workaround for your use-case with this kind of issue. See snippet below:
const db = getFirestore();
const colName = "users";
const arrayName = "books";
const usersCol = collection(db, colName);
const userRef = doc(db, colName, `${auth.currentUser.uid}`);
const arrayRef = `${col}.${arrayName}`;
const q = query(usersCol, where(arrayRef, "array-contains", `${bookToDelete}`));
const querySnapshot = await getDocs(q)
.then((querySnapshot) => {
// Removal of object will not proceed if the querySnapshot is empty.
if ((querySnapshot.empty)) {
console.log("No object found!");
}
else {
// Proceeds to removal of object.
updateDoc(userRef, {
[arrayRef]: arrayRemove(`${bookToDelete}`)
})
.then(() => {
// Check again if the object was deleted successfully.
const querySnapshot = getDocs(q)
.then((querySnapshot) => {
if ((querySnapshot.empty)) {
console.log("Book Deleted!");
}
else {
console.log("Failed!");
}
})
});
}
})
// Catch if there are any Firebase errors.
.catch(error => console.log('Failed!', error));
The workaround that I created will query the object in the array then remove the object in the array if it exist. After removing, it will query again to check if the object has been deleted and logs Book Deleted!. Vise versa for checking if the object doesn't exist on the 1st query, it will not proceed on removing them and logs No object found!.
The workaround itself can still be improved. You can add any logic you want for your use-case.
I'd also recommend to create a Feature Request if you want to have this kind of feature together with the arrayRemove Method.

Trouble batch setting a document field by docId in Firestore

I have been using firebase (firestore) for a while but I'm a little stuck and was wondering if anyone can think of a solution.
On the firestore DB I have a single collection of users, each user has an email address and several other fields. In this instance I am checking if a user email exists and if it does, I want to create a list field for that particular user with a listUid. I am referencing the users by email, grabbing the docId for those users and then trying to set a list field for each of them.
I am not getting any error's from firestore, it's simply not updating in the DB for some reason and I can't figure out where I am going wrong. Thanks in advance
export const addListUidToExistingUserList = (
{ firestore },
emailArray,
listUid
) => {
return async () => {
let docIds = [];
emailArray.forEach((emailAddress) => {
//find users by email (works)
const query = db
.collection("users")
.where("email", "==", emailAddress);
//get docId's for user with matching email (works)
query.get().then((querySnapshot) => {
querySnapshot.forEach((doc) => {
docIds.push(doc.id);
});
});
//add a new list with corresponding listUid (does not work)
docIds.forEach((id) => {
let userRef = db.collection("users").doc(id);
batch.set(userRef, { lists: [{ listUid }] });
});
});
return await batch.commit();
};
};

You are running into this issue because your docIds array is always empty at the time you call docIds.forEach.
That's because query.get().then runs asynchronously, and so docIds.forEach is not waiting for it to complete.
You could either:
await query.get().then; or
Add the docIds.forEach function INSIDE the then callback of query.get.
Here are your possible fixes:
await query.get().then
//get docId's for user with matching email (works)
await query.get().then((querySnapshot) => {
querySnapshot.forEach((doc) => {
docIds.push(doc.id);
});
});
OR:
docIds.forEach inside then
//get docId's for user with matching email (works)
query.get().then((querySnapshot) => {
querySnapshot.forEach((doc) => {
docIds.push(doc.id);
});
docIds.forEach((id) => {
let userRef = db.collection("users").doc(id);
batch.set(userRef, { lists: [{ listUid }] });
});
});
Note: Of course, you could also add batch.set directly into your first iteration of querySnapshot.docs.forEach to prevent an unnecessary iteration.

How to append to an array in IndexedDB filtered by ID?

Init code:
let dbPormise = null;
const OBJECT_STORE_NAME = 'pages';
const DB_NAME = 'tracking-log';
To initiate an ObjectStore:
dbPromise = idb.open(DB_NAME, 3, upgradeDB => {
upgradeDB.createObjectStore(OBJECT_STORE_NAME, {
autoIncrement: true,
keypath: 'id'
});
});
This is how I generate a blank record in the IndexedDB:
const tx = db.transaction(OBJECT_STORE_NAME, 'readwrite');
tx.objectStore(OBJECT_STORE_NAME).put(
{ id: newBucketID, data: [] });
Now, at a later point, I have some elements that I want to append to the data array for a particular id.
This is how I tried doing it:
const tx = db.transaction(OBJECT_STORE_NAME, 'readwrite');
tx.objectStore(OBJECT_STORE_NAME).put(
{ id: localStorage.getItem("currentBucket"), data: item }
);
Schema
{
data: Array
}
Every item has a unique key generated and provided by me.
However, this doesn't work and returns an error: "Key already exists in the object store."
So, how can I append a value to a field inside a IDB objectt?

Not sure about the error, but regardless of that, the basic way of adding an item would be something like this:
function addItem(db, bucketId, item) {
return new Promise(addItemExecutor.bind(null, db, bucketId, item));
}
function addItemExecutor(db, bucketId, item, resolve, reject) {
// Start a single writable transaction that we will use for two requests. One to
// find the corresponding bucket, and one to update it.
const tx = db.transaction(OBJECT_STORE_NAME, 'readwrite');
// If all requests completed without error, we are done
tx.oncomplete = resolve;
// If any request fails, the operation fails
tx.onerror = event => reject(event.target.error);
const store = tx.objectStore(OBJECT_STORE_NAME);
// Go find the corresponding bucket object to update
const findRequest = store.get(bucketId);
findRequest.onsuccess = findRequestOnsuccess.bind(findRequest, bucketId, item, reject);
}
// React to the resolution of the get request
function findRequestOnsuccess(bucketId, item, reject, event) {
const bucket = event.target.result;
// If no bucket exists for that id then fail
if(!bucket) {
const error = new Error('No bucket found for id ' + bucketId);
reject(error);
return;
}
// Lazily init the data array property
if(!bucket.data) {
bucket.data = [];
}
// Add our item to the data array
bucket.data.push(item);
// Save the bucket object back into the bucket object store, completely replacing
// the bucket that was there before.
const bucketStore = event.target.source;
bucketStore.put(bucket);
}
async function someCallingCodeExampleAvoidingTopLevelAwait() {
const bucketId = localStorage.currentBucket;
const item = {foo:bar};
const db = evilUnreliableGlobalDbVariableFromSomewhereMagicalForeverOpenAssumeInitialized;
try {
await addItem(db, bucketId, item);
} catch(error) {
console.debug(error);
}
// Leave the database connection open for page lifetime
}

Without a reduced example it's difficult to figure out what's going on. The best way to get help is to create a reduced example of the problem, as in, the smallest amount of code needed to recreate the issue you're seeing, then put it on something like jsbin.com or glitch.com so folks only have to click a link to see the error you're seeing.
I wasn't able to recreate the error you're seeing. You have keypath when it should be keyPath, but I don't think that creates the error you're seeing.
Anyway, here's how to modify a record in IDB:
async function main() {
// Set up the database.
const OBJECT_STORE_NAME = 'pages';
const DB_NAME = 'tracking-log';
const db = await idb.open(DB_NAME, 1, upgradeDB => {
upgradeDB.createObjectStore(OBJECT_STORE_NAME, {
autoIncrement: true,
keyPath: 'id'
});
});
// The OP didn't make it clear what this value was, so I'll guess.
const newBucketID = 1;
{
// Create the record.
const tx = db.transaction(OBJECT_STORE_NAME, 'readwrite');
tx.objectStore(OBJECT_STORE_NAME).put({ id: newBucketID, data: ['first value'] });
}
{
const tx = db.transaction(OBJECT_STORE_NAME, 'readwrite');
// Get the record.
const record = await tx.objectStore(OBJECT_STORE_NAME).get(newBucketID);
// Modify it.
record.data.push('second value');
// Put the modified record back.
tx.objectStore(OBJECT_STORE_NAME).put(record);
}
{
// Read the value to confirm everything worked.
const tx = db.transaction(OBJECT_STORE_NAME);
const value = await tx.objectStore(OBJECT_STORE_NAME).get(newBucketID);
console.log(value);
}
}
main();
And here's that example running: https://jsbin.com/dineguq/edit?js,console

Batch update in knex

I'd like to perform a batch update using Knex.js
For example:
'UPDATE foo SET [theValues] WHERE idFoo = 1'
'UPDATE foo SET [theValues] WHERE idFoo = 2'
with values:
{ name: "FooName1", checked: true } // to `idFoo = 1`
{ name: "FooName2", checked: false } // to `idFoo = 2`
I was using node-mysql previously, which allowed multiple-statements. While using that I simply built a mulitple-statement query string and just send that through the wire in a single run.
I'm not sure how to achieve the same with Knex. I can see batchInsert as an API method I can use, but nothing as far as batchUpdate is concerned.
Note:
I can do an async iteration and update each row separately. That's bad cause it means there's gonna be lots of roundtrips from the server to the DB
I can use the raw() thing of Knex and probably do something similar to what I do with node-mysql. However that defeats the whole knex purpose of being a DB abstraction layer (It introduces strong DB coupling)
So I'd like to do this using something "knex-y".
Any ideas welcome.

I needed to perform a batch update inside a transaction (I didn't want to have partial updates in case something went wrong).
I've resolved it the next way:
// I wrap knex as 'connection'
return connection.transaction(trx => {
const queries = [];
users.forEach(user => {
const query = connection('users')
.where('id', user.id)
.update({
lastActivity: user.lastActivity,
points: user.points,
})
.transacting(trx); // This makes every update be in the same transaction
queries.push(query);
});
Promise.all(queries) // Once every query is written
.then(trx.commit) // We try to execute all of them
.catch(trx.rollback); // And rollback in case any of them goes wrong
});

Assuming you have a collection of valid keys/values for the given table:
// abstract transactional batch update
function batchUpdate(table, collection) {
return knex.transaction(trx => {
const queries = collection.map(tuple =>
knex(table)
.where('id', tuple.id)
.update(tuple)
.transacting(trx)
);
return Promise.all(queries)
.then(trx.commit)
.catch(trx.rollback);
});
}
To call it
batchUpdate('user', [...]);
Are you unfortunately subject to non-conventional column names? No worries, I got you fam:
function batchUpdate(options, collection) {
return knex.transaction(trx => {
const queries = collection.map(tuple =>
knex(options.table)
.where(options.column, tuple[options.column])
.update(tuple)
.transacting(trx)
);
return Promise.all(queries)
.then(trx.commit)
.catch(trx.rollback);
});
}
To call it
batchUpdate({ table: 'user', column: 'user_id' }, [...]);
Modern Syntax Version:
const batchUpdate = (options, collection) => {
const { table, column } = options;
const trx = await knex.transaction();
try {
await Promise.all(collection.map(tuple =>
knex(table)
.where(column, tuple[column])
.update(tuple)
.transacting(trx)
)
);
await trx.commit();
} catch (error) {
await trx.rollback();
}
}

You have a good idea of the pros and cons of each approach. I would recommend a raw query that bulk updates over several async updates. Yes you can run them in parallel, but your bottleneck becomes the time it takes for the db to run each update. Details can be found here.
Below is an example of an batch upsert using knex.raw. Assume that records is an array of objects (one obj for each row we want to update) whose values are the properties names line up with the columns in the database you want to update:
var knex = require('knex'),
_ = require('underscore');
function bulkUpdate (records) {
var updateQuery = [
'INSERT INTO mytable (primaryKeyCol, col2, colN) VALUES',
_.map(records, () => '(?)').join(','),
'ON DUPLICATE KEY UPDATE',
'col2 = VALUES(col2),',
'colN = VALUES(colN)'
].join(' '),
vals = [];
_(records).map(record => {
vals.push(_(record).values());
});
return knex.raw(updateQuery, vals);
}
This answer does a great job explaining the runtime relationship between the two approaches.
Edit:
It was requested that I show what records would look like in this example.
var records = [
{ primaryKeyCol: 123, col2: 'foo', colN: 'bar' },
{ // some other record, same props }
];
Please note that if your record has additional properties than the ones you specified in the query, you cannot do:
_(records).map(record => {
vals.push(_(record).values());
});
Because you will hand too many values to the query per record and knex will fail to match the property values of each record with the ? characters in the query. You instead will need to explicitly push the values on each record that you want to insert into an array like so:
// assume a record has additional property `type` that you dont want to
// insert into the database
// example: { primaryKeyCol: 123, col2: 'foo', colN: 'bar', type: 'baz' }
_(records).map(record => {
vals.push(record.primaryKeyCol);
vals.push(record.col2);
vals.push(record.colN);
});
There are less repetitive ways of doing the above explicit references, but this is just an example. Hope this helps!

The solution works great for me! I just include an ID parameter to make it dynamic across tables with custom ID tags. Chenhai, here's my snippet including a way to return a single array of ID values for the transaction:
function batchUpdate(table, id, collection) {
return knex.transaction((trx) => {
const queries = collection.map(async (tuple) => {
const [tupleId] = await knex(table)
.where(`${id}`, tuple[id])
.update(tuple)
.transacting(trx)
.returning(id);
return tupleId;
});
return Promise.all(queries).then(trx.commit).catch(trx.rollback);
});
}
You can use
response = await batchUpdate("table_name", "custom_table_id", [array of rows to update])
to get the returned array of IDs.

The update can be done in batches, i.e 1000 rows in a batch
And as long as it does it in batches, the bluebird map could be used.
For more information on bluebird map: http://bluebirdjs.com/docs/api/promise.map.html
const limit = 1000;
const totalRows = 50000;
const seq = count => Array(Math.ceil(count / limit)).keys();
map(seq(totalRows), page => updateTable(dbTable, page), { concurrency: 1 });
const updateTable = async (dbTable, page) => {
let offset = limit* page;
return knex(dbTable).pluck('id').limit(limit).offset(offset).then(ids => {
return knex(dbTable)
.whereIn('id', ids)
.update({ date: new Date() })
.then((rows) => {
console.log(`${page} - Updated rows of the table ${dbTable} from ${offset} to ${offset + batch}: `, rows);
})
.catch((err) => {
console.log({ err });
});
})
.catch((err) => {
console.log({ err });
});
};
Where pluck() is used to get ids in array form