I am trying to batch commit but it does not work as expected here is my code
exports.test = functions.https.onRequest(async (req, res) => {
const firestore = admin.firestore();
const batch = firestore.batch();
try {
let snapUser = await admin.firestore().collection("users").get();
const batch = firestore.batch();
snapUser.docs.forEach(async (doc) => {
let snapObject = await doc.ref.collection("objects").get();
if (!objectSnapShot.empty) {
objectSnapShot.docs.forEach((doc) => {
batch.set(
doc.ref,
{
category: doc.data().category,
},
{
merge: true,
}
);
});
await batch.commit()
}
});
res.status(200).send("success");
} catch (error) {
functions.logger.log("Error while deleting account", error);
}
});
Am I doing something wrong ?
Any help would be appreciated
The await batch.commit() runs in every iteration of the loop but you can only commit a batch once. You must commit it after the execution of the loop.
snapUser.docs.forEach(async (doc) => {
// add operations to batch
});
await batch.commit()
Also, it might be better to use for loop when using async-await instead of a forEach() loop. Checkout Using async/await with a forEach loop for more information.
You can receive this errors in cases where transactions have been committed and or
you are using the same batch object to perform several commits. You can go ahead and create a new batch object for every commit, if necessary. You may try to start small and break things down to really understand the concepts.
It seems the WriteBatch is being closed before asynchronous operations try to write to it.
A batch process needs to have two parts, a main function and a callback from each batch.
The issue is you are trying to redeclare the batch process once it's been committed for the first batch.
Your main function is an async which will continue to run all methods asynchronously, ignoring any internal functions. You must make sure that all jobs being applied to the batch are added before attempting to commit, at the core this is a race condition.
You can handle these in multiple ways, such as wrapping it with a promise chain or creating a callback.
Check the following documentation for Managing Data Transactions.
Related
I have a collection of user posts with documents that contain unique id's. In each document is a timestamp created by the server. To visualize the structure looks like this:
userPosts (collection) -> uniqueID (document) -> timeStamp (field)
There are around 100 or so uniqueID's with different timeStamp values. I have queried and ordered my data by the timeStamp field, using .orderBy() and have the ordered results and wish to write to a document in another collection called "mostRecent" with roughly this structure:
mostRecent (collection) -> recentTwenty(document) -> posts (array of maps)
await db.collection(userPosts)
.orderBy("timeStamp").limit(20).get()
.then((querySnapshot) => {
querySnapshot.docs.map((item,key) => {
console.log(item.data())
db.collection("mostRecent").doc("recentTwenty")
.update(
{
posts: FieldValue.arrayUnion(item.data())
},
{ merge: true }
)
}
});
When console.log(item.data) the data is correctly ordered by the timestamp field. However, when I call .update() with the correctly ordered items in item.data() to the recentTwenty doc the order is completely out of whack. I would love to understand what is going on. Thanks in advance for whoever helps me out!
You are NOT waiting for each .update() to complete - it IS an asynchronous operation. Whether it works at ALL also depends on what code you're running outside of what you show - I see no proof that the operations are guaranteed to be competed. The OUTER await doesn't guarantee the synchronicity of the INNER loop.
Firestore arrays ARE ordered by the order they were inserted (if that's how you create them) -but the order the DATABASE will see the operations, as you have shown it here, will depend on Network conditions, etc - you've set up a race condition. At a MINIMUM, you need to await the result of EACH .update() operation to get them in sequence.
A bigger question, though, is WHY use individual .update() at all? Just collect your information using the .map, and write the array in a single operation:
await db.collection(userPosts)
.orderBy("timeStamp").limit(20).get()
.then((querySnapshot) => {
//the following .map() is SYNCHRONOUS
let result_array = querySnapshot.docs.map((item,key) => {
console.log(item.data())
return item.data();
});
//return the PROMISE from the .set(), for the OUTER await
return db.collection("mostRecent").doc("recentTwenty")
.set(
{
posts: result_array;
},
{ merge: true }
)
});
a single write to the database (which is less expensive)
Your problem is that the map runs over a callback function and in your case it is a function with an asynchronous .update() function call. Therefore, you will be never be able to be 100% sure that this will be in order. For that you need the execute them in a synchronous call in for loop.
return new Promise((resolve, reject) => {
db.collection("mostRecent").doc("recentTwenty")
.update(
{
posts: FieldValue.arrayUnion(item.data())
},
{ merge: true }
)
.then((result) => resolve(result))
.catch((err) => reject(err))
;
});
};
await db.collection(userPosts)
.orderBy("timeStamp").limit(20).get()
.then(async (querySnapshot) => {
for(let i = 0; i < querySnapshot.docs.length; i++) {
await updateItem(querySnapshot.docs[i]);
}
})
;
You could also find a way to bulk update all the querySnapshot.docs in a single operation, which would have a better performance. However, I don't know firebase very well so I can't help you on this.
const deleteTaskCount = async(id) => {
const task = Task.findByIdAndDelete(id);
const count = await Task.countDocument({completed:false})
return count;
}
deleteTaskCount("1234").then((count)=>{
console.log(count);
})
I understood that with await in const task = Task.findByIdAndDelete(id); it deletes the user with the id.But without the await keyword(like in the above function) the function still happens, that is it deletes the user but in a non-blocking way after some time.When i ran the above code the count is showing properly but the const task = Task.findByIdAndDelete(id); is not deleting the user.
You should use await for the Task.findByIdAndDelete(id) as well. When you use await, NodeJS will go do some other things until that resolves, and it will only then continue to execute the function (it will not block your code at all).
const deleteTaskCount = async(id) => {
const task = await Task.findByIdAndDelete(id);
const count = await Task.countDocument({completed:false})
return count;
}
deleteTaskCount("1234").then((count)=>{
console.log(count);
})
I don't know much about the mongoose API but I suspect mongoose.query behaves similar to mongo cursors and many other asynchronous database drivers/ORMs. Queries often don't execute unless awaited or chained upon.
In any case, without awaiting Task.findByIdAndDelete, there's no guarantee in concurrency between the two queries. If you want the result of countDocument to reflect the changes in findByIdAndDelete, you must wait on the client-side for that query to complete and then run countDocument.
You would achieve this by adding the await keyword before Task.findByIdAndDelete.
I'd presonally recommend using the promises/thenables as opposed to await/async.
Task.findByIdAndDelete("1234")
.then(task => Task.countDocument({completed:false}))
.then(count => console.log(count));
Imho, it's cleaner, less error prone, more functional, less confusing. YMMV
When I await Promises.all, my unit-test process (npm with Mocha) never exits, but when I await individual promises, it does.
async function f(){...}
async function getVals1() {
const vals = await Promise.all([f(), f()]);
return vals;
}
async function getVals2() {
const vals = []
vals.push(await f());
vals.push(await f());
return vals;
}
Important points:
The values are all sccessfully returned from await Promises.all. They are the expected values (i.e., not themselves Promises), the same as with individual awaits. Everything works fine, but the npm does not exit.
The problem does not always occur, but I don't know what, if anything, conditions the problem.
The application uses MongoDB and non-closure of MongoDB connections can cause applications to hang. But I did check that MongoDB connections are closed (at the end of the test run, long after this await call; and in any case the difference between await Promise.all and individual await is hard to explain.
This was caused by multiple opening of MongoDB clients, so that afterwards, while some were closed, some were not. A lock was needed around mongoDbClient = await MongoClient.connect(...).
It seems the problem is that one or more of your functions is not asynchronous. But without code it hard to understand
I'm implementing a query engine that mass fetches and processes requests. I am using async/await.
Right now the flow of execution runs in a hierarchy where there is a list of items containing queries, and each of those queries have a fetch.
What I am trying to do is bundle the items in groups of n, so even if each of them have m queries with fetches inside, only n*m requests run simultaneously; and specially only one request will be made simultaneously to the same domain.
The problem is, when I await the execution of the items (at the outer level, in a while that groups items and will stop iterations until the promises resolve), those promises are resolving when the execution of an inner query is deferred because of the inner await of the fetch.
That causes my queuing while to only stop momentarily, instead of awaiting for the inner promises to resolve to.
This is the outer, queuing class:
class AsyncItemQueue {
constructor(items, concurrency) {
this.items = items;
this.concurrency = concurrency;
}
run = async () => {
let itemPromises = [];
const bundles = Math.ceil(this.items.length / this.concurrency);
let currentBundle = 0;
while (currentBundle < bundles) {
console.log(`<--------- FETCHING ITEM BUNDLE ${currentBundle} OF ${bundles} --------->`);
const lowerRange = currentBundle * this.concurrency;
const upperRange = (currentBundle + 1) * this.concurrency;
itemPromises.push(
this.items.slice(lowerRange, upperRange).map(item => item.run())
);
await Promise.all(itemPromises);
currentBundle++;
}
};
}
export default AsyncItemQueue;
This is the simple item class that queue is running. I'm omitting superfluous code.
class Item {
// ...
run = async () => {
console.log('Item RUN', this, this.name);
return await Promise.all(this.queries.map(query => {
const itemPromise = query.run(this.name);
return itemPromise;
}));
}
}
And this is the queries contained inside items. Every item has a list of queries. Again, some code is removed as it's not interesting.
class Query {
// ...
run = async (item) => {
// Step 1: If requisites, await.
if (this.requires) {
await this.savedData[this.requires];
}
// Step 2: Resolve URL.
this.resolveUrl(item);
// Step 3: If provides, create promise in savedData.
const fetchPromise = this.fetch();
if (this.saveData) {
this.saveData.forEach(sd => (this.savedData[sd] = fetchPromise));
}
// Step 4: Fetch.
const document = await fetchPromise;
// ...
}
}
The while in AsyncItemQueue is stopping correctly, but only until the execution flow reaches step 3 in Query. As soon as it reaches that fetch, which is a wrapper for the standard fetch functions, the outer promise resolves, and I end up with all the requests being performed at the same time.
I suspect the problem is somewhere in the Query class, but I am stumped as to how to avoid the resolution of the outer promise.
I tried making the Query class run function return document, just in case, but to no avail.
Any idea or guidance would be greatly appreciated. I'll try to answer any questions about the code or provide more if needed.
Thanks!
PS: Here is a codesandbox with a working example: https://codesandbox.io/s/goofy-tesla-iwzem
As you can see in the console exit, the while loop is iterating before the fetches finalize, and they are all being performed at the same time.
I've solved it.
The problem was in the AsyncItemQueue class. Specifically:
itemPromises.push(
this.items.slice(lowerRange, upperRange).map(item => item.run())
);
That was pushing a list of promises into the list, and so, later on:
await Promise.all(itemPromises);
Did not find any promises to wait in that list (because it contained more lists, with promises inside).
The solution was to change the code to:
await Promise.all(this.items.slice(lowerRange, upperRange).map(item => item.run()));
Now it is working perfectly. Items are being run in batches of n, and a new batch will not run until the previous has finished.
I'm not sure this will help anyone but me, but I'll leave it here in case somebody finds a similar problem someday. Thanks for the help.
I will appreciate if you help me with the following case:
Given function:
async function getAllProductsData() {
try {
allProductsInfo = await getDataFromUri(cpCampaignsLink);
allProductsInfo = await getCpCampaignsIdsAndNamesData(allProductsInfo);
await Promise.all(allProductsInfo.map(async (item) => {
item.images = await getProductsOfCampaign(item.id);
}));
allProductsInfo = JSON.stringify(allProductsInfo);
console.log(allProductsInfo);
return allProductsInfo;
} catch(err) {
handleErr(err);
}
}
That function is fired when server is started and it gathers campaigns information from other site: gets data(getDataFromUri()), then extracts from data name and id(getCpCampaignsIdsAndNamesData()), then gets products images for each campaign (getProductsOfCampaign());
I have also express.app with following piece of code:
app.get('/products', async (req, res) => {
if (allProductsInfo.length === undefined) {
console.log('Pending...');
allProductsInfo = await getAllProductsData();
}
res.status(200).send(allProductsInfo);
});
Problem description:
Launch server, wait few seconds until getAllProductsData() gets executed, do '/products' GET request, all works great!
Launch server and IMMEDIATELY fire '/products' GET request' (for that purpose I added IF with console.log('Pending...') expression), I get corrupted result back: it contains all campaigns names, ids, but NO images arrays.
[{"id":"1111","name":"Some name"},{"id":"2222","name":"Some other name"}],
while I was expecting
[{"id":"1111","name":"Some name","images":["URI","URI"...]}...]
I will highly appreciate your help about:
Explaining the flow of what is happening with async execution, and why the result is being sent without waiting for images arrays to be added to object?
If you know some useful articles/specs part covering my topic, I will be thankful for.
Thanks.
There's a huge issue with using a global variable allProductsInfo, and then firing multiple concurrent functions that use it asynchronously. This creates race conditions of all kinds, and you have to consider yourself lucky that you got only not images data.
You can easily solve this by making allProductsInfo a local variable, or at least not use it to store the intermediate results from getDataFromUri and getCpCampaignsIdsAndNamesData - use different (local!) variables for those.
However, even if you do that, you're potentially firing getAllProductsData multiple times, which should not lead to errors but is still inefficient. It's much easier to store a promise in the global variable, initialise this once with a single call to the info gathering procedure, and just to await it every time - which won't be noticeable when it's already fulfilled.
async function getAllProductsData() {
const data = await getDataFromUri(cpCampaignsLink);
const allProductsInfo = await getCpCampaignsIdsAndNamesData(allProductsInfo);
await Promise.all(allProductsInfo.map(async (item) => {
item.images = await getProductsOfCampaign(item.id);
}));
console.log(allProductsInfo);
return JSON.stringify(allProductsInfo);
}
const productDataPromise = getAllProductsData();
productDataPromise.catch(handleErr);
app.get('/products', async (req, res) => {
res.status(200).send(await productDataPromise);
});
Of course you might also want to start your server (or add the /products route to it) only after the data is loaded, and simply serve status 500 until then. Also you should consider what happens when the route is hit after the promise is rejected - not sure what handleErr does.