This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 2 years ago.
I am trying to create a function that handles file uploads in bulk. I have what I call a floor, and each floor has rooms. Every room can have an audio file. This function should handle the upload of these files. My problem is that the updated "floor" gets written before the files have been uploaded. Meaning I can't make it work asynchronously properly.
This is my function:
- tracks is an object with the keys being the index of the room and the values the files themselves (using an object and not an array because this is an updating funtction so I might not have all the rooms in this object).
export const saveFloor = (floor, tracks, cb) => async () => {
const newFloor = { ...floor };
if (tracks) {
await Object.keys(tracks).forEach(async (trackKey) => {
const track = tracks[trackKey];
const ref = storage.ref(
`/audio/floor_tracks/${floor.id}/${trackKey}/${track.name}`
);
const upload = await ref.put(track);
if (!upload) return;
const downloadUrl = await ref.getDownloadURL();
if (!downloadUrl) return;
newFloor.rooms[trackKey].track = { name: track.name, file: downloadUrl };
console.log("this is the floor", "Just finished one track");
});
console.log("this is the floor", "Just finished all tracks");
}
console.log("this is the floor", newFloor.rooms);
db.collection("floors")
.doc(floor.id)
.set(newFloor)
.then(() => {
cb();
})
.catch((e) => {
console.log("Saving to db failed", e);
});
};
I get Just finished all tracks and newFloor.rooms printed way before Just finished one track.
Edit after the answer from RikkusRukkus, this is my current function, and I am getting the error
Unhandled Rejection (TypeError): undefined is not iterable (cannot read property Symbol(Symbol.iterator))
export const saveFloor = (floor, logoFile, tracks, cb) => async () => {
const newFloor = { ...floor };
if (logoFile) {
const ref = storage.ref(`/images/floor_logos/${floor.id}`);
const upload = await ref.put(logoFile);
if (!upload) return;
const downloadUrl = await ref.getDownloadURL();
if (!downloadUrl) return;
newFloor.logo = downloadUrl;
}
if (tracks) {
await Promise.all(
Object.keys(tracks).forEach(async (trackKey) => {
const track = tracks[trackKey];
const ref = storage.ref(
`/audio/floor_tracks/${floor.id}/${trackKey}/${track.name}`
);
const upload = await ref.put(track);
if (!upload) return;
const downloadUrl = await ref.getDownloadURL();
if (!downloadUrl) return;
newFloor.rooms[trackKey].track = {
name: track.name,
file: downloadUrl,
};
console.log("this is the floor", "Just finsihed one track");
})
);
console.log("this is the floor", "Just finsihed all tracks");
}
console.log("this is the floor", newFloor.rooms);
db.collection("floors")
.doc(floor.id)
.set(newFloor)
.then(() => {
cb();
})
.catch((e) => {
console.log("Saving to db failed", e);
});
};
await Object.keys(tracks).forEach(async (trackKey) => {
Array.prototype.forEach returns undefined, this will not throw an error because you can await any kind of value, not just Promises. So what happens is:
Your array is looped over
Async processes are started
undefined is returned by forEach
It's not a promise so it immediately resolves and with almost no delay, the next line is executed ("finished all tracks")
Some time later, the promises from #2 resolve. Nothing is done with their resolved values (there are no values in the return statement, so the promises will resolve to undefined).
You can await an array of promises like so:
await Promise.all( Object.keys(tracks).map(async (trackKey) => { ... }) )
Array.prototype.map does return values, in this case Promises. Promise.all takes an array of Promises & resolves when all resolve or rejects ("throws") when any one rejects.
_
If you were to include return values in your async functions then Promise.all will resolve to an array of these returned values, in order.
Related
I've followed several guides on how to correctly wait for the data from my ListFile() function. I have logged the output to the console in ListFile() so I know the data is correct. When I try and await for the function to complete and log the data, I get one of two outputs when experimenting with awaits and promises:
getData error: TypeError: undefined is not an object (evaluating 'response.json')
or
Promise {
"_U": 0,
"_V": 0,
"_W": null,
"_X": null,
}
I'm creating a React Native app and the function that calls ListFile() contains a bunch of other components that are rendered so I can't make the entire function an async function. Not sure what else to do here.
Calls ListFile()
const getData = async () => {
try {
const response = await ListFile();
const data = response.json();
console.log(data);
} catch (err) {
console.log("getData error: " + err);
}
}
getData(); //always returns getData error: TypeError: undefined is not an object (evaluating 'response.json')
ListFile()
async function ListFile() {
// Reference Firebase Container
var filesInStorageList = [];
let content = [];
const listRef = ref(fbStorage, filePath);
console.log("Listing Files");
// List all files in container
listAll(listRef)
.then((res) => {
res.prefixes.forEach((folderRef) => {
// All the prefixes under listRef.
// You may call listAll() recursively on them.
console.log(res);
});
res.items.forEach((itemRef) => {
// All the items under listRef.
let fileInStorage = itemRef["_location"]["path_"].slice(filePath.length);
filesInStorageList.push(fileInStorage);
// CORRECT DATA IS LOGGED console.log("Pushing " + fileInStorage);
// CORRECT DATA IS LOGGED console.log(filesInStorageList);
});
return filesInStorageList;
}).catch((error) => {
// Uh-oh, an error occurred!
console.log("ListFile - ERROR");
});
}
You're mixing async/await and Promises. Although they serve similar purposes, generally they are used separately to avoid situations like this where they are used together and then create unexpected results if one doesn't know what to look for.
You annotate your ListFile function with async, which means that it's expecting to find await somewhere. Instead, you use a Promise-based .then and .catch.
To convert your code, you could change it to this:
async function ListFile() {
// Reference Firebase Container
var filesInStorageList = [];
let content = [];
const listRef = ref(fbStorage, filePath);
console.log("Listing Files");
// List all files in container
try {
const res = await listAll(listRef);
//...do your forEach loops
return filesInStorageList;
} catch(err) {
//handle your error
}
}
You should add a return on the listAll method to return its value when calling it from getData(). See sample code below:
const getData = async () => {
try {
ListFile()
.then((response) => {
// do something here.
console.log(response);
});
} catch (err) {
console.log("getData error: " + err);
}
}
getData();
async function ListFile() {
// Reference Firebase Container
var filesInStorageList = [];
let content = [];
const filePath = "test/";
const listRef = ref(storage, filePath);
// List all files in container
return listAll(listRef)
.then((res) => {
res.prefixes.forEach((folderRef) => {
// All the prefixes under listRef.
// You may call listAll() recursively on them.
console.log(res);
});
res.items.forEach((itemRef) => {
// All the items under listRef.
let fileInStorage = itemRef["_location"]["path_"].slice(filePath.length);
filesInStorageList.push(fileInStorage);
// CORRECT DATA IS LOGGED console.log("Pushing " + fileInStorage);
// CORRECT DATA IS LOGGED console.log(filesInStorageList);
// console.log(filesInStorageList);
});
return filesInStorageList;
}).catch((error) => {
// Uh-oh, an error occurred!
console.log(error);
});
}
Also, as #jnpdx stated, you have to use a Promise-based .then and .catch on your ListFile()
The following code is for my FCM function where I am listening to firestore the getting tokens before constructing payload to send. Every time its sent the system logs that the tokens are empty. How can I make sure its not empty when sending the fcm?
let functions = require('firebase-functions');
let admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.sendNotification =functions.firestore.document('chatrooms/{chatRoomId}/messages/{messageId}')
.onWrite((snap, context) => {
let message = snap.after.data().messageBody;
let messageSender = snap.after.data().senderName;
let messageUserId = snap.after.data().userId;
let chatRoomId = context.params.chatRoomId;
let tokens = [];
let chatRoomRef = admin.firestore().collection("chatrooms").doc(chatRoomId);
return admin.firestore().runTransaction(t => {
return t.get(chatRoomRef)
.then(chatroom => {
let usersArray = chatroom.data().chatMembers;
usersArray.forEach(user_id => {
let userIdRef = admin.firestore().collection("tokens").doc(user_id);
return t.get(userIdRef).then(doc => {
if (doc.exists) {
let user_token = doc.data().token;
functions.logger.log('token: ', token);
tokens.push(user_token);
}
}).catch(err => {
functions.logger.error(err);
})
});
});
}).then(() => {
//The transaction has run successfully, we expect tokens array to not be empty
functions.logger.log("Construction the notification message.");
const payload = {
data: {
data_type: "data_type_chat_message",
title: "Tuchat",
message: message,
sender_id: messageUserId,
sender_name: messageSender,
chatRoom_id: chatRoomId
}
};
const options = {
priority: "high",
timeToLive: 60 * 60 * 24
};
return admin.messaging().sendToDevice(tokens, payload).catch(err => {
functions.logger.error(err);
});
}).catch(err => {
functions.logger.error('Transaction error: ', err);
})
});
Also before trying transactions it was returning empty tokens.
The problem is because of the way you're dealing with promises inside the forEach loop. The loop will not wait for promises to be resolved for the code inside it. It is currently just iterating as fast as possible over each user ID and not waiting for the queries to complete inside it. That means the code will continue before tokens can get populated.
You should instead collect each of the promises inside the loop and use Promise.all() to wait for the entire batch, then return the promise from that to indicate when all the work is complete. That will ensure that tokens contains everything you want before the next then in the chain is executed.
The general form of the code will be like this:
.then(() => {
const promises = []
collection.forEach(item => {
const promise = doSomeWork()
promises.push(promise)
})
return Promise.all(promises)
})
.then(() => {
// this will continue only after all promises are resolved
})
See also:
How to use promise in forEach loop of array to populate an object
Node JS Promise.all and forEach
This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Closed 2 years ago.
My code looks something like this:
(async () => {
try {
const results = await heavyCalculation();
saveResultsToFiles(results);
} catch (e) {
handleError(e);
} finally {
process.exit(0);
}
})();
const saveResultsToFiles = (results) => {
results.forEach(result => {
(async () => await saveResultFile(result));
})
}
const saveResultFile = (result) => {
return promiseToPreprocess(result)
.then(processedResult => saveToFile(processedResult))
}
const promiseToPreprocess = async (result) => {
// this function returns a promise to preprocess the data
}
const saveToFile = (data) => {
// this function synchronously saves data to a file
}
I thought this code would
perform calculations
wait for each piece of the results to be preprocessed and saved to a file
then exit
The first step works, as the program seems to await the heavy calculation results. However, it appears the finally clause is entered before the promise within the forEach-loop is resolved, causing the program to exit early. What am I doing wrong?
You have two problems here:
Your forEach loop in saveResultsToFiles does not return anything so you have no way to make other parts of your code "wait" for each item's promise to resolve.
saveResultFile returns a promise but this promise is not awaited in your try block.
The result of both of these issues is that the try block only "starts" the process of saving to files but doesn't wait for it to finish before yieling to the finally block.
Here are solutions you could try.
You need to be able to await each of the saveResultFile calls and for that you need to access the array of promises instanciated in saveResultsToFiles. With .map you will actually get an array of results (as opposed to .forEach):
const saveResultsToFiles = (results) => {
return results.map(result => saveResultFile(result));
}
Now that saveResultsToFiles actually returns an array of promises, you should await all of them before proceeding. This is exactly what Promise.all is for:
try {
const results = await heavyCalculation();
await Promise.all(saveResultsToFiles(results));
}
You are not awaiting saveResultsToFiles(results);
Try:
(async () => {
try {
const results = await heavyCalculation();
saveResultsToFiles(results);
} catch (e) {
handleError(e);
} finally {
process.exit(0);
}
})();
const saveResultsToFiles = async (results) => {
results.forEach(result => {
await saveResultFile(result);
})
}
const saveResultFile = (result) => {
return promiseToPreprocess(result)
.then(processedResult => saveToFile(processedResult))
}
const promiseToPreprocess = async (result) => {
// this function returns a promise to preprocess the data
}
const saveToFile = (data) => {
// this function synchronously saves data to a file
}
// The Below code already contains the suggestions from the answers and hence works :)
within the below script I tried to fully execute the 'createDatabase' function before the .then call at the end starts to run. Unfortunately I couldn't figure out a solution to achieve just that.
In generell the flow should be as followed:
GetFiles - Fully Execute it
CreateDatabase - Fully Execute it (while awaiting each .map call to finish before starting the next)
Exit the script within the .then call
Thanks a lot for any advise :)
const db = require("../database")
const fsp = require("fs").promises
const root = "./database/migrations/"
const getFiles = async () => {
let fileNames = await fsp.readdir(root)
return fileNames.map(fileName => Number(fileName.split(".")[0]))
}
const createDatabase = async fileNumbers => {
fileNumbers.sort((a, b) => a - b)
for (let fileNumber of fileNumbers) {
const length = fileNumber.toString().length
const x = require(`.${root}${fileNumber.toString()}.js`)
await x.create()
}
}
const run = async () => {
let fileNumbers = await getFiles()
await createDatabase(fileNumbers)
}
run()
.then(() => {
console.log("Database setup successfully!")
db.end()
process.exitCode = 0
})
.catch(err => {
console.log("Error creating Database!", err)
})
The x.create code looks as follows:
const dbQ = (query, message) => {
return new Promise((resolve, reject) => {
db.query(query, (err, result) => {
if (err) {
console.log(`Error: ${err.sqlMessage}`)
return reject()
}
console.log(`Success: ${message}!`)
return resolve()
})
})
}
x.create = async () => {
const query = `
CREATE TABLE IF NOT EXISTS Country (
Code CHAR(2) UNIQUE NOT NULL,
Flag VARCHAR(1024),
Name_de VARCHAR(64) NOT NULL,
Name_en VARCHAR(64) NOT NULL,
Primary Key (Code)
)`
const result = await dbQ(query, "Created Table COUNTRY")
return result
}
If you want each x.create to fully execute before the next one starts, i.e. this is what I interpret where you say while awaiting each .map call to finish before starting the next - then you could use async/await with a for loop as follows:
const createDatabase = async fileNumbers => {
fileNumbers.sort((a, b) => a - b);
for (let fileNumber of fileNumbers) {
const x = require(`.${root}${fileNumber.toString()}.js`);
await x.create();
})
}
However, this also assumes that x.create() returns a Promise - as you've not shown what is the typical content of .${root}${fileNumber.toString()}.js file is, then I'm only speculating
The other interpretation of your question would simply require you to change createDatabase so that promises is actually an array of promises (at the moment, it's an array of undefined
const createDatabase = async fileNumbers => {
fileNumbers.sort((a, b) => a - b);
const promises = fileNumbers.map(fileNumber => {
const x = require(`.${root}${fileNumber.toString()}.js`);
return x.create();
})
await Promise.all(promises);
}
Now all .create() run in "parallel", but createDatabase only resolves onces all promises returned by x.create() resolve - again, assumes that x.create() returns a promise
I know that promise.all() fails when even 1 of the promise is failed. I only want to try for failed promises and don't want to run promise.all() again.
Any recommendations on how I can achieve this in minimal way?
Promises are eager construct and model a value obtained asynchronously,
a Promise is produced using some kind of producer, like fetch for instance.
If you retain a reference to this producer then you can replay the nmechanism
that produced the Promise in the first place.
// producer function
function getData (arg) {
const result = new Promise();
return result.then(value => {
return { ok:true, value };
}, error => {
return {
ok: false,
value: error,
// retry is a function which calls the producer with the same arguments
retry: () => getData(arg)
};
})
}
Then if you have something like:
const data = [];
// Promise<{ok: boolean, value: any, retry?: function}>
// No promises will fail in this array
const asyncResults = data.map(getResults);
Promise.all(asyncResults)
.then((results) => {
const successes = results.filter(res => res.ok);
const retrys = results.filter(res => !res.ok).map(res => res.retry()); // retry all failed promises
})
Memory leaks, stack overflow: because I retain a reference to original arguments in order to retry and the algorithm is recursive there could be a memory leak. However the algorithm cannot "stack overflow":
getData calls do not get "deeper" over time (see retry definition)
the asyncrhonicity of the algorithm prevent this behaviour if a promise was never resolved
old data is properly discarded when accessing the results as const resultData = results.filter(res => res.ok).map(res => res.value);
However the algorithm could take a long time to settle if a promise keep on not getting resolved and prevent access to the rest of the values.
In an alternative I suggest you take a look at another async primitive, not yet part of the language (maybe some day) : Observables which are designed for this kind of tasks: lazy, retry-able async operations.
You may use async package and wrap all promise calls with closure with done argument.
Then simply resolve results.
const async = require('async');
const promiseAll = promises => {
return new Promise((resolve) => {
// preparing array of functions which has done method as callback
const parallelCalls = promises.map(promise => {
return done => {
promise
.then(result => done(null, result)
.catch(error => {
console.error(error.message);
done();
});
}
});
// calling array of functions in parallel
async.parallel(
parallelCalls,
(_, results) => resolve(results.filter(Boolean))
);
});
};
router.get('/something', async (req, res) => {
...
const results = await promiseAll(promises);
...
});
Or we can simply do Promise.all without using async package:
router.get('/something', async (req, res) => {
...
const results = (
await Promise.all(
promises.map(promise => {
return new Promise(resolve => {
promise.then(resolve).catch(e => resolve());
});
});
)
).filter(Boolean);
...
});