I just wanted to know if it is considered good practice to nest promises like in this example, or is there better alternatives ?
getDatabaseModel.searchId(idNoms).then(function([noms, elements]) {
getDatabaseModel.getLocalisations().then(function(localisations){
getDatabaseModel.getStates().then(function(states){
//Some code
})
})
})
Obviously, your promises are independent. So you should use Promise.all() to make it run parallel with the highest performance.
The Promise.all() method takes an iterable of promises as an input,
and returns a single Promise that resolves to an array of the results
of the input promises
var searchById = getDatabaseModel.searchId(idNoms);
var getLocalisations = getDatabaseModel.getLocalisations();
var getStates = getDatabaseModel.getStates();
var result = Promise.all([searchById, getLocalisations, getStates]);
result.then((values) => {
console.log(values);
});
For example, Let's say each promise takes 1s - So it should be 3s in total, But with Promise.all, actually it just takes 1s in total.
var tick = Date.now();
const log = (v) => console.log(`${v} \n Elapsed: ${Date.now() - tick}`);
log("Staring... ");
var fetchData = (name, ms) => new Promise(resolve => setTimeout(() => resolve(name), ms));
var result = Promise.all(
[
fetchData("searchById", 1000),
fetchData("getLocalisations", 1000),
fetchData("getStates", 1000)
]);
result.then((values) => {
log("Complete...");
console.log(values);
});
Besides, If you're concern about asyn/await with more elegant/concise/read it like sync code, then await keyword much says the code should wait until the async request is finished and then afterward it'll execute the next thing. While those promises are independent. So promise.all is better than in your case.
var tick = Date.now();
const log = (v) => console.log(`${v} \n Elapsed: ${Date.now() - tick}`);
var fetchData = (name, ms) => new Promise(resolve => setTimeout(() => resolve(name), ms));
Run();
async function Run(){
log("Staring... ");
var a = await fetchData("searchById", 1000);
var b = await fetchData("getLocalisations", 1000);
var c = await fetchData("getStates", 1000);
log("Complete...");
console.log([a, b, c]);
}
Promises were made to avoid callback hell, but they are not too good at it too. People like promises until they find async/await. The exact same code can be re-written in async/await as
async getModel(idNoms){
const [noms, elements] = await getDatabaseModel.searchId(idNoms);
const localisations = await getDatabaseModel.getLocalisations();
const state = await getDatabaseModel.getStates():
// do something using localisations & state, it'll work
}
getModel(idNoms);
Learn async/await here
IMO it's a little hard to read and understand. Compare with this:
getDatabaseModel.searchId(idNoms)
.then(([noms, elements]) => getDatabaseModel.getLocalisations())
.then(localization => getDatabaseModel.getStates());
As #deceze pointed out there are two things to note:
These functions are called serially
They don't seem to depend on each other as the noms, elements and localization are not used at all.
With Promise.all you can mix and match however you want:
// Call `searchId` and `getState` at the same time
// Call `getLocalisations` after `searchId` is done
// wait for all to finish
Promise.all([
getDatabaseModel.searchId(idNoms).then(([noms, elements]) => getDatabaseModel.getLocalisations()),
getDatabaseModel.getStates()
]).then(([result1, result2]) => console.log('done'));
// Call all 3 at the same time
// wait for all to finish
Promise.all([
getDatabaseModel.searchId(idNoms),
getDatabaseModel.getLocalisations(),
getDatabaseModel.getStates(),
]).then(([result1, result2, result3]) => console.log('done'));
There is a huge performance hit when I use await keyword for multiple API calls,
In the below example suppose persons array have 50 objects therefore it make 50 API calls (for..of loop) to get transaction details of a person (one api call) one at a time and do operation on it.
for (const person of persons) {
const trx = await transactionHistory(person.id); // method which trigger single API call (https endpoint)
// do some business operation on trx object
}
It takes around 7+ seconds to execute everything.
But when I tweak a little bit by making all the API call in foreach loop and assign it to a property and later use it, in await keyword. Then it takes roughly less than a second to execute everything.
persons.forEach(person => {
person.trxHistory = transactionHistory(person.id); // method which trigger single API call (https endpoint)
});
for (const person of persons) {
const trx = await person.trxHistory
// do some business operation on trx object
}
I am a little confused and unable to understand how it improved the performance with a big difference. Can anyone explain the difference between the two logic? I am a beginner level :)
In the second snippet, you are triggering the operations one after another asynchronously in the forEach before they are awaited. So it happens in the background. The result of the transactionHistory(person.id) is a Promise which is then assigned to the person.trxHistory.
Since the operations were already started in the background, when you ultimately come to the for-of loop to await on them they are already in-progress and you need to wait fewer time to get the result.
But in the earlier for-of loop in your first snippet, you were triggering the operations one by one:
Start the operation
Wait for it to finish
This was the sequence in your first example. Hence this was slow as it was sequential, whereas the latter one was concurrent.
persons.forEach(() => {
//operation already started in the background in an async way
person.trxHistory = transactionHistory(person.id); // method which trigger single API call (https endpoint)
});
for (const person of persons) {
const trx = await person.trxHistory
// do some business operation on trx object
}
So when you eventually come to the for-of loop, you are only awaiting on the last result synchronously.
You can trigger all the operations inside a Promise.all call and await those results one by one:
Promise.all(persons.map(p => transactionHistory(p.id)).then(async res => {
const trx = await res
// do some business operation on trx object
});
To make it clear take this mock API for instance:
This first snippet is the concurrent one, where all tasks are triggered before hand and then awaited so total time is the max time for each individual task (approx 7 secs):
const mockAPICall = (time) => {
return new Promise((res, rej) => {
console.log("Started execution");
setTimeout(() => res("Mocked return value"), time);
});
}
const waits = [{wait: 1000}, {wait: 2000}, {wait: 7000}]
waits.forEach(w => {
w.result = mockAPICall(w.wait);
});
const start = performance.now();
(async () => {
for (const w of waits){
let result = await w.result;
console.log(result);
}
console.log(performance.now() - start)
})();
This snippet is sequential, waits for each task to complete one-by-one so total time is the sum of all the waiting periods (approx 10 secs):
const mockAPICall = (time) => {
return new Promise((res, rej) => {
console.log("Started execution");
setTimeout(() => res("Mocked return value"), time);
});
}
const waits = [{wait: 1000}, {wait: 2000}, {wait: 7000}]
const start = performance.now();
(async () => {
for (const w of waits){
let result = await mockAPICall(w.wait);
console.log(result);
}
console.log(performance.now() - start);
})();
I’ve nodes program which I need to run two function in the beginning of the program
And later on access the function results, currently with await each function at a time this works,
However in order to save a time and not waiting to GetService and GetProcess as I need the data later on in the project
It takes about 4 seconds to get this data and I want to run it on the background as I don’t need the results immediately,
How I can do it in node js, If I run promise.all It would wait until the getService and getProcess and then go to rest of the program.
an example
function main() {
//I want to run this both function in background to save time
let service = await GetServices();
this.process = await GetProcess();
…..//Here additional code is running
//let say that after 30 second this code is called
Let users = GetUser(service);
Let users = GetAdress(this.process);
}
im actually running yeoman generator
https://yeoman.io/authoring/
https://yeoman.io/authoring/user-interactions.html
export default class myGenerator extends Generator {
//here I want run those function in background to save time as the prompt to the user takes some time (lets say user have many questions...)
async initializing() {
let service = await GetServices();
this.process = await GetProcess();
}
async prompting() {
const answers = await this.prompt([
{
type: "input",
name: "name",
message: "Your project name",
default: this.appname // Default to current folder name
},
{
type: "confirm",
name: "list",
choises: this.process //here I need to data from the function running in background
}
]);
}
Let's assume that getServices() may take 3 seconds and getProcess() may take 4 seconds, so if you run these both functions at the same time you will be returned in total 4 seconds with the return values from both promises.
You can execute the code while this process is running in the background there will be a callback when the promises resolved, your late functions will be called at this stage.
Check the below simple example;
let service;
let process;
function main() {
// Both functions will execute in background
Promise.all([getServices(), getProcess()]).then((val) => {
service = val[0];
process = val[1];
console.log(service, process);
// Aafter completed this code will be called
// let users = GetUser(service);
// let users = GetAdress(process);
console.log('I am called after all promises completed.')
});
// Current example.
// let service = await GetServices();
// this.process = await GetProcess();
/* Code blocks.. */
console.log('Code will execute without delay...')
}
function getServices() {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve("service is returned")
}, 3000);
});
}
function getProcess() {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve("process is returned")
}, 4000);
});
}
main();
You can start the asynchronous operation but not await it yet:
function suppressUnhandledRejections(p) {
p.catch(() => {});
return p;
}
async function main() {
// We have to suppress unhandled rejections on these promises. If they become
// rejected before we await them later, we'd get a warning otherwise.
const servicePromise = suppressUnhandledRejections(GetServices());
this.processPromise = suppressUnhandledRejections(GetProcess());
// Do other stuff
const service = await servicePromise;
const process = await this.processPromise;
}
Also consider using Promise.all() which returns a promise for the completion of all promises passed to it.
async function main() {
const [ services, process, somethingElse ] = await Promise.all([
GetServices(),
GetProcess(),
SomeOtherAsyncOperation(),
]);
// Use the results.
}
To do what who you need, you have to understand the event loop.
Nodejs is designed to work in a single thread unlike languages like go, however nodejs handle proccess on different threads. so you can use nextTick () to add a new event to the main thread and it will be executed at the end of the whole block.
function main() {
//I want to run this both function in background to save time
let service = await GetServices();
this.process = await GetProcess();
…..//Here additional code is running
//Let say that after 30 second this code is called
Let users = GetUser(service);
Let users = GetAdr(this.process);
}
function someFunction(){
// do something...
}
main();
process.nextTick(someFunction());// happens after all main () processes are terminated...
I have the following code
module.exports = async function (req, res) {
const station1 = await getStation('one')
const station2 = await getStation('two')
return { stations: [station1, station2] }
}
Can I be guaranteed that when the final return value is sent it will definitely have both station1 and station2 data in them, or do I need to wrap the function call in a Promise.all()
As you have it, it is guaranteed that the return statement will only be executed when the two getStation() promises have resolved.
However, the second call to getStation will only happen when the first promise has resolved, making them run in serial. As there is no dependency between them, you could gain performance, if you would run them in parallel.
Although this can be achieved with Promise.all, you can achieve the same by first retrieving the two promises and only then performing the await on them:
module.exports = async function (req, res) {
const promise1 = getStation('one');
const promise2 = getStation('two');
return { stations: [await promise1, await promise2] }
}
Now both calls will be performed at the "same" time, and it will be just the return statement that will be pending for both promises to resolve. This is also illustrated in MDN's "simple example".
The await keyword actually makes you "wait" on the line of code, while running an async action.
That means that you don't proceed to the next line of code until the async action is resolved. This is good if your code has a dependency with the result.
Example:
const res1 = await doSomething();
if(res1.isValid)
{
console.log('do something with res1 result');
}
The following code example will await a promise that gets resolved after three seconds. Check the date prints to the console to understand what await does:
async function f1() {
console.log(new Date());
// Resolve after three seconds
var p = new Promise(resolve => {
setTimeout(() => resolve({}),3000);
});
await p;
console.log(new Date());
}
f1();
ES6Console
BTW, In your case, since you don't use the result of station1 it's better using Promise.all to work parallel.
Check this example (it will run for 3 seconds instead of 4 seconds the way you coded above):
async function f1() {
console.log(new Date());
// Resolve after three seconds
var p1 = new Promise(resolve => {
setTimeout(() => resolve({a:1}),3000);
});
// Resolve after one second
var p2 = new Promise(resolve => {
setTimeout(() => resolve({a:2}),1000);
});
// Run them parallel - Max(three seconds, one second) -> three seconds.
var res = await Promise.all([p1,p2]);
console.log(new Date());
console.log('result:' + res);
}
f1();
ES6Console.
If either of await getStation('one') or await getStation('two') fails an exception will be thrown from the async function. So you should always get the resolved value from both promises.
You can rewrite your function as follows to use Promise.all
module.exports = async function (req, res) {
try{
const [station1, station2] = await Promise.all([
getStation('one'),
getStation('two')
]);
return { stations: [station1, station2] };
} catch (e) {
throw e;
}
}
I would like to clarify this point, as the documentation is not too clear about it;
Q1: Is Promise.all(iterable) processing all promises sequentially or in parallel? Or, more specifically, is it the equivalent of running chained promises like
p1.then(p2).then(p3).then(p4).then(p5)....
or is it some other kind of algorithm where all p1, p2, p3, p4, p5, etc. are being called at the same time (in parallel) and results are returned as soon as all resolve (or one rejects)?
Q2: If Promise.all runs in parallel, is there a convenient way to run an iterable sequencially?
Note: I don't want to use Q, or Bluebird, but all native ES6 specs.
Is Promise.all(iterable) executing all promises?
No, promises cannot "be executed". They start their task when they are being created - they represent the results only - and you are executing everything in parallel even before passing them to Promise.all.
Promise.all does only await multiple promises. It doesn't care in what order they resolve, or whether the computations are running in parallel.
is there a convenient way to run an iterable sequencially?
If you already have your promises, you can't do much but Promise.all([p1, p2, p3, …]) (which does not have a notion of sequence). But if you do have an iterable of asynchronous functions, you can indeed run them sequentially. Basically you need to get from
[fn1, fn2, fn3, …]
to
fn1().then(fn2).then(fn3).then(…)
and the solution to do that is using Array::reduce:
iterable.reduce((p, fn) => p.then(fn), Promise.resolve())
In parallel
await Promise.all(items.map(async (item) => {
await fetchItem(item)
}))
Advantages: Faster. All iterations will be started even if one fails later on. However, it will "fail fast". Use Promise.allSettled, to complete all iterations in parallel even if some throw. Technically, these are concurrent invocations not in parallel.
In sequence
for (const item of items) {
await fetchItem(item)
}
Advantages: Variables in the loop can be shared by each iteration. Behaves like normal imperative synchronous code.
NodeJS does not run promises in parallel, it runs them concurrently since it’s a single-threaded event loop architecture. There is a possibility to run things in parallel by creating a new child process to take advantage of the multiple core CPU.
Parallel Vs Concurent
In fact, what Promise.all does is, stacking the promises function in the appropriate queue (see event loop architecture) running them concurrently (call P1, P2,...) then waiting for each result, then resolving the Promise.all with all the promises results.
Promise.all will fail at the first promise which fails unless you have to manage the rejection yourself.
There is a major difference between parallel and concurrent, the first one will run a different computation in a separate process at exactly the same time and they will progress at their rhythm, while the other one will execute the different computation one after another without waiting for the previous computation to finish and progress at the same time without depending on each other.
Finally, to answer your question, Promise.all will execute neither in parallel nor sequentially but concurrently.
Bergi's answer got me on the right track using Array.reduce.
However, to actually get the functions returning my promises to execute one after another I had to add some more nesting.
My real use case is an array of files that I need to transfer in order one after another due to limits downstream...
Here is what I ended up with:
getAllFiles().then( (files) => {
return files.reduce((p, theFile) => {
return p.then(() => {
return transferFile(theFile); //function returns a promise
});
}, Promise.resolve()).then(()=>{
console.log("All files transferred");
});
}).catch((error)=>{
console.log(error);
});
As previous answers suggest, using:
getAllFiles().then( (files) => {
return files.reduce((p, theFile) => {
return p.then(transferFile(theFile));
}, Promise.resolve()).then(()=>{
console.log("All files transferred");
});
}).catch((error)=>{
console.log(error);
});
didn't wait for the transfer to complete before starting another and also the "All files transferred" text came before even the first file transfer was started.
Not sure what I did wrong, but wanted to share what worked for me.
Edit: Since I wrote this post I now understand why the first version didn't work. then() expects a function returning a promise. So, you should pass in the function name without parentheses! Now, my function wants an argument so then I need to wrap in in a anonymous function taking no argument!
You can also process an iterable sequentially with an async function using a recursive function. For example, given an array a to process with asynchronous function someAsyncFunction():
var a = [1, 2, 3, 4, 5, 6]
function someAsyncFunction(n) {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log("someAsyncFunction: ", n)
resolve(n)
}, Math.random() * 1500)
})
}
//You can run each array sequentially with:
function sequential(arr, index = 0) {
if (index >= arr.length) return Promise.resolve()
return someAsyncFunction(arr[index])
.then(r => {
console.log("got value: ", r)
return sequential(arr, index + 1)
})
}
sequential(a).then(() => console.log("done"))
Just to elaborate on #Bergi's answer (which is very succinct, but tricky to understand ;)
This code will run each item in the array and add the next 'then chain' to the end:
function eachorder(prev,order) {
return prev.then(function() {
return get_order(order)
.then(check_order)
.then(update_order);
});
}
orderArray.reduce(eachorder,Promise.resolve());
Using async await an array of promises can easily be executed sequentially:
let a = [promise1, promise2, promise3];
async function func() {
for(let i=0; i<a.length; i++){
await a[i]();
}
}
func();
Note: In above implementation, if a promise is rejected, the rest wouldn't be executed.If you want all your promises to be executed, then wrap your await a[i](); inside try catch
parallel
see this example
const resolveAfterTimeout = async i => {
return new Promise(resolve => {
console.log("CALLED");
setTimeout(() => {
resolve("RESOLVED", i);
}, 5000);
});
};
const call = async () => {
const res = await Promise.all([
resolveAfterTimeout(1),
resolveAfterTimeout(2),
resolveAfterTimeout(3),
resolveAfterTimeout(4),
resolveAfterTimeout(5),
resolveAfterTimeout(6)
]);
console.log({ res });
};
call();
by running the code it'll console "CALLED" for all six promises and when they are resolved it will console every 6 responses after timeout at the same time
I stumbled across this page while trying to solve a problem in NodeJS: reassembly of file chunks. Basically:
I have an array of filenames.
I need to append all those files, in the correct order, to create one large file.
I must do this asynchronously.
Node's 'fs' module does provide appendFileSync but I didn't want to block the server during this operation. I wanted to use the fs.promises module and find a way to chain this stuff together. The examples on this page didn't quite work for me because I actually needed two operations: fsPromises.read() to read in the file chunk, and fsPromises.appendFile() to concat to the destination file. Maybe if I was better with JavaScript I could have made the previous answers work for me. ;-)
I stumbled across this and I was able to hack together a working solution:
/**
* sequentially append a list of files into a specified destination file
*/
exports.append_files = function (destinationFile, arrayOfFilenames) {
return arrayOfFilenames.reduce((previousPromise, currentFile) => {
return previousPromise.then(() => {
return fsPromises.readFile(currentFile).then(fileContents => {
return fsPromises.appendFile(destinationFile, fileContents);
});
});
}, Promise.resolve());
};
And here's a jasmine unit test for it:
const fsPromises = require('fs').promises;
const fsUtils = require( ... );
const TEMPDIR = 'temp';
describe("test append_files", function() {
it('append_files should work', async function(done) {
try {
// setup: create some files
await fsPromises.mkdir(TEMPDIR);
await fsPromises.writeFile(path.join(TEMPDIR, '1'), 'one');
await fsPromises.writeFile(path.join(TEMPDIR, '2'), 'two');
await fsPromises.writeFile(path.join(TEMPDIR, '3'), 'three');
await fsPromises.writeFile(path.join(TEMPDIR, '4'), 'four');
await fsPromises.writeFile(path.join(TEMPDIR, '5'), 'five');
const filenameArray = [];
for (var i=1; i < 6; i++) {
filenameArray.push(path.join(TEMPDIR, i.toString()));
}
const DESTFILE = path.join(TEMPDIR, 'final');
await fsUtils.append_files(DESTFILE, filenameArray);
// confirm "final" file exists
const fsStat = await fsPromises.stat(DESTFILE);
expect(fsStat.isFile()).toBeTruthy();
// confirm content of the "final" file
const expectedContent = new Buffer('onetwothreefourfive', 'utf8');
var fileContents = await fsPromises.readFile(DESTFILE);
expect(fileContents).toEqual(expectedContent);
done();
}
catch (err) {
fail(err);
}
finally {
}
});
});
You can do it by for loop.
async function return promise:
async function createClient(client) {
return await Client.create(client);
}
let clients = [client1, client2, client3];
if you write following code then client are created parallelly:
const createdClientsArray = yield Promise.all(clients.map((client) =>
createClient(client);
));
But if you want to create client sequentially then you should use for loop:
const createdClientsArray = [];
for(let i = 0; i < clients.length; i++) {
const createdClient = yield createClient(clients[i]);
createdClientsArray.push(createdClient);
}
Bergi's answer helped me to make the call synchronous. I have added an example below where we call each function after the previous function is called:
function func1 (param1) {
console.log("function1 : " + param1);
}
function func2 () {
console.log("function2");
}
function func3 (param2, param3) {
console.log("function3 : " + param2 + ", " + param3);
}
function func4 (param4) {
console.log("function4 : " + param4);
}
param4 = "Kate";
//adding 3 functions to array
a=[
()=>func1("Hi"),
()=>func2(),
()=>func3("Lindsay",param4)
];
//adding 4th function
a.push(()=>func4("dad"));
//below does func1().then(func2).then(func3).then(func4)
a.reduce((p, fn) => p.then(fn), Promise.resolve());
I've been using for of in order to solve sequential promises. I'm not sure if it helps here but this is what I've been doing.
async function run() {
for (let val of arr) {
const res = await someQuery(val)
console.log(val)
}
}
run().then().catch()
Yes, you can chain an array of promise returning functions as follows
(this passes the result of each function to the next). You could of course edit it to pass the same argument (or no arguments) to each function.
function tester1(a) {
return new Promise(function(done) {
setTimeout(function() {
done(a + 1);
}, 1000);
})
}
function tester2(a) {
return new Promise(function(done) {
setTimeout(function() {
done(a * 5);
}, 1000);
})
}
function promise_chain(args, list, results) {
return new Promise(function(done, errs) {
var fn = list.shift();
if (results === undefined) results = [];
if (typeof fn === 'function') {
fn(args).then(function(result) {
results.push(result);
console.log(result);
promise_chain(result, list, results).then(done);
}, errs);
} else {
done(results);
}
});
}
promise_chain(0, [tester1, tester2, tester1, tester2, tester2]).then(console.log.bind(console), console.error.bind(console));
see this sample
Promise.all working parallel
const { range, random, forEach, delay} = require("lodash");
const run = id => {
console.log(`Start Task ${id}`);
let prom = new Promise((resolve, reject) => {
delay(() => {
console.log(`Finish Task ${id}`);
resolve(id);
}, random(2000, 15000));
});
return prom;
}
const exec = () => {
let proms = [];
forEach(range(1,10), (id,index) => {
proms.push(run(id));
});
let allPromis = Promise.all(proms);
allPromis.then(
res => {
forEach(res, v => console.log(v));
}
);
}
exec();