I'm a bit new to Node.js. I've run into a problem where I want to prevent a callback from running while it is already being executed. For example:
items.forEach(function(item) {
doLongTask(item, function handler(result) {
// If items.length > 1, this will get executed multiple times.
});
});
How do I make the other invocations of handler wait for the first one to finish before going ahead? I'm thinking something along the lines of a queue, but I'm a newbie to Node.js so I'm not exactly sure what to do. Ideas?
There are already libraries which take care of that, the most used being async.
You will be interested in the async.eachSeries() function.
As for an actual example...
const async = require('async')
async.eachSeries(
items,
(item, next) => {
// Do stuff with item, and when you are done, call next
// ...
next()
},
err => {
// either there was an error in one of the handlers and
// execution was stopped, or all items have been processed
}
)
As for how the library does this, you are better of having a look at the source code.
It should be noted that this only ever makes sense if your item handler ever performs an asynchronous operation, like interfacing with the filesystem or with internet etc. There exists no operation in Node.js that would cause a piece of JS code to be executed in parallel to another JS code within the same process. So, if all you do is some calculations, you don't need to worry about this at all.
How to prevent two callbacks from running simultaneously?
They won't run simultaneously unless they're asynchronous, because Node runs JavaScript on a single thread. Asynchronous operations can overlap, but the JavaScript thread will only ever be doing one thing at a time.
So presumably doLongTask is asynchronous. You can't use forEach for what you'd like to do, but it's still not hard: You just keep track of where you are in the list, and wait to start processing the next until the previous one completes:
var n = 0;
processItem();
function processItem() {
if (n < items.length) {
doLongTask(items[n], function handler(result) {
++n;
processItem();
});
}
}
Related
I want to preface by saying I've viewed a lot of stackoverflow questions regarding this topic, but I haven't found any 'duplicates' per se since none of them contain solutions that would solve this specific case.
I've mainly looked at How do I return the response from an asynchronous call?, and the section on 'Promises with async/await' would work inside an asynchronous function, but the function I'm working on is not async and is part of an existing codebase that I can't change easily. I wouldn't be able to change the function to async.
The section on callbacks wouldn't work either, which is explained further below. Anyway, I'll jump into my question:
I'm editing a function (standard function, not async) in JavaScript by adding an asynchronous function call. I want to wait until the async call finishes before I return from the function (since the result of the async call needs to be included in the return value). How would I do so?
I looked into using callbacks, which would allow me to write code which is guaranteed to run only after the async call completes. However, this wouldn't interrupt the flow of the program in the original function, and the original function could still return before the callback is run. A callback would allow me to execute something sequentially after the async function, but it wouldn't allow me to wait for asynchronous call to complete at the highest level.
Example code, which wouldn't return the desired result:
function getPlayers() {
... other code ...
let outfieldPlayers = asyncGetOutfieldPlayersCall()
... other code ...
allPlayers.add(outfieldPlayers)
return allPlayers // the returned value may or may not include outfield players
}
The actual problem I'm facing is a little more complicated - I'm calling the async function in each iteration of a for loop, and need to wait until all calls have completed before returning. But, I think if I can solve this simpler problem, I can solve the problem with a for loop.
Sadly, it is pretty much impossible to wait for async code in a synchronous way. This is because there is no threading in JS (most JS runtimes, but some are). So code is either synchronous or asynchronous.
Asynchronous code is possible because of the event loop. The event loop is part of the javascript runtime. It works by keeping a stack of callback functions that run when events trigger them - usually either timeout events (which you can set with setTimeout()) or IO events (which happen when you make disk or HTTP requests, or on user interaction). However, these callbacks only run when no other code is running, so only when the program is idle and all functions have returned.
This means that techniques like "spin loops" (where you just run a loop until a condition is changed by another thread) that work in threaded environments don't work because the async code won't run until the spin loop finishes.
More Info: https://medium.com/front-end-weekly/javascript-event-loop-explained-4cd26af121d4
If you are using NodeJS, this is possible through execSync.
This requires you to place your asynchronous code in a separate file, spawn a separate process using execSync, which will wait until it exits.
For example, consider the following async function which prints an array.
// task.js
(async function() {
await new Promise((resolve) => setTimeout(() => {
console.log(JSON.stringify([3,4,5]));
resolve();
}, 1000));
})();
Now, you can invoke this from your main process:
function asyncGetOutfieldPlayersCall() {
const execSync = require('child_process').execSync;
return JSON.parse(execSync("node task.js"));
}
function getPlayers() {
let allPlayers = [1,2];
// ... other code ...
let outfieldPlayers = asyncGetOutfieldPlayersCall();
// ... other code ...
allPlayers = allPlayers.concat(outfieldPlayers)
return allPlayers;
}
I am working with this while loop and it is not working. I decided to use the Google Chrome debugger and I saw that the code inside is not being executed.
All the time it checks the condition, starts the first line of the code inside, and goes back again to check the condition.
It is a NodeJS server and I am using the Spotify API.
app.get('/process', ensureAuthenticated, function (req, res) {
let init_array = text.split(" ");
let modtext = init_array;
while (init_array.length != 0) {
spotifyApi.searchTracks(modtext.join(" "))
.then(function (data) {
try {
console.log(data.body.tracks.items[0].name);
for (let i = 0; i < modtext.length; i++) {
init_array.shift();
}
modtext = init_array;
} catch (err) {
console.log("No song");
modtext.pop();
}
});
}
res.redirect('/');
});
This question is best understood by understanding how node.js uses an event loop. At its core, node.js runs your Javascript in a single thread and it uses an event loop in order to manage the completion of things outside that single thread such as timers, network operations, file operations, etc...
Let's first start with a very simple while() loop:
let done = false;
setTimeout(() => {
done = true;
}, 100);
while(!done) {
// do whatever you want here
}
console.log("done with while loop"); // never gets called
At first blush, you would think that the while loop would run for 100ms and then done would be set to true and the while loop would exit. That is not what happens. In fact, this is an infinite while loop. It runs and runs and runs and the variable done is never set to true. The console.log() at the end never runs.
It has this issue because setTimeout() is an asynchronous operation and it communicates its completion through the event loop. But, as we described above, node.js runs its Javascript as single threaded and only gets the next event from the event loop when that single thread finishes what it's doing. But, the while can't finish what it's doing until done gets set to true, but done can't get set to true until the while loop finishes. It's a stand-off and the while loop just runs forever.
So, in a nutshell, while any sort of loop is running, NO asynchronous operation ever gets its result processed (unless it's using await inside the loop which is something different). Asynchronous operations (or anything that uses the event loop) has to wait until the current running Javascript is done and then the interpreter can go back to the event loop.
Your while loop has the exact same issue. spotifyApi.searchTracks() is an asynchronous operation that returns a promise and all promises communicate their results via the event queue. So, you have the same standoff. Your .then() handler can't get called until the while loop finishes, but your while loop can't finish until the .then() handler gets called. Your while loop will just loop infinitely until you exhaust some system resource and your .then() handlers never get a chance to execute.
Since you haven't included code in your request handler that actually produces some result or action (all it appears to do is just modify some local variables), it's not obvious what exactly you're trying to accomplish and thus how to better write this code.
You appear to have N searches to do and you're logging something in each search. You could do them all in parallel and just use Promise.all() to track when they are all done (no while loop at all). Or, you can sequence them so you run one, get its result, then run another. Your question doesn't give us enough info to know what the best option would be.
Here's one possible solution:
Sequence the operations using async/await
Here the request handler is declared async so we can use await inside the while loop. That will suspend the while loop and allow other events to process while waiting for the promise to resolve.
app.get('/process', ensureAuthenticated, async function (req, res) {
let init_array = text.split(" ");
let modtext = init_array;
while (init_array.length != 0) {
try {
let data = await spotifyApi.searchTracks(modtext.join(" "));
console.log(data.body.tracks.items[0].name);
for (let i = 0; i < modtext.length; i++) {
init_array.shift();
}
modtext = init_array;
} catch (err) {
console.log("No song");
modtext.pop();
}
}
res.redirect('/');
});
The reason you're only seeing one line execute is because it's asynchronous. When you call an asynchronous function, it returns immediately and continues to do its work in the background. Once it's done, it calls another function (a "callback") and passes the results of the function to that. That's why your code has to go inside of a call to then() rather than just being on the next line of code.
In this case, spotifyApi.searchTracks() returns a Promise. Once the Spotify API has completed the search, the function in then() will run.
Lets use async/await to solve this problem, I have no clue what data you get in a text but I think it is good example to understand a concept of asynchronous processing.
app.get('/process', ensureAuthenticated, async function (req, res, next) {
try {
const modtext = text.split(" ");
const promises = modtext.map(item => spotify.searchTracks(item));
const response = await Promise.all(promises);
response.forEach(data => {
// if you use lodash, simply use a get function `get(data, 'body.tracks.items[0].name')` => no need to check existence of inner attributes
// or use something like this
if (data && data.body && data.body.track && data.body.tracks.items && data.body.tracks.items[0]) {
console.log(data.body.tracks.items[0].name);
} else {
console.log('No song');
}
});
res.redirect('/');
} catch(err) {
next(err)
}
});
For me much simpler and cleaner code, again, i dont know what is structure of your text attribute and logic behind it, so maybe you will have to make some changes.
Firstly I am familiar with the concept of asynchronous/synchronous function.
There is also a lot of questions related to mine. But I can't find my answer anywhere.
So the question is:
Is there a way to return a value instead of a Promise using async/await ? As a synchronous function do.
For example:
async doStuff(param) {
return await new Promise((resolve, reject) => {
setTimeout(() => {
console.log('doStuff after a while.');
resolve('mystuffisdone'+param);
}, 2000);
});
}
console.log(doStuff('1'));
The only way to get the value of this function is by using the .then function.
doStuff('1').then(response => {
console.log(response); // output: mystuffisdone1
doOtherStuffWithMyResponse(response);
// ...
});
Now, what I want is:
const one = doStuff('1');
console.log(one) // mystuffisdone1
const two = doStuff('2');
console.log(two) // mystuffisdone2
To explain myself, I have an asynchronous library full of callbacks. I can turn this asynchronous behavior to a synchronous behavior by using Promises and async/await to faking a synchronous behavior.
But there is still a problem, it is still asynchronous in the end; outside of the scope of the async function.
doStuff('1').then((r) => {console.log(r)};
console.log('Hello wolrd');
It will result in: Hello world then mystuffisdone1. This is the expected behavior when using async/await functions. But that's not what I want.
Now my question would be: Is there a way to do the same thing as await do without the keyword async ? To make the code being synchronous ? And if not possible, why ?
Edit:
Thank you for all you answers, I think my question is not obsvious for all. To clear up what I think here is my comment to #Nikita Isaev answer.
"I understand why all I/O operations are asynchronously done; or done in parallel. But my question is more about the fact that why the engine doesn't block the caller of the sync function in an asynchronous manner ? I mean const a = doStuff(...) is a Promise. We need to call .then to get the result of this function. But why JavaScript or Node engine does not block the caller (just the block where the call is made). If this is possible, we could do const a = doStuff(...), wait and get the result in a without blocking the main thread. As async/await does, why there is no place for sync/wait ?"
Hope this is more clear now, feel free to comment or ask anything :)
Edit 2:
All precisions of the why of the answer are in the comments of the accepted answer.
There are some hacky ways to do what is desired, but that would be an anti-pattern. I’ll try to explain. Callbacks is one of the core concepts in javascript. When your code launches, you may set up event listeners, timers, etc. You just tell the engine to schedule some tasks: “when A happens, do B”. This is what asynchrony is. But callbacks are ugly and difficult to debug, that’s why promises and async-await were introduced. It is important to understand that this is just a syntax sugar, your code still is asynchronous when using async-await. As there are no threads in javascript, waiting for some events to fire or some complicated operations to finish in a synchronous way would block your entire application. The UI or the server would just stop responding to any other user interactions and would keep waiting for a single event to fire.
Real world cases:
Example 1.
Let’s say we have a web UI. We have a button that downloads the latest information from the server on click. Imagine we do it synchronously. What happens?
myButton.onclick = function () {
const data = loadSomeDataSync(); // 0
useDataSomehow(data);
}
Everything’s synchronous, the code is flat and we are happy. But the user is not.
A javascript process can only ever execute a single line of code in a particular moment. User will not be able to click other buttons, see any animations etc, the app is stuck waiting for loadSomeDataSync() to finish. Even if this lasts 3 seconds, it’s a terrible user experience, you can neither cancel nor see the progress nor do something else.
Example 2.
We have a node.js http server which has over 1 million users. For each user, we need to execute a heavy operation that lasts 5 seconds and return the result. We can do it in a synchronous or asynchronous manner. What happens if we do it in async?
User 1 connects
We start execution of heavy operation for user 1
User 2 connects
We return data for user 1
We start execution of heavy operation for user 2
…
I.e we do everything in parallel and asap. Now imagine we do the heavy operation in a sync manner.
User 1 connects
We start execution of heavy operation for user 1, everyone else is waiting for it to accomplish
We return data for user 1
User 2 connects
…
Now imagine the heavy operation takes 5 seconds to accomplish, and our server is under high load, it has over 1 million users. The last one will have to wait for nearly 5 million seconds, which is definitely not ok.
That’s why:
In browser and server API, most of the i/o operations are asynchronous
Developers strive to make all heavy calculation asynchronous, even React renders in an asynchronous manner.
No, going from promise to async/await will not get you from async code to sync code. Why? Because both are just different wrapping for the same thing. Async function returns immediately just like a promise does.
You would need to prevent the Event Loop from going to next call. Simple while(!isMyPromiseResolved){} will not work either because it will also block callback from promises so the isMyPromiseResolved flag will never be set.
BUT... There are ways to achieve what you have described without async/await. For example:
OPTION 1: using deasync approach. Example:
function runSync(value) {
let isDone = false;
let result = null;
runAsync(value)
.then(res => {
result = res;
isDone = true;
})
.catch(err => {
result = err;
isDone = true;
})
//magic happens here
require('deasync').loopWhile(function(){return !isDone;});
return result;
}
runAsync = (value) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
// if passed value is 1 then it is a success
if(value == 1){
resolve('**success**');
}else if (value == 2){
reject('**error**');
}
}, 1000);
});
}
console.log('runSync(2): ', runSync(2));
console.log('runSync(1): ', runSync(1));
OR
OPTION 2: calling execFileSync('node yourScript.js') Example:
const {execFileSync} = require('child_process');
execFileSync('node',['yourScript.js']);
Both approaches will block the user thread so they should be used only for automation scripts or similar purposes.
Wrap the outer body in an asynchronous IIFE:
/**/(async()=>{
function doStuff(param) { // no need for this one to be async
return new Promise((resolve, reject) => { // just return the original promise
setTimeout(() => {
console.log('doStuff after a while.');
resolve('mystuffisdone'+param);
}, 2000);
});
}
console.log(await doStuff('1')); // and await instead of .then
/**/})().then(()=>{}).catch(e=>console.log(e))
The extra cleanup of the doStuff function isn't strictly necessary -- it works either way -- but I hope it helps clarify how async functions and Promises are related. The important part is to wrap the outer body into an async function to get the improved semantics throughout your program.
It's also not strictly necessary to have the final .then and .catch, but it's good practice. Otherwise, your errors might get swallowed, and any code ported to Node will whine about uncaught Promise rejections.
I have a node application that is not a web application - it completes a series of asynchronous tasks before returning 1. Immediately before returning, the results of the program are printed to the console.
How do I make sure all the asynchronous work is completed before returning? I was able to achieve something similar to this in a web application by making sure all tasks we completed before calling res.end(), but I haven't any equivalent for a final 'event' to call before letting a script return.
See below for my (broken) function currently, attempting to wait until callStack is empty. I just discovered that this is a kind of nonsensical approach because node waits for processHub to complete before entering any of the asynchronous functions called in processObjWithRef.
function processHub(hubFileContents){
var callStack = [];
var myNewObj = {};
processObjWithRef(samplePayload, myNewObj, callStack);
while(callStack.length>0){
//do nothing
}
return 1
}
Note: I have tried many times previously to achieve this kind of behavior with libraries like async (see my related question at How can I make this call to request in nodejs synchronous?) so please take the answer and comments there into account before suggesting any answers based on 'just use asynch'.
You cannot wait for an asynchronous event before returning--that's the definition of asynchronous! Trying to force Node into this programming style will only cause you pain. A naive example would be to check periodically to see if callstack is empty.
var callstack = [...];
function processHub(contents) {
doSomethingAsync(..., callstack);
}
// check every second to see if callstack is empty
var interval = setInterval(function() {
if (callstack.length == 0) {
clearInterval(interval);
doSomething()
}
}, 1000);
Instead, the usual way to do async stuff in Node is to implement a callback to your function.
function processHub(hubFileContents, callback){
var callStack = [];
var myNewObj = {};
processObjWithRef(samplePayload, myNewObj, callStack, function() {
if (callStack.length == 0) {
callback(some_results);
}
});
}
If you really want to return something, check out promises; they are guaranteed to emit an event either immediately or at some point in the future when they are resolved.
function processHub(hubFileContents){
var callStack = [];
var myNewObj = {};
var promise = new Promise();
// assuming processObjWithRef takes a callback
processObjWithRef(samplePayload, myNewObj, callStack, function() {
if (callStack.length == 0) {
promise.resolve(some_results);
}
});
return promise;
}
processHubPromise = processHub(...);
processHubPromise.then(function(result) {
// do something with 'result' when complete
});
The problem is with your design of the function. You want to return a synchronous result from a list of tasks that are executed asynchronously.
You should implement your function with an extra parameter that will be the callback where you would put the result (in this case, 1) for some consumer to do something with it.
Also you need to have a callback parameter in your inner function, otherwise you won't know when it ends. If this last thing is not possible, then you should do some kind of polling (using setInterval perhaps) to test when the callStack array is populated.
Remember, in Javascript you should never ever do a busy wait. That will lock your program entirely as it runs on a single process.
deasync is desinged to address your problem exactly. Just replace
while(callStack.length>0){
//do nothing
}
with
require('deasync').loopWhile(function(){return callStack.length>0;});
The problem is that node.js is single-threaded, which means that if one function runs, nothing else runs (event-loop) until that function has returned. So you can not block a function to make it return after async stuff is done.
You could, for example, set up a counter variable that counts started async tasks and decrement that counter using a callback function (that gets called after the task has finished) from your async code.
Node.js runs on A SINGLE threaded event loop and leverages asynchronous calls for doing various things, like I/O operations.
if you need to wait for a number of asynchronous operations to finish before executing additional code
you can try using Async -
Node.js Async Tutorial
You'll need to start designing and thinking asynchronously, which can take a little while to get used to at first. This is a simple example of how you would tackle something like "returning" after a function call.
function doStuff(param, cb) {
//do something
var newData = param;
//"return"
cb(newData);
}
doStuff({some:data}, function(myNewData) {
//you're done with doStuff in here
});
There's also a lot of helpful utility functions in the async library available on npm.
I'd like to be able to dispatch a bunch of work via JavaScript to be done in the browser in such a way that the browser stays responsive throughout.
The approach I'm trying to take is to chunk up the work, passing each chunk to a function that is then queued with a setTimeout(func, 0) call.
I need to know when all the work is done, so I'm storing the returned timer ID in a map (id -> true|false). This mapping is set to false in the next block of code after I have the timer ID, and the queued function sets the mapping to true when it completes... except, of course, the queued function doesn't know its timer ID.
Maybe there's a better/easier way... or some advice on how I can manipulate my map as I need to?
I would queue the work in an array, use one timeout to process the queue and call a callback once the queue is empty. Something like:
var work = [...];
var run = function(work, callback) {
setTimeout(function() {
if(work.length > 0) {
process(work.shift());
setTimeout(arguments.callee, 25);
}
else {
callback();
}
}, 25);
};
run(work, function() {
alert('Work is done!');
});
As JavaScript in browsers is single threaded there is no real advantage to run multiple timeouts (at least I think this is what you are doing). It may even slow down the browser.
I'd like to add that although javascript is single threaded you can still have multiple ajax calls going at once. I recently had a site that needed to do potentially hundreds of ajax calls and the browser just couldn't handle it. I created a queue that used setTimeOut to run 5 calls at once. When one of the ajax calls returned it fired a callback (which is handled by a single thread) and then made the next call on the stack.
Imagine you're a manager that can only talk to one person at a time, you give 5 employees assignments, then wait for their responses, which may come in any order. Once the first employee comes back and gives you the information, you give them a new assignment and wait for the next employee (or perhaps even the same employee) to come back. So although you're "single threaded" 5 things are going on at once.
There is an example right in the HTML Standard, how it is best to handle it:
To run tasks of several milliseconds back to back without any delay,
while still yielding back to the browser to avoid starving the user
interface (and to avoid the browser killing the script for hogging the
CPU), simply queue the next timer before performing work:
function doExpensiveWork() {
var done = false;
// ...
// this part of the function takes up to five milliseconds
// set done to true if we're done
// ...
return done;
}
function rescheduleWork() {
var handle = setTimeout(rescheduleWork, 0); // preschedule next iteration
if (doExpensiveWork())
clearTimeout(handle); // clear the timeout if we don't need it
}
function scheduleWork() {
setTimeout(rescheduleWork, 0);
}
scheduleWork(); // queues a task to do lots of work
The moment of finishing the work is pretty clear, when clearTimeout is called.