While studying nodejs streams, I wanted to check what happens if the end listener is added after a filestream has been consumed and closed.
I am puzzled by what happens: no error event, no exception thrown, but the promise is not awaited and the node process is closed with return value 0:
const fs = require("node:fs");
const { setTimeout } = require("node:timers/promises");
run()
.then(() => console.log("done"))
.catch(console.error);
async function run() {
const myStream = fs.createReadStream(__filename);
myStream.on("data", () => console.log("stream data"));
myStream.on("error", (error) => console.log("stream error", error));
myStream.on("close", () => console.log("stream close"));
console.log("data listener added");
await setTimeout(2000);
await new Promise((resolve) => {
console.log("add end listener");
myStream.on("end", () => {
console.log("stream end");
resolve();
});
console.log("end listener added");
});
console.log("run end");
}
Output (the process stops immediately after printing this, it does not hang):
$ node stream1.js
data listener added
stream data
stream close
add end listener
end listener added
I get that "stream end" is not reached, not pleasant but I can work with it by first checking for stream.isClosed.
As a consequence it seems logical that "run end" is not reached: I expect the run function to wait for the promise forever.
But why the process does not hang?
Can some nodejs expert explain what is happening?
Did I make an error with my promise handling?
I am using node v18.12.1 on Linux.
When you add the end listener to your stream, it has already ended, therefore the listener will never be invoked. resolve() is never executed and the await new Promise(...) never comes to an end. Therefore, run end is not logged either. And the promise returned by run never resolves so that done is also not logged.
On the other hand, after end listener added has been logged, the event loop of Node.js is empty: there is nothing more to expect, because the file has been read. In such a situation, the Node.js process exits.
Just attaching a listener to an event does not make Node.js wait. Perhaps the following analogy helps:
http.createServer().on("request", express());
registers a request handler but then exists immediately. By contrast, the .listen method makes the HTTP server wait for incoming requests, even if they are ignored because no request handler is registered at all:
http.createServer().listen(8080);
never exits.
Also, pending promises do not make Node.js wait. Compare
new Promise(function(resolve, reject) {
setTimeout(resolve, 5000);
});
which exits after 5 seconds, to
new Promise(function(resolve, reject) {
});
which exists immediately. It is not the promise that makes Node.js wait, but the setTimeout.
You made no error, only a wrong expectation perhaps.
I'd like to understand better under what conditions nodejs stops a running process. My guess was that it stops the process when both the stack and the event loop are empty. But the following program only prints hello once, whereas I was expecting it to loop forever, printing hello every second.
(async () => {
while (true) {
await new Promise(resolve => setTimeout(() => console.log("hello"), 1000))
}
})();
How does the while (true) loop interact with the event loop?
You haven't mistaken how NodeJS works. Your code just has a bug: resolve never gets called.
If you change it to the following, "hello" prints forever at 1 second intervals:
(async () => {
while (true) {
await new Promise(resolve => setTimeout(() => {
console.log("hello")
resolve();
}, 1000))
}
})();
The reason your code would still end is because, in NodeJS, the resolve function falls out of scope, indicating to the V8 JS engine that the Promise can never resolve. Therefore it ends the async () => {...}, which in turn quits since it's the last function still running.
You need to call your resolve() method ,such that loop can proceed further,like below
(async() => {
while (true) {
const data = await new Promise(resolve => setTimeout(() => resolve("hello"), 1000))
console.log(data)
}
})();
You haven't resolved your promise.
Node can tell that there are no more event sources that might make something interesting happen (no more timers are scheduled) so it exits, even though there is an unresolved promise.
Compare this version, which will print hello forever, because the promise is resolved when the timeout completes and a new timeout is scheduled, thus ensuring that there is always something in the event queue, giving Node.js a reason to carry on running your program.
(async () => {
while (true) {
await new Promise(resolve => setTimeout(() => {
console.log("hello");
resolve();
}, 1000))
}
})();
If resolve is not called, then the promise is not resolved, and the await never completes and no new timeout is ever scheduled. The event queue empties and node decides that to go on doing nothing would be futile, so it exits.
See this example:
// CLOSEWORKER.JS:
self.postMessage('foo');
self.close();
self.postMessage('bar');
setTimeout(() => self.postMessage('baz'), 0);
// MAIN.JS:
const worker = new Worker('./worker.js');
worker.onmessage = ({data}) => console.log(data);
Even if I have invoked self.close(),the next line self.postMessage('bar'); still executed and log 'bar'.I can't understand that since the close method should just kill the worker thread right after being called.
Also ,see another example:
TERMINATEWORKER.JS:
self.onmessage = ({data}) => console.log(data);
MAIN.JS:
const worker = new Worker('./worker.js');
setTimeout(() => {
worker.postMessage('foo');
worker.terminate();
worker.postMessage('bar');
setTimeout(() => worker.postMessage('baz'), 0);
}, 1000)
When using the terminate method instead of the close method, the code executed as I expected. It just logs 'foo', no-log any other thing, which suggests that the thread has been killed right after calling the terminate method.
WorkerGlobalScope.close() when called discards any tasks that have been scheduled and sets the closing flag of the WorkerGlobalScope, effectively preventing any new task to get queued.
However, it doesn't stop the execution of the current task.
So indeed, in your first example, it will run the current script until its end and still execute the self.postMessage('bar') operation, that your main thread will be able to handle.
Worker.terminate() on the other hand will also "[a]bort the script currently running in the worker."
I have an event listener in Node JS, as shown below.
client.on('collect', async reaction => {
await external.run(reaction, ...);
});
The function I called external.run returns a promise, and takes around 5 seconds to complete. If this event is triggered again while the previous trigger is still in execution (i.e before the 5 seconds it takes), it messes with my program.
Is there a way to wait for the previous execution to finish before running the new one?
Thanks.
Yes, what you want is called a Lock in other languages ... JS doesn't provide that mechanism natively, but its easy to write one yourself:
const createLock = () => {
let queue = Promise.resolve();
return task => queue = queue.then(() => task());
};
const externalLock = createLock();
client.on('collect', reaction => externalLock(async () => {
await external.run(reaction, ...);
}));
For sure this is only a contrived example, you might want to handle errors properly ... or you just use one of the libraries out there that do this
Take the following loop:
for(var i=0; i<100; ++i){
let result = await some_slow_async_function();
do_something_with_result();
}
Does await block the loop? Or does the i continue to be incremented while awaiting?
Is the order of do_something_with_result() guaranteed sequential with regard to i? Or does it depend on how fast the awaited function is for each i?
Does await block the loop? Or does the i continue to be incremented while awaiting?
"Block" is not the right word, but yes, i does not continue to be incremented while awaiting. Instead the execution jumps back to where the async function was called, providing a promise as return value, continuing the rest of the code that follows after the function call, until the code stack has been emptied. Then when the awaiting is over, the state of the function is restored, and execution continues within that function. Whenever that function returns (completes), the corresponding promise -- that was returned earlier on -- is resolved.
Is the order of do_something_with_result() guaranteed sequential with regard to i? Or does it depend on how fast the awaited function is for each i?
The order is guaranteed. The code following the await is also guaranteed to execute only after the call stack has been emptied, i.e. at least on or after the next microtask can execute.
See how the output is in this snippet. Note especially where it says "after calling test":
async function test() {
for (let i = 0; i < 2; i++) {
console.log('Before await for ', i);
let result = await Promise.resolve(i);
console.log('After await. Value is ', result);
}
}
test().then(_ => console.log('After test() resolved'));
console.log('After calling test');
As #realbart says, it does block the loop, which then will make the calls sequential.
If you want to trigger a ton of awaitable operations and then handle them all together, you could do something like this:
const promisesToAwait = [];
for (let i = 0; i < 100; i++) {
promisesToAwait.push(fetchDataForId(i));
}
const responses = await Promise.all(promisesToAwait);
You can test async/await inside a "FOR LOOP" like this:
(async () => {
for (let i = 0; i < 100; i++) {
await delay();
console.log(i);
}
})();
function delay() {
return new Promise((resolve, reject) => {
setTimeout(resolve, 100);
});
}
async functions return a Promise, which is an object that will eventually "resolve" to a value, or "reject" with an error. The await keyword means to wait until this value (or error) has been finalized.
So from the perspective of the running function, it blocks waiting for the result of the slow async function. The javascript engine, on the other hand, sees that this function is blocked waiting for the result, so it will go check the event loop (ie. new mouse clicks, or connection requests, etc.) to see if there are any other things it can work on until the results are returned.
Note however, that if the slow async function is slow because it is computing lots of stuff in your javascript code, the javascript engine won't have lots of resources to do other stuff (and by doing other stuff would likely make the slow async function even slower). Where the benefit of async functions really shine is for I/O intensive operations like querying a database or transmitting a large file where the javascript engine is well and truly waiting on something else (ie. database, filesystem, etc.).
The following two bits of code are functionally equivalent:
let result = await some_slow_async_function();
and
let promise = some_slow_async_function(); // start the slow async function
// you could do other stuff here while the slow async function is running
let result = await promise; // wait for the final value from the slow async function
In the second example above the slow async function is called without the await keyword, so it will start execution of the function and return a promise. Then you can do other things (if you have other things to do). Then the await keyword is used to block until the promise actually "resolves". So from the perspective of the for loop it will run synchronous.
So:
yes, the await keyword has the effect of blocking the running function until the async function either "resolves" with a value or "rejects" with an error, but it does not block the javascript engine, which can still do other things if it has other things to do while awaiting
yes, the execution of the loop will be sequential
There is an awesome tutorial about all this at http://javascript.info/async.
No Event loop isn't blocked, see example below
function sayHelloAfterSomeTime (ms) {
return new Promise((resolve, reject) => {
if (typeof ms !== 'number') return reject('ms must be a number')
setTimeout(() => {
console.log('Hello after '+ ms / 1000 + ' second(s)')
resolve()
}, ms)
})
}
async function awaitGo (ms) {
await sayHelloAfterSomeTime(ms).catch(e => console.log(e))
console.log('after awaiting for saying Hello, i can do another things ...')
}
function notAwaitGo (ms) {
sayHelloAfterSomeTime(ms).catch(e => console.log(e))
console.log('i dont wait for saying Hello ...')
}
awaitGo(1000)
notAwaitGo(1000)
console.log('coucou i am event loop and i am not blocked ...')
Here is my test solution about this interesting question:
import crypto from "crypto";
function diyCrypto() {
return new Promise((resolve, reject) => {
crypto.pbkdf2('secret', 'salt', 2000000, 64, 'sha512', (err, res) => {
if (err) {
reject(err)
return
}
resolve(res.toString("base64"))
})
})
}
setTimeout(async () => {
console.log("before await...")
const a = await diyCrypto();
console.log("after await...", a)
}, 0);
setInterval(() => {
console.log("test....")
}, 200);
Inside the setTimeout's callback the await blocks the execution. But the setInterval is keep runnning, so the Event Loop is running as usual.
Let me clarify a bit because some answers here have some wrong information about how Promise execution works, specifically when related to the event loop.
In the case of the example, await will block the loop. do_something_with_result() will not be called until await finishes it's scheduled job.
https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous/Async_await#handling_asyncawait_slowdown
As for the other points, Promise "jobs" run before the next event loop cycle, as microtasks. When you call Promise.then() or the resolve() function inside new Promise((resolve) => {}), you creating a Job. Both await and async are wrapper, of sorts, for Promise, that will both create a Job. Microtasks are meant to run before the next event loop cycle. That means adding a Promise Job means more work before it can move on to the next event loop cycle.
Here's an example how you can lock up your event loop because your promises (Jobs) take too long.
let tick = 0;
let time = performance.now();
setTimeout(() => console.log('Hi from timeout'), 0);
const tock = () => console.log(tick++);
const longTask = async () => {
console.log('begin task');
for(let i = 0; i < 1_000_000_000; i++) {
Math.sqrt(i);
}
console.log('done task');
}
requestAnimationFrame(()=> console.log('next frame after', performance.now() - time, 'ms'));
async function run() {
await tock();
await tock();
await longTask(); // Will stall your UI
await tock(); // Will execute even though it's already dropped frames
await tock(); // This will execute too
}
run();
// Promise.resolve().then(tock).then(tock).then(longTask).then(tock).then(tock);
In this sample, 5 total promises are created. 2 calls for tock, 1 for longTask and then 2 calls for tock. All 5 will run before the next event loop.
The execution would be:
Start JS execution
Execute normal script
Run 5 scheduled Promise jobs
End JS execution
Event Loop Cycle Start
Request Animation Frame fire
Timeout fire
The last line commented line is scheduling without async/await and results in the same.
Basically, you will stall the next event loop cycle unless you tell your JS execution where it can suspend. Your Promise jobs will continue to run in the current event loop run until it finishes its call stack. When you call something external, (like fetch), then it's likely using letting the call stack end and has a callback that will resolve the pending Promise. Like this:
function waitForClick() {
return new Promise((resolve) => {
// Use an event as a callback;
button.onclick = () => resolve();
// Let the call stack finish by implicitly not returning anything, or explicitly returning `undefined` (same thing).
// return undefined;
})
}
If you have a long job job that want to complete, either use a Web Worker to run it without pausing, or insert some pauses with something like setTimeout() or setImmediate().
Reshaping the longTask function, you can do something like this:
const longTask = async () => {
console.log('begin task');
for(let i = 0; i < 1_000_000_000; i++)
if (i && i % (10_000_000) === 0) {
await new Promise((r) => setTimeout(r,0));
}
Math.sqrt(i);
console.log('done task');
}
Basically, instead of doing 1 billion records in one shot, you only do 10 million and then wait until the next event (setTimeout) to run the next one. The bad here is it's slower because of how much you hand back to the event loop. Instead, you can use requestIdleCallback() which is better, but still not as good as multi-threading via Web Workers.
But be aware that just slapping on await or Promise.resolve().then() around a function won't help with the event loop. Both will wait until the function returns with either a Promise or a value before letting up for the event loop. You can mostly test by checking to see if the function you're calling returns an unresolved Promise immediately.
Does await block the loop? Or does the i continue to be incremented while awaiting?
No, await won't block the looping. Yes, i continues to be incremented while looping.
Is the order of do_something_with_result() guaranteed sequential with regard to i? Or does it depend on how fast the awaited function is for each i?
Order of do_something_with_result() is guaranteed sequentially but not with regards to i. It depends on how fast the awaited function runs.
All calls to some_slow_async_function() are batched, i.e., if do_something_with_result() was a console then we will see it printed the number of times the loop runs. And then sequentially, after this, all the await calls will be executed.
To better understand you can run below code snippet:
async function someFunction(){
for (let i=0;i<5;i++){
await callAPI();
console.log('After', i, 'th API call');
}
console.log("All API got executed");
}
function callAPI(){
setTimeout(()=>{
console.log("I was called at: "+new Date().getTime())}, 1000);
}
someFunction();
One can clearly see how line console.log('After', i, 'th API call'); gets printed first for entire stretch of the for loop and then at the end when all code is executed we get results from callAPI().
So if lines after await were dependent on result obtained from await calls then they will not work as expected.
To conclude, await in for-loop does not ensure successful operation on result obtained from await calls which might take some time to finish.
In node, if one uses neo-async library with waterfall, one can achieve this.