I have an event listener in Node JS, as shown below.
client.on('collect', async reaction => {
await external.run(reaction, ...);
});
The function I called external.run returns a promise, and takes around 5 seconds to complete. If this event is triggered again while the previous trigger is still in execution (i.e before the 5 seconds it takes), it messes with my program.
Is there a way to wait for the previous execution to finish before running the new one?
Thanks.
Yes, what you want is called a Lock in other languages ... JS doesn't provide that mechanism natively, but its easy to write one yourself:
const createLock = () => {
let queue = Promise.resolve();
return task => queue = queue.then(() => task());
};
const externalLock = createLock();
client.on('collect', reaction => externalLock(async () => {
await external.run(reaction, ...);
}));
For sure this is only a contrived example, you might want to handle errors properly ... or you just use one of the libraries out there that do this
Related
I was reading up on the eventEmitter documentation and came across the section where they discuss async vs synchronous code. They have this example that showcases how to process an event asynchronously:
const myEmitter = new MyEmitter();
myEmitter.on('event', (a, b) => {
setImmediate(() => {
console.log('this happens asynchronously');
});
});
My question is: would using the async keyword not achieve the same thing?
Wouldn't the following implementation also run asynchronously?
const myEmitter = new MyEmitter();
myEmitter.on('event', async () => {
console.log('does this happens asynchronously????');
});
It may help if you tell us why you think those would be the same.
The second function will execute on the event even though it is marked async (it will just return a promise that won't be used anywhere).
The first will also execute on the event but schedule its inner contents (the function that you pass to setImmediate) to be run when the event queue is empty.
async doesn't mean "do later" it just means "wrap my return in a promise automagically and allow me to use await in my function body".
I know one must not block the loop, I know use call backs, and I know ES6 await. The more I research this the more it reaffirms it.
But sometimes your hands are tied. Is there a way to tell JavaScript, please go check on your event queue, and service those, then come back here before continuing execution.
Something like inspired by the MDN docs:
if (queue.hasNextMessage()) {
queue.processNextMessage()
}
There are similar threads about use datetime to wait a duration, but I don't know how the long other event thread will take, been looking at polling the promise status, but it appears to be a dead end.
The context is I have to override a validation callback. The caller of the callback does not wait for a promise to resolve (That I cant change).
Here is the test setup showing the concept. I have made a few attempts, but none of them work because they are always stuck in the main loop.
// The validate function depends on a fetch call which takes time.
// Free to change this.
function validate() {
return fetch(url).then(response => response.json())
.then(data => {console.log(data); return true;})
.catch(msg => {console.log(msg); return false;})
}
// Cannot change this function, I am not in control of it
function CallValidate() {
console.log("Validation Result: ", Boolean(validate()));
}
// This is the setup for when test passes
let url = 'http://api.open-notify.org/astros.json';
CallValidate();
// This is the setup for when test fails
// This currently fails because the promise objects is being evaluated
// to true, instead of waiting for its response.
url = 'http://DUMMY.NOT.WORKING.URL';
CallValidate();
Is there a way to tell JavaScript, please go check on your event queue, and service those, then come back here before continuing execution.
No, but you can do something very similar: divide your computation into several tasks. In Node.js, you can use setImmediate() to queue a task. In the browser, you can use messages, as outlined by this answer.
Example:
function setImmediate(callback) {
const channel = new MessageChannel();
channel.port1.onmessage = () => {
callback();
};
channel.port2.postMessage("");
}
console.log("Task 1");
setImmediate( () => {
console.log("Task 2");
});
Boolean(validate()) will always evaluate to true, because validate returns a Promise which is a truthy value no matter what it resolves to.
If you can't change CallValidate, you might still be able to lift up the fetch call. But I'm not sure if that's an option for you.
async function validateUrl(url) {
const valid = await fetch(url).then(response => response.json())
.then(data => {console.log(data); return true;})
.catch(msg => {console.log(msg); return false;});
function validate() {
return valid;
}
function CallValidate() {
console.log("Validation Result: ", Boolean(validate()));
}
CallValidate();
}
// assuming you are in async function context
await validateUrl('http://api.open-notify.org/astros.json');
await validateUrl('http://DUMMY.NOT.WORKING.URL');
If moving the function is also no option I would ditch the CallValidate code and either write something yourself, or pick another library/helper/etc that does handle asynchronous functions (with promise return values).
I've created some tests of using async and sync event handlers. I've came to conclusion that using async handlers could make huge improvement in our code.
The only difference in the below two snippets is that in one, customEventHandler is async, and in that handler, it uses await sleep(customEventHandlerSleepTime); instead of sleep(customEventHandlerSleepTime);:
async test:
<body>
<div id="event1">
<div id="event2">
<div id="event3">
<div id="event4"></div>
</div>
</div>
</div>
</body>
<script>
const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
const customEventHandlerIterationsCount = 1000000;
const customEventHandlerSleepTime = 500;
const customEventName = 'customevent';
const customEvent = new Event('customevent');
const customEventHandler = async() => {
for (let i = 0; i < customEventHandlerIterationsCount; ++i) {
await sleep(customEventHandlerSleepTime);
}
};
document.getElementById('event4').addEventListener(customEventName, customEventHandler);
document.getElementById('event3').addEventListener(customEventName, customEventHandler);
document.getElementById('event2').addEventListener(customEventName, customEventHandler);
document.getElementById('event1').addEventListener(customEventName, customEventHandler);
(() => {
const start = new Date().getTime();
document.getElementById('event4').dispatchEvent(customEvent);
const end = new Date().getTime();
console.log('Time: ', (end - start));
})();
</script>
sync test:
<body>
<div id="event1">
<div id="event2">
<div id="event3">
<div id="event4"></div>
</div>
</div>
</div>
</body>
<script>
const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
const customEventHandlerIterationsCount = 1000000;
const customEventHandlerSleepTime = 500;
const customEventName = 'customevent';
const customEvent = new Event('customevent');
const customEventHandler = () => {
for (let i = 0; i < customEventHandlerIterationsCount; ++i) {
sleep(customEventHandlerSleepTime).then(() => {});
}
};
document.getElementById('event4').addEventListener(customEventName, customEventHandler);
document.getElementById('event3').addEventListener(customEventName, customEventHandler);
document.getElementById('event2').addEventListener(customEventName, customEventHandler);
document.getElementById('event1').addEventListener(customEventName, customEventHandler);
(() => {
const start = new Date().getTime();
document.getElementById('event4').dispatchEvent(customEvent);
const end = new Date().getTime();
console.log('Time: ', (end - start));
})();
</script>
The result of the above tests are:
async test execution time: ~1ms
sync test execution time: ~1500ms
Am I doing something wrong or is it true? If we remove "await" and ".then()" from sleep function, the sync handler prints "Time" message faster with minimal difference of time.
Based on this test, I am wondering if it's better to always (or almost always) use async handlers, if we e.g. don't know what will be going on in nested functions of this handler or maybe if we don't use "await" in our handler directly it's better to avoid using async? Maybe there is a better way to test this?
You're doing heavy processing in both snippets. The main difference is that in the second (sync) snippet, you're creating all Promises at once, synchronously. There are a large number of promises, so the overhead of creating so many is significant. In the first (async) snippet, when the event is dispatched and the handler runs, you only create one sleep Promise synchronously - this takes next to no time at all, and then the event finishes. Then, as a microtask, the second Promise is created, and you wait for it to resolve. Then, as a microtask, the third Promise is created, and you wait for it to resolve. Etc.
You're doing heavy processing in both snippets, but in one, it's staggered out over a long period of time (the Promises run in serial), but in the other, the Promises all run in parallel, being initialized immediately. If the event is fired via Javascript (rather than, for example, a native button click), it will take some time to get to the line after the manual firing of the event if all the heavy processing is synchronous.
So, sure, sometimes this async technique may help you (though, processing this intensive is pretty rare in Javascript, so often it won't be noticeable at all).
A better option would probably be to move the heavy processing into a web worker instead - that way, the processing is done on a completely separate thread.
No, there's no advantage to using async functions for DOM event handlers, all it does is add (a very tiny bit of) overhead.
What your test is missing is that the functions still take the same amount of time to run, they just do it later, after you're done measuring, so you don't see it. But they still take that time, and it's still on the main UI thread.
I've got a really weird issue whereby awaiting a Promise that has passed its resolve to an event-emitter callback just exits the process without error.
const {EventEmitter} = require('events');
async function main() {
console.log("entry");
let ev = new EventEmitter();
let task = new Promise(resolve=>{
ev.once("next", function(){resolve()}); console.log("added listener");
});
await task;
console.log("exit");
}
main()
.then(()=>console.log("exit"))
.catch(console.log);
process.on("uncaughtException", (e)=>console.log(e));
I'm expecting the process to halt when I run this because clearly "next" is currently never emitted. but the output I get is:
entry
added listener
and then the nodejs process terminates gracefully.
I thought it was something to do with the Garbage Collector, but ev and task are clearly still in scope on main. So I'm really at a loss as to why the process exits entirely without error.
Obviously I would eventually emit the event, but I've simplified my code to the above to reproduce. I'm on node v8.7.0. Is there something wrong with my code or is this a node bug?
This question is basically: how does node decide whether to exit the event loop or go around again?
Basically node keeps a reference count of scheduled async requests — setTimeouts, network requests, etc.. Each time one is scheduled, that count increases, and each time one is finished, the count decreases. If you arrive at the end of an event loop cycle and that reference count is zero node exits.
Simply creating a promise or event emitter does not increase the reference count — creating these objects isn't actually an async operation. For example, this promise's state will always be pending but the process exits right away:
const p = new Promise( resolve => {
if(false) resolve()
})
p.then(console.log)
In the same vein this also exits after creating the emitter and registering a listener:
const ev = new EventEmitter()
ev.on("event", (e) => console.log("event:", e))
If you expect Node to wait on an event that is never scheduled, then you may be working under the idea that Node doesn't know whether there are future events possible, but it does because it keeps a count every time one is scheduled.
So consider this small alteration:
const ev = new EventEmitter()
ev.on("event", (e) => console.log("event:", e))
const timer = setTimeout(() => ev.emit("event", "fired!"), 1000)
// ref count is not zero, event loop will go again.
// after timer fires ref count goes back to zero and node exits
As a side note, you can remove the reference to the timer with: timeout.unref(). This, unlike the previous example, will exit immediately:
const ev = new EventEmitter()
ev.on("event", (e) => console.log("event:", e))
const timer = setTimeout(() => ev.emit("event", "fired!"), 1000)
timer.unref()
There's a good talk about the event loop by Bert Belder here that clears up a lot of misconceptions: https://www.youtube.com/watch?v=PNa9OMajw9w
I was debugging for several hours why one of our scripts exits (without any errors) after one line of code in the middle of main function. It was a line await connectToDatabase(config). And you know what?
I found that difference between these two functions is CRUCIAL:
first:
async function connectToDatabase(config = {}) {
if (!config.port) return;
return new Promise(resolve => {
resolve();
})
}
second:
async function connectToDatabase(config = {}) {
return new Promise(resolve => {
if (!config.port) return;
resolve();
})
}
second function sometimes (when config.port is empty) creates never-resolved promise, it makes event loop empty, and node.js exits thinking that "nothing more to do here"
check it yourself:
// index.js - start it as node index.js
(async function main() {
console.log('STARTED')
await connectToDatabase()
console.log('CONNECTED')
console.log('DOING SOMETHING ELSE')
})()
'CONNECTED' and 'DOING SOMETHING ELSE' are NOT printed if you use second function and are printed, if you use first
As a general note, your code is combining three similar, but different methods: async/await, promises, event listeners. I'm not sure what you mean by "bombs out." But looking at the code, the result seems expected.
Your process exits, because you called promise on adding your event listener. It successfully resolves, and therefore exits. If you try to log task, it will give you undefined. Instead of logging "exit" in your then statement, log the result. Task will be undefined since the program does not wait to resolve its value and its "code block has finished".
You can simplify your code to the following. As you can see it resolves immediately since you call the resolve function.
const { EventEmitter } = require('events');
let ev = new EventEmitter()
var p = new Promise(( resolve ) => {
ev.once("next", resolve("Added Event Listener"));
})
p
.then(res => console.log(res))
.catch(e => console.log(e))
Take the following loop:
for(var i=0; i<100; ++i){
let result = await some_slow_async_function();
do_something_with_result();
}
Does await block the loop? Or does the i continue to be incremented while awaiting?
Is the order of do_something_with_result() guaranteed sequential with regard to i? Or does it depend on how fast the awaited function is for each i?
Does await block the loop? Or does the i continue to be incremented while awaiting?
"Block" is not the right word, but yes, i does not continue to be incremented while awaiting. Instead the execution jumps back to where the async function was called, providing a promise as return value, continuing the rest of the code that follows after the function call, until the code stack has been emptied. Then when the awaiting is over, the state of the function is restored, and execution continues within that function. Whenever that function returns (completes), the corresponding promise -- that was returned earlier on -- is resolved.
Is the order of do_something_with_result() guaranteed sequential with regard to i? Or does it depend on how fast the awaited function is for each i?
The order is guaranteed. The code following the await is also guaranteed to execute only after the call stack has been emptied, i.e. at least on or after the next microtask can execute.
See how the output is in this snippet. Note especially where it says "after calling test":
async function test() {
for (let i = 0; i < 2; i++) {
console.log('Before await for ', i);
let result = await Promise.resolve(i);
console.log('After await. Value is ', result);
}
}
test().then(_ => console.log('After test() resolved'));
console.log('After calling test');
As #realbart says, it does block the loop, which then will make the calls sequential.
If you want to trigger a ton of awaitable operations and then handle them all together, you could do something like this:
const promisesToAwait = [];
for (let i = 0; i < 100; i++) {
promisesToAwait.push(fetchDataForId(i));
}
const responses = await Promise.all(promisesToAwait);
You can test async/await inside a "FOR LOOP" like this:
(async () => {
for (let i = 0; i < 100; i++) {
await delay();
console.log(i);
}
})();
function delay() {
return new Promise((resolve, reject) => {
setTimeout(resolve, 100);
});
}
async functions return a Promise, which is an object that will eventually "resolve" to a value, or "reject" with an error. The await keyword means to wait until this value (or error) has been finalized.
So from the perspective of the running function, it blocks waiting for the result of the slow async function. The javascript engine, on the other hand, sees that this function is blocked waiting for the result, so it will go check the event loop (ie. new mouse clicks, or connection requests, etc.) to see if there are any other things it can work on until the results are returned.
Note however, that if the slow async function is slow because it is computing lots of stuff in your javascript code, the javascript engine won't have lots of resources to do other stuff (and by doing other stuff would likely make the slow async function even slower). Where the benefit of async functions really shine is for I/O intensive operations like querying a database or transmitting a large file where the javascript engine is well and truly waiting on something else (ie. database, filesystem, etc.).
The following two bits of code are functionally equivalent:
let result = await some_slow_async_function();
and
let promise = some_slow_async_function(); // start the slow async function
// you could do other stuff here while the slow async function is running
let result = await promise; // wait for the final value from the slow async function
In the second example above the slow async function is called without the await keyword, so it will start execution of the function and return a promise. Then you can do other things (if you have other things to do). Then the await keyword is used to block until the promise actually "resolves". So from the perspective of the for loop it will run synchronous.
So:
yes, the await keyword has the effect of blocking the running function until the async function either "resolves" with a value or "rejects" with an error, but it does not block the javascript engine, which can still do other things if it has other things to do while awaiting
yes, the execution of the loop will be sequential
There is an awesome tutorial about all this at http://javascript.info/async.
No Event loop isn't blocked, see example below
function sayHelloAfterSomeTime (ms) {
return new Promise((resolve, reject) => {
if (typeof ms !== 'number') return reject('ms must be a number')
setTimeout(() => {
console.log('Hello after '+ ms / 1000 + ' second(s)')
resolve()
}, ms)
})
}
async function awaitGo (ms) {
await sayHelloAfterSomeTime(ms).catch(e => console.log(e))
console.log('after awaiting for saying Hello, i can do another things ...')
}
function notAwaitGo (ms) {
sayHelloAfterSomeTime(ms).catch(e => console.log(e))
console.log('i dont wait for saying Hello ...')
}
awaitGo(1000)
notAwaitGo(1000)
console.log('coucou i am event loop and i am not blocked ...')
Here is my test solution about this interesting question:
import crypto from "crypto";
function diyCrypto() {
return new Promise((resolve, reject) => {
crypto.pbkdf2('secret', 'salt', 2000000, 64, 'sha512', (err, res) => {
if (err) {
reject(err)
return
}
resolve(res.toString("base64"))
})
})
}
setTimeout(async () => {
console.log("before await...")
const a = await diyCrypto();
console.log("after await...", a)
}, 0);
setInterval(() => {
console.log("test....")
}, 200);
Inside the setTimeout's callback the await blocks the execution. But the setInterval is keep runnning, so the Event Loop is running as usual.
Let me clarify a bit because some answers here have some wrong information about how Promise execution works, specifically when related to the event loop.
In the case of the example, await will block the loop. do_something_with_result() will not be called until await finishes it's scheduled job.
https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous/Async_await#handling_asyncawait_slowdown
As for the other points, Promise "jobs" run before the next event loop cycle, as microtasks. When you call Promise.then() or the resolve() function inside new Promise((resolve) => {}), you creating a Job. Both await and async are wrapper, of sorts, for Promise, that will both create a Job. Microtasks are meant to run before the next event loop cycle. That means adding a Promise Job means more work before it can move on to the next event loop cycle.
Here's an example how you can lock up your event loop because your promises (Jobs) take too long.
let tick = 0;
let time = performance.now();
setTimeout(() => console.log('Hi from timeout'), 0);
const tock = () => console.log(tick++);
const longTask = async () => {
console.log('begin task');
for(let i = 0; i < 1_000_000_000; i++) {
Math.sqrt(i);
}
console.log('done task');
}
requestAnimationFrame(()=> console.log('next frame after', performance.now() - time, 'ms'));
async function run() {
await tock();
await tock();
await longTask(); // Will stall your UI
await tock(); // Will execute even though it's already dropped frames
await tock(); // This will execute too
}
run();
// Promise.resolve().then(tock).then(tock).then(longTask).then(tock).then(tock);
In this sample, 5 total promises are created. 2 calls for tock, 1 for longTask and then 2 calls for tock. All 5 will run before the next event loop.
The execution would be:
Start JS execution
Execute normal script
Run 5 scheduled Promise jobs
End JS execution
Event Loop Cycle Start
Request Animation Frame fire
Timeout fire
The last line commented line is scheduling without async/await and results in the same.
Basically, you will stall the next event loop cycle unless you tell your JS execution where it can suspend. Your Promise jobs will continue to run in the current event loop run until it finishes its call stack. When you call something external, (like fetch), then it's likely using letting the call stack end and has a callback that will resolve the pending Promise. Like this:
function waitForClick() {
return new Promise((resolve) => {
// Use an event as a callback;
button.onclick = () => resolve();
// Let the call stack finish by implicitly not returning anything, or explicitly returning `undefined` (same thing).
// return undefined;
})
}
If you have a long job job that want to complete, either use a Web Worker to run it without pausing, or insert some pauses with something like setTimeout() or setImmediate().
Reshaping the longTask function, you can do something like this:
const longTask = async () => {
console.log('begin task');
for(let i = 0; i < 1_000_000_000; i++)
if (i && i % (10_000_000) === 0) {
await new Promise((r) => setTimeout(r,0));
}
Math.sqrt(i);
console.log('done task');
}
Basically, instead of doing 1 billion records in one shot, you only do 10 million and then wait until the next event (setTimeout) to run the next one. The bad here is it's slower because of how much you hand back to the event loop. Instead, you can use requestIdleCallback() which is better, but still not as good as multi-threading via Web Workers.
But be aware that just slapping on await or Promise.resolve().then() around a function won't help with the event loop. Both will wait until the function returns with either a Promise or a value before letting up for the event loop. You can mostly test by checking to see if the function you're calling returns an unresolved Promise immediately.
Does await block the loop? Or does the i continue to be incremented while awaiting?
No, await won't block the looping. Yes, i continues to be incremented while looping.
Is the order of do_something_with_result() guaranteed sequential with regard to i? Or does it depend on how fast the awaited function is for each i?
Order of do_something_with_result() is guaranteed sequentially but not with regards to i. It depends on how fast the awaited function runs.
All calls to some_slow_async_function() are batched, i.e., if do_something_with_result() was a console then we will see it printed the number of times the loop runs. And then sequentially, after this, all the await calls will be executed.
To better understand you can run below code snippet:
async function someFunction(){
for (let i=0;i<5;i++){
await callAPI();
console.log('After', i, 'th API call');
}
console.log("All API got executed");
}
function callAPI(){
setTimeout(()=>{
console.log("I was called at: "+new Date().getTime())}, 1000);
}
someFunction();
One can clearly see how line console.log('After', i, 'th API call'); gets printed first for entire stretch of the for loop and then at the end when all code is executed we get results from callAPI().
So if lines after await were dependent on result obtained from await calls then they will not work as expected.
To conclude, await in for-loop does not ensure successful operation on result obtained from await calls which might take some time to finish.
In node, if one uses neo-async library with waterfall, one can achieve this.