I'm trying to build a queue with nodejs and async module but it's not working as desired.
Here is my code:
const async = require('async');
const queueSize = 10;
const taskHandler = function (task, done) {
task(done);
};
const myQueue = async.queue(taskHandler, queueSize);
myQueue.drain = function () {
console.log('The queue is now empty.');
};
function delay() {
return new Promise(resolve => setTimeout(resolve, 1000));
}
async function delayedLog(item) {
await delay();
console.log(item);
}
const run = async () => {
for (let item = 0; item < 30; item++) {
myQueue.push(async function (done) {
await delayedLog(item)
done();
});
}
}
(async () => {
console.log('START');
await run();
console.log('END);
})()
What I want:
START
// logs from delayedLog
END
Output:
START
END
// logs from delayedLog
As you can see await isn't working. I tried to promisify the module but the problem is still here. I tried with d3-queue and I had exactly the same problem.
Any suggestion ?
As #Chiến Nghê said you are using some async/await where there is no need to but this is another problem.
Your await isn't working at the end because your drain function is called at the end of your queue and it's not returning a promise.
You have to promisify your function, here is a sample example to do that:
function end() {
return new Promise((resolve, reject) => {
myQueue.drain = function () {
console.log('The queue is now empty.');
resolve();
};
})
}
And then you can use your async await on your end function:
(async () => {
console.log('START');
run();
await end();
console.log('END');
})()
Output:
START
// a lot of logs
END
Your program is working as it is. Actually you dont need to declare async for function run(). This function is simply to put 30 async tasks to the queue for later execution then it ends after that.
I don't know exact what you want, but if you want the console.log('END') logic is executed only every async tasks in queue are finished. Please move it to queue's drain() function.
Related
I have a function that looks like this:
async function sync(req, res, done){
await createRecords().then(async ()=> {
await Promise.all(
[
quantityReport(),
listings(),
productIdentifiers(),
productChildren()
])
}).then(async ()=>{
await saveAll()
} ).then(await createCSV);
}
module.exports = sync
I am calling it like this inside a switch:
// const saveRecords = require('../scripts/saveRecords.js') <- for reference
await saveRecords;
My problem is that the program continues before saveRecords finishes and I cannot figure out why.
All of the functions in Promise.all are asynchronous functions.
If I call sync() directly in saveRecords.js it works fine.
Thanks.
edit
createCSV also works fine in other locations in the program. It's imported to this file like this:
const {createCSV, uploadReports} = require('../scripts/createCSV.js')
//in createCSV.js
module.exports = createCSV;
I'd refactor your function as such (by the way, sync doesn't sounds like a great name for your function, write something more obvious).
async function sync(req, res, done){
try{
await createRecords()
const _res = await Promise.all([
quantityReport(),
listings(),
productIdentifiers(),
productChildren()
])
if(_res) {
await saveAll()
await createCSV()
}
return
}
catch(err){
throw new Error(err)
}
}
module.exports = sync
As I mentioned in the comments using async/await with then from the Promise API (see also fetch) is an odd thing to do. Use one or the other. But the key issue is that you're not calling the sync function await sync().
Here's an quick example of simply using async/await.
function mockCall(n) {
return new Promise((res, rej) => {
setTimeout(() => res(n), 1000);
});
}
async function sync() {
const first = await mockCall(1);
const twoToFive = await Promise.all([
mockCall(2),
mockCall(3),
mockCall(4),
mockCall(5)
]);
const six = await mockCall(6);
const seven = await mockCall(7);
console.log([ first, twoToFive, six, seven ]);
}
(async function main() {
await sync();
console.log('Let the processing resume!');
})();
Description
I want to conditionally call a db SELECT within a non-async function. I wrapped it in a IIAEF? like so
async function getScopedDataPolicies() {
const test = new Promise((res) => {
console.log('2')
res('test1')
})
return test;
}
function nonAsyncMethod() {
(async () => {
console.log('1')
const test = getScopedDataPolicies();
const policies = await test;
console.log('3')
})()
console.log('4')
}
nonAsyncMethod()
My expectation is
1
2
3
4
But I get
1
2
4
3
What wouldn't the await work in the immediately invoking method??
Look at these examples; am I missing something?
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/async_function
All await does is pause the execution of the current function, until an async task has finished, at which point execution will continue. It does not pause all of Javascript, that kind of behavior is synchronous programming, and Javascript is built to avoid it, because it makes for a really bad user experience if Javascript has to stall every time you make a rest request, etc.
update
Some possible workarounds:
await the IIFE, or take out the IIFE entirely. The IIFE is an async function, so it'll return a promise when it'scalled. You're not awaiting that promise, which is why your outer sync function continues on without waiting for it to finish.
async function nonAsyncMethod() {
await (async () => {
console.log('1')
const test = getScopedDataPolicies();
const policies = await test;
console.log('3')
})()
console.log('4')
}
Move console.log('4') into your IIFE. I know that console.log('4') might represent a whole log of code, but in some cases, you've just got to move it all inside - break up your function into smaller functions if you have to.
await inside the IIFE doesn't effect the outer function
To do so, you could either do this:
async function getScopedDataPolicies() {
const test = new Promise((res) => {
console.log('2')
res('test1')
})
return test;
}
async function nonAsyncMethod() {
await (async () => {
console.log('1')
const test = getScopedDataPolicies();
const policies = await test;
console.log('3')
})()
console.log('4')
}
nonAsyncMethod()
or this:
async function getScopedDataPolicies() {
const test = new Promise((res) => {
console.log('2')
res('test1')
})
return test;
}
function nonAsyncMethod() {
(async () => {
console.log('1')
const test = getScopedDataPolicies();
const policies = await test;
console.log('3')
})().then(() => {
console.log('4')
})
}
nonAsyncMethod()
Note: if you need to "wait" for nonAsyncMethod - you can only do await nonAsyncMethod(), or nonAsyncMethod.then(() => { continue here })
I can't seem to wrap my head around async await, let's say I want to process one item of an array at a time, with a 1 second delay:
myfunction() {
clearTimeout(this.timer);
this.timer = setTimeout(() => {
for (const item of this.myarray) {
this.dosomething(item).then((res: string) => {
setTimeout(() => {
await this.final_thing_before_the_next_item.push(res);
}, 1000);
});
}
}, 200);
}
Where would I put the "async"?
Stackblitz demo
You can put an async inside the setTimeout:
myfunction() {
setTimeout(async () => {
for (const item of this.myarray) {
await this.wait(1000);
console.log(item);
}
});
}
wait(timer: number): Promise<void> {
return new Promise((resolve, reject) => {
setTimeout(() => resolve(), timer);
});
}
In a simplified form, await is syntax sugar for consuming promises. The following snippets are pretty much equivalent:
doSomethingAsync().then(value => {
console.log('it finished', value);
});
const value = await doSomethingAsync();
The caveat is that if you want to use await, you need to mark the function as async. I will assume you want to have a 1s delay after full completion. Your code might look something like this:
function wait() {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
async function processItems(items) {
for (const item of items) {
const res = await this.doSomething(item);
// This isn't processed until doSomething completes.
const other = await this.doFinalThing(res);
// This isn't run until doFinalThing completes.
await wait(1000);
// The loop now continues after waiting 1s.
}
}
you marked this as angular, so I'm going to say, consider rxjs...
// imports
import {timer, from, concat} from 'rxjs'
import {switchMap, delay} from 'rxjs/operators'
// wait 200 ms
timer(200).pipe(
// switch
switchMap(() => {
// map array into streams
const obs$ = this.myArray.map(i => {
// from promise, do whatever
return from(this.dosomething(i)).pipe(
// switch again, to do final thing, again from promise
switchMap(res => from(this.finalThing(res))),
// delay 1s
delay(1000)
)
});
return concat(...obs$); // execute one after the other
})
).subscribe(v => console.log('values 1 by 1, spaced 1s apart', v))
If you want to process one asynchronous operation at a time, you could do the following:
// "wait" for a 1000ms before resolving the Promise
const wait = (ms = 1000) => {
return new Promise(resolve => {
setTimeout(() => resolve(), ms);
});
};
// array of asynchronous operations
const promises = Array.from({ length: 4 }).map((_, idx) =>
Promise.resolve(idx)
);
async function main() {
const data = [];
for (let item of promises) {
const result = await item;
await wait(1000);
data.push(result);
console.log("result", result);
}
console.log("final data", data);
}
main();
If I understand your question (and code sample) correctly, you basically want to
Sleep 200 ms
Iterate over a list of items. For each item you want to:
Call a function, passing the current item to it and getting a response in return.
Pause for 1 second.
Call a second function, passing it the response
To do that you need a sleep() function that returns a promise that will resolve when the specified time has elapsed:
function sleep(ms = 1000) {
const p = new Promise( (resolve, reject) => {
setTimeout(() => resolve());
});
return p;
}
Having that, you need an async function to do the actual work. It has to be async because that is what lets it await other things:
async function process_items( items, delayInMs = 1000 ) {
for ( item of items ) {
const res = await doSomething( item, delayInMs );
await sleep(delay);
await doSomethingElse( res );
}
}
As you can see from the above bit of code, the advantage of using async/await over callbacks or promise chains is that it gives you a rather more succinct and declarative syntax.
Then you can wrap it all up in another async function:
async function myfunction() {
await sleep(200);
await process_items( this.myarray, 1000 );
}
Marking a function as async does two things: it
Enables the use of await within that function, and
Converts the function into a function returning a promise, regardless of what its ostensible return value is.
If you take this function:
function foo() {
return 1;
}
and mark it as async:
async function foo() {
return 1;
}
it is (more or less) as if you had changed to to read:
function foo() {
return Promise.resolve(1);
}
The easiest way, without looking at the specifics of your code is before myFunction ie async myFunction(). I see 'this' being used so I imagine this is a method of some object/class in which case that would be one way to go about it.
I'm using socket.io and mongoose in my express server.
My socket is listening for events using the following code:
socket.on('do something', async () => {
try {
await doA();
doX();
await doB();
doY();
await doC();
} catch (error) {
console.log(error);
}
});
doA, doB and doC are async operations that writes on database using mongoose, but in general they can be any method returning a promise.
I want that 'do something' runs synchronously.
If the event queue processes more events at the same time I have consistency problems in my mongodb.
In other words if the server receives two 'do something' events, I want that the second event received is processed only when the first event is fully processed (after the await doC). Unfortunately the 'do something' callback is async.
How to handle this?
It's possible to implement a queue by adding the functions you want to run to an array, and then running them one by one. I've created an example below.
let queue = [];
let running = false;
const delay = (t, v) => {
return new Promise((resolve) => {
setTimeout(resolve.bind(null, "Returned value from Promise"), t)
});
}
const onSocketEvent = async () => {
console.log("Got event");
if (!running) {
console.log("Nothing in queue, fire right away");
return doStuff();
}
// There's something in the queue, so add it to it
console.log("Queuing item")
queue.push(doStuff);
}
const doStuff = async () => {
running = true;
const promiseResult = await delay(2000);
console.log(promiseResult);
if (queue.length > 0) {
console.log("There's more in the queue, run the next one now")
queue.shift()();
} else {
console.log("Queue empty!")
running = false;
}
}
onSocketEvent();
setTimeout(() => onSocketEvent(), 1000);
setTimeout(() => onSocketEvent(), 1500);
setTimeout(() => onSocketEvent(), 2000);
setTimeout(() => onSocketEvent(), 2500);
I would suggest adding a delay between each await. This will prevent deadlocks from occurring and fix your issue. For such things, I would suggest using the Caolan's async library.
Task delay example:
setTimeout(function() { your_function(); }, 5000); // 5 seconds
If your function has no parameters and no explicit receiver, you can call directly setTimeout(func, 5000)
Useful jQuery timers plugin
With Promises I can have two separate "threads" both waiting for the same value:
let trigger;
const promise = new Promise(r => {
console.log('promise is created *once*');
trigger = value => {
console.log('trigger is called *once*');
r(value);
}
});
(async () => {
console.log('A waiting');
const value = await promise;
console.log(`A finished, got ${value}`);
})();
(async () => {
console.log('B waiting');
const value = await promise;
console.log(`B finished, got ${value}`);
})();
trigger('hello');
console.log('And *two* things are waiting on the single promise');
I've tried to replicate this with async/await but to no avail.
The following snippet does not work:
let trigger = async () => {
console.log('trigger should be called *once*');
return 'hello';
};
(async () => {
console.log('A waiting');
const value = await trigger; // <-- What do I need to put here?
console.log(`A finished, got ${value}`);
})();
(async () => {
console.log('B waiting');
const value = await trigger; // <-- What do I need to put here?
console.log(`B finished, got ${value}`);
})();
trigger(); // <-- How can this "kick off" the two awaits above?
Is it possible to write the same functionality in the first snippet, using async/await?
I'm okay with falling back to using Promise if that's needed.
To await, you need to have a reference to the singular promise, so you can't call a function on demand and have that function create a promise and then use that same promise elsewhere (unless the function that creates the promise also holds it in state to return to other callers, like a singleton).
I'd create a singular promise initially, and then send it to the async functions:
const trigger = async () => {
console.log('trigger should be called *once*');
return 'hello';
};
async function as1(prom) {
console.log('A waiting');
const value = await prom;
console.log(`A finished, got ${value}`);
}
async function as2(prom) {
console.log('B waiting');
const value = await prom;
console.log(`B finished, got ${value}`);
}
const thePromise = trigger();
as1(thePromise);
as2(thePromise);
Don't use async just to return a promise, though - if the purpose of a function is to create a promise, then do it explicitly - that way, it's more clear what your code is meant to do. Async and await have not made the Promise keyword obsolete, it's just syntax sugar which is useful in some situations (and unnecessary in others).