in an async function i, want to main function to return only in a specific point and not to return void at the end of the function
const fun = () => {
const list = [];
let streamFinished = 0;
let streamCount = files.length;
await fs.readdir(JSON_DIR, async(err, files) => {
await files.forEach((filename) => {
const readStream = fs.createReadStream(path.join("directory", filename));
const parseStream = json.createParseStream();
await parseStream.on('data', async(hostlist: Host[]) => {
hostlist.forEach(async host => {
list.push(host);
});
});
parseStream.on('end', () => {
streamFinished++;
if (streamFinished === streamCount) {
// End of all streams...
return list; //this should be the only point when the function return something
}
})
readStream.pipe(parseStream);
})
});
};
There are lots of things wrong with this code. It's a big mix of event-driven streams, callback-driven functions and promises. The first order of business is to make the main business logic just be promise-driven (as you can see in the new parseData() function and switch all control flow outside of that to just use promises. Here's one way to do it:
const fsp = fs.promises;
function parseData(filename) {
return new Promise((resolve, reject) => {
const readStream = fs.createReadStream(path.join("directory", filename));
const parseStream = json.createParseStream();
const list = [];
parseStream.on('data', (hostlist: Host[]) => {
list.push(...hostlist);
}).on('end', () => {
resolve(list);
}).on('error', reject);
readStream.pipe(parseStream);
});
}
const fun = async () => {
const list = [];
const files = await fsp.readdir(JSON_DIR);
for (let filename of files) {
const listData = await parseData(filename);
list.push(...listData);
}
return list;
};
fun().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
A few thoughts here:
The easiest way to "wait" for a stream operation to complete is to encapsulate it in a promise which you can then use await or .then() with. That's the purpose of the parseData() function. In addition, this also enables error handling by hooking stream errors to the promise rejection.
For processing a loop of asynchronous operations, you can either do them one at a time, using await on each asynchronous operation in the loop or you can run them in parallel by collecting an array of promises and using let data = await Promise.all(arrayOfPromises); on that array of promises.
It is only useful to use await if the thing you're awaiting is a promise that is connected to your asynchronous operation. So, things like await parseStream.on('data', ...) and await files.forEach(...) are pointless because neither of those return promises. Do NOT just stick await in places unless you KNOW you are awaiting a promise.
You will generally NOT want to use .forEach() with an asynchronous operation in the loop. Because .forEach() has no return value and no loop control, you can't really control much. Use a regular for loop instead which gives you full control. I consider .forEach() pretty much obsolete for asynchronous programming.
Don't mix promises and callbacks and don't mix promises and streams. If you have those, then "promisify" the stream or callback so all your main logic/control flow can be promises. This will vastly simplify your code and make error handling possible/practical.
Related
In node.js I am trying to get a list of bid and ask prices from a trade exchange website (within an async function). Within the foreach statement I can console.info() the object data(on each iteration) but when I put all of this into an array and then return it to another function it passes as 'undefined'.
const symbolPrice = async() => {
let symbolObj = {}
let symbolList = []
await bookTickers((error, ticker) => {
ticker.forEach(symbol => {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolObj = {
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
}
console.info(symbolObj);
}
symbolList.push(symbolObj)
});
const results = Promise.all(symbolList)
return results;
});
}
const symbolPriceTest = async() => {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
I have tried pretty much everything I can find on the internet like different awaits/Promise.all()'s. I do admit I am not as familiar with async coding as I would like to be.
So, if the basic problem here is to call bookTickers(), retrieve the asynchronous result from that call, process it and then get that processed result as the resolved value from calling symbolPrice(), then you can do that like this:
const { promisify } = require('util');
const bookTickersP = promisify(bookTickers);
async function symbolPrice(/* declare arguments here */) {
let symbolList = [];
const ticker = await bookTickersP(/* fill in arguments here */);
for (let symbol of ticker) {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolList.push({
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
});
}
}
return symbolList;
}
async function symbolPriceTest() {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
Things to learn from your original attempt:
Only use await when you are awaiting a promise.
Only use Promise.all() when you are passing it an array of promises (or an array of a mixture of values and promises).
Don't mix plain callback asynchronous functions with promises. If you don't have a promise-returning version of your asynchronous function, then promisify it so you do (as shown in my code above with bookTickersP().
Do not guess with async and await and just throw it somewhere hoping it will do something useful. You MUST know that you're awaiting a promise that is connected to the result you're after.
Don't reuse variables in a loop.
Your original implementation of symbolPrice() had no return value at the top level of the function (the only return value was inside a callback so that just returns from the callback, not from the main function). That's why symbolPrice() didn't return anything. Now, because you were using an asynchronous callback, you couldn't actually directly return the results anyway so other things had to be redone.
Just a few thoughts on organization that might be reused in other contexts...
Promisify book tickers (with a library, or pure js using the following pattern). This is just the api made modern:
async function bookTickersP() {
return new Promise((resolve, reject) => {
bookTickers((error, ticker) => {
error ? reject(error) : resolve(ticker);
})
});
}
Use that to shape data in the way that the app needs. This is your app's async model getter:
// resolves to [{ symbol, bid, ask }, ...]
async function getSymbols() {
const result = await bookTickersP();
return result.map(({ symbol, bidPrice, askPrice }) => {
return { symbol: symbol.toUpperCase(), bid: bidPrice, ask: askPrice }
});
}
Write business logic that does things with the model, like ask about particular symbols:
async function symbolPriceTest(search) {
const prefix = search.toUpperCase();
const symbols = await getSymbols();
return symbols.filter(s => s.symbol.startsWith(prefix));
}
symbolPriceTest('ETH').then(console.log);
I have a very simple code I'm trying to debug. I have an async function:
async function updateResult(event){
let result = db.fetchResult(event.ProcessId);
return result;
}
And I'm calling this from another simple async function:
exports.processEvent = async (event) => {
try {
let promises = [];
const controls = await db.fetchControls(event.EventCode);
if (controls) {
promises = controls.map((control) => {
console.log(control);
control.map(x => this.updateResult(event));
});
}
return Promise.allSettled(promises);
}
catch (err) {
console.error(err);
}
};
The problem is that in exports.processEvent, the content of db.fetchControls(event.EventCode) is executed to completion (a few logics and a db getItem call).
But the call to db.fetchResult(event.ProcessId) via this.updateResult(x.Id, x.Version, event) does NOT complete executing the fetchResult tasks (a few other logics a db GET call).
I feel like fetchResult returns prematurely. What am I doing wrong?
.map() is NOT promise-aware. It does not wait for any of the promises in your callback to complete. So, in this piece of code:
control.map(x => this.updateResult(x.Id, x.Version, event));
It just runs through the entire array ignoring all the promises that are returned and thus NOTHING waits for those promises.
You can fix that by returning those promises to the higher level .map() which you are then awaiting with Promise.all():
promises = controls.map((control) => {
console.log(control);
return Promise.all(control.map(x => this.updateResult(x.Id, x.Version, event)));
});
How can I stream data using promises.
I have two different functions in two different files. In one calls an API service and that returns a promise.
async myPromise(){
return new Promise((resolve, reject) => {
callToAnAPI().then(()=>{
resolve ("pending");
}).then(()=>{
resolve(callToAnotherAPI());
})
.catch(err=>{
// error handling
});
});
}
In another file I have a function like so:
async myPromise2(){
functionFromOtherFile().then((data)=>{
// how can I get 'pending' here?
}).then(data =>{
// how can I get data fromncallToAnotherAPI() here?
})
}
I want to know that the api has been called and that it is in 'pending' state. How can I achieve this?
Stream is just an async iterator... So we could just use callback, much like node.js
function myPromise(cl) {
cl(null, "pending")
setTimeout(() => {
cl(null, "data")
}, 2000);
}
function myPromise2() {
myPromise((err, data) => {
console.log(data)
})
}
myPromise2()
The most elegant way in my opinion is to return two promises and process them separately.
function myPromise(){
const api1Status = callToAnAPI().then(()=>{
resolve ("pending");
});
return [
api1status,
api1status.then(()=>{
resolve(callToAnotherAPI());
})
.catch(err=>{
// error handling
})
];
}
Then the second file would use it like this:
async myPromise2(){
const [api1, api2] = functionFromOtherFile();
const shouldSayPending = await api1;
const shoudHaveData = await api2;
}
The first function doesn't need to be an async one then, you just return a number of promises.
You could also consider async generators, which would give you a nicer code in the first method, but less nice in the second, like this:
async function* myPromise() {
try {
yield await callToAnApi(); // we need to await here so that
yield callToAnotherApi(); // this method is executed after.
} catch(e) {
// error handling
}
}
The other side would result in something like this:
async myPromise2() {
const progress = theIteratorFromOtherFile(); // just call the function*
const shouldBePending = (await progress.next()).value;
const theOtherResult = (await progress.next()).value;
}
Performance wise there's very little difference between the two - you're doing async operations so these are your bottlenecks. The choice is then up to your personal preference.
If I have the following code
new Promise(res => res(1))
.then(val => console.log(val))
is this equivalent to
let val = await new Promise(res => res(1))
console.log(val)
I know one difference is that I have to wrap the second one in an async function, but otherwise are they equivalent?
Because your promise always resolves (never rejects), they are equivalent. You could also do:
Promise.resolve(1).then(val => console.log(val));
Keep in mind that a major difference with await (besides it needs to be wrapped in an async function) is what happens when the promise rejects. Though your example is hard-wired to resolve, not reject, lets look at what they look like with actual error handling (which you should always have):
new Promise(res => res(1))
.then(val => console.log(val))
.catch(err => console.log(err));
And:
try {
let val = await new Promise(res => res(1));
console.log(val);
} catch(e) {
console.log(err);
}
Or, if you didn't have the try/catch, then any rejects would automatically be sent to the promise that was automatically returned by the async function. This automatic propagation of errors (both synchronous exceptions and asynchronous rejections) is very useful in an async function.
It becomes more apparent when you have multiple asynchronous operations in series:
const fsp = require('fs').promises;
async function someFunc() {
let handle = await fsp.open("myFile.txt", "w");
try {
await handle.write(...);
await handle.write(...);
} finally {
await handle.close().catch(err => console.log(err));
}
}
someFunc().then(() => {
console.log("all done");
}).catch(err => {
console.log(err);
});
Here, the async wrapper catches errors form any of the three await statements and automatically returns them all to the caller. The finally statement catches either of the last two errors in order to close the file handle, but lets the error continue to propagate back to the caller.
I know that promise.all() fails when even 1 of the promise is failed. I only want to try for failed promises and don't want to run promise.all() again.
Any recommendations on how I can achieve this in minimal way?
Promises are eager construct and model a value obtained asynchronously,
a Promise is produced using some kind of producer, like fetch for instance.
If you retain a reference to this producer then you can replay the nmechanism
that produced the Promise in the first place.
// producer function
function getData (arg) {
const result = new Promise();
return result.then(value => {
return { ok:true, value };
}, error => {
return {
ok: false,
value: error,
// retry is a function which calls the producer with the same arguments
retry: () => getData(arg)
};
})
}
Then if you have something like:
const data = [];
// Promise<{ok: boolean, value: any, retry?: function}>
// No promises will fail in this array
const asyncResults = data.map(getResults);
Promise.all(asyncResults)
.then((results) => {
const successes = results.filter(res => res.ok);
const retrys = results.filter(res => !res.ok).map(res => res.retry()); // retry all failed promises
})
Memory leaks, stack overflow: because I retain a reference to original arguments in order to retry and the algorithm is recursive there could be a memory leak. However the algorithm cannot "stack overflow":
getData calls do not get "deeper" over time (see retry definition)
the asyncrhonicity of the algorithm prevent this behaviour if a promise was never resolved
old data is properly discarded when accessing the results as const resultData = results.filter(res => res.ok).map(res => res.value);
However the algorithm could take a long time to settle if a promise keep on not getting resolved and prevent access to the rest of the values.
In an alternative I suggest you take a look at another async primitive, not yet part of the language (maybe some day) : Observables which are designed for this kind of tasks: lazy, retry-able async operations.
You may use async package and wrap all promise calls with closure with done argument.
Then simply resolve results.
const async = require('async');
const promiseAll = promises => {
return new Promise((resolve) => {
// preparing array of functions which has done method as callback
const parallelCalls = promises.map(promise => {
return done => {
promise
.then(result => done(null, result)
.catch(error => {
console.error(error.message);
done();
});
}
});
// calling array of functions in parallel
async.parallel(
parallelCalls,
(_, results) => resolve(results.filter(Boolean))
);
});
};
router.get('/something', async (req, res) => {
...
const results = await promiseAll(promises);
...
});
Or we can simply do Promise.all without using async package:
router.get('/something', async (req, res) => {
...
const results = (
await Promise.all(
promises.map(promise => {
return new Promise(resolve => {
promise.then(resolve).catch(e => resolve());
});
});
)
).filter(Boolean);
...
});