How can I stream data using promises.
I have two different functions in two different files. In one calls an API service and that returns a promise.
async myPromise(){
return new Promise((resolve, reject) => {
callToAnAPI().then(()=>{
resolve ("pending");
}).then(()=>{
resolve(callToAnotherAPI());
})
.catch(err=>{
// error handling
});
});
}
In another file I have a function like so:
async myPromise2(){
functionFromOtherFile().then((data)=>{
// how can I get 'pending' here?
}).then(data =>{
// how can I get data fromncallToAnotherAPI() here?
})
}
I want to know that the api has been called and that it is in 'pending' state. How can I achieve this?
Stream is just an async iterator... So we could just use callback, much like node.js
function myPromise(cl) {
cl(null, "pending")
setTimeout(() => {
cl(null, "data")
}, 2000);
}
function myPromise2() {
myPromise((err, data) => {
console.log(data)
})
}
myPromise2()
The most elegant way in my opinion is to return two promises and process them separately.
function myPromise(){
const api1Status = callToAnAPI().then(()=>{
resolve ("pending");
});
return [
api1status,
api1status.then(()=>{
resolve(callToAnotherAPI());
})
.catch(err=>{
// error handling
})
];
}
Then the second file would use it like this:
async myPromise2(){
const [api1, api2] = functionFromOtherFile();
const shouldSayPending = await api1;
const shoudHaveData = await api2;
}
The first function doesn't need to be an async one then, you just return a number of promises.
You could also consider async generators, which would give you a nicer code in the first method, but less nice in the second, like this:
async function* myPromise() {
try {
yield await callToAnApi(); // we need to await here so that
yield callToAnotherApi(); // this method is executed after.
} catch(e) {
// error handling
}
}
The other side would result in something like this:
async myPromise2() {
const progress = theIteratorFromOtherFile(); // just call the function*
const shouldBePending = (await progress.next()).value;
const theOtherResult = (await progress.next()).value;
}
Performance wise there's very little difference between the two - you're doing async operations so these are your bottlenecks. The choice is then up to your personal preference.
Related
in an async function i, want to main function to return only in a specific point and not to return void at the end of the function
const fun = () => {
const list = [];
let streamFinished = 0;
let streamCount = files.length;
await fs.readdir(JSON_DIR, async(err, files) => {
await files.forEach((filename) => {
const readStream = fs.createReadStream(path.join("directory", filename));
const parseStream = json.createParseStream();
await parseStream.on('data', async(hostlist: Host[]) => {
hostlist.forEach(async host => {
list.push(host);
});
});
parseStream.on('end', () => {
streamFinished++;
if (streamFinished === streamCount) {
// End of all streams...
return list; //this should be the only point when the function return something
}
})
readStream.pipe(parseStream);
})
});
};
There are lots of things wrong with this code. It's a big mix of event-driven streams, callback-driven functions and promises. The first order of business is to make the main business logic just be promise-driven (as you can see in the new parseData() function and switch all control flow outside of that to just use promises. Here's one way to do it:
const fsp = fs.promises;
function parseData(filename) {
return new Promise((resolve, reject) => {
const readStream = fs.createReadStream(path.join("directory", filename));
const parseStream = json.createParseStream();
const list = [];
parseStream.on('data', (hostlist: Host[]) => {
list.push(...hostlist);
}).on('end', () => {
resolve(list);
}).on('error', reject);
readStream.pipe(parseStream);
});
}
const fun = async () => {
const list = [];
const files = await fsp.readdir(JSON_DIR);
for (let filename of files) {
const listData = await parseData(filename);
list.push(...listData);
}
return list;
};
fun().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});
A few thoughts here:
The easiest way to "wait" for a stream operation to complete is to encapsulate it in a promise which you can then use await or .then() with. That's the purpose of the parseData() function. In addition, this also enables error handling by hooking stream errors to the promise rejection.
For processing a loop of asynchronous operations, you can either do them one at a time, using await on each asynchronous operation in the loop or you can run them in parallel by collecting an array of promises and using let data = await Promise.all(arrayOfPromises); on that array of promises.
It is only useful to use await if the thing you're awaiting is a promise that is connected to your asynchronous operation. So, things like await parseStream.on('data', ...) and await files.forEach(...) are pointless because neither of those return promises. Do NOT just stick await in places unless you KNOW you are awaiting a promise.
You will generally NOT want to use .forEach() with an asynchronous operation in the loop. Because .forEach() has no return value and no loop control, you can't really control much. Use a regular for loop instead which gives you full control. I consider .forEach() pretty much obsolete for asynchronous programming.
Don't mix promises and callbacks and don't mix promises and streams. If you have those, then "promisify" the stream or callback so all your main logic/control flow can be promises. This will vastly simplify your code and make error handling possible/practical.
In node.js I am trying to get a list of bid and ask prices from a trade exchange website (within an async function). Within the foreach statement I can console.info() the object data(on each iteration) but when I put all of this into an array and then return it to another function it passes as 'undefined'.
const symbolPrice = async() => {
let symbolObj = {}
let symbolList = []
await bookTickers((error, ticker) => {
ticker.forEach(symbol => {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolObj = {
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
}
console.info(symbolObj);
}
symbolList.push(symbolObj)
});
const results = Promise.all(symbolList)
return results;
});
}
const symbolPriceTest = async() => {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
I have tried pretty much everything I can find on the internet like different awaits/Promise.all()'s. I do admit I am not as familiar with async coding as I would like to be.
So, if the basic problem here is to call bookTickers(), retrieve the asynchronous result from that call, process it and then get that processed result as the resolved value from calling symbolPrice(), then you can do that like this:
const { promisify } = require('util');
const bookTickersP = promisify(bookTickers);
async function symbolPrice(/* declare arguments here */) {
let symbolList = [];
const ticker = await bookTickersP(/* fill in arguments here */);
for (let symbol of ticker) {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolList.push({
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
});
}
}
return symbolList;
}
async function symbolPriceTest() {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
Things to learn from your original attempt:
Only use await when you are awaiting a promise.
Only use Promise.all() when you are passing it an array of promises (or an array of a mixture of values and promises).
Don't mix plain callback asynchronous functions with promises. If you don't have a promise-returning version of your asynchronous function, then promisify it so you do (as shown in my code above with bookTickersP().
Do not guess with async and await and just throw it somewhere hoping it will do something useful. You MUST know that you're awaiting a promise that is connected to the result you're after.
Don't reuse variables in a loop.
Your original implementation of symbolPrice() had no return value at the top level of the function (the only return value was inside a callback so that just returns from the callback, not from the main function). That's why symbolPrice() didn't return anything. Now, because you were using an asynchronous callback, you couldn't actually directly return the results anyway so other things had to be redone.
Just a few thoughts on organization that might be reused in other contexts...
Promisify book tickers (with a library, or pure js using the following pattern). This is just the api made modern:
async function bookTickersP() {
return new Promise((resolve, reject) => {
bookTickers((error, ticker) => {
error ? reject(error) : resolve(ticker);
})
});
}
Use that to shape data in the way that the app needs. This is your app's async model getter:
// resolves to [{ symbol, bid, ask }, ...]
async function getSymbols() {
const result = await bookTickersP();
return result.map(({ symbol, bidPrice, askPrice }) => {
return { symbol: symbol.toUpperCase(), bid: bidPrice, ask: askPrice }
});
}
Write business logic that does things with the model, like ask about particular symbols:
async function symbolPriceTest(search) {
const prefix = search.toUpperCase();
const symbols = await getSymbols();
return symbols.filter(s => s.symbol.startsWith(prefix));
}
symbolPriceTest('ETH').then(console.log);
On the way of learning the concepts of asynchronous JavaScript, I got struggled with the idea behind the situation when they can be chained. As an example consider the following situation: a webhook calls a cloud function and as a requirement there is set a time interval by which the cloud function should response to the webhook. In the cloud function is called an operation for fetching some data from a database, which can be short- or long-running task. For this reason we may want to return a promise to the webhook just to "register" a future activity and later provide results from it e.g.:
async function main () {
return new Promise ((resolve, reject) => {
try {
db.getSomeData().then(data=> {
resolve({ result: data});
});
} catch (err) {
reject({ error: err.message });
}
});
}
Another way is to use async/await instead of a promise for fetching the data, e.g.:
async function main () {
return new Promise (async (resolve, reject) => {
try {
const data = await db.getSomeData();
resolve({ result: data });
} catch (err) {
reject({ error: err.message });
}
});
}
(The code is a pseudo-code and therefore all checks on the returned types are not considered as important.)
Does the await block the code in the second example and prevent returning of the promise?
async functions always return a Promise. It is up to the function caller to determine how to handle it: either by chaining with then/catch or using await. In your example, there is no need to create and return a new Promise.
You could do something like this for example:
async function main () {
try {
const data = await db.getSomeData()
return {result: data}
} catch (err) {
return {error: err.message}
}
}
// On some other part of the code, you could handle this function
// in one of the following ways:
//
// 1.
// Chaining the Promise with `then/catch`.
main()
.then((result) => console.log(result)) // {result: data}
.catch((err) => console.log(err)) // {error: err.message}
// 2.
// Using await (provided we are working inside an asyn function)
try {
const result = await main()
console.log(result) // {result: data}
} catch (err) {
console.log(err) // {error: err.message}
}
Try to avoid combining both methods of handling asynchronous operations as a best practice.
I'm trying to understand how to chain two different API calls including one with a for loop in a 'notes' Vue component. I have a really basic experience of promises and I'm looking to improve.
I'm making a first API call to get all the notes and pushing them into an array using a Vuex mutation. During that first API call I'm also mapping the different users emails into an Object.
Using this mapped object, I'm making a second API call inside a for loop to get all the users avatars.
Here's what the first API call looks like :
getAllNotesAPI(entity) {
noteService.getNotes(entity)
.then((response) => {
if (response.data.length === '0') {
// Set hasData to false if the response is 0
this.hasData = false;
} else {
// Push data into the note array using a store mutation
this.setAllNotes(response.data);
}
// Mapping all users emails into 'userEmails'
this.userEmails = [...new Set(response.data.map(x => x.userEmail))];
// Calling my second API call here to get all the avatars associated with these emails
for (let i = 0; i < this.userEmails.length; i++) {
this.getAvatarAPI(this.userEmails[i])
}
})
.catch((error) => {
console.log(error);
})
.finally(() => {
this.endLoader('notes');
});
},
this.getAvatarAPI is the second API call which looks like this :
getAvatarAPI(login) {
userService.getAvatar(login)
.then((response) => {
let newAvatar = {
userEmail: login,
picture: response.data.picture
};
// Push the response into a userAvatar Object using a store mutation
this.setUserAvatar(newAvatar);
}).catch((error) => {
console.log(error)
})
},
I've tried using async / await but couldn't figure out how to bind this inside of an async function (this.getAvatarAPI(this.userEmails)) was undefined, I've tried chaining using multiples then but couldn't figure out how to : get all my notes then all my avatars then end the 'note' loader once both those API calls are done.
If any of you could give me some pointers or the beginning of an answer that would be truly appreciated !
First whilst not related to your problem, avoid for loop when non necessary:
Do you need the i index?
for (let i = 0; i < this.userEmails.length; i++) {
this.getAvatarAPI(this.userEmails[i])
}
no. You need the userMail. Then
this.userEmails.forEach(userMail => {
this.getAvatarAPI(userEmail)
})
Now, to synchronize promises, you need to return a promise (let's not talk about async yet)
make getAvatarAPI return a promise
getAvatarAPI(login) {
return userService.getAvatar(login).then(blabla) // notice the return here
retrieve the promises of getAvatar API
let promises = this.userEmails.map(userMail => {
return getAvatarAPI(userEmail)
})
return after all promises have fulfilled
let promises = this.userEmails.map(userMail => {
return getAvatarAPI(userEmail)
})
return Promise.all(promises)
On a side note with async/await
If you use it you are not forced anymore to write return, you need to write async/await though
The underlying idea stay the same. Specifying the async keywords says that your function will return a promise-like.
e.g
async function p () {
return 5
}
p.then(x => console.log(x)) // does print 5 even though we didn't explicitely write return Promise.resolve(5)
Now you have to ensure you await the async function when you call it:
getAvatarAPI: async login => {
return userService.getAvatar(login).then(blabla)
}
// DO NOT do it
this.userEmails.forEach(userMail => {
return await this.getAvatarAPI(userEmail)
})
In forEach loop above, you will do your getAvatarAPI call in sequence because await "stops" iterating as long as getAvatarAPI has not resolved.
The proper way would be
getAllNotesAPI: async entity => {
try { // notice the necesary try-catch block
const response = await noteService.getNotes(entity)
blabla
let promises = this.userEmails.map(userMail => {
return this.getA...
})
let result = await Promise.all(promises)
// eventually return result, or simply await Promise... without lefthand-side assignment
} catch (error) {
console.log(error);
}
console.log(this.end('loader'))
}
I have the following functions with promises:
const ajaxRequest = (url) => {
return new Promise(function(resolve, reject) {
axios.get(url)
.then((response) => {
//console.log(response);
resolve(response);
})
.catch((error) => {
//console.log(error);
reject();
});
});
}
const xmlParser = (xml) => {
let { data } = xml;
return new Promise(function(resolve, reject) {
let parser = new DOMParser();
let xmlDoc = parser.parseFromString(data,"text/xml");
if (xmlDoc.getElementsByTagName("AdTitle").length > 0) {
let string = xmlDoc.getElementsByTagName("AdTitle")[0].childNodes[0].nodeValue;
resolve(string);
} else {
reject();
}
});
}
I'm trying to apply those functions for each object in array of JSON:
const array = [{"id": 1, "url": "www.link1.com"}, {"id": 1, "url": "www.link2.com"}]
I came up with the following solution:
function example() {
_.forEach(array, function(value) {
ajaxRequest(value.url)
.then(response => {
xmlParser(response)
.catch(err => {
console.log(err);
});
});
}
}
I was wondering if this solution is acceptable regarding 2 things:
Is it a good practice to apply forEach() on promises in the following matter.
Are there any better ways to pass previous promise results as parameter in then() chain? (I'm passing response param).
You can use .reduce() to access previous Promise.
function example() {
return array.reduce((promise, value) =>
// `prev` is either initial `Promise` value or previous `Promise` value
promise.then(prev =>
ajaxRequest(value.url).then(response => xmlParser(response))
)
, Promise.resolve())
}
// though note, no value is passed to `reject()` at `Promise` constructor calls
example().catch(err => console.log(err));
Note, Promise constructor is not necessary at ajaxRequest function.
const ajaxRequest = (url) =>
axios.get(url)
.then((response) => {
//console.log(response);
return response;
})
.catch((error) => {
//console.log(error);
});
The only issue with the code you provided is that result from xmlParser is lost, forEach loop just iterates but does not store results. To keep results you will need to use Array.map which will get Promise as a result, and then Promise.all to wait and get all results into array.
I suggest to use async/await from ES2017 which simplifies dealing with promises. Since provided code already using arrow functions, which would require transpiling for older browsers compatibility, you can add transpiling plugin to support ES2017.
In this case your code would be like:
function example() {
return Promise.all([
array.map(async (value) => {
try {
const response = await ajaxRequest(value.url);
return xmlParser(response);
} catch(err) {
console.error(err);
}
})
])
}
Above code will run all requests in parallel and return results when all requests finish. You may also want to fire and process requests one by one, this will also provide access to previous promise result if that was your question:
async function example(processResult) {
for(value of array) {
let result;
try {
// here result has value from previous parsed ajaxRequest.
const response = await ajaxRequest(value.url);
result = await xmlParser(response);
await processResult(result);
} catch(err) {
console.error(err);
}
}
}
Another solution is using Promise.all for doing this, i think is a better solution than looping arround the ajax requests.
const array = [{"id": 1, "url": "www.link1.com"}, {"id": 1, "url": "www.link2.com"}]
function example() {
return Promise.all(array.map(x => ajaxRequest(x.url)))
.then(results => {
return Promise.all(results.map(data => xmlParser(data)));
});
}
example().then(parsed => {
console.log(parsed); // will be an array of xmlParsed elements
});
Are there any better ways to pass previous promise results as
parameter in then() chain?
In fact, you can chain and resolve promises in any order and any place of code. One general rule - any chained promise with then or catch branch is just new promise, which should be chained later.
But there are no limitations. With using loops, most common solution is reduce left-side foldl, but you also can use simple let-variable with reassign with new promise.
For example, you can even design delayed promises chain:
function delayedChain() {
let resolver = null
let flow = new Promise(resolve => (resolver = resolve));
for(let i=0; i<100500; i++) {
flow = flow.then(() => {
// some loop action
})
}
return () => {
resolver();
return flow;
}
}
(delayedChain())().then((result) => {
console.log(result)
})