In my Node.js app I'm trying to use a helper function to gather and format some data to ultimately be returned via an API endpoint. I need to loop through an array, and for each entry, make an asynchronous call to my database. However, I'm unable to return this new array of data in my route once I map over it. I believe that this has something to do with the nature of 'async.map' but have been unable to figure it out from the docs. Am I missing something? Better way to do this?
router.get('/route',async (req,res) => {
const formattedData = await module.formatData(data);
// formattedData IS RETURNED AS UNDEFINED HERE
res.json({data:formattedData});
};
Relevant helper functions:
formatData: async(data) => {
// gets entire object from the id that I originally had
getNewData: (dataID) => {
const data = Data.find({_id:dataID},(error,response)=>{
return response;
})
return data;
},
const formatDataHelper = async(data,done) =>{
// THIS function queries database (reason I have to use async)
const newData = await getNewData(data);
done(null, newData);
}
function dataMap(data, callback) {
async.map(data, formatDataHelper, callback);
}
const returnData = await dataMap(data,(err,response)=>{
return response;
})
return returnData;
},
await only awaits something when you await a function that returns a promise that is linked to whatever you want to await. Your dataMap() function returns nothing, therefore await on that function doesn't await anything.
So, in order to work with await, your dataMap() function needs to return a promise that is tied to the asynchronous operation it is doing.
Related
In node.js I am trying to get a list of bid and ask prices from a trade exchange website (within an async function). Within the foreach statement I can console.info() the object data(on each iteration) but when I put all of this into an array and then return it to another function it passes as 'undefined'.
const symbolPrice = async() => {
let symbolObj = {}
let symbolList = []
await bookTickers((error, ticker) => {
ticker.forEach(symbol => {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolObj = {
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
}
console.info(symbolObj);
}
symbolList.push(symbolObj)
});
const results = Promise.all(symbolList)
return results;
});
}
const symbolPriceTest = async() => {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
I have tried pretty much everything I can find on the internet like different awaits/Promise.all()'s. I do admit I am not as familiar with async coding as I would like to be.
So, if the basic problem here is to call bookTickers(), retrieve the asynchronous result from that call, process it and then get that processed result as the resolved value from calling symbolPrice(), then you can do that like this:
const { promisify } = require('util');
const bookTickersP = promisify(bookTickers);
async function symbolPrice(/* declare arguments here */) {
let symbolList = [];
const ticker = await bookTickersP(/* fill in arguments here */);
for (let symbol of ticker) {
if (symbol.symbol.toUpperCase().startsWith(starts.toUpperCase())) {
symbolList.push({
symbol: symbol.symbol,
bid: symbol.bidPrice,
ask: symbol.askPrice
});
}
}
return symbolList;
}
async function symbolPriceTest() {
const res = await symbolPrice(null, 'ETH', true);
console.log(res)
}
Things to learn from your original attempt:
Only use await when you are awaiting a promise.
Only use Promise.all() when you are passing it an array of promises (or an array of a mixture of values and promises).
Don't mix plain callback asynchronous functions with promises. If you don't have a promise-returning version of your asynchronous function, then promisify it so you do (as shown in my code above with bookTickersP().
Do not guess with async and await and just throw it somewhere hoping it will do something useful. You MUST know that you're awaiting a promise that is connected to the result you're after.
Don't reuse variables in a loop.
Your original implementation of symbolPrice() had no return value at the top level of the function (the only return value was inside a callback so that just returns from the callback, not from the main function). That's why symbolPrice() didn't return anything. Now, because you were using an asynchronous callback, you couldn't actually directly return the results anyway so other things had to be redone.
Just a few thoughts on organization that might be reused in other contexts...
Promisify book tickers (with a library, or pure js using the following pattern). This is just the api made modern:
async function bookTickersP() {
return new Promise((resolve, reject) => {
bookTickers((error, ticker) => {
error ? reject(error) : resolve(ticker);
})
});
}
Use that to shape data in the way that the app needs. This is your app's async model getter:
// resolves to [{ symbol, bid, ask }, ...]
async function getSymbols() {
const result = await bookTickersP();
return result.map(({ symbol, bidPrice, askPrice }) => {
return { symbol: symbol.toUpperCase(), bid: bidPrice, ask: askPrice }
});
}
Write business logic that does things with the model, like ask about particular symbols:
async function symbolPriceTest(search) {
const prefix = search.toUpperCase();
const symbols = await getSymbols();
return symbols.filter(s => s.symbol.startsWith(prefix));
}
symbolPriceTest('ETH').then(console.log);
I am querying AMPS SOW using javascript API. My functions look like this:
sow_query = async(topic, filter, options) => {
await this.connection
await this.client.sow(this.onMessage, topic, filter, options)
return this.message
}
onMessage = (message)=>{
this.message.push(message.data)
}
//Calling functions
const getData = async(date) =>{
let data = await getAmpsData(date)
}
async getAmpsData(date){
const filter = "some filter"
const data = await ampsClient.sow_query(topic, filter)
data.forEach((element) => console.log(element))
}
It looks like my call to ampsClient.sow_query doesnt wait for the callback to finish so returns an empty list. Therefore, I cannot loop through the data in my calling function getAmpsData.
I know it has to do with async/await because if I simply do console.log(data) in getAmpsData i can see the data (perhaps after 2 seconds) when the promise is resolved. Is there anyway i can get this working?
If I understand you correctly, data in getAmpsData is an array as expected, but data in getData is undefined. It's not an async/await problem, you just forgot to add return data; to getAmpsData.
I not sure about what package you are using. But, maybe, it using a callback function to get the result of .sow function - message.data.
With your logic, onMessage function will be called after data.forEach done, you can try adding a console.log line to onMessage function.
Maybe, the package has an important reason to do that. But, to fix this issue, you can wrap .sow function into a promise.
sow_query = async (topic, filter, options) => {
await this.connection // ???
return new Promise((resolve) => { // wrap in a promise
this.client.sow( // no need to wait .sow
(message) => {
resolve(message.data); // just resolve when we get the message's data
},
topic, filter, options)
});
}
//Calling functions
const getData = async (date) => {
let data = await getAmpsData(date)
}
async getAmpsData(date) {
const filter = "some filter"
const data = await ampsClient.sow_query(topic, filter)
data.forEach((element) => console.log(element))
}
I've been struggling for the past four hours on this. I am writing a function to see if a property named "data" exists in my Firebase storage. If it does, I want to do one thing, if it doesn't I want to do something else. However, I cannot figure out how this asynchronous stuff works for the life of me. I simplified my code below. Basically I just want to wait for the data to be fetched before I reach the if/else. I've been playing around with different options but keep getting errors or some other issue. The code below is the closest I've gotten to working where the code doesn't crash but even if "data" does not exist in the Firestore, I'm always going through the else clause and I don't know why. Can someone help me figure out what I am doing wrong?
const fetchDataFromDB = async (user) => {
let query = db
.collection("users")
.doc(user)
.get()
.then((doc) => {
doc.data();
console.log(doc.data().data);
});
return await query;
};
export const getSchedule = (miles, user) => {
const firebaseData = fetchDataFromDB(user);
console.log(firebaseData);
// WAIT FOR FETCH BEFORE CONTINUING
if (!firebaseData) {
console.log("NOT getting data from FB");
return;
} else {
console.log("getting data from FB");
return;
}
};
Change up the code as follows:
const fetchDataFromDB = (user) => {
return db
.collection("users")
.doc(user)
.get()
.then((doc) => {
const data = doc.data();
console.log(data);
return data;
});
};
export const getSchedule = async (miles, user) => {
const firebaseData = await fetchDataFromDB(user);
console.log(firebaseData);
if (!firebaseData) {
console.log("NOT getting data from FB");
return;
} else {
console.log("getting data from FB");
return;
}
};
The point to remember about async await is that it doesn't really make asynchronous calls synchronous, it just makes them look that way so that your code is a bit less unwieldy with wrapped promises and the like. Every async function returns a promise, so if you want to deal with what it returns, you need to either deal with the promise directly (using .then...), or by using another await. In the latter case, you of course need to declare the consuming function as async as well.
With regards to the first function, there's no need for the async await there. Just return the promise. (Thanks #charlieftl for pointing out the problem in that code)
I used Papa Parse to parse a .csv file, and pushed the result to an empty array called parsed_data. I am able to use console.log(parsed_data), and view the arrays produced. However, when I try to index the data, for example, console.log(parsed_data[0]), the result is undefined. Not sure what's going wrong here.
Example code:
let parsed_data = [];
const data_url = "acdata.csv";
async function getData() {
const response = await fetch(data_url);
const blob = await response.blob();
const data = Papa.parse(blob, {
complete: function(results) {
//console.log("Finished:", results.data);
parsed_data.push(results.data);
}
});
};
console.log(parsed_data);
getData();
Since Papa parse complete callback is called asynchronously, you'll need to wait for that to complete - however, papa parse doesn't seem to use Promises, so, you can "promisify" the parse function like so
const papaParsePromise = blob => new Promise(resolve => Papa.parse(blob, { complete: resolve }));
Another way of looking at that function, if you don't understand the => notation, is
function papaParsePromise(blob) {
return new Promise(function(resolve) {
Papa.parse(blob, {
complete: function(data) {
resolve(data);
}
);
});
}
that returns a promise that resolves to the data that is passed to the complete callback
Your code would also need to wait for the promise returned by getData before it can use anything in that data. Unless your code is inside another async function, you'll need to use promise .then method, as below
const data_url = "acdata.csv";
async function getData() {
// create a function that returns a Promise that resolves when papa parse complete is called
const papaParsePromise = blob => new Promise(resolve => Papa.parse(blob, { complete: resolve }));
const response = await fetch(data_url);
const blob = await response.blob();
const data = await papaParsePromise(blob);
return data;
};
getData()
.then(parse_data => {
console.log(parsed_data);
console.log(parsed_data[0]);
// i.e. do what you need with parsed_data here
});
if, however, you are calling getData inside an async function - along with the changes to getData above, you can simply use await getData() to wait for the value - i.e.
async function someFunction() {
const parsed_data = await getData();
// do what you need with parsed_data here
}
was following this tutorial(https://javascript.info/fetch) on javascript's promise and async await and had trouble understanding the exercise it provided.
The question is about retrieving multiple users info from Github. One fetch request per user.
And requests shouldn’t wait for each other. So that the data arrives as soon as possible.
The solution it provided is
async function getUsers(names) {
let jobs = [];
for(let name of names) {
let job = fetch(`https://api.github.com/users/${name}`).then(
successResponse => {
if (successResponse.status != 200) {
return null;
} else {
return successResponse.json();
}
},
failResponse => {
return null;
}
);
jobs.push(job);
}
let results = await Promise.all(jobs);
return results;
}
My first question is, can we use await for the fetch. i.e. is the following snippet equivalent to the solution he provided?
async function getUsers2(names) {
let jobs = [];
for(let name of names) {
let response
try {
response = await fetch(`https://api.github.com/users/${name}`);
} catch(e) {
response = null
}
const job = response && response.json()
jobs.push(job);
}
let results = await Promise.all(jobs);
return results;
}
Furthermore, the tutorial said
.then call is attached directly to fetch, so that when we have the response, it doesn’t wait for other fetches, but starts to read .json() immediately.
If we used await Promise.all(names.map(name => fetch(...))), and call .json() on the results, then it would wait for all fetches to respond. By adding .json() directly to each fetch, we ensure that individual fetches start reading data as JSON without waiting for each other.
Does he mean that if we write the solution this way
async function getUser(name) {
const response = await fetch(`https://api.github.com/users/${name}`)
return response.ok ? await response.json : null
}
async function getUsers(names) {
return await Promise.all(names.map(name => getUser(name)))
}
we wouldn't be able to achieve the effect such that we don't want requests shouldn’t wait for each other?
My first question is, can we use await for the fetch. i.e. is the following snippet equivalent to the solution he provided?
No. When in the immediate body of an async function, whenever there's an await, the function will completely pause until the following Promise resolves. So, the loop
for(let name of names) {
let response
try {
response = await fetch(`https://api.github.com/users/${name}`);
} catch(e) {
response = null
}
needs to wait in serial for the headers of each response to be received before continuing on to initialize the next request.
Does he mean that if we write the solution this way
First, the syntax needs to be adjusted: .json is a method, so it needs to be called:
async function getUser(name) {
const response = await fetch(`https://api.github.com/users/${name}`)
return response.ok ? await response.json() : null
// ^^
}
But that's perfectly fine to do. The only await in the getUsers function is waiting for the whole Promise.all to resolve; the .mapping of the array to the getUser call is carried out synchronously, so all requests get sent out at once, so none of the network requests need to wait for any of the others to finish in order to work.
The problem the author was referring to was calling Promise.all on an array of the fetch calls, rather than on an array of the .json() calls:
// Bad, do not use:
const getUsers = async (names) => {
const responses = await Promise.all(names.map(
name => fetch(`https://api.github.com/users/${name}`)
));
return Promise.all(responses.map(
response => response.ok ? response.json() : null
));
}
The problem with the above is that the script must wait for all of the response headers from every request to be received before the response body for any of them can start to be parsed.
Another parallel solution:
const getUsers = names => Promise.all(names.map(
async (name) => {
const res = await fetch(`https://api.github.com/users/${name}`);
return res.ok ? res.json() : null;
}
));