I am struggling to understand how to return a single array from a function that calls another function several times. Currently, the console.log in the code below outputs a growing array each time the scrapingfunction runs.
The final last time that the scrapingfunction runs is actually what I want, but I want to find a way to return a single array at the end of the hello function so that I can drop each object into my database. I'm guessing this is just not me understanding javascript well enough yet.
const hello = async () => {
//[launch puppeteer and navigate to page I want to scrape]
await scrapingfunction(page)
//[navigate to the next page I want to scrape]
await scrapingfunction(page)
//[navigate to the next page I want to scrape]
await scrapingfunction(page)
}
const scrapingfunction = async (page) => {
const html = await page.content()
const $ = cheerio.load(html)
const data = $('.element').map((index, element)=>{
const dateElement = $(element).find('.sub-selement')
const date = dateElement.text()
return {date}
}).get()
console.log(data)
}
hello();
The problem you've encountered is that Promises (which is what async/await uses "under the covers") cannot return values outside the Promise chain.
Think of it this way.
You ask me to write a StackOverflow article for you and immediately demand the result of that task, without waiting for me to finish it.
When you set me the task, I haven't yet completed it, so I cannot provide a response.
You will need to restructure your request to return values from your awaits which can then be operated upon by the surrounding async function, such as:
# Assume doubleValue() takes some unknown time to return a result, like
# waiting for the result of an HTTP query.
const doubleValue = async (val) => return val * 2
const run = async () => {
const result = []
result.push(await doubleValue(2))
result.push(await doubleValue(4))
result.push(await doubleValue(8))
console.log(result)
}
which will print [4, 8, 16] to the console.
You might think you could return the result from run() and print it to the console as in:
const run = async () => {
const result = []
result.push(await doubleValue(2))
result.push(await doubleValue(4))
result.push(await doubleValue(8))
return result
}
console.log(run())
But since Node has no idea when run() has everything it needs to create a result, console.log will not print anything. (That's not expressly true since an async function returns a Promise, but the explanation works for this example.)
The rule is that you can await the result of other functions from within a function marked as async, but you cannot return any useful result to its surrounding context.
Since an async function does return a Promise, you could:
run().then(result => console.log(result))
But note that the result never leaves the Promise chain.
Related
I am trying out a Chrome extension.
How do I get a replaceAll's callback function to use await ?
window.onload = async () =>
{
const elem = document.getElementById('#myID');
const elem_modified = elem.replaceAll(regex, await myFunction);
document.getElementById('#myID').innerHTML = elem_modified;
}
myFunction :
const myFunction = async (str) =>
{
const items = [...str.matchAll(regex)][0];
// Do some manipulation
let amount = number * value;
const amount_modified = await calculate(amount);
console.log("amount_modified = " + amount_modified); // This shows correctly
return items[0] + ' (' + amount_modified + ')';
}
The result is like
over [object Promise] to many people
You don't, replaceAll — like lots of functions that support callbacks — doesn't expect a promise as a return value from the callback. In the vast majority of cases, you can't usefully pass a promise to code that doesn't expect one. (map is one exception to that rule.)
You haven't provided a runnable example or an example of your regular expression, but the general shape of the solution here will be:
Split the string into parts (the segments you want to pass through your async function and the segments between them).
Start the calculation for each of the segments you want to process and get a promise (from your async function) of the result; for the segments in-between, just create a promise fulfilled to the original segment value.
Wait for all of those to complete by using Promise.all, awaiting the promise it provides. (That's assuming it's okay to start processing all the segments at once, rather than waiting for the processing of one to end before starting processing the next.)
Assemble a new string from the updated segments and the segments that used to be between them.
Here's a rough sketch that isolates the numbers in a string and builds a new string with the numbers doubled. This version doesn't, of course, need to be async at all, but you can substitute a truly async operation for calculate:
async function calculate(str) {
await new Promise((resolve) => setTimeout(resolve, 10)); // Just so we do actually do something async
return str ? String(Number(str) * 2) : str;
}
async function example(str) {
const rex1 = /(\d+)/; // Note the capture group, so `split` retains the delimiter between matches
const rex2 = /^\d+$/; // Same as above, but full string match and no capture group
// #1
const parts = str.split(rex1);
// #2 and #3 (the `await`)
const updated = await Promise.all(parts.map(async (part) => {
if (rex2.test(part)) {
return await calculate(part);
}
return part;
}));
// #4
return updated.join("");
}
example("Testing 1 2 3 testing")
.then((result) => console.log(result))
.catch((error) => console.error(error));
Side note: The DOM event system won't pay any attention to the promise the async function you're using as an event handler returns, so it's important not to allow any errors (promise rejections) to terminate that handler, since those rejections will be unhandled.
This question already has answers here:
How do I convert an existing callback API to promises?
(24 answers)
Closed 2 years ago.
I'm building a trading bot that needs to get stock names from separate files. But even I have used async function and await in my code, that doesn't work.
My index file init method.
const init = async () => {
const symbols = await getDownTrendingStock();
console.log("All stocks that are down: " + symbols);
const doOrder = async () => {
//do stuff
}
doOrder();
}
my getDownTrendeingStock file
const downStocks = []
function getDownTrendingStock () {
for(i = 0; i < data.USDTPairs.length; i++){
const USDTPair = data.USDTPairs[i] + "USDT";
binance.prevDay(USDTPair, (error, prevDay, symbol) => {
if(prevDay.priceChangePercent < -2){
downStocks.push(symbol)
}
});
}
return downStocks;
}
I have tried to also use async in for loop because the getDownTrendinStock function returns an empty array before for loop is finished. I didn't find the right way to do that because I was confused with all async, promise and callback stuff. What is the right statement to use in this situation?
Output:
All stocks that are down:
Wanted output:
All stocks that are down: [BTCUSDT, TRXUSDT, ATOMUSDT...]
I think the main issue in the code you posted is that you are mixing callbacks and promises.
This is what's happening in your getDownTrendingStock function:
You start iterating over the data.USDTPairs array, picking the first element
You call binance.prevDay. This does nothing yet, because its an asynchronous function that takes a bit of time. Notably, no data is added to downStocks yet.
You continue doing 1-2, still, no data is added
You return downStocks, which is still empty.
Your function is complete, you print the empty array
Now, at some point, the nodejs event loop continues and starts working on those asynchronous tasks you created earlier by calling binance.prevDay. Internally, it probably calls an API, which takes time; once that call is completed, it calls the function you provided, which pushes data to the downStocks array.
In summary, you didn't wait for the async code to complete. You can achieve this in multiple ways.
One is to wrap this in a promise and then await that promise:
const result= await new Promise((resolve, reject) => {
binance.prevDay(USDTPair, (error, prevDay, symbol) => {
if (error) {
reject(error);
} else {
resolve({prevDay, symbol});
}
});
});
if(result.prevDay.priceChangePercent < -2){
downStocks.push(result.symbol)
}
Note that you can probably also use promisify for this. Also, this means that you will wait for one request to finish before starting the next, which may slow down your code considerably, depending on how many calls you need; you may want to look into Promise.all as well.
Generally speaking, I use two technics:
const asyncFunc = () => {smthAsync};
const arrayToProcess = [];
// 1
const result = await arrayToProcess.reduce((acc, value) => acc.then(() => asyncFunc(value)), Promise.resolve(someInitialValue));
// 2
// here will be eslint errors
for(let i = 0 i < arrayToProcess.length; i+=1) {
const processResult = await asyncFunc(value);
// do with processResult what you want
};
I am new to JavaScript and have been trying to read up a lot on why this is not working. Here is my code. I have also read a number of articles here on stack overflow but still feeling dense
Also if my title does not make sense, please suggest an edit
listRef.listAll()
.then(response => {
let files = []
response.items.forEach(item => {
var text
getText(item.name).then(res=>{text = res});
const id = {uid: guid()}
const url = item.getDownloadURL().then(url => {return url} )
const gsurl = `gs://archivewebsite.appspot.com/${folder}/${item.name}`
files.push({...item, name:item.name, url, gsurl, id:id.uid, text})
});
this.files = files;
})
.catch(error => console.log(error));
async function getText(docID) {
var docRef = firestore.collection("recipes").doc(docID);
let doc = await docRef.get()
if (doc.exists){
return doc.data().text
}
}
that code "works" in that it logs the response to the console but the text variable is a pending promise object.
I understand that async functions return a promise so when I call getText I need to use .then - what I am struggling with and have refactored this code a few times is this:
how can I assign the value of doc.data().text to a variable to be used later in other words, how can var text be an actual string and not a promise object pending
Also for my own learning on javascript inside the async function if I replace
if (doc.exists){
return doc.data().text
}
with
if (doc.exists){
return Promise.resolve(doc.data().text)
}
I get the same result in console.log - is this expected? is return simply short hand for the handler to resolve the promise?
I have also refactored this code to be non async and I get the same result where my var text is basically a pending promise and never the resolved data
Thanks for your help - also any articles to help explain this to me would be great! I have been going through courses on udemy but little confused by this right now
Actually you are assigning the complete promise to the variable text
Replace
var text = getText(item.name).then(res=>console.log(res))
by
var text = await getText(item.name);
OR
var text
getText(item.name).then(res=>{text = res});
Of course text is going to be a Promise. Promise.then() always returns a Promise.
Consider this code:
function doA(n) {
// do something here...
console.log("A" + n);
}
asnyc function doB(n) {
// do something here...
console.log("B" + n);
}
doA(1);
doA(2);
doB(3); // async
doA(4);
doB(5); // async
doB(6); // async
doA(7);
What do you expect the output to be?
1, 2, 4, and 7 will always be in order, because they are executed synchronously.
3 will not ever print before 1 and 2. Likewise, 5 and 6 will not ever print before 1, 2, and 4.
However, 3, 5, and 6 can be printed in any order, because async functions do not guarantee execution order once created.
Also, 4 can print before 3. Likewise, 7 can print before 5 and 6.
Basically, think of async functions as a parallel task that runs independently (although not really; single thread JS only simulates this behavior). It can return (fulfill/reject) at any moment. For this reason, you cannot just simply assign a return value of an async function to a variable using synchronous code - the value is not guaranteed to be (and probably is not) available at the moment of synchronous execution.
Therefore you need to put all the code that requires the value of text to be set into the callback block of the Promise, something like this:
getText(item.name).then((text) => {
// put everything that uses text here
});
This can of course lead to the infamous "callback hell", where you have layers inside layers of async callback. See http://callbackhell.com for details and mitigation techniques.
async/await is just one of the newer ways to do the same thing: MDN has an excellent article here: https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous/Async_await
OK I worked with someone at work and found a solution - it was related to this post
https://stackoverflow.com/a/37576787/5991792
I was using async function inside a for each loop
the refactored code is here
async function buildFiles(){
let items = await listRef.listAll()
let files = []
for (const item of item.items){
const text = await getText(item.name)
const url = await item.getDownloadURL()
const gsurl = `gs://archivewebsite.appspot.com/${folder}/${sermon.name}`
files.push({...item, name:item.name, url, gsurl, text})
}
return files
}
async function getText(docID) {
var docRef = firestore.collection("recipies").doc(docID);
let doc = await docRef.get()
if (doc.exists){return await doc.data().text}}
buildFiles().then(res=>this.files = res)
Thanks also to #cyqsimon and #Nav Kumar V
I have a function which makes two asynchronous calls to the database inside a loop. The issue is that the return function works before the data from the loop is retrieved.
const myFunc = async (customers) => {
const customerList = customers.map(async (customer) => {
const cashCollected = await knex('cash_collections')
.sum('amount_collected as amount')
.where('account_code', customer.code)
.first();
const orderValue = await knex('orders')
.sum('discounted_price as amount')
.where('customer_id', customer.id)
.first();
const customerData = {
name: customer.name,
outstandingBalance: (orderValue.amount - cashCollected.amount),
};
// This line works after console.log(customerList);
console.log(customerData);
return customerData;
});
// console and return works before data is retrieved
// (before console.log(customerData) is run)
console.log(customerList);
return customerList;
};
// Function is called in another place
myFunc()
You're making all of those calls in parallel by doing them in the map callback. If you really want to do that, you need to wait for those calls to settle by using Promise.all:
const customerList = await Promise.all(customers.map(async (customer) => {
// ...
}));
If you want to do them in series instead, use a for loop and await each response. :-) But it looks like parallel is okay.
You need to await the map and then it will wait for it otherwise being async doesn't make it wait it actually means it will go on to the next code unless you tell it to await. like so:
const customerList = await customers.map....
Now since map does not return a promise you would need to wrap it in a promise either with Promise.all or an individual promise.
By default the Promise.All([]) function returns a number based index array that contains the results of each promise.
var promises = [];
promises.push(myFuncAsync1()); //returns 1
promises.push(myFuncAsync1()); //returns 2
Promise.all(promises).then((results)=>{
//results = [0,1]
}
What is the best vanilla way to return a named index of results with Promise.all()?
I tried with a Map, but it returns results in an array this way:
[key1, value1, key2, value2]
UPDATE:
My questions seems unclear, here is why i don't like ordered based index:
it's crappy to maintain: if you add a promise in your code you may have to rewrite the whole results function because the index may have change.
it's awful to read: results[42] (can be fixed with jib's answer below)
Not really usable in a dynamic context:
var promises = [];
if(...)
promises.push(...);
else{
[...].forEach(... => {
if(...)
promises.push(...);
else
[...].forEach(... => {
promises.push(...);
});
});
}
Promise.all(promises).then((resultsArr)=>{
/*Here i am basically fucked without clear named results
that dont rely on promises' ordering in the array */
});
ES6 supports destructuring, so if you just want to name the results you can write:
var myFuncAsync1 = () => Promise.resolve(1);
var myFuncAsync2 = () => Promise.resolve(2);
Promise.all([myFuncAsync1(), myFuncAsync2()])
.then(([result1, result2]) => console.log(result1 +" and "+ result2)) //1 and 2
.catch(e => console.error(e));
Works in Firefox and Chrome now.
Is this the kind of thing?
var promises = [];
promises.push(myFuncAsync1().then(r => ({name : "func1", result : r})));
promises.push(myFuncAsync1().then(r => ({name : "func2", result : r})));
Promise.all(promises).then(results => {
var lookup = results.reduce((prev, curr) => {
prev[curr.name] = curr.result;
return prev;
}, {});
var firstResult = lookup["func1"];
var secondResult = lookup["func2"];
}
If you don't want to modify the format of result objects, here is a helper function that allows assigning a name to each entry to access it later.
const allNamed = (nameToPromise) => {
const entries = Object.entries(nameToPromise);
return Promise.all(entries.map(e => e[1]))
.then(results => {
const nameToResult = {};
for (let i = 0; i < results.length; ++i) {
const name = entries[i][0];
nameToResult[name] = results[i];
}
return nameToResult;
});
};
Usage:
var lookup = await allNamed({
rootStatus: fetch('https://stackoverflow.com/').then(rs => rs.status),
badRouteStatus: fetch('https://stackoverflow.com/badRoute').then(rs => rs.status),
});
var firstResult = lookup.rootStatus; // = 200
var secondResult = lookup.badRouteStatus; // = 404
If you are using typescript you can even specify relationship between input keys and results using keyof construct:
type ThenArg<T> = T extends PromiseLike<infer U> ? U : T;
export const allNamed = <
T extends Record<string, Promise<any>>,
TResolved extends {[P in keyof T]: ThenArg<T[P]>}
>(nameToPromise: T): Promise<TResolved> => {
const entries = Object.entries(nameToPromise);
return Promise.all(entries.map(e => e[1]))
.then(results => {
const nameToResult: TResolved = <any>{};
for (let i = 0; i < results.length; ++i) {
const name: keyof T = entries[i][0];
nameToResult[name] = results[i];
}
return nameToResult;
});
};
A great solution for this is to use async await. Not exactly ES6 like you asked, but ES8! But since Babel supports it fully, here we go:
You can avoid using only the array index by using async/await as follows.
This async function allows you to literally halt your code inside of it by allowing you to use the await keyword inside of the function, placing it before a promise. As as an async function encounters await on a promise that hasn't yet been resolved, the function immediately returns a pending promise. This returned promise resolves as soon as the function actually finishes later on. The function will only resume when the previously awaited promise is resolved, during which it will resolve the entire await Promise statement to the return value of that Promise, allowing you to put it inside of a variable. This effectively allows you to halt your code without blocking the thread. It's a great way to handle asynchronous stuff in JavaScript in general, because it makes your code more chronological and therefore easier to reason about:
async function resolvePromiseObject(promiseObject) {
await Promise.all(Object.values(promiseObject));
const ret = {};
for ([key, value] of Object.entries(promiseObject)) {
// All these resolve instantly due to the previous await
ret[key] = await value;
};
return ret;
}
As with anything above ES5: Please make sure that Babel is configured correctly so that users on older browsers can run your code without issue. You can make async await work flawlessly on even IE11, as long as your babel configuration is right.
in regards to #kragovip's answer, the reason you want to avoid that is shown here:
https://medium.com/front-end-weekly/async-await-is-not-about-making-asynchronous-code-synchronous-ba5937a0c11e
"...it’s really easy to get used to await all of your network and I/O calls.
However, you should be careful when using it multiple times in a row as the await keyword stops execution of all the code after it. (Exactly as it would be in synchronous code)"
Bad Example (DONT FOLLOW)
async function processData() {
const data1 = await downloadFromService1();
const data2 = await downloadFromService2();
const data3 = await downloadFromService3();
...
}
"There is also absolutely no need to wait for the completion of first request as none of other requests depend on its result.
We would like to have requests sent in parallel and wait for all of them to finish simultaneously. This is where the power of asynchronous event-driven programming lies.
To fix this we can use Promise.all() method. We save Promises from async function calls to variables, combine them to an array and await them all at once."
Instead
async function processData() {
const promise1 = downloadFromService1();
const promise2 = downloadFromService2();
const promise3 = downloadFromService3();
const allResults = await Promise.all([promise1, promise2, promise3]);