I'm new to JS asynchronous and I have a question about: how to start working on created array only if all queries are done. I fetch pages in for loop. That's my code:
var allOrgReposData = [];
var repoContributorsUrls = [];
for (var i=1; i <= orgPageIterations; i++) {
var orgReposUrl = 'https://api.github.com/orgs/angular/repos?page='+i;
fetch(orgReposUrl)
.then(response => response.json())
.then(orgReposData => {
allOrgReposData = allOrgReposData.concat(orgReposData);
console.log(allOrgReposData);
})
}
As You can see the allOrgReposData array is created on for loop, but If I try to do something on this array, script do It on every iteration so my operations are multipicated instead execution single time for exapmle (30 item on page): 30; 60; 90; 120; 150; 171 = 621 instead 171.
Is It possible to "wait" for finish fetching and get access to array without this. "multiplication"?
Greetings!
You can use Promise.all which will wait until all promises are complete:
var allOrgReposData = [];
var repoContributorsUrls = [];
var promises = [];
let orgPageIterations = 1;
for (var i = 1; i <= orgPageIterations; i++) {
let orgReposUrl = 'https://api.github.com/orgs/angular/repos?page=' + i;
promises.push(fetch(orgReposUrl).then(response => response.json()));
}
Promise.all(promises)
.then(data => {
allOrgReposData = data;
console.log(allOrgReposData);
})
.catch(err => console.error(err));
Please note that I've also changed var orgReposUrl to let orgReposUrl to make us of block scoping.
You could keep track of the number of calls you did with a variable :
var allOrgReposData = [];
var repoContributorsUrls = [];
var callSuccess = 1; //Variable keeping track of your ajax calls
for (var i=1; i <= orgPageIterations; i++) {
var orgReposUrl = 'https://api.github.com/orgs/angular/repos?page='+i;
fetch(orgReposUrl)
.then(response => response.json())
.then(orgReposData => {
allOrgReposData = allOrgReposData.concat(orgReposData);
console.log(allOrgReposData);
callSuccess++; //Increment your var for each call
if(callSuccess == orgPageIterations){ //If every call has already been made, then continue
//DO YOUR THING HERE
}
})
}
Related
I have an array candleRealTimeDataQueue which is not getting updated properly. Please find the code below:
let candleCurrentJSONDataWS = null;
var candleRealTimeDataQueue = [];
let tempDateTime = null;
let candleJsonData = {};
useEffect(() => {
getDataFromAPI();
}, []);
...
const getDataFromAPI = async () => {
let apiDataFetch = await fetch('https:/api/endpoint');
let response = await apiDataFetch.json(); // data from api obtained correctly
// total 4 values
for (var i = 0; i < 4; i++) {
tempDateTime = new Date(parseInt(response[i][0]));
candleJsonData['time'] = tempDateTime.toString();
candleJsonData['open'] = parseFloat(response[i][1]);
candleJsonData['high'] = parseFloat(response[i][2]);
candleJsonData['low'] = parseFloat(response[i][3]);
candleJsonData['close'] = parseFloat(response[i][4]);
console.log(candleJsonData); // this correctly prints different
// data for each different i
candleRealTimeDataQueue.push(candleJsonData);
console.log(candleRealTimeDataQueue); // PROBLEM is here: At the end
// candleRealTimeDataQueue array all
// have SAME elements. Its wrong. All
// 4 elements are of i = 3
}
}
Problem is at the end candleRealTimeDataQueue has 4 elements and all the elements are same. This should not happen because I am pushing DIFFERENT candleJsonData elements in the candleRealTimeDataQueue array in the for loop. Please help.
I believe the problem here is that you are reusing the candleJsonData object. When you run candleRealTimeDataQueue.push(candleJsonData), you are pushing the reference to candleJsonData into candleRealTimeDataQueue. So at the end of the loop, you have four references to the same object inside candleRealTimeDataQueue. And since you are modifying the same candleJsonData object over and over again, you'll just see four identical json data inside the queue when you log it and all four elements will be of i = 3.
Instead, you should be creating new candleJsonData objects inside your loop. So something like
for (var i = 0; i < 4; i++) {
tempDateTime = new Date(parseInt(response[i][0]));
let candleJsonData = {}
candleJsonData['time'] = tempDateTime.toString();
candleJsonData['open'] = parseFloat(response[i][1]);
candleJsonData['high'] = parseFloat(response[i][2]);
candleJsonData['low'] = parseFloat(response[i][3]);
candleJsonData['close'] = parseFloat(response[i][4]);
candleRealTimeDataQueue.push(candleJsonData);
}
it is because of the candleJsonData variable which is declared outside, so latest value is overriding previous value. In face there is no need of that variable and it can directly push in the array.
var candleRealTimeDataQueue = [];
React.useEffect(() => {
getDataFromAPI().then((data) => {
for (let i = 0; i < 4; i++) {
candleRealTimeDataQueue.push({
time: new Date(parseInt(data[i][0])).toString(),
open: parseFloat(data[i][1]),
low: parseFloat(data[i][3]),
close: parseFloat(data[i][4]),
});
}
});
return () => {
// do clean up here
};
}, []);
const getDataFromAPI = () => {
return fetch('https:/api/endpoint');
};
I'm trying to retrieve an array of cards for a project. However, in my function, the final contacts array returns an empty array.
I know that, because I have an async call to another funcion inside the forEach loop, the loop doesn't execute as intended. However, I'm very newbie when it comes to deal with this issues, so I want to ask you what's the best approach to deal with this.
This is my code:
export const extractsIDSForUser = async (currentUser: User) : Promise <Object> => {
let contactCards = currentUser.contacts;
const contacts = [];
const usersRef = await firebase.firestore().collection('Users').get();
const usersSnapshot = usersRef.docs.map(doc => doc.data());
contactCards.forEach(async folder => {
const ids = [];
folder.forEach(contact => {
ids.push(contact);
});
for (let i = 0; i < ids.length; i +=1) {
const contact = ids[i];
for (let j = 0; j < usersSnapshot.length; j += 1) {
const userId = usersSnapshot[j].id;
// Async call to function
const cardsFromUser = await extractCardsFromUser(userId);
const arrayCards = Object.values(cardsFromUser);
if (arrayCards.length > 0) {
for (let j = 0; j < arrayCards.length; j += 1) {
const arrayId = arrayCards[j].id;
const sameCardId = arrayId === contact;
if (sameCardId) {
// Where I insert the values into the array
contacts.push(arrayCards[j]);
}
}
}
}
}
});
// this is empty
return contacts;
}
What will be the best approach to deal with this?
I think you have already found a solution, but I had a similar problem and found this article quite helpful.
You could use a traditional for (const contactCard of contactCards) and it will work, but it will be less efficient than using a Promise.all approach.
app.get("/indsalesx/:store/:mm", (req, res) => {
connect();
let ddd = [];
let staffarray = [{}];
let store = req.params.store;
let mm = req.params.mm;
const SP = mongoose.model(`sales${store}`, Sales);
let num = stafflist[store].length - 1;
for (i = 0; i <= num; i++) {
let staffname = stafflist[store][i];
let calc = 0;
SP.find(
{ v_salesperson: stafflist[store][i], v_month: mm },
"v_amount",
(err, doc) => {
let t = doc.length - 1;
doc.map((res) => {
calc = calc + res.v_amount;
});
ddd.name = staffname;
ddd.amount = calc;
staffarray.push(ddd);
}
);
}
console.log(staffarray);
});
The issue I have is: Why is staffarray returning an empty array? staffarray was declared as an empty array of objects, and in a loop function, objects were pushed to to array. But when I console.log(staffarray), it returns the empty array of objects declared initially.
Any help on what to do?
When using find(), you can use 2 approaches.
Pass a callback function
await the function to execute and return the results.
It appears that you used the first approach which means that you are passing a callback into the find() method which handles the result once received.
The console.log() code line will execute before the result will return since it's the next line to execute after the for loop.
So, let's go through what it happening here:
Javascript is executing the find() code line.
That line of code is being placed in the web API which are the pieces of the browser in which concurrency kicks in and makes the call to the server for us.
The console.log() line is being executed with an empty array (since the results haven't been received yet.
After some time, results came back and the callback is being set in the callback queue.
The JS event loop takes the callback from the callback queue and executes it.
This is part of the javascript event loop. you could read more about this here
Mongoose documentation: Model.find()
you can use for of with async/await instead of for
app.get("/indsalesx/:store/:mm", async(req, res) => {
connect();
let ddd = [];
let staffarray = [{}];
let store = req.params.store;
let mm = req.params.mm;
const SP = mongoose.model(`sales${store}`, Sales);
let num = stafflist[store].length - 1;
var list = Array.from(Array(num).keys());
for (let i of list) {
let staffname = stafflist[store][i];
let calc = 0;
let doc = await SP.find(
{ v_salesperson: stafflist[store][i], v_month: mm },
"v_amount"
);
let t = doc.length - 1;
doc.map((res) => {
calc = calc + res.v_amount;
});
ddd.name = staffname;
ddd.amount = calc;
staffarray.push(ddd);
}
console.log(staffarray);
});
I have been able to solve it, all I needed was proper structuring with the async and await statements.
app.get("/indsalesx/:store/:mm", async (req, res) => {
connect();
let ddd = {};
let staffarray = [];
let store = req.params.store;
let mm = req.params.mm;
const SP = mongoose.model(`sales${store}`, Sales);
let num = stafflist[store].length - 1;
for (i = 0; i <= num; i++) {
let staffname = stafflist[store][i];
let calc = 0;
await SP.find(
{ v_salesperson: stafflist[store][i], v_month: mm },
"v_amount",
(err, doc) => {
let t = doc.length - 1;
doc.map((res) => {
calc = calc + res.v_amount;
});
staffarray.push({ name: staffname, amount: calc });
}
);
}
console.log(staffarray);
res.send({ data: staffarray });
});
So guys, I've got scraping function, where I create object of scraped data. Code of scraper is:
const axios = require('axios');
const cheerio = require('cheerio');
const db = require('../config/db.config');
const Article = db.article;
const prices = new Array();
const ids = new Array();
const descs = new Array();
const links = new Array();
for (p = 1; p < 3; p++) {
function again() {
const url = `https://www.olx.ba/pretraga?vrsta=samoprodaja&kategorija=23&sort_order=desc&kanton=9&sacijenom=sacijenom&stranica=${p}`;
axios
.get(url)
.then((response) => {
let $ = cheerio.load(response.data);
$('div[class="naslov"] > a').each((i, el) => {
const id = $(el).attr('href'); // ID, description and link are in the same div class
const desc = id;
const link = id;
descs.push(desc.substring(36)); //Retriving description with substring and push into array
ids.push(id.substring(27, 35)); //Retriving id with substring and push into array
links.push(link); //Retriving link and push into array
for (var i = 0; i < descs.length; i++) {
descs[i] = descs[i].replace('/', '').replace('-', ' ');
}
});
$('div[class="datum"] > span').each((i, el) => {
$('span[class="prekrizenacijena"]').remove();
const price = $(el).text();
prices.push(price); //Retriving price and push into array
});
for (var i = prices.length - 1; i >= 0; i--) {
if (prices[i] === 'PO DOGOVORU') {
prices.splice(i, 1);
}
}
async function asy() {
const sqm = new Array();
for (k = 0; k < links.length; k++) {
const res = await axios
.get(`${links[k]}`)
.then((result) => {
let $ = cheerio.load(result.data);
const pr = $('div[class="df2 "]').first().text();
sqm.push(pr);
for (var i = 0; i < sqm.length; i++) {
sqm[i] = sqm[i].replace('m2', '');
}
})
.catch((err) => {
//handle error
console.log(err);
});
}
const object = ids.map((element, index) => {
const ppm2 =
parseFloat(
prices[index].replace(/\.| ?€$/g, '').replace(',', '.')
) / parseFloat(sqm[index]);
const ppm2final = Math.round(ppm2);
return {
id: element,
price: prices[index],
descr: descs[index],
link: links[index],
sqm: sqm[index],
ppm2: ppm2final + ' KM',
};
});
console.log(object);
console.log(Object.keys(object).length);
/*const ins = await Article.bulkCreate(object)
.then(console.log('Data added to DB'))
.catch((err) => console.log(err));*/
}
asy();
})
.catch((e) => {
console.log(e);
});
}
again();
}
Now when I delete first for lop and function again() and instead of ${p} in url insert eg. 1,2,3 etc. it's working perfect - sqm is fetched for correct link.
Now the problem:
I want to run this url multiple times because ${p} is number of page on that url. Now first problem I got:
sqm isn't correct - sqm data is thrown all over the object and isn't correct for that link.(it's correct when I don't use ${p}
First time i get sqm data(but not correct for that link), when function needs to get ran second time (for second page - to ${p}=2) - sqm isn't fetched at all (it throws NaN).
Also I've got console.log(Object.keys(object).length); where I expect first time to be 30, then after is runned second time to I get 60.(each page contains 30 articles), but I get 60, then again 60.
I've tried with many things: async functions, putting axios to await etc. but nothing really work - sometimes I get only 30 articles, sometimes 60 but with incorrect values.
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 4 years ago.
I've been working on a project and I need some help regarding this:
I have a promise inside a function and I need to return the value of "current" so that I can use it elsewhere.
I'm retrieving data from Firebase, then I shuffle the result and extract only 10 elements from the result.
function retriveData() {
//declaration of variables ...
axios.get("Firebase link")
.then((response) => {
keyArray = Object.keys(response.data);
let k = shuffle(keyArray);
//shuffle is a function to shuffle the key results
for (var i = 0; i < 10; ++i) {
current[i] = response.data[k[i]];
}
});
return current;} //I want this variable to end up with the promise result in current
I know, this is not how promises work but I need a solution to solve this problem.
Thanks!
axios.get is asynchronous, so either you pass a callback to retrieveData, or it needs to return a Promise itself. There's no way around that.
Using a callback (no error handling):
function retriveData(callback) {
axios.get("Firebase link")
.then((response) => {
keyArray = Object.keys(response.data);
let k = shuffle(keyArray);
//shuffle is a function to shuffle the key results
for (var i = 0; i < 10; ++i) {
current[i] = response.data[k[i]];
}
callback(null, current);
});
}
retrieveData((err, result) => console.log(result));
Using a Promise (no error handling):
function retriveData() {
return new Promise((resolve) => {
axios.get("Firebase link")
.then((response) => {
keyArray = Object.keys(response.data);
let k = shuffle(keyArray);
//shuffle is a function to shuffle the key results
for (var i = 0; i < 10; ++i) {
current[i] = response.data[k[i]];
}
resolve(current);
});
}
retrieveData().then((result) => console.log(result));
[EDIT] The above example is mean for illustrative purposes. Since axios.get already returns a Promise, it can be returned back directly from retrieveData.
function retriveData() {
return axios.get("Firebase link")
.then((response) => {
keyArray = Object.keys(response.data);
let k = shuffle(keyArray);
//shuffle is a function to shuffle the key results
for (var i = 0; i < 10; ++i) {
current[i] = response.data[k[i]];
}
return current;
});
}
retrieveData().then((result) => console.log(result));
Try this: I am making your retriveData function as a promise so you can use it anywhere in your program
function retriveData() {
//declaration of variables ...
return new Promise((resolve, reject) => {
axios.get("Firebase link")
.then((response) => {
keyArray = Object.keys(response.data);
let k = shuffle(keyArray);
//shuffle is a function to shuffle the key results
for (var i = 0; i < 10; ++i) {
current[i] = response.data[k[i]];
}
// if everything fine, if you get any conditional error then call reject();
resolve(current);
});
})
}
//call the function like promise
retriveData().then(result => {
//current will come here
console.log('result comes here');
}).catch(error => {
//error comes here (reject error)
console.log('error');
})