I'm writing an app that needs to query an RestAPI with a set of input. However, that API is very bad at scaling and it will return code 429 if it receives too many request. So I was trying to add a sleep function between each axios call but it seems not working. Wondering if I have this sleep function set correctly.
const sleep = (milliseconds) => {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
useEffect(() => {
let results = [];
async function fetchData(id) {
let request = someAPI + id + query;
axios.get(request).then((result) => {
results.push(result);
}).catch((error) => {console.log(error);});
await sleep(2000);
}
for (let i = 1; i <= 20; i++) {
fetchData(i);
}
},[]);
You didn't await the fetchData function, you can try this.
useEffect(() => {
let results = [];
async function fetchData(id) {
try {
const result = await axios.get('')
results.push(result);
await sleep(2000)
}
catch(error) {
console.log(error);
}
}
(async function() {
for (let i = 1; i <= 20; i++) {
await fetchData(i);
}
})()
},[]);
In your for loop, you have to await the fetchData function. This way, for each iteration, the process will wait for fetchData (and therefore the nested call to the sleep function) to complete.
Currently, you're just firing off all 20 calls to fetchData at once, without actually waiting for your sleep function.
async function fetchAll() {
for (let i = 0; i < 20; i++) {
await fetchData()
}
}
fetchAll()
Note that you will have to wrap your for-loop in an async function, in order to be able to use await within the loop.
Related
I have the following structure of code:
var cdict = new Map();
fetch("randomurl")
.then(res => res.json())
.then(data => {
for (const obj of data.result) {
// insert stuff into Map cdict
}
counter(0).then(res => {
console.log(cdict);
})
// ^ here after calling of counter i need to do stuff
});
const cbegin = 0;
const ccount = 10;
function counter(cnt) {
if (cnt < ccount) {
setTimeout( () => {
cnt++;
fetch(someurl)
.then(res => res.json())
.then(data => {
for (const obj of data.result) {
// do stuff
}
})
.catch(err => {
console.log(err);
});
counter(cnt);
}, 1000);
}
}
Here after execution of counter(0) call and all its fetch requests, I wish to execute a line console.log(cdict);
How can this be achieved? And is this proper way to call fetch requests with delay of 1 second?
Don't mix setTimeout event queue callbacks with promise-based code -
const sleep = time =>
new Promise(resolve => setTimeout(resolve, time))
async function main()
{ console.log("please wait 5 seconds...")
await sleep(5000)
console.log("thanks for waiting")
return "done"
}
main().then(console.log, console.error)
Don't write .then(res => res.json()) every time you need some JSON. Write it once and reuse it -
const fetchJSON(url, options = {}) =>
fetch(url, options).then(res => res.json())
async function main()
{ const data = await fetchJSON("path/to/data.json")
console.log("data received", data)
return ...
}
main().then(console.log, console.error)
Don't attempt to declare variables outside of Promises and modify them later. You cannot return the result of an asynchronous call. Asynchronous data needs to stay contained within the promise or you will be chasing hard-to-find bugs in your code -
async function main(urls)
{ const result = []
for (const u of urls) // for each url,
{ result.push(await getJSON(u)) // await fetch and append to result
sleep(1000) // wait 1 second
}
return result
}
const myUrls =
[ "foo/path/data.json"
, "another/something.json"
, "and/more/here.json"
]
main(urls)
.then(result => /* counter logic */)
.then(console.log, console.error)
Continue abstracting as you see fit -
// give reusable functions a name; use parameters for configurable behaviour
async function fetchAll(urls, delay = 100)
{ const result = []
for (const u of urls)
{ result.push(await getJSON(u))
sleep(delay)
}
return result
}
async function counter(urls)
{ const results = await fetchAll(urls) // use our generic fetchAll
const cdict = /* counter logic... */
return cdict
}
const myUrls =
[ "foo/path/data.json"
, "another/something.json"
, "and/more/here.json"
]
counter(urls).then(console.log, console.error)
As you can see async and await prevent nesting that occurs with the use of setTimeout or .then callbacks. If you use them correctly, your code remains flat and you can think about your code in a synchronous way.
The Problem is with the uplines.push.
I always get an empty uplines array so the last part of the code doesn't run. The promises resolve later and I get the correct data. May I know how to go about doing it the correct way?
const getAllUplines = async () => {
uplines = [];
const findUser = async (userFid) => {
const userDoc = await firestore.collection("users").doc(userFid).get();
if (userDoc.exists) {
const user = { ...userDoc.data(), id: userDoc.id };
console.log(user);
uplines.push(user);
if (user.immediateUplineFid) {
findUser(user.immediateUplineFid); //self looping
}
} else {
console.log("No User Found");
return null;
}
};
sale.rens.forEach(async (ren) => {
findUser(ren.userFid);
});
console.log(uplines);
return uplines;
};
let uplines = await getAllUplines();
console.log(uplines);
uplines = uplines.filter(
(v, i) => uplines.findIndex((index) => index === v) === i
); //remove duplicates
uplines.forEach((user) => {
if (user.chatId) {
sendTelegramMessage(user.chatId, saleToDisplay, currentUser.displayName);
console.log("Telegram Message Sent to " + user.displayName);
} else {
console.log(user.displayName + " has no chatId");
}
});
There are a few things that you have missed out while implementing the async call, which are explained in the inline comments in the code snippet.
A short explanation for what happened in your code is that in the line sale.rens.forEach you are passing an async function in the argument, which does not make any difference to the function forEach, it will execute it without waiting for it to complete.
Therefore in my answer I am using Promise.all to wait for all the async function calls to complete before returning the result.
// This is wrapped in an immediately executed async function because await in root is not supported here
(async () => {
const mockGetData = () => new Promise(resolve => setTimeout(resolve, 1000));
const sale = {
rens: [
{ userFid: 1 },
{ userFid: 2 },
{ userFid: 3 }
]
};
const getAllUplines = async () => {
const uplines = [];
const findUser = async (userFid) => {
// Simulating an async function call
const userDoc = await mockGetData();
console.log("User data received");
uplines.push(`User ${userFid}`);
};
const promises = [];
sale.rens.forEach(ren => { // This function in foreach does not have to be declared as async
// The function findUser is an async function, which returns a promise, so we have to keep track of all the promises returned to be used later
promises.push(findUser(ren.userFid));
});
await Promise.all(promises);
return uplines;
};
let uplines = await getAllUplines();
console.log(uplines);
})();
In order to get the results of getAllUplines() properly, you need to add await to all async functions called in getAllUplines().
const getAllUplines = async () => {
uplines = [];
const findUser = async (userFid) => {
const userDoc = await firestore.collection("users").doc(userFid).get();
if (userDoc.exists) {
const user = { ...userDoc.data(), id: userDoc.id };
console.log(user);
uplines.push(user);
if (user.immediateUplineFid) {
await findUser(user.immediateUplineFid); //self looping
}
} else {
console.log("No User Found");
return null;
}
};
sale.rens.forEach(async (ren) => {
await findUser(ren.userFid);
});
console.log(uplines);
return uplines;
};
I have an API to call every 2.5 seconds. Initially, the data inside the response object is null as the database is still updating it through a transaction. But on the subsequent 3rd or 4th try, I get the data. I am writing a reusable function for the same, however I get undefined. My goal is to keep calling the API until I get the value in my path and close the connection. Please advice.
P.S: The below API URL doesnt have any delay, but my private API has.
const getData = (url, path) => {
const interval = setInterval(async () => {
const result = await axios.get(url);
if (_.has(result.data, path) && result.data[path]) {
return result[path]
}
}, 2500)
return clearInterval(interval)
}
getData('https://api.oceandrivers.com/static/resources.json', 'swaggerVersion')
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.15/lodash.js"></script>
JS Fiddle URL
Please advice.
You
return clearInterval(interval); // undefined
If you want to return a Promise which will resolve when the data is available, you could do something like this:
const getData = (url, path) => {
return new Promise((resolve, reject) => {
const interval = setInterval(async() => {
const result = await axios.get(url);
if (_.has(result.data, path) && result.data[path]) {
clearInterval(interval); // Clear the interval
resolve(result.data[path]); // Resolve with the data
}
}, 2500);
});
}
getData('https://api.oceandrivers.com/static/resources.json', 'swaggerVersion')
.then(data => {
console.log(data); // Your data is available here
});
// OR
(async () => {
const version = await getData('https://api.oceandrivers.com/static/resources.json', 'swaggerVersion');
console.log(version);
})();
Its because javascript is asynchronous as above comment is already mentioned. You can use either callbacks or promise in javascript. Here is the code:
const getData = (url, path, cb) => {
const interval = setInterval(async () => {
const result = await axios.get(url);
if (_.has(result.data, path) && result.data[path]) {
clearInterval(interval); //We found it remove the interval
cb(result.data[path]);
}
}, 2500);
};
getData(
"https://api.oceandrivers.com/static/resources.json",
"swaggerVersion",
data => {
console.log("test",data);
}
);
Here is the fiddle: https://jsfiddle.net/7pc4hq6t/3/
You could create an asynchronous delay:
const delay = milliseconds => new Promise(resolve, setTimeout(resolve, milliseconds));
Then use like this:
const getDataAsync = async (url, path) => {
while (true) {
const result = await axios.get(url);
if (_.has(result.data, path) && result.data[path]) {
return result.data[path];
}
await delay(2500);
}
}
const data = await getDataAsync('https://api.oceandrivers.com/static/resources.json', 'swaggerVersion');
This avoids the multiple layers of nested callbacks, and produces much more readable code.
shortly, I was trying to simulate async / await behavior in JavaScript but getting not expected
const urls = ['api1', 'api2', 'api3']
async function start() {
for (i = 0; i < urls.length; i++) {
result = await getFromApi(urls[i])
console.log(result)
}
}
async function getFromApi(apiUrl) {
return await new Promise((resolve, reject) => {
resolve(apiUrl)
}).then(apiUrl => apiUrl)
}
console.log('start ....')
start()
console.log('done ... ')
so the expected result should be
start ....
api1
api2
api3
done ...
but I am getting
start ....
done ...
api1
api2
api3
The function called start() needs be used with await. Also in the same time your code needs to be wrapped with async function.
Try as the following:
(async () => {
const urls = ['api1', 'api2', 'api3']
async function start() {
for (i = 0; i < urls.length; i++) {
result = await getFromApi(urls[i])
console.log(result)
}
}
async function getFromApi(apiUrl) {
return await new Promise((resolve, reject) => {
resolve(apiUrl)
}).then(apiUrl => apiUrl)
}
console.log('start ....')
await start()
console.log('done ... ')
})();
I hope this helps!
start() isn't being awaited. If this is at the top-level scope then you would probably use .then() on the returned Promise object. For example:
console.log('start ....');
start().then(() => {
console.log('done ... ');
});
How to insert data into db with index as the title? I think this is typical async question but I can't solve it. The order of i been inserted not in sequence.
const { times } = require('lodash')
module.exports = (async () => {
try {
const createJob = async (i) => {
console.log(i) //executed first 1 - 50 first
const { data } = await axios
.post('http://localhost:3000/job/create', {
"title": i,
"created_at": Date.now()
})
if(data) {
console.log('job created. ', data)
}
}
times(50, (i) => {
createJob(++i)
});
} catch(e) {
console.log('Error creating ad. ', e)
}
})()
You can chain your promises so each createJob is called after the previous one is finished. You can create an array of indexes and use array.reduce to do the chaining. In the code below, I replace your axios call with a new Promise(...), just for simulating:
var createJob = async (i) => {
console.log(i) //executed first 1 - 50 first
const { data } = await new Promise((resolve, reject) => setTimeout(() => resolve({ data: i}), 1000));
if (data) {
console.log('job created. ', data);
}
return data;
}
var arr = Array.from(new Array(10), (e, i) => i);
arr.reduce((m, o) => m.then(() => createJob(o)), Promise.resolve());
You can also use Promise.all to solve this. Essentially you save all the promises of the request in an array. Once, all the requests are complete, you can iterate over them.
const axios = require('axios');
module.exports = (async () => {
try {
const createJob = async (i, url) => {
console.log(i, url) //executed first 1 - 50 first
return axios.get(url);
}
const a = ['http://httpbin.org/anything/123', 'http://httpbin.org/anything/456']
const promiseArray = [];
for (let j = 0; j < 4; j++) {
promiseArray.push(createJob(j, a[j % 2]));
}
Promise.all(promiseArray).then((result) => {
console.log('result', typeof (result));
for (let i = 0; i < result.length; i++) {
console.log(result[i].data.url);
}
});
} catch (e) {
console.log('Error creating ad. ', e)
}
})()
I have used httpbin for making actual calls and random payload to make sure the execution order is always the same.