I'm querying a REST api which includes pagination. After an initial query which gets total count and max per page, I'm trying to loop through total pages to get all of the items.
const instance = axios.create({
baseURL: 'https://api.com',
headers: {'X-Application-Key': 'APPLICATION_KEY'}
});
instance.get('/trips')
.then((response) => {
const totalTrips = response.data.count;
const totalPages = Math.ceil(totalTrips / response.data.max_per_page);
let i;
for (i = 0; i < (totalPages + 1); i++) {
instance.get(`/trips?page=${i}`);
console.log(response.data.results)
};
})
.catch(function (error) {
console.log(error);
})
This isn't working since the 'response' argument is still referencing the initial query, but it's my understanding that all .then blocks should be on the same level and not actually nested, so I'm not sure how to actually loop through the pages. What is the proper way to accomplish this?
Here is a slightly opinionated response, using async await to handle the multiple asynchronous requests. And I think you'll want to use a function like promise.all, which allows you to make all the requests at the same time, and handle the response when all the http requests have completed.
const instance = axios.create({
baseURL: 'https://api.com',
headers: {
'X-Application-Key': 'APPLICATION_KEY'
}
});
getAlltrips();
async function getAlltrips() {
try {
const response = await instance.get('/trips');
const totalTrips = response.data.count;
const totalPages = Math.ceil(totalTrips / response.data.max_per_page);
const promiseArray = [];
for (let i = 0; i < (totalPages + 1); i++) {
promiseArray.push(instance.get(`/trips?page=${i}`));
};
// promise.all allows you to make multiple axios requests at the same time.
// It returns an array of the results of all your axios requests
let resolvedPromises = await Promise.all(promiseArray)
for (let i = 0; i < resolvedPromises.length; i++) {
// This will give you access to the output of each API call
console.log(resolvedPromises[i])
}
} catch (err) {
console.log('Something went wrong.');
}
}
Related
Trying to get a better grasp on Promises with async/await syntax. .then().catch() feels more straightforward with the added visual cues and structure.
Do these two async/await functions make any logical sense for making multiple API calls and acting on the data once they're resolved?
The goal is to have a function that takes the url, number of additional pages and returns data from all calls in one output. It works but is slow enough to lead me to believe its pretty inefficient. (I'm sure the nested loops dont help)
async function getData(url, numOfPages) {
try {
const response = await axios.get(url)
const { results, next: nextUrl } = response.data
const nextPages = await getNextPages(nextUrl, numOfPages)
const finalData = results.concat(nextPages)
finalData.forEach(data => console.log(data.name))
}
catch (err) {
console.log(err)
}
}
getData('https://swapi.dev/api/planets/', 3)
async function getNextPages(nextUrl, numOfPages) {
try {
const response = await axios.get(nextUrl)
const data = response.data.results.map(res => res)
let nextPage = response.data.next
for (let i = 1; i < numOfPages; i++) {
let response = await axios.get(nextUrl)
let results = response.data.results
nextPage = response.data.next
for (let i = 0; i < results.length; i++) {
data.push(results[i])
}
}
return data
}
catch (err) {
console.log(err)
}
}
I am trying to work with an api where I have to send a request for each item in a list.
However, I see that the loop doesn't seem to wait for every request, i.e, the loop doesn't work as expected. Here's the code below
getInfo = async () => {
const mylist = ["item1","item2","item3","item4","item5","item6","item7"]
const responses = []
const len = mylist.length
for (let i = 0; i < len; i++) {
//console.log("inside loop")
await axios.get("some_url/"+mylist[i])
.then(res => {
responses.push(res.data)
})
}
When I run the program, all the console.log("inside loop") executes immediately without waiting for the request to be complete.
How can I modify the code so as to wait for each response to be completed before updating the for loop counter variable?
You could try re-arranging the code to something like this. But using a Promise.all with Array.prototype.map would be more idiomatic solution for the problem.
await the async call (remove unnecessary .then call) and then console.log
getInfo = async () => {
const mylist = ["item1","item2","item3","item4","item5","item6","item7"]
const responses = []
const len = mylist.length
for (let i = 0; i < len; i++) {
responses.push((await axios.get("some_url/"+mylist[i])).data)
console.log("inside loop")
}
}
Internally, await is translated into a Promise chain. Since the for loop can't be transformed into a Promise-continuation, you'll need to convert it to a Promise-based construct.
Depending on what you want to achieve there are multiple ways to go about it.
Constructing the responses array could be done with a map statement.
const promises = mylist.map(item => {
return axios.get("some_url/"+item).then(res => { return res.data; })
});
const data = await Promise.all(promises);
No manual pushing items around or fiddling with the array length.
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 2 years ago.
I am trying to return a few lists with the axios api call i make from the javascript, however, i do not understand how to console.log() the result and pass it for my future use (trying to use it for a data visualization)
heres what i have
const axios = require('axios');
var years = []
var totals = []
var rail = []
var bus = []
var para = []
const getTotal = async () => {
const url="https://data.cityofchicago.org/resource/w8km-9pzd.json";
var totals = []
try {
let res = await axios.get(url);
for (i = 0; i < res.data.length; i++) {
totals.push(res.data[i].total);
years.push(res.data[i].year);
rail.push(res.data[i].rail);
bus.push(res.data[i].bus);
para.push(res.data[i].para);
}
}catch(error) {
console.log(error);
}
return(totals,years,rail,bus,para)
}
//data = axiosDataFetch().bus;
console.log(getTotal())
^^^ how do i print totals here instead of having it show as undefined?
i edited the code based on some answers i received, essentially i want to be able to call and use the 5 lists i got from the API into a data visualization.
Add an await keyword before the axios call, then instead of using the .then callback, do stuff with the res object.
const axios = require('axios');
async function axiosDataFetch() {
const url="https://data.cityofchicago.org/resource/w8km-9pzd.json";
var totals = []
let res = await axios.get(url);
for (i = 0; i < res.data.length; i++) {
totals += res.data[i].total;
}
console.log(res.result)
}
axiosDataFetch();
Javascript is designed with asynchronous behavior at its core (eg. it's unknown when the http call made by axios will return, so it's not guaranteed to be in sync with your program).
You can read more about the many, many ways to handle asynchronous behavior here. Long article, but will save you countless hours in the future if you can understand its content.
Try logging from inside your then. Your other log is being called before axios returns back a result.
const axios = require('axios');
const url="https://data.cityofchicago.org/resource/w8km-9pzd.json";
var totals = []
let res = axios.get(url)
.then(result=>function(response){
for (i = 0; i < response.data.length; i++) {
totals += response.data[i].total;
}
console.log(totals); // add this
}
)
.catch(function (error) {
console.log(error)
});
const axios = require('axios');
const url="https://data.cityofchicago.org/resource/w8km-9pzd.json";
const getTotal = async () => {
var totals = []
try {
let response = await axios.get(url);
for (i = 0; i < response.data.length; i++) {
totals.push(response.data[i].total);
}
console.log(totals);
} catch(error) {
console.log(error);
}
return totals;
}
sorry my fault, I didn't know that you wanted to execute the query at the top level, you have the pending response because the function getTotals is returning a function that is a promise. You can solve it doing the following
getTotal().then(result => {
console.log(result);
});
Usually we don't have this behaviour because we have our code modularised and when we import our file, our code is internally inside a IIFE.
This question already has answers here:
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
(7 answers)
Closed 2 years ago.
I know Js is async but how to prevent my callback from returning an empty array ?
Here is my code :
function getPicturesArray(nbr,callback){
var avatarArr = [];
for(var i =0; i<nbr ; ++i){
request("https://randomuser.me/api/",{json: true},(err,response,body)=> {
avatarArr.push(body.results[0].picture.large);
});
}
callback(avatarArr);
}
getPicturesArray(12, (e)=>{
console.log(e);
});
console.log('here');
And the sout is:
[]
here
I'm mocking the calls with a public service for demonstration purposes, but the key point here is the asynchronous nature of HTTP requests.
When you invoke callback(avatarArr);, your for loop IS done but the (in this case 12) requests you sent are not, so avatarArr does not have any items yet.
Notice async before function getPicturesArray, and await before fetch
await tells the execution to wait for the fetch call to finish
You MUST check out async/await.
async function getPicturesArray(nbr,callback){
var avatarArr = [];
console.log('loading...');
for(var i =0; i<nbr ; ++i){
const response = await fetch(`https://jsonplaceholder.typicode.com/todos/${i+1}`);
const json = await response.json();
avatarArr.push(json.title);
}
console.log('done loading...');
callback(avatarArr);
}
getPicturesArray(12, (e)=>{
console.log(e);
});
console.log('see how this is executed before done loading...');
Don`t use callbacks. Use Promises or async/await.
getUserPic = async () => {
const url = 'https://randomuser.me/api/'
const response = await fetch(url)
const users = await response.json()
return users.results[0].picture.large
}
;(async () => {
const userAvatars = []
for (let i = 0; i < 12; i++) {
userAvatars.push(await getUserPic())
}
console.log(userAvatars)
})()
Also try to use this.
In this case I advice you to use Set instead of Array to exclude duplicate items.
getUserPic = async () => {
const url = 'https://randomuser.me/api/'
const response = await fetch(url)
const users = await response.json()
return users.results[0].picture.large
}
getMultipleUserPics = async limit => {
const userAvatars = new Set()
while (userAvatars.size != limit) {
userAvatars.add(await getUserPic())
}
return userAvatars
}
;(async () => {
const userPics = await getMultipleUserPics(5)
console.log(userPics)
})()
I'm trying to write the following code and make it work synchronously, but the only problem that it works correctly with console.log, which prints me every item in array with delay in 1 second, but don't work with the following structure:
for (let i = 0; i < array.length; i++) {
setTimeout(function () {
1.http request via rp or request.get (I receive a huge data array)
2. .map results
3.insert to Mongo via mongoose
}
}
as for now I have the following code inside:
request.get({url: array[i].url}), function (error, body) {
body.map(element => {
//do stuff, it works fine
});
collection.insertMany(body, function (err, docs) {
//#justloggerthings
}
Or I have almost the same version with rp instead of request.get
By default I have mongoose.Promise = global.Promise;
Why this cause a problem? Because body.length is very huge dataset which eat a lot of RAM. (Now imagine 20+ arrays with insertMany)
So Mongo trying to insertMany all responses from request at once (when they ready, w/o 1000s delay). Actually that's why I choose request instead of rp (request-promise) but it seems look async too. So should I choose another http get module from npm and switch to it. And not to worry about it?
Or should I wrap this operations to promise || made an async function and recall it inside loop every time (1000s for example) when I it's correctly finished. In this case, the only thing which I found actual on StackOverflow is:
How to insert data to mongo synchronously (Nodejs, Express)
Bit it's a bit outdated. So any ideas?
Well, i dont have your actual code so i will write in pseudo code what you can do.
const chunkArray = (array, chunkSize) => {
let i,j,chunks = [];
for (i = 0, j = array.length; i < j; i += chunkSize) {
chunks.push(array.slice(i, i + chunkSize));
}
return chunks;
}
for (let i = 0; i < array.length; i++) {
let bigArray = await request('your data url');
// do some mapping or whatever
// then break your large array into smaller chunks
let chunks = chunkArray(bigArray, 10);
let j;
for (j = 0; j < chunks.length; j++) {
await collection.insertMany(chunks[j]);
}
}
Actual code that solve my problem is:
async function test (name, url, lastModified) {
try {
const response = await rp({uri: url, json: true});
response.map(async (element) => {
if (element.buyout > 0) {
element.price = (element.buyout / element.quantity);
}
element.lastModified = lastModified
});
return collection.insertMany(response);
} catch (err) {
console.error(err)
}
}
async function addAsync() {
const z = await test();
console.log(z);
}
addAsync();