I have an empty array of documents.
let arrayOfDocuments = [];
I want to call http requests (using superagent) to download a text file and put its contents into my arrayOfDocuments.
request.get('docs.google.com/document/d/SOME_FILE_NAME').then((res) => {
arrayOfDocuments.push(res.text);
});
That part I get, but here is the tricky part. I want to put this in a for loop and do something after the for loop. So like:
for (let i = 0; i < numOfLinks; i++) {
// send the http requests as above
}
//do stuff here but only after the above for loop is finished.
How do I only do the last line if the loop is finished? The way my program is running right now, the code after the for loop runs before the http requests get a response and finish. I think there is a way to do this using Bluebird Promises, but I'm not sure. Thanks!
You can use the Promise.map method:
Promise.map(arrayOfLinks, function(link) {
return request.get(link);
}).then(function(arrayOfDocuments) {
// ...
});
Use promise.all as shown here http://bluebirdjs.com/docs/api/promise.all.html
In practice it might look something like:
var promises = []
var links = ['a.com/a', 'a.com/b']
for (let i = 0; i < links.length; i++) {
promises.push(request.get(links[i])
}
Promise.all(promises).then(function(allRes) {
//do anything you want with allRes or iterate
for (var promise in promises){
promise.then(function(singleRes){/*do something with each promise after all resolve*/}
}
});
Related
I've spent the last few days trying to tackle this issue and have read all sorts of solutions on StackOverflow and other sites.
I'm building a site that grabs XML data from an outside source and then gets more XML depending on the results to build a network graph. The problem is that I have to essentially wait until this loop of AJAX calls (which may loop into more AJAX calls) is finished before drawing.
I don't know if this just has an especially high cognitive load, but it really has me stumped.
My Code:
function cont(outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
// I want the code to wait until loop is done, and then draw.
draw(nodes, edges);
}
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
var outerNodes = process(result, fromId, term);
cont(outerNodes);
}
});
}
Note: I understand I may be completely misunderstanding JavaScript synchronicity here, and I very likely am. I have used callbacks and promises successfully in the past, I just can't seem to wrap my head around this one.
If I have not been totally clear, please let me know.
I did try implementing a counter of sorts that is incremented in the process() function, like so:
if (processCount < 15) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
} else {
draw(nodes, edges);
}
However, this ended up with several draw() calls which made my performance abysmal.
There are nice new well-supported APIs and language constructs we can use. The Fetch API, await, and for...of loops.
The Fetch API uses Promises. Promises can be awaited. The for...of loop is aware of await and won't continue the loop until the await has passed.
// Loop through, one-at-a-time
for (const node of outerNodes) {
// Make the HTTP request
const res = await fetch(someUrl);
// Do something with the response here...
}
Don't forget a try/catch (which also works with await), and check res.ok.
Brad's answer changes the code to by synchronious and to me that defeats the purpose. If you are constantly waiting on all request to be finished then it could take a while, while normal browsers can handle multiple requests.
The problem you have in your original questions is with scope. Since each call to cont(outerNodes) will trigger it's own scope, it has no idea what are calls are doing. So basically if you call cont(outerNodes) twice, each call will handle it's own list of outernodes and then call draw.
The solution is to share information between the different scopes, which can be done with one, but preferably two global variables: 1 to track active processes and 1 to track errors.
var inProcess = 0;
var nrErrors = 0;
function cont(outerNodes) {
//make sure you have outerNodes before you call outerNodes.length
if (outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
inProcess++; //add one more in process
getXML(node["label"], node["id"]);
}
}
//only trigger when nothing is in proces.
if (inProcess==0) {
// I want the code to wait until loop is done, and then draw.
draw(nodes, edges);
}
}
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
var outerNodes = process(result, fromId, term);
inProcess--; //one is done
cont(outerNodes);
},
error: function() {
inProcess--; //one is done
nrErrors++; //one more error
cont(null); //run without new outerNodes, to trigger a possible draw
}
});
}
Please note that I track nrErrors but dont do anything with it. You could use it to stop further processing all together or warn the user that the draw is not complete.
[important] Keep in mind that this works in javascript because at best it mimics multithreading. That means the the call to inProcess--; and then right after cont(outerNodes); is always execute directly after eachother.
If you would port this to a true multithreading environment, it could very well be that another scope/version of cont(null); would cut in between the two lines and there would still be multiple draws.
The best way to solve this question should be using either promise or callback.
If you really want to avoid promise or callback(Although i don't know why...)
You can try with a counter.
let processCount = 0;
// Increasing the processCount in getXML callback method
function getXML(term, fromId) {
var url = someURL;
$.ajax({
url: url,
dataType: "xml",
success: function(result) {
processCount++;
var outerNodes = process(result, fromId, term);
cont(outerNodes);
}
});
}
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
while (processCount < outerNodes.length) {
// do nothing, just wait'
}
draw(nodes, edges);
If after testing it many times, you know that it will never take more than say 5 seconds... you can use a setTimeout.
function cont(outerNodes) {
for (var i = 0; i < outerNodes.length; i++) {
var node = outerNodes.pop();
getXML(node["label"], node["id"]);
}
// Display a 5 second progress bar here
setTimeout(function(){ draw(nodes, edges); },5000);
}
So, without boring anyone with the backstory, I need to access data from a number of APIs in order to run my script. The data needs to all be loaded before I execute the script, which I'm normally comfortable doing: I just declare some fetch requests, write a Promise.all, then continue on with the function.
HOWEVER, I've hit something of a snafu with a certain API that limits the number of results I can pull from one request to 100 and I need to query all of the results. I didn't think this was a huge deal since I figured I can just make a couple extra requests by affixing "&page=X" to the end of the request.
The plan, then, is to request the total number of pages from the API and then feed that into a for loop to push a number of fetch requests into an array of promises (i.e., link://to/api/data&page=1, link://to/api/data&page=2, etc). When I actually attempt to create this array with a for loop, though, the array returns empty. Here's my work:
const dataUrlNoPage = 'link/to/api/data&page=';
const totalPages = 3; //usually pulled via a function, but just using a static # for now
let apiRequestLoop = function(inp) {
return new Promise(function(resolve){
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
fetch(dataUrlLoop).then(function(response){
promiseArray.push(response.json());
})
}
resolve(promiseArray);
})
}
let finalPromiseArray = apiRequestLoop(totalPages).then(result => {
let requestArray = [apiRequest1,apiRequest2];
//requestArray contains earlier fetch promises
result.forEach(foo => {
requestArray.push(foo);
}
);
return requestArray;
});
So, what's tripping me up is really the loop, and how it's not returning an array of promises. When I look at it in the console, it shows up as a blank array, but I can expand it and see the promises I was expecting. I see the "Value below was evaluated just now" response. No matter how many promises or .thens, I write, however, the array is never actually populated at run time.
What's going on? Can I not generate fetch promises via a for loop?
(Also, just to cut this line of questioning off a bit, yes, the API I'm trying to access is Wordpress. Looking around, most people suggest creating a custom endpoint, but let's assume for the purpose of this project I am forbidden from doing that.)
You have several problems here.
The first is that you have the function provided to new Promise itself containing promise creations. Don't do this! It's a definite anti-pattern and doesn't keep your code clean.
The second is this basic bit of code:
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
fetch(dataUrlLoop).then(function(response){
promiseArray.push(response.json());
})
}
resolve(promiseArray);
This says:
create an empty array
loop through another array, doing HTTP requests
resolve your promise with the empty array
when the HTTP requests are completed, add them to the array
Step four will always come after step three.
So, you need to add the promises to your array as you go along, and have the overall promise resolve when they are all complete.
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop).then(function(response) {
return response.json();
}));
}
return Promise.all(promiseArray);
}
or, with an arrow function to clean things up:
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop).then(response => response.json()));
}
return Promise.all(promiseArray);
}
A few points:
you want to actually put the promises themselves into the array, not push to the array in a .then() chained to the promise.
you probably want to skip creating a new Promise in your function. Just get an array of all the promises from your loop, then return a Promise.all on the array.
Like this:
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop))
}
return Promise.all(promiseArray);
}
In your final .then statement, in finalPromiseArray, your result will be an array of the results from all the promises. like this [response1, response2, response3, ...]
See the Promise.all documentation for more details.
How to stop nodeJS to execute statements written outside for loop till for loop is completed?
for(i=0;i<=countFromRequest;i++)
{
REQUEST TO MODEL => then getting result here (its an object)
licensesArray.push(obj.key);
}
res.status(200).send({info:"Done Releasing Bulk Licenses!!!",licensesArray:licensesArray})
The problem is that the statement after For Loop is being executed before For loop, so the licensesArray is empty when i receive the API Data.
Any clue how to do that?
Will be thankful to you.
Using async/await:
const licensesArray = [];
for(let i = 0; i <= countFromRequest; i++) {
const obj = await requestModel(); // wait for model and get resolved value
licensesArray.push(obj.key);
}
res.status(200).send({licensesArray});
Using Promise.all:
const pArr = [];
const licensesArray = [];
for(let i = 0; i <= countFromRequest; i++) {
pArr.push(requestModel().then(obj => {
licensesArray.push(obj.key);
}));
}
Promise.all(pArr).then(() => { // wait for all promises to resolve
res.status(200).send({licensesArray});
});
I would go with async/await if your environment supports it, as it makes things easier to read and lets you program with a synchronous mindset (under the hood it's still asynchronous). If your environment does not support it, you can go with the Promise.all approach.
Further reading:
async function
Promise.all
I'm learning Node.js and am working through the Learnyounode course from Node school which is amazing although it is quite a challenge and I'm and a bit confused...
The question I'm stuck on is Excercise 9- Juggling Async...
I'm trying to solve the problem:
"You must collect the complete content provided to you by each of the URLs and print it to the console (stdout). You don't need to print out the length, just the data as a String; one line per URL. The catch is that you must print them out in the same order as the URLs are provided to you as command-line arguments."
I've tried counting the callbacks in a variable called 'waiting' and then calling a complete() function when it has counted 3 callbacks, but the data still always comes back in the wrong order and I don't know why....
var http = require('http');
var bl = require('bl');
var url = []
url[0] = process.argv[2];
url[1] = process.argv[3];
url[2] = process.argv[4];
var waiting = 0;
for(i=0; i < url.length; i++) {
var output = [];
http.get( url[i], function(res) {
res.setEncoding('utf8');
res.pipe(bl(function(err,data) {
output.push(data.toString());
waiting++;
if(waiting == 3) {
complete(output);
}
}));
});
}
var complete = function(output) {
for(i=0; i < output.length; i++) {
console.log(output[i]);
}
}
The async callbacks are not guaranteed to occur in the order that the HTTP requests are made.
You could maintain the order by changing:
output.push(data.toString());
to:
output[i] = data.toString();
it's because your http get requests are asynchronous.
instead of pushing to output array do this
output[i]=data.toString();
Save the value of the 'i' index for each of you requests, either by closure or by adding it as a property to the request. You can then use that index to order your responses.
I have a page than can make a different number of $http requests depending on the length of a variables, and then I want to send the data to the scope only when all the requests are finished. For this project I do not want to use jQuery, so please do not include jQuery in your answer. At the moment, the data is sent to the scope as each of the requests finish, which isn't what I want to happen.
Here is part of the code I have so far.
for (var a = 0; a < subs.length; a++) {
$http.get(url).success(function (data) {
for (var i = 0; i < data.children.length; i++) {
rData[data.children.name] = data.children.age;
}
});
}
Here is the part that I am sceptical about, because something needs to be an argument for $q.all(), but it is not mentioned on the docs for Angular and I am unsure what it is meant to be.
$q.all().then(function () {
$scope.rData = rData;
});
Thanks for any help.
$http call always returns a promise which can be used with $q.all function.
var one = $http.get(...);
var two = $http.get(...);
$q.all([one, two]).then(...);
You can find more details about this behaviour in the documentation:
all(promises)
promises - An array or hash of promises.
In your case you need to create an array and push all the calls into it in the loop. This way, you can use $q.all(…) on your array the same way as in the example above:
var arr = [];
for (var a = 0; a < subs.length; ++a) {
arr.push($http.get(url));
}
$q.all(arr).then(function (ret) {
// ret[0] contains the response of the first call
// ret[1] contains the second response
// etc.
});