I've come to the realization that since promises in ECMAScript 6 allow for synchronous coding of asynchronous functions, for every promise-laden piece of code there's a synchronous corollary. For instance:
var data = processData(JSON.parse(readFile(getFileName())));
Is the same as:
var data = getFileName()
.then(readFile)
.then(JSON.parse)
.then(processData);
Now for my current use-case I want to write code to pull data from a massive public API. The API is paginated, so in a purely synchronous world I would write something like the following:
var data = [];
var offset = 0;
var total = 10000; // For example - not actually how this would work
while( offset < total ) {
data.concat(getDataFromAPI(offset));
offset = data.length;
}
Now my question is, how would I do this with promises? I could write something like:
var data = [];
var offset = 0;
var total = 10000;
getDataFromAPI(offset)
.then(function(newData){
data.concat(newData);
return getDataFromAPI(data.length);
});
But at this point I'm forced to just chain infinite .thens -- there's no looping logic. I feel like something should be possible using recursion, but I have no idea how to do it.
I'm using BluebirdJS as my promise library, so I have access to all of their helper methods.
I feel like something should be possible using recursion
Exactly. You can name the callback so you can reference it again. As long as the condition isn't met, return a promise from the callback. Otherwise return the final result:
getDataFromAPI(offset)
.then(function next(newData){
data.concat(newData);
var newOffset = data.length;
return newOffset < total ? getDataFromAPI(newOffset).then(next) : data;
})
.then(function(data) {
console.log(data); // final result
});
Related
I know this code is probably awful, but I'm trying to get data from my Firebase database and everything works but the function within the for loop always returns Promise {pending}, and I'm not sure what to do to fix that.
for(var i = 0; i < querySnapshot.size; i++){
var blah = firestore.collection("testList").doc(i.toString());
lines[i] = blah.get().then(function(doc){
return doc.data().text;
});
console.log(lines[i]);
}
var finalText = lines.join("\n");
If you want to wait for all documents to be loaded, you're looking for Promise.all():
const promises = [];
for (var i = 0; i < querySnapshot.size; i++){
var blah = firestore.collection("testList").doc(i.toString());
promises.push(blah.get().then();
}
Promise.all(promises).then(function(docs){
return docs.map(function(doc)
return doc.data().text;
});
}).then(function(lines) {
var finalText = lines.join("\n");
console.log(finalText);
});
You might want to read up on asynchronous code and promises, as this is quite common and you'll keep bumping into learning problems until you grok these concepts.
Some good starting points:
How do I return the response from an asynchronous call?
Learn JavaScript Promises (Pt.1) with HTTP Triggers in Cloud Functions
Graceful asynchronous programming with Promises
The Function Promise.all - does have the job, to wait of asynchronious Functions and do certain logic after those asynchronious functions are done. I couldn't find how exactly does this function work i.e. if you can do certain things.
Example:
In this Code the Upperhalf till Promise.all, does fill the datelist with data. The Asynchronious function attachRequestCompleted - must first be done, so that datelist will be filled with data.
Within Promise.all i want to iterate through the datelist which was filled with data in attachRequestCompleted, so that i can later add them as Special Dates in the Calendar
var datelist = [];
var oModel = new sap.ui.model.json.JSONModel();
console.log(oModel.oData, datelist.length, datelist);
oModel.attachRequestCompleted(function() {
var oFeiertageBerlin = oModel.getData().BE;
for (var prop in oFeiertageBerlin) {
datelist.push(oFeiertageBerlin[prop].datum);
}
});
var jDatum = new Date();
var jLink = "https://feiertage-api.de/api/?jahr=" + jDatum.getFullYear();
oModel.loadData(jLink);
Promise.all([
this.oModel.attachRequestCompleted
]).then(
for (var j = 0; j < datelist.length; j++) {
console.log(datelist[j]);
}
)
Expected Result: Possibility to iterate through the List
Actual Result: Syntax Error
As was indicated by the other comments, for the Promise function you would need to wrap the event callback in a Promise. However, your last comment lets me to believe that your root issue with the "this.byId" is that you'll need to bind the context for the callback. Hence:
oModel.attachRequestCompleted(function() {
// now this.byId(...) should work here
}.bind(this));
However, I would propose not using this setup and avoid the loadData function of the JSON model. The attaching of event handlers for completed / failed etc seems not be very elegant for me. I would rather go for jQuery.ajax() (https://api.jquery.com/jQuery.ajax/) or a manual XMLHttpRequest.
With jQuery it would look like this:
var oRequest = jQuery.ajax({
url: jLink
});
oRequest.done(function() {
// this.byId(...)
}.bind(this));
So, without boring anyone with the backstory, I need to access data from a number of APIs in order to run my script. The data needs to all be loaded before I execute the script, which I'm normally comfortable doing: I just declare some fetch requests, write a Promise.all, then continue on with the function.
HOWEVER, I've hit something of a snafu with a certain API that limits the number of results I can pull from one request to 100 and I need to query all of the results. I didn't think this was a huge deal since I figured I can just make a couple extra requests by affixing "&page=X" to the end of the request.
The plan, then, is to request the total number of pages from the API and then feed that into a for loop to push a number of fetch requests into an array of promises (i.e., link://to/api/data&page=1, link://to/api/data&page=2, etc). When I actually attempt to create this array with a for loop, though, the array returns empty. Here's my work:
const dataUrlNoPage = 'link/to/api/data&page=';
const totalPages = 3; //usually pulled via a function, but just using a static # for now
let apiRequestLoop = function(inp) {
return new Promise(function(resolve){
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
fetch(dataUrlLoop).then(function(response){
promiseArray.push(response.json());
})
}
resolve(promiseArray);
})
}
let finalPromiseArray = apiRequestLoop(totalPages).then(result => {
let requestArray = [apiRequest1,apiRequest2];
//requestArray contains earlier fetch promises
result.forEach(foo => {
requestArray.push(foo);
}
);
return requestArray;
});
So, what's tripping me up is really the loop, and how it's not returning an array of promises. When I look at it in the console, it shows up as a blank array, but I can expand it and see the promises I was expecting. I see the "Value below was evaluated just now" response. No matter how many promises or .thens, I write, however, the array is never actually populated at run time.
What's going on? Can I not generate fetch promises via a for loop?
(Also, just to cut this line of questioning off a bit, yes, the API I'm trying to access is Wordpress. Looking around, most people suggest creating a custom endpoint, but let's assume for the purpose of this project I am forbidden from doing that.)
You have several problems here.
The first is that you have the function provided to new Promise itself containing promise creations. Don't do this! It's a definite anti-pattern and doesn't keep your code clean.
The second is this basic bit of code:
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
fetch(dataUrlLoop).then(function(response){
promiseArray.push(response.json());
})
}
resolve(promiseArray);
This says:
create an empty array
loop through another array, doing HTTP requests
resolve your promise with the empty array
when the HTTP requests are completed, add them to the array
Step four will always come after step three.
So, you need to add the promises to your array as you go along, and have the overall promise resolve when they are all complete.
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop).then(function(response) {
return response.json();
}));
}
return Promise.all(promiseArray);
}
or, with an arrow function to clean things up:
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop).then(response => response.json()));
}
return Promise.all(promiseArray);
}
A few points:
you want to actually put the promises themselves into the array, not push to the array in a .then() chained to the promise.
you probably want to skip creating a new Promise in your function. Just get an array of all the promises from your loop, then return a Promise.all on the array.
Like this:
let apiRequestLoop = function(inp) {
let promiseArray = [];
for (let i = 1; i <= inp; i++) {
let dataUrlLoop = dataUrlNoPage + i;
promiseArray.push(fetch(dataUrlLoop))
}
return Promise.all(promiseArray);
}
In your final .then statement, in finalPromiseArray, your result will be an array of the results from all the promises. like this [response1, response2, response3, ...]
See the Promise.all documentation for more details.
There's a async call I'm making that queries a database on a service, but this service has a limit of how many it can output at once, so I need to check if it hit its limit through the result it sends and repeat the query until it doesn't.
Synchronous mockup :
var query_results = [];
var limit_hit = true; #While this is true means that the query hit the record limit
var start_from = 0; #Pagination parameter
while (limit_hit) {
Server.Query(params={start_from : start_from}, callback=function(result){
limit_hit = result.limit_hit;
start_from = result.results.length;
query_result.push(result.results);
}
}
Obviously the above does not work, I've seen some other questions here about the issue, but they don't mention what to do when you need each iteration to wait for the last one to finish and you don't know before hand the number of iterations.
How can I turn the above asynchronous? I'm open to answers using promise/deferred-like logic, but preferably something clean.
I can probably think of a monstruous and horrible way of doing this using waits/timeouts, but there has to be a clean, clever and modern way to solve it.
Another way is to make a "pre-query" to know the number of features before hand so you know the number of loops, I'm not sure if this is the correct way.
Here we use Dojo sometimes, but the examples I found does not explain what to do when you have an unknown amount of loops https://www.sitepen.com/blog/2015/06/10/dojo-faq-how-can-i-sequence-asynchronous-operations/
although many answers already, still I believe async/await is the cleanest way.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
and you might need babel
https://babeljs.io/
JS async logic syntax changed from callback to promise then to async/await, they all do the same thing, when callback nests a lot we need something like a chain, then promise come, when promise goes in loop, we need something make the chain more plain more simple, then async/await come. But not all browsers support the new syntax, so babel come to compile new syntax to old syntax, then you can always code in new syntax.
getData().then((data) => {
//do something with final data
})
async function getData() {
var query_results = [];
var limit_hit = true;
var start_from = 0;
//when you use await, handle error with try/catch
try {
while (limit_hit) {
const result = await loadPage(start_from)
limit_hit = result.limit_hit;
start_from = result.results.length;
query_result.push(result.results);
}
} catch (e) {
//when loadPage rejects
console.log(e)
return null
}
return query_result
}
async function loadPage(start_from) {
//when you use promise, handle error with reject
return new Promise((resolve, reject) => Server.Query({
start_from
}, (result, err) => {
//error reject
if (err) {
reject(err)
return
}
resolve(result)
}))
}
If you want to use a loop then I think there is no (clean) way to do it without Promises.
A different approach would be the following:
var query_results = [];
var start_from = 0;
funciton myCallback(result) {
if(!result) {
//first call
Server.Query({ start_from: start_from}, myCallback);
} else {
//repeated call
start_from = result.results.length
query_result.push(result.results);
if(!result.limit_hit) {
//limit has not been hit yet
//repeat the query with new start value
Server.Query({ start_from: start_from}, myCallback);
} else {
//call some callback function here
}
}
}
myCallback(null);
You could call this recursive, but since the Query is asynchronous you shouldn't have problems with call stack limits etc.
Using promises in an ES6 environment you could make use of async/await. Im not sure if this is possible with dojo.
You don't understand callbacks until you have written a rate limiter or queue ;) The trick is to use a counter: Increment the counter before the async request, and decrement it when you get the response, then you will know how many requests are "in flight".
If the server is choked you want to put the item back in the queue.
There are many things you need to take into account:
What will happen to the queue if the process is killed ?
How long to wait before sending another request ?
Make sure the callback is not called many times !
How many times should you retry ?
How long to wait before giving up ?
Make sure there are no loose ends ! (callback is never called)
When all edge cases are taken into account you will have a rather long and not so elegant solution. But you can abstract it into one function! (that returns a Promise or whatever you fancy).
If you have a user interface you also want to show a loading bar and some statistics!
You must await for the server response every time. Here a encapsulated method
var query = (function(){
var results = [];
var count = 0;
return function check(fun){
Server.Query({ start_from: count}, function(d){
count = d.results.length;
results.push(d.results);
if (d.limit_hit && fun) fun(results);
else check(fun);
});
};
})();
// Call here
var my_query = query(function(d){
// --> retrive all data when limit_hit is true)
});
You can use a generator function Generators to achieve this
For POC:
some basics
- You define a generator with an asterick *
- it exposes a next function which returns the next value
- generators can pause with yield statement internally and can resume externally by calling the next()
- While (true) will ensure that the generator is not done until limit has reached
function *limitQueries() {
let limit_hit = false;
let start_from = 0;
const query_result = [];
while (true) {
if (limit_hit) {break;}
yield Server.Query(params={start_from : start_from},
callback=function* (result) {
limit_hit = result.limit_hit;
start_from = result.results.length;
yield query_result.push(result.results);
}
}
}
So apparently, the generator function maintains its own state. Generator function exposes two properties { value, done } and you can call it like this
const gen = limitQueries();
let results = [];
let next = gen.next();
while(next.done) {
next = gen.next();
}
results = next.value;
You might have to touch your Server.Query method to handle generator callback. Hope this helps! Cheers!
I have a page than can make a different number of $http requests depending on the length of a variables, and then I want to send the data to the scope only when all the requests are finished. For this project I do not want to use jQuery, so please do not include jQuery in your answer. At the moment, the data is sent to the scope as each of the requests finish, which isn't what I want to happen.
Here is part of the code I have so far.
for (var a = 0; a < subs.length; a++) {
$http.get(url).success(function (data) {
for (var i = 0; i < data.children.length; i++) {
rData[data.children.name] = data.children.age;
}
});
}
Here is the part that I am sceptical about, because something needs to be an argument for $q.all(), but it is not mentioned on the docs for Angular and I am unsure what it is meant to be.
$q.all().then(function () {
$scope.rData = rData;
});
Thanks for any help.
$http call always returns a promise which can be used with $q.all function.
var one = $http.get(...);
var two = $http.get(...);
$q.all([one, two]).then(...);
You can find more details about this behaviour in the documentation:
all(promises)
promises - An array or hash of promises.
In your case you need to create an array and push all the calls into it in the loop. This way, you can use $q.all(…) on your array the same way as in the example above:
var arr = [];
for (var a = 0; a < subs.length; ++a) {
arr.push($http.get(url));
}
$q.all(arr).then(function (ret) {
// ret[0] contains the response of the first call
// ret[1] contains the second response
// etc.
});