Develop Reference

Best practice for sequential url requests nodejs

I've got a list of urls i need to request from an API, however in order to avoid causing a lot of load i would ideally like to perform these requests with a gap of x seconds. Once all the requests are completed, certain logic that doesnt matter follows.
There are many ways to go about it, i've implemented a couple.
A) Using a recursive function that goes over an array that holds all the urls and calls itself when each request is done and a timeout has happened
B) Setting timeouts for every request in a loop with incremental delays and returning promises which upon resolution using Promise.all execute the rest of the logic and so on.
These both work. However, what would you say is the recommended way to go about this? This is more of an academic type of question and as im doing this to learn i would rather avoid using a library that abstracts the juice.

Your solutions are almost identical. Thought I would choose a bit different approach. I would make initial promise and sleep promise function, then I would chain them together.
function sleep(time){
return new Promise(resolve => setTimeout(resolve, ms));
}
ApiCall()
.then(sleep(1000))
.then(nextApiCall())...
Or more modular version
var promise = Promise.resolve()
myApiCalls.forEach(call => {
promise = promise.then(call).then(() => sleep(1000))
})
In the end, go with what you understand, what make you most sense and what you will understand in month. The one that you can read best is you preferred solution, performance won’t matter here.

You could use something like this to throttle per period.
If you want all urls to be processed even when some fail you could catch the failed ones and pick them out in the result.
Code would look something like this:
const Fail = function(details){this.details=details;};
twoPerSecond = throttlePeriod(2,1000);
urls = ["http://url1","http://url2",..."http://url100"];
Promise.all(//even though a 100 promises are created only 2 per second will be started
urls.map(
(url)=>
//pass fetch function to twoPerSecond, twoPerSecond will return a promise
// immediately but will not start fetch untill there is an available timeslot
twoPerSecond(fetch)(url)
.catch(e=>new Fail([e,url]))
)
)
.then(
results=>{
const failed = results.map(result=>(result&&result.constuctor)===Fail);
const succeeded = results.map(result=>(result&&result.constuctor)!==Fail);
}
)

removing promise dependencies

I'm aware of the power of promises, however I have several old functions that are synchronous:
function getSomething() {
return someExternalLibrary.functionReturnsAValue()
}
console.log(getSomething()); // eg prints 'foo'
Unfortunately, when someExternalLibrary updated, it has removed functionReturnsAValue() and has lumped me with functionReturnsAPromise():
function getSomething() {
return someExternalLibrary.functionReturnsAPromise()
}
console.log(getSomething()); // now prints '[object]'
This of course, breaks absolutely everything written that depends on what used to be a simple value.
Obviously, I'd prefer two things:
ask the original library to keep a synchronous return value. (Not going to happen -- b/c they have refused)
A way to actually wait for a value
I have read numerous articles on why promises are great, ad nauseam, but the simple fact is: If I embrace promises, all I really do is shuffle promises onto some other part of the code, which then must deal with the promise of a value...
Is there a way (in nodejs) to actually wait for a promise to get itself together?
The best I can find is to use coroutines and yield, but really, it's still passing the buck. To be clear, I want the function getSomething to continue to return a value. Is there a way to do it?
Clearly, I fear I've misunderstood something about Promises...
The app is for non-browser implementations and runs purely from the command line. I've been trying to understand how bluebird's reflect() might help, to no avail.
(Yes, I'm aware this question has been asked many times in various formats, but I can't find a suitable answer to the core issue. If anything, I'm looking for the opposite of this question. The closest related (but unhelpful) question I can find is: Managing promise dependencies.)

There's the concept of generator functions. These are a special kind of function in both syntax (asterisk notation) and semantics. Unlike regular functions, generator functions return something that's also new to ECMAScript: iterators. Iterators happen to be objects made specifically to be iterated on, e.g. with the all new for...of loop. They can be also iterated on manually by calling their 'next' method. Each such call produces an object containing two properties: 'value' (iterator's current value) and 'done' (a boolean indicating whether we reached the last value of the iterable). However, the best thing about generator functions is their ability to suspend their execution each time a keyword 'yield' is encountered. Let's have a glimpse of how it all works together:
'use strict';
let asyncTask = () =>
new Promise((resolve, reject) => {
if (Math.random() > 0.5) {
resolve(1);
} else {
reject(new Error('Something went wrong'));
}
});
let makeMeLookSync = fn => {
let iterator = fn();
let loop = result => {
!result.done && result.value.then(
res => loop(iterator.next(res)),
err => loop(iterator.throw(err))
);
};
loop(iterator.next());
};
makeMeLookSync(function* () {
try {
let result = yield asyncTask();
console.log(result);
} catch (err) {
console.log(err.message);
}
});

The short answer
I am told repeatedly: You can't undo functions that have been promisified.
Edit: An upcoming solution
It appears that the ES2017 (although still draft), goes a long way in making promisified code easier to work with:
https://ponyfoo.com/articles/understanding-javascript-async-await
It seems that there is also a node library ready for this support too: https://github.com/normalize/mz.
Using this methodology, having apis converted to Promises won't be so bad (although it still appears that promises still poison the rest of the codebase):
const fs = require('mz/fs')
async function doSomething () {
if (await fs.exists(__filename)) // do something
}
The rest of this answer is just a general commentary on the problem.
Why we need a solution
Let's start with a sample piece of traditional synchronous code, in 3 flavours from more 'older-fashioned' to 'newer':
This is the traditional javascript way, requiring exception based programming to handle unexpected errors:
function getSomething() {
if (someproblem) throw new Error('There is a problem');
return 'foo';
}
However, adding try/ catch statements becomes very laborious and tedious, very quickly.
With the advent of node.js, callbacks were made popular, which nicely circumvented the issue, since each caller was explicitly forced to deal with error conditions in the same callback. This meant less errors in the caller's code:
function getSomething(callback) {
if (callback) {
if (someproblem)
callback(new Error('There is a problem'), null);
else
callback(null, 'foo');
}
return 'foo';
}
Then, the after some teething issues, node.js quickly proved itself for server-side communications, and people were amazed at the speed that asynchronous solutions provided. Node application frameworks like Express and Meteor grew, which focused on this.
Unfortunately, using the same callback scheme quickly became troublesome and the developers dealing in asynchronous code started using Promises in an effort to linearize the code, to make it readable, like the traditional (try/catch) code was.
The problem is that it got evangenlized too much. Everyone started thinking that Promises are the way to go. Personally, I call it a poison on a codebase. Once you have anything that uses Promises, your whole codebase must become asynchronous. This is not always a sensible nor a practical solution, IMHO.
The worst of all side effects is that the above function, even though it is completely synchronous, can be written in Promises too:
var bluebird = require('bluebird');
function getSomething() {
// IMHO, this is ridiculous code, but is increasingly popular.
if (someproblem) return Promise.reject(new Error('There is a problem'));
return Promise.resolve('foo');
}
For those who doubt this is a problem, perhaps should look at the SO question: How do I convert an existing callback API to promises?. Pay particular attention to #3, Node-style callback.
So, for anyone who cares, I would like to suggest that there needs to be a 'pill' for Promises. I urge that we need more than promises: we need results, and sometimes in a timely manner.
Take a look at the default node.js api. It does not use Promises. It also provides both synchronous and asynchronous calls to appropriate parts of the api (eg File System).
For those of you who feel tempted to downvote this answer: that is your prerogative, but there are clear issues on when Promises are not the answer, and I feel strongly that there are cases when we need to be able to re-synchronize decoupled code.
I also apologize for this 'blog-post' styled answer.

Chrome system calls waiting on asynchronous results

I've working on a Chrome extension which is going to collect information from the Chrome.System apis and make use of all that data at once. The problem of course is that these calls are asynchrounous. I've not got a ton of experience with JS, so I want to make sure I'm doing this the easiest way possible.
The only way I can think of is making nested functions in the callbacks.
Something Like:
chrome.identity.getProfileUserInfo(function(userinfo){
getLocalIPs(userinfo.email, function(email, ips){
//keep passing data and nesting here.... and then do something after all calls are made
}
}
This seems like it will get very hard to read the code very quickly. What's the recommened method to do something like this. With synchronous programming, I would want to accomplish something like this:
var email = getEmail();
var ip = getIP();
var processor = getProcessor();
dosomething(email, ip, processor);

Why reinvent the wheel when JavaScript already natively has tools to deal with it?
Specifically, Promises. There's a very good article at HTML5Rock introducing the concept.
You make functions that return promises:
function getEmailPromise() {
return new Promise(function(resolve, reject) {
chrome.identity.getProfileUserInfo(function(userinfo) {
resolve(userinfo.email);
});
});
}
And then you can do exactly what you want:
var email = getEmailPromise();
var ip = getIPPromise();
var processor = getProcessorPromise();
Promise.all([email, ip, processor]).then(function(data) {
// data is [email, ip, processor]
});

Update:
It's 2019 and Promises are well supported in browsers. So see Xan's answer
Original Answer:
The nested functions become a burden a few levels in. I would suggest running each asynchronous call in order, and setting a flag for each callback on completion.
I would then create a function runIfReady() which checks if all required flags are set before launching doSomething(). You can add runIfReady() to the end of every asynchronous callback. Once all of the details are in, the doSomething() will run immediately.
You do not have to worry about launching doSomething() multiple times, because although the callbacks are asynchronous, only one will run at a time.

IndexedDb : read and return the object

Here is the code I'm using to retrieve informations from localdb :
function getById(idObject, typeObjectStore, cb_function)
{
idObject = parseInt(idObject);
var objectStore = db.transaction(typeObjectStore).objectStore(typeObjectStore);
var request = objectStore.get(idObject);
request.onerror = function(event) {
console.log('Error when loading ' + typeObjectStore);
};
request.onsuccess = function(event) {
cb_function(request.result);
};
}
I've read a lot of documentations and I can see that the retrieved content is usually printed directely in the cb_function (callback function).
I'd like to know if it's possible to return the object directly, so I would be able to exploit it in the rest of my code?
Doing something like this doesn't work, but you'll get what I'd like to do :
request.onsuccess = function(event) {
return(request.result);
};
I'm afraid it's not possible considering the asynchronous way of retrieving the data, but I'd like to have a confirmation.
Thank you in advance.
Regards,
Bdloul

It is not possible to return the value in this manner. The callback function is called asynchronously because request.onsuccess is called asynchronously. A return statement is synchronous. You need to know how to work with asynchronous Javascript (AJAX) in order to use indexedDB. Writing AJAX is primarily accomplished with passing callbacks, a continuation passing style (CPS).
Not everyone likes CPS, but it is a very powerful and simple way to express asynchronous operations in code, and sooner or later in every JavaScript programmer's career, you should learn about it.
If you are absolutely set on trying to return something, you can return a promise. Promises are not simple to learn how to use and will take a lot of understanding so proceed with caution.
Personally (my subjective opinion) I would not recommend trying to use Promises. I recommend learning more about Function.prototype.bind. You can use bind to avoid writing nested functions. Bind is also difficult to learn how to use but it also presents an elegant way to avoid nested functions.

Understanding jQuery.Deferred in the context of jQuey.AJAX (again)

I concede that, despite hours of reading and attempting, I am fundamentally unable to grasp something about Deferred promises and asynchrony in general.
The goal on my end is real, real simple: send some data to the server, and react to the contents of the response conditionally.
The response will always be a JSON object with save and error keys:
{ "save": true, "error":false}
// or
{ "save" : false,
"error" : "The server has run off again; authorities have been notifed."}
I have tried dozens and dozens of variations from the jQuery API, from other stackexchange answers, from tutorials, etc.. The examples all seem concerned with local asynchronous activity. When I need is some ability to be made aware when the AJAX request has either finished and returned a response I can inspect and make decisions about, or else to know that it's failed. Below, I've used comments to explain what I think is happening so someone can show me where I'm failing.
I know this is a repost; I am, apprently, worse than on average at grasping this.
var postData = {"id":7, "answer":"Ever since I went to Disneyland..."};
/* when(), as I understand it, should fire an event to be
responded to by then() when it's contents have run their course */
var result = $.when(
/* here I believe I'm supposed to assert what must complete
before the when() event has fired and before any chained
functions are subsequently called */
/* this should return a jqXHR object to then(), which is,
I'd thought, a queue of functions to call, in order,
UPON COMPLETION of the asynchronous bit */
$.post("my/restful/url", postData))
.then( function() {
/* since "this" is the jqXHR object generated in the $.post()
call above, and since it's supposed to be completed by now,
it's data key should be populated by the server's response—right? */
return this.data;
});
// alas, it isn't
console.log(result.data);
// >> undefined
Most examples I can find discuss a timeout function; but this seems, as I understand, to be a failsafe put in place to arbitrarily decide when the asynchronous part is said to have failed, rather than a means of stalling for time so the request can complete. Indeed, if all we can do is just wait it out, how's that any different from a synchronous request?
I'll even take links to a new read-mes, tutorials, etc. if they cover the material in a different way, use something other than modified examples from the jQuery API, or otherwise help this drooling idiot through the asynchronous mirk; here's where I've been reading to date:
jQuery API: Deferred
JQuery Fundamentals
jQuery Deferreds promises asynchronous bliss (blog)
StackOverflow: timeout for function (jQuery)
Update
This is in response to #Kevin B below:
I tried this:
var moduleA = {
var moduleB = {
postData: {"id":7, "answer":"Ever since I went to Disneyland..."};
save: function() {
return $.post("path/to/service", postData, null, "JSON");
}
};
var result = this.moduleB.save();
result.done(function(resp) {
if (resp.saved == true) {
// never reached before completion
console.log("yahoo");
} else {
console.log("Error: " + resp.error);
// >> undefined
}
});
}

You are over-complicating your code. You cannot get the data to outside of the callback, no matter how many deferred/promises you create/use (your sample creates 3 different deferred objects!)
Use the done callback.
var postData = {"id":7, "answer":"Ever since I went to Disneyland..."};
$.post("my/restful/url", postData).done(function (result) {
console.log(result.save, result.error);
});

You seem to have a misunderstanding of both asynchronous requests, the Promise pattern, and Javascripts mechanism of passing functions as an argument.
To understand what's really happening in your code I suggest you use a debugger and set some breakpoints in the code. Or, alternatively, add some console.logs in your code. This way you can see the flow of the program and might understand it better. Also be sure to log the arguments of the function you pass as an argument in the then()-method, so you understand what is passed.

ok you got it half right. the problem is that when you execute the console.log the promised is not yet fulfilled the asynchronous nature of the promises allows the code to execute before that ajax operation is done. also result is a deferred not a value, you need to handle your promised with .done instead of .then if you wish to return a value otherwise you'll continue passing promises.
so that said
var result={};
$.when(
$.post("my/restful/url", postData))
.done( function(data) {
result.data=data;
});
// here result is an object and data is a undefined since the promised has no yet been resolve.
console.log(result.data);