Promises inside of setInterval - javascript

I've got this routine function that runs every 60000ms on a setInterval function. Inside of this routine function, I loop over all the usernames on a parsed JSON(db) and check whether or not they're available via a promise (checkUsername), which is a network request.
However, It's pretty clear to me that this is a terrible approach because the promises may take more than 60 seconds to complete, and I've been getting ETIMEDOUT errors all over the place. But I just don't understand promises and asynchrony enough to think of a solution.
What would be a better approach to this?
Would async/await fit here?
function routine() {
db.users.forEach(userObj => {
userObj.username.forEach(username => {
checkUsername(username).then(hasUser => {
if(!hasUser) {
bot.sendMessage(userObj.chatId, `‼️ Username #${username} is now AVAILABLE ‼️`);
removeName(username, userObj.chatId);
}
}).catch(err => {
console.log(err);
})
})
});
}
setInterval(routine, 120000);

var interval = 60000;
function routine() {
return Promise.all(db.users.map(userObj => {
return Promise.all(userObj.username.map(username => {
return checkUsername(username).then(hasUser => {
if(!hasUser){
return removeName(username, userObj.chatId).then(function(){
return bot.sendMessage(userObj.chatId, `‼️ Username #${username} is now AVAILABLE ‼️`)
})
}
}))
})).then(function(){
setTimeout(routine, interval);
}, function(error){
console.error(error);
setTimeout(routine, interval);
})
}
routine();
This will run every 60 seconds plus the time it takes to resolve all the requests. If it fails, it will run again in 60 seconds. You could be running into timeout issues if you have too many users. Also, if "removeName" fails, "bot.sendMessage" won't run.
You can end promises many different ways. It depends on what you need to do with the promise.
.then(function(){
//do stuff
setTimeout(routine, interval);
}, function(error){
console.error(error);
setTimeout(routine, interval);
})
or
.then(function(){
//do stuff
setTimeout(routine, interval);
}).catch(function(error){
console.error(error);
setTimeout(routine, interval);
})
or
.catch(function(error){
console.error(error);
}).finally(function(){
setTimeout(routine, interval);
})
or
.finally(function(){
setTimeout(routine, interval);
})

I made a code snippet that you can run that makes use of Promise.all as well as async/await ES7 to make your code a bit easier to handle and understand. I'm also hitting a real API endpoint I found online just for the sake of a complete example.
I've also added a way to stop the timeout for good if you ever want that option.
// How often the timeout will run.
// Since the timeout is dependent on when all the requests finish, the timeout will run this many ms after all the requests finish.
var interval = 5000;
// This is what your db seems to resemble.
var db = {
users: [{
username: ['1']
},
{
username: ['2']
},
{
username: ['3']
},
{
username: ['4']
},
]
};
// This will hold the timeout function so you can cancel it at a later time.
var timeoutFn;
// Returns returns a single Promise that resolves when all of the promises it contains have resolved/rejected. It rejects with the first promise that rejects.
function routine() {
console.log("-- Running routine function --");
// Return an array of promises. Please see my comments at the bottom of this whole answer which questions db architecture with username being an array.
// I'm also using map instead of forEach because map automatically returns and array.
let promiseArray = db.users.map(userObj => {
return Promise.all(userObj.username.map(username => {
// This processUsername() function should do all the work related to the username. It helps to keep the routine function as clean as possible since there's already a lot happening in here.
return processUsername(username);
}));
});
// Returns an array of array of promises. This means that every single promise within each array (see above) has to resolve before the `then` runs. If any reject, `catch` will run instead.
return Promise.all(promiseArray).then(() => {
runRoutineAgain();
}).catch((err) => {
console.log('err:', err)
});
}
// This will create a timeout and run routine after interval.
function runRoutineAgain() {
timeoutFn = setTimeout(routine, interval);
}
// This async function returns a promise
async function processUsername(username) {
// Make API call to get data
console.log('Processing username for', username);
// I'm using this free API endpoint online just for the sake of making a complete example. Obviously your API will be returning very different data.
return await fetch(`https://jsonplaceholder.typicode.com/todos/${username}`)
.then(response => response.json())
.then((json) => {
console.log(json);
// This is where you can do your processing.
// if(!hasUser) {
// bot.sendMessage(userObj.chatId, `‼️ Username #${username} is now AVAILABLE ‼️`);
// removeName(username, userObj.chatId);
// }
});
}
function stopTimeout() {
clearTimeout(timeoutFn);
}
routine();
Basically, I'm using Promise.all to capture and wait on the result of individual promises which is very useful since you have many users you need to get data for.
Feel free to open up Web Console to better see the output data.
I'm also using async/await ES7 syntax just to demonstrate other (easier, some might say) ways to write Promises. I understand Promises can be daunting so here are some links that really hit the nail on the head when trying to learn them.
https://javascript.info/promise-basics - This covers Promises
https://javascript.info/async-await - This covers async/await
Also is there a reason why you are looping over each username for a user in your original code?
db.users.forEach(userObj => {
userObj.username.forEach(username => {…
If there’s only 1 username for a userObj, that second loop adds unnecessary complexity. But if your db has multiple usernames for a single userObj, then that’s completely fine!

One way is to clear the interval as soon as you start handling the interval-function. Then when your interval-function is done, you can start the interval again. This way your interval is not ticking while your interval-function is being executed. Take a look at below code.
let interval = setInterval(routine, 1000)
function routine() {
let sequence = Promise.resolve()
sequence
.then(function () {
clearInterval(interval)
return Promise.resolve()
})
// ... chain rest of your promise here.
.then(function (value) {
return new Promise(function (resolve) {
setTimeout(function () {
console.log('I take long time.');
resolve()
}, 5000)
})
})
.then(function () { // in last promise, start the timer again.
interval = setInterval(routine, 1000)
})
}
Here you interval-function is supposed to run every 1s, but the promise itself takes 5s. So the first thing the method does is stop the timer, then do its job, and finally re-enable the timer.
With timeout,
let timeout = setTimeout(routine, 1000)
function routine() {
let sequence = Promise.resolve()
sequence
.then(function () {
return Promise.resolve()
})
// ... chain rest of your promise here.
.then(function (value) {
return new Promise(function (resolve) {
setTimeout(function () {
console.log('I take long time.');
resolve()
}, 5000)
})
})
.then(function () { // in last promise, start the timer again.
timeout = setTimeout(routine, 1000)
})
}

Related

nodejs/javascript on stream data post loop delay

I am trying to use a twitter npm to search for tweets in realtime and like them. It streams the tweets data and then uses .post to create the likes.
Currently works but I keep running into 429 too many request errors because of the api rate limit. Ive been trying to get it to pause after each like, however nothing I've tried seems to work. At most it effects the loop before or after but never in between the post/like action.
Any ideas how to get it to delay after each post(like)? I've commented out some of the things I've already tried.
// Returns a Promise that resolves after "ms" Milliseconds
const timer = ms => new Promise(res => setTimeout(res, ms))
const wait = (duration, ...args) => new Promise(resolve => {
setTimeout(resolve, duration, ...args);
});
function LikeTweets() {
client.stream('statuses/filter', { track: terms }, function (stream) {
stream.on('data', async function (tweet) {
// try {
// for (var i = 0; i < 3;) {
v1Client.post('favorites/create', { id: tweet.id_str })
.then(async (result) => {
console.log(result.text);
i++;
console.log(i);
await timer(10000);
}).then(async (newresult) => {
console.log(newresult);
await timer(10000);
}).catch(error => {
console.log(error);
return;
});
//await timer(3000); // then the created Promise can be awaited
// }
// } catch(err) {
// console.log("or catching here?");
// setTimeout(function() {
// LikeTweets();
// }, 15000);
// }
});
});
}
setTimeout(function() {
LikeTweets();
}, 15000);
You make one request per invocation of the stream.on("data", ...) event handler, therefore if 100 data events arrive within a minute, you will make 100 requests within that minute. This exceeds the rate limit.
You must ensure that the sequence of requests made is slower than the sequence of incoming events. The following code illustrates how this decoupling of sequences can be achieved:
/* Make one request every 20 seconds. */
var requestQueue = [];
function processQueue() {
var r = requestQueue.shift();
if (r) v1Client.post("favorites/create", r.payload).then(r.resolve, r.reject);
setTimeout(processQueue, 20000);
}
processQueue();
/* Use this function to schedule another request. */
function makeRequest(payload) {
var r = {payload};
requestQueue.push(r);
return new Promise(function(resolve, reject) {
r.resolve = resolve;
r.reject = reject;
});
}
stream.on("data", function(tweet) {
makeRequest({id: tweet.id_str}).then(async (result) => {...});
});
The promise returned by makeRequest can take a while until it resolves, therefore the code {...} may be executed only after several seconds or even minutes. In other words: The code uses the power of promises to keep within the strictures of the API rate limit.
This works only if, in the long run average, the number of incoming data events does not exceed the possible rate of outgoing requests, which is 1 every 20 seconds. This is nothing you can get around without a mass-update API (which would not be in the interest of the Twitter community, I assume).

How to start next asynchronous task with Observable

In some cases we need to start asynchronous task like:
setTimeout(() => {
// do something
}, 0);
It starts asynchronous task for content.
But it is ugly and not recommended.
How could I do that with Observable?
I would use Promise rather than Observable. It's native and you can acheive the same things with it :
let myPromise = new Promise((resolve, reject) => {
// we do something
if(/* an error has occured */) {
// we don't want to return in the then function if there is an error.
reject(theError);
}
let valueToReturn = "we want to return this from the promise";
resolve(valueToReturn);
});
myPromise.then((data) => {
//the async process is done
console.log(data) // <-- "we want to return this from the promise"
}).catch((error) => {
// we print the error if there is one.
console.error(error);
});
Sure, you can use some wonderful function with Observable, such as pipe, but if you don't need it, you don't need an whole new library just for that.
For this scenario (setTimeout, 0) you can use Schedulers. Internally the asyncScheduler works with an setInterval.
source$.pipe(
observeOn(asyncScheduler)
);

promises with a recursive call in error Nodejs

requestReport()
.then(getReportData)
.then(checkReportStatus)
.then(handleData)
checkReportStatus = (data) => {
return new Promise((resolve, reject) => {
if(data.status === 'completed')
resolve(data)
else {
setTimeout(() => getReportData(), 1000)
}
So I make a report request first, then I need to check whether report was produced(status would turn into completed). If it is not completed yet , I need to call getReportData again. But I'm really confused with promises. My code is actually many lines and there is around 15 chainings going. What is the correct way to make call if report is not completed, so that when it becomes completed it can just continue from handleData?
edit1: typo
If getReportData() returns a promise that resolves with the data (which it appears to in your .then() chain) and doesn't need any input parameters from requestReport(), then you can just do this:
// utility function that returns a promise that resolves after a delay
// useful for inserting a delay into a promise chain
function delay(t) {
return new Promise(function(resolve) {
setTimeout(resolve, t);
});
}
requestReport()
.then(getReportData)
.then(checkReportStatus)
.then(handleData)
function checkReportStatus(data) {
if (data.status === 'completed') {
return data;
} else {
// retry after a delay
// add retry promise to the current promise chain
return delay(1000).then(getReportData);
}
}
If you need the results of requestReport() as arguments to getReportData(), then you will have pass that info into the promise chain so the retry on getReportData() can use it. You'd have to show us more detail (what data is returned from requestReport() and what data is needed by getReportData()) for us to make a specific recommendation on how best to do that.
checkReportStatus (data) => {
if (data.status === 'completed') {
return data;
} else {
// retry after a delay
delay(1000).then(checkReportStatus(data));
}
}
try call checkReportStatus(data).

JS Promise - instantly retrieve some data from a function that returns a Promise

Can anyone recommend a pattern for instantly retrieving data from a function that returns a Promise?
My (simplified) example is an AJAX preloader:
loadPage("index.html").then(displayPage);
If this is downloading a large page, I want to be able to check what's happening and perhaps cancel the process with an XHR abort() at a later stage.
My loadPage function used to (before Promises) return an id that let me do this later:
var loadPageId = loadPage("index.html",displayPage);
...
doSomething(loadPageId);
cancelLoadPage(loadPageId);
In my new Promise based version, I'd imagine that cancelLoadPage() would reject() the original loadPage() Promise.
I've considered a few options all of which I don't like. Is there a generally accepted method to achieve this?
Okay, let's address your bounty note first.
[Hopefully I'll be able to grant the points to someone who says more than "Don't use promises"... ]
Sorry, but the answer here is: "Don't use promises". ES6 Promises have three possible states (to you as a user): Pending, Resolved and Rejected (names may be slightly off).
There is no way for you to see "inside" of a promise to see what has been done and what hasn't - at least not with native ES6 promises. There was some limited work (in other frameworks) done on promise notifications, but those did not make it into the ES6 specification, so it would be unwise of you to use this even if you found an implementation for it.
A promise is meant to represent an asynchronous operation at some point in the future; standalone, it isn't fit for this purpose. What you want is probably more akin to an event publisher - and even that is asynchronous, not synchronous.
There is no safe way for you to synchronously get some value out of an asynchronous call, especially not in JavaScript. One of the main reasons for this is that a good API will, if it can be asynchronous, will always be asynchronous.
Consider the following example:
const promiseValue = Promise.resolve(5)
promiseValue.then((value) => console.log(value))
console.log('test')
Now, let's assume that this promise (because we know the value ahead of time) is resolved synchronously. What do you expect to see? You'd expect to see:
> 5
> test
However, what actually happens is this:
> test
> 5
This is because even though Promise.resolve() is a synchronous call that resolves an already-resolved Promise, then() will always be asynchronous; this is one of the guarantees of the specification and it is a very good guarantee because it makes code a lot easier to reason about - just imagine what would happen if you tried to mix synchronous and asynchronous promises.
This applies to all asynchronous calls, by the way: any action in JavaScript that could potentially be asynchronous will be asynchronous. As a result, there is no way for you do any kind of synchronous introspection in any API that JavaScript provides.
That's not to say you couldn't make some kind of wrapper around a request object, like this:
function makeRequest(url) {
const requestObject = new XMLHttpRequest()
const result = {
}
result.done = new Promise((resolve, reject) => {
requestObject.onreadystatechange = function() {
..
}
})
requestObject.open(url)
requestObject.send()
return requestObject
}
But this gets very messy, very quickly, and you still need to use some kind of asynchronous callback for this to work. This all falls down when you try and use Fetch. Also note that Promise cancellation is not currently a part of the spec. See here for more info on that particular bit.
TL:DR: synchronous introspection is not possible on any asynchronous operation in JavaScript and a Promise is not the way to go if you were to even attempt it. There is no way for you to synchronously display information about a request that is on-going, for example. In other languages, attempting to do this would require either blocking or a race condition.
Well. If using angular you can make use of the timeout parameter used by the $http service if you need to cancel and ongoing HTTP request.
Example in typescript:
interface ReturnObject {
cancelPromise: ng.IPromise;
httpPromise: ng.IHttpPromise;
}
#Service("moduleName", "aService")
class AService() {
constructor(private $http: ng.IHttpService
private $q: ng.IQService) { ; }
doSomethingAsynch(): ReturnObject {
var cancelPromise = this.$q.defer();
var httpPromise = this.$http.get("/blah", { timeout: cancelPromise.promise });
return { cancelPromise: cancelPromise, httpPromise: httpPromise };
}
}
#Controller("moduleName", "aController")
class AController {
constructor(aService: AService) {
var o = aService.doSomethingAsynch();
var timeout = setTimeout(() => {
o.cancelPromise.resolve();
}, 30 * 1000);
o.httpPromise.then((response) => {
clearTimeout(timeout);
// do code
}, (errorResponse) => {
// do code
});
}
}
Since this approach already returns an object with two promises the stretch to include any synchronous operation return data in that object is not far.
If you can describe what type of data you would want to return synchronously from such a method it would help to identify a pattern. Why can it not be another method that is called prior to or during your asynchronous operation?
You can kinda do this, but AFAIK it will require hacky workarounds. Note that exporting the resolve and reject methods is generally considered a promise anti-pattern (i.e. sign you shouldn't be using promises). See the bottom for something using setTimeout that may give you what you want without workarounds.
let xhrRequest = (path, data, method, success, fail) => {
const xhr = new XMLHttpRequest();
// could alternately be structured as polymorphic fns, YMMV
switch (method) {
case 'GET':
xhr.open('GET', path);
xhr.onload = () => {
if (xhr.status < 400 && xhr.status >= 200) {
success(xhr.responseText);
return null;
} else {
fail(new Error(`Server responded with a status of ${xhr.status}`));
return null;
}
};
xhr.onerror = () => {
fail(networkError);
return null;
}
xhr.send();
return null;
}
return xhr;
case 'POST':
// etc.
return xhr;
// and so on...
};
// can work with any function that can take success and fail callbacks
class CancellablePromise {
constructor (fn, ...params) {
this.promise = new Promise((res, rej) => {
this.resolve = res;
this.reject = rej;
fn(...params, this.resolve, this.reject);
return null;
});
}
};
let p = new CancellablePromise(xhrRequest, 'index.html', null, 'GET');
p.promise.then(loadPage).catch(handleError);
// times out after 2 seconds
setTimeout(() => { p.reject(new Error('timeout')) }, 2000);
// for an alternative version that simply tells the user when things
// are taking longer than expected, NOTE this can be done with vanilla
// promises:
let timeoutHandle = setTimeout(() => {
// don't use alert for real, but you get the idea
alert('Sorry its taking so long to load the page.');
}, 2000);
p.promise.then(() => clearTimeout(timeoutHandle));
Promises are beautiful. I don't think there is any reason that you can not handle this with promises. There are three ways that i can think of.
The simplest way to handle this is within the executer. If you would like to cancel the promise (like for instance because of timeout) you just define a timeout flag in the executer and turn it on with a setTimeout(_ => timeout = true, 5000) instruction and resolve or reject only if timeout is false. ie (!timeout && resolve(res) or !timeout && reject(err)) This way your promise indefinitely remains unresolved in case of a timeout and your onfulfillment and onreject functions at the then stage never gets called.
The second is very similar to the first but instead of keeping a flag you just invoke reject at the timeout with proper error description. And handle the rest at the then or catch stage.
However if you would like to carry the id of your asych operation to the sync world then you can also do it as follows;
In this case you have to promisify the async function yourself. Lets take an example. We have an async function to return the double of a number. This is the function
function doubleAsync(data,cb){
setTimeout(_ => cb(false, data*2),1000);
}
We would like to use promises. So normally we need a promisifier function which will take our async function and return another function which when run, takes our data and returns a promise. Right..? So here is the promisifier function;
function promisify(fun){
return (data) => new Promise((resolve,reject) => fun(data, (err,res) => err ? reject(err) : resolve(res)));
}
Lets se how they work together;
function promisify(fun){
return (data) => new Promise((resolve,reject) => fun(data, (err,res) => err ? reject(err) : resolve(res)));
}
function doubleAsync(data,cb){
setTimeout(_ => cb(false, data*2),1000);
}
var doubleWithPromise = promisify(doubleAsync);
doubleWithPromise(100).then(v => console.log("The asynchronously obtained result is: " + v));
So now you see our doubleWithPromise(data) function returns a promise and we chain a then stage to it and access the returned value.
But what you need is not only a promise but also the id of your asynch function. This is very simple. Your promisified function should return an object with two properties; a promise and an id. Lets see...
This time our async function will return a result randomly in 0-5 secs. We will obtain it's result.id synchronously along with the result.promise and use this id to cancel the promise if it fails to resolve within 2.5 secs. Any figure on console log Resolves in 2501 msecs or above will result nothing to happen and the promise is practically canceled.
function promisify(fun){
return function(data){
var result = {id:null, promise:null}; // template return object
result.promise = new Promise((resolve,reject) => result.id = fun(data, (err,res) => err ? reject(err) : resolve(res)));
return result;
};
}
function doubleAsync(data,cb){
var dur = ~~(Math.random()*5000); // return the double of the data within 0-5 seconds.
console.log("Resolve in " + dur + " msecs");
return setTimeout(_ => cb(false, data*2),dur);
}
var doubleWithPromise = promisify(doubleAsync),
promiseDataSet = doubleWithPromise(100);
setTimeout(_ => clearTimeout(promiseDataSet.id),2500); // give 2.5 seconds to the promise to resolve or cancel it.
promiseDataSet.promise
.then(v => console.log("The asynchronously obtained result is: " + v));
You can use fetch(), Response.body.getReader(), where when .read() is called returns a ReadableStream having a cancel method, which returns a Promise upon cancelling read of the stream.
// 58977 bytes of text, 59175 total bytes
var url = "https://gist.githubusercontent.com/anonymous/"
+ "2250b78a2ddc80a4de817bbf414b1704/raw/"
+ "4dc10dacc26045f5c48f6d74440213584202f2d2/lorem.txt";
var n = 10000;
var clicked = false;
var button = document.querySelector("button");
button.addEventListener("click", () => {clicked = true});
fetch(url)
.then(response => response.body.getReader())
.then(reader => {
var len = 0;
reader.read().then(function processData(result) {
if (result.done) {
// do stuff when `reader` is `closed`
return reader.closed.then(function() {
return "stream complete"
});
};
if (!clicked) {
len += result.value.byteLength;
}
// cancel stream if `button` clicked or
// to bytes processed is greater than 10000
if (clicked || len > n) {
return reader.cancel().then(function() {
return "read aborted at " + len + " bytes"
})
}
console.log("len:", len, "result value:", result.value);
return reader.read().then(processData)
})
.then(function(msg) {
alert(msg)
})
.catch(function(err) {
console.log("err", err)
})
});
<button>click to abort stream</button>
The method I am currently using is as follows:
var optionalReturnsObject = {};
functionThatReturnsPromise(dataToSend, optionalReturnsObject ).then(doStuffOnAsyncComplete);
console.log("Some instant data has been returned here:", optionalReturnsObject );
For me, the advantage of this is that another member of my team can use this in a simple way:
functionThatReturnsPromise(data).then(...);
And not need to worry about the returns object. An advanced user can see from the definitions what is going on.

How to finish one operation in Node.js if some limit time is exceeded?

Consider that we want to perform some operation which takes some time to complete in Node.js, say doHeavyWork(). What we usually do is that we pass a callback function like that:
doHeavyWork(params, callbackFunction);
Now, I want to stablish one limit time for the operation to complete. What I want is that if the uptime of that operation exceeds the limit time, the function terminates and informs the callback that the limit time was exceeded.
How can this be accomplished in Node.js?
You'd think this is possible with a simple callback, but I'm going to illustrate a problem for you
function doHeavyWork(params, callback) {
// set the timeout
var timeout = setTimeout(callback, 5000, Error('timeout'))
// some debug message
console.log('Please wait...')
// begin heavy work
// we sill simulate heavy work with setTimeout here
setTimeout(function() {
// if the function completed in time, clear the timeout
clearTimeout(timeout)
// callback without error
callback(null, 'done')
}, 3000)
}
// check to see it work
doHeavyWork('some params', function(err, res) {
if (err)
throw error
else
console.log(res)
})
Now watch what happens if you change the 3000 to a value that exceeds the 5000 ms timeout — for example, try 10000. You will see that the callback is called twice. This is not good! And not trivial to fix unless you want to clutter up your function with a ton of rubbish.
If we use a Promise, we can fix this quite easily tho
function doHeavyWork(params) {
return new Promise(function(resolve, reject) {
// set the timeout
var timeout = setTimeout(reject, 5000, Error('timeout'))
// some debug message
console.log('Please wait...')
// begin heavy work
// we sill simulate heavy work with setTimeout here
setTimeout(function() {
// if the function completed in time, clear the timeout
clearTimeout(timeout)
// callback without error
resolve('done')
}, 3000)
})
}
// check to see it work
doHeavyWork('some params').then(
function(res) { console.log(res) },
function(err) { console.log(err.message) }
)
Now watch what happens if you change the 3000 to 10000 again. The Promise "callback" will only be called once
Lastly, this code can even be improved just a little bit more using Promise.race. Promise.race will return a new Promise that resolves or rejects as soon as one of the input Promises resolves or rejects. This is perfect for our use case.
Most crucially, notice how the timeout logic is not coupled with doHeavyWork. This keeps your functions clean and nicely organized.
function doHeavyWork(params) {
return new Promise(function(resolve, reject) {
// some debug message
console.log('Please wait...')
// begin heavy work
// we sill simulate heavy work with setTimeout here
setTimeout(resolve, 3000, 'done')
})
}
function timeout(ms) {
return new Promise(function(resolve, reject) {
setTimeout(reject, ms, Error('timeout'))
})
}
// using Promise.race
Promise.race([
timeout(5000),
doHeavyWork('some params')
]).then(
function(res) { console.log(res) },
function(err) { console.log(err.message) }
)
In the code example above, change the timeout value of 5000 to something shorter like 1000 to see what the error looks like

Categories

Resources