I am trying to use a twitter npm to search for tweets in realtime and like them. It streams the tweets data and then uses .post to create the likes.
Currently works but I keep running into 429 too many request errors because of the api rate limit. Ive been trying to get it to pause after each like, however nothing I've tried seems to work. At most it effects the loop before or after but never in between the post/like action.
Any ideas how to get it to delay after each post(like)? I've commented out some of the things I've already tried.
// Returns a Promise that resolves after "ms" Milliseconds
const timer = ms => new Promise(res => setTimeout(res, ms))
const wait = (duration, ...args) => new Promise(resolve => {
setTimeout(resolve, duration, ...args);
});
function LikeTweets() {
client.stream('statuses/filter', { track: terms }, function (stream) {
stream.on('data', async function (tweet) {
// try {
// for (var i = 0; i < 3;) {
v1Client.post('favorites/create', { id: tweet.id_str })
.then(async (result) => {
console.log(result.text);
i++;
console.log(i);
await timer(10000);
}).then(async (newresult) => {
console.log(newresult);
await timer(10000);
}).catch(error => {
console.log(error);
return;
});
//await timer(3000); // then the created Promise can be awaited
// }
// } catch(err) {
// console.log("or catching here?");
// setTimeout(function() {
// LikeTweets();
// }, 15000);
// }
});
});
}
setTimeout(function() {
LikeTweets();
}, 15000);
You make one request per invocation of the stream.on("data", ...) event handler, therefore if 100 data events arrive within a minute, you will make 100 requests within that minute. This exceeds the rate limit.
You must ensure that the sequence of requests made is slower than the sequence of incoming events. The following code illustrates how this decoupling of sequences can be achieved:
/* Make one request every 20 seconds. */
var requestQueue = [];
function processQueue() {
var r = requestQueue.shift();
if (r) v1Client.post("favorites/create", r.payload).then(r.resolve, r.reject);
setTimeout(processQueue, 20000);
}
processQueue();
/* Use this function to schedule another request. */
function makeRequest(payload) {
var r = {payload};
requestQueue.push(r);
return new Promise(function(resolve, reject) {
r.resolve = resolve;
r.reject = reject;
});
}
stream.on("data", function(tweet) {
makeRequest({id: tweet.id_str}).then(async (result) => {...});
});
The promise returned by makeRequest can take a while until it resolves, therefore the code {...} may be executed only after several seconds or even minutes. In other words: The code uses the power of promises to keep within the strictures of the API rate limit.
This works only if, in the long run average, the number of incoming data events does not exceed the possible rate of outgoing requests, which is 1 every 20 seconds. This is nothing you can get around without a mass-update API (which would not be in the interest of the Twitter community, I assume).
In some cases we need to start asynchronous task like:
setTimeout(() => {
// do something
}, 0);
It starts asynchronous task for content.
But it is ugly and not recommended.
How could I do that with Observable?
I would use Promise rather than Observable. It's native and you can acheive the same things with it :
let myPromise = new Promise((resolve, reject) => {
// we do something
if(/* an error has occured */) {
// we don't want to return in the then function if there is an error.
reject(theError);
}
let valueToReturn = "we want to return this from the promise";
resolve(valueToReturn);
});
myPromise.then((data) => {
//the async process is done
console.log(data) // <-- "we want to return this from the promise"
}).catch((error) => {
// we print the error if there is one.
console.error(error);
});
Sure, you can use some wonderful function with Observable, such as pipe, but if you don't need it, you don't need an whole new library just for that.
For this scenario (setTimeout, 0) you can use Schedulers. Internally the asyncScheduler works with an setInterval.
source$.pipe(
observeOn(asyncScheduler)
);
requestReport()
.then(getReportData)
.then(checkReportStatus)
.then(handleData)
checkReportStatus = (data) => {
return new Promise((resolve, reject) => {
if(data.status === 'completed')
resolve(data)
else {
setTimeout(() => getReportData(), 1000)
}
So I make a report request first, then I need to check whether report was produced(status would turn into completed). If it is not completed yet , I need to call getReportData again. But I'm really confused with promises. My code is actually many lines and there is around 15 chainings going. What is the correct way to make call if report is not completed, so that when it becomes completed it can just continue from handleData?
edit1: typo
If getReportData() returns a promise that resolves with the data (which it appears to in your .then() chain) and doesn't need any input parameters from requestReport(), then you can just do this:
// utility function that returns a promise that resolves after a delay
// useful for inserting a delay into a promise chain
function delay(t) {
return new Promise(function(resolve) {
setTimeout(resolve, t);
});
}
requestReport()
.then(getReportData)
.then(checkReportStatus)
.then(handleData)
function checkReportStatus(data) {
if (data.status === 'completed') {
return data;
} else {
// retry after a delay
// add retry promise to the current promise chain
return delay(1000).then(getReportData);
}
}
If you need the results of requestReport() as arguments to getReportData(), then you will have pass that info into the promise chain so the retry on getReportData() can use it. You'd have to show us more detail (what data is returned from requestReport() and what data is needed by getReportData()) for us to make a specific recommendation on how best to do that.
checkReportStatus (data) => {
if (data.status === 'completed') {
return data;
} else {
// retry after a delay
delay(1000).then(checkReportStatus(data));
}
}
try call checkReportStatus(data).
Can anyone recommend a pattern for instantly retrieving data from a function that returns a Promise?
My (simplified) example is an AJAX preloader:
loadPage("index.html").then(displayPage);
If this is downloading a large page, I want to be able to check what's happening and perhaps cancel the process with an XHR abort() at a later stage.
My loadPage function used to (before Promises) return an id that let me do this later:
var loadPageId = loadPage("index.html",displayPage);
...
doSomething(loadPageId);
cancelLoadPage(loadPageId);
In my new Promise based version, I'd imagine that cancelLoadPage() would reject() the original loadPage() Promise.
I've considered a few options all of which I don't like. Is there a generally accepted method to achieve this?
Okay, let's address your bounty note first.
[Hopefully I'll be able to grant the points to someone who says more than "Don't use promises"... ]
Sorry, but the answer here is: "Don't use promises". ES6 Promises have three possible states (to you as a user): Pending, Resolved and Rejected (names may be slightly off).
There is no way for you to see "inside" of a promise to see what has been done and what hasn't - at least not with native ES6 promises. There was some limited work (in other frameworks) done on promise notifications, but those did not make it into the ES6 specification, so it would be unwise of you to use this even if you found an implementation for it.
A promise is meant to represent an asynchronous operation at some point in the future; standalone, it isn't fit for this purpose. What you want is probably more akin to an event publisher - and even that is asynchronous, not synchronous.
There is no safe way for you to synchronously get some value out of an asynchronous call, especially not in JavaScript. One of the main reasons for this is that a good API will, if it can be asynchronous, will always be asynchronous.
Consider the following example:
const promiseValue = Promise.resolve(5)
promiseValue.then((value) => console.log(value))
console.log('test')
Now, let's assume that this promise (because we know the value ahead of time) is resolved synchronously. What do you expect to see? You'd expect to see:
> 5
> test
However, what actually happens is this:
> test
> 5
This is because even though Promise.resolve() is a synchronous call that resolves an already-resolved Promise, then() will always be asynchronous; this is one of the guarantees of the specification and it is a very good guarantee because it makes code a lot easier to reason about - just imagine what would happen if you tried to mix synchronous and asynchronous promises.
This applies to all asynchronous calls, by the way: any action in JavaScript that could potentially be asynchronous will be asynchronous. As a result, there is no way for you do any kind of synchronous introspection in any API that JavaScript provides.
That's not to say you couldn't make some kind of wrapper around a request object, like this:
function makeRequest(url) {
const requestObject = new XMLHttpRequest()
const result = {
}
result.done = new Promise((resolve, reject) => {
requestObject.onreadystatechange = function() {
..
}
})
requestObject.open(url)
requestObject.send()
return requestObject
}
But this gets very messy, very quickly, and you still need to use some kind of asynchronous callback for this to work. This all falls down when you try and use Fetch. Also note that Promise cancellation is not currently a part of the spec. See here for more info on that particular bit.
TL:DR: synchronous introspection is not possible on any asynchronous operation in JavaScript and a Promise is not the way to go if you were to even attempt it. There is no way for you to synchronously display information about a request that is on-going, for example. In other languages, attempting to do this would require either blocking or a race condition.
Well. If using angular you can make use of the timeout parameter used by the $http service if you need to cancel and ongoing HTTP request.
Example in typescript:
interface ReturnObject {
cancelPromise: ng.IPromise;
httpPromise: ng.IHttpPromise;
}
#Service("moduleName", "aService")
class AService() {
constructor(private $http: ng.IHttpService
private $q: ng.IQService) { ; }
doSomethingAsynch(): ReturnObject {
var cancelPromise = this.$q.defer();
var httpPromise = this.$http.get("/blah", { timeout: cancelPromise.promise });
return { cancelPromise: cancelPromise, httpPromise: httpPromise };
}
}
#Controller("moduleName", "aController")
class AController {
constructor(aService: AService) {
var o = aService.doSomethingAsynch();
var timeout = setTimeout(() => {
o.cancelPromise.resolve();
}, 30 * 1000);
o.httpPromise.then((response) => {
clearTimeout(timeout);
// do code
}, (errorResponse) => {
// do code
});
}
}
Since this approach already returns an object with two promises the stretch to include any synchronous operation return data in that object is not far.
If you can describe what type of data you would want to return synchronously from such a method it would help to identify a pattern. Why can it not be another method that is called prior to or during your asynchronous operation?
You can kinda do this, but AFAIK it will require hacky workarounds. Note that exporting the resolve and reject methods is generally considered a promise anti-pattern (i.e. sign you shouldn't be using promises). See the bottom for something using setTimeout that may give you what you want without workarounds.
let xhrRequest = (path, data, method, success, fail) => {
const xhr = new XMLHttpRequest();
// could alternately be structured as polymorphic fns, YMMV
switch (method) {
case 'GET':
xhr.open('GET', path);
xhr.onload = () => {
if (xhr.status < 400 && xhr.status >= 200) {
success(xhr.responseText);
return null;
} else {
fail(new Error(`Server responded with a status of ${xhr.status}`));
return null;
}
};
xhr.onerror = () => {
fail(networkError);
return null;
}
xhr.send();
return null;
}
return xhr;
case 'POST':
// etc.
return xhr;
// and so on...
};
// can work with any function that can take success and fail callbacks
class CancellablePromise {
constructor (fn, ...params) {
this.promise = new Promise((res, rej) => {
this.resolve = res;
this.reject = rej;
fn(...params, this.resolve, this.reject);
return null;
});
}
};
let p = new CancellablePromise(xhrRequest, 'index.html', null, 'GET');
p.promise.then(loadPage).catch(handleError);
// times out after 2 seconds
setTimeout(() => { p.reject(new Error('timeout')) }, 2000);
// for an alternative version that simply tells the user when things
// are taking longer than expected, NOTE this can be done with vanilla
// promises:
let timeoutHandle = setTimeout(() => {
// don't use alert for real, but you get the idea
alert('Sorry its taking so long to load the page.');
}, 2000);
p.promise.then(() => clearTimeout(timeoutHandle));
Promises are beautiful. I don't think there is any reason that you can not handle this with promises. There are three ways that i can think of.
The simplest way to handle this is within the executer. If you would like to cancel the promise (like for instance because of timeout) you just define a timeout flag in the executer and turn it on with a setTimeout(_ => timeout = true, 5000) instruction and resolve or reject only if timeout is false. ie (!timeout && resolve(res) or !timeout && reject(err)) This way your promise indefinitely remains unresolved in case of a timeout and your onfulfillment and onreject functions at the then stage never gets called.
The second is very similar to the first but instead of keeping a flag you just invoke reject at the timeout with proper error description. And handle the rest at the then or catch stage.
However if you would like to carry the id of your asych operation to the sync world then you can also do it as follows;
In this case you have to promisify the async function yourself. Lets take an example. We have an async function to return the double of a number. This is the function
function doubleAsync(data,cb){
setTimeout(_ => cb(false, data*2),1000);
}
We would like to use promises. So normally we need a promisifier function which will take our async function and return another function which when run, takes our data and returns a promise. Right..? So here is the promisifier function;
function promisify(fun){
return (data) => new Promise((resolve,reject) => fun(data, (err,res) => err ? reject(err) : resolve(res)));
}
Lets se how they work together;
function promisify(fun){
return (data) => new Promise((resolve,reject) => fun(data, (err,res) => err ? reject(err) : resolve(res)));
}
function doubleAsync(data,cb){
setTimeout(_ => cb(false, data*2),1000);
}
var doubleWithPromise = promisify(doubleAsync);
doubleWithPromise(100).then(v => console.log("The asynchronously obtained result is: " + v));
So now you see our doubleWithPromise(data) function returns a promise and we chain a then stage to it and access the returned value.
But what you need is not only a promise but also the id of your asynch function. This is very simple. Your promisified function should return an object with two properties; a promise and an id. Lets see...
This time our async function will return a result randomly in 0-5 secs. We will obtain it's result.id synchronously along with the result.promise and use this id to cancel the promise if it fails to resolve within 2.5 secs. Any figure on console log Resolves in 2501 msecs or above will result nothing to happen and the promise is practically canceled.
function promisify(fun){
return function(data){
var result = {id:null, promise:null}; // template return object
result.promise = new Promise((resolve,reject) => result.id = fun(data, (err,res) => err ? reject(err) : resolve(res)));
return result;
};
}
function doubleAsync(data,cb){
var dur = ~~(Math.random()*5000); // return the double of the data within 0-5 seconds.
console.log("Resolve in " + dur + " msecs");
return setTimeout(_ => cb(false, data*2),dur);
}
var doubleWithPromise = promisify(doubleAsync),
promiseDataSet = doubleWithPromise(100);
setTimeout(_ => clearTimeout(promiseDataSet.id),2500); // give 2.5 seconds to the promise to resolve or cancel it.
promiseDataSet.promise
.then(v => console.log("The asynchronously obtained result is: " + v));
You can use fetch(), Response.body.getReader(), where when .read() is called returns a ReadableStream having a cancel method, which returns a Promise upon cancelling read of the stream.
// 58977 bytes of text, 59175 total bytes
var url = "https://gist.githubusercontent.com/anonymous/"
+ "2250b78a2ddc80a4de817bbf414b1704/raw/"
+ "4dc10dacc26045f5c48f6d74440213584202f2d2/lorem.txt";
var n = 10000;
var clicked = false;
var button = document.querySelector("button");
button.addEventListener("click", () => {clicked = true});
fetch(url)
.then(response => response.body.getReader())
.then(reader => {
var len = 0;
reader.read().then(function processData(result) {
if (result.done) {
// do stuff when `reader` is `closed`
return reader.closed.then(function() {
return "stream complete"
});
};
if (!clicked) {
len += result.value.byteLength;
}
// cancel stream if `button` clicked or
// to bytes processed is greater than 10000
if (clicked || len > n) {
return reader.cancel().then(function() {
return "read aborted at " + len + " bytes"
})
}
console.log("len:", len, "result value:", result.value);
return reader.read().then(processData)
})
.then(function(msg) {
alert(msg)
})
.catch(function(err) {
console.log("err", err)
})
});
<button>click to abort stream</button>
The method I am currently using is as follows:
var optionalReturnsObject = {};
functionThatReturnsPromise(dataToSend, optionalReturnsObject ).then(doStuffOnAsyncComplete);
console.log("Some instant data has been returned here:", optionalReturnsObject );
For me, the advantage of this is that another member of my team can use this in a simple way:
functionThatReturnsPromise(data).then(...);
And not need to worry about the returns object. An advanced user can see from the definitions what is going on.
Consider that we want to perform some operation which takes some time to complete in Node.js, say doHeavyWork(). What we usually do is that we pass a callback function like that:
doHeavyWork(params, callbackFunction);
Now, I want to stablish one limit time for the operation to complete. What I want is that if the uptime of that operation exceeds the limit time, the function terminates and informs the callback that the limit time was exceeded.
How can this be accomplished in Node.js?
You'd think this is possible with a simple callback, but I'm going to illustrate a problem for you
function doHeavyWork(params, callback) {
// set the timeout
var timeout = setTimeout(callback, 5000, Error('timeout'))
// some debug message
console.log('Please wait...')
// begin heavy work
// we sill simulate heavy work with setTimeout here
setTimeout(function() {
// if the function completed in time, clear the timeout
clearTimeout(timeout)
// callback without error
callback(null, 'done')
}, 3000)
}
// check to see it work
doHeavyWork('some params', function(err, res) {
if (err)
throw error
else
console.log(res)
})
Now watch what happens if you change the 3000 to a value that exceeds the 5000 ms timeout — for example, try 10000. You will see that the callback is called twice. This is not good! And not trivial to fix unless you want to clutter up your function with a ton of rubbish.
If we use a Promise, we can fix this quite easily tho
function doHeavyWork(params) {
return new Promise(function(resolve, reject) {
// set the timeout
var timeout = setTimeout(reject, 5000, Error('timeout'))
// some debug message
console.log('Please wait...')
// begin heavy work
// we sill simulate heavy work with setTimeout here
setTimeout(function() {
// if the function completed in time, clear the timeout
clearTimeout(timeout)
// callback without error
resolve('done')
}, 3000)
})
}
// check to see it work
doHeavyWork('some params').then(
function(res) { console.log(res) },
function(err) { console.log(err.message) }
)
Now watch what happens if you change the 3000 to 10000 again. The Promise "callback" will only be called once
Lastly, this code can even be improved just a little bit more using Promise.race. Promise.race will return a new Promise that resolves or rejects as soon as one of the input Promises resolves or rejects. This is perfect for our use case.
Most crucially, notice how the timeout logic is not coupled with doHeavyWork. This keeps your functions clean and nicely organized.
function doHeavyWork(params) {
return new Promise(function(resolve, reject) {
// some debug message
console.log('Please wait...')
// begin heavy work
// we sill simulate heavy work with setTimeout here
setTimeout(resolve, 3000, 'done')
})
}
function timeout(ms) {
return new Promise(function(resolve, reject) {
setTimeout(reject, ms, Error('timeout'))
})
}
// using Promise.race
Promise.race([
timeout(5000),
doHeavyWork('some params')
]).then(
function(res) { console.log(res) },
function(err) { console.log(err.message) }
)
In the code example above, change the timeout value of 5000 to something shorter like 1000 to see what the error looks like