Develop Reference

nodejs/javascript on stream data post loop delay

I am trying to use a twitter npm to search for tweets in realtime and like them. It streams the tweets data and then uses .post to create the likes.
Currently works but I keep running into 429 too many request errors because of the api rate limit. Ive been trying to get it to pause after each like, however nothing I've tried seems to work. At most it effects the loop before or after but never in between the post/like action.
Any ideas how to get it to delay after each post(like)? I've commented out some of the things I've already tried.
// Returns a Promise that resolves after "ms" Milliseconds
const timer = ms => new Promise(res => setTimeout(res, ms))
const wait = (duration, ...args) => new Promise(resolve => {
setTimeout(resolve, duration, ...args);
});
function LikeTweets() {
client.stream('statuses/filter', { track: terms }, function (stream) {
stream.on('data', async function (tweet) {
// try {
// for (var i = 0; i < 3;) {
v1Client.post('favorites/create', { id: tweet.id_str })
.then(async (result) => {
console.log(result.text);
i++;
console.log(i);
await timer(10000);
}).then(async (newresult) => {
console.log(newresult);
await timer(10000);
}).catch(error => {
console.log(error);
return;
});
//await timer(3000); // then the created Promise can be awaited
// }
// } catch(err) {
// console.log("or catching here?");
// setTimeout(function() {
// LikeTweets();
// }, 15000);
// }
});
});
}
setTimeout(function() {
LikeTweets();
}, 15000);

You make one request per invocation of the stream.on("data", ...) event handler, therefore if 100 data events arrive within a minute, you will make 100 requests within that minute. This exceeds the rate limit.
You must ensure that the sequence of requests made is slower than the sequence of incoming events. The following code illustrates how this decoupling of sequences can be achieved:
/* Make one request every 20 seconds. */
var requestQueue = [];
function processQueue() {
var r = requestQueue.shift();
if (r) v1Client.post("favorites/create", r.payload).then(r.resolve, r.reject);
setTimeout(processQueue, 20000);
}
processQueue();
/* Use this function to schedule another request. */
function makeRequest(payload) {
var r = {payload};
requestQueue.push(r);
return new Promise(function(resolve, reject) {
r.resolve = resolve;
r.reject = reject;
});
}
stream.on("data", function(tweet) {
makeRequest({id: tweet.id_str}).then(async (result) => {...});
});
The promise returned by makeRequest can take a while until it resolves, therefore the code {...} may be executed only after several seconds or even minutes. In other words: The code uses the power of promises to keep within the strictures of the API rate limit.
This works only if, in the long run average, the number of incoming data events does not exceed the possible rate of outgoing requests, which is 1 every 20 seconds. This is nothing you can get around without a mass-update API (which would not be in the interest of the Twitter community, I assume).

How to create your own setTimeout function?

I understand how to use setTimeout function, but I can't find a way to create a function like it.
I have an example:
setTimeout(() => {
console.log('3s');
}, 3000);
while(1);
The result is setTimeout callback never call so I think it use the same thread like every js other functions. But when it's check the time reach or not? and how it can do that?
Updated
To avoid misunderstanding I update my question.
I can't find a way to create a async function with callback after specify time (without using setTimeout and don't block entire thread). This function setTimeout seen like a miracle to me. I want to understand how it work.

Just for the game since I really don't see why you couldn't use setTimeout...
To create a non-blocking timer, without using the setTimeout/setInterval methods, you have only two ways:
event based timer
run your infinite loop in a second thread
Event based timer
One naive implementation would be to use the MessageEvent interface and polling until the time has been reached. But that's not really advice-able for long timeouts as this would force the event-loop to constantly poll new tasks, which is bad for trees.
function myTimer(cb, ms) {
const begin = performance.now();
const channel = myTimer.channel ??= new MessageChannel();
const controller = new AbortController();
channel.port1.addEventListener("message", (evt) => {
if(performance.now() - begin >= ms) {
controller.abort();
cb();
}
else if(evt.data === begin) channel.port2.postMessage(begin);
}, { signal: controller.signal });
channel.port1.start();
channel.port2.postMessage(begin);
}
myTimer(() => console.log("world"), 2000);
myTimer(() => console.log("hello"), 100);
So instead, if available, one might want to use the Web Audio API and the AudioScheduledSourceNode, which makes great use of the high precision Audio Context's own clock:
function myTimer(cb, ms) {
if(!myTimer.ctx) myTimer.ctx = new (window.AudioContext || window.webkitAudioContext)();
var ctx = myTimer.ctx;
var silence = ctx.createGain();
silence.gain.value = 0;
var note = ctx.createOscillator();
note.connect(silence);
silence.connect(ctx.destination);
note.onended = function() { cb() };
note.start(0);
note.stop(ctx.currentTime + (ms / 1000));
}
myTimer(()=>console.log('world'), 2000);
myTimer(()=>console.log('hello'), 200);
Infinite loop on a different thread
Yes, using Web Workers we can run infinite loops without killing our web page:
function myTimer(cb, ms) {
var workerBlob = new Blob([mytimerworkerscript.textContent], {type: 'application/javascript'});
var url = URL.createObjectURL(workerBlob);
var worker = new Worker(url);
worker.onmessage = function() {
URL.revokeObjectURL(url);
worker.terminate();
cb();
};
worker.postMessage(ms);
}
myTimer(()=>console.log('world'), 2000);
myTimer(()=>console.log('hello'), 200);
<script id="mytimerworkerscript" type="application/worker-script">
self.onmessage = function(evt) {
var ms = evt.data;
var now = performance.now();
while(performance.now() - now < ms) {}
self.postMessage('done');
}
</script>
And for the ones who like to show off they know about the latest features not yet really available (totally not my style), a little mention of the incoming Prioritized Post Task API and its delayed tasks, which are basically a more powerful setTimeout, returning a promise, on which we can set prioritization.
(async () => {
if(globalThis.scheduler) {
const p1 = scheduler.postTask(()=>{ console.log("world"); }, { delay: 2000} );
const p2 = scheduler.postTask(()=>{ console.log("hello"); }, { delay: 1000} );
await p2;
console.log("future");
}
else {
console.log("Your browser doesn't support this API yet");
}
})();

The reason callback of setTimeout() is not being called is, you have while(1) in your code which acts as infinite loop. It will keep your javascript stack busy whole time and that is the reason event loop will never push callback function of setTimeout() in stack.
If you remove while(1) from your code, callback for setTimeout() should get invoked.
setTimeout(() => {
console.log('3s');
}, 3000);

To create your own setTimeout function, you can use the following function, setMyTimeout() to do that without using setTimeout.
var foo= ()=>{
console.log(3,"Called after 3 seconds",new Date().getTime());
}
var setMyTimeOut = (foo,timeOut)=>{
let timer;
let currentTime = new Date().getTime();
let blah=()=>{
if (new Date().getTime() >= currentTime + timeOut) {
clearInterval(timer);
foo()
}
}
timer= setInterval(blah, 100);
}
console.log(1,new Date().getTime());
setMyTimeOut(foo,3000)
console.log(2,new Date().getTime());

Following is the implementation of custom setTimeout and setInterval, clearTimeout and clearInterval. I created them to use in sandbox environments where builtin setTimeout and setInterval doesn't work.
const setTimeouts = [];
export function customSetTimeout(cb, interval) {
const now = window.performance.now();
const index = setTimeouts.length;
setTimeouts[index] = () => {
cb();
};
setTimeouts[index].active = true;
const handleMessage = (evt) => {
if (evt.data === index) {
if (window.performance.now() - now >= interval) {
window.removeEventListener('message', handleMessage);
if (setTimeouts[index].active) {
setTimeouts[index]();
}
} else {
window.postMessage(index, '*');
}
}
};
window.addEventListener('message', handleMessage);
window.postMessage(index, '*');
return index;
}
export function customClearTimeout(setTimeoutId) {
if (setTimeouts[setTimeoutId]) {
setTimeouts[setTimeoutId].active = false;
}
}
const setIntervals = [];
export function customSetInterval(cb, interval) {
const intervalId = setIntervals.length;
setIntervals[intervalId] = function () {
if (setIntervals[intervalId].active) {
cb();
customSetTimeout(setIntervals[intervalId], interval);
}
};
setIntervals[intervalId].active = true;
customSetTimeout(setIntervals[intervalId], interval);
return intervalId;
}
export function customClearInterval(intervalId) {
if (setIntervals[intervalId]) {
setIntervals[intervalId].active = false;
}
}

Hi you can try this. ]
HOpe it will help. Thanks
function customSetTimeOut (callback, ms) {
var dt = new Date();
var i = dt.getTime();
var future = i + ms;
while(Date.now() <= future) {
//do nothing - blocking
}
return callback();
}
customSetTimeOut(function(){
console.log("Timeout success");
},1000);

Handle promise resolves indvidually in promise.all()

There is a lot of information on how to handle errors when using promise.all() using catch but what I'm trying to achieve is to handle every time a promise inside of this promise.all() resolves. The reason I'm trying to do this is because I am attempting to setup a custom progress bar in console and I need to call the tick method every time a promise is resolved.
this.getNewSources = function () {
var bar = new ProgressBar(':bar', {total: this.getSourceMap().size});
var timer = setInterval(function () {
bar.tick();
if (bar.complete) {
console.log('\ncomplete\n');
clearInterval(timer);
}
}, 100);
let promiseArr = [];
for (let x of this.getSourceMap().values()) {
promiseArr.push(this.requestArticles(x.getName(), x.getCat(), x.getKey()));
}
return Promise.all(promiseArr).then(() => {
console.log("Articles loaded this round: " + this.articles.size);
console.log('all sources updated');
this.loadedArticles = true;
console.log(this.articleCount);
console.log(this.articles.size);
}).catch(e => {
console.log(e);
});
};
I'm trying to figure out a way of being able to call the bar.tick() method when each individual promise resolves.

(Answering my own question.)
I handled it by adding a then handler where I'm getting the promise from requestArticles (where I'm pushing them into the promiseArr array). I had to be sure to pass the value that handler receives out of the handler so it propagates to Promise.all, see *** lines:
this.getNewSources = function () {
var bar = new ProgressBar(':bar', {total: this.getSourceMap().size});
var timer = setInterval(function () {
if (bar.complete) {
console.log('\ncomplete\n');
clearInterval(timer);
}
}, 100);
function updateProgressBar() {
bar.tick()
}
let promiseArr = [];
for (let x of this.getSourceMap().values()) {
promiseArr.push(this.requestArticles(x.getName(), x.getCat(), x.getKey())
.then(value => { // ***
updateProgressBar(); // ***
return value; // ***
}) // ***
);
}
return Promise.all(promiseArr).then(() => {
console.log("Articles loaded this round: " + this.articles.size);
console.log('all sources updated');
this.loadedArticles = true;
console.log(this.articleCount);
console.log(this.articles.size);
}).catch(e => {
console.log(e);
});
};
That way, my handlers get called as the promises complete individually, and since I'm returning the value I receive, the promise created by my call to then will resolve with that value, which Promise.all will see. Rejections will skip over that handler and go straight to the handlers hooked up by Promise.all.
The ascii progress library on npm
Result output in console:
(With thanks to T.J. Crowder for his initial explanation, which made me realize I could do this where I'm pushing onto the array. He said he preferred deleting that answer and having me post this instead.)

JavaScript checking if resource is reachable with fetch

I'm basically just trying to verify if a resource is reachable from the executing client. I can not use XHR, because the target resource doesn't allow that.
I'm pretty new to JS and am currently working with this ( executable here ):
var done = false;
var i = 1;
var t = "https://i.stack.imgur.com/Ya15i.jpg";
while(!done && i < 4)
{
console.log("try "+i);
done = chk(t);
sleep(1000);
i = i+1;
if (done)
{
console.log("Reachable!");
break;
}
else
{
console.log("Unreachable.");
}
}
function chk(target)
{
console.log("checking "+target)
fetch(target, {mode: 'no-cors'}).then(r=>{
return true;
})
.catch(e=>{
return false;
});
}
// busy fake sleep
function sleep(s)
{
var now = new Date().getTime();
while(new Date().getTime() < now + s){ /* busy sleep */ }
}
I was expecting this code to check for the resource, print the result, then wait for a sec. Repeat this until 3 tries were unsuccessful or one of them was successful.
Instead the execution blocks for a while, then prints all of the console.logs at once and the resource is never reachable (which it is).
I do know that the fetch operation is asynchronous, but I figured if I previously declare done and implement a sleep it should work. In the worst case, the while loop would use the previously declared done.
How do I achieve the described behavior? Any advice is welcome.

Your sleep function is blocking, what you really want is a recursive function that returns a promise after checking the url n times with a delay of y seconds etc.
Something like this
function chk(target, times, delay) {
return new Promise((res, rej) => { // return a promise
(function rec(i) { // recursive IIFE
fetch(target, {mode: 'no-cors'}).then((r) => { // fetch the resourse
res(r); // resolve promise if success
}).catch( err => {
if (times === 0) // if number of tries reached
return rej(err); // don't try again
setTimeout(() => rec(--times), delay ) // otherwise, wait and try
}); // again until no more tries
})(times);
});
}
To be used like this
var t = "https://i.stack.imgur.com/Ya15i.jpg";
chk(t, 3, 1000).then( image => {
console.log('success')
}).catch( err => {
console.log('error')
});
And note that this does not fail on 404 or 500, any response is a successful request.

The main problem is that you are trying to return from callback. That makes no sense.
But fetch is Promise based request you can use Promise to simulate delays as well
Something like this should do the trick
// promise based delay
const delay = timeout => new Promise(resolve => setTimeout(resolve, timeout))
// check if target can be fetched
const check = target => fetch(target, {...})
.then(response => response.ok)
const ping = (target, times = 3, timeout = 1000) => check(target)
.then(found => {
if(!found && times) { // still can check
// wait then ping one more time
return delay(timeout).then(() => ping(target, times - 1, timeout))
}
return found
})
ping('https://i.stack.imgur.com/Ya15i.jpg')
.then(found => {
console.log(found ? 'Reachable': 'Unreachable')
})

Your chk function returns undefined, you return true/false from promise callbacks not from container function.
You should use recursion and timeout in catch callback.
It will be something like this:
var i = 0;
var done = false;
var t = "https://i.stack.imgur.com/Ya15i.jpg";
(function chk(target){
console.log("checking "+target)
fetch(target, {mode: 'no-cors'}).then(r=>{
done = true;
console.log("Reachable!");
})
.catch(e=>{
console.log("Unreachable.");
if(i<4){
setTimeout(function(){
chk(target)
},1000)
}
});
})(t)

You can't return within a callback. When you do, it is the callback that is returning, not the parent function. If fact, the function chk is never returning anything.
What it sounds like you are intending to do is return the promise returned by fetch. And attempt to fetch three times.
Try this:
const numberOfTries =3;
currentTry = 1;
var t = "https://i.stack.imgur.com/Ya15i.jpg";
chk(t);
function tryCheck(resource, currentTry) {
chk(resource).done(function(){
console.log("Reachable!");
}).catch(function(e) {
console.log("Unreachable.");
if (currentTry >= numberOfTries) return;
sleep(1000);
tryCheck(resource, currentTry + 1);
});
}
function chk(resource) {
console.log("checking "+target);
return fetch(target, {mode: 'no-cors'});
}

Try this, Hope it works
var myHeaders = new Headers();
myHeaders.append('Content-Type', 'image/jpeg');
var myInit = { method: 'GET',
headers: myHeaders,
mode: 'no-cors',
cache: 'default' };
var myRequest = new Request('https://i.stack.imgur.com/Ya15i.jpg');
fetch(myRequest,myInit).then(function(response) {
...
});

What is wrong with this recursive Promise function with jQuery Deferred

I am working on my scripts for ReST API processes. Now I need a function which keeps retrying to request to the API as many as possible in few seconds.
So I wrote some Promise abstractions and I made something like below:
$(function () {
var $el = $('#test');
function api (message) {
return $.ajax('/echo/json/', {
method: 'POST',
data: { json: JSON.stringify({ message: message }), delay: 2000 },
timeout: 1000
});
}
// Process to keep retrying an API request as many as possible in 3 seconds
retried(_.bind(api, undefined, 'Hello, world.'), 3000)
.then(function (dat) {
$el.text(dat.message);
}, function (err) {
if (err instanceof Error) $el.css('color', 'red');
$el.text(err.message);
});
});
Some functions I made for the above are below:
// Promise wrapper
var promisify = function (func) {
var funcPartial = function () {
var funcArgs = _.toArray(arguments);
var dfr = new $.Deferred(), promiseArgs = [ dfr.resolve, dfr.reject ];
var timeoutId = setTimeout(function () {
clearTimeout(timeoutId);
func.apply(undefined, _.union(promiseArgs, funcArgs));
}, 1);
return dfr.promise();
};
return funcPartial;
};
// Promise abstraction for recursive call
retried = promisify(function (resolve, reject, done, duration, start) {
if (!_.isNumber(start)) start = +(new Date());
return done()
.then(resolve, function (err) {
var stop = +(new Date());
if (duration <= stop - start) {
reject(err);
} else {
return retried(done, duration, start);
}
});
});
The process finishes in a correct condition, but the retried function won't return Promise chain.
I am really stacked. Can someone point me out the wrong points to correct the implementation above?
Here's the entire demo script.
DEMO
Thank you.
SOLVED
Thanks to #BenjaminGruenbaum below, I have just noticed that I did not need promisify to make retried function at all. This was completely shame question, but thanks again to all the people replied on this question.
This is revised retried function which does not need promisify at all...
var retried = function (done, duration, start) {
if (!_.isNumber(start)) start = +(new Date());
return done()
.then(function (dat) {
return dat;
}, function (err) {
var stop = +(new Date());
if (duration > stop - start) return retried(done, duration, start);
return err;
});
};
I updated the demo and it works fine now XD
DEMO (revised)

Your logic for retrying is very complicated. You're "promisifying" a function with resolve and reject and performing explicit construction. How would you implement retry in synchronous code?
do {
var failed = false;
try {
var val = fn();
} catch(e){
failed = true;
}
} while(failed);
But we can't really do that, because we can't use loops, we can however use recursion instead:
function retry(fn){
try {
return fn(); // call the function
} catch(e){
return retry(fn); // retry it again
}
}
Now, adding promises in just means that failure isn't through try/catch but through the promise's resolution/lack of:
function retry(fn) {
return fn().then(null, function(e){
return retry(fn); // on failure handler, retry again.
});
}
This implementation has the benefit of not depending on the promise library used (it'll use the same type fn returns).
As a side note, please don't ignore errors. Even network errors - at least log them somewhere :)