First of all, I'm not experienced in asynchronous programming, so I'm sorry if I missed something obvious.
I see this question pop up a lot. I know that people don't like forcing synchronicity in Javascript, but it is necessary in this case. I'm making heavy calls to a production database that can't take too much load due to how frequently it is used. Because of this, I'm setting up my code in a loop to make a request, wait for confirmation that it is finished, sleep for 2 seconds, and then make the next request. This is because I'm going to be pulling a LOT of data from this server on a weekly basis over the course of around 10-20 minutes.
Here's the code that I have. Sleep is a function that forces the program to wait using the Date class.
var thread = function(cb){
cb();
};
do{
var x = thread(function(){
request.post(options, function(e, r, body){
console.log(e);
console.log(r.statusCode);
issues.push(body["issues"]);
maxResults = body["total"];
options.body.startAt += 25;
console.log("Going to sleep");
});
sleep(2000);
});
console.log("Waking up and moving to the next cycle");
}while(issues.length < maxResults);
console.log("Finished with the requests");
}
although I have a callback set up, my code is still running the requests asynchronously. Because I leave maxResults null, it is plainly obvious that my callback isn't working. This is my output:
Waking up and moving to the next cycle
Finished with the requests
Going to sleep
You need to make a recursive asynchronous function.
It would look something like this:
function fetch(existingIssues) {
return sendRequest().then(function() {
existingIssues.push(...);
if (existingIssues.length >= maxResults)
return existingIssues;
else
return fetch(existingIssues);
});
}
fetch([]).then(...);
Related
Inside the catch of a promise
.catch((message) => {
console.log(message)
var newUrl = url
chrome.tabs.create({url: newUrl}, function(response) {
console.log(response.status)
status = 'loading'
while (status == 'loading') {
setTimeout(function() {
console.log(response.status)
status = response.status
}, 3000)
}
})
})
I'm trying to write the catch in the way that it will open up a new page, wait for it to finish loading, then grab the new cookies
I feel like Im taking crazy pills as this seems super straight forward. However its never printing out response.status
I want it to wait to check response.status every 3 seconds and once the page has loaded it will end the loop.
What am I doing wrong?
The way You've wrote it you've made an infinite loop, which will put tons of setTimeouts on browser's event queue.
setTimeout also put's code there, but it puts it with "3sec plus" delay note.
In practice you tell your browser - set infinite timeouts for me, and after it's finished, please do console.log after 3 seconds. This won't happen.
You should probably use setInterval instead
I have a NodeJS server managing some files. It's going to watch for a known filename from an external process and, once received, read it and then delete it. However, sometimes it's attempted to be read/deleted before the file has "unlocked" from previous use so likely will fail occasionally. What I'd like to do is retry this file asap, either as soon as it's finished or continuously at a fast pace.
I'd rather avoid a long sleep where possible, because this needs to be handled ASAP and every second counts.
fs.watchFile(intput_json_file, {interval: 10}, function(current_stats, previous_stats) {
var json_data = "";
try {
var file_cont = fs.readFileSync(input_json_file); // < TODO: async this
json_data = JSON.parse(file_cont.toString());
fs.unlink(input_json_file);
} catch (error) {
console.log("The JSON in the could not be parsed. File will continue to be watched.");
console.log(error);
return;
}
// Else, this has loaded properly.
fs.unwatchFile(input_json_file);
// ... do other things with the file's data.
}
// set a timeout for the file watching, just in case
setTimeout(fs.unwatchFile, CLEANUP_TIMEOUT, input_json_file);
I expect "EBUSY: resource busy or locked" to turn up occasionally, but fs.watchFile isn't always called when the file is unlocked.
I thought of creating a function and then calling it with a delay of 1-10ms, where it could call itself if that fails too, but that feels like a fast route to a... cough stack overflow.
I'd also like to steer clear of synchronous methods so that this scales nicely, but being relatively new to NodeJS all the callbacks are starting to turn into a maze.
May be it will be over for this story, but you can create own fs with full control. In this case other programs will write data directly to your program. Just search by word fuse and fuse-binding
I'm working in a project where I need to make requests over an API. The requests return data about a support ticket, but the problem is that i have about 500 tickets to get data about and each one requires one request. To speed up the requests, i tried to build a async routine that generate many requests at the same time. But, since the API that i'm integrating with has a rate limiter of 10 requests per second, some of the routines get the answer "Limit Exceed". If I make the requests sequentially, it's take about 5 minutes.
That way, someone has a tip for me in that task? I tried some solutions like rate-limiter of NodeJS, but it just generate 10 requests simultaneously, and didn't give any kind of error treatment or retry if the request fail.
About the language, it not have restriction, the project is written in NodeJS but have some python code too and didn't have problem to integrate another language.
Something like this isn't too difficult to create yourself, and it'd give you the flexibility you need.
There are fancy ways like tracking the start and completion time of each and checking if you've sent 10 in the second.
The system probably also limits it to 10 active requests going (i.e., you can't spin up 100 requests, 10 each second, and let them all process).
If you assume this, I'd say launch 10 all at once, then let them complete, then launch the next batch. You could also launch 10, then start 1 additional each time one finishes. You could think of this like a "thread pool".
You can easily track this with a simple variable tracking how many calls are going. Then, just check how many calls are going once a second (to avoid the 1 second limit) and if you have available "threads", fire off that many more new requests.
It could look something like this:
const threadLimit = 10;
const rateLimit = 1000; // ms
let activeThreads = 0;
const calls = new Array(100).fill(1).map((_, index) => index); // create an array 0 through 99 just for an example
function run() {
if (calls.length == 0) {
console.log('complete');
return;
}
// threadLimit - activeThreads is how many new threads we can start
for (let i = 0; i < threadLimit - activeThreads && calls.length > 0; i++) {
activeThreads++; // add a thread
call(calls.shift())
.then(done);
}
setInterval(run, rateLimit);
}
function done(val) {
console.log(`Done ${val}`);
activeThreads--; // remove a thread
}
function call(val) {
console.log(`Starting ${val}`);
return new Promise(resolve => waitToFinish(resolve, val));
}
// random function to simulate a network call
function waitToFinish(resolve, val) {
const done = Math.random() < .1; // 10% chance to finish
done && resolve(val)
if (!done) setInterval(() => waitToFinish(resolve, val), 10);
return done;
}
run();
Basically, run() just starts up however many new threads it can, based on the limit and how many are done. Then, it just repeats the process every second, adding new ones as it can.
You might need to play with the threadLimit and rateLimit values, as most rate limiting systems don't actually let you go up right to the limit and don't release it as soon as it's done.
I'm writing Javascript which is counting up to a certain number for a project. The number could be around 100,000 and It will take roughly 10-15 seconds to complete processing.
I want the script to run as soon as the user calls the page and when the script completes it does a redirect.
Is it possible to pause for even 10ms to update the DOM while it is running to give feedback such as "Still working"?
I would like to avoid the use of jQuery and web-workers are not an option in this situation.
I realise this isn't a real world application!
EDIT: Added some of the code as a sample:
In the head
function myCounter (target) {
var t = target;
var count = 0;
while (t != count){
if (t == count) {
window.location.replace("http://example.com"); // redirect
}
count++;
}
}
In the body
<script>myCounter(100000);</script>
In most browsers JavaScript and the UI run in the same thread. You can give the thread back to the browser by using setTimeout (or setInterval).
var myNumber = 0;
updateNumber();
function updateNumber(){
// do heavy work for great good, ideally divided into smaller chunks
document.getElementById('updateDiv').innerHTML = 'still working, up to ' + myNumber;
if(myNumber < limit) {
setTimeout(updateNumber, 20);
}
}
For a lot more details on the general process, this answer is worth a read: https://stackoverflow.com/a/4575011/194940
Edit: We can close. Isn't truly asynchronous, non-blocking javascript impossible?
var PATH = require ("path");
var URL = require ("url");
var sleep = function (ms){
var start = new Date ().getTime ();
while ((new Date ().getTime () - start) < ms);
}
require ("http").createServer (function (req, res){
if (URL.parse (req.url).pathname === "/1"){
console.log ("tab 1: I'm in");
PATH.exists ("test", function (exists){
sleep (5000);
res.writeHead (200, {"Content-Type": "text/plain"});
res.end ("1");
console.log ("tab 1: I'm done");
});
}else{
console.log ("tab 2: I'm in");
res.writeHead (200, {"Content-Type": "text/plain"});
res.end ("2");
console.log ("tab 2: I'm done");
}
}).listen (80);
Copy the content into a file.
Execute the file.
Open a new tab in browser. Set the url to localhost/1. Don't go yet.
Open a new tab in browser. Set the url to localhost/2. Don't go yet.
Go back to the first tab. Press enter and immediately after change to the second tab and press enter.
Result:
console log:
tab 1: I'm in
tab 1: I'm done
tab 2: I'm in
tab 2: I'm done
Tab 1 waits 5 seconds to receive the result "1".
Tab 2 also has to wait 5 seconds because tab 1 is sleeping for 5 seconds.
The docs says that all is asynchronous except the code. Only one thread. Only one request at a time. Requests are enqueued.
I/O calls are supposed to be asynchronous, right? Then why tab 2 has to wait to tab 1 if the callback comes from an asynchronous I/O process?
Thanks.
Because your sleep is blocking the event loop.
Replace it with setTimemout(function() { /* Code to run */ }, 5000); and watch /2 respond immediately.
The actual I/O is asynchronous, but all actions you're performing on the I/O happen in the event loop. If something is blocking the event loop, everything else has to wait, just like you said.
EDIT. For more clarity, look at the following ASCII graphic:
Event Loop Thread: ------+req1HandlerExistsCall+-------+req1Wait5Sec++++++++++++++++++++++++++req2ExistsCall+-------+req2Immediate+-------------
HTTP I/O: -+req1+--------------------------------------+req2+--------------------------------------+req1Response+--------+req2Response+
File I/O: ----------------------------+exists1+----------------------------------------------------+exists2+---------------------------
Basically, only one at a time for each thread. Because the first request handler blocks for 5 seconds (and it's essentially impossible to beat your filesystem with your fingers in a speed test), the second response doesn't even start to be handled until the first request is almost done.
You don't have any I/O calls in your code
You are calling a busy sleep method in your code ie your code is taking 5 seconds to process. And while it is processing nothing else can run on that instance
Honestly, is this a real question or are you just trying to bash node?
Sleep is implemented in blocking manner and runs on the very engine that is being single threaded.
SetTimeout method is the equivalent version of waiting some time in JavaScript.
Also consider, that in JavaScript most thing should involve a resultHandler as continuation is handled with function typed parameters to other functions to be invoked when the job is done.