Develop Reference

Javascript Workers - why is the worker message treated so lately and can I do something against it?

I have a Worker that shares a SharedArrayBuffer with the "main thread". To work correctly, I have to make sure that the worker has access to the SAB before the main thread accesses to it. (EDIT: The code creating the worker has to be in a seperate function (EDIT2: which returns an array pointing to the SAB).) (Maybe, already this is not possible, you'll tell me).
The initial code looks like this:
function init() {
var code = `onmessage = function(event) {
console.log('starting');
var buffer=event.data;
var arr = new Uint32Array(buffer);// I need to have this done before accessing the buffer again from the main
//some other code, manipulating the array
}`
var buffer = new SharedArrayBuffer(BUFFER_ELEMENT_SIZE);
var blob = new Blob([code], { "type": 'application/javascript' });
var url = window.URL || window.webkitURL;
var blobUrl = url.createObjectURL(blob);
var counter = new Worker(blobUrl);
counter.postMessage(buffer);
let res = new Uint32Array(buffer);
return res;
}
function test (){
let array = init();
console.log('main');
//accessing the SAB again
};
The worker code is always executed after test(), the console shows always main, then starting.
Using timeouts does not help. Consider the following code for test:
function test (){
let array = [];
console.log('main');
setTimeout(function(){
array = initSAB();
},0);
setTimeout(function(){
console.log('main');
//accessing the SAB again
},0);
console.log('end');
};
The console shows end first, followed by main, followed by starting.
However, assigning the buffer to a global array outside the test() function does the job, even without timeouts.
My questions are the following:
why does the worker does not start directly after the message was send (= received?). AFAIK, workers have their own event queue, so they should not rely on the main stack becoming empty?
Is there a specification detailing when a worker starts working after sending a message?
Is there a way to make sure the worker has started before accessing the SAB again without using global variables? (One could use busy waiting, but I beware...) There is probably no way, but I want to be sure.
Edit
To be more precise:
In a completly parallel running scenario, the Worker would be able to
handle the message immediately after it was posted. This is obviously
not the case.
Most Browser API (and Worker is such an API) use a callback queue to handle calls to the API. But if this applied, the message would be
posted/handled before the timeout calbacks were executed.
To go even further: If I try busy waiting after postMessage by reading from the SAB until it changes one value will block the
program infinitely. For me, it means that the Browser does
not posts the message until the call stack is empty As far as
I know, this behaviour is not documentated and I cannot explain it.
To summerize: I want to know how the browser determines when to post the message and to handle it by the worker, if the call of postMessage is inside a function. I already found a workaround (global variables), so I'm more interested in how it works behind the scenes. But if someone can show me a working example, I'll take it.
EDIT 2:
The code using the global variable (the code that works fine) looks like this
function init() {
//Unchanged
}
var array = init(); //global
function test (){
console.log('main');
//accessing the SAB again
};
It prints starting, then main to the console.
What is also worth noticing : If I debug the code with the Firefox Browser (Chrome not tested) I get the result I want without the global variable (starting before main) Can someone explain?

why does the worker does not start directly after the message was sen[t] (= received?). AFAIK, workers have their own event queue, so they should not rely on the main stack becoming empty?
First, even though your Worker object is available in main thread synchronously, in the actual worker thread there are a lot of things to do before being able to handle your message:
it has to perform a network request to retrieve the script content. Even with a blobURI, it's an async operation.
it has to initialize the whole js context, so even if the network request was lightning fast, this would add up on parallel execution time.
it has to wait the event loop frame following the main script execution to handle your message. Even if the initialization was lightning fast, it will anyway wait some time.
So in normal circumstances, there is very little chances that your Worker could execute your code at the time you require the data.
Now you talked about blocking the main thread.
If I try busy waiting after postMessage by reading from the SAB until it changes one value will block the program infinitely
During the initialization of your Worker, the message are temporarily being kept on the main thread, in what is called the outside port. It's only after the fetching of the script is done that this outside port is entangled with the inside port, and that the messages actually pass to that parallel thread.
So if you do block the main thread before the ports have been entangled it won't be able to pass it to the worker's thread.
Is there a specification detailing when a worker starts working after sending a message?
Sure, and more specifically, the port message queue is enabled at the step 26, and the Event loop is actually started at the step 29.
Is there a way to make sure the worker has started before accessing the SAB again without using global variables? [...]
Sure, make your Worker post a message to the main thread when it did.
// some precautions because all browsers still haven't reenabled SharedArrayBuffers
const has_shared_array_buffer = window.SharedArrayBuffer;
function init() {
// since our worker will do only a single operation
// we can Promisify it
// if we were to use it for more than a single task,
// we could promisify each task by using a MessagePort
return new Promise((resolve, reject) => {
const code = `
onmessage = function(event) {
console.log('hi');
var buffer= event.data;
var arr = new Uint32Array(buffer);
arr.fill(255);
if(self.SharedArrayBuffer) {
postMessage("done");
}
else {
postMessage(buffer, [buffer]);
}
}`
let buffer = has_shared_array_buffer ? new SharedArrayBuffer(16) : new ArrayBuffer(16);
const blob = new Blob([code], { "type": 'application/javascript' });
const blobUrl = URL.createObjectURL(blob);
const counter = new Worker(blobUrl);
counter.onmessage = e => {
if(!has_shared_array_buffer) {
buffer = e.data;
}
const res = new Uint32Array(buffer);
resolve(res);
};
counter.onerror = reject;
if(has_shared_array_buffer) {
counter.postMessage(buffer);
}
else {
counter.postMessage(buffer, [buffer]);
}
});
};
async function test (){
let array = await init();
//accessing the SAB again
console.log(array);
};
test().catch(console.error);

According to MDN:
Data passed between the main page and workers is copied, not shared. Objects are serialized as they're handed to the worker, and subsequently, de-serialized on the other end. The page and worker do not share the same instance, so the end result is that a duplicate is created on each end. Most browsers implement this feature as structured cloning.
Read more about transferring data to and from workers
Here's a basic code that shares a buffer with a worker. It creates an array with even values (i*2) and it sends it to the worker. It uses Atomic operations to change the buffer values.
To make sure the worker has started you can just use different messages.
var code = document.querySelector('[type="javascript/worker"]').textContent;
var blob = new Blob([code], { "type": 'application/javascript' });
var blobUrl = URL.createObjectURL(blob);
var counter = new Worker(blobUrl);
var sab;
var initBuffer = function (msg) {
sab = new SharedArrayBuffer(16);
counter.postMessage({
init: true,
msg: msg,
buffer: sab
});
};
var editArray = function () {
var res = new Int32Array(sab);
for (let i = 0; i < 4; i++) {
Atomics.store(res, i, i*2);
}
console.log('Array edited', res);
};
initBuffer('Init buffer and start worker');
counter.onmessage = function(event) {
console.log(event.data.msg);
if (event.data.edit) {
editArray();
// share new buffer with worker
counter.postMessage({buffer: sab});
// end worker
counter.postMessage({end: true});
}
};
<script type="javascript/worker">
var sab;
self['onmessage'] = function(event) {
if (event.data.init) {
postMessage({msg: event.data.msg, edit: true});
}
if (event.data.buffer) {
sab = event.data.buffer;
var sharedArray = new Int32Array(sab);
postMessage({msg: 'Shared Array: '+sharedArray});
}
if (event.data.end) {
postMessage({msg: 'Time to rest'});
}
};
</script>

Zapier: Task timed out after 1.00 seconds

I am using the Zapier Code application in the javascript language, I am making a request in an api but in almost all attempts at the time of executing the script, I get the error message: "We had trouble sending your test through. Please try again. Error:
2018-03-09T14:32:54.748Z c0958e0a-23a6-11e8-9be1-a515bc24f853 Task timed out after 1.00 seconds". Sometimes script execution happens successfully, but most of the time it gives this error.
The calling code in the api I'm using is this:
var promises = [];
var retornoDaChamada;
promises.push(fetch(urls));
Promise.all(promises).then(function(res){
var blobPromises = [];
for (var i = res.length - 1; i >= 0; i--) {
blobPromises.push(res[i].text());
}
return Promise.all(blobPromises);
}).then(function(body){
retornoDaChamada=JSON.parse(body);
var titulosDaApi = [retornoDaChamada.length];
var duracao = [retornoDaChamada.length];
var ids = [retornoDaChamada.length];
for(var i=0; i<retornoDaChamada.length; i++){
titulosDaApi[i]=retornoDaChamada[i].title;
duracao[i]=milissegundosParaHorasMinutosSegundos(retornoDaChamada[i].files[0].fileInfo.duration);
ids[i]=retornoDaChamada[i].id;
}
var output = {titulosDaApi, duracao, ids};
callback(null, output);
}).catch(callback);
I read the documentation of the application Code and I kind of understood that free user only has the time of up to 1 second for calls in Api, is there any way I can get around this problem even though I am a free user?

David here, from the Zapier Platform team.
It's a little tough to understand the context of your code, but it looks like you're doing multiple HTTP requests. Due to their nature, they're a very slow operation. If you're doing more than 1 (maybe two if the external resource responds really quickly), you're unlikely to be able to fit everything into 1 second.
Sorry for the bad news!

Saving To MongoDB In A Loop

i am having trouble saving a new record to mongoDB. i am pretty sure there is something i am using in my code that i don't fully understand and i was hoping someone might be able to help.
i am trying to save a new record to mongoDB for each of the cats. this code is for node.js
for(var x = 0; x < (cats.length - 1); x++){
if (!blocked){
console.log("x = "+x);
var memberMessage = new Message();
memberMessage.message = message.message;
memberMessage.recipient = room[x].userId;
memberMessage.save(function(err){
if (err) console.log(err);
console.log(memberMessage + " saved for "+cats[x].name);
});
}
});
}
i log the value of "cats" before the loop and i do get all the names i expect so i would think that looping through the array it would store a new record for each loop.
what seems to happen is that when i look ta the the database, it seems to have only saved for the last record for every loop cycle. i don't know how/why it would be doing that.
any help on this is appreciated because I'm new to node.js and mongoDB.
thanks.

That's because the save is actually a I/O operation which is Async. Now, the for loop is actually sync.
Think of it this way: your JS engine serially executes each line it sees. Assume these lines are kept one-after-another on a stack. When it comes to the save, it keeps it aside on a different stack (as it is an I/O operation, and thus would take time) and goes ahead with the rest of the loop. It so turns out that the engine would only check this new stack after it has completed every line on the older one. Therefore, the value of the variable cats will be the last item in the array. Thus, only the last value is saved.
To fight this tragedy, you can use mutiple methods:
Closures - Read More
You can make closure like so: cats.forEach()
Promises - Read More. There is a sweet library which promisifies the mongo driver to make it easier to work with.
Generators, etc. - Read More. Not ready for primetime yet.
Note about #2 - I'm not a contributor of the project, but do work with the author. I've been using the library for well over an year now, and it's fast and awesome!

You can use a batch create feature from mongoose:
var messages = [];
for(var x = 0; x < (cats.length - 1); x++) {
if (!blocked) {
var message = new Message();
message.message = message.message;
message.recipient = room[x].userId;
messages.push(message);
}
}
Message.create(messages, function (err) {
if (err) // ...
});

How to structure my code to return a callback?

So I've been stuck on this for quite a while. I asked a similar question here: How exactly does done() work and how can I loop executions inside done()?
but I guess my problem has changed a bit.
So the thing is, I'm loading a lot of streams and it's taking a while to process them all. So to make up for that, I want to at least load the streams that have already been processed onto my webpage, and continue processing stream of tweets at the same time.
loadTweets: function(username) {
$.ajax({
url: '/api/1.0/tweetsForUsername.php?username=' + username
}).done(function (data) {
var json = jQuery.parseJSON(data);
var jsonTweets = json['tweets'];
$.Mustache.load('/mustaches.php', function() {
for (var i = 0; i < jsonTweets.length; i++) {
var tweet = jsonTweets[i];
var optional_id = '_user_tweets';
$('#all-user-tweets').mustache('tweets_tweet', { tweet: tweet, optional_id: optional_id });
configureTweetSentiment(tweet);
configureTweetView(tweet);
}
});
});
}};
}
This is pretty much the structure to my code right now. I guess the problem is the for loop, because nothing will display until the for loop is done. So I have two questions.
How can I get the stream of tweets to display on my website as they're processed?
How can I make sure the Mustache.load() is only executed once while doing this?

The problem is that the UI manipulation and JS operations all run in the same thread. So to solve this problem you should just use a setTimeout function so that the JS operations are queued at the end of all UI operations. You can also pass a parameter for the timeinterval (around 4 ms) so that browsers with a slower JS engine can also perform smoothly.
...
var i = 0;
var timer = setInterval(function() {
var tweet = jsonTweets[i++];
var optional_id = '_user_tweets';
$('#all-user-tweets').mustache('tweets_tweet', {
tweet: tweet,
optional_id: optional_id
});
configureTweetSentiment(tweet);
configureTweetView(tweet);
if(i === jsonTweets.length){
clearInterval(timer);
}
}, 4); //Interval between loading tweets
...
NOTE
The solution is based on the following assumptions -
You are manipulating the dom with the configureTweetSentiment and the configureTweetView methods.
Ideally the solution provided above would not be the best solution. Instead you should create all html elements first in javascript only and at the end append the final html string to a div. You would see a drastic change in performance (Seriously!)
You don't want to use web workers because they are not supported in old browsers. If that's not the case and you are not manipulating the dom with the configure methods then web workers are the way to go for data intensive operations.

How to stop intense Javascript loop from freezing the browser

I'm using Javascript to parse an XML file with about 3,500 elements. I'm using a jQuery "each" function, but I could use any form of loop.
The problem is that the browser freezes for a few seconds while the loop executes. What's the best way to stop freezing the browser without slowing the code down too much?
$(xmlDoc).find("Object").each(function() {
//Processing here
});

I would ditch the "each" function in favour of a for loop since it is faster. I would also add some waits using the "setTimeout" but only every so often and only if needed. You don't want to wait for 5ms each time because then processing 3500 records would take approx 17.5 seconds.
Below is an example using a for loop that processes 100 records (you can tweak that) at 5 ms intervals which gives a 175 ms overhead.
var xmlElements = $(xmlDoc).find('Object');
var length = xmlElements.length;
var index = 0;
var process = function() {
for (; index < length; index++) {
var toProcess = xmlElements[index];
// Perform xml processing
if (index + 1 < length && index % 100 == 0) {
setTimeout(process, 5);
}
}
};
process();
I would also benchmark the different parts of the xml processing to see if there is a bottleneck somewhere that may be fixed. You can benchmark in firefox using firebug's profiler and by writing out to the console like this:
// start benchmark
var t = new Date();
// some xml processing
console.log("Time to process: " + new Date() - t + "ms");
Hope this helps.

Set a timeOut between processing to prevent the loop cycle from eating up all the browser resources. In total it would only take a few seconds to process and loop through everything, not unreasonable for 3,500 elements.
var xmlElements = $(xmlDoc).find('Object');
var processing = function() {
var element = xmlElements.shift();
//process element;
if (xmlElements.length > 0) {
setTimeout(processing, 5);
}
}
processing();

I'd consider converting the 3500 elements from xml to JSON serverside or even better upload it to server converted, so that it's native to JS from the getgo.
This would minimize your load and prolly make the file size smaller too.

you can setTimeout() with duration of ZERO and it will yield as desired

Long loops without freezing the browser is possible with the Turboid framework. With it, you can write code like:
loop(function(){
// Do something...
}, number_of_iterations, number_of_milliseconds);
More details in this turboid.net article: Real loops in Javascript

Javascript is single-threaded, so aside from setTimeout, there's not much you can do. If using Google Gears is an option for your site, they provide the ability to run javascript in a true background thread.

You could use the HTML5 workers API, but that will only work on Firefox 3.1 and Safari 4 betas atm.

I had the same problem which was happening when user refreshed the page successively. The reason was two nested for loops which happened more than 52000 times. This problem was harsher in Firefox 24 than in Chrome 29 since Firefox would crash sooner (around 2000 ms sooner than Chrome). What I simply did and it worked was that I user "for" loops instead of each and then I refactored the code so that I divided the whole loop array to 4 separated calls and then merged the result into one. This solution has proven that it has worked.
Something like this:
var entittiesToLoop = ["..."]; // Mainly a big array
loopForSubset(0, firstInterval);
loopForSubset(firstInterval, secondInterval);
...
var loopForSubset = function (startIndex, endIndex) {
for (var i=startIndex; i < endIndex; i++) {
//Do your stuff as usual here
}
}
The other solution which also worked for me was the same solution implemented with Worker APIs from HTML5. Use the same concept in workers as they avoid your browser to be frozen because they run in the background of your main thread. If just applying this with Workers API did not work, place each of instances of loopForSubset in different workers and merge the result inside the main caller of Worker.
I mean this might not be perfect but this has worked. I can help with more real code chunks, if someone still thinks this might suite them.

You could try shortening the code by
$(xmlDoc).find("Object").each(function(arg1) {
(function(arg1_received) {
setTimeout(function(arg1_received_reached) {
//your stuff with the arg1_received_reached goes here
}(arg1_received), 0)
})(arg1)
}(this));
This won't harm you much ;)

As a modification of #tj111 answer the full usable code
//add pop and shift functions to jQuery library. put in somewhere in your code.
//pop function is now used here but you can use it in other parts of your code.
(function( $ ) {
$.fn.pop = function() {
var top = this.get(-1);
this.splice(this.length-1,1);
return top;
};
$.fn.shift = function() {
var bottom = this.get(0);
this.splice(0,1);
return bottom;
};
})( jQuery );
//the core of the code:
var $div = $('body').find('div');//.each();
var s= $div.length;
var mIndex = 0;
var process = function() {
var $div = $div.first();
//here your own code.
//progress bar:
mIndex++;
// e.g.: progressBar(mIndex/s*100.,$pb0);
//start new iteration.
$div.shift();
if($div.size()>0){
setTimeout(process, 5);
} else {
//when calculations are finished.
console.log('finished');
}
}
process();