As far as I can tell, web workers need to be written in a separate JavaScript file, and called like this:
new Worker('longrunning.js')
I'm using the closure compiler to combine and minify all my JavaScript source code, and I'd rather not have to have my workers in separate files for distribution. Is there some way to do this?
new Worker(function() {
//Long-running work here
});
Given that first-class functions are so crucial to JavaScript, why does the standard way to do background work have to load a whole other JavaScript file from the web server?
http://www.html5rocks.com/en/tutorials/workers/basics/#toc-inlineworkers
What if you want to create your worker script on the fly, or create a self-contained page without having to create separate worker files? With Blob(), you can "inline" your worker in the same HTML file as your main logic by creating a URL handle to the worker code as a string
Full example of BLOB inline worker:
<!DOCTYPE html>
<script id="worker1" type="javascript/worker">
// This script won't be parsed by JS engines because its type is javascript/worker.
self.onmessage = function(e) {
self.postMessage('msg from worker');
};
// Rest of your worker code goes here.
</script>
<script>
var blob = new Blob([
document.querySelector('#worker1').textContent
], { type: "text/javascript" })
// Note: window.webkitURL.createObjectURL() in Chrome 10+.
var worker = new Worker(window.URL.createObjectURL(blob));
worker.onmessage = function(e) {
console.log("Received: " + e.data);
}
worker.postMessage("hello"); // Start the worker.
</script>
The html5rocks solution of embedding the web worker code in HTML is fairly horrible.
And a blob of escaped JavaScript-as-a-string is no better, not least because it complicates work-flow (Closure compiler can't operate on strings).
Personally I really like the toString methods, but #dan-man THAT regex!
My preferred approach:
// Build a worker from an anonymous function body
var blobURL = URL.createObjectURL( new Blob([ '(',
function(){
//Long-running work here
}.toString(),
')()' ], { type: 'application/javascript' } ) ),
worker = new Worker( blobURL );
// Won't be needing this anymore
URL.revokeObjectURL( blobURL );
Support is the intersection of these three tables:
http://caniuse.com/#feat=webworkers
http://caniuse.com/#feat=blobbuilder
http://caniuse.com/#feat=bloburls
This won't work for a SharedWorker however, because the URL must be an exact match, even if the optional 'name' parameter matches. For a SharedWorker, you'll need a separate JavaScript file.
2015 update - The ServiceWorker singularity arrives
Now there's an even more powerful way of solving this problem.
Again, store the worker code as a function, (rather than a static string) and convert using .toString(), then insert the code into CacheStorage under a static URL of your choice.
// Post code from window to ServiceWorker...
navigator.serviceWorker.controller.postMessage(
[ '/my_workers/worker1.js', '(' + workerFunction1.toString() + ')()' ]
);
// Insert via ServiceWorker.onmessage. Or directly once window.caches is exposed
caches.open( 'myCache' ).then( function( cache )
{
cache.put( '/my_workers/worker1.js',
new Response( workerScript, { headers: {'content-type':'application/javascript'}})
);
});
There are two possible fall-backs. ObjectURL as above, or more seamlessly, put a real JavaScript file at /my_workers/worker1.js
Advantages of this approach are:
SharedWorkers can also be supported.
Tabs can share a single cached copy at a fixed address. The blob approach proliferates random objectURLs for every tab.
You can create a single JavaScript file that is aware of its execution context and can act as both a parent script and a worker. Let's start off with a basic structure for a file like this:
(function(global) {
var is_worker = !this.document;
var script_path = is_worker ? null : (function() {
// append random number and time to ID
var id = (Math.random()+''+(+new Date)).substring(2);
document.write('<script id="wts' + id + '"></script>');
return document.getElementById('wts' + id).
previousSibling.src;
})();
function msg_parent(e) {
// event handler for parent -> worker messages
}
function msg_worker(e) {
// event handler for worker -> parent messages
}
function new_worker() {
var w = new Worker(script_path);
w.addEventListener('message', msg_worker, false);
return w;
}
if (is_worker)
global.addEventListener('message', msg_parent, false);
// put the rest of your library here
// to spawn a worker, use new_worker()
})(this);
As you can see, the script contains all code for both the parent's and the worker's point of view, checking if its own individual instance is a worker with !document. The somewhat unwieldy script_path computation is used to accurately calculate the script's path relative to the parent page, as the path supplied to new Worker is relative to the parent page, not the script.
Using the Blob method, how about this for a worker factory:
var BuildWorker = function(foo){
var str = foo.toString()
.match(/^\s*function\s*\(\s*\)\s*\{(([\s\S](?!\}$))*[\s\S])/)[1];
return new Worker(window.URL.createObjectURL(
new Blob([str],{type:'text/javascript'})));
}
So you could use it like this...
var myWorker = BuildWorker(function(){
//first line of worker
self.onmessage(){....};
//last line of worker
});
EDIT:
I've just extended this idea further to make it easier to do cross-thread communication: bridged-worker.js.
EDIT 2:
The above link is to a gist I created. Someone else later turned it into an actual repo.
Web workers operate in entirely separate contexts as individual Program's.
This means that code cannot be moved from one context to another in object form, as they would then be able to reference objects via closures belonging to the other context.
This is especially crucial as ECMAScript is designed to be a single threaded language, and since web workers operate in separate threads, you would then have the risk of non-thread-safe operations being performed.
This again means that web workers need to be initialized with code in source form.
The spec from WHATWG says
If the origin of the resulting
absolute URL is not the same as the
origin of the entry script, then throw
a SECURITY_ERR exception.
Thus, scripts must be external files
with the same scheme as the original
page: you can't load a script from a
data: URL or javascript: URL, and an
https: page couldn't start workers
using scripts with http: URLs.
but unfortunately it doesn't really explain why one couldn't have allowed passing a string with source code to the constructor.
Recent answer (2018)
You can use Greenlet:
Move an async function into its own thread. A simplified single-function version of Workerize.
Example:
import greenlet from 'greenlet'
const getName = greenlet(async username => {
const url = `https://api.github.com/users/${username}`
const res = await fetch(url)
const profile = await res.json()
return profile.name
})
console.log(await getName('developit'))
a better to read way for a inline worker..
var worker_fn = function(e)
{
self.postMessage('msg from worker');
};
var blob = new Blob(["onmessage ="+worker_fn.toString()], { type: "text/javascript" });
var worker = new Worker(window.URL.createObjectURL(blob));
worker.onmessage = function(e)
{
alert(e.data);
};
worker.postMessage("start");
Taking Adria's response and putting it in a copy-pastable function which works with current Chrome and FF but not IE10 (worker from blob causes a security error).
var newWorker = function (funcObj) {
// Build a worker from an anonymous function body
var blobURL = URL.createObjectURL(new Blob(
['(', funcObj.toString(), ')()'],
{type: 'application/javascript'}
));
var worker = new Worker(blobURL);
// Won't be needing this anymore
URL.revokeObjectURL(blobURL);
return worker;
}
And here's a working example http://jsfiddle.net/ubershmekel/YYzvr/
A simple promisified version, Function#callAsWorker, that takes a thisArg and arguments (just like call), and returns a promise:
Function.prototype.callAsWorker = function (...args) {
return new Promise( (resolve, reject) => {
const code = `self.onmessage = e => self.postMessage((${this.toString()}).call(...e.data));`,
blob = new Blob([code], { type: "text/javascript" }),
worker = new Worker(window.URL.createObjectURL(blob));
worker.onmessage = e => (resolve(e.data), worker.terminate());
worker.onerror = e => (reject(e.message), worker.terminate());
worker.postMessage(args);
});
}
// Demo
function add(...nums) {
return nums.reduce( (a,b) => a+b );
}
// Let the worker execute the above function, with the specified arguments
add.callAsWorker(null, 1, 2, 3).then(function (result) {
console.log('result: ', result);
});
So I think we have another cool option for this now, thanks to template literals in ES6. That allows us to dispense with the extra worker function (and its weird scope) and just write the code that's intended for the worker as multiline text, much like the case where we were using to store text, but without actually needing a document or DOM to do that in. Example:
const workerScript = `
self.addEventListener('message', function(e) {
var data = e.data;
console.log('worker recieved: ',data);
self.postMessage('worker added! :'+ addOne(data.value));
self.close();//kills the worker
}, false);
`;
Here's a gist of the rest of that approach.
Note that we can pull in any extra function dependencies we want into the worker just by collecting them into an array and running .toString on each of them to reduce them down into strings as well (should work as long as they are function declarations) and then just prepending that to the script string. That way we don't have to importScripts that we might already have bundled into the scope of the code we're writing.
The only real downside to this particular version is that linters won't be able to lint the service worker code (since it's just a string), which is an advantage for the "separate worker function approach."
Depending on your use case you can use something like
task.js Simplified interface for getting CPU intensive code to run on all cores (node.js, and web)
A example would be
function blocking (exampleArgument) {
// block thread
}
// turn blocking pure function into a worker task
const blockingAsync = task.wrap(blocking);
// run task on a autoscaling worker pool
blockingAsync('exampleArgumentValue').then(result => {
// do something with result
});
Take a look at the vkThread plugin. With htis plugin you can take any function in your main code and execute it in a thread (web worker). So, you don't need to create a special "web-worker file".
http://www.eslinstructor.net/vkthread/
--Vadim
I think the better way to do this is using a Blob object, below you can see a simple example.
// create a Blob object with a worker code
var blob = new Blob(["onmessage = function(e) { postMessage('msg from worker'); }"]);
// Obtain a blob URL reference to our worker 'file'.
var blobURL = window.URL.createObjectURL(blob);
// create a Worker
var worker = new Worker(blobURL);
worker.onmessage = function(e) {
console.log(e.data);
};
worker.postMessage("Send some Data");
Try to use jThread. https://github.com/cheprasov/jThread
// You can use simple calling like this
jThread(
function(arr){
//... some code for Worker
return arr;
}
,function(arr){
//... done code
}
)( [1,2,3,4,5,6,7] ); // some params
here console:
var worker=new Worker(window.URL.createObjectURL(new Blob([function(){
//Long-running work here
postMessage('done');
}.toString().split('\n').slice(1,-1).join('\n')],{type:'text/javascript'})));
worker.addEventListener('message',function(event){
console.log(event.data);
});
https://developer.mozilla.org/es/docs/Web/Guide/Performance/Using_web_workers
// Syntax: asyncEval(code[, listener])
var asyncEval = (function () {
var aListeners = [], oParser = new Worker("data:text/javascript;charset=US-ASCII,onmessage%20%3D%20function%20%28oEvent%29%20%7B%0A%09postMessage%28%7B%0A%09%09%22id%22%3A%20oEvent.data.id%2C%0A%09%09%22evaluated%22%3A%20eval%28oEvent.data.code%29%0A%09%7D%29%3B%0A%7D");
oParser.onmessage = function (oEvent) {
if (aListeners[oEvent.data.id]) { aListeners[oEvent.data.id](oEvent.data.evaluated); }
delete aListeners[oEvent.data.id];
};
return function (sCode, fListener) {
aListeners.push(fListener || null);
oParser.postMessage({
"id": aListeners.length - 1,
"code": sCode
});
};
})();
Use my tiny plugin https://github.com/zevero/worker-create
var worker_url = Worker.createURL(function(e){
self.postMessage('Example post from Worker'); //your code here
});
var worker = new Worker(worker_url);
This is just an addition to above - I have a nice templates for testing web workers in jsFiddle. Rather than Blob it uses jsFiddles ?js api:
function workerFN() {
self.onmessage = function(e) {
switch(e.data.name) {
case "" :
break;
default:
console.error("Unknown message:", e.data.name);
}
}
}
// This is a trick to generate real worker script that is loaded from server
var url = "/echo/js/?js="+encodeURIComponent("("+workerFN.toString()+")()");
var worker = new Worker(url);
worker.addEventListener("message", function(e) {
switch(e.data.name) {
case "" :
break;
default:
console.error("Unknown message:", e.data.name);
}
})
Normal web worker and shared worker templates are available.
I discovered that CodePen currently does not syntax-highlight inline <script> tags that are not type="text/javascript" (or which have no type attribute).
So I devised a similar but slightly different solution using labeled blocks with break, which is the only way you can bail from a <script> tag without creating a wrapper function (which is unnecessary).
<!DOCTYPE html>
<script id="worker1">
worker: { // Labeled block wrapper
if (typeof window === 'object') break worker; // Bail if we're not a Worker
self.onmessage = function(e) {
self.postMessage('msg from worker');
};
// Rest of your worker code goes here.
}
</script>
<script>
var blob = new Blob([
document.querySelector('#worker1').textContent
], { type: "text/javascript" })
// Note: window.webkitURL.createObjectURL() in Chrome 10+.
var worker = new Worker(window.URL.createObjectURL(blob));
worker.onmessage = function(e) {
console.log("Received: " + e.data);
}
worker.postMessage("hello"); // Start the worker.
</script>
One-liner for running functions in workers:
const FunctionalWorker = fn => new Worker(window.URL.createObjectURL(new Blob(["(" + fn.toString() + ")()"], {type: "text/javascript"})));
Example usage:
let fn = FunctionalWorker(() => {
self.postMessage("hi");
});
fn.onmessage = msg => {
console.log(msg);
};
my take on it:
function BuildWorker(fn){
var str = fn.toString().match(/^[^{]+{([\s\S]+)}\s*$/m)[1];
return new Worker(window.URL.createObjectURL(
new Blob([str],{type:'text/javascript'})));
}
function createAsyncWorker(fn){
// asyncworker=createAsyncWorker(function(){
// importScripts('my_otherscript.js');
// self.onmessage = function([arg1,arg2]) {
// self.postMessage('msg from worker');
// };
// })
// await asyncworker.postMessage('arg1','value')
// await asyncworker.postMessage('arg1','value')
// asyncworker.worker.terminate()
var worker = BuildWorker(fn);
function postMessage(...message){
let external={}, promise= new Promise((resolve,reject)=>{external.resolve=resolve;external.reject=reject;})
worker.onmessage = function(message){ external.resolve(message.data)};
worker.postMessage(message); // Start the worker.
return promise;
}
return {worker,postMessage};
}
usage example:
autoarima = createAsyncWorker(function(){
importScripts("https://127.0.0.1:11000/arima.js")
self.onmessage=(message)=>{
let [action,arg1,arg2]=message.data
if(action=='load')
{
ARIMAPromise.then(ARIMA1 => {
ARIMA=ARIMA1
autoarima = new ARIMA({ auto: true });
// const ts = Array(10).fill(0).map((_, i) => i + Math.random() / 5)
// const arima = new ARIMA({ p: 2, d: 1, q: 2, P: 0, D: 0, Q: 0, S: 0, verbose: false }).train(ts)
// const [pred, errors] = arima.predict(10)
postMessage('ok')
});
}
if(action=='fit')
{
autoarima.fit(arg1)
postMessage('ok')
}
if(action=='predict')
{
postMessage(autoarima.predict(arg1,arg2))
}
};
})
autoarima.terminate=function(){ this.worker.terminate(); }
autoarima.load=async function(...args){return await this.postMessage('load',...args)}
autoarima.fit=async function(...args){return await this.postMessage('fit',...args)}
autoarima.predict=async function(...args){return await this.postMessage('predict',...args)}
await autoarima.load()
await autoarima.fit(b_values)
await autoarima.predict(1)
You can use web workers in same javascript file using inline webworkers.
The below article will address you to easily understand the webworkers and their limitations and debugging of webworkers.
Mastering in webworkers
I use code like this, you can define your onmessage as a function other than plain text, so the editor can highlight your code and jshint works.
const worker = createWorker();
createWorker() {
const scriptContent = getWorkerScript();
const blob = new Blob([
scriptContent,
], {
type: "text/javascipt"
});
const worker = new Worker(window.URL.createObjectURL(blob));
return worker;
}
getWorkerScript() {
const script = {
onmessage: function (e) {
console.log(e);
let result = "Hello " + e.data
postMessage(result);
}
};
let content = "";
for (let prop in script){
content += `${prop}=${script[prop].toString()}`;
}
return content;
}
Yes, it is possible, I did it using Blob files and passing a callback
I'll show you what a class I wrote does and how it manages the execution of callbacks in the background.
First you instantiate the GenericWebWorker with whatever data you'd like to pass to callback that'll be executing in the Web Worker, that includes functions you want to use, in this case a number, a date and a function called blocker
var worker = new GenericWebWorker(100, new Date(), blocker)
This blocker function will execute an infinite while for n miliseconds
function blocker (ms) {
var now = new Date().getTime();
while(true) {
if (new Date().getTime() > now +ms)
return;
}
}
and then you use it like this
worker.exec((num, date, fnBlocker) => {
/*Everithing here does not block the main thread
and this callback has access to the number, date and the blocker */
fnBlocker(10000) //All of this run in backgrownd
return num*10
}).then(d => console.log(d)) //Print 1000
Now, time to see the magic in the example below
/*https://github.com/fercarvo/GenericWebWorker*/
class GenericWebWorker {
constructor(...ags) {
this.args = ags.map(a => (typeof a == 'function') ? {type:'fn', fn:a.toString()} : a)
}
async exec(cb) {
var wk_string = this.worker.toString();
wk_string = wk_string.substring(wk_string.indexOf('{') + 1, wk_string.lastIndexOf('}'));
var wk_link = window.URL.createObjectURL( new Blob([ wk_string ]) );
var wk = new Worker(wk_link);
wk.postMessage({ callback: cb.toString(), args: this.args });
var resultado = await new Promise((next, error) => {
wk.onmessage = e => (e.data && e.data.error) ? error(e.data.error) : next(e.data);
wk.onerror = e => error(e.message);
})
wk.terminate(); window.URL.revokeObjectURL(wk_link);
return resultado
}
async parallel(arr, cb) {
var res = [...arr].map(it => new GenericWebWorker(it, ...this.args).exec(cb))
var all = await Promise.all(res)
return all
}
worker() {
onmessage = async function (e) {
try {
var cb = new Function(`return ${e.data.callback}`)();
var args = e.data.args.map(p => (p.type == 'fn') ? new Function(`return ${p.fn}`)() : p);
try {
var result = await cb.apply(this, args); //If it is a promise or async function
return postMessage(result)
} catch (e) { throw new Error(`CallbackError: ${e}`) }
} catch (e) { postMessage({error: e.message}) }
}
}
}
function blocker (ms) {
var now = new Date().getTime();
while(true) {
if (new Date().getTime() > now +ms)
return;
}
}
setInterval(()=> console.log("Not blocked " + Math.random()), 1000)
console.log("\n\nstarting blocking code in Worker\n\n")
var worker = new GenericWebWorker(100, new Date(), blocker)
worker.exec((num, date, fnBlocker) => {
fnBlocker(7000) //All of this run in backgrownd
return num*10
})
.then(d => console.log(`\n\nEnd of blocking code: result ${d}\n\n`)) //Print 1000
You can place the contents of your worker.js file inside backticks (which allows a multiline string constant) and create the worker from a blob like this:
var workerScript = `
self.onmessage = function(e) {
self.postMessage('message from worker');
};
// rest of worker code goes here
`;
var worker =
new Worker(createObjectURL(new Blob([workerScript], { type: "text/javascript" })));
This is handy if for whatever reason you don't want to have separate script tags for the worker.
Another solution is just to wrap the Worker in a function, then creating a blob invoking the function like so:
function workerCode() {
self.onmessage = function (e) {
console.log("Got message from parent", e.data);
};
setTimeout(() => {
self.postMessage("Message From Worker");
}, 2000);
}
let blob = new Blob([
"(" + workerCode.toString() + ")()"
], {type: "text/javascript"});
// Note: window.webkitURL.createObjectURL() in Chrome 10+.
let worker = new Worker(window.URL.createObjectURL(blob));
worker.onmessage = function (e) {
console.log("Received: " + e.data);
};
worker.postMessage("hello"); // Start the worker.
there have been a few answers, but here is another inline version.
note: "self" argument is purely cosmetic for linting purposes, actual worker code starts after first brace, self is as normal
inlineWorker (
"hello world",// initial message to send to worker
function(self){
// inline worker code.
self.onmessage = function (e) {
self.postMessage("thinking...");
for (var i=0;i<100000000;i++) {
var r = Math.random();
}
self.postMessage(e.data.toUpperCase());
}
},function(e){
// optional message handler
document.getElementById("log").innerHTML= "from worker:"+e.data;
});
function inlineWorker (msg,fn,onMsg) {
var
w=window,
U=!!w.webkitURL?w.webkitURL:w.URL,
src=fn.toString(),
s=src.indexOf('{'),
e=src.lastIndexOf('}'),
worker = new Worker(U.createObjectURL(
new Blob([ src.substring(s+1,e-1) ], { type: "text/javascript" })
));
if (typeof onMsg==="function") {
worker.addEventListener("message",onMsg);
}
if (msg) {
worker.postMessage(msg);
}
return worker;
}
<div id="log"></div>
For a Node.js implementation, the following adaptation of Trincot's answer can be employed. Note again that Function.prototype.callAsWorker() takes a thisArg and arguments, just like Function.prototype.call(), and returns a promise.
const { Worker } = require ( 'worker_threads' );
Function.prototype.callAsWorker = function ( ...args ) {
return new Promise( ( resolve, reject ) => {
const code = `
const { parentPort, workerData } = require ( 'worker_threads' );
parentPort.postMessage( ( ${this.toString()} ).call( ...workerData ) )
`;
const worker = new Worker( code, { eval: true, workerData: args } );
worker.on('message', ( msg ) => { resolve( msg ), worker.terminate() } );
worker.on('error', ( err ) => { reject( err ), worker.terminate() } );
worker.on('exit', ( code ) => {
if ( code !== 0 ) {
reject( new Error( `Worker stopped with exit code ${code}.` ) );
}
});
});
}
// Demo
function add( ...nums ) {
return nums.reduce( ( a, b ) => a + b );
}
// Let the worker execute the above function, with the specified arguments
let result = await add.callAsWorker( null, 1, 2, 3 );
console.log( 'result: ', result );
I liked the answer that ifbamoq gave but was not able to comment because of stack overflow's points policy. Therefor I'll give an example that shows some intensive work being done - and how it does not lock the main thread.
All without running into CORS problems with the null origin - if you're like me and like double-clicking the html files and treating them like little programs. :-)
<!DOCTYPE html>
<html>
<head>
<title>Worker example: One-core computation</title>
</head>
<body>
<p>The highest prime number discovered so far is: <div id="result"></div></p>
</body>
<script>
// let worker = new Worker('WebWorker.js'); // lets skip this to avoid null origin issues
let WorkerFn = (event) =>
{
let isPrime = false;
for (let n = 2; n <= 1_000_000; n++)
{
isPrime = true;
for(let i = 2; i <= Math.sqrt(n); i++)
if (n % i == 0)
isPrime = false; // If you can get thru all this shit and survive, ur prime!
if (isPrime)
postMessage(n);
}
}
let worker = new Worker(window.URL.createObjectURL(new Blob(["(" + WorkerFn.toString() + ")()"], {type: "text/javascript"})));
worker.onmessage = (event) =>
{
result.innerHTML = event.data;
}
</script>
</html>
#Trincot's seems to be the best so far. Yet perhaps we can develop it a little further. So the idea is,
Let's not modify the Function.prototype.
Obtain a promisified / threadable version of a function for threaded operation.
Make sure that the function can still be invoked synchronously if need be.
So we define a Threadable class with a spawn method. Once we make our function a member of this class then it is threadable :)
class Threadable extends Function {
constructor(f){
super("...as",`return ${f.toString()}.apply(this,as)`);
}
spawn(...as){
var code = `self.onmessage = m => self.postMessage((${this.toString()}).apply(self,m.data));`,
blob = new Blob([code], {type: "text/javascript"}),
wrkr = new Worker(window.URL.createObjectURL(blob));
return new Promise( (v,x) => ( wrkr.onmessage = m => (v(m.data), wrkr.terminate())
, wrkr.onerror = e => (x(e.message), wrkr.terminate())
, wrkr.postMessage(as)
)
);
}
}
function add(...nums) {
return nums.reduce((a,b) => a+b);
}
var addT = new Threadable(add);
addT.spawn(1,2,3,4)
.then(m => console.log(`Promisified thread returned ${m}`));
console.log(`Synchronous invocation of addT returned ${addT(1,2,3,4)}`);
Related
In a Chrome Extension, I have no problem adding, updating, and removing data to/from an IndexedDB database accessed by my service worker with Chrome Runtime Messaging sent from my content script. My trouble is doing a full table read from my content script. I do a console.log() to dump out the property before I send it back in my sendResponse in the Chrome Runtime Messaging, and I see the data there properly, but the content script receives an undefined. I assume this is because of the asynchronous nature of getting the data. I tried promises and async/await and the combination thereof and I just can't seem to get anything except an undefined in my content script on the message back from the service worker. I also ran sending a test array back and that worked just fine -- but receiving the IndexedDB table data does not work in the message passing. I also tried to JSONify the data and that didn't help either. What's the catch?
service-worker.js
importScripts('modules/idb.js');
var SW = {};
SW.onReady = function(){
chrome.runtime.onMessage.addListener(function(o, sender, sendResponse) {
(o.readTable) && sendResponse(SW.readTable(o,sender));
});
};
SW.readTable = function(o,sender){
var sTable = o.table;
new Promise((resolve) => {
IDB.readTable(sTable,function(asEntries){
resolve(asEntries);
});
}).then((asEntries) => {
console.log('SW asEntries',asEntries); // this shows me valid data in tests
var o = {};
// can also change this to fake data with asEntries being a string array and bug goes away in content.js
o.response = asEntries;
return o;
});
};
SW.onReady();
modules/idb.js
var IDB = {};
// Requires storage (or, even better, unlimitedStorage) permission in your manifest.json file.
// Note also that dev console of service worker will not show data -- have to use toolbar button popup panel (if you have one) and
// dev console from there, or code to access it, which sucks.
IDB.connectStore = function(sTable,sReadWriteSetting,fn){
var conn = indexedDB.open('unlimitedStorage', 1);
conn.onupgradeneeded = function(e) {
var db = e.target.result;
db.createObjectStore(sTable);
};
conn.onsuccess = function(e) {
var db = e.target.result;
var tx = db.transaction(sTable,sReadWriteSetting);
var store = tx.objectStore(sTable);
fn(db,tx,store);
};
};
IDB.addToTable = function(sTable,sKey,sVal){
IDB.connectStore(sTable,'readwrite',function(db,tx,store){
if ((sKey === undefined) || (sKey === '') || (sKey === null) || (sKey === false)) { // auto key by increment
var req = store.count();
req.onsuccess = function(e){
sKey = e.target.result + 1;
store.add(sVal,sKey);
tx.complete;
}
} else {
store.add(sVal,sKey);
tx.complete;
}
});
};
IDB.removeFromTable = function(sTable,sKey){
IDB.connectStore(sTable,'readwrite',function(db,tx,store){
store.delete(sKey);
tx.complete;
});
};
IDB.readTableByKey = function(sTable,sKey,fn){
IDB.connectStore(sTable,'readonly',function(db,tx,store){
var req = store.get(sKey);
req.onerror = function(e){
fn(e.target.result);
}
req.onsuccess = function(e){
fn(e.target.result);
}
});
};
IDB.readTable = function(sTable,fn){
IDB.connectStore(sTable,'readonly',function(db,tx,store){
var req = store.getAll();
req.onerror = function(e){
fn(e.target.result);
}
req.onsuccess = function(e){
fn(e.target.result);
}
});
};
content.js
var CONTENT = {};
CONTENT.onReady = function(){
var o = {};
o.readTable = true;
o.table = 'loadTimes';
chrome.runtime.sendMessage(o,function(response){
if (response.response) { // errors here with response property being undefined
console.log('CONTENT RCVD asEntries',response.response);
}
});
};
CONTENT.onReady();
Chrome extensions API, unlike Firefox WebExtensions, can't handle Promise returned from a callback or provided in sendResponse, https://crbug.com/1185241.
There's also a bug in your readTable: you need to add return before new Promise((resolve)
The solution is two-fold:
Use return true from the callback to allow asynchronous sendResponse
Call sendReponse inside .then of a Promise chain.
chrome.runtime.onMessage.addListener(function(o, sender, sendResponse) {
if (o.readTable) {
SW.readTable(o,sender).then(sendResponse);
return true;
} else {
sendResponse(); // Chrome 99-101 bug workaround, https://crbug.com/1304272
}
});
Do not use this answer. It is here for posterity reasons and is just a workaround. The chosen solution works.
The fix is to return data in a different message thread:
In the service worker in SW.readTable(), just return variable o with o.response = true and then ignore the response in the content script.
Before returning the variable o from SW.readTable(), do a chrome.runtime.sendMessage({readTableResult = true, data: asEntries},function(response){ /* ignore response */});
In the content script, ignore any response back from the readTable message. So, the if (response.response) {...} condition can be eliminated.
In the content script, add a message listener with chrome.runtime.onMessage.addListener(o, sender, sendResponse) and look for the condition (o.readTableResult). Once received, the o.data will now contain the asEntries data.
I want to bombard the receiver with say 1M messages, but also that the receiver will get each message as soon as possible. The naive way is to loop 1M times and just send a message to the receiver with postmessage. Doesn't work.
What i get is that the whole 1M messages are queued, and only when the code finishes, the receiver starts processing them.
What i need to happen is that the sender will send 1M messages and as he keeps on sending the messages the receiver simultaneously will process them.
For example, what i have now is something like this:
sender : send m1.
sender : send m2.
sender : send m3.
receiver : received m1.
receiver : received m2.
receiver : received m3.
What i want:
sender : send m1.
receiver : received m1.
sender : send m2.
receiver : received m2.
sender : send m3.
receiver : received m3.
How can i achieve this? I can not make the receiver send acks. My goal is to send as many massages as i can the fastest.
Edit: The code i have now:
Sender:
function sendx(x){
console.log("start spam");
for(let i=0; i<200000; i++){
window.opener.postMessage(x, '*');
}
console.log("done");
}
Receiver:
window.addEventListener("message", r_function );
function r_function(event)
{
let index = event.data;
let junk = something(index);
return junk;
}
Where the sender is a new window created by the receiver. What i get in practice is that only when the 'sendx' function ends, the receiver start receiving messages.
What i need to happen is that the sender will send 1M messages and as he keeps on sending the messages the receiver simultaneously will process them.
That's what happens already.
const worker = new Worker(URL.createObjectURL(
new Blob([worker_script.textContent])
));
let logged_first = false;
worker.onmessage = e => {
if(e.data === "spam") {
if(!logged_first) {
console.log('received first message at', new Date().toLocaleString());
logged_first = true; // ignore next messages
}
}
else {
console.log(e.data);
}
}
<script type="text/worker-script" id="worker_script">
const now = performance.now();
postMessage("start spamming at " + new Date().toLocaleString());
while(performance.now() - now < 5000) {
postMessage('spam');
}
postMessage("end spamming at " + new Date().toLocaleString());
</script>
However, for it to work, there is one big condition that needs to be met:
Your two JavaScript instances (sender & receiver) must run on different threads.
That is, if you were doing it using a MessageChannel on the same thread, then it would obviously be unable to treat the messages at the same time it's sending it:
const channel = new MessageChannel();
channel.port1.onmessage = e => {
console.log('received first message at', new Date().toLocaleString());
channel.port1.onmessage = null; // ignore next messages
};
const now = performance.now();
console.log("start spamming at ", new Date().toLocaleString());
while(performance.now() - now < 5000) {
channel.port2.postMessage('spam');
}
console.log("end spamming at ", new Date().toLocaleString());
And if you are dealing with an iframe or an other window, you can not be sure that you'll meet this condition. Browsers all behave differently, here, but they will all run at least some windows on the same process. You have no control as to which process will be used, and hence can't guarantee that you'll run in an other one.
So the best you can do, is to run your loop in a timed-loop, which will let the browser some idle time where it will be able to process other windows event loops correctly.
And the fastest timed-loop we have is actually the one postMessage offers us.
So to do what you wish, the best would be to run each iteration of your loop in the message event of a MessageChannel object.
For this, generator function* introduced in ES6 are quite useful:
/***************************/
/* Make Fake Window part */
/* ONLY for DEMO */
/***************************/
const fake_win = new MessageChannel();
const win = fake_win.port1; // window.open('your_url', '')
const opener = fake_win.port2; // used in Receiver
/********************/
/* Main window part */
/********************/
const messages = [];
win.onmessage = e => {
messages.push(e.data);
};
!function log_msg() {
document.getElementById('log').textContent = messages.length;
requestAnimationFrame(log_msg);
}();
/*******************/
/* Receiver part */
/*******************/
// make our loop a Generator function
function* ourLoopGen(i) {
while(i++ < 1e6) {
opener.postMessage(i);
yield i;
}
}
const ourLoop = ourLoopGen(0);
// here we init our time-loop
const looper = new MessageChannel();
looper.port2.onmessage = e => {
const result = ourLoop.next();
if(!result.done)
looper.port1.postMessage(''); // wait next frame
};
// start our time-loop
looper.port1.postMessage('');
<pre id="log"></pre>
We could also do the same using ES6 async/await syntax, since we can be sure that nothing else in our MessageChannel powered timed-loop will interfere (unlike in a Window's postMessage), we can promisify it:
/***************************/
/* Make Fake Window part */
/* ONLY for DEMO */
/***************************/
const fake_win = new MessageChannel();
const win = fake_win.port1; // window.open('your_url', '')
const opener = fake_win.port2; // used in Receiver
/********************/
/* Main window part */
/********************/
const messages = [];
win.onmessage = e => {
messages.push(e.data);
};
! function log_msg() {
document.getElementById('log').textContent = messages.length;
requestAnimationFrame(log_msg);
}();
/*******************/
/* Receiver part */
/*******************/
const looper = makeLooper();
// our async loop function
async function loop(i) {
while (i++ < 1e6) {
opener.postMessage(i);
await looper.next()
}
}
loop(0);
// here we init our promisified time-loop
function makeLooper() {
const engine = new MessageChannel();
return {
next() {
return new Promise((res) => {
engine.port2.onmessage = e => res();
engine.port1.postMessage('');
});
}
};
};
<pre id="log"></pre>
But it could obviously also be made entirely ES5 style with callbacks and everything:
/***************************/
/* Make Fake Window part */
/* ONLY for DEMO */
/***************************/
var fake_win = new MessageChannel();
var win = fake_win.port1; // window.open('your_url', '')
var opener = fake_win.port2; // used in Receiver
/********************/
/* Main window part */
/********************/
var messages = [];
win.onmessage = function(e) {
messages.push(e.data);
};
!function log_msg() {
document.getElementById('log').textContent = messages.length;
requestAnimationFrame(log_msg);
}();
/*******************/
/* Receiver part */
/*******************/
var i = 0;
var looper = makeLooper(loop);
// our callback loop function
function loop() {
if (i++ < 1e6) {
opener.postMessage(i);
looper.next(loop);
}
}
loop(0);
// here we init our promisified time-loop
function makeLooper(callback) {
var engine = new MessageChannel();
return {
next: function() {
engine.port2.onmessage = function(e) {
callback();
}
engine.port1.postMessage('');
}
};
};
<pre id="log"></pre>
But note that browsers will anyway throttle the pages that are not in focus, so you may have slower results than in these snippets.
I have simple image uploader in a website and a javascript function which uses FileReader and converts image to base64 to display it for user without uploading it to actual server.
function generateThumb(file) {
var fileReader = new FileReader();
fileReader.readAsDataURL(file);
fileReader.onload = function (e) {
// Converts the image to base64
file.dataUrl = e.target.result;
};
}
Now I am trying to write tests for this method using Mocha and Chai. My thinking is that I want to check if the file.dataUrl was successfully created and it is base64. So I would like to mock the local file somehow in testing environment (not sure how to do this). Or I should not test this at all and assume that this is working ?
The answer here depends a little. Does your "generateThumbs" method have other logic than loading a files contents and assign that to a property of a passed in object? Or does it have other logic such as generating the thumbnail from the image data, reading out file properties and assigning them to the file object? and so on?
If so then I would infact suggest you mock out the FileReader object with your own, so that you can control your test results. However, it isn't the FileReaders functionality you want to test, it is your own logic. So you should assume that FileReader works and test that your code that depends upon it works.
Now since the method you posted was a bit on the small side, In that case I would just assume it works, rename the method and work on testing the rest of your code. But there is a place for having such a mock, and I must admit it was quite fun to figure out how to mock the event target, so I will give an example here, using your method as a basis:
//This implements the EventTarget interface
//and let's us control when, where and what triggers events
//and what they return
//it takes in a spy and some fake return data
var FakeFileReader = function(spy, fakeData) {
this.listeners = {};
this.fakeData = fakeData;
this.spy = spy;
this.addEventListener('load', function () {
this.spy.loaded = true;
});
};
//Fake version of the method we depend upon
FakeFileReader.prototype.readAsDataURL = function(file){
this.spy.calledReadAsDataURL = true;
this.spy.file = file;
this.result = this.fakeData;
this.dispatchEvent({type:'load'}); //assume file is loaded, and send event
};
FakeFileReader.prototype.listeners = null;
FakeFileReader.prototype.addEventListener = function(type, callback) {
if(!(type in this.listeners)) {
this.listeners[type] = [];
}
this.listeners[type].push(callback);
};
FakeFileReader.prototype.removeEventListener = function(type, callback) {
if(!(type in this.listeners)) {
return;
}
var stack = this.listeners[type];
for(var i = 0, l = stack.length; i < l; i++) {
if(stack[i] === callback){
stack.splice(i, 1);
return this.removeEventListener(type, callback);
}
}
};
FakeFileReader.prototype.dispatchEvent = function(event) {
if(!(event.type in this.listeners)) {
return;
}
var stack = this.listeners[event.type];
event.target = this;
for(var i = 0, l = stack.length; i < l; i++) {
stack[i].call(this, event);
}
};
// Your method
function generateThumb(file, reader){
reader.readAsDataURL(file);
reader.addEventListener('load', function (e) {
file.dataUrl = base64(e.target.result);
});
}
Now we have a nice little object that behaves like a FileReader, but we control it's behavior, and we can now use our method and inject it into, thus enabling us to use this mock for testing and the real thing for production.
So we can now write nice unit tests to test this out:
I assume you have mocha and chai setup
describe('The generateThumb function', function () {
var file = { src: 'image.file'};
var readerSpy = {};
var testData = 'TESTDATA';
var reader = new FakeFileReader(readerSpy, testData);
it('should call the readAsDataURL function when given a file name and a FileReader', function () {
generateThumb(file, reader);
expect(readerSpy.calledReadAsDataURL).to.be.true;
expect(readerSpy.loaded).to.be.true;
});
it('should load the file and convert the data to base64', function () {
var expectedData = 'VEVTVERBVEE=';
generateThumb(file, reader);
expect(readerSpy.file.src).to.equal(file.src);
expect(readerSpy.file.dataUrl).to.equal(expectedData);
});
});
Here is a working JSFiddle example:
https://jsfiddle.net/workingClassHacker/jL4xpwwv/2/
The benefits here are you can create several versions of this mock:
what happens when correct data is given?
what happens when incorrect data is given?
what happens when callback is never called?
what happens when too large of a file is put in?
what happens when a certain mime type is put in?
You can offcourse massively simplify the mock if all you need is that one event:
function FakeFileReader(spy, testdata){
return {
readAsDataURL:function (file) {
spy.file = file;
spy.calledReadAsDataURL = true;
spy.loaded = true;
this.target = {result: testdata};
this.onload(this);
}
};
}
function generateThumb(file, reader){
reader.onload = function (e) {
file.dataUrl = base64(e.target.result);
};
reader.readAsDataURL(file);
}
Which is how I would actually do this. And create several of these for different purposes.
Simple version:
https://jsfiddle.net/workingClassHacker/7g44h9fj/3/
I am a NodeJS newbie from C# world still trying to understand the complex JavaScript maze.
I am trying to implement the multi-threading from the example on https://www.npmjs.com/package/webworker-threads.
Can not understand why the below works:
var Worker = require('webworker-threads').Worker;
var FibCalculator = require('./FibCalculator.js');
require('http').createServer(function (req,res) {
console.log('request received.');
var fibo = new Worker(function() {
var calcFib = function (n) {
return n > 1 ? calcFib(n - 1) + calcFib(n - 2) : 1;
};
this.onmessage = function (event) {
postMessage(calcFib(event.data));
};
});
fibo.onmessage = function (event) {
var msg = 'fib(5) = ' + event.data;
console.log(msg);
res.end(msg);
};
fibo.postMessage(5);
}).listen(3000);
But the below just does not
var Worker = require('webworker-threads').Worker;
var FibCalculator = require('./FibCalculator.js');
//the below function just does not get called
var calcFib = function (n) {
console.log('***This will not print. Can someone explain why?***');
return n > 1 ? calcFib(n - 1) + calcFib(n - 2) : 1;
};
require('http').createServer(function (req,res) {
console.log('request received.');
var fibo = new Worker(function() {
this.onmessage = function (event) {
postMessage(calcFib(event.data));
};
});
fibo.onmessage = function (event) {
var msg = 'fib(5) = ' + event.data;
console.log(msg);
res.end(msg);
};
fibo.postMessage(5);
}).listen(3000);
Why is it that if I take out the scope of calcFib, it cannot get called. My idea was to implement a wrapper over webworker-threads that could be used to perform any CPU intrinsic operations on another thread. However, I am not even able to call an external function
Below is reply to Benjamin Gruenbaum answer:
How can I call console.log inside onmessage but not my function defined outside the scope in the same file?
this.onmessage = function (event) {
console.log('can call console.log even if it is external');
//cannot call calcFib(event.data) if it is defined outside the scope.
postMessage(calcFib(event.data));
};
Workers work by copying the code over so all closure data is lost. You cannot access closure scope from within the worker so calcFib is undefined.
You need to define calcFib inside the worker code or send it via a message and eval it.
var Worker = require('webworker-threads').Worker;
var FibCalculator = require('./FibCalculator.js');
require('http').createServer(function (req,res) {
console.log('request received.');
var fibo = new Worker(function() {
//the below function just does not get called
var calcFib = function (n) {
console.log('***This will not print. Can someone explain why?***');
return n > 1 ? calcFib(n - 1) + calcFib(n - 2) : 1;
};
this.onmessage = function (event) {
postMessage(calcFib(event.data));
};
});
fibo.onmessage = function (event) {
var msg = 'fib(5) = ' + event.data;
console.log(msg);
res.end(msg);
};
fibo.postMessage(5);
}).listen(3000);
I am a collaborator on the node-webworker-threads project.
Why your code doesn't work
node-webworker-threads workers must be taught all of the functionality they are asked to perform. They exist in a separate namespace, so they can't see things present even in the same file unless you specifically tell them about it. You tell them about the necessary functions using node-webworker-threads's load and eval APIs.
What node-webworker-threads workers can do
node-webworker-threads workers can perform native JavaScript functionality, like arrays and objects, interacting with JSON and regular expressions and so on. These workers also support some of the Node.js-specific built-ins, like console.log. This is why you can call console.log (a supported built-in) but not your function (outside of the worker's namespace). As noted elsewhere, in some cases the node-webworker-threads implementations of the built-in APIs are not a perfect match.
I am trying to initialize a Breeze manager inside a 'Web Worker'.
RequireJs, knockout, q, breeze are being imported inside the worker.
After a call to:EntityQuery.from('name').using(manager).execute(),
the following error appears:
Uncaught Error: Q is undefined. Are you missing Q.js? See https://github.com/kriskowal/q.
A live preview is uploaded here http://plnkr.co/edit/meXjKa?p=preview
(plunk supports downloading for easier debug).
EDIT -- relevant code
Worker.js
importScripts('knockout.js', 'q.js', 'breeze.js', 'require.js');
define('jquery', function () { return jQuery; });
define('knockout', ko);
define('q', Q); //Just trying to assign q since breeze requests Q as q
require(function () {
var self = this;
this.q = this.Q; //Just trying to assign q since breeze requests Q as q
breeze.NamingConvention.camelCase.setAsDefault();
var manager = new breeze.EntityManager("breeze/Breeze");
var EntityQuery = breeze.EntityQuery;
// Q or q here is defined (TESTED)
var test = function (name) {
return EntityQuery.from(name)
.using(manager).execute() // <-- Here q/Q breaks (I think on execute)
};
var primeData = function () {
return test('Languages')
.then(test('Lala'))
.then(test('Lala2'))
};
primeData();
setTimeout(function () { postMessage("TestMan"); }, 500);
});
Worker will be initialized on main page as:
var myWorker = new Worker("worker.js");
Ok here it goes:
Create a new requireJs and edit the
isBrowser = !!(typeof window !== 'undefined' && typeof navigator !== 'undefined' && window.document)
to
isBrowser = false
Create a new Jquery so it uses nothing related to window and generally anything that a WebWorker cannot access. Unfortunatelly i can't remember where i got this Custom JQueryJs but i have uploaded it here "https://dl.dropboxusercontent.com/u/48132252/jqueydemo.js".
Please if you find the author or the original change link and give credit.
My workerJs file looks like:
importScripts('Scripts/test.js', 'Scripts/jqueydemo.js', 'Scripts/q.js', 'Scripts/breeze.debug.js', 'Scripts/require2.js');
define('jquery', function () { return jQuery; });
require(
{
baseUrl: "..",
},
function () {
var manager = new breeze.EntityManager("breeze/Breeze");
var EntityQuery = breeze.EntityQuery;
var primeData = function () {
return EntityQuery.from(name)
.using(manager).execute() // Get my Data
.then(function (data) {
console.log("fetced!\n" + ((new Date()).getTime()));
var exportData = manager.exportEntities(); // Export my constructed entities
console.log("created!\n" + ((new Date()).getTime()));
var lala = JSON.stringify(exportData)
postMessage(lala); // Send them as a string to the main thread
})
};
primeData();
});
Finally on my mainJs i have something like:
this.testWorker = function () {
var myWorker = new Worker("worker.js"); // Init Worker
myWorker.onmessage = function (oEvent) { // On worker job finished
toastr.success('Worker finished and returned');
var lala = JSON.parse(oEvent.data); // Reverse string to JSON
manager.importEntities(lala); // Import the pre-Constructed Entities to breezeManager
toastr.success('Import done');
myWorker.terminate();
};
};
So we have managed to use breeze on a WebWorker enviroment to fetch and create all of our entities, pass our exported entities to our main breeze manager on the main thread(import).
I have tested this with 9 tables fully related to each other and about 4MB of raw data.
PROFIT: UI stays fully responsive all the time.
No more long execution script, application not responding or out of memory errors) at least for chrome
*As it makes sense breeze import entities is way more faster than the creation a full 4MB raw data plus the association process following for these entities.
By having all the heavy work done on the back, and only use import entities on the front, breeze allows you to handle large datasets 'like a breeze'.