I have a C++ project that I compile to Javascript using emscripten. This works, however, for resource limits and interactivity reasons I would like to run this inside a webworker.
However, my project uses the stdin. I found a way to provide my own implementation of stdin by overwriting Module['stdin'] with a function that returns a single character at a time of the total stdin, and closes with 0 as EOF.
This works when the script runs inside the page, as the Module object present in the html file is shared with the script.
When you run as a webworker though, this module object is not shared. Instead, message passing makes sure the regular functionality of Module still works. This does not include 'stdin'.
I worked around this by modifying the output javascript:
A: Adding an implementation of a Module object that includes this stdin specification. This function is modified to read a variable of the webworker as if it were the stdin and feed this on a per-character basis.
B: Changing the onmessage of the webworker to call an additional function handling my own events.
C: This additional function listens to the events and reacts when the event is the content of stdin, by setting the variable that the stdin function I specified reads.
D: adding and removing run dependencies on this additional event to prevent the c++ code running without the stdin specified.
In code:
Module['stdin_pointer'] = 0;
Module['stdin_content'] = "";
Module['stdin']=(function () {
if (Module['stdin_pointer'] < Module['stdin_content'].length) {
code = Module['stdin_content'].charCodeAt(Module['stdin_pointer']);
Module['stdin_pointer']=Module['stdin_pointer']+1;
return code;
} else {
return null;
}
});
external = function(message){
switch(message.data.target){
case 'stdin' : {
Module['idpCode'] = message.data.content;
removeRunDependency('stdin');
break;
}
default: throw 'wha? ' + message.data.target;
}
};
[...]
addRunDependency("stdin");
[...]
//Change this in the original onmessage function:
// default: throw 'wha? ' + message.data.target;
//to
default: {external(message);}
Clearly, this a & c part is quite easy because it can be added at the start (or near the start) of the js file, but b & d (adding your own dependencies and getting your own messagehandler in the loop) requires you to edit the code inline.
As my project is very large, finding the necessary lines to edit can be very cumbersome, only more so in optimized and mimified emscripten code.
Automatic scripts to do this, as well as the workaround itself, are likely to break on new emscripten releases.
Is there a nicer, more proper way to reach the same behavior?
Thank you!
//EDIT:
The --separate-asm flag is quite helpful, in the respect that the file that I must edit is now only a few lines long (in mimified form). It greatly reduces the burden, but it is still not a proper way, so I'm reluctant to mark this as resolved.
The only way I know of achieving what you want is to not use the Emscripten-supplied worker API, and roll your own. All the details are probably beyond the scope of a single question, but at a high level you'll need to...
Compile the worker module with your processing code, but not using the BUILD_AS_WORKER flag
At both the UI and worker ends, you'll need to write some JavaScript code to communicate in/out of the C++ worlds, using one of the techniques at http://kripken.github.io/emscripten-site/docs/porting/connecting_cpp_and_javascript/Interacting-with-code.html, that then directly calls the JavaScript worker API https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
At the Worker side of this, you will be able to control the Module object, setting stdin as you see fit
As a side-note, I have found that the Emscripten-supplied C++ wrappers for JavaScript functionality, such as workers, graphics, audio, http requests etc, are good to get going at first, but have limitations and don't expose everything that is technically possible. I have often had to roll my own to get the functionally needed. Although not for the same reasons, I have also had to write my own API for workers.
Related
I'm learning v8 now, but I have encountered some problems.
How to set a breakpoint a method's start address in memory if I want to debug a method's C++ implementation.
e.g. var a= new Array(0,1); a.indexOf(1) ; I want to set a breakpoint at slice's beginning, or are there other ways to track the assembler code ?
There are a lot of functions will be complied and writed into a file named snapshot.bin. so I can't set a breakpoint at the beginning of these functions.
You need to check the source code and find the implementation of slice. Then set a gdb/lldb break point in that .cc file: byiltins-typedarray.cc
A lot of functions are defined as builtin or runtime functions.
It depends on the kind of function you want to inspect.
You can compile without snapshot to get around snapshot-related debugging difficulties (at the cost of making startup quite a bit slower: several seconds in Debug mode).
You can modify the respective code generator to emit a break instruction at the beginning of the function. For the example of Array.indexOf, that's probably the easiest solution; the CodeStubAssembler instruction is called DebugBreak().
You can break somewhere else using GDB, find your way to the function in question (e.g. via isolate->builtins), and set a breakpoint on the address of its entry. (This requires a bit of V8 knowledge and/or code reading skills, but it's not difficult.)
You can use various --print-*-code flags to print code to stdout (without breaking on it).
this questions pops up again and again across the internet (even on SO), but I haven't found a satisfying solution to this problem:
How can we change/replace Javascript code in a running web application, without reloading the page?
Many people answer this with "you cannot, because it is impossible". Some experiments with IntelliJ IDEAs live edit plugin proves me that it is possible. But I don't want to be bound to an IDE for this feature. (Bonus: browser independent)
Here is what I tried:
add //# sourceURL=whatever.js to my dynamically loaded script
add folder to Chrome containing whatever.js
mapping the local whatever.js to the network whatever.js
changing code in either does not affect the web-page at all. In fact editing the network-side file results in a oddish "flashing" of the dev tools.
Please understand that I do not expect the changed JS to magically apply to the webpage once I change it, but I expect it to use the new code when the execution point is passed again.
Example:
Given a button that triggers 'alert(1);'
Change to 'alert(2);'
I expect the button to trigger 'alert(2);'
Having many dependencies and a huge script that is triggered pretty late in a workflow it is really a big problem for me to refresh the page, so I need to find a solution that works on-the-fly.
First of all: What you ask for is really tricky and you can find security problems if you allow this in your applications, anyway it is not impossible.
BUT if you want to achieve your example follow this steps:
Make a code snippet like this:
var message = "1"; // this must be a global variable!!!!
function showMessage() {
alert(message);
}
Given a button that triggers 'alert(1);'
Make button call a function ie: onclick='showMessage()'
Change to 'alert(2);'
I expect the button to trigger 'alert(2);'
Now it's easy, When you detect the event that implies to change the alert message to 2 you just need to change message value:
message = "2";
That's all.
Option 1: Livereload
I would say as long it's for develop reasons you can use livereload on your server.
Depends of your server type. I'm note big expert in apach, glassfish and other java's world stuff, but in world of JS (nodejs) this is a shorter way.
(link for npm-livereload)
Hack: You can handle static-files such as js, css with simple node.js server with built-in livereload.
Option 2: jRebel
I'm not sure about js but perhaps JRebel can handle this issue. Anyway it's a good addition to the develop process - at least it would make a java's "hot reload: for you.
Option 3: Monkey-patching
You can use monkey-patching techniques: Each function in js it's just a string, you can turn string -> function with new Function().
just like:
var foo = {
sum: function (a, b) {return a+b;}
}
//...
obj.sum = new Function(....) //Now you're replaced the original code
check this article about graceful way to do monkey-patching.
And small advertising of my lib for monkey-patching: monkey-punch
Option 4: Attach new tag
You can attach js files with:
var s = document.createElement("script");
s.type = "text/javascript";
s.src = "http://somedomain.com/somescript";
$("head").append(s);
You're also able to remove dom elements (scripts, styles) and attach new at anytime.
I'm just starting to play around with Emscripten, but I've run into something odd, when trying to export functions, for use in JavaScript. The test project has to do with libsquish, but it's not really relevant for the question, other than the header/code filenames.
So - anyway, simple test.
c/c++
//squish.h
extern "C" int main();
int main();
extern "C" int gsr();
int gsr();
//squish.cpp
int main()
{
return 99;
}
int gsr()
{
//return squish::GetStorageRequirements(width,height,flags);
return 8675309;
}
Additional Options/Command Line
-s EXPORTED_FUNCTIONS="['_gsr','_main']"
Javascript
main=Module.cwrap('main','number',null);
console.log(main());
GetStorageRequirements = Module.cwrap('gsr', 'number',null);
console.log(GetStorageRequirements());
Javascript Console (Chrome)
99
Assertion failed: Cannot call unknown function gsr (perhaps LLVM optimizations or closure removed it?)
Assertion failed: Cannot call unknown function gsr (perhaps LLVM optimizations or closure removed it?)
And on top of everything else, optimization is disabled (O0).
So, what's the deal here? Same return types, same number of parameters, defined in exactly the same way, with the only difference being the name of the function. Yet one works, and the other is "unknown".
The only possible, logical cause is that it's automatically exporting main, and ignoring the EXPORTED_FUNCTIONS all together. But if that's the case, I have no idea why.
Thanks to Charles Ofria's work in narrowing down the issue, the only real oddity ended up being with the Emscripten Visual Studio plugin. I was on the right track, when I thought that it might just be automatically exporting main - which was correct. The reason it wasn't exporting the other function, was because I put it in the Additional Options, for the Clang/C++ section, instead of the Emcc Linker.
The trouble was, and this is where the "oddity" comes in, the Emcc Linker section doesn't appear when you choose Console Application (.js) as the Configuration Type. So, after changing the Target Extension to .html and the Configuration Type to Browser Application (.html) - the Linker section re-appeared, and I was able to put the EXPORTED_FUNCTIONS option, in the correct location.
After building, and copying the output over, both functions were defined, and returned the proper values in the Javascript console.
Probably the issue here is C++ name mangling. That's a C++ filename, so "gsr" turns into something like "_Z3gsr". You can avoid that by making it a C file, or using extern "C" around that method.
For more see http://kripken.github.io/emscripten-site/docs/getting_started/FAQ.html#why-do-functions-in-my-c-c-source-code-vanish-when-i-compile-to-javascript-and-or-i-get-no-functions-to-process
I'm developing a networked application between the browser and a server running node.js. I'm sharing a lot of code right now, but when I actually deploy this I'd like the client to only get client specific code. My options right now are:
1.) Implement any browser/node.js differences using inheritance. I've tried this in a few places and I end up with a lot of classes that are very, very basic customizations of their parent often only partially specializing a single function. This is not a style I like very much because it means a lot indirection when you're trying to find out what's actually going on.
2.) Define a constant like IS_BROWSER at global scope and then check it whenever I need to change code paths on the browser vs node.js. Then closure compile all js with advanced optimizations to remove dead code on the browser (setting IS_BROWSER = true). Are there any problems with this approach assuming I do whatever I need to do to get advanced optimizations going in closure compiler?
3.) ?? I'm open to suggestions.
If you use advanced compilation, any unused code should be removed; if you use the compiler's export system correctly, any server-side code that your client code does not call will not be in the compiled version of the client code.
You could write all of your code in one big blob then, for your client, add one file with contents like
goog.require('my.client.app');
goog.exportSymbol('my.app.entryPoint', my.client.app.entryPoint);
the compiled code will not include anything that is not in the call tree of my.client.app.entryPoint. Likewise, if your compilation only exports a server entry point, client code will be excluded.
The above style is for writing your script to provide some function which will then get called by an inline script; to make the whole thing into a single script you could do something much simpler:
goog.require('my.client.app');
my.client.app.entryPoint();
To verify that you are not getting a lot of dead code in your compilation output, you could play around with something like this: ScriptCover
One of the theoretical benefits from working with Node.js is the possibility to share the same scripts between clients and the server. That would make it possible to degrade the same functionality to the server if the client does not support javascript.
However, the Node.js require() method works on it's own. In the script you load, you can add stuff to this or exports that will later be available in the object that fetched the script:
var stuff = require('stuff');
stuff.show();
In stuff.js:
this.show = function() {
return 'here is my stuff';
}
So, when re-using this script on the client, the .show() method will be added to the window scope. That is not what we want here, instead we would like to add it to a custom namespace.
My only solution so far is something like (in stuff.js):
var ns = typeof exports == 'undefined' ? (function() {
return window['stuff'] = {};
})() : exports;
ns.show = function() {
return 'here is my stuff';
}
delete ns; // remove ns from the global scope
This works quite well since I can call stuff.show() on the server and client. But it looks quirky. I tried searching for solutions but node.js is still very new (even to me) so there are few reliable resources. Does anyone have a better idea on how to solve this?
In short, if you want to re-use scripts don't use Node.js specific stuff you have to go with the lowest common denominator here, the Browser.
Solutions are:
Go overkill and use RequireJS, this will make it work in both Node.js and the Browser. But you need to use the RequireJS format on the server side and you also need to plug in an on the fly converted script...
Do your own loader
Wrap your re-use scripts both on the server and client side with an anonymous function
Now create some code that users call(module) on that function, on the Node side you pass in this for the module, on the client side you pass in a name space object
Keep it simple and stupid, as it is now, and don't use this in the module scope on the Node.js side of things
I wish I could give you a simple out of the box solution, but both environments differ to much in this case. If you really have huge amounts of code, you might consider a build script which generates the files.