Node.js/ v8: How to make my own snapshot to accelerate startup - javascript

I have a node.js (v0.6.12) application that starts by evaluating a Javascript file, startup.js. It takes a long time to evaluate startup.js, and I'd like to 'bake it in' to a custom build of Node if possible.
The v8 source directory distributed with Node, node/deps/v8/src, contains a SconScript that can almost be used to do this. On line 302, we have
LIBRARY_FILES = '''
runtime.js
v8natives.js
array.js
string.js
uri.js
math.js
messages.js
apinatives.js
date.js
regexp.js
json.js
liveedit-debugger.js
mirror-debugger.js
debug-debugger.js
'''.split()
Those javascript files are present in the same directory. Something in the build process apparently evaluates them, takes a snapshot of state, and saves it as a byte string in node/out/Release/obj/release/snapshot.cc (on Mac OS). This file seems to be baked into Node.
Some customization of the startup snapshot is possible by altering the SconScript. For example, I can change the definition of the builtin Date.toString by altering date.js. I can even add new global variables by adding startup.js to the list of library files, with contents global.test = 1.
However, I can't put just any javascript code in startup.js. If it contains Date.toString = 1;, an error results even though the code is valid at the node repl:
Build failed: -> task failed (err #2):
{task: libv8.a SConstruct -> libv8.a}
make: *** [program] Error 1
And it obviously can't make use of code that depends on libraries Node adds to v8. global.underscore = require('underscore'); causes the same error.
I'd ideally like a tool, customSnapshot, where customSnapshot startup.js evaluates startup.js with Node and then dumps a snapshot to a file, snapshot.cc, which I can put into the Node source directory. I can then build node and tell it not to rebuild the snapshot.

I just added an option to the mksnapshot command (which runs while you are building V8). The new --extra-file=filename.js flag lets you specify a file that is to be loaded and run in the process and then put in the snapshot. It's on the trunk version of V8, not the 3.11 branch that is being used for node 0.8 so you will have to run node 0.8 with V8 version 3.11. As far as I know at the moment that works, but you will be somewhat on your own.
Please file bugs if you try this and it doesn't work for you.

Related

What's the meaning of "rc" in the config file names [duplicate]

In my home folder in Linux I have several config files that have "rc" as a file name extension:
$ ls -a ~/|pcregrep 'rc$'
.bashrc
.octaverc
.perltidyrc
.screenrc
.vimrc
What does the "rc" in these names mean?
It looks like one of the following:
run commands
resource control
run control
runtime configuration
Also I've found a citation:
The ‘rc’ suffix goes back to Unix's grandparent, CTSS. It had a command-script feature called "runcom". Early Unixes used ‘rc’ for the name of the operating system's boot script, as a tribute to CTSS runcom.
Runtime Configuration normally if it's in the config directory. I think of them as resource files. If you see rc in file name this could be version i.e. Release Candidate.
Edit: No, I take it back officially... "run commands"
[Unix: from runcom files on the CTSS system 1962-63, via the startup script /etc/rc]
Script file containing startup instructions for an application program (or an entire operating system), usually a text file containing commands of the sort that might have been invoked manually once the system was running but are to be executed automatically each time the system starts up.
Thus, it would seem that the "rc" part stands for "runcom", which I believe can be expanded to "run commands". In fact, this is exactly what the file contains, commands that bash should run.
Quoted from What does “rc” in .bashrc stand for?
I learnt something new! :)
In the context of Unix-like systems, the term rc stands for the phrase "run commands". It is used for any file that contains startup information for a command. It is believed to have originated somewhere in 1965 from a runcom facility from the MIT Compatible Time-Sharing System (CTSS).
Reference: https://en.wikipedia.org/wiki/Run_commands
In Unix world, RC stands for "Run Control".
http://www.catb.org/~esr/writings/taoup/html/ch10s03.html
To understand rc files, it helps to know that Ubuntu boots into several different runlevels. They are 0-6, 0 being "halt", 1 being "single-user", 2 being "multi-user"(the default runlevel), etc. This system has now been outdated by the Upstart and initd programs in most Linux Distros. It is still maintained for backwards compatibility.
Within the /etc directory are several folders labeled "rc0.d, rc1.d" etc, through rc6.d. These are the directories the kernel refers to to know which init scripts it should run for that runlevel. They are symbolic links to the system service scripts residing in the /etc/init.d directory.
In the context you are using it, it would appear that you are listing any files with rc in the name. The code in these files will set the way the services/tasks startup and run when initialized.

How do I use the Google V8 Engine in a different c++ Project?

If I import the .lib and .h files in the Google V8 Engine, it will attempt to use them in a different c ++ project.
Do I need to import a file of any name?
When the source of the v8 was compile and build,
cctest
fuzzer_support
generate-bytecode-expectations
gmock
gtest
icui18n
icuuc
inspector_protocol
json_fuzzer_lib
mksnapshot
parser_fuzzer_lib
regexp_fuzzer_lib
unittests
v8
v8_base_0
v8_base_1
v8_base_2
v8_base_3
v8_external_snapshot
v8_libbase
v8_libplatform
v8_libsampler
v8_nosnapshot
v8_simple_json_fuzzer
v8_simple_parser_fuzzer
v8_simple_regexp_fuzzer
v8_simple_wasm_asmjs_fuzzer
v8_simple_wasm_fuzzer
wasm_fuzzer_lib
wasm_asmjs_fuzzer_lib
name file is generated.
I simply use the v8, and it outputs a run value of javascript.
My guess, seems to use only "v8.lib, v8_base (0,1,2,3) .lib" and "v8.h".
Is the sample code in a new c ++ project by copying the contents of the "hello-world.cc", made a cpp file. And I run, "LNK2019", "LNK1120" error occurs.
1>----- Build started: Project: v8Application, Configuration: Debug Win32 -----
1> v8Application.cpp
1>v8Application.obj : error LNK2019: unresolved external symbol _main referenced in function"class v8::Platform * __cdecl v8::platform::CreateDefaultPlatform(int)" (?CreateDefaultPlatform#platform#v8##YAPAVPlatform#2#H#Z) 1>c:\users\kito\documents\visual studio 2015\Projects\v8Application\Debug\v8Application.exe : fatal error LNK1120: 1 unresolved externals
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========
It seems must not defined "platform :: CreateDefaultPlatform" part is turned only in 'libplatform.h' declaration.
However, I would add "src \ libplatform \ default-platform.cc" the platform :: CreateDefaultPlatform has been defined, other error occurs in the add.
"Ws2_32.lib, winmm.lib" was also added to the project of the Additional Dependencies property.
How do I do this?
You must create your project and then LINK with V8, but you must not add V8's source files to your project.
The first thing to do is to compile V8 and take note of where the libraries were generated (or copy them to the directory of your choice). The same applies to V8's include files, they must be available in order to be included in your project (it is not advisable to copy them to your project directory).
In your console project's settings go to Linker -> Input -> Additional Dependencies and put v8.lib there. If the error is still present work you might want to add v8_libplatform.lib too.
Something similar applies to the include files, in your project settings you should go to C/C++ -> General -> Additional Include Directories and add V8's include directory there.
But remember, you must first compile v8 alone and expose the lib and include directories to your new project.
Update:
Your project file indicates that you configured Your project only for the configuración "Release", but the output above indicates that you are compiling in Debug mode.
Project config:
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
...
<Link>
<AdditionalDependencies>ws2_32.lib;winmm.lib;v8.lib;v8_base_0.lib;v8_base_1.lib...
</Link>
<ItemDefinitionGroup
Compiler output above:
1>----- Build started: Project: v8Application, Configuration: Debug Win32 -----
You must configure your project for all configurations, or at least test it using the configuration you've setup.
On the other hand, v8_libplatform.lib is not in the list of additional dependencies, you may want to add it.

Concat then delete source files in Gulp

New to Gulp, using v4 (alpha), learning as I go. I'm thinking of an optimized build process and am at a point where I want to concatenate files (gulp-concat) and delete (del) the source files in 1 step. (my build process copies files from the source location to the build destination, and then only works on files there, rather than directly on source files (as a practice). Once I concat the files, I don't want them in the build, and want to delete them afterwards. FYI, the dest variable just points to the build folder based on flags set when run (./build/ for development builds and ./dist/ for deployment builds.)
gulp.src(dest + 'assets/scripts/**/*.js')
.pipe(gulp.concat('combined.js')
// ??? (stuck here)
Correct me if I'm wrong, but at this point in the pipe, I've lost reference to my "source" files, and only the combined.js file is sitting in the pipe. I could gulp.dest() and save it, but as I understand it that creates a writeStream which I cannot continue to the next .pipe(), right? On top of that, the del plugin does not seem to return a stream, so I'm not sure if I could use it in a pipe, but I just want to ensure it only runs after I've done the concatenation.

WIKI: How to use Lime (how to use closure-compiler with 3rd party (closure) libraries)

The following post inspired me to have a look at limeJS, as a side project I'm working on and off an a Yatzee game (mostly off) and thought that might be a nice library to use.
As a beginner in google-closure I had some difficulties running uncompiled code and getting code compiled, mostly due to not knowing what the options and commands are and because of dependencies.
For other beginners with google-closuse I have written this tutorial.
Note that jQuery can be used by your closure compiled code but you need the jQuery externs file. You can't compile jQuery into your code, closure library has a dressed down dojo that can be found in third_party. When using that you can compile your code to one file.
What you need:
Python
Git client
Closure compiler for compiling code (minifying and merging all files into one)
Closure library like jQuery and jQuery ui but can be compiled along with your code
Python (I use 2.7)
LimeJS As a 3rd party library (use git client to get it, the url:https://github.com/digitalfruit/limejs.git)
JAVA JRE
Directory structure
I'm using Windows and have my projects in D:\projects, if you have your projects somewhere else you have to replace D:\projects references to your own.
In D:\projects I make a directory called libraries in that directory I copy the goog and third_party directories from closure library (goog is under the closure directory) since you'll use them for many projects I keep it at projects root, you can include a copy of it in every project you create but in this case I'll share the library among all projects.
Now I copy the contents of the src directory from limeJS ([lime clone dir]\lime\src) to D:\projects\libraries\lime (the one containing the sub directories called animation, audio ...).
Next I'll use an existing project from the limeJS library, copy the [lime clone dir]\lime\demos\roundball to D:\projects\roundball
At this time your directory structure should look like this:
D:
projects
libraries
goog
lime
animation
audio
css
...
third_party
closure
...
roundball
assets
... other roundball stuff
cacldeps.py
When you open D:\projects\roundball\rb.html and check out the console (press F12 in Chrome or in FireFox preferably having firebug plugin installed) you'll see an error: "ReferenceError: goog is not defined"
Open up D:\projects\roundball\rb.html and change:
<script type="text/javascript" src="../../../closure/closure/goog/base.js"></script>
to
<script type="text/javascript" src="../libraries/goog/base.js"></script>
Now when you open up rb.html again you get a different error: "goog.require could not find: lime.Director"
This is because closure uses deps.js to find dependencies and since lime is not in there it can't find it. Lucky for us there is a tool called calcdeps.py in the bin directory of the closure library that can create this file for us. It'll look in your code and and use goog.require to figure out what other files are needed. If your file structure is good than this tool will find it.
It will expect that Director is in a file called director.js in a directory called lime (and it is). That js file should have a goog.provide statement providing lime.Director.
You can add directories for calcdeps.py to look in with the --path parameter.
In D:\projects\roundball I'll create a makedeps.bat with the following content:
set calc="D:\software\closure compiler\library\closure\bin\calcdeps.py"
c:\Python27\python.exe %calc% ^
--path D:\projects\roundball\ ^
--path D:\projects\libraries\ ^
--input D:\projects\roundball\rb.js ^
--output_mode deps ^
--output_file D:\projects\libraries\goog\deps.js
pause
Uncompiled code uses /goog/deps.js to load dependencies. caclcdeps.py will check your code starting with rb.js (as this is the starting point of the code) and add entries to deps.js according to what your project uses.
Once again, cacldeps.py can be found in the bin directory of closure library.
Note that when you have another project; let's say D:\projects\project2 then you have to create a makedeps.bat in that project directory that re creates the deps.js for you before you can run the uncompiled code. This because multiple projects share one google closure library so when you switch projects make sure you'll build the deps.js first before running your uncompiled code.
Resolving missing dependency
Opening the rt.html we still get an error but a different one: Error: "Undefined nameToPath for lime.css"
Looking in goog/deps.js we can find that lime.css is needed by lime but doesn't have an entry in deps.js. Why is this? Looking in D:\projects\libraries\lime there is no css directory or css.js file. But in the directory where you cloned the git lime project there is a directory called css.
Copy the css directory to D:\projects\libraries\lime and run makedeps.bat again, now when you open rt.html it will run.
The whole thing is uncompiled and you can set breakpoints to step into 3rd party code.
Compile your code
In production you would want to compile the code to one file. To compile the code I created a compile.bat in D:\projects\roundball with the following content:
set calc="D:\software\closure compiler\library\closure\bin\calcdeps.py"
c:\Python27\python.exe %calc% ^
--path D:\projects\roundball\ ^
--path D:\projects\libraries\ ^
--input D:\projects\roundball\rb.js ^
--compiler_jar "D:\software\closure compiler\compiler.jar" ^
--output_mode compiled ^
--compiler_flags="--compilation_level=ADVANCED_OPTIMIZATIONS" ^
--compiler_flags="--formatting=PRETTY_PRINT" ^
--output_file D:\projects\roundball\compiled\roundball.js
pause
;Need this flag for production compile:
;--compiler_flags="--define goog.DEBUG=false" ^
;Remove the following flag from production compile:
;--compiler_flags="--formatting=PRETTY_PRINT" ^
;lime libraries is giving me errors with stricter compilation
; so had to remove this flag (have to fix the errors or no type checking for your code)
;--compiler_flags="--warning_level=VERBOSE" ^
The file compiler.jar can be found where you downloaded (and extracted) closure compiler
Now when you run the compile.bat and open D:\projects\roundball\compiled\roundball.html you'll see in the developer tools of your browser that only roundball.js is loaded.
The answer is in the question because it's a howto article that could help someone (like me in the future when I forgot how to do it).

Python and the Spidermonkey Javascript engine on Linux

I have successfully installed Spidermonkey JS engine on my Linux machine ( Ubuntu ).
Basically my goal is to make it execute Ajax (js) scripts and return the result back to my Python script. I'm basically trying to build a good O.O. web scraper. But it's pretty hard for me to get all of this working.
I'm now at the point where when I type JS in my terminal I can start executing Javascript.
I've been Googling and found this little snipet on Stackoverflow :
import urllib2
import spidermonkey
js = spidermonkey.Runtime()
js_ctx = js.new_context()
script = urllib2.urlopen('http://etherhack.co.uk/hashing/whirlpool/js/whirlpool.js').read()
js_ctx.eval_script(script)
js_ctx.eval_script('var s="abc"')
js_ctx.eval_script('print(HexWhirpool(s))')
but it failed to run with the error that module Spidermonkey can not be found.
I'm a bit lost now. Anyone able to help?
I also tried easy_install python-spidermonkey with no luck, for libnspr-dev package is absent.
So, I've built package from source. Instructions from project page (Debian Stretch):
Building
Check out the Python-Spidermonkey module from the SVN repository ( I downloaded it as source archive, direct link )
Unpack, and cd to ./python-spidermonkey/trunk
CPPFLAGS="-Wno-format-security" python setup.py build (these flags for Debian)
Error jsemit.h:508:32: error: expected ‘(’ before ‘)’ token uintN decltype); means that decltype cannot be used as variable (maybe it's a macro or something else), fix it this way:
sed -e 's/decltype/dectyp/' -i.ORIG ./js/src/jsemit.h
sed -e 's/decltype/dectyp/' -i.ORIG ./js/src/jsemit.cpp
Error jsemit.cpp:6490:1: error: narrowing conversion of ‘-1’ from ‘int’ to ‘uint8 {aka unsigned char}’ inside { } [-Wnarrowing] means illegal variable conversion, recompile it manually:
cd js/src
g++ -o Linux_All_DBG.OBJ/jsemit.o -c -Wall -Wno-narrowing -Wno-format -MMD -g3 -DXP_UNIX -DSVR4 -DSYSV -D_BSD_SOURCE -DPOSIX_SOURCE -DHAVE_LOCALTIME_R -DHAVE_VA_COPY -DVA_COPY=va_copy -DPIC -fPIC -DDEBUG -DDEBUG_user -DEDITLINE -ILinux_All_DBG.OBJ jsemit.cpp
Error spidermonkey.c:1:2: error: #error Do not use this file, it is the result of a failed Pyrex compilation. - some trouble with pyrex. There is a patch. Do it this way:
wget -O - https://storage.googleapis.com/google-code-attachments/python-spidermonkey/issue-14/comment-4/cinit.patch | patch -p1 ./spidermonkey.pyx
Installation
su, and python setup.py install as root.
Running
By default, setup script installs libjs.so to /usr/local/lib/, so I did ln -s /usr/local/lib/libjs.so /usr/lib/libjs.so (but you'd better use solution from Seagal82)
Without this step, python keeps complaining about import ImportError: libjs.so: cannot open shared object file: No such file or directory
I also had an error ImportError: cannot import name Runtime after from spidermonkey import Runtime. The reason possibly was in old easy_install data in ~/.local/lib/python2.7/site-packages/spidermonkey/. After removing it, all runs smooth
Recently i got a task need to do something like Web scraping,
and for the javascript part, currently want to try using python-spidermonkey to resolve it and see if this might work for me ...
and i seem to meet situation might alike, after i think i finished install python-spidermonkey, i execute the script above, i got this error:
Traceback (most recent call last):
File "spidermonkeytest.py", line 2, in <module>
import spidermonkey
ImportError: libjs.so: cannot open shared object file: No such file or directory
then after some searching by google...i found the solution probably in the end of here:
http://hi.baidu.com/peizhongyou/item/ec1575c3f0e00e31e80f2e48
i setup these things:
$sudo vi /etc/ld.so.conf.d/libjs.so.conf
fill in this line:
/usr/local/lib/
save & exit, execute ldconfig:
$sudo ldconfig
then i can run the script provided above by #Synbitz Prowduczions
don't know if this is the answer you need, or this still helps?
You need to try libnspr4. If that doesn't work, you can always download it from Mozilla and build the code yourself.
It is not difficult to type ./config && make && make install to build the library yourself after untarring the source. If you build yourself, files will likely be in
/usr/local/{include,lib}
Also just try Googling for "YOUR_OS_NAME install nspr4".
I believe someone wrote a C/C++ header file translator for Python ctypes. Although I can't say much else because I don't use Python.
SpiderMonkey also has its own implementation of ctypes modeled after Python. So technically if you know javascript you could forego using Python altogether since you want to do some ajax with it. You will need to brush up on the NSPR or C runtime sockets to meet the requirements for your projects using only Spidermonkey.
OR a web search for Python +AJAX might turn up exactly what you need.

Categories

Resources