I have a project for work where I am basically creating a form of CMS to which we will add applications as time moves forward.
The issue we're having is getting those applications loaded in (and more specifically modified) on run-time within the server.
The reason we're requiring this form of "hot loading" is because we don't want the server to restart whenever a change has been made, and more specifically, we'd like to add the new applications through an admin panel.
Nodemon is a useful tool for development, but for our production environment we want to be able to replace an existing application (or module/plugin if you will) without having to restart the server (whether it's manually or through nodemon, the server needs to be running at all time).
You could compare this to how CMS' like Drupal, Yoomla, or Wordpress do things, but for our needs, we decided that Node was the better way to go for many reasons.
Code wise, I am looking for something like this, but that will work:
let applications = []
//add a new application through the web interface calling the appropiate class method, within the method the following code runs:
applications.push(require('path/to/application');
//when an application gets modified:
applications.splice(index,1);
applications.push('path/to/application');
But I require existing instances of said application to be adjusted as well.
Example:
// file location: ./Applications/application/index.js
class application {
greet() {
console.log("Hello");
}
}
module.exports = application;
the app loader would load in said application:
class appLoader {
constructor() {
this.List = new Object();
}
Add(appname) {
this.List[appname] = require(`./Applications/${appname}/index`);
}
Remove(appname) {
delete require.cache[require.resolve(`./Applications/${appname}/index`)]
delete this.List[appname];
}
Reload(appname) {
this.Remove(appname);
this.Add(appname);
}
}
The running code:
const AppLoader = require('appLoader');
const applications = new AppLoader();
applications.add('application'); // adds the application created above
var app = new applications.List['application']();
app.greet();
// Change is made to the application file, .greet() now outputs "Hello World" instead of "Hello"
//do something to know it has to reload, either by fs.watch, or manual trigger
applications.Reload('application');
app.greet();
The expected behavior is:
Hello
Hello World
In reality, I'm getting:
Hello
Hello
If anyone can help me figure out a way to dynamically load in applications like this, but also remove/reload them during run-time, it would be greatly appreciated!
Edit: if there is a way to run my application code without the use of require that would allow a dynamic load/reload/remove, that is also a welcome solution
Ok, thanks to #jfriend00 I realized I need to fix something else with my code, so his comments can still be useful for other people. As to my issue of unloading required modules or reloading them without a server restart, I figured out a relatively elegant way of making it happen.
Let me start by showing you all my test class and app.js and I'll explain what I did and how it works.
Class.js:
"use strict";
class Class {
constructor() {
// this.file will be put in comments post run-time, and this.Output = "Hey" will be uncommented to change the source file.
var date = new Date()
this.Output = date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds() + "." + date.getMilliseconds();
this.file = global.require.fs.readFileSync('./file.mov');
//this.Output = "Hey";
}
}
module.exports = Class;
app.js:
'use strict';
global.require = {
fs: require('fs')
};
const arr = [];
const mod = './class.js'
let Class = [null];
Class[0] = require(mod);
let c = [];
c.push(new Class[0]());
console.log(c[0].Output);
console.log(process.memoryUsage());
setTimeout(() => {
delete require.cache[require.resolve(mod)];
delete Class[0];
Class[0] = require(mod);
console.log(Class)
delete c[0];
c[0] = new Class[0]();
console.log(c[0].Output);
console.log(process.memoryUsage());
}, 10000);
Now let me explain here for a bit, and mind you, this is testing code so the naming is just horrid.
This is how I went to work:
Step 1
I needed a way to separate required modules (like fs, or websocket, express, etc.) so it wouldn't mess with the whole delete require_cache() part of the code, my solution was making those globally required:
global.required = {
fs: require('fs')
}
Step 2
Figure out a way to make sure the Garbage Collector removes the unloaded code, I achieved this by putting my requires and class declarations inside of a variable so that I could use the delete functionality within Node/Javascript. (I used let in my test code because I was testing another method beforehand, haven't tested if const would work again).
I also made a variable that contains the path string for the file (in this case './Class.js' but for my explanation below I'll just write it in as is)
let Class = [null] //this declares an array that has an index '0'
Class[0] = require('./Class');
let c = [new Class[0]()] // this declares an array that has the class instantiated inside of index '0'
As for the garbage collection, I'm simply able to do the following:
delete Class[0];
delete c[0];
After this I am able to redo the declaration of the required class and subsequently the class itself and keep my code working without requiring a restart.
Take in mind that his takes a lot of work to implement in an actual project, but you could split it up by adding an unload() method to a class to unload underlying custom classes. But my initial testing shows that this works like a charm!
Edit: I feel required to note that without jfriend00's comments I'd never have figured out this solution
Output
When the project start, it outputs the current time and the process.memoryUsage()
13:49:13.540
{ rss: 50343936,
heapTotal: 7061504,
heapUsed: 4270696,
external: 29814377 }
during the 10 second wait, I change the Class.js file to not read the file.mov and say "Hey" instead of the time, after the 10s timout this is the output:
Hey
{ rss: 48439296,
heapTotal: 7585792,
heapUsed: 4435408,
external: 8680 }
I'm running against a wall here, maybe it's just a small problem where I can't see the solution due to my inexperience with NodeJS.
Right now I'm constructing a BT device which will be controlled by a master application and I have settled for the prototyping on a Raspberry PI 3 with NodeJS using the Bleno module.
So far everything worked fine, the device gets found and I can set and get values over Bluetooth. But to separate the different "programs" which the device could execute from the Bluetooth logic (because of loops etc.) I have opted to extract these into external NodeJS files.
The idea here was to use the NodeJS fork module and control the starting and stoppping of those processes through the main process.
But herein my problems start. I can fork the different JavaScript files without problem and these get executed, but I can't get them to stop and I don't know how to fix it.
Here's the code (simplified):
var util = require('util');
var events = require('events');
var cp = require('child_process');
...
var ProgramTypeOne = {
NONE: 0,
ProgramOne: 1,
...
};
...
var currentProgram=null;
...
function BLEDevice() {
events.EventEmitter.call(this);
...
this.currentProgram=null;
...
}
util.inherits(BLELamp, events.EventEmitter);
BLELamp.prototype.setProgram = function(programType, programNumber) {
var self = this;
var result=0;
if(programType=="ProgramTypeOne "){
if(programNumber==1){
killProgram();
this.currentProgram=cp.fork('./programs/programOne');
result=1;
}else if(programNumber==2){
...
}
self.emit('ready', result);
};
...
module.exports.currentProgram = currentProgram;
...
function killProgram(){
if(this.currentProgram!=null){
this.currentProgram.kill('SIGTERM');
}
}
There seems to be a problem with the variable currentProgram which, seemingly, never gets the childprocess from the fork call.
As I have never worked extensivley with JavaScript, except some small scripts on websites, I have no idea where exactly my error lies.
I think it has something to do with the handling of class variables.
The starting point for me was the Pizza example of Bleno.
Hope the information is enough and that someone can help me out.
Thanks in advance!
Since killProgram() is a standalone function outside of the scope of BLELamp, you need to call killProgram with the correct scope by binding BLELamp as this. Using apply (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/apply) should resolve it. The following I would expect would fix it (the only line change is the one invoking killProgram):
BLELamp.prototype.setProgram = function(programType, programNumber) {
var self = this;
var result=0;
if(programType=="ProgramTypeOne "){
if(programNumber==1){
killProgram.apply(this);
this.currentProgram=cp.fork('./programs/programOne');
result=1;
}else if(programNumber==2){
...
}
self.emit('ready', result);
};
As a side note, your code is kind of confusing because you have a standalone var currentProgram then a couple prototypes with their own bound this.currentPrograms. I would change the names to prevent confusion.
I have written my code across several files for my node server.
If I have a file, say basket.js:
var Basket = {
fruits : 0,
addFruit : function() {
fruits++;
},
removeFruit : function() {
fruits--;
},
printFruit : function() {
console.log(this.fruits);
}
}
module.export = Basket;
And I have another file called give.js:
var Basket1 = require("./basket.js");
Basket1.addFruit();
Basket1.printFruit();
And another file called take.js:
var Basket2 = require("./basket.js");
Basket2.removeFruit();
Basket2.printFruit();
Will both files write into the same instance of Basket?
In other words, will they both have control over the property, fruits?
Does node manage race conditions on its own? i.e. if two commands to modify fruit come in at the same time from add and sub, does node know how to handle it?
If I want to make a way in which two files can look at a singleton at the same time and access it, is this the way to go?? Or how else does one do it?
Yes, they will access the same object.
Modules are cached after the first time they are loaded. This means (among other things) that every call to require('foo') will get exactly the same object returned, if it would resolve to the same file.
– Modules docs
No, node does not manage race conditions on its own, because race conditions will not be caused by node itself. Node is single-threaded and thus no code can be executed at the same time as other code. See for example this answer for some more explanation.
I'm a beginner but I think the correct syntax is module.exports not modules.export - if you may correct so that people don't wonder why it does not work like I just did :)
I am trying to use requirejs in an Apple TV project. We have a lot of requirejs modules written for web, would be cool if we could re-use them.
Apple TV platform has certain limitations and it's sorta impossible to use requirejs "as is". There's no DOM in common sense.
One possible way I found to overcome the problem is: first to load require.js itself and then override its .load() method, so whenever require('foo') gets called it would load foo.js via a simple XHR call:
requirejs.load = (context, moduleName, moduleUrl) ->
reqModule = new XMLHttpRequest()
reqModule.open('GET', appRoot+moduleUrl, true)
reqModule.send(null)
reqModule.onreadystatechange = ->
if reqModule.readyState is 4 and reqModule.status is 200
fn = (new Function(reqModule.responseText))() # parse module
context[moduleName] = fn
context.completeLoad(moduleName)
So this works for normally defined modules like this:
define [], ->
someField: 'empty field'
Even works for self executing functions like this (with shim configured):
(myFoo = ->
someField:"empty field"
)()
for example Undercore.js contains itself in a self executing wrapper
However, that doesn't work with modules defined like this:
myFoo = ->
someField:"empty field"
Question: how can I make it work for all 3 cases? When used in browser, requirejs successfully loads all of them.
One solution I found is to wrap the function in define block for non-wrapped modules like in the last example, so instead of doing fn = (new Function(reqModule.responseText))() I would do:
fn = define [], (new Function("return "+reqModule.responseText))()
But then that would break load for both first and second cases. Is there a way to find out if a function wrapped in a self-executing block or not? How can I distinguish first two cases from the last one?
Using the code in the question as a starting point, I was able to get the following code to work. I don't have Apple TV so I cannot test it on Apple TV. I've tested it in a browser. It is able to load all 3 types of modules you've shown in your question, provided that the 2nd and 3rd modules have appropriate shims. So the logic is sound. The missing piece is what needs to stand in for window in eval.call(window, ...). In Node.js, it would be global. I don't know the equivalent in Apple TV.
requirejs.load = function(context, moduleName, moduleUrl) {
var reqModule = new XMLHttpRequest();
reqModule.open('GET', moduleUrl, true);
reqModule.send(null);
return reqModule.onreadystatechange = function() {
if (reqModule.readyState === 4 && reqModule.status === 200) {
eval.call(window, reqModule.responseText);
return context.completeLoad(moduleName);
}
};
};
If I were you, I would use Browserify
Write your browser code with node.js-style requires.
I'm wondering if it's possible to sandbox JavaScript running in the browser to prevent access to features that are normally available to JavaScript code running in an HTML page.
For example, let's say I want to provide a JavaScript API for end users to let them define event handlers to be run when "interesting events" happen, but I don't want those users to access the properties and functions of the window object. Am I able to do this?
In the simplest case, let's say I want to prevent users calling alert. A couple of approaches I can think of are:
Redefine window.alert globally. I don't think this would be a valid approach because other code running in the page (i.e., stuff not authored by users in their event handlers) might want to use alert.
Send the event handler code to the server to process. I'm not sure that sending the code to the server to process is the right approach, because the event handlers need to run in the context of the page.
Perhaps a solution where the server processes the user defined function and then generates a callback to be executed on the client would work? Even if that approach works, are there better ways to solve this problem?
Google Caja is a source-to-source translator that "allows you to put untrusted third-party HTML and JavaScript inline in your page and still be secure."
Have a look at Douglas Crockford's ADsafe:
ADsafe makes it safe to put guest code (such as third party scripted advertising or widgets) on any web page. ADsafe defines a subset of JavaScript that is powerful enough to allow guest code to perform valuable interactions, while at the same time preventing malicious or accidental damage or intrusion. The ADsafe subset can be verified mechanically by tools like JSLint so that no human inspection is necessary to review guest code for safety. The ADsafe subset also enforces good coding practices, increasing the likelihood that guest code will run correctly.
You can see an example of how to use ADsafe by looking at the template.html and template.js files in the project's GitHub repository.
I created a sandboxing library called jsandbox that uses web workers to sandbox evaluated code. It also has an input method for explicitly giving sandboxed code data it wouldn't otherwise be able to get.
The following is an example of the API:
jsandbox
.eval({
code : "x=1;Math.round(Math.pow(input, ++x))",
input : 36.565010597564445,
callback: function(n) {
console.log("number: ", n); // number: 1337
}
}).eval({
code : "][];.]\\ (*# ($(! ~",
onerror: function(ex) {
console.log("syntax error: ", ex); // syntax error: [error object]
}
}).eval({
code : '"foo"+input',
input : "bar",
callback: function(str) {
console.log("string: ", str); // string: foobar
}
}).eval({
code : "({q:1, w:2})",
callback: function(obj) {
console.log("object: ", obj); // object: object q=1 w=2
}
}).eval({
code : "[1, 2, 3].concat(input)",
input : [4, 5, 6],
callback: function(arr) {
console.log("array: ", arr); // array: [1, 2, 3, 4, 5, 6]
}
}).eval({
code : "function x(z){this.y=z;};new x(input)",
input : 4,
callback: function(x) {
console.log("new x: ", x); // new x: object y=4
}
});
An improved version of RyanOHara's web workers sandbox code, in a single file (no extra eval.js file is necessary).
function safeEval(untrustedCode)
{
return new Promise(function (resolve, reject)
{
var blobURL = URL.createObjectURL(new Blob([
"(",
function ()
{
var _postMessage = postMessage;
var _addEventListener = addEventListener;
(function (obj)
{
"use strict";
var current = obj;
var keepProperties =
[
// Required
'Object', 'Function', 'Infinity', 'NaN', 'undefined', 'caches', 'TEMPORARY', 'PERSISTENT',
// Optional, but trivial to get back
'Array', 'Boolean', 'Number', 'String', 'Symbol',
// Optional
'Map', 'Math', 'Set',
];
do
{
Object.getOwnPropertyNames(current).forEach(function (name)
{
if (keepProperties.indexOf(name) === -1)
{
delete current[name];
}
});
current = Object.getPrototypeOf(current);
}
while (current !== Object.prototype)
;
})(this);
_addEventListener("message", function (e)
{
var f = new Function("", "return (" + e.data + "\n);");
_postMessage(f());
});
}.toString(),
")()"],
{type: "application/javascript"}));
var worker = new Worker(blobURL);
URL.revokeObjectURL(blobURL);
worker.onmessage = function (evt)
{
worker.terminate();
resolve(evt.data);
};
worker.onerror = function (evt)
{
reject(new Error(evt.message));
};
worker.postMessage(untrustedCode);
setTimeout(function ()
{
worker.terminate();
reject(new Error('The worker timed out.'));
}, 1000);
});
}
Test it:
https://jsfiddle.net/kp0cq6yw/
var promise = safeEval("1+2+3");
promise.then(function (result) {
alert(result);
});
It should output 6 (tested in Chrome and Firefox).
As mentioned in other responces, it's enough to jail the code in a sandboxed iframe (without sending it to the server-side) and communicate with messages.
I would suggest to take a look at a small library I created mostly because of the need to providing some API to the untrusted code, just like as described in the question: there's an opportunity to export the particular set of functions right into the sandbox where the untrusted code runs. And there's also a demo which executes the code submitted by a user in a sandbox:
http://asvd.github.io/jailed/demos/web/console/
I think that js.js is worth mentioning here. It's a JavaScript interpreter written in JavaScript.
It's about 200 times slower than native JavaScript, but its nature makes it a perfect sandbox environment. Another drawback is its size – almost 600 KB, which may be acceptable for desktops in some cases, but not for mobile devices.
All the browser vendors and the HTML5 specification are working towards an actual sandbox property to allow sandboxed iframes -- but it's still limited to iframe granularity.
In general, no degree of regular expressions, etc. can safely sanitise arbitrary user provided JavaScript as it degenerates to the halting problem :-/
An ugly way, but maybe this works for you:
I took all the globals and redefined them in the sandbox scope, as well I added the strict mode so they can't get the global object using an anonymous function.
function construct(constructor, args) {
function F() {
return constructor.apply(this, args);
}
F.prototype = constructor.prototype;
return new F();
}
// Sanboxer
function sandboxcode(string, inject) {
"use strict";
var globals = [];
for (var i in window) {
// <--REMOVE THIS CONDITION
if (i != "console")
// REMOVE THIS CONDITION -->
globals.push(i);
}
globals.push('"use strict";\n'+string);
return construct(Function, globals).apply(inject ? inject : {});
}
sandboxcode('console.log( this, window, top , self, parent, this["jQuery"], (function(){return this;}()));');
// => Object {} undefined undefined undefined undefined undefined undefined
console.log("return of this", sandboxcode('return this;', {window:"sanboxed code"}));
// => Object {window: "sanboxed code"}
https://gist.github.com/alejandrolechuga/9381781
An independent JavaScript interpreter is more likely to yield a robust sandbox than a caged version of the built-in browser implementation.
Ryan has already mentioned js.js, but a more up-to-date project is JS-Interpreter. The documentation covers how to expose various functions to the interpreter, but its scope is otherwise very limited.
As of 2019, vm2 looks like the most popular and most regularly-updated solution to running JavaScript in Node.js. I'm not aware of a front-end solution.
With NISP you'll be able to do sandboxed evaluation.
Though the expression you write is not exactly JavaScript code, instead you'll write S-expressions. It is ideal for simple DSLs that doesn't demand extensive programming.
Suppose you have code to execute:
var sCode = "alert(document)";
Now, suppose you want to execute it in a sandbox:
new Function("window", "with(window){" + sCode + "}")({});
These two lines when executed will fail, because "alert" function is not available from the "sandbox"
And now you want to expose a member of window object with your functionality:
new Function("window", "with(window){" + sCode + "}")({
'alert':function(sString){document.title = sString}
});
Indeed you can add quotes escaping and make other polishing, but I guess the idea is clear.
Where is this user JavaScript code coming from?
There is not much you can do about a user embedding code into your page and then calling it from their browser (see Greasemonkey). It's just something browsers do.
However, if you store the script in a database, then retrieve it and eval() it, then you can clean up the script before it is run.
Examples of code that removes all window. and document. references:
eval(
unsafeUserScript
.replace(/\/\/.+\n|\/\*.*\*\/, '') // Clear all comments
.replace(/\s(window|document)\s*[\;\)\.]/, '') // Removes window. Or window; or window)
)
This tries to prevent the following from being executed (not tested):
window.location = 'http://example.com';
var w = window;
There are a lot of limitations you would have to apply to the unsafe user script. Unfortunately, there isn't any 'sandbox container' available for JavaScript.
I've been working on a simplistic JavaScript sandbox for letting users build applets for my site. Although I still face some challenges with allowing DOM access (parentNode just won't let me keep things secure =/), my approach was just to redefine the window object with some of its useful/harmless members, and then eval() the user code with this redefined window as the default scope.
My "core" code goes like this... (I'm not showing it entirely ;)
function Sandbox(parent){
this.scope = {
window: {
alert: function(str){
alert("Overriden Alert: " + str);
},
prompt: function(message, defaultValue){
return prompt("Overriden Prompt:" + message, defaultValue);
},
document: null,
.
.
.
.
}
};
this.execute = function(codestring){
// Here some code sanitizing, please
with (this.scope) {
with (window) {
eval(codestring);
}
}
};
}
So, I can instantiate a Sandbox and use its execute() function to get code running. Also, all new declared variables within eval'd code will ultimately bound to the execute() scope, so there will not be clashing names or messing with existing code.
Although global objects will still be accessible, those which should remain unknown to the sandboxed code must be defined as proxies in the Sandbox::scope object.
You can wrap the user's code in a function that redefines forbidden objects as parameters -- these would then be undefined when called:
(function (alert) {
alert ("uh oh!"); // User code
}) ();
Of course, clever attackers can get around this by inspecting the JavaScript DOM and finding a non-overridden object that contains a reference to the window.
Another idea is scanning the user's code using a tool like JSLint. Make sure it's set to have no preset variables (or: only variables you want), and then if any globals are set or accessed do not let the user's script be used. Again, it might be vulnerable to walking the DOM -- objects that the user can construct using literals might have implicit references to the window object that could be accessed to escape the sandbox.