Develop Reference

NodeJS On Run-Time Adding/Removing/Reloading requires WITHOUT server restart (No nodemon either)

I have a project for work where I am basically creating a form of CMS to which we will add applications as time moves forward.
The issue we're having is getting those applications loaded in (and more specifically modified) on run-time within the server.
The reason we're requiring this form of "hot loading" is because we don't want the server to restart whenever a change has been made, and more specifically, we'd like to add the new applications through an admin panel.
Nodemon is a useful tool for development, but for our production environment we want to be able to replace an existing application (or module/plugin if you will) without having to restart the server (whether it's manually or through nodemon, the server needs to be running at all time).
You could compare this to how CMS' like Drupal, Yoomla, or Wordpress do things, but for our needs, we decided that Node was the better way to go for many reasons.
Code wise, I am looking for something like this, but that will work:
let applications = []
//add a new application through the web interface calling the appropiate class method, within the method the following code runs:
applications.push(require('path/to/application');
//when an application gets modified:
applications.splice(index,1);
applications.push('path/to/application');
But I require existing instances of said application to be adjusted as well.
Example:
// file location: ./Applications/application/index.js
class application {
greet() {
console.log("Hello");
}
}
module.exports = application;
the app loader would load in said application:
class appLoader {
constructor() {
this.List = new Object();
}
Add(appname) {
this.List[appname] = require(`./Applications/${appname}/index`);
}
Remove(appname) {
delete require.cache[require.resolve(`./Applications/${appname}/index`)]
delete this.List[appname];
}
Reload(appname) {
this.Remove(appname);
this.Add(appname);
}
}
The running code:
const AppLoader = require('appLoader');
const applications = new AppLoader();
applications.add('application'); // adds the application created above
var app = new applications.List['application']();
app.greet();
// Change is made to the application file, .greet() now outputs "Hello World" instead of "Hello"
//do something to know it has to reload, either by fs.watch, or manual trigger
applications.Reload('application');
app.greet();
The expected behavior is:
Hello
Hello World
In reality, I'm getting:
Hello
Hello
If anyone can help me figure out a way to dynamically load in applications like this, but also remove/reload them during run-time, it would be greatly appreciated!
Edit: if there is a way to run my application code without the use of require that would allow a dynamic load/reload/remove, that is also a welcome solution

Ok, thanks to #jfriend00 I realized I need to fix something else with my code, so his comments can still be useful for other people. As to my issue of unloading required modules or reloading them without a server restart, I figured out a relatively elegant way of making it happen.
Let me start by showing you all my test class and app.js and I'll explain what I did and how it works.
Class.js:
"use strict";
class Class {
constructor() {
// this.file will be put in comments post run-time, and this.Output = "Hey" will be uncommented to change the source file.
var date = new Date()
this.Output = date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds() + "." + date.getMilliseconds();
this.file = global.require.fs.readFileSync('./file.mov');
//this.Output = "Hey";
}
}
module.exports = Class;
app.js:
'use strict';
global.require = {
fs: require('fs')
};
const arr = [];
const mod = './class.js'
let Class = [null];
Class[0] = require(mod);
let c = [];
c.push(new Class[0]());
console.log(c[0].Output);
console.log(process.memoryUsage());
setTimeout(() => {
delete require.cache[require.resolve(mod)];
delete Class[0];
Class[0] = require(mod);
console.log(Class)
delete c[0];
c[0] = new Class[0]();
console.log(c[0].Output);
console.log(process.memoryUsage());
}, 10000);
Now let me explain here for a bit, and mind you, this is testing code so the naming is just horrid.
This is how I went to work:
Step 1
I needed a way to separate required modules (like fs, or websocket, express, etc.) so it wouldn't mess with the whole delete require_cache() part of the code, my solution was making those globally required:
global.required = {
fs: require('fs')
}
Step 2
Figure out a way to make sure the Garbage Collector removes the unloaded code, I achieved this by putting my requires and class declarations inside of a variable so that I could use the delete functionality within Node/Javascript. (I used let in my test code because I was testing another method beforehand, haven't tested if const would work again).
I also made a variable that contains the path string for the file (in this case './Class.js' but for my explanation below I'll just write it in as is)
let Class = [null] //this declares an array that has an index '0'
Class[0] = require('./Class');
let c = [new Class[0]()] // this declares an array that has the class instantiated inside of index '0'
As for the garbage collection, I'm simply able to do the following:
delete Class[0];
delete c[0];
After this I am able to redo the declaration of the required class and subsequently the class itself and keep my code working without requiring a restart.
Take in mind that his takes a lot of work to implement in an actual project, but you could split it up by adding an unload() method to a class to unload underlying custom classes. But my initial testing shows that this works like a charm!
Edit: I feel required to note that without jfriend00's comments I'd never have figured out this solution
Output
When the project start, it outputs the current time and the process.memoryUsage()
13:49:13.540
{ rss: 50343936,
heapTotal: 7061504,
heapUsed: 4270696,
external: 29814377 }
during the 10 second wait, I change the Class.js file to not read the file.mov and say "Hey" instead of the time, after the 10s timout this is the output:
Hey
{ rss: 48439296,
heapTotal: 7585792,
heapUsed: 4435408,
external: 8680 }

nodeJS prevent access to code for variable passed into a function

I'm creating a plugin system using the following:
function Plugin(thingy, code)
{
var GLOBAL = null;
var arguments = null;
var process = null;
var require = null;
eval(code);
};
plugins.push(new Plugin(thingy, code));
Please don't get too excited about the eval(), using ('vm') or a sandbox is not an option as this will be a long running object until the user unloads it. It will also be running in it's own nodeJS instance so they can't affect other users. I'd still have the same problem passing in this object reference to a sandbox system anyway.
What I am concerned about is someone seeing the code of the thingy object that has functions they need to use e.g shoot()
console.log(thingy.shoot.toString());
A way around this was the following:
function thingy()
{
// They can't see this
var _shoot = function(someone)
{
// Load weapon
// Aim
// Fire
};
// They can see this
this.shoot = function(someone)
{
_shoot(someone);
};
};
This way if they console.log(thingy.shoot.toString()) they'll only see _shoot(someone); and not the actual code that handles the shooting process.
Please could someone help me with the following:
Is there an easier way to limit access to a passed in variables code?
I'm setting GLOBAL, arguments, process and require to null; are there others I need to worry about?

Nodejs: Global variables across multiple files

I have written my code across several files for my node server.
If I have a file, say basket.js:
var Basket = {
fruits : 0,
addFruit : function() {
fruits++;
},
removeFruit : function() {
fruits--;
},
printFruit : function() {
console.log(this.fruits);
}
}
module.export = Basket;
And I have another file called give.js:
var Basket1 = require("./basket.js");
Basket1.addFruit();
Basket1.printFruit();
And another file called take.js:
var Basket2 = require("./basket.js");
Basket2.removeFruit();
Basket2.printFruit();
Will both files write into the same instance of Basket?
In other words, will they both have control over the property, fruits?
Does node manage race conditions on its own? i.e. if two commands to modify fruit come in at the same time from add and sub, does node know how to handle it?
If I want to make a way in which two files can look at a singleton at the same time and access it, is this the way to go?? Or how else does one do it?

Yes, they will access the same object.
Modules are cached after the first time they are loaded. This means (among other things) that every call to require('foo') will get exactly the same object returned, if it would resolve to the same file.
– Modules docs
No, node does not manage race conditions on its own, because race conditions will not be caused by node itself. Node is single-threaded and thus no code can be executed at the same time as other code. See for example this answer for some more explanation.

I'm a beginner but I think the correct syntax is module.exports not modules.export - if you may correct so that people don't wonder why it does not work like I just did :)

JavaScript Autoloader: How To Recover From Errors

I have a proof-of-concept working for a JavaScript autoloader, but it currently suffers from a major flaw: it requires the code to be re-executed in its entirety rather than simply trying again from the line that failed.
Here is the prototype:
<!doctype html>
<html>
<head>
</head>
<body>
<script type="text/javascript">
var app = function(){
console.log('Initialize App');
var test = new Test();
var foo = new Foo();
var bar = new Bar();
};
var autoload = function(app){
var recover = function(error){
var name = error.message.split(' ')[0];
console.log('Loading '+name);
//A file could be synchronously loaded here instead
this[name] = function(){
console.log(name+' has been dynamically created');
};
load(app);
};
var load = function(app){
try {
app();
} catch (error){
if (error.name == "ReferenceError"){
console.log(error.message);
recover(error, app);
}
}
};
load(app);
};
autoload(app);
</script>
</body>
</html>
How It's Supposed To Work
The idea is that all of your application code would get executed within the app function. Eventually, if I can get it working properly, you could also pass in a dependency map to autoloader with the app function to synchronously load dependencies when a function is not defined. The dependency map would simply be an object mapping function names to file names.
How It Currently Works
If you don't feel like trying it out, the above code outputs the following to the console:
Initialize App
Test is not defined
Loading Test
Initialize App
Test has been dynamically created
Foo is not defined
Loading Foo
Initialize App
Test has been dynamically created
Foo has been dynamically created
Bar is not defined
Loading Bar
Initialize App
Test has been dynamically created
Foo has been dynamically created
Bar has been dynamically created
The complete app function is re-executed each time the autoloader catches an error. Obviously, this is less than ideal for a number of reasons.
Recovering From The Error
To move to the next step for making this work, I need to find a way to recover from the error without re-executing the entire app function. The error object from the catch block does provide both the line number and file name where the error occurred, but so far, I haven't been able to find a way to take advantage of that information. There are three general approaches that I can think of:
Restart script execution at the given line
Restart script execution at the beginning, but skip all lines until the given line
Grab the file as a string, split it into an array by line number, and eval the remaining lines.
Unfortunately, I wasn't able to find information on either of the first two approaches. Of the three, #1 seems like it would be more ideal, but I would certainly be open to other creative suggestions as well. As far as I can tell, JavaScript doesn't provide a way to start script execution at an arbitrary line number. #3 might work, but I'm not sure it would be very performant. The only way I can think of doing it would be to require an extra request each time to load the file text into a string.
The Questions
This is admittedly pushing the boundaries of how dependencies could be loaded in JavaScript. I'm not even sure if it is possible because I don't know if JavaScript allows for this type of error recovery. That said, I'm interested in exploring it further until I find out it's absolutely impossible.
In the interests of getting this working:
Is there a way to start script execution at an arbitrary line of JavaScript?
Are there other approaches to this that might be more fruitful? Feel free to be creative!
Taking a step back to look at the bigger picture (assuming I can get this to work):
Is a JavaScript autoloader something that people would even want?
What are the advantages/disadvantages to an autoloader like this vs. an approach like AMD?
What kind of performance issues would be encountered with this approach? Would the performance hit be too much to make it worth it?

It's kind of complicated to make, throwing and catching is kind of expensive as well. You could use typeof window["Test"]!=="function" and then create it instead of using the try catch like this.
But for a general recover and continue approach the following code would do the trick.
var app = (function(i){
var objects=new Array(3),
fnNames=["Test","Foo","Bar"];
return function(){
var len=fnNames.length
,test,foo,bar;
//check if closure vars have been reset, if so
// this has ran successfully once so don't do anything?
if(objects===false){
console.log("nothing to do, initialized already");
return;
}
while(i<len){
try{
objects[i] = new window[fnNames[i]]();
i++;
}catch(e){
if (e.name == "TypeError"){//different syntax different error
throw {"fnName":fnNames[i]};
}
}
}
//store instances in the variables
test = objects[0];
foo = objects[1];
bar = objects[2];
//reset closure vars assuming you only call app once
// when it's successful
i=null;objects=false;
console.log("Got all the instances",test,foo,bar);
};
}(0));
var autoload = function(app){
var recover = function(name){
console.log('Loading '+name);
//A file could be synchronously loaded here instead
this[name] = function(){
this.name=name;
console.log(this.name+' has been dynamically created');
};
load(app);
};
var load = function(app){
try {
app();
} catch (error){
//if statement here no longer needed
// object thrown has fnName (function name)
recover(error.fnName, app);
}
};
load(app);
};
autoload(app);
autoload(app);

Why can't I extend everyone's pocket in nowjs?

I'm trying to provide functions in everyone's pocket of nowjs. I'd like to do so by _.extending everyone's pocket, i.e. everyone.now. For some reason which I cannot understand, _.extend fails to properly provide the function at the client side.
This is my current code:
var _ = require("underscore"),
everyone = require("nowjs").initialize(app);
everyone.now.foo = function() {};
_.extend(everyone.now, {
bar: function() {}
});
console.log(everyone.now.foo); // [Function]
console.log(everyone.now.bar); // undefined
On both the server and client sides, I can do now.foo() just fine. On the other hand, now.bar() fails because now.bar is not defined. This is the case on both the client and server sides. I tried to check for existence at the server side, as shown above on the last line. However, this line logs undefined.
Underscore's extend function (obviously) does work on other objects so I guess it has something to do with the "magical namespace" that nowjs uses.
How come extending doesn't work with everyone.now and how can I get it to work?
Edit 2: I digged some more into proxies. It seems like setting a property on a proxy by passing a variable as its name does not work. I removed my first edit because this testcase is more narrowed down.
Why is this not working? Is this a bug? (Most of the times I ask this myself I know it isn't, but this is really making me clueless...)
var proxy = Proxy.create({
get: function(pr, name) {
console.log("get called");
return null;
},
set: function(pr, name, value) {
console.log("set called");
}
});
var key = "foo";
proxy["foo"] = "bar";
proxy[ key ] = "bar";
proxy["foo"];
proxy[ key ];
Log result:
set called
get called
get called
Apparently, proxy[ key ] = "bar"; does not cause set to be called on the proxy. Why is that?

They posted a blog entry on the NowJS website on how to use node-proxy on Windows, instead of the native V8 implementation using the --harmony_proxies flag.
It appeared that the V8 version that Node currently uses contains several bugs with regard to proxies, which were causing the weird behaviour as outlined in the question. node-proxy, however, is a module that enables proxies (the core of the "magical namespace" of NowJS) without those bugs. (The bugs are fixed in a newer version of V8 as well, but that would require a custom build of Node.)
I just couldn't figure out how to build node-proxy on Windows (it's a .node addon; not a pure JavaScript one). In the above blog post they distributed the compiled module, and everything now works like a charm.
To fix:
Download the compiled module
Extract the folder to the node_modules folder and rename it to now
Don't run Node with the proxy flag
Edit: Node 0.7.0 uses V8 3.8.6 which also solves this issue. Just run with the --harmony flag and remove the reference to node-proxy.