We are building an Electron app that allows users to supply their own 'modules' to run. We are looking for a way to require the modules but then delete or kill the modules if need be.
We have looked a few tutorials that seem to discuss this topic but we can't seem to get the modules to fully terminate. We explored this by using timers inside the modules and can observe the timers still running even after the module reference is deleted.
https://repl.it/repls/QuerulousSorrowfulQuery
index.js
// Load module
let Mod = require('./mod.js');
// Call the module function (which starts a setInterval)
Mod();
// Delete the module after 3 seconds
setTimeout(function () {
Mod = null;
delete Mod;
console.log('Deleted!')
}, 3000);
./mod.js
function Mod() {
setInterval(function () {
console.log('Mod log');
}, 1000);
}
module.exports = Mod;
Expected output
Mod log
Mod log
Deleted!
Actual output
Mod log
Mod log
Deleted!
Mod log
...
(continues to log 'Mod log' indefinitely)
Maybe we are overthinking it and maybe the modules won't be memory hogs, but the modules we load will have very intensive workloads and having the ability to stop them at will seems important.
Edit with real use-case
This is how we are currently using this technique. The two issues are loading the module in the proper fashion and unloading the module after it is done.
renderer.js (runs in a browser context with access to document, etc)
const webview = document.getElementById('webview'); // A webview object essentially gives us control over a webpage similar to how one can control an iframe in a regular browser.
const url = 'https://ourserver.com/module.js';
let mod;
request({
method: 'get',
url: url,
}, function (err, httpResponse, body) {
if (!err) {
mod = requireFromString(body, url); // Module is loaded
mod(webview); // Module is run
// ...
// Some time later, the module needs to be 'unloaded'.
// We are currently 'unloading' it by dereferencing the 'mod' variable, but as mentioned above, this doesn't really work. So we would like to have a way to wipe the module and timers and etc and free up any memory or resources it was using!
mod = null;
delete mod;
}
})
function requireFromString(src, filename) {
var Module = module.constructor;
var m = new Module();
m._compile(src, filename);
return m.exports;
}
https://ourserver.com/module.js
// This code module will only have access to node modules that are packaged with our app but that is OK for now!
let _ = require('lodash');
let obj = {
key: 'value'
}
async function main(webview) {
console.log(_.get(obj, 'key')) // prints 'value'
webview.loadURL('https://google.com') // loads Google in the web browser
}
module.exports = main;
Just in case anyone reading is not familiar with Electron, the renderer.js has access to 'webview' elements which are almost identical to iframes. This is why passing it to the 'module.js' will allow the module to access manipulate the webpage such as change URL, click buttons on that webpage, etc.
There is no way to kill a module and stop or close any resources that it is using. That's just not a feature of node.js. Such a module could have timers, open files, open sockets, running servers, etc... In addition node.js does not provide a means of "unloading" code that was once loaded.
You can remove a module from the module cache, but that doesn't affect the existing, already loaded code or its resources.
The only foolproof way I know of would be to load the user's module in a separate node.js app loaded as a child process and then you can exit that process or kill that process and then the OS will reclaim any resources it was using and unload everything from memory. This child process scheme also has the advantage that the user's code is more isolated from your main server code. You could even further isolate it by running this other process in a VM if you wanted to.
Related
I am building an Electron app for editing .txt files on a windows computer. I have used electron builders fileAssociations to open .txt files however I cannot get the path of the file. When I looked in the Electron docs I found this which is exactly what I want except it is mac-only... is there a way to get the same functionality on Windows?
As Lucas Roland mentioned, you have to parse process.argv to get the filepath
if (process.argv.length >= 2) { // or electron.remote.process.argv
let filePath = process.argv[1];
//open, read, handle file
}
On Windows, you have to parse process.argv (in the main process) to
get the filepath.
from https://github.com/electron/electron/blob/master/docs/api/app.md#event-open-file-macos
On Windows, you need to use process.argv in the main process to read the file path. According to this answer, you can use the fs package to open, read & write files. There are a few more ways described to do the same thing.
Also, the following snippet from this blog post might be helpful.
How to configure your app to open linked files in Windows
On Windows, you have to parse process.argv to get the file path.
Then, you can use the IPC module to handle messages from the renderer process (web page) and retrieve a datastore from a file. This is how we did it:
In the main process:
var ipc = require('ipc');
var fs = require('fs');
// read the file and send data to the render process
ipc.on('get-file-data', function(event) {
var data = null;
if (process.platform == 'win32' && process.argv.length >= 2) {
var openFilePath = process.argv[1];
data = fs.readFileSync(openFilePath, 'utf-8');
}
event.returnValue = data;
});
I am not very well versed with electron, else I would have tried to give you a better answer, but this is what I could find with my understanding about it. I hope this helps!
For windows, parsing process.argv on app ready worked for me. So in the main process somewhere:
const { app } = require('electron');
const devEnv = /electron/.test(process.argv[0]);
app.on('ready', () => {
if (process.platform.startsWith('win') && !devEnv && process.argv.length >= 2) {
const filePath = process.argv[1];
// In my app, I initialise a new window here and send filePath through
// to the renderer process so it can be read and displayed.
} else {
// Create whatever your default window is (in my case, an empty document)
}
});
On windows, this event is fired every time you open a new file even if another file is already open in the application.
Remember also that during development, we launch the application with electron . or electron . --debug or something similar. Because of this, I also test whether the first argument is electron and skip over file opening logic in this case (the case that devEnv evaluates to 'true').
The best alternative that I have found for this problem is using second-instance event on the app module with process.argv.
When a user clicks a file associated with your app, on Windows (yeah! I only tried it on Windows), it will try to open a new instance of the app with the path of the related file inside process.argv. If you are not using multiple instances of the app, then you should prevent the app from opening multiple instances of the app and try to fetch the data (mainly the process.argv) to the main instance using the second-instance event from the app module.
First, you should get the singleInstanceLock from app.requestSingleInstanceLock() to make the current instance the only instance of the app. This will return a boolean. If it returns false, this means that another instance has the singleInstanceLock. Then you should close that instance using app.quit().
// Behaviour on second instance for parent process
const gotSingleInstanceLock = app.requestSingleInstanceLock();
if (!gotSingleInstanceLock) app.quit(); // Quits the app if app.requestSingleInstanceLock() returns false.
If the boolean is true, then this means the current app session is the main session of the app.
Then we should use the second-instance event of the app module to detect second instances from that point onwards. If another instance emerges, it will be closed, but second-instance event will be fired with the relevant arguments needed to identify what opened that instance to the process.argv. The second parameter of the callback added to the second-instance event can be used to get those arguments sent to that instance. From that point onwards, you can filter the strings inside the process.argv and get the relevant path of the file that opened the other instance.
// Behaviour on the second instance for the parent process
const gotSingleInstanceLock = app.requestSingleInstanceLock();
if (!gotSingleInstanceLock) app.quit();
else {
app.on('second-instance', (_, argv) => {
//User requested a second instance of the app.
//argv has the process.argv arguments of the second instance.
if (app.hasSingleInstanceLock()) {
if (mainWindow?.isMinimized()) mainWindow?.restore();
mainWindow?.focus();
process.argv = argv; // I tried to add the argv from the second instance to my main instance.
}
});
}
Please keep in mind that arguments in process.argv will be added differently when the app is packaged. Therefore, you should loop through the arguments and check whether an argument is a path in the system using the stat() function in the fs module and if it has the .txt extension at the end using the path.extname()
I found this method the hard way, so I hope that anyone with this problem wouldn't have to go through all of that.
I have a project for work where I am basically creating a form of CMS to which we will add applications as time moves forward.
The issue we're having is getting those applications loaded in (and more specifically modified) on run-time within the server.
The reason we're requiring this form of "hot loading" is because we don't want the server to restart whenever a change has been made, and more specifically, we'd like to add the new applications through an admin panel.
Nodemon is a useful tool for development, but for our production environment we want to be able to replace an existing application (or module/plugin if you will) without having to restart the server (whether it's manually or through nodemon, the server needs to be running at all time).
You could compare this to how CMS' like Drupal, Yoomla, or Wordpress do things, but for our needs, we decided that Node was the better way to go for many reasons.
Code wise, I am looking for something like this, but that will work:
let applications = []
//add a new application through the web interface calling the appropiate class method, within the method the following code runs:
applications.push(require('path/to/application');
//when an application gets modified:
applications.splice(index,1);
applications.push('path/to/application');
But I require existing instances of said application to be adjusted as well.
Example:
// file location: ./Applications/application/index.js
class application {
greet() {
console.log("Hello");
}
}
module.exports = application;
the app loader would load in said application:
class appLoader {
constructor() {
this.List = new Object();
}
Add(appname) {
this.List[appname] = require(`./Applications/${appname}/index`);
}
Remove(appname) {
delete require.cache[require.resolve(`./Applications/${appname}/index`)]
delete this.List[appname];
}
Reload(appname) {
this.Remove(appname);
this.Add(appname);
}
}
The running code:
const AppLoader = require('appLoader');
const applications = new AppLoader();
applications.add('application'); // adds the application created above
var app = new applications.List['application']();
app.greet();
// Change is made to the application file, .greet() now outputs "Hello World" instead of "Hello"
//do something to know it has to reload, either by fs.watch, or manual trigger
applications.Reload('application');
app.greet();
The expected behavior is:
Hello
Hello World
In reality, I'm getting:
Hello
Hello
If anyone can help me figure out a way to dynamically load in applications like this, but also remove/reload them during run-time, it would be greatly appreciated!
Edit: if there is a way to run my application code without the use of require that would allow a dynamic load/reload/remove, that is also a welcome solution
Ok, thanks to #jfriend00 I realized I need to fix something else with my code, so his comments can still be useful for other people. As to my issue of unloading required modules or reloading them without a server restart, I figured out a relatively elegant way of making it happen.
Let me start by showing you all my test class and app.js and I'll explain what I did and how it works.
Class.js:
"use strict";
class Class {
constructor() {
// this.file will be put in comments post run-time, and this.Output = "Hey" will be uncommented to change the source file.
var date = new Date()
this.Output = date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds() + "." + date.getMilliseconds();
this.file = global.require.fs.readFileSync('./file.mov');
//this.Output = "Hey";
}
}
module.exports = Class;
app.js:
'use strict';
global.require = {
fs: require('fs')
};
const arr = [];
const mod = './class.js'
let Class = [null];
Class[0] = require(mod);
let c = [];
c.push(new Class[0]());
console.log(c[0].Output);
console.log(process.memoryUsage());
setTimeout(() => {
delete require.cache[require.resolve(mod)];
delete Class[0];
Class[0] = require(mod);
console.log(Class)
delete c[0];
c[0] = new Class[0]();
console.log(c[0].Output);
console.log(process.memoryUsage());
}, 10000);
Now let me explain here for a bit, and mind you, this is testing code so the naming is just horrid.
This is how I went to work:
Step 1
I needed a way to separate required modules (like fs, or websocket, express, etc.) so it wouldn't mess with the whole delete require_cache() part of the code, my solution was making those globally required:
global.required = {
fs: require('fs')
}
Step 2
Figure out a way to make sure the Garbage Collector removes the unloaded code, I achieved this by putting my requires and class declarations inside of a variable so that I could use the delete functionality within Node/Javascript. (I used let in my test code because I was testing another method beforehand, haven't tested if const would work again).
I also made a variable that contains the path string for the file (in this case './Class.js' but for my explanation below I'll just write it in as is)
let Class = [null] //this declares an array that has an index '0'
Class[0] = require('./Class');
let c = [new Class[0]()] // this declares an array that has the class instantiated inside of index '0'
As for the garbage collection, I'm simply able to do the following:
delete Class[0];
delete c[0];
After this I am able to redo the declaration of the required class and subsequently the class itself and keep my code working without requiring a restart.
Take in mind that his takes a lot of work to implement in an actual project, but you could split it up by adding an unload() method to a class to unload underlying custom classes. But my initial testing shows that this works like a charm!
Edit: I feel required to note that without jfriend00's comments I'd never have figured out this solution
Output
When the project start, it outputs the current time and the process.memoryUsage()
13:49:13.540
{ rss: 50343936,
heapTotal: 7061504,
heapUsed: 4270696,
external: 29814377 }
during the 10 second wait, I change the Class.js file to not read the file.mov and say "Hey" instead of the time, after the 10s timout this is the output:
Hey
{ rss: 48439296,
heapTotal: 7585792,
heapUsed: 4435408,
external: 8680 }
I have an app and it uses webworkers to help do some computation with other workers. The error it throws is essentially complaining that the data it's expecting isn't there since the computations weren't computed.
The computations are performed locally with the webworkers running, but in prod I am pretty much out of ideas why it's not working.
One thing to note is that because I need the context of certain functions (defined outside of the worker context) imported in, I've followed the advice suggested here: ( Creating a web worker inside React ) and am stringing it in first and turning it into a blob first.
`
export const buildBlobUrlFromCode = (code) => {
// returns a DOMString that links context to the code within the blob
code = code.substring(code.indexOf("{") + 1, code.lastIndexOf("}"));
const blob = new Blob([code], { type: "application/javascript" });
return URL.createObjectURL(blob);
}
export class WebWorker {
constructor(worker) {
let code = worker.toString();
// const blobUrl = strDecoratedBuildBlobUrlFromCode(code);
const blobUrl = buildBlobUrlFromCode(code);
return new Worker(blobUrl);
}
}
and how I use the above code is:
const workerInstance = new WebWorker(MyWorker);
where MyWorker is a fn exported with onmessage defined and all the operations defined inside.
I added some logs, and essentially the code break when I start to kick the computation off with: workerInstance.postMessage([data]);
Also to note is that I'm using the react-create-app to build my app and I haven't ejected yet so the webpack configs are untouched.
Also -- the source files for these workers are completely blank when deployed, but are filled on local ( as in I can see the actual code in the worker.js files )
Heroku -- Broken:
Local -- Working:
We have been struggling with a similar issue and we found the following...
We are also using Webpack and importantly a production configuration which uglified the code. During this process we found that the onmessage function variable was being discarded as it was not being called anywhere else. This explained why no code was visible in the production build worker instance.
We also found another potential pitfall in that the uglification process was changing all variable names to single characters. This process would also render the onmessage function useless.
To fix it, instead of declaring the variable onmessage, assign onmessage to the global object (this)
this.onmessage = (event) => {}
This way the uglification did not discard the code and the function name was not altered.
I know this is late but hope it helps.
In a "normal" require.js function module, the module is considered "loaded" as soon as the module function returns:
define(function() {
// As soon as this function returns, the module is "loaded"
});
But I have a module that needs to do some asynchronous script loading (specifically including some Google Javascript API-s) and I don't want my module to be considered "loaded" until I say it is.
When creating a loader plugin for require.js, you are supplied with an "onload" function that you can call when the plugin is done loading. This would be perfect for my case, but I don't want my Google API wrapper to be a plugin, I want it to appear to be a "normal" module. Plugins are treated differently by the optimizer and I don't want that headache. Also plugins must be required using special syntax, and I'd like to avoid having to remember that every time I use it.
I have combed through the API several times without finding a way to accomplish what I'm trying to do. Is there an undocumented (or poorly documented) method of defining a module, where the module itself gets to decide when it should be considered "loaded"?
As an example, an implementation like this would be awesome, if it existed:
define(["onload"], function(onload) {
setTimeout(onload, 5000);
});
The first time this module was required, it should take 5 seconds to "load".
We bootstrap a lot of stuff using the convention below which is based on early releases (0.3.x) of the MEAN stack and uses the awesome async library.
Using your example, it might look something like this:
// bootstrap.js
// Array of tasks to pass to async to execute.
var tasks = [];
// Require path
var thePath = __dirname + '/directoryContainingRequires';
// Build array of tasks to be executed in parallel.
fs.readdirSync(thePath).forEach(function (file) {
if (~file.indexOf('.js')) {
var filePath = thePath + '/' + file;
tasks.push(function(callback) {
require(filePath)(callback);
});
}
});
// Execute parallel methods.
async.parallel(
tasks,
function(err) {
if(err) console.error(err);
console.log('All modules loaded!');
process.exit(0);
}
);
The file being required looks similar to this:
// yourModule.js
module.exports = function(moduleCallback) {
setTimeout(function() {
console.log('5 seconds elapsed');
moduleCallback(null);
}, 5000);
};
Is it possible to hot reload external js files in node.js based on its timestamp?
I know node.js caches module after first time load from here: http://nodejs.org/docs/latest/api/modules.html#modules_caching
Modules are cached after the first time they are loaded. This means
(among other things) that every call to require('foo') will get
exactly the same object returned, if it would resolve to the same
file.
And I also know if I need to reload it I can do like this:
// first time load
var foo = require('./foo');
foo.bar()
...
// in case need to reload
delete require.cache[require;.resolve('./foo')]
foo = require('./foo')
foo.bar();
But I wonder if there's any native support in node.js that it watches the file and reload it if there's any changes. Or I need to do it by myself?
pseudo code like
// during reload
if (timestamp for last loaded < file modified time)
reload the file as above and return
else
return cached required file
P.S. I am aware of supervisor and nodemon and don't want to restart server for reloading some specific modules.
There is native support, although it isn't consistent across operating systems. That would be either using fs.watch(), or fs.watchFile(). The first function will use file change events, while the second one will use stat polling.
You can watch a file, and the check it's modification time when it has changed.
var fs = require('fs');
var file = './module.js';
var loadTime = new Date().getTime();
var module = require(file);
fs.watch(file, function(event, filename) {
fs.stat(file, function(err, stats) {
if (stats.mtime.getTime() > loadTime) {
// delete the cached file and reload
} else {
// return the cached file
}
});
});
With fs.watchFile(), you don't need to use fs.stats() since the functions already callbacks instances of fs.Stat.
fs.watchFile(file, function(curr, prev) {
if (curr.mtime.getTime() > loadTime) {
// delete the cached file and reload
} else {
// return the cached file
}
});