Develop Reference

Dynamic require() in pkg (node.js)

IMPORTANT: If you want to help me or you have the same issue discuss it here: https://github.com/vercel/pkg/discussions/1580
I'm writing a node.js app which downloads repositories from GitHub at runtime. These contain javascript files which I require at runtime. But now I want to package my code into an executable using pkg, but I read that I can't just simply require packages at runtime using pkg. So my question is: How can I require these files at runtime? I'ts important for me to access its exported function, as I need to pass variables into it. Here is an example dynamically downloaded file:
module.exports = function PluginServerSideRenderer (app, dir, config) {
/*
*
* app is the express app
* dir is the current-working-directory of the server or mostly the root
* config is the config.json file as a javascript object
*
*/
app.get('/__plugin-template__', (req, res) => {
// Uncomment if user must be logged in to access this page
// if (req.session.loggedin !== true) return res.redirect('/login') // Login Check
res.send('Hello World from the Plugin-Template!')
})
}
Normally I would use this code to import these files:
// Plugin Server Side Renderer
const plugindirx = path.join(dir, '.tvt', 'plugins')
const files = fs.readdirSync(plugindirx)
const activeplugins = JSON.parse(fs.readFileSync(path.join(plugindirx, 'active.json'), 'utf8').toString())
for (const plugindiroutoffiles of files) {
if (plugindiroutoffiles === 'active.json') continue
if (!(activeplugins[genFileName(plugindiroutoffiles)])) continue
const filesx = fs.readdirSync(path.join(plugindirx, plugindiroutoffiles))
const ssrfile = path.join(plugindirx, plugindiroutoffiles, filesx[0], 'serverside.js')
require(ssrfile)(app, dir, config, Logger, IServConfig)
}
// End Plugin Server Side Renderer

Application modularity with Vue.js and local NPM packages

I'm trying to build a modular application in Vue via the vue-cli-service. The main app and the modules are separated projects living in different folders, the structure is something like this:
-- app/package.json
/src/**
-- module1/package.json
/src**
-- module2/package.json
/src**
The idea is to have the Vue app completely agnostic about the application modules that can be there at runtime, the modules themself are compiled with vue-cli-service build --target lib in a local moduleX/dist folder, pointed with the package.json "main" and "files" nodes.
My first idea (now just for development speed purposes) was to add the modules as local NPM packages to the app, building them with a watcher and serving the app with a watcher itself, so that any change to the depending modules would (I think) be distributed automatically to the main app.
So the package.json of the app contains dependencies like:
...
"module1": "file:../module1",
"module2": "file:../module2",
...
This dependencies are mean to be removed at any time, or in general be composed as we need, the app sould just be recompiled and everything should work.
I'm trying to understand now how to dynamically load and activate the modules in the application, as I cannot use the dynamic import like this:
import(/* webpackMode: "eager" */ `module1`).then(src => {
src.default.boot();
resolve();
});
Because basically I don't know the 'module1', 'module2', etc...
In an OOP world I would just use dependency injection retrieving classes implementing a specific interface, but in JS/TS I'm not sure it is viable.
There's a way to accomplish this?

Juggling with package.json doesn't sound like a good idea to me - doesn't scale. What I would do:
Keep all available "modules" in package.json
Create separate js file (or own prop inside package.json) with all available configurations (for different clients for example)
module.exports = {
'default': ['module1', 'module2', 'module3'],
'clientA': ['module1', 'module2', 'module4'],
'clientB': ['module2', 'module3', 'module4']
}
tap into VueCLI build process - best example I found is here and create js file which will run before each build (or "serve") and using simple template (for example lodash) generate new js file which will boot configured modules based on the value of some ENV variable. See following (pseudo)code (remember this runs inside node during build):
const fs = require('fs')
const _ = require('lodash')
const modulesConfig = require(`your module config js`)
const configurationName = process.env.MY_APP_CONFIGURATION ?? 'default'
const modules = modulesConfig[configurationName]
const template = fs.loadFileSync('name of template file')
const templateCompiled = _.template(template)
const generatedJS = templateCompiled({ `modules`: modules })
fs.writeFileSync('bootModules.js', generatedJS)
Write your template for bootModules.js. Simplest would be:
<% _.forEach(modules , function(module) { %>import '<%= module %>' as <%= module %><% }); %>;
import bootModules.js into your app
Use MY_APP_CONFIGURATION ENV variable to switch desired module configuration - works not just during development but you can also setup different CI processes targeting same repo with just different MY_APP_CONFIGURATION values
This way you have all configurations at one place, you don't need to change package.json before every build, you have simple mechanism to switch between different module configurations and every build (bundle) contains only the modules needed....

In an OOP world I would just use dependency injection retrieving classes implementing a specific interface, but in JS/TS I'm not sure it is viable.
Why not?
More than this, with JS/TS you are not restricted to use classes implementing a specific interface: you just need to define the interface (i.e. the module.exports) of your modules and respecting it in the libraries entries (vue build lib).
EDIT: reading comments probably I understood the request.
Each module should respect following interface (in the file which is the entry of the vue library)
export function isMyAppModule() {
return true;
}
export function myAppInit() {
return { /* what you need to export */ };
}
Than in your app:
require("./package.json").dependencies.forEach(name => {
const module = require(name);
if(! module.isMyAppModule || module.isMyAppModule() !== true) return;
const { /* the refs you need */ } = module.myAppInit();
// use your refs as you need
});

Dynamic bundle loading at runtime

Consider a bundle in some project packed with webpack and all dependencies to bundle1.js:
// entry.ts
import * as _ from 'lodash';
// other imported libraries
import { something } from './util';
// other imported bundle project local code
export function doBundleSomething(args: any): any {
// actually does something necessary
console.log('doing things');
return 'whatever result';
}
Shell program is another project also packed with webpack and all dependencies to main.js, except bundle1.js, because it is decided at run time from environment:
// main.ts
import * as _ from 'lodash';
// other imported libraries
import { something } from './shell-util';
// other imported shell program local code
// do something useful
const ifBundleNeeded = true; // decided by useful something
if (ifBundleNeeded) {
const bundlePath = process.env.BUNDLE_PATH; // could be CLI argument
// somehow load bundle
const doBundleSomething = ???
const result = doBundleSomething({arg: 'hello-bundle'});
console.log('bundle said', result);
}
Should be executed like:
BUNDLE_PATH=/opt/path/some-bundle.js node /srv/program/path/main.js
Notes:
bare minimum possible configurations are used for everything
tsconfig targets for both bundles is es6
typescript >= 3.x with webpack awesome-typescript-loader (i.e. latest and greatest)
duplication of code in both bundles is unavoidable and acceptable
security issues are not of concern (like code injection etc.)
How to achieve this behavior with bare minimum configs?
If BUNDLE_PATH is known (i.e. hard-coding path, so that webpack sees it) at webpacking of shell program, it works in different variations with require, import() including eval("require")().
Otherwise, using different alternatives fail at different stages with various errors from Cannot find module to undefined when used.

Bundling with webpack from script

I am using webpack to bundle my Javascript files in my project:
webpack --config myconfig.webpack.config.
From commandline it is ok.
Building
However I would like to create a build task, I am using jake, so in order to create the bundle I need to invoke webpack from Javascript.
I could not find the API online, I basically need something like this:
// Jakefile.js
var webpack = require("webpack");
desc('This is the default build task which also bundles stuff.');
task('default', function (params) {
webpack.bundle("path-to-config"); // Something like this?
});
How do I achieve this?
Attempt 1
I have tried the following:
// Jakefile.js
var webpack = require("webpack");
var config = require("./webpack.config.js");
desc('This is the default build task which also bundles stuff.');
task('default', function (params) {
webpack(config);
});
webpack.config.js is my config for webpack. When I use from commandline and reference that file the bundle is correctly created. But when using the above code it does not work. When I execute it, no errors, but the bundle is not emitted.

In your Attempt 1, you seem to be consuming the webpack's Node.js API by passing the config to webpack method. If you take this approach, webpack method will return a compiler object and you need to handle it correctly.
For e.g.,
import webpack from 'webpack';
var config = {}; // Your webpack config
var wpInstanceCompiler = webpack(config);
wpInstanceCompiler.run(function(err, stats) {
if (stats.hasErrors()) {
console.log(stats.toJson("verbose");
}
});
This is how you execute a webpack config via the Node.js API. Unless you run the compiler instance, the output will not get generated.

This worked for me as well:
var webpack = require("webpack");
var lib = require(path.join(__dirname, "webpack.config.js"));
desc('Builds the projects and generates the library.');
task('default', function() {
webpack(lib, function() {
console.log("Bundle successfully created!");
});
});

Avoiding relative require paths in Node.js

I prefer to import dependencies without lots of relative filesystem navigation like ../../../foo/bar.
In the front-end I have traditionally used RequireJS to configure a default base path which enables "absolute" pathing e.g. myapp/foo/bar.
How can I achieve this in Node.js?

What you can do is use require.main.require which would always be the module path of the starting file. See https://nodejs.org/api/modules.html#modules_accessing_the_main_module
This means, that when you run
const x = require.main.require('some/file/path/y')
It would require it based on the base path of your starting file (that was invoked , node app.js meaning the app.js file).
Other options include using process.cwd() to get the starting path of your app, but that would be depending on where you invoke your node process, not the location of the file. Meaning, node app.js would be different than of you would start it one level up, node src/app.js.

Just to add to the other answer, you can put this few lines in your main file:
(function (module) {
let path = require('path')
let __require = module.constructor.prototype.require
let __cpath = __dirname
module.constructor.prototype.require = function (str) {
if (str.startsWith('package:')) {
let package = str.substr(8)
let p = path.resolve(__dirname, package)
return __require(p)
} else {
return __require(str)
}
}
})(module)
It extends your require in such way:
if path begins with package: (i.e. package:src/foo/bar) it translates it to require('/absolute/path/to/your/app/src/foo/bar'),
if it doesn't start with package: then it behaves like a normal require would.