Develop Reference

Next.js - best way to serve static JS from a node module's "dist" folder

I'm working with an application that uses Tesseract (OCR) to read text from images.
I would like to take some JS files from node_modules/tesseract.js/dist and make them downloadable in the browser.
I know I can just copy the files to ./public and next.js will serve it statically from there, but then if I update my version of Tesseract, I may need to update those files as well. So maintenance becomes a problem.
My 1st thought is to customize my webpack config to copy the files from node_modules/tesseract.js/dist to ./public/tesseract (or something like that). That would make sure the files get re-copied if I update Tesseract. But I'm not particularly savvy with webpack and haven't figured out how to do that yet.
My 2nd thought was to "proxy" the retrieval of the JS file's content and just make the content available as a "page" in next.js (but this seems super hacktastic).
I feel like this is something that shouldn't be so complicated ... but I've not been able to figure it out myself yet.
Thanks in advance!

Yup agreed, updating your server to serve a node_modules path sounds a bit dangerous.
I personally would just copy over these files with webpack like you mentioned.
Here are the docs on Next.js on how to set up a custom webpack config.
next.config.js
const CopyPlugin = require("copy-webpack-plugin");
module.exports = {
webpack: (config) => {
// append the CopyPlugin to copy the file to your public dir
config.plugins.push(
new CopyPlugin({
patterns: [
{ from: "node_modules/tesseract.js/dist", to: "public/" },
],
}),
)
// Important: return the modified config
return config
}
};
I purposefully didn't include the public/tesseract path, I'm not sure if the CopyPlugin will automatically generate missing directories if they don't exist at build time.

How to bundle react components which would be needed dynamically?

We have several applications deployed which were with plain html and js.
These applications should be completely independent modules without affecting other applications deployed.
But we should be able to load from one application to another if the situation needs which will be dynamically decided by business.
We were able to load different application screens by simply giving the relative path as there was no dependency bundling previously.
Now we are converting them to react applications with webpack as bundler.
Here we have to use dynamic import if we are going to need something dynamically. That also works on patterns which should be given at the build time.
So is there any way to achieve this kind of dynamic pattern using webpack bundler?
For importing screen components, code is something like below,
Promise.all(reqList.map(modulePath =>
{
return import(/* webpackMode: "lazy-once" */`../../../${modulePath}.jsx`)
})).then(modules => {doStuff()})
As they all are parallelly deployed applications we are trying to go back 3 folders (i.e, root folder like webapps in tomcat) and access the other application path, which will be derived dynamically in modulePath variable.
So while importing, webpack tries to import from the chunks which has already been loaded on first application launch. But this loaded chunks are not having the screens from other applications yet.
We have tried giving each jsx files as entry points in webpack which did created independent files but if we make them as entry files, they should be attached to the index.html manually, Which would cease export in jsx to work.
My wepack config is something like below,
function getEntries(pattern) {
const entries = {};
glob.sync(pattern).forEach((file) => {
let fileName = file.substr(0,file.indexOf("."));
entries[fileName.replace('src/', '')] = path.join(__dirname, file);
});
return entries;
}
let jsxFiles = getEntries('src/**/*.jsx');
console.log(Object.keys(jsxFiles));
module.exports = {
entry: jsxFiles,
output: {
path: path.resolve(__dirname, './dist'),
filename: '[name].js',
chunkFilename: "AppId"+'.js'
}
}
The first application is loading fine as webpack is able to find the bundled chunks but when we try to load different application screen components, dynamic importing fails saying module not found.
Is there any way we can achieve this kind of dynamic imports?
Thanks for going through such a long post, I didn't had a choice but to explain.. :)

Essentially, you are looking for dynamic module loading. You would need to webpack each module separately.
To load them, you have several options
Use the browser's native import(), but IE, Edge, and Firefox don't support it, yet
Use a module loader library
SystemJS
RequireJS, although it's rather old
In case of going with a module loader, you need to webpack your modules to the corresponding System.register or AMD formats

How to manage configuration for Webpack/Electron app?

I am using Webpack 2 and Electron to build nodejs application on Mac.
In my project in the root I have directory 'data' where I store configuration in a json like data/configurations/files.json (in practices there are different files with dynamic names)
After webpackaing though when I call: fs.readdirSync(remote.app.getAppPath()); to get files in the root I get only these packed: [ "default_app.js", "icon.png", "index.html", "main.js", "package.json", "renderer.js" ]
path.join(remote.app.getAppPath(), 'data/tests/groups.json'); called with FS ReadSync leads to an issue Error: ENOENT, data/tests/groups.json not found in /Users/myuser/myproject/node_modules/electron/dist/Electron.‌​app/Contents/Resourc‌​es/default_app.asar. So it seems that the whole data folder is not picked up by webpacker.
Webpack config is using json-loader and I did not find any documentation mentioning anything special about including specific files or jsons. Or do I have to reference json files in my code differently as they might be packed under main.js.
What is the best practice for Electron/Webpack for managing JSON config files? Am I doing something wrong when webpacking the project?
My project is based of https://github.com/SimulatedGREG/electron-vue using webpack/electron/vue

The Webpack Misconception
One thing to understand upfront is that webpack does not bundle files required through fs or other modules that ask for a path to a file. These type of assets are commonly labeled as Static Assets, as they are not bundled in any way. webpack will only bundle files that are required or imported (ES6). Furthermore, depending on your webpack configuration, your project root may not always match what is output within your production builds.
Based on the electron-vue documentation's Project Structure/File Tree, you will find that only webpack bundles and the static/ directory are made available in production builds. electron-vue also has a handy __static global variable that can provide a path to that static/ folder within both development and production. You can use this variable similar to how one would with __dirname and path.join to access your JSON files, or really any files.
A Solution to Static Assets
It seems the current version of the electron-vue boilerplate already has this solved for you, but I'm going to describe how this is setup with webpack as it can apply to not only JSON files and how it can also apply for any webpack + electron setup. The following solution assumes your webpack build outputs to a separate folder, which we'll use dist/ in this case, assumes your webpack configuration is located in your project's root directory, and assumes process.env.NODE_ENV is set to development during development.
The static/ directory
During development we need a place to store our static assets, so let's place them in a directory called static/. Here we can put files, such as JSONs, that we know we will need to read with fs or some other module that requires a full path to the file.
Now we need to make that static/ assets directory available in production builds.
But webpack isn't handling this folder at all, what can we do?
Let's use the simple copy-webpack-plugin. Within our webpack configuration file we can add this plugin when building for production and configure it to copy the static/ folder into our dist/ folder.
new CopyWebpackPlugin([
{
from: path.join(__dirname, '/static'),
to: path.join(__dirname, '/dist/static'),
ignore: ['.*']
}
])
Okay so the assets are in production, but how do I get a path to this folder in both development and production?
Creating a global __static variable
What's the point of making this __static variable?
Using __dirname is not reliable in webpack + electron setups. During development __dirname could be in reference to a directory that exists in your src/ files. In production, since webpack bundles our src/ files into one script, that path you formed to get to static/ doesn't exist anymore. Furthermore, those files you put inside src/ that were not required or imported never make it to your production build.
When handling the project structure differences from development and production, trying to get a path to static/ will be highly annoying during development having to always check your process.env.NODE_ENV.
So let's simplify this by creating one source of truth.
Using the webpack.DefinePlugin we can set our __static variable only in development to yield a path that points to <projectRoot>/static/. Depending if you have multiple webpack configurations, you can apply this for both a main and renderer process configuration.
new webpack.DefinePlugin({
'__static': `"${path.join(__dirname, '/static').replace(/\\/g, '\\\\')}"`
})
In production, we need to set the __static variable manually in our code. Here's what we can do...
index.html (renderer process)
<!-- Set `__static` path to static files in production -->
<script>
if (process.env.NODE_ENV !== 'development') window.__static = require('path').join(__dirname, '/static').replace(/\\/g, '\\\\')
</script>
<!-- import webpack bundle -->
main.js (main process)
// Set `__static` path to static files in production
if (process.env.NODE_ENV !== 'development') {
global.__static = require('path').join(__dirname, '/static').replace(/\\/g, '\\\\')
}
// rest of application code below
Now start using your __static variable
Let's say we have a simple JSON file we need to read with fs, here's what we can accomplish now...
static/someFile.json
{"foo":"bar"}
someScript.js (renderer or main process)
import fs from 'fs'
import path from 'path'
const someFile = fs.readFileSync(path.join(__static, '/someFile.json'), 'utf8')
console.log(JSON.parse(someFile))
// => { foo: bar }
Conclusion
webpack was made to bundle assets together that are required or imported into one nice bundle. Assets referenced with fs or other modules that need a file path are considered Static Assets, and webpack does not directly handle these. Using copy-webpack-plugin and webpack.DefinePlugin we can setup a reliable __static variable that yields a path to our static/ assets directory in both development and production.
To end, I personally haven't seen any other webpack + electron boilerplates handle this situation as it isn't a very common situation, but I think we can all agree that having one source of truth to a static assets directory is a wonderful approach to alleviate developer fatigue.

I think the confusion, (if there is any), might come from the fact that webpack not only "packs", embeds, things, code, etc... but also process content with its plugins.
html plugin being a good example, as it simply generates an html file at build-time.
And how this relates to the config file issue?,
well depending on how you are "requiring" the "config" file, what plug-in you are using to process that content.
You could be embedding it, or simply loading it as text, from file system or http, or else...
In the case of a config file, that I guess you want it to be parsed at runtime,
otherwise it's just fancy hardcoding values that perhaps you could be better simply typing it in your source code as simple objects.
And again in that case I think webpack adds little to nothing to the runtime needs, as there is nothing to pre-pack to read at later use,
so I would possibly instead or "require"it, i'll read it from the file system, with something like :
// read it parse it relative to appPath/cwd,
const config = JSON.parse(
fs.readfileSync(
path.join( app.getAppPath(), "config.json" ),
"utf-8"
))
//note: look fs-extra, it does all that minus the app.path plus async
and electron will read it from the file system , or if using Electron.require will read it from asar|fileSystem (in that order if I remember correctly, I could be wrong),

Webpack design philosophy is focused around very simple yet powerful concept:
Transform and bundle everything that is actually used by your app.
To achieve that webpack introduces a powerful concept of dependency graph, which is able to manage virtually any kind of dependencies (not only *.js modules) by the means of so-called loaders.
The purpose of a loader is to transform your dependency in a way that makes statement import smth from 'your_dependency' meaningful. For instance, json-loader calls JSON.parse(...) during loading of *.json file and returns configuration object. Therefore, in order to take advantage of webpack dependency resolution system for managing JSONs, start from installing json-loader:
$ npm install --save-dev json-loader
Then modify your webpack.config.js in the following way:
module.exports = {
...
module: {
rules: [
{test: /\.json$/, use: 'json-loader'}
]
}
...
};
At this point webpack should be able to resolve your JSON dependencies by their absolute paths, so the following should work (I assume here that you have a subdirectory config of your root context dir, containing file sample.json):
import sampleCfg from './config/sample.json';
But importing physical paths doesn't lead to elegant, robust and maintainable code (think of testability, for example), so it is considered a good practice to add aliases to your webpack.config.js for abstracting away your physical .config/ folder from your import statements
module.exports = {
...
resolve: {
alias: {
cfg: './config'
}
}
...
}
Then you'll be able to import your JSON config like that:
import sampleCfg from 'cfg/sample.json'
Finally, if you use SimulatedGREG/electron-vue Electron project template (as you mentioned in your post), then you have three webpack configuration files:
.electron-vue/webpack.web.config.js - use this config file if you use this template just for ordinary web development (i.e. not for building native Electron projects);
.electron-vue/webpack.main.config.js - use this file to configure webpack module that will run inside Electron's main process;
.electron-vue/webpack.renderer.config.js - use this file for Electron's renderer process.
You can find more information on main and renderer processes in the official Electron documentation.

How to bundle isomorphic commonJS code with webpack

I have a project that uses nodeJS module format (commonJS) and should also (in parts) run in the browser.
I do have non-isomorphic code paths where I conditionally include modules:
var impl;
try {
// in node, use node-impl
impl = require('../node-impl');
} catch (err) {
// running in browser, use browser-impl
impl = require('../browser-impl');
}
Now, I want to use webpack to create a bundle that runs in the browser. I therefore need to defined the external (nodeJS specific) modules as external in the webpack.config.js so that they don't get included in the bundle:
external: {
'../node-impl': true
}
I verified that the '../node-impl' code is actually not included in bundle, but the emitted code looks like this:
/***/ },
/* 33 */
/***/ function(module, exports) {
module.exports = ../node-impl;
/***/ },
This is syntactically wrong JS and the browser will throw a syntax error there.
How is this scenario properly handled with webpack.js? Be aware that I do not wish to use webpack for running with nodeJS, only the browser bundles should be created with webpack.

// Your actual situation:
var impl;
try {
impl = require('../node-impl');
} catch(e) {
impl = require('../browser-impl');
}
You need to refactor this snippet to:
var impl = require('../node-impl');
After this rework, your code is able to work only in a node js environment, that's is good because we are going to mock this request when bundling for browsers...
// webpack.browser.config.js
module.exports = {
resolve: {
alias: {
'../node-impl': '../browser-impl'
}
}
};
Webpack - Resolve.Alias
Or using a package.json#browser, or https://webpack.github.io/docs/configuration.html#resolve-packagealias

I don't think this is the intended purpose of the externals config. Per the docs,
Specify dependencies that shouldn’t be resolved by webpack, but should become dependencies of the resulting bundle. The kind of the dependency depends on output.libraryTarget.
So you're telling webpack that your build requires that module, but not to bundle it in. It's leaving it out, but attempting to require it.
There are probably a couple of ways to do what you want. Worth mentioning that you can easily have webpack produce multiple builds, with different/shared config, from a single config file, which opens up a lot of possibilities. But the way I'd suggest is to use the DefinePlugin to define a boolean 'constant' that represents the execution context (e.g. IN_BROWSER = true). Check that constant in your conditional around the require(). Webpack's parser isn't that smart but it can evaluate boolean variables, so it will resolve the conditional correctly and only require the appropriate module. (Using a non-boolean, like CONTEXT = 'browser' is too confusing for webpack, and it will parse each require statement.) You can then use the Uglify plugin to remove the 'dead code' in the conditional so it doesn't bloat your production build.

With the help of #Hitmands I could come up with a solution that is still not perfect, but fits my needs. I introduce a fictious nonexistingmodule and declare it as external; I then declare the node-impl specific modules to be available as nonexistingmodule:
externals: {
'nonexistingmodule': true,
'../node-impl': 'nonexistingmodule'
}
This way I can keep the try/catch pattern to load specific implementations and it will still run on node. In the browser the loading of nonexistingmodule fails, and the browser-impl module is loaded - just as I intended.
I am a big fan of "don't refactor your code to match a tool", so I am going with this solution.

Require JS files dynamically on runtime using webpack

I am trying to port a library from grunt/requirejs to webpack and stumbled upon a problem, that might be a game-breaker for this endeavor.
The library I try to port has a function, that loads and evaluates multiple modules - based on their filenames that we get from a config file - into our app. The code looks like this (coffee):
loadModules = (arrayOfFilePaths) ->
new Promise (resolve) ->
require arrayOfFilePaths, (ms...) ->
for module in ms
module ModuleAPI
resolve()
The require here needs to be called on runtime and behave like it did with requireJS. Webpack seems to only care about what happens in the "build-process".
Is this something that webpack fundamentally doesn't care about? If so, can I still use requireJS with it? What is a good solution to load assets dynamically during runtime?
edit: loadModule can load modules, that are not present on the build-time of this library. They will be provided by the app, that implements my library.

So I found that my requirement to have some files loaded on runtime, that are only available on "app-compile-time" and not on "library-compile-time" is not easily possible with webpack.
I will change the mechanism, so that my library doesn't require the files anymore, but needs to be passed the required modules. Somewhat tells me, this is gonna be the better API anyways.
edit to clarify:
Basically, instead of:
// in my library
load = (path_to_file) ->
(require path_to_file).do_something()
// in my app (using the 'compiled' libary)
cool_library.load("file_that_exists_in_my_app")
I do this:
// in my library
load = (module) ->
module.do_something()
// in my app (using the 'compiled' libary)
module = require("file_that_exists_in_my_app")
cool_library.load(module)
The first code worked in require.js but not in webpack.
In hindsight i feel its pretty wrong to have a 3rd-party-library load files at runtime anyway.

There is concept named context (http://webpack.github.io/docs/context.html), it allows to make dynamic requires.
Also there is a possibility to define code split points: http://webpack.github.io/docs/code-splitting.html
function loadInContext(filename) {
return new Promise(function(resolve){
require(['./'+filename], resolve);
})
}
function loadModules(namesInContext){
return Promise.all(namesInContext.map(loadInContext));
}
And use it like following:
loadModules(arrayOfFiles).then(function(){
modules.forEach(function(module){
module(moduleAPI);
})
});
But likely it is not what you need - you will have a lot of chunks instead of one bundle with all required modules, and likely it would not be optimal..
It is better to define module requires in you config file, and include it to your build:
// modulesConfig.js
module.exports = [
require(...),
....
]
// run.js
require('modulesConfig').forEach(function(module){
module(moduleAPI);
})

You can also try using a library such as this: https://github.com/Venryx/webpack-runtime-require
Disclaimer: I'm its developer. I wrote it because I was also frustrated with the inability to freely access module contents at runtime. (in my case, for testing from the console)