Related
I have a Vue project using TypeScript and built by Vue-CLI3.
What I'm trying to achieve is to get Webpack to build separate bundles for my workers. I've read about Webpack Code Splitting and about configureWebpack in vue.config.js, but so far had no luck in putting them together.
The project setup is the standard vue create type. I have a ./src/main.ts as the main entry point and a bunch of TypeScript modules, I want as separate bundles with their own dependency trees (I'm fine with code duplication if it can't be avoided).
I'd like to get
./dist/js/all main stuff
./dist/js/workers/worker1.6e3ebec8.js
./dist/js/workers/worker2.712f2df5.js
./dist/js/workers/worker3.83041b4b.js
So I could do new Worker(worker1.6e3ebec8.js) in the main code.
I could launch workers from the main package by generating javascript code, putting it into a blob and instantiating from that, but it looks rather awkward. Besides, my worker code import other modules, so it doesn't seem to be an option anyway.
I'm quite new to all of this, so maybe I'm not even heading in the right direction.
What is the usual way of doing that on this stack?
You can use import(), it will return a Promise and will resolve your module.
As you are using Vue-CLI 3, webpack is ready and it should split your bundle automatically.
const moduleName = 'coolModuleName'
import (
/* webpackChunkName: "[moduleName]" */
`#/my/module/path/${moduleName}.js`
).then(moduleCode => {
// use your module
})
// load them in parallel
const getModuleDynamically(path, moduleName) => import(
/* webpackChunkName: "[moduleName]" */
`#/${path}/${moduleName}.js`
)
Promise.all([
getModuleDynamically(path, moduleName1),
getModuleDynamically(path, moduleName2),
getModuleDynamically(path, moduleName3)
])
Got there! #aquilesb's answer did help, although I've failed to get getModuleDynamically() from the answer working after plenty of experimenting.
Update: Looks like with this solution I'm not able to use imports of npm modules. I've tried experimenting with worker-loader but haven't got anywhere so far.
Here are a few takeaways:
Create a separate webpack config for packing workers. The target: 'webworker' must be there. Call it with webpack --config ./webpack.config.workers.js, as Vue wouldn't know about that.
Create a separate tsconfig.json for workers TypeScript. The lib setting for workers must be there: "lib": ["esnext","webworker","scripthost"] as well as the proper include:[...]/exclude:[...] settings.
You may need to tell Vue to use the main tsconfig.json that has it's own "lib":["esnext","dom","dom.iterable","scripthost"] and include/exclude. This is done in vue.config.js, you will probably need to create it. I use chainWebpack configuration option of Vue config.
Let Webpack know you have dynamic loading by making calls to import() with static (i.e. not variable) names. I haven't found a way to do so in config file, but it doesn't matter: you can't help hardcoding the names somewhere, how else Webpack would know it has to bundle and split the code?
Somehow get the name(s) of generated files, as you must have at least one of them at runtime to do new Worker(filename). I used the --json option of Webpack CLI for that.
There are many ways all of this can be achieved. This is what this ended up looking like in my project:
Folder structure:
webpack.config.workers.js
vue.config.js
tsconfig.base.json
src/main/
src/main/tsconfig.json -- extends tsconfig.base.json
src/shared/ -- this code may be duplicated by the Vue app bundles and by workers bundle
src/workers/
src/workers/tsconfig.json -- extends tsconfig.base.json
webpack.config.workers.js: contains a single entry – the main worker file, that loads the other stuff.
entry: {
worker: './src/workers/worker.ts'
}
build.workers.sh: the script calls Webpack CLI and produces a JSON file with the resulting workers names (trivial actions on folders are omitted). The only one I need is called "worker". The rest is to be dynamically loaded by it.
#!/bin/bash
# Map entry name -> bundle file name
# "assetsByChunkName":{"entryN":"entryN.[hash].js", ...}
json=$(webpack --config ./webpack.config.workers.js --json $#|tr -d "\n\r\t "|grep -Eo '"assetsByChunkName":.+?}')
# Remove "assetsByChunkName"
json=$(echo "${json:20}")
echo $json
echo $json > "$target/$folder/workers.json"
Load workers.json at runtime. The other option would be to use it at compile time by providing Vue config with const VUE_APP_MAIN_WORKER = require("path to/workers.json").worker and using this env constant.
Now that we have the name of the main worker file, we can do new Worker("main worker file path we've got from webpack").
The main worker file contains the function that statically references other modules and dynamically loads them. This way Webpack knows what to bundle and how to split the code.
enum WorkerName {
sodium = "sodium",
socket = "socket"
}
function importModule(name: WorkerName): Promise<any> {
switch (name) {
case WorkerName.sodium:
return import(
/* webpackChunkName: "sodium" */
"workers/sodium"
);
case WorkerName.socket:
return import(
/* webpackChunkName: "socket" */
"workers/socket"
);
}
}
Use the postMessage/message event API to tell your main worker code what to load.
const messageHandler = (e: MessageEvent) => {
// here goes app-specific implementation of events
// that gets you the moduleName in the end
importModule(moduleName).then((work) => {
// do smth
});
};
Now, to the correct answer.
To achieve the following:
Using webworkers
Using both dynamic imports and normal imports in webworker code
Sharing code between webworkers and main app
I had to add a separate rule for worker-loader in vue.config.js and also to add babel-loader. It took me some time to find the correct solution, but I dropped the previous one (in my other answer) in the end. I still use separate tsconfig.js for main and for webworkers.
What I'm still not happy with, is that vue-cli–or rather fork-ts-checker plugin–doesn't seem to know the webworker-specific types in my worker classes (so I can't use DedicatedWorkerScope, for instance).
Scenario
Consider the following directory structure
/views
/one.js
/two.js
/components
/Header.js
/Footer.js
/other
...
I need to transpile the /views directory with Babel. I am doing it programmatically but the equivalent cli command would look like something like this:
babel views --out-dir .views
But files in the /views directory depend on files in other directories, i.e.
/*
views/one.js
*/
const Header = require('../components/Header');
...
Given this dependency, I also need /components/Header.js to be transpiled.
Question
How can I use Babel, possibly with another library, to only transpile the specified directory and any recursive dependencies without transpiling the entire codebase (e.g. /other, etc.)?
There are probably multiple options for getting dependency graphs in Node. You can use node-dependency-tree to get the dependency graph you want to transpile with babel. Madge would be another possible option (uses dependency-tree internally).
How does it work?
Dependency tree takes in a starting file, extracts its declared dependencies via precinct, resolves each of those dependencies to a file on the filesystem via filing-cabinet, then recursively performs those steps until there are no more dependencies to process.
When using dependency-tree the toList method
Returns a post-order traversal (list form) of the tree with duplicate sub-trees pruned.
This is useful for bundling source files, because the list gives the concatenation order.
Note: you can pass the same arguments as you would to dependencyTree()
The two arguments used by toList are
filename The path to the file containing the initial dependency, i.e. the dependency graph's entry point. This is used by the module filing-cabinet.
directory The path to all the files, i.e. the root for resolving dependencies. This is also used by the module filing-cabinet.
This build script is using CommonJS modules, Node v16.16.0, and Babel 7.19.3. I'm sure you can convert this to ES modules with no issues, or be Promise based if you prefer.
build.js
const fs = require('node:fs')
const { dirname, resolve } = require('node:path')
const dependencyTree = require('dependency-tree')
const babel = require('#babel/core')
dependencyTree.toList({
filename: './views/one.js',
directory: './views'
}).forEach(path => {
const basepath = path.split('views/')
const filepath = resolve('./.views', basepath[1])
const directory = dirname(filepath)
fs.mkdirSync(directory, { recursive: true })
fs.writeFileSync(
filepath,
babel.transformSync(fs.readFileSync(path).toString()).code
)
})
node build.js
I am using Webpack 2 and Electron to build nodejs application on Mac.
In my project in the root I have directory 'data' where I store configuration in a json like data/configurations/files.json (in practices there are different files with dynamic names)
After webpackaing though when I call: fs.readdirSync(remote.app.getAppPath()); to get files in the root I get only these packed: [ "default_app.js", "icon.png", "index.html", "main.js", "package.json", "renderer.js" ]
path.join(remote.app.getAppPath(), 'data/tests/groups.json'); called with FS ReadSync leads to an issue Error: ENOENT, data/tests/groups.json not found in /Users/myuser/myproject/node_modules/electron/dist/Electron.app/Contents/Resources/default_app.asar. So it seems that the whole data folder is not picked up by webpacker.
Webpack config is using json-loader and I did not find any documentation mentioning anything special about including specific files or jsons. Or do I have to reference json files in my code differently as they might be packed under main.js.
What is the best practice for Electron/Webpack for managing JSON config files? Am I doing something wrong when webpacking the project?
My project is based of https://github.com/SimulatedGREG/electron-vue using webpack/electron/vue
The Webpack Misconception
One thing to understand upfront is that webpack does not bundle files required through fs or other modules that ask for a path to a file. These type of assets are commonly labeled as Static Assets, as they are not bundled in any way. webpack will only bundle files that are required or imported (ES6). Furthermore, depending on your webpack configuration, your project root may not always match what is output within your production builds.
Based on the electron-vue documentation's Project Structure/File Tree, you will find that only webpack bundles and the static/ directory are made available in production builds. electron-vue also has a handy __static global variable that can provide a path to that static/ folder within both development and production. You can use this variable similar to how one would with __dirname and path.join to access your JSON files, or really any files.
A Solution to Static Assets
It seems the current version of the electron-vue boilerplate already has this solved for you, but I'm going to describe how this is setup with webpack as it can apply to not only JSON files and how it can also apply for any webpack + electron setup. The following solution assumes your webpack build outputs to a separate folder, which we'll use dist/ in this case, assumes your webpack configuration is located in your project's root directory, and assumes process.env.NODE_ENV is set to development during development.
The static/ directory
During development we need a place to store our static assets, so let's place them in a directory called static/. Here we can put files, such as JSONs, that we know we will need to read with fs or some other module that requires a full path to the file.
Now we need to make that static/ assets directory available in production builds.
But webpack isn't handling this folder at all, what can we do?
Let's use the simple copy-webpack-plugin. Within our webpack configuration file we can add this plugin when building for production and configure it to copy the static/ folder into our dist/ folder.
new CopyWebpackPlugin([
{
from: path.join(__dirname, '/static'),
to: path.join(__dirname, '/dist/static'),
ignore: ['.*']
}
])
Okay so the assets are in production, but how do I get a path to this folder in both development and production?
Creating a global __static variable
What's the point of making this __static variable?
Using __dirname is not reliable in webpack + electron setups. During development __dirname could be in reference to a directory that exists in your src/ files. In production, since webpack bundles our src/ files into one script, that path you formed to get to static/ doesn't exist anymore. Furthermore, those files you put inside src/ that were not required or imported never make it to your production build.
When handling the project structure differences from development and production, trying to get a path to static/ will be highly annoying during development having to always check your process.env.NODE_ENV.
So let's simplify this by creating one source of truth.
Using the webpack.DefinePlugin we can set our __static variable only in development to yield a path that points to <projectRoot>/static/. Depending if you have multiple webpack configurations, you can apply this for both a main and renderer process configuration.
new webpack.DefinePlugin({
'__static': `"${path.join(__dirname, '/static').replace(/\\/g, '\\\\')}"`
})
In production, we need to set the __static variable manually in our code. Here's what we can do...
index.html (renderer process)
<!-- Set `__static` path to static files in production -->
<script>
if (process.env.NODE_ENV !== 'development') window.__static = require('path').join(__dirname, '/static').replace(/\\/g, '\\\\')
</script>
<!-- import webpack bundle -->
main.js (main process)
// Set `__static` path to static files in production
if (process.env.NODE_ENV !== 'development') {
global.__static = require('path').join(__dirname, '/static').replace(/\\/g, '\\\\')
}
// rest of application code below
Now start using your __static variable
Let's say we have a simple JSON file we need to read with fs, here's what we can accomplish now...
static/someFile.json
{"foo":"bar"}
someScript.js (renderer or main process)
import fs from 'fs'
import path from 'path'
const someFile = fs.readFileSync(path.join(__static, '/someFile.json'), 'utf8')
console.log(JSON.parse(someFile))
// => { foo: bar }
Conclusion
webpack was made to bundle assets together that are required or imported into one nice bundle. Assets referenced with fs or other modules that need a file path are considered Static Assets, and webpack does not directly handle these. Using copy-webpack-plugin and webpack.DefinePlugin we can setup a reliable __static variable that yields a path to our static/ assets directory in both development and production.
To end, I personally haven't seen any other webpack + electron boilerplates handle this situation as it isn't a very common situation, but I think we can all agree that having one source of truth to a static assets directory is a wonderful approach to alleviate developer fatigue.
I think the confusion, (if there is any), might come from the fact that webpack not only "packs", embeds, things, code, etc... but also process content with its plugins.
html plugin being a good example, as it simply generates an html file at build-time.
And how this relates to the config file issue?,
well depending on how you are "requiring" the "config" file, what plug-in you are using to process that content.
You could be embedding it, or simply loading it as text, from file system or http, or else...
In the case of a config file, that I guess you want it to be parsed at runtime,
otherwise it's just fancy hardcoding values that perhaps you could be better simply typing it in your source code as simple objects.
And again in that case I think webpack adds little to nothing to the runtime needs, as there is nothing to pre-pack to read at later use,
so I would possibly instead or "require"it, i'll read it from the file system, with something like :
// read it parse it relative to appPath/cwd,
const config = JSON.parse(
fs.readfileSync(
path.join( app.getAppPath(), "config.json" ),
"utf-8"
))
//note: look fs-extra, it does all that minus the app.path plus async
and electron will read it from the file system , or if using Electron.require will read it from asar|fileSystem (in that order if I remember correctly, I could be wrong),
Webpack design philosophy is focused around very simple yet powerful concept:
Transform and bundle everything that is actually used by your app.
To achieve that webpack introduces a powerful concept of dependency graph, which is able to manage virtually any kind of dependencies (not only *.js modules) by the means of so-called loaders.
The purpose of a loader is to transform your dependency in a way that makes statement import smth from 'your_dependency' meaningful. For instance, json-loader calls JSON.parse(...) during loading of *.json file and returns configuration object. Therefore, in order to take advantage of webpack dependency resolution system for managing JSONs, start from installing json-loader:
$ npm install --save-dev json-loader
Then modify your webpack.config.js in the following way:
module.exports = {
...
module: {
rules: [
{test: /\.json$/, use: 'json-loader'}
]
}
...
};
At this point webpack should be able to resolve your JSON dependencies by their absolute paths, so the following should work (I assume here that you have a subdirectory config of your root context dir, containing file sample.json):
import sampleCfg from './config/sample.json';
But importing physical paths doesn't lead to elegant, robust and maintainable code (think of testability, for example), so it is considered a good practice to add aliases to your webpack.config.js for abstracting away your physical .config/ folder from your import statements
module.exports = {
...
resolve: {
alias: {
cfg: './config'
}
}
...
}
Then you'll be able to import your JSON config like that:
import sampleCfg from 'cfg/sample.json'
Finally, if you use SimulatedGREG/electron-vue Electron project template (as you mentioned in your post), then you have three webpack configuration files:
.electron-vue/webpack.web.config.js - use this config file if you use this template just for ordinary web development (i.e. not for building native Electron projects);
.electron-vue/webpack.main.config.js - use this file to configure webpack module that will run inside Electron's main process;
.electron-vue/webpack.renderer.config.js - use this file for Electron's renderer process.
You can find more information on main and renderer processes in the official Electron documentation.
I've been using Gulp for a while now. Then I was introduced to Browserify. Now Webpack is on my horizon given I'm moving towards React. I digress...
I'd like to achieve these steps:
require front end dependencies, such as jQuery and Backbone, in a node application so as to have one source of truth - npm.
Concatenate those dependencies (in whatever order I choose) into a dependencies.js file suitable for the browser (Browserify, right?).
Concatenate the above file with my own JavaScript files (in whatever order I choose) into a all.min.js file, minified, wrapped in a self executing anonymous function - (function(){ /*all code here*/})() - so as to avoid any global variables / variables on the window object / global pollution. (<-- this one is the key).
I'd love to be able to handle this all in just Gulp as I'm used to it, but the whole global pollution thing is killing me. Take a look at the gulp file:
var gulp = require('gulp');
var concat = require('gulp-concat-util'); // Makes concat.header, concat.footer available.
var uglify = require('gulp-uglify');
var browserify = require('browserify');
var source = require('vinyl-source-stream');
var buffer = require('vinyl-buffer');
gulp.task('browserify', function() {
return browserify('libraries.js') // File containing a list of requires's.
.bundle()
.pipe(source('dependencies.js')) // Concatenated file.
.pipe(buffer())
.pipe(gulp.dest('./dev/dependencies')); // Destination directory.
});
gulp.task('scripts', function() {
return gulp.src([
'dev/dependencies/dependencies.js',
'dev/js/models/**/*.js', // Models before views & collections.
'dev/js/**/!(app|templates)*.js',
'dev/js/templates.js',
'dev/js/app.js'
])
.pipe(concat('all.min.js')) // Final file.
.pipe(concat.header('(function(){')) // Attempts to wrap in a SEAF.
.pipe(concat.footer('\n})();'))
.pipe(uglify())
.pipe(gulp.dest('public'));
});
All of that code will indeed produce a single file, minified, wrapped in a SEAF, but the libraries I required (say Backbone & jQuery) are still accessible in the global scope. My life! Is this just the way it works? Those libraries are attaching themselves to window (I looked in the code) but I thought maybe some Browserify magic could deal with that. Any thoughts are appreciated!
A lot of modules will use UMD (Universal Module Definition) patterns. One of the patterns is to always declare the module as a global resource so that other libraries that do not use a module loader will have the library available globally. Having single libraries globally registered is not terribly bad as long as they do not collide. Since you are wrapping your app code in an IIFE and hopefully not adding to the window object directly the globals should be limited to just the 3rd party libraries.
I'm using webpack for processing js and css and I'm pretty happy with it.
All library dependencies are installed by npm install and saved in package.json. In the result I have bunch of minified files that contains all my app. JS modules are stored in function scopes, so, no global scope is spoiled. Easy to maintain, easy to understand and track all dependencies used in each module.
I'll give you some basic example.
File: webpack.config.js - configuration for webpack
module.exports = {
// entrypoint for webpack to start processing
entry: {
bundle: './js/main.js'
},
output: {
path: __dirname + '/' + 'assets',
filename: '[name].js',
publicPath : '/assets/',
}
};
File: ./js/main.js - entrypoint for webpack
// This is require from node_modules
var $ = require('jquery');
// this is require for my local module
var MyModule1 = require('./modules/module1.js');
File: ./modules/module1.js - local module
var OtherLib = require('other-lib');
...
// exporting some data for others
module.exports = {
// export object
};
Pros:
One tool for everything
Easy dependency resolving and installing
Bunch of plugins/loaders from community https://webpack.github.io/docs/list-of-loaders.html
Easy to use from the box
Cons that I faced:
webpack is JS preprocessor, so, if you need to have processed CSS as separate file (not in minified JS), you need to extract CSS from JS result. So, I'm using ExtractTextPlugin.
It quite slow while processing lots of SASS files in --watch mode.
I'm working on an application that needs to pull in the ReadiumJS library, which uses AMD modules. The app itself is written in es6 w/ webpack and babel. I've gotten the vendor bundle working correctly, and it's pulling in the built Readium file, but when I try to require any of the modules Webpack says it can't resolve them. Anyone ever do this before with Webpack and RequireJS? Here's some info that may help - not sure what else to include as this is my first time really using Webpack..
Folder Structure
/readium-src
/readium-js
/ *** all readium-specific files and build output (have to pull down repo and build locally)
/node_modules
/src
/app.js -> main entry for my app
/webpack.config.babel.js
webpack.config.js entries
entry: {
vendorJs: [
'jquery',
'angular',
'../readium-src/readium-js/build-output/_single-bundle/readium-js_all.js',
'bootstrap/js/alert.js' //bootstrap js example
],
appJs: './app.js'
}
Trying to require it in app.js
var readiumSharedGlobals = require('readium_shared_js/globals');
I never really got into using RequireJS, so really struggling to understand how to consume that type of module along side other types of modules with webpack. Any help greatly appreciated :)
Update
If I change my app.js to use this instead:
window.rqReadium = require('../readium-src/readium-js/build-output/_single-bundle/readium-js_all.js');
Then it appears to try to load all the modules, but I get a strange error:
Uncaught Error: No IPv6
At this point, I'm unsure of
Should I have to require the entire path like that?
Is this error something from webpack, requirejs, or Readium? Tried debugging, but couldn't find anything useful...
UPDATE 8/12/2016
I think this is related to an issue with a library that Readium is depending on: https://github.com/medialize/URI.js/issues/118
However, I'm still not clear on how to correctly import AMD modules with webpack. Here's what I mean:
Let's say I have an amd module defined in moneyService.amd.js like this:
define('myMoneyService', ['jquery'], function($) {
//contrived simple example...
return function getDollaz() { console.log('$$$'); }
});
Then, in a sibling file, app.js, I want to pull in that file.
//this works
var getDollaz = require('./moneyService.amd.js');
//this works
require(['./moneyService.amd.js'], function(getDollaz) { getDollaz(); }
//this does not
require(['myMoneyService' /*require by its ID vs file name*/], function(getDollaz) {
getDollaz();
}
So, if we cannot require named modules, how would we work with a third party lib's dist file that has all the modules bundled into a single file?
Ok, so there's a repo out there for an Electron ePub reader using Readium, and it's using webpack: https://github.com/clebeaupin/readium-electron This shows a great way to handle pulling in RequireJS modules with webpack.
One super awesome thing I found is that you can specify output.library and output.libraryTarget and webpack will transpose from one module format to another... freaking awesome! So, I can import the requirejs module, set output library and libraryTarget to 'readium-js' and 'commonjs2' respectively, then inside my application code I can do import Readium from 'readium-js';