Using Webpack, I have multiple chunks/bundles being created so that the entire app is not loaded at once. I've hand-chosen which dependencies I want to be moved into their own chunks. Here is the important part of my config:
module.exports = {
devtool: 'inline-source-map',
mode: process.env.NODE_ENV,
entry: {
main: './src/index.tsx',
},
optimization: {
runtimeChunk: {
name: 'runtime',
},
splitChunks: {
cacheGroups: {
...makeChunkCacheGroup('chunk_1', /\/node_modules\/(... list of deps ...)(\/|$)/),
...makeChunkCacheGroup('chunk_2', /\/node_modules\/(... list of deps ...)(\/|$)/),
},
},
},
// ...
};
function makeChunkCacheGroup(name, ...moduleNameRegexps) {
return {
[name]: {
name,
test: module => moduleNameRegexps.some(pattern => pattern.test(module.context)),
chunks: 'all',
minChunks: 1,
minSize: 0,
maxSize: Infinity,
reuseExistingChunk: true,
enforce: true,
},
};
}
This config gives me runtime, main, chunk_, and chunk_2. However, all of these chunks are injected into index.html, thus they all load during the initial page load instead of dynamically (as I naively expected).
I've used SystemJS in the past to bundle things up into multiple bundles and it would only download a given bundle as it was required by the app. I now realize that Webpack does not work this way.
Is there a way to make Webpack only download the runtime and main bundles initially, and then download the other bundles as they're needed?
Note 1: I realize that I can use dynamic imports e.g. import('some-dep').then(...), but it's not reasonable to do so based on the size of the codebase, and also, I think this sort of thing is better left to configuration (a module shouldn't have to pick and choose which deps it should load dynamically).
Note 2: I did try to specify multiple entry points but never got it working. The app really only has a single entry point. But, for instance, we have multiple directories under src/app/elements/, and it'd be perfect if each of those directories ended up in its own bundle which was then dynamically loaded. I couldn't get this working in an automated/smart way.
Related
I have a Gatsby site that consumes a number of packages. One of those packages is published from our monorepo: #example/forms. That package contains a number of named exports, one for each form component that we use on our site. There are quite a large number of forms and some are relatively complex multistep forms of a non-trivial size.
Currently, Gatsby/Webpack does a good job of treeshaking, and does produce a large number of bundles, some common bundles and one for each page of the site, containing any local assets or components that are only used on that page. However, all the components from #example/forms are being added to the commons bundle, despite the fact that most are used on only a single page. This is causing unnecessary bloat of the commons bundle which is loaded on every page.
If feels like it should be possible for the individual components within #example/forms to be split out into the page-specific bundles, but I'm not sure if I'm hoping for too much. For example, if Form A is only used on page 4, I'd like Form A to only be added to the bundle for page 4.
Is this something that is supported, and if so, what could be preventing this from happening.
#example/forms is transpiled with ES6 exports left intact, and sideEffects is set to false in its package.json.
Its main file is index.js which (re)exports all the form components from their own files as separate named exports:
export {default as FormA} from './forms/formA'
export {default as FormB} from './forms/formB'
...
Another thing that might be relevant is that all the exports from #example/forms are used within the Gatsby site, just on separate pages. It appears that perhaps tree-shaking cannot be used across bundles, i.e. tree shaking is performed first, then what is left is split into bundles. Using that logic, #example/forms would have been used on multiple pages and would be moved to commons. However this is definitely not optimal, and hopefully isn't what is happening.
Tree-shaking process is part of the minification step which is really at late stage, since that it is not possible to reverse the order, first tree-shake then chunk split.
But you can split your forms lib before hand using some heuristics
The basic one is to use splitChunks in order to split this #example/forms module into a separate chunk as whole, this will decrease the commons bloat & on the first form usage all forms will be loaded.
// webpack.config.js
module.exports = {
...
optimization: {
splitChunks: {
cacheGroups: {
formComponents: {
chunks: 'all',
enforce: true,
minChunks: 1,
name: 'form-components',
priority: 10,
test: /[\\\/]node_modules[\\\/]#example[\\\/]forms[\\\/]/,
},
},
},
},
...
};
Make a chunk for each form inside the #example/forms module based on some heuristics, in my example I'm "grouping" all items based on the form path.
// webpack.config.js
module.exports = {
...
optimization: {
splitChunks: {
cacheGroups: {
formComponents: {
chunks: 'all',
minChunks: 1,
name(module) {
const libPath = path.resolve(path.join('node_modules/#example/forms'))
const folderBasedChunk = path.relative(libPath, module.context);
const hash = crypto.createHash('sha1');
hash.update(folderBasedChunk)
return hash.digest('hex').substring(0, 8)
},
priority: 10,
reuseExistingChunk: true,
enforce: true,
test: /[\\\/]node_modules[\\\/]#example[\\\/]forms[\\\/]src[\\\/]forms[\\\/]/,
},
},
},
...
};
You can checkout a small example that I've created that mimics the scenario.
https://github.com/felixmosh/webpack-module-split-example
Hope this helps
I am building an angular app, which contains a single component which depends on a library in my node_modules. I would like to create a single chunk specifically for the dependency so that my users are able to cache it, as the component may change regularly, but the dependency will only be updated every few weeks (whenever it has an update).
I have tried various splits in my webpack config, which always resulted in either all node_modules in a single chunk, or the the component and its dependency together in one chunk.
Is there a way to configure webpack to always split a vendor into its own chunk if it is not loaded initially?
I achieved the desired effect with the following config:
optimization: {
splitChunks: {
cacheGroups: {
initVendors: {
chunks: 'initial',
test: /[\\/]node_modules[\\/]/,
},
asyncVendors: {
chunks: 'async',
test: /[\\/]node_modules[\\/]/,
},
},
},
},
initVendors
With chunks: 'initial' we advise webpack to only group modules that are required for the initial pageload.
initVendors: {
chunks: 'initial',
test: /[\\/]node_modules[\\/]/,
}
asyncVendors
With chunks: 'async' we advise wepack to only group modules that can be loaded into the page async, while browsing, for features that are not required on pageload.
This will also place these modules into their own chunks, instead of bundling them with the code in your application that is using them. This allows long term caching of a chunk.
asyncVendors: {
chunks: 'async',
test: /[\\/]node_modules[\\/]/,
},
using webpack v2, i have two entry points which should generate two bundles that can be loaded on the same page or different pages separately. I'm also using the CommonsChunkPlugin to extract all dependencies from the node_modules folder to a 3rd bundle which is only needed by the first bundle. When loading the second bundle alone on a page, it does not get initialized, because it seems to need the commonChucnks bundle to have been loaded before (hosting the webpackJsonp function) is there an option other than having two webpack configs to solve my problem?
module.exports = {
context: path.resolve(__dirname),
entry: {
"first": "./src/js/first.js",
"second": "./src/js/second.js",
},
output: {
path: path.resolve(__dirname, "target/js"),
filename: "[name].js",
libraryTarget: "umd"
},
plugins: [
new webpack.optimize.CommonsChunkPlugin({
names: "vendor",
// automatically determine which files are from the node modules. And put them in a separate "lib" bundle
// #see https://webpack.js.org/guides/code-splitting-libraries/#commonschunkplugin
// this assumes your vendor imports exist in the node_modules directory
minChunks: module => module.context && module.context.indexOf("node_modules") !== -1
}),
"(two entrypoints)... that can be loaded on the same page".
This is not how it supposed to be. Entry point - is an entry point - it can be only once loaded.
What you need is: two entrypoints chunks, with "specific" code chunks" and one common code chunk.
I'm learning React and want to understand how web pack is configured for a project.
It would be great if someone can tell me what the following lines of code are doing.
const fs = require('fs')
const path = require('path')
const webpack = require('webpack')
function isDirectory(dir) {
return fs.lstatSync(dir).isDirectory()
}
const SubjectsDir = path.join(__dirname, 'subjects')
const SubjectDirs = fs.readdirSync(SubjectsDir).filter(function (dir) {
return isDirectory(path.join(SubjectsDir, dir))
})
module.exports = {
devtool: 'source-map',
entry: SubjectDirs.reduce(function (entries, dir) {
if (fs.existsSync(path.join(SubjectsDir, dir, 'exercise.js')))
entries[dir + '-exercise'] = path.join(SubjectsDir, dir, 'exercise.js')
if (fs.existsSync(path.join(SubjectsDir, dir, 'solution.js')))
entries[dir + '-solution'] = path.join(SubjectsDir, dir, 'solution.js')
if (fs.existsSync(path.join(SubjectsDir, dir, 'lecture.js')))
entries[dir + '-lecture'] = path.join(SubjectsDir, dir, 'lecture.js')
return entries
}, {
shared: [ 'react', 'react-dom' ]
}),
output: {
path: '__build__',
filename: '[name].js',
chunkFilename: '[id].chunk.js',
publicPath: '__build__'
},
resolve: {
extensions: [ '', '.js', '.css' ]
},
module: {
loaders: [
{ test: /\.css$/, loader: 'style!css' },
{ test: /\.js$/, exclude: /node_modules|mocha-browser\.js/, loader: 'babel' },
{ test: /\.woff(2)?$/, loader: 'url?limit=10000&mimetype=application/font-woff' },
{ test: /\.ttf$/, loader: 'file' },
{ test: /\.eot$/, loader: 'file' },
{ test: /\.svg$/, loader: 'file' },
{ test: require.resolve('jquery'), loader: 'expose?jQuery' }
]
},
plugins: [
new webpack.optimize.CommonsChunkPlugin({ name: 'shared' })
],
devServer: {
quiet: false,
noInfo: false,
historyApiFallback: {
rewrites: [
{ from: /ReduxDataFlow\/exercise.html/,
to: 'ReduxDataFlow\/exercise.html'
}
]
},
stats: {
// Config for minimal console.log mess.
assets: true,
colors: true,
version: true,
hash: true,
timings: true,
chunks: false,
chunkModules: false
}
}
}
This informaiton is coming from a training course, but they do not explain what the lines are doing.
Webpack is what we call a module bundler for JavaScript applications. You can do a whole slew of things with it that help a client browser download and run your code. In the case of React, it helps convert JSX code into plain JS so that the browser can understand it. JSX itself will not run in the browser. We can even use plugins to help minify code, inject HTML, bundle various groups of code together, etc. Now that the introduction to Webpack is out of the way, let's take a look at the code. I will be starting from the very top. Feel free to skip down to #3 if you are only interested in the Webpack configuration object.
The following code will require the modules that are needed in this file. fs is short for "filesystem" and is a module that gives you functions to run that can access the project's filesystem. path is a common module used to resolve or create pathnames to files and is very easy to use! And then we have the webpack module through which we can access webpack specific functions (ie: webpack plugins like webpack.optimize.UglifyJsPlugin).
const fs = require('fs')
const path = require('path')
const webpack = require('webpack')
These next few lines first set up a helper function to determine whether or not something in the filesystem being read is a directory.
function isDirectory(dir) {
return fs.lstatSync(dir).isDirectory()
}
const SubjectsDir = path.join(__dirname, 'subjects')
const SubjectDirs = fs.readdirSync(SubjectsDir).filter(function (dir) {
return isDirectory(path.join(SubjectsDir, dir))
})
devtool specifies which developer tool you want to use to help with debugging. Options are listed here : https://webpack.github.io/docs/configuration.html#devtool. This can be very useful in helping you determine exactly which files and lines and columns errors are coming from.
devtool: 'source-map'
These next few lines tell Webpack where to begin bundling your files. These initial files are called entry points. The entry property in a Webpack configuration object should be an object whose keys determine the name of a bundle and values point to a relative path to the entry file or the name of a node_module. You can also pass in an array of files to each entry point. This will cause each of those files to be bundled together into one file under the name specified by the key - ie: react and react-dom will each be parsed and have their outputs bundled under the name shared.
entry: SubjectDirs.reduce(function (entries, dir) {
if (fs.existsSync(path.join(SubjectsDir, dir, 'exercise.js')))
entries[dir + '-exercise'] = path.join(SubjectsDir, dir, 'exercise.js')
if (fs.existsSync(path.join(SubjectsDir, dir, 'solution.js')))
entries[dir + '-solution'] = path.join(SubjectsDir, dir, 'solution.js')
if (fs.existsSync(path.join(SubjectsDir, dir, 'lecture.js')))
entries[dir + '-lecture'] = path.join(SubjectsDir, dir, 'lecture.js')
return entries
}, {
shared: [ 'react', 'react-dom' ]
}),
In the reduce function we simply read through the SubjectsDir, determine whether files exercise.js, lecture.js & solution.js exist, and then provide the path to those files as values associated with the key names identified by dir + '-' + filename (ie: myDir-exercise). This may end up looking like the following if only exercise.js exists:
entry : {
'myDir-exercise': 'subjectDir/myDir/exercise.js',
share: ['react', 'react-dom']
}
After we provide entry points to the Webpack configuration object, we must specify where we want Webpack to output the result of bundling those files. This can be specified in the output property.
output: {
path: '__build__',
filename: '[name].js',
chunkFilename: '[id].chunk.js',
publicPath: '__build__'
},
The path property defines the absolute path to the output directory. In this case we call it __build__.
The filename property defines the output name of each entry point file. Webpack understands that by specifying '[name]' you are referring to the key you assigned to each entry point in the entry property (ie: shared or myDir-exercise).
The chunkFilename property is similar to the filename property but for non-entry chunk files which can be specified by the CommonChunksPlugin (see below). The use of [id] is similar to the use of [name].
The publicPath property defines the public URL to where your files are located, as in the URL from which to access your files through a browser.
The resolve property tells Webpack what how to resolve your files if it can not find them for some reason. There are several properties we can pass here with extensions being one of them. The extensions property tells Webpack which file extensions to try on a file if one is not specified in your code.
resolve: {
extensions: [ '', '.js', '.css' ]
},
For example, let's say we have the following code
const exercise = require('./exercise');
We can leave out the .js because we have provided that string in the resolve property of the webpack configuration and Webpack will try and append .js to this at bundling time to find your file. As of Webpack 2 we also no longer need to specify an empty string as the first element of the resolve property.
The module property tells Webpack how modules within our project will be treated. There are several properties we can add here and I suggest taking a look at the documentation for more details. loaders is a common property to use and with that we can tell Webpack how to parse particular file types within our project. The test property in each loader is simply a Regex that tells Webpack which files to run this loader on. This /\.js$/ for example will run the specified loader on files that end with .js. babel-loader is a widely used JavaScript + ES6 loader. The exclude property tells Webpack which files to not run with the specified loader. The loader property is the name of the loader. As of Webpack 2 we are no longer able to drop the -loader from the string as we see here.
Plugins have a wide range of functions. As mentioned earlier we can use plugins to help minify code or to build chunks that are used across our entire application, like react and react-dom. Here we see the CommonChunksPlugin being used which will bundle the files under the entry name shared together as a chunk so that they can be separated from the rest of the application.
Finally we have the devServer property which specifies certain configurations for the behavior of webpack-dev-server, which is a separate module from webpack. This module can be useful for development in that you can opt out of building your own web server and allow webpack-dev-server to serve your static files. It also does not write the outputs to your filesystem and serves the bundle from a location in memory at the path specified by the publicPath property in the output property of the Webpack configuration object (see #5). This helps make development faster. To use it you would simply run webpack-dev-server in place of webpack. Take a look at the documentation online for more details.
The configuration object that we've taken a look at follows the Webpack 1 standard. There are many changes between Webpack 1 and 2 in terms of configuration syntax that would be important to take a look at. The concepts are still the same, however. Take a look at the documentation for information on migrating to Webpack 2 and for further Webpack 2 details.
I'm trying out the multi-compiler option in webpack and am following the example at their github. However, I can't seem to understand how I can split out the common code amongst the multiple configurations.
For example, I may have same vendor libraries used in the different set of configurations. I would like to have these shared codes to be bundled to one single common file.
I tried the following but it ended up creating an individual bundles of the vendors entry for each compile configuration.
var path = require("path");
var webpack = require("webpack");
module.exports = [
{
name: "app-mod1",
entry: {
vendors: ['jquery', 'react', 'react-dom'],
pageA: ['./mod1/pageA'],
pageB: ['./mod1/pageB']
},
output: {
path: path.join(__dirname, "/mod1/js"),
filename: "[name].bundle.js"
},
plugins: [
new webpack.optimize.CommonsChunkPlugin({
names: ['vendors'],
minChunks: Infinity
})
]
},
{
name: "app-mod2",
entry: {
vendors: ['lodash', 'react', 'react-dom'],
pageA: ['./mod2/pageA'],
pageB: ['./mod2/pageB']
},
output: {
path: path.join(__dirname, "/mod2/js"),
filename: "[name].bundle.js"
},
plugins: [
new webpack.optimize.CommonsChunkPlugin({
names: ['vendors'],
minChunks: Infinity
})
]
}
];
Since react, react-dom are shared between the 2 compilations, my intention is for them to be bundled as a single file which can be shared instead of creating a same bundle for each compilation.
How can I extract the common chunks out of multiple compiler configurations?
Brief answer
You can't do that job in the way you want.
TL;DR
#Carven, I am afraid that you can't achieve your goal via MultiCompiler of Webpack, MultiCompiler is not meant to do that job, at least for the near feature.
See the source code for initiating the MultiCompiler instance, it actually initiates separate Compiler instances. These compiler have no data shared between.
See also the source of running MultiCompiler instance, the compilers instance also run separately without sharing data.
The only thing these compilers share is the Stats instance they produce and merge into a MultiStats.
By the way, there is no clue in the example you mentioned that some modules are shared between multi-compilers.
Alternative
As described by #Tzook-Bar-Noy, IMHO, you have to use muti-entries to achieve MPA(Multi-page Application) bundling.
Other worth mentioning
I noticed a library called webpack-multi-configurator is using the multi-compiler feature. But I don't think it will share common chunk(s).
You can extract the shared code into another compilation, and bundle it with DllBundlesPlugin.
later consume this DLL via DLLReferencePlugin and add it to your page either manually or via HTMLWebpackPlugin's add-asset-html-webpack-plugin
Bolierplate can be reduced by using autodll-webpack-Plugin
I learned about it now, and this topic seems quite hard to understand in webpack docs. I managed to create something that works, as it created 2 separate files and extracted the common dependencies to another file.
Here is my webpack config:
{
entry: {
pageA: "./first/first",
pageB: "./second/second"
},
output: {
path: path.join(__dirname, "js"),
filename: "[name].js"
},
plugins: [
new webpack.optimize.CommonsChunkPlugin({
names: ["vendor", "common"],
})
]
};
the output of this will be:
./js/
common.js
vendor.js
pageA.js
pageB.js
I created a repo with the example I worked on: https://github.com/tzookb/webpack-common-vendor-chunks
when I open a new html file I load these files:
first page:
common.js
vendor.js
pageA.js
sec page:
common.js
vendor.js
pageB.js