Related
I’m working with Webpack 5 and trying to optimize the splitChunks configuration in a multi page application with a PHP backend and a Vue frontend.
In order to reduce the size of the vendor file I have started excluding some libraries by customizing the test function of the vendor cacheGroup.
test(module /* , chunk */) {
if (module.context) {
// only node_modules are needed
const isNodeModule = module.context.includes('node_modules');
// but we exclude some specific node_modules
const isToBundleSeparately = [
'marked', // only needed in one component
'braintree-web', // only payment page
'bootstrap/js',
].some(str => module.context.includes(str));
if (isNodeModule && !isToBundleSeparately) {
return true;
}
}
return false;
},
By doing so, some libraries that are not used in all pages are only imported in the components that need them, because those components are imported via dynamic imports and are extracted to separate chunks.
This has proven to work as expected until I encountered a strange behavior with one specific chunk and with one specific library (BootstrapVue).
If I add 'bootstrap-vue' to the list of excluded libraries, what happens is that two components of the library, BTab and BTabs, are extracted to a very large chunk which also includes the whole code for the payment page and all the libraries used in that page.
If you look at the screenshot, the file is the one whose name starts with “init-initPaymentGateway”.
So now all the pages that need those two BootstrapVue components are loading that large file, including the product page and other pages that only need the two tiny BootstrapVue components.
Here you can see that the chunk is imported in the product page:
I would expect that with my current configuration those two components would go to a separate chunk, or, if they are too small, they should be duplicated. Using Webpack Bundle Analyzer I see both very small files and duplicated library files, so I don’t understand why this is not happening with those specific components. The BootstrapVue BAlert component, for example, which is also small, is duplicated in different components.
I suspect that the issue comes from the fact that the components are small, but by setting minSize to 0 (or 10) I would expect to not have a minimum size for the creation of chunks.
Here are the imports in the payment page:
import { BTab, BTabs } from 'bootstrap-vue';
import dataCollector from 'braintree-web/data-collector';
(Then other inner components import other files from braintree-web).
Here is the import in one component of the product page:
import { BTabs, BTab } from 'bootstrap-vue';
And here is the complete splitChunks configuration (I have removed some of the excluded libraries from the list in the test function as they are not relevant).
splitChunks: {
chunks: 'all',
minSize: 10,
minChunks: 1,
maxAsyncRequests: 30,
maxInitialRequests: 30,
enforceSizeThreshold: 50000,
name(module, chunks /* , cacheGroupKey */) {
const allChunksNames = chunks
.filter(item => item.name !== null)
.map(item => item.name.replace('View', ''))
.join('~');
return allChunksNames.slice(0, 30);
},
cacheGroups: {
default: {
minChunks: 2,
priority: -20,
reuseExistingChunk: true,
},
defaultVendors: {
name: 'vendor',
priority: -10,
chunks: 'all',
minChunks: 1,
reuseExistingChunk: true,
test(module /* , chunk */) {
if (module.context) {
// only node_modules are needed
const isNodeModule = module.context.includes('node_modules');
// but we exclude some specific node_modules
const isToBundleSeparately = [
'marked', // only needed in one component
'bootstrap-vue',
'braintree-web',
'bootstrap/js',
].some(str => module.context.includes(str));
if (isNodeModule && !isToBundleSeparately) {
return true;
}
}
return false;
},
},
},
},
To fix the issue I removed 'bootstrap-vue' from the exclusion in the test function, but I’d like to understand if I can improve my configuration or at least the reason for this behavior.
By trying to fix another issue caused by a naming collision, I have modified the name function and fixed the issue.
Basically the slice function was causing a naming collision, which resulted in a chunk containing the code for more than one module, because both had the same name.
An explanation is given in the Webpack docs:
https://webpack.js.org/plugins/split-chunks-plugin/#splitchunksminsize, but it wasn't very clear to me until I experienced this issue:
Warning
When assigning equal names to different split chunks, all vendor modules are placed into a single shared chunk, though it's not recommend since it can result in more code downloaded.
A simple fix was to increase the length of the name by changing the last line from
return allChunksNames.slice(0, 30);
to
return allChunksNames.slice(0, 40);
But I still had the feeling that in some moment another naming clash could come up. I tried removing the slice part but it resulted in a Webpack error complaining about a name that was too long.
So in the end the long term solution was to simply delete the name function, and let Webpack assign a name.
name(module, chunks /* , cacheGroupKey */) {
const allChunksNames = chunks
.filter(item => item.name !== null)
.map(item => item.name.replace('View', ''))
.join('~');
return allChunksNames.slice(0, 40);
}
The resulting names are less clear, but I can always figure out what each chunk contains by using the Webpack Bundle Analyzer plugin.
Here is the chunk containing only the two modules that are needed in the product page.
I have the following gulp task which I want to change the filename and contents of a file replacing any matching strings with the replacement.
The matching strings in the file contents get changed, but the file's name does not. I thought it would as my code appears to match the examples on https://www.npmjs.com/package/gulp-replace
What am I doing wrong?
function renameFileContents() {
return gulp.src([
'**/*',
'!.github/**',
'!languages/**',
'!node_modules/**',
'!.babelrc',
'!.editconfig',
'!.gitignore',
'!.travis.yml',
'!CHANGELOG.md',
'!codesniffer.ruleset.xml',
'!composer.json',
'!composer.lock',
'!config.yml',
'!config-default.yml',
'!gulpfile.babel.js',
'!MIT-LICENSE.txt',
'!package-lock.json',
'!package.json',
'!phpunit.xml.dist',
'!README.md',
'!webpack.config.js'
])
.pipe($.replace('BigTest', 'Tester'))
.pipe($.replace('Bigtest', 'Tester'))
.pipe($.replace('bigtest', 'tester'))
.pipe(gulp.dest('./'));
}
Use gulp-rename to alter filenames. Add:
const rename = require('gulp-rename');
and before .pipe(gulp.dest('./'));:
.pipe(
rename(function(path) {
path.basename = path.basename.replace(/BigTest|Bigtest|bigtest/, function(matched) {
return { BigTest: 'Tester', Bigtest: 'Tester', bigtest: 'tester' }[matched];
});
})
)
You asked in a comment why new files are created (with the new names) but the original files still remain. Why does gulp-rename not actually rename the original files as you might expect?
Gulp-rename is not working with the original files. This can be a little confusing.
It's called gulp-rename because it renames an in-memory gulp file
object. gulp is like functional programming, each plugin takes in
input and produces output in-memory without causing side effects. [emphasis added]
gulp works like this:
read file (gulp.src)
do some stuff, modify the file in-memory (plugins)
commit file changes back to fs (gulp.dest/or others)
From gulp-rename issues: not renaming the original files.
The suggested fix (from gulp recipes: deleting files from a pipeline) which I tested is:
const del = require('del');
const vinylPaths = require('vinyl-paths');
and add this pipe before the replace pipe:
.pipe(vinylPaths(del))
.pipe(
rename(function(path) { ......
and your original files will be deleted, leaving only the newly named files. Obviously, make sure you test this on good test cases before deleting any of your files!
I want to use WebPack to build my JavaScript application. The application is supposed to be deployed on several sites, in several different modifications. Therefore, I have set up the config to use multiple entry points, one per each site:
entry: {
s1: "s1",
s2: "s2",
s3: "s3"
},
The application also uses several dependencies, which are loaded via AMD modules (I used RequireJS before):
c1
c2
...
Let's say that s1 requires both c1 and c2, whereas s2 needs only c1. The build works OK, and it creates s1, s2 and s3 as the entry points, and several chunks containing various combinations of the components, as the sites need:
/* s1 */
define(['c1', 'c2'], function(c1, c2) { ... }
The question I have is: What do I need to specify in the configuration in order to get each site package built as one standalone file? I appreciate the ability of WebPack to split the application into chunks, but right now I need every build to be one JS file (for various reasons).
I am able to achieve that easily when I configure only a single entry point (such as entry: "s1"), using the following:
webpack.optimize.LimitChunkCountPlugin({maxChunks: 1})
However, for multiple entry points, this configuration creates one additional chunk on top of all entry points. How can I make sure each built entry point (such as s1.bundle.js, s2.bundle.js, ...) contains all JavaScript inside that one file, and doesn't need to load any chunks on demand?
Sounds like the easiest way is to have multiple builds (each with one entry point) rather than a single build with multiple entry points. If your webpack.config.js exports an array of config objects instead of a single config object, webpack will automatically do a separate build for each config. So you could, for example, put your entry points in an array and loop through it, creating a config object for each. Now each config has only a single entry point, and the LimitChunkCountPlugin should give you the single file.
In Webpack 5, you can use chunkLoading: false in your entry definition.
{
// ...
entry: {
s1: {
import: "s1",
chunkLoading: false
}
s2: {
import: "s2",
chunkLoading: false
}
s3: {
import: "s3",
chunkLoading: false
}
},
// ...
}
https://webpack.js.org/configuration/entry-context/#entry-descriptor
Copying the code example from the link above in case the link breaks in the future:
module.exports = {
//...
entry: {
home: './home.js',
shared: ['react', 'react-dom', 'redux', 'react-redux'],
catalog: {
import: './catalog.js',
filename: 'pages/catalog.js',
dependOn: 'shared',
chunkLoading: false, // Disable chunks that are loaded on demand and put everything in the main chunk.
},
personal: {
import: './personal.js',
filename: 'pages/personal.js',
dependOn: 'shared',
chunkLoading: 'jsonp',
asyncChunks: true, // Create async chunks that are loaded on demand.
layer: 'name of layer', // set the layer for an entry point
},
},
};
I am trying to merge css and scss files into a main.css file that goes in my build directory.
Its working, but not in the right order. The style attributes from the scss files need to be in the bottom of the main.css file so they overrule the rest.
my Gulp task looks like this:
//CSS
gulp.task('css', function () {
var cssTomincss = gulp.src(['dev/css/reset.css', 'dev/css/style.css','dev/css/typography.css', 'dev/css/sizes.css']);
var cssFromscss = gulp.src(['dev/css/*.scss'])
.pipe(sass());
return es.merge(cssTomincss, cssFromscss)
.pipe(concat('main.css'))
.pipe(minifyCSS())
.pipe(gulp.dest('build/css'))
});
I am defining the sources first with variables. I am using the gulp-sass plugin to convert the scss file into normal css (.pipe(sass)) and later merging the two with the es.merge function and concatenating them into main.css.
The problem is that the style attributes van the .scss files end up somewhere in the top end of the main.css file. I need them to be at the bottom. So they need to be concatenated at the bottom.
Any clue on how to do this?
Try streamqueue.
var streamqueue = require('streamqueue');
gulp.task('css', function () {
return streamqueue({ objectMode: true },
gulp.src(['dev/css/reset.css', 'dev/css/style.css', 'dev/css/typography.css', 'dev/css/sizes.css']),
gulp.src(['dev/css/*.scss']).pipe(sass())
)
.pipe(concat('main.css'))
.pipe(minifyCSS())
.pipe(gulp.dest('build/css'))
});
This cheatsheet will help you. PDF is here.
It seems that the plugin gulp-order fits perfectly well in your case.
It allows you to re-order the passed stream with your own glob pattern, for example based on your code :
return es.merge(cssTomincss, cssFromscss)
.pipe(order([
'dev/css/reset.css',
'dev/css/style.css',
'dev/css/typography.css',
'dev/css/sizes.css',
'dev/css/*.css',
]))
.pipe(concat('main.css'))
.pipe(minifyCSS())
.pipe(gulp.dest('build/css'))
One drawback of this is that you have to re-declare your globs, but you can get around by assign your globs to a value and then concat them in you order pipe, much cleaner.
You may have to set the base option to . of gulp-order as stated in their Readme if the files were not ordered correctly.
One another way would be to use stream-series, basically the same as event-stream, but the order of your stream is preserved, and you don't have to rewrite your globs.
I tried gulp-order without success: the order somehow wasn't taken into account.
The solution which worked for me was using stream-series, mentioned by Aperçu.
return streamSeries(
cssTomincss,
cssFromscss)
.pipe(concat('main.css'))
.pipe(minifyCSS())
.pipe(gulp.dest('build/css'));
I failed with all provided answers, they produced some silent errors. Finally merge2 worked for me (seems like there was gulp-merge and later the project was renamed into merge2). I'm not sure why there is a need in streamify plugin, e.g. streams created with Rollup may produce "stream-not-supported-errors" with gulp-concat, gulp-uglify or gulp-insert.
const mergeStreams = require('merge2');
const streamify = require('streamify');
...
gulp.task('build', () => {
const streams = sources.map(createJSFile);
return mergeStreams(...streams)
.pipe(streamify(concat('bundle.js')))
.pipe(streamify(uglify()))
.pipe(gulp.dest('./dist'));
});
Say, for example, you are building a project on Backbone or whatever and you need to load scripts in a certain order, e.g. underscore.js needs to be loaded before backbone.js.
How do I get it to concat the scripts so that they’re in order?
// JS concat, strip debugging and minify
gulp.task('scripts', function() {
gulp.src(['./source/js/*.js', './source/js/**/*.js'])
.pipe(concat('script.js'))
.pipe(stripDebug())
.pipe(uglify())
.pipe(gulp.dest('./build/js/'));
});
I have the right order of scripts in my source/index.html, but since files are organized by alphabetic order, gulp will concat underscore.js after backbone.js, and the order of the scripts in my source/index.html does not matter, it looks at the files in the directory.
So does anyone have an idea on this?
Best idea I have is to rename the vendor scripts with 1, 2, 3 to give them the proper order, but I am not sure if I like this.
As I learned more I found Browserify is a great solution, it can be a pain at first but it’s great.
I had a similar problem recently with Grunt when building my AngularJS app. Here's a question I posted.
What I ended up doing is to explicitly list the files in order in the grunt config. The config file will then look like this:
[
'/path/to/app.js',
'/path/to/mymodule/mymodule.js',
'/path/to/mymodule/mymodule/*.js'
]
Grunt is able to figure out which files are duplicates and not include them. The same technique will work with Gulp as well.
Another thing that helps if you need some files to come after a blob of files, is to exclude specific files from your glob, like so:
[
'/src/**/!(foobar)*.js', // all files that end in .js EXCEPT foobar*.js
'/src/js/foobar.js',
]
You can combine this with specifying files that need to come first as explained in Chad Johnson's answer.
I have used the gulp-order plugin but it is not always successful as you can see by my stack overflow post gulp-order node module with merged streams. When browsing through the Gulp docs I came across the streamque module which has worked quite well for specifying order of in my case concatenation. https://github.com/gulpjs/gulp/blob/master/docs/recipes/using-multiple-sources-in-one-task.md
Example of how I used it is below
var gulp = require('gulp');
var concat = require('gulp-concat');
var handleErrors = require('../util/handleErrors');
var streamqueue = require('streamqueue');
gulp.task('scripts', function() {
return streamqueue({ objectMode: true },
gulp.src('./public/angular/config/*.js'),
gulp.src('./public/angular/services/**/*.js'),
gulp.src('./public/angular/modules/**/*.js'),
gulp.src('./public/angular/primitives/**/*.js'),
gulp.src('./public/js/**/*.js')
)
.pipe(concat('app.js'))
.pipe(gulp.dest('./public/build/js'))
.on('error', handleErrors);
});
With gulp-useref you can concatenate every script declared in your index file, in the order in which you declare it.
https://www.npmjs.com/package/gulp-useref
var $ = require('gulp-load-plugins')();
gulp.task('jsbuild', function () {
var assets = $.useref.assets({searchPath: '{.tmp,app}'});
return gulp.src('app/**/*.html')
.pipe(assets)
.pipe($.if('*.js', $.uglify({preserveComments: 'some'})))
.pipe(gulp.dest('dist'))
.pipe($.size({title: 'html'}));
});
And in the HTML you have to declare the name of the build file you want to generate, like this:
<!-- build:js js/main.min.js -->
<script src="js/vendor/vendor.js"></script>
<script src="js/modules/test.js"></script>
<script src="js/main.js"></script>
In your build directory you will have the reference to main.min.js which will contain vendor.js, test.js, and main.js
The sort-stream may also be used to ensure specific order of files with gulp.src. Sample code that puts the backbone.js always as the last file to process:
var gulp = require('gulp');
var sort = require('sort-stream');
gulp.task('scripts', function() {
gulp.src(['./source/js/*.js', './source/js/**/*.js'])
.pipe(sort(function(a, b){
aScore = a.path.match(/backbone.js$/) ? 1 : 0;
bScore = b.path.match(/backbone.js$/) ? 1 : 0;
return aScore - bScore;
}))
.pipe(concat('script.js'))
.pipe(stripDebug())
.pipe(uglify())
.pipe(gulp.dest('./build/js/'));
});
I just add numbers to the beginning of file name:
0_normalize.scss
1_tikitaka.scss
main.scss
It works in gulp without any problems.
I have my scripts organized in different folders for each package I pull in from bower, plus my own script for my app. Since you are going to list the order of these scripts somewhere, why not just list them in your gulp file? For new developers on your project, it's nice that all your script end-points are listed here. You can do this with gulp-add-src:
gulpfile.js
var gulp = require('gulp'),
less = require('gulp-less'),
minifyCSS = require('gulp-minify-css'),
uglify = require('gulp-uglify'),
concat = require('gulp-concat'),
addsrc = require('gulp-add-src'),
sourcemaps = require('gulp-sourcemaps');
// CSS & Less
gulp.task('css', function(){
gulp.src('less/all.less')
.pipe(sourcemaps.init())
.pipe(less())
.pipe(minifyCSS())
.pipe(sourcemaps.write('source-maps'))
.pipe(gulp.dest('public/css'));
});
// JS
gulp.task('js', function() {
gulp.src('resources/assets/bower/jquery/dist/jquery.js')
.pipe(addsrc.append('resources/assets/bower/bootstrap/dist/js/bootstrap.js'))
.pipe(addsrc.append('resources/assets/bower/blahblah/dist/js/blah.js'))
.pipe(addsrc.append('resources/assets/js/my-script.js'))
.pipe(sourcemaps.init())
.pipe(concat('all.js'))
.pipe(uglify())
.pipe(sourcemaps.write('source-maps'))
.pipe(gulp.dest('public/js'));
});
gulp.task('default',['css','js']);
Note: jQuery and Bootstrap added for demonstration purposes of order. Probably better to use CDNs for those since they are so widely used and browsers could have them cached from other sites already.
Try stream-series. It works like merge-stream/event-stream.merge() except that instead of interleaving, it appends to the end. It doesn't require you to specify the object mode like streamqueue, so your code comes out cleaner.
var series = require('stream-series');
gulp.task('minifyInOrder', function() {
return series(gulp.src('vendor/*'),gulp.src('extra'),gulp.src('house/*'))
.pipe(concat('a.js'))
.pipe(uglify())
.pipe(gulp.dest('dest'))
});
merge2 looks like the only working and maintained ordered stream merging tool at the moment.
Update 2020
The APIs are always changing, some libraries become unusable or contain vulnerabilities, or their dependencies contain vulnerabilities, that are not fixed for years. For text files manipulations you'd better use custom NodeJS scripts and popular libraries like globby and fs-extra along with other libraries without Gulp, Grunt, etc wrappers.
import globby from 'globby';
import fs from 'fs-extra';
async function bundleScripts() {
const rootPaths = await globby('./source/js/*.js');
const otherPaths = (await globby('./source/**/*.js'))
.filter(f => !rootFiles.includes(f));
const paths = rootPaths.concat(otherPaths);
const files = Promise.all(
paths.map(
// Returns a Promise
path => fs.readFile(path, {encoding: 'utf8'})
)
);
let bundle = files.join('\n');
bundle = uglify(bundle);
bundle = whatever(bundle);
bundle = bundle.replace(/\/\*.*?\*\//g, '');
await fs.outputFile('./build/js/script.js', bundle, {encoding: 'utf8'});
}
bundleScripts.then(() => console.log('done');
An alternative method is to use a Gulp plugin created specifically for this problem. https://www.npmjs.com/package/gulp-ng-module-sort
It allows you to sort your scripts by adding in a .pipe(ngModuleSort()) as such:
var ngModuleSort = require('gulp-ng-module-sort');
var concat = require('gulp-concat');
gulp.task('angular-scripts', function() {
return gulp.src('./src/app/**/*.js')
.pipe(ngModuleSort())
.pipe(concat('angularAppScripts.js))
.pipe(gulp.dest('./dist/));
});
Assuming a directory convention of:
|——— src/
| |——— app/
| |——— module1/
| |——— sub-module1/
| |——— sub-module1.js
| |——— module1.js
| |——— module2/
| |——— sub-module2/
| |——— sub-module2.js
| |——— sub-module3/
| |——— sub-module3.js
| |——— module2.js
| |——— app.js
Hope this helps!
For me I had natualSort() and angularFileSort() in pipe which was reordering the files. I removed it and now it works fine for me
$.inject( // app/**/*.js files
gulp.src(paths.jsFiles)
.pipe($.plumber()), // use plumber so watch can start despite js errors
//.pipe($.naturalSort())
//.pipe($.angularFilesort()),
{relative: true}))
I just use gulp-angular-filesort
function concatOrder() {
return gulp.src('./build/src/app/**/*.js')
.pipe(sort())
.pipe(plug.concat('concat.js'))
.pipe(gulp.dest('./output/'));
}
I'm in a module environnement where all are core-dependents using gulp.
So, the core module needs to be appended before the others.
What I did:
Move all the scripts to an src folder
Just gulp-rename your core directory to _core
gulp is keeping the order of your gulp.src, my concat src looks like this:
concat: ['./client/src/js/*.js', './client/src/js/**/*.js', './client/src/js/**/**/*.js']
It'll obviously take the _ as the first directory from the list (natural sort?).
Note (angularjs):
I then use gulp-angular-extender to dynamically add the modules to the core module.
Compiled it looks like this:
angular.module('Core', ["ui.router","mm.foundation",(...),"Admin","Products"])
Where Admin and Products are two modules.
if you would like to order third party libraries dependencies, try wiredep. This package basically checks each package dependency in bower.json then wire them up for you.
I tried several solutions from this page, but none worked. I had a series of numbered files which I simply wanted be ordered by alphabetical foldername so when piped to concat() they'd be in the same order. That is, preserve the order of the globbing input. Easy, right?
Here's my specific proof-of-concept code (print is just to see the order printed to the cli):
var order = require('gulp-order');
var gulp = require('gulp');
var print = require('gulp-print').default;
var options = {};
options.rootPath = {
inputDir: process.env.INIT_CWD + '/Draft',
inputGlob: '/**/*.md',
};
gulp.task('default', function(){
gulp.src(options.rootPath.inputDir + options.rootPath.inputGlob, {base: '.'})
.pipe(order([options.rootPath.inputDir + options.rootPath.inputGlob]))
.pipe(print());
});
The reason for the madness of gulp.src? I determined that gulp.src was running async when I was able to use a sleep() function (using a .map with sleeptime incremented by index) to order the stream output properly.
The upshot of the async of src mean dirs with more files in it came after dirs with fewer files, because they took longer to process.
In my gulp setup, I'm specifying the vendor files first and then specifying the (more general) everything, second. And it successfully puts the vendor js before the other custom stuff.
gulp.src([
// vendor folder first
path.join(folder, '/vendor/**/*.js'),
// custom js after vendor
path.join(folder, '/**/*.js')
])
Apparently you can pass in the "nosort" option to gulp.src gulp.src.