Concatenate JS files after webpack build process - javascript

I'm trying to concatenate two js files after the webpack build process. The goal is to provide a single js file with the ES6 modules and the legacy code in it.
Already tried plugins like webpack-concat-files-plugin without success. Which makes sense to me, because the output files are not there when the plugin gets executed.
Another thought would be a small script executed in the afterCompile hook, which handles the concatenation of the two files. Is this the common way to do something like this or are there other ways to achieve the goal?
Thanks for your help.
Basic example:
module.exports = {
entry: {
app: 'app.js',
legacy: [
'legacy-1.js',
'legacy-2.js',
// ...
]
},
output: {
filename: path.join('dist/js', '[name].js'),
}
}

Solved this as suggested:
FileMergeWebpackPlugin.js
const fs = require('fs');
class FileMergeWebpackPlugin {
constructor({ files, destination, removeSourceFiles }) {
this.files = files;
this.destination = destination;
this.removeSourceFiles = removeSourceFiles;
}
apply(compiler) {
compiler.hooks.afterEmit.tap('FileMergeWebpackPlugin', () => {
const fileBuffers = [];
this.files
.filter(file => fs.existsSync(file))
.forEach(file => fileBuffers.push(fs.readFileSync(file)))
fs.writeFileSync(this.destination, fileBuffers.concat(), { encoding: 'UTF-8' })
if (this.removeSourceFiles) {
this.files.forEach(file => fs.unlinkSync(file));
}
});
}
}
module.exports = FileMergeWebpackPlugin;
webpack.config.js
const FileMergeWebpackPlugin = require('./FileMergeWebpackPlugin');
module.exports = {
entry: {
app: 'app.js',
legacy: [
'legacy-1.js',
'legacy-2.js',
]
},
output: {
filename: path.join('dist/js', '[name].js'),
},
plugins: [
new FileMergeWebpackPlugin({
destination: 'dist/js/bundle.js',
removeSourceFiles: true,
files: [
'dist/js/app.js',
'dist/js/legacy.js',
]
})
]
}
Eventually i will release this as a npm package, will update the post when i do so

Yes.
You can create your own Plugin that is activate in emit hook - the parameter for this hook is compilation. You can get the chunks you look to concat from this object and create a new chunk with the concated value (or just add one to the other)

Related

Reload Webpack Html Template on Certain Files Change

I'm using revealjs to create responsive presentations. The problem with revealjs is that all the slides code is written in a single HTML file which can be messy to some level (Some presentations' HTML code reached about 3500 lines of HTML in that single file).
I'm now restructuring this system and I would like to have a directory named slides that contains each slide HTML file. Each of these files is named slidenumber.html. Finally, I want to bundle all of the files with webpack 5 into a single HTML file in dist. I managed to achieve this but it has an issue with the dev server.
webpack.config.js
// ... imports ....
module.exports = {
...,
plugins: [
....,
new HtmlWebpackPlugin({
filename: "index.html",
inject: true,
templateContent: getTemplate(),
}),
new WatchExternalFilesPlugin({
files: ["./slides/*.html"],
}),
],
module: {
rules: [...],
},
devServer: {
port: 8080,
},
};
The getTemplate function loops over the HTML files in the slides directory and returns them wrapped with the template boilerplate
This is the function for reference
getTemplate.js
const fs = require("fs/promises");
const path = require("path");
const { parse } = require("node-html-parser");
module.exports = async () => {
const template = parse(
await fs.readFile(path.join(__dirname, "../templates/index.html"))
);
const files = await fs.readdir(path.join(__dirname, "../slides"));
for await (const fileName of files) {
const slide = parse(
await fs.readFile(path.join(__dirname, `../slides/${fileName}`))
);
template.querySelector("#slides").appendChild(slide);
}
return template.toString();
};
all of the above code is working fine on build but when running the dev server, I can't get the HtmlWebpackPlugin to re-execute the templateContent: getTemplate() on the change of any HTML slide file in the slides directory and as a result, when I edit any file of the slides HTML files in the slides directory, I don't get any update.
I'm aware that templateContent is supposed to run only on the start of the server but I'm asking if there is any other feature that can get me to the required behavior.
Thanks if you made it to here and excuse my English, I'm not a native speaker.
I could achieve the behavior I described in the question by setting a middleware from the dev server options that catches any request and returns the output of the getTemplate function.
This is the configurations for the dev server
webpack.config.dev.js
// ...imports...
module.exports = {
mode: "development",
entry: { index: "./main.js" },
output: {...},
module: {
rules: [...],
},
devServer: {
port: 8080,
watchFiles: ["./slides/*.html"],
hot: true,
onBeforeSetupMiddleware: function (devServer) {
devServer.app.get("/", async function (req, res) {
res.send(await getTemplate());
});
},
},
};
These are the configurations for the production server
webpack.config.production.js
// ...imports...
module.exports = {
mode: "production",
entry: { index: "./main.js" },
output: {...},
plugins: [
new HtmlWebpackPlugin({
filename: "index.html",
templateContent: getTemplate(),
inject: false,
}),
],
module: {
rules: [...],
},
};
I used the webpackHtmlPlugin in production as usual but in development, I didn't use it at all since it can't reload the templates on the build
In development, though I lost the ability to add the hash number to the compiled js file as I won't be able to predict the hash and inject its script tag. The compiled file had the same name as the original file and I added the script tag manually in the HTML template file.
Hope this helps anyone!

How to compile template string at build time using Babel (not Webpack)

I am using Babel (without Webpack) to transpile some ES6 code. The code contains a template literal which I would like to evaluate at build time.
The code contains the following import where I would like to inject the version:
const Component = React.lazy(() => import(`projectVersion${__VERSION__}/FooBar`))
I have used a number of different plugins to try and achieve this such as babel-plugin-inline-replace-variables, babel-plugin-transform-inline-environment-variables, but the code compiles to something like this:
var Component = /*#__PURE__*/_react.default.lazy(function () {
return Promise.resolve("projectVersion".concat("1", "/FooBar")).then(function (s) {
return _interopRequireWildcard(require(s));
});
});
But what I'm after is something along the lines of this (as if I just manually added the version to projectVersion):
const Component = /* #__PURE__*/_react.default.lazy(() => Promise.resolve().then(() => _interopRequireWildcard(require('projectVersion1/FooBar'))))
One of the reasons for this is because when running the code with concat, I get the error Critical dependency: the request of a dependency is an expression.
Thanks in advance for any help and suggestions.
More Information
Command: babel src --out-dir build
Babel Config:
module.exports = api => {
api.cache(true)
return {
presets: [
['#babel/preset-env', {
corejs: 3,
targets: '> 0.25%, not dead',
useBuiltIns: 'usage',
}],
['#babel/preset-react', {
runtime: 'automatic',
}],
'#babel/preset-typescript',
],
plugins: [],
}
}
Having tried many different potential solutions, I eventually landed on writing a custom babel plugin to replace the TemplateLiteral with a StringLiteral.
The code below is written quite lazily and is specific for my use-case, but hopefully it will help someone. If I have time, I will make it more robust and re-useable and share it on npm.
module.exports = ({ types: t }) => ({
visitor: {
TemplateLiteral(path, state) {
const { version } = state.opts
const { expressions, quasis } = path.node
const expressionStrings = expressions.map(exp => exp.name)
if (expressionStrings.length === 1 && expressionStrings.includes('__VERSION__')) {
const startString = quasis.find(q => q.tail === false).value.raw
const endString = quasis.find(q => q.tail === true).value.raw
path.replaceWith(
t.StringLiteral(`${startString}${version}${endString}`),
)
}
},
},
})
Using the plugin is as simple as adding it to your babel plugins:
plugins: [
['./babel/interpolate-template-literal', {
version: "99",
}],
],

How to inject Webpack build hash to application code

I'm using Webpack's [hash] for cache busting locale files. But I also need to hard-code the locale file path to load it from browser. Since the file path is altered with [hash], I need to inject this value to get right path.
I don't know how can get Webpack [hash] value programmatically in config so I can inject it using WebpackDefinePlugin.
module.exports = (env) => {
return {
entry: 'app/main.js',
output: {
filename: '[name].[hash].js'
}
...
plugins: [
new webpack.DefinePlugin({
HASH: ***???***
})
]
}
}
In case you want to dump the hash to a file and load it in your server's code, you can define the following plugin in your webpack.config.js:
const fs = require('fs');
class MetaInfoPlugin {
constructor(options) {
this.options = { filename: 'meta.json', ...options };
}
apply(compiler) {
compiler.hooks.done.tap(this.constructor.name, stats => {
const metaInfo = {
// add any other information if necessary
hash: stats.hash
};
const json = JSON.stringify(metaInfo);
return new Promise((resolve, reject) => {
fs.writeFile(this.options.filename, json, 'utf8', error => {
if (error) {
reject(error);
return;
}
resolve();
});
});
});
}
}
module.exports = {
// ... your webpack config ...
plugins: [
// ... other plugins ...
new MetaInfoPlugin({ filename: 'dist/meta.json' }),
]
};
Example content of the output meta.json file:
{"hash":"64347f3b32969e10d80c"}
I've just created a dumpmeta-webpack-plugin package for this plugin. So you might use it instead:
const { DumpMetaPlugin } = require('dumpmeta-webpack-plugin');
module.exports = {
...
plugins: [
...
new DumpMetaPlugin({
filename: 'dist/meta.json',
prepare: stats => ({
// add any other information you need to dump
hash: stats.hash,
})
}),
]
}
Please refer to the Webpack documentation for all available properties of the Stats object.
Seems like it should be a basic feature but apparently it's not that simple to do.
You can accomplish what you want by using wrapper-webpack-plugin.
plugins: [
new WrapperPlugin({
header: '(function (BUILD_HASH) {',
footer: function (fileName) {
const rx = /^.+?\.([a-z0-9]+)\.js$/;
const hash = fileName.match(rx)[1];
return `})('${hash}');`;
},
})
]
A bit hacky but it works — if u don't mind the entire chunk being wrapped in an anonymous function.
Alternatively you can just add var BUILD_HASH = ... in the header option, though it could cause problem if it becomes a global.
I created this plugin a while back, I'll try to update it so it provides the chunk hash naturally.
On server, you can get the hash by reading the filenames (example: web.bundle.f4771c44ee57573fabde.js) from your bundle folder.
You can pass the version to your build using webpack.DefinePlugin
If you have a package.json with a version, you can extract it like this:
const version = require("./package.json").version;
For example (we stringified the version):
new webpack.DefinePlugin({
'process.env.VERSION': JSON.stringify(version)
}),
then in your javascript, the version will be available as:
process.env.VERSION
The WebpackManifestPlugin is officially recommended in the output management guide. It writes a JSON to the output directory mapping the input filenames to the output filenames. Then you can inject those mapped values into your server template.
It's similar to Dmitry's answer, except Dmitry's doesn't appear to support multiple chunks.
That can be done with Webpack Stats Plugin. It gives you nice and neat output file with all the data you want. And it's easy to incorporate it to the webpack config files where needed.
E.g. To get hash generated by Webpack and use it elsewhere.
Could be achieved like:
# installation
npm install --save-dev webpack-stats-plugin
yarn add --dev webpack-stats-plugin
# generating stats file
const { StatsWriterPlugin } = require("webpack-stats-plugin")
module.exports = {
plugins: [
// Everything else **first**.
// Write out stats file to build directory.
new StatsWriterPlugin({
stats: {
all: false,
hash: true,
},
filename: "stats.json" // Default and goes straight to your output folder
})
]
}
# usage
const stats = require("YOUR_PATH_TO/stats.json");
console.log("Webpack's hash is - ", stats.hash);
More usage examples in their repo
Hope that helps!

Vue Cli 3 is not allowing me to process SVG's in Webpack

Vue Cli defaults to file-loader for SVG assets, but I want to use svg-sprite-loader (as well as a few others) instead.
I updated the vue.config.js file to do this and it still seems to use file-loader. Almost as though it's not picking up my config at all.
vue.config.js
module.exports = {
configureWebpack: {
module: {
rules: [
{
test: /\.(svg)(\?.*)?$/,
use: [
{
loader: 'svg-sprite-loader',
options: {
name: '[name]-[hash:7]',
prefixize: true
}
},
'svg-fill-loader',
'svgo-loader'
]
}
]
}
}
}
Is there anything wrong with my setup?
I'm still getting SVG files imported into my component as a URL string / path when it should be an object with properties.
Many thanks.
This took me a while to find a work around. Basically you need to stop file-loader matching on .svg. The best way I have found to do this is using chainWebpack and returning false from the test method on file-loader. I have included my working config.
module.exports = {
lintOnSave: false,
configureWebpack: {
module: {
rules: [
{
test: /\.(svg)(\?.*)?$/,
use: [
{
loader: 'svg-inline-loader',
options: {
limit: 10000,
name: 'assets/img/[name].[hash:7].[ext]'
}
}
]
}
]
}
},
chainWebpack: config => {
config.module
.rule('svg')
.test(() => false)
.use('file-loader')
}
}
The Webpack docs for Vue CLI 3.0 beta got updated with an example on how to replace an existing Base Loader. For svg-sprite-loader this means that you'll have to add the following configuration to your vue.config.js:
chainWebpack: config => {
config.module
.rule('svg')
.use('file-loader')
.loader('svg-sprite-loader')
}
I'm using Vue CLI 3.0.3 and this config works for me 😉
const path = require('path');
const glob = require('glob');
const SpriteLoaderPlugin = require('svg-sprite-loader/plugin');
module.exports = {
lintOnSave: false,
configureWebpack: {
plugins: [
new SpriteLoaderPlugin()
]
},
chainWebpack: config => {
config.module.rules.delete('svg');
config
.entry('app')
.clear()
.add(path.resolve(__dirname, './src/main.ts'))
config
.entry('sprite')
.add(...glob.sync(path.resolve(__dirname, `./src/assets/icons/*.svg`)));
config.module.rule('svg')
.test(/\.(svg)(\?.*)?$/)
.use('file-loader')
.loader('svg-sprite-loader')
.options({
extract: true,
spriteFilename: 'icons.svg'
})
}
};
Vue CLI docs for version 3.x in webpack section suggests to use something like this:
// vue.config.js
module.exports = {
chainWebpack: config => {
const svgRule = config.module.rule('svg')
// clear all existing loaders.
// if you don't do this, the loader below will be appended to
// existing loaders of the rule.
svgRule.uses.clear()
// add replacement loader(s)
svgRule
.use('vue-svg-loader')
.loader('vue-svg-loader')
}
}
Even vue-svg-loader configuration guide suggests same approach.
module.exports = {
chainWebpack: config => {
const svgRule = config.module.rule('svg')
svgRule.clear()
svgRule
.use('vue-svg-loader')
.loader('vue-svg-loader')
}
}

Compiling Webpack in memory but resolving to node_modules on disk

I'm trying to use web pack to compile an in memory string of valid javascript code. I'm using memory fs as outlined here: https://webpack.github.io/docs/node.js-api.html#compile-to-memory.
So I'm taking a string containing raw javascript, writing that to memory fs, and then web pack resolves to that entry point. But the compilation fails on the first require statement, presumably because it's not able to look in the real fs for node_modules.
Any ideas on how can I accomplish this?
import webpack from 'webpack';
import MemoryFS from 'memory-fs';
import thenify from 'thenify';
function* compile(code) {
const fs = new MemoryFS();
fs.writeFileSync('/file.js', code);
const compiler = webpack({
entry: { file: '/file.js' },
output: {
path: '/build',
filename: '[name].js'
},
module: {
loaders: [
{ test: /\.json$/, loader: 'json' }
],
}
});
compiler.run = thenify(compiler.run);
compiler.inputFileSystem = fs;
compiler.resolvers.normal.fileSystem = fs; //this is needed for memfs
compiler.outputFileSystem = fs;
const stats = yield compiler.run();
//retrieve the output of the compilation
const res = stats.compilation.assets['file.js'].source();
return res;
}
Usage
var code = "var _ = require('underscore'); console.log(_);";
var bundle = yield compile(code); //should be a bundle containing the underscore source.
The error is
ModuleNotFoundError: Module not found: Error: Cannot resolve module
underscore in /
This question indicates that others have tried the same thing: https://github.com/webpack/webpack/issues/1562. there's a gist referenced at https://gist.github.com/DatenMetzgerX/2a96ebf287b4311f4c18 that I believe was intended to do what I'm hoping to accomplish, but in it's current form I don't see how. It assigns an instance of MemoryFs to all of the resolvers. I've tried assigning node's fs module, but no dice.
So in short, I'm trying to set an entry point to an in memory string of raw javascript, but still have require and import statements resolved to node_modules on disk.
UPDATE
I've been able to get the result I'm looking for but it's not pretty. I'm basically overriding the implementation of #stat and #readFile in MemoryFS
to check the real filesystem if it gets any request for a file that doesn't exist in memory. I could clean this up a bit by subclassing MemoryFS instead of swapping method implementations at runtime, but the idea would still be the same.
Working solution
import webpack from 'webpack';
import JsonLoader from 'json-loader';
import MemoryFS from 'memory-fs';
import UglifyJS from "uglify-js";
import thenify from 'thenify';
import path from 'path';
import fs from 'fs';
import root from 'app-root-path';
/*
* Provide webpack with an instance of MemoryFS for
* in-memory compilation. We're currently overriding
* #stat and #readFile. Webpack will ask MemoryFS for the
* entry file, which it will find successfully. However,
* all dependencies are on the real filesystem, so any require
* or import statements will fail. When that happens, our wrapper
* functions will then check fs for the requested file.
*/
const memFs = new MemoryFS();
const statOrig = memFs.stat.bind(memFs);
const readFileOrig = memFs.readFile.bind(memFs);
memFs.stat = function (_path, cb) {
statOrig(_path, function(err, result) {
if (err) {
return fs.stat(_path, cb);
} else {
return cb(err, result);
}
});
};
memFs.readFile = function (path, cb) {
readFileOrig(path, function (err, result) {
if (err) {
return fs.readFile(path, cb);
} else {
return cb(err, result);
}
});
};
export default function* compile(code) {
// Setup webpack
//create a directory structure in MemoryFS that matches
//the real filesystem
const rootDir = root.toString();
//write code snippet to memoryfs
const outputName = `file.js`;
const entry = path.join(rootDir, outputName);
const rootExists = memFs.existsSync(rootDir);
if (!rootExists) {
memFs.mkdirpSync(rootDir);
}
memFs.writeFileSync(entry, code);
//point webpack to memoryfs for the entry file
const compiler = webpack({
entry: entry,
output: {
filename: outputName
},
module: {
loaders: [
{ test: /\.json$/, loader: 'json' }
]
}
});
compiler.run = thenify(compiler.run);
//direct webpack to use memoryfs for file input
compiler.inputFileSystem = memFs;
compiler.resolvers.normal.fileSystem = memFs;
//direct webpack to output to memoryfs rather than to disk
compiler.outputFileSystem = memFs;
const stats = yield compiler.run();
//remove entry from memory. we're done with it
memFs.unlinkSync(entry);
const errors = stats.compilation.errors;
if (errors && errors.length > 0) {
//if there are errors, throw the first one
throw errors[0];
}
//retrieve the output of the compilation
const res = stats.compilation.assets[outputName].source();
return res;
}
Usage
var code = "var _ = require('underscore'); console.log(_);";
var bundle = yield compile(code); //is a valid js bundle containing the underscore source and a log statement logging _.
If there's not a better way, then I'll definitely encapsulate this into a subclass of MemoryFS, but I'm hoping there's a more sane way to accomplish this with Webpack's api.
Instead of memory-fs, the combination of unionfs/memfs/linkfs should help.
https://npmjs.com/unionfs
https://npmjs.com/memfs
https://npmjs.com/linkfs
I have created this snippet untested. I think you want the inputFS to be the real one and the output fs to be the in memory one. On the other hand you want all the dependencies of file.js to be constructed separately. For that I figured the webpack.optimize.CommonsChunkPlugin plugin could help. I expect webpack to write everything to the memory. I hope it works.
import webpack from 'webpack';
import MemoryFS from 'memory-fs';
import thenify from 'thenify';
import realFS from 'fs';
function* compile(code) {
const fs = new MemoryFS();
const compiler = webpack({
entry: {
file: '/file.js',
vendor: [
'underscore',
'other-package-name'
]
},
output: {
path: '/build',
filename: '[name].js'
},
module: {
loaders: [
{ test: /\.json$/, loader: 'json' }
],
},
plugins: [
new webpack.optimize.CommonsChunkPlugin('vendor', 'vendor.js')
]
});
compiler.run = thenify(compiler.run);
compiler.inputFileSystem = realFS;
compiler.resolvers.normal.fileSystem = fs; //this is needed for memfs
compiler.outputFileSystem = fs;
const stats = yield compiler.run();
//retrieve the output of the compilation
const res = stats.compilation.assets['file.js'].source();
return res;
}
You're using MemoryFS, which is a JavaScript reimplementation of a feature normally handled by the Operating System. I wonder, could you mount a directory using tmpfs at the Operating System level, then use that? webpack would then not know or care that the input file is actually stored in memory.
Assuming that you have mounted a memory-based filesystem at /media/memory, the webpack configuration code could be as simple as this:
resolve: {
root: ['/media/memory', ...other paths...],
},
output: {
path: '/wherever/you/want/the/output/files'
}
}
This approach also has a hidden benefit: If you want to debug the input code, you just mount /media/memory with a non-RAM-based filesystem and you can see what's being generated.
I know it's late but for the record here comes a code snippet.
import * as fs from 'fs';
import { resolve } from 'path';
import { Volume } from 'memfs';
import { ufs } from 'unionfs';
const volume = Volume.fromJSON({
[resolve(process.cwd(), 'test.js')]: 'this file is on memory not on disk'
});
ufs.use(fs).use(volume);
// Reads from memory
console.log(ufs.readFileSync(resolve(process.cwd(), 'test.js'), 'utf8'));
// Reads from disk
console.log(ufs.readFileSync(resolve(process.cwd(), 'package.json'), 'utf8'));
// Writing into memory
volume.writeFileSync(resolve(process.cwd(), 'test.memory'), 'This should be
on memory');
console.log(ufs.readFileSync(resolve(process.cwd(), 'test.memory'), 'utf8'));
// Writing into disk
ufs.writeFileSync(resolve(process.cwd(), 'test.disk'), 'This should be on disk');
console.log(ufs.readFileSync(resolve(process.cwd(), 'test.disk'), 'utf8'));
Hers the console output:
user1#pc playground % node inMem.mjs
this file is on memory not on disk
{
"name": "playground",
"version": "1.0.0",
"main": "index.js",
"license": "MIT",
"dependencies": {
"memfs": "^3.3.0",
"unionfs": "^4.4.0"
}
}
This should be on memory
This should be on disk
user1#pc playground % ls .
inMem.mjs node_modules package.json yarn.lock

Categories

Resources