Right now really need help to understand webpack, although I'm trying so much, im still not understand how I can use webpack as a task runner rather than just module bundler. This is a learning process, so please lets not discuss of why gulp/webpack is better than other for this kind of job.
You see the code is actually pretty simple, the idea is to crawl every htmlfile based on a template that injected with a variable from css and js with the same name. This gulpfile is what I use in order to create Polymerjs Html File.
Is there any way I could convert this into Webpack specific build tools? Where should I put the information to crawl the folder in the webpack? Should I put it into index.js?
I understand the concept of using a specific entry files for Webpack, but when facing this kind of usecase, what should I do?
gulp.task('templates', function() {
// crawl folder pathToFolder
readdir(pathToFolder, ['*.scss', '*.js', '*.json'], function(err, files) {
// loop every file founds
files.forEach(function(file) {
let data = file.substr(10);
// make sure the path is correct when used on windows device.
data = data.replace(String.fromCharCode(92), String.fromCharCode(47));
let index = data.lastIndexOf('/');
let path = data.substr(0, index);
let scss = file.substr(0, file.lastIndexOf('.')) + '.scss';
let js = file.substr(0, file.lastIndexOf('.')) + '.js';
let json = file.substr(0, file.lastIndexOf('.')) + '.json';
// process all files into their respective loader and squash it into one variable to use on the template.
let process = {
css: '',
form: '',
js: '',
json: '',
};
if (fs.existsSync(scss)) {
process = Object.assign(process, {
css: sass.compiler.renderSync({
file: scss,
}).css,
});
}
if (fs.existsSync(js)) {
process = Object.assign(process, {
js: fs.readFileSync(js, 'utf8'),
});
}
if (fs.existsSync(json)) {
let x = gulp.src('./ramen-renderer.html')
.pipe(template({
json: json,
}));
process = Object.assign(process, {
json: x,
});
// jsonProcess = Object.assign(jsonProcess, {json: fs.readFileSync(json, "utf8")});
}
// render the final html path with gulp-template
return gulp.src(file)
.pipe(template(process))
.pipe(gulp.dest('src/' + path));
});
});
});
webpack is not a task runner but a module bundler. That means you need another way to run tasks. The most common way to do this is to use NPM for this. You define your tasks in your package.json under the scripts section.
So, I'm moving from grunt to gulp (or trying to anyway), and I'm having trouble getting gulp to do what I'm doing in grunt. Specifically the $templateCache stuff.
My angular app is broken up into several components/modules. Each module contains everything it needs to run (controllers, directives, partials, scss, etc.).
Using Grunt, I've been able to boil each module down into 5 files:
module.min.css // all module scss files compiled and concatenated
module.min.js // all module controllers, directives, services, etc. concatenated
module.tpls.min.js // all partials in $templateCache for this module
module.mocks.min.js // all unit test mock objects for this module
module.specs.min.js // all unit test specs for this module
This has worked really well for 2 years now and been a cornerstone of my modular architecture. My only reasons to try out gulp was 1) Curiosity, 2) My grunt file is getting kinda hairy as we add in deployment and environment specific stuff and so far gulp has really slimmed that down.
For the most part, I've figured out how to do all my grunt tasks in gulp, but I'm having trouble figuring out how to generate a template cache file for each module. All the gulp-ng|angular-templates|templatecache plugins take all my partials and create one file. I'd like to take all my files under module/partials/*.html and create a single module.tpls.min.js; and do that for each module.
This was actually a problem with grunt too, but I figured it out with grunt.file.expand().forEach() like this:
grunt.registerTask('prepModules', '...', function(){
// loop through our modules directory and create subtasks
// for each module, modifying tasks that affect modules.
grunt.file.expand("src/js/modules/*").forEach(function (dir) {
// get the module name by looking at the directory we're in
var mName = dir.substr(dir.lastIndexOf('/') + 1);
// add ngtemplate subtasks for each module, turning
// all module partials into $templateCache objects
ngtemplates[mName] = {
module: mName,
src: dir + "/partials/**/*.html",
dest: 'dev/modules/' + mName + '/' + mName + '.tpls.min.js'
};
grunt.config.set('ngtemplates', ngtemplates);
});
});
My current gulp for this same task:
var compileTemplates = gulp.src('./src/js/modules/**/partials/*.html', {base:'.'})
.pipe(ngTemplates())
.pipe(gulp.dest('.'));
I've only really looked at the options, but none of them seemed to do what I wanted. They were all around changing the file name, or the final destination of the file, or a module name, or whatever else; nothing that said anything about doing it for only the directory it happens to be in.
I had thought about using gulp-rename because it worked well for me when doing the CSS compilation:
var compileScss = gulp.src('./src/js/modules/**/scss/*.scss', {base:'.'})
.pipe(sass({includePaths: ['./src/scss']}))
.pipe(rename(function(path){
path.dirname = path.dirname.replace(/scss/,'css');
}))
.pipe(gulp.dest('.'));
However, when I pipe rename() after doing ngTemplates() it only has the path of the final output file (one log entry). When you console.log() path after sass(), it has all the paths of all the files that it found (lots of log entries).
Any ideas? Thanks!
This SO post has the correct answer, but the wasn't coming up in my searches for this specific usage. I was going to vote to close my question, but since someone else might search using my own specific terms (since I did), it seems more appropriate to leave it alone and just redirect to the original question as well as show how I solved my own particular problem.
var fs = require('fs');
var ngTemplates = require('gulp-ng-templates');
var rename = require('gulp-rename');
var modulesDir = './src/js/modules/';
var getModules = function(dir){
return fs.readdirSync(dir)
.filter(function(file){
return fs.statSync(path.join(dir, file)).isDirectory();
});
};
gulp.task('default', function(){
var modules = getModules(modulesDir);
var moduleTasks = modules.map(function(folder){
// get all partials for this module
// parse into $templateCache file
// rename to be /dev/modules/_____/______.tpls.min.js
return gulp.src(modulesDir + folder + '/partials/*.html', {basedir:'.'})
.pipe(ngTemplates({module:folder}))
.pipe(rename(function(path){
path.dirname = './dev/apps/' + folder + '/';
path.basename = folder + '.tpls.min';
}))
.pipe(gulp.dest('.'));
});
});
It's essentially like the tasks per folder recipe but with a change to use gulp-ng-templates. I'll probably be using this same pattern for my SCSS and JS now that I'm more aware of it.
Seems like the gulp equivalent of grunt.file.expand().forEach().
Whenever I deal with scss/sass for gulp tasks, I will only put one scss file as the source parameter. This parameter file is composed of a list of imports. This way you don't need to rely on gulp to concat the scss file contents for you.
//in gulpfile
gulp.src('./src/js/modules/**/scss/main.scss', {base:'.'})
//in main.scss
#import 'a', 'b', 'c';
a, b, and c would represent your other scss files.
I have list of glob patterns:
var assets = [
'../projects.a/dir1/dir2/file1.js',
'../projects.b/dir1/dir2/file2.js',
'./app/scripts/dir1/dir2/module.js'
];
I want to create a task that will copy all the files that represented by those patterns to my dest folder, and keep the structure. For example, in my dest folder I want the following:
'dest/projects.a/dir1/dir2/file1.js',
'dest/projects.b/dir1/dir2/file2.js',
'dest/app/scripts/dir1/dir2/module.js'
I built the following task:
gulp.task('scripts', ['clean'], function() {
return gulp.src(assets)
.pipe(gulp.dest('dest'));
});
The problem is that I get under 'dest' only one level of directories. For example:\
'dest/dir2/file1.js',
'dest/dir2/file2.js',
'dest/dir2/module.js'
Any idea what is wrong? How can I solve it?
What is the common approach of handling assets from different folders with gulp?
UPDATE
Looking on Looking for way to copy files in gulp and rename based on parent directory, I tried to set {base: './'} as a parameter to gulp.src(assets, {base: './'}) but I get the files that under my current directory nested under another folder. I didn't get the needed behavior. Any idea?
You could handle the files which are not under the current directory separately. You could use separate tasks, or join streams using event-stream:
var projectAssets = [
'../projects.a/dir1/dir2/file1.js',
'../projects.b/dir1/dir2/file2.js'
];
var assets = [
'./app/scripts/dir1/dir2/module.js'
];
var es = require('event-stream');
gulp.task('scripts', ['clean'], function() {
return es.merge(gulp.src(assets, {base: './'}), gulp.src(projectAssets, {base: '../'}))
.pipe(gulp.dest('dest'));
});
Say, for example, you are building a project on Backbone or whatever and you need to load scripts in a certain order, e.g. underscore.js needs to be loaded before backbone.js.
How do I get it to concat the scripts so that they’re in order?
// JS concat, strip debugging and minify
gulp.task('scripts', function() {
gulp.src(['./source/js/*.js', './source/js/**/*.js'])
.pipe(concat('script.js'))
.pipe(stripDebug())
.pipe(uglify())
.pipe(gulp.dest('./build/js/'));
});
I have the right order of scripts in my source/index.html, but since files are organized by alphabetic order, gulp will concat underscore.js after backbone.js, and the order of the scripts in my source/index.html does not matter, it looks at the files in the directory.
So does anyone have an idea on this?
Best idea I have is to rename the vendor scripts with 1, 2, 3 to give them the proper order, but I am not sure if I like this.
As I learned more I found Browserify is a great solution, it can be a pain at first but it’s great.
I had a similar problem recently with Grunt when building my AngularJS app. Here's a question I posted.
What I ended up doing is to explicitly list the files in order in the grunt config. The config file will then look like this:
[
'/path/to/app.js',
'/path/to/mymodule/mymodule.js',
'/path/to/mymodule/mymodule/*.js'
]
Grunt is able to figure out which files are duplicates and not include them. The same technique will work with Gulp as well.
Another thing that helps if you need some files to come after a blob of files, is to exclude specific files from your glob, like so:
[
'/src/**/!(foobar)*.js', // all files that end in .js EXCEPT foobar*.js
'/src/js/foobar.js',
]
You can combine this with specifying files that need to come first as explained in Chad Johnson's answer.
I have used the gulp-order plugin but it is not always successful as you can see by my stack overflow post gulp-order node module with merged streams. When browsing through the Gulp docs I came across the streamque module which has worked quite well for specifying order of in my case concatenation. https://github.com/gulpjs/gulp/blob/master/docs/recipes/using-multiple-sources-in-one-task.md
Example of how I used it is below
var gulp = require('gulp');
var concat = require('gulp-concat');
var handleErrors = require('../util/handleErrors');
var streamqueue = require('streamqueue');
gulp.task('scripts', function() {
return streamqueue({ objectMode: true },
gulp.src('./public/angular/config/*.js'),
gulp.src('./public/angular/services/**/*.js'),
gulp.src('./public/angular/modules/**/*.js'),
gulp.src('./public/angular/primitives/**/*.js'),
gulp.src('./public/js/**/*.js')
)
.pipe(concat('app.js'))
.pipe(gulp.dest('./public/build/js'))
.on('error', handleErrors);
});
With gulp-useref you can concatenate every script declared in your index file, in the order in which you declare it.
https://www.npmjs.com/package/gulp-useref
var $ = require('gulp-load-plugins')();
gulp.task('jsbuild', function () {
var assets = $.useref.assets({searchPath: '{.tmp,app}'});
return gulp.src('app/**/*.html')
.pipe(assets)
.pipe($.if('*.js', $.uglify({preserveComments: 'some'})))
.pipe(gulp.dest('dist'))
.pipe($.size({title: 'html'}));
});
And in the HTML you have to declare the name of the build file you want to generate, like this:
<!-- build:js js/main.min.js -->
<script src="js/vendor/vendor.js"></script>
<script src="js/modules/test.js"></script>
<script src="js/main.js"></script>
In your build directory you will have the reference to main.min.js which will contain vendor.js, test.js, and main.js
The sort-stream may also be used to ensure specific order of files with gulp.src. Sample code that puts the backbone.js always as the last file to process:
var gulp = require('gulp');
var sort = require('sort-stream');
gulp.task('scripts', function() {
gulp.src(['./source/js/*.js', './source/js/**/*.js'])
.pipe(sort(function(a, b){
aScore = a.path.match(/backbone.js$/) ? 1 : 0;
bScore = b.path.match(/backbone.js$/) ? 1 : 0;
return aScore - bScore;
}))
.pipe(concat('script.js'))
.pipe(stripDebug())
.pipe(uglify())
.pipe(gulp.dest('./build/js/'));
});
I just add numbers to the beginning of file name:
0_normalize.scss
1_tikitaka.scss
main.scss
It works in gulp without any problems.
I have my scripts organized in different folders for each package I pull in from bower, plus my own script for my app. Since you are going to list the order of these scripts somewhere, why not just list them in your gulp file? For new developers on your project, it's nice that all your script end-points are listed here. You can do this with gulp-add-src:
gulpfile.js
var gulp = require('gulp'),
less = require('gulp-less'),
minifyCSS = require('gulp-minify-css'),
uglify = require('gulp-uglify'),
concat = require('gulp-concat'),
addsrc = require('gulp-add-src'),
sourcemaps = require('gulp-sourcemaps');
// CSS & Less
gulp.task('css', function(){
gulp.src('less/all.less')
.pipe(sourcemaps.init())
.pipe(less())
.pipe(minifyCSS())
.pipe(sourcemaps.write('source-maps'))
.pipe(gulp.dest('public/css'));
});
// JS
gulp.task('js', function() {
gulp.src('resources/assets/bower/jquery/dist/jquery.js')
.pipe(addsrc.append('resources/assets/bower/bootstrap/dist/js/bootstrap.js'))
.pipe(addsrc.append('resources/assets/bower/blahblah/dist/js/blah.js'))
.pipe(addsrc.append('resources/assets/js/my-script.js'))
.pipe(sourcemaps.init())
.pipe(concat('all.js'))
.pipe(uglify())
.pipe(sourcemaps.write('source-maps'))
.pipe(gulp.dest('public/js'));
});
gulp.task('default',['css','js']);
Note: jQuery and Bootstrap added for demonstration purposes of order. Probably better to use CDNs for those since they are so widely used and browsers could have them cached from other sites already.
Try stream-series. It works like merge-stream/event-stream.merge() except that instead of interleaving, it appends to the end. It doesn't require you to specify the object mode like streamqueue, so your code comes out cleaner.
var series = require('stream-series');
gulp.task('minifyInOrder', function() {
return series(gulp.src('vendor/*'),gulp.src('extra'),gulp.src('house/*'))
.pipe(concat('a.js'))
.pipe(uglify())
.pipe(gulp.dest('dest'))
});
merge2 looks like the only working and maintained ordered stream merging tool at the moment.
Update 2020
The APIs are always changing, some libraries become unusable or contain vulnerabilities, or their dependencies contain vulnerabilities, that are not fixed for years. For text files manipulations you'd better use custom NodeJS scripts and popular libraries like globby and fs-extra along with other libraries without Gulp, Grunt, etc wrappers.
import globby from 'globby';
import fs from 'fs-extra';
async function bundleScripts() {
const rootPaths = await globby('./source/js/*.js');
const otherPaths = (await globby('./source/**/*.js'))
.filter(f => !rootFiles.includes(f));
const paths = rootPaths.concat(otherPaths);
const files = Promise.all(
paths.map(
// Returns a Promise
path => fs.readFile(path, {encoding: 'utf8'})
)
);
let bundle = files.join('\n');
bundle = uglify(bundle);
bundle = whatever(bundle);
bundle = bundle.replace(/\/\*.*?\*\//g, '');
await fs.outputFile('./build/js/script.js', bundle, {encoding: 'utf8'});
}
bundleScripts.then(() => console.log('done');
An alternative method is to use a Gulp plugin created specifically for this problem. https://www.npmjs.com/package/gulp-ng-module-sort
It allows you to sort your scripts by adding in a .pipe(ngModuleSort()) as such:
var ngModuleSort = require('gulp-ng-module-sort');
var concat = require('gulp-concat');
gulp.task('angular-scripts', function() {
return gulp.src('./src/app/**/*.js')
.pipe(ngModuleSort())
.pipe(concat('angularAppScripts.js))
.pipe(gulp.dest('./dist/));
});
Assuming a directory convention of:
|——— src/
| |——— app/
| |——— module1/
| |——— sub-module1/
| |——— sub-module1.js
| |——— module1.js
| |——— module2/
| |——— sub-module2/
| |——— sub-module2.js
| |——— sub-module3/
| |——— sub-module3.js
| |——— module2.js
| |——— app.js
Hope this helps!
For me I had natualSort() and angularFileSort() in pipe which was reordering the files. I removed it and now it works fine for me
$.inject( // app/**/*.js files
gulp.src(paths.jsFiles)
.pipe($.plumber()), // use plumber so watch can start despite js errors
//.pipe($.naturalSort())
//.pipe($.angularFilesort()),
{relative: true}))
I just use gulp-angular-filesort
function concatOrder() {
return gulp.src('./build/src/app/**/*.js')
.pipe(sort())
.pipe(plug.concat('concat.js'))
.pipe(gulp.dest('./output/'));
}
I'm in a module environnement where all are core-dependents using gulp.
So, the core module needs to be appended before the others.
What I did:
Move all the scripts to an src folder
Just gulp-rename your core directory to _core
gulp is keeping the order of your gulp.src, my concat src looks like this:
concat: ['./client/src/js/*.js', './client/src/js/**/*.js', './client/src/js/**/**/*.js']
It'll obviously take the _ as the first directory from the list (natural sort?).
Note (angularjs):
I then use gulp-angular-extender to dynamically add the modules to the core module.
Compiled it looks like this:
angular.module('Core', ["ui.router","mm.foundation",(...),"Admin","Products"])
Where Admin and Products are two modules.
if you would like to order third party libraries dependencies, try wiredep. This package basically checks each package dependency in bower.json then wire them up for you.
I tried several solutions from this page, but none worked. I had a series of numbered files which I simply wanted be ordered by alphabetical foldername so when piped to concat() they'd be in the same order. That is, preserve the order of the globbing input. Easy, right?
Here's my specific proof-of-concept code (print is just to see the order printed to the cli):
var order = require('gulp-order');
var gulp = require('gulp');
var print = require('gulp-print').default;
var options = {};
options.rootPath = {
inputDir: process.env.INIT_CWD + '/Draft',
inputGlob: '/**/*.md',
};
gulp.task('default', function(){
gulp.src(options.rootPath.inputDir + options.rootPath.inputGlob, {base: '.'})
.pipe(order([options.rootPath.inputDir + options.rootPath.inputGlob]))
.pipe(print());
});
The reason for the madness of gulp.src? I determined that gulp.src was running async when I was able to use a sleep() function (using a .map with sleeptime incremented by index) to order the stream output properly.
The upshot of the async of src mean dirs with more files in it came after dirs with fewer files, because they took longer to process.
In my gulp setup, I'm specifying the vendor files first and then specifying the (more general) everything, second. And it successfully puts the vendor js before the other custom stuff.
gulp.src([
// vendor folder first
path.join(folder, '/vendor/**/*.js'),
// custom js after vendor
path.join(folder, '/**/*.js')
])
Apparently you can pass in the "nosort" option to gulp.src gulp.src.
I have a library lib.js that I want to create from lib/a.js and lib/b.js and to be able to use it from a script client.js using var a = require('lib/a.js'); and that it works when I just include the compiled lib.js library before client.js (therefore, lib.js has to declare a require function that knows about lib/a.js)
I guess I have to use external and alias but I am not sure what is the proper way to do it
Also, is it possible to have a Gulp file that creates all the alias automatically for the folders in my library? eg. creates an alias for all the files in the lib/ dir?
Here are a couple of gulp tasks that would help to build your common lib.js and the client.js bundles separately.
Note that you have to tell browserify to b.require() lib/*.js when bundling lib.js, and you have to tell it to b.external() the libraries that will be loaded separately when bundling client.js
var path = require('path');
var gulp = require('gulp');
var browserify = require('browserify');
var concat = require('gulp-concat');
var transform = require('vinyl-transform');
gulp.task('build-lib', function () {
// use `vinyl-transform` to wrap around the regular ReadableStream returned by b.bundle();
// so that we can use it down a vinyl pipeline as a vinyl file object.
// `vinyl-transform` takes care of creating both streaming and buffered vinyl file objects.
var browserified = transform(function(filename) {
// basename, for eg: 'a.js'
var basename = path.basename(filename);
// define the exposed name that your client.js would use to require();
// for eg: require('lib/a.js'); // -> exposed name should be 'lib/a.js'
var expose = 'lib/' + basename;
return browserify(filename)
.require(filename, { expose: expose})
.bundle();
});
return gulp.src(['./lib/*.js'])
.pipe(browserified)
.pipe(concat('lib.js'))
.pipe(gulp.dest('./dist'));
});
gulp.task('build-client', function () {
var browserified = transform(function(filename) {
// filename = './client.js'
// let browserify know that lib/a.js and and lib/b.js are external files
// and will be loaded externally (in your case, by loading the bundled lib.js
// for eg: <script src='dist/lib.js'>)
return browserify(filename)
.external('lib/a.js')
.external('lib/b.js')
.bundle();
});
return gulp.src(['./client.js'])
.pipe(browserified)
.pipe(gulp.dest('./dist'));
});
gulp.task('default', ['build-lib', 'build-client']);
Are you looking for external requires?
To use with gulp-browserify, check the README
.on('prebundle', function(bundle) {
bundle.external('domready');
bundle.external('react');
})
Should work with bundle.require as well.