Good day!
I want to automate the conversion of images to webp format. To avoid doing it manually or via online converters, I decided to use gulp 4 and gulp-webp.
This is the structure of nesting folders in my project:
I want Gulp, when it finds a picture, so that it creates a folder called "webp" at the same nesting level and places the converted picture in this folder.
I want the following result:
My Gulpfile.js:
let gulp = require('gulp'),
webp = require('gulp-webp');
gulp.task('webp', () => {
// './dev/img/**/*.{png,gif,jpg}' - all files in img and all files in subfolders in img
return gulp.src('./dev/img/**/*.{png,gif,jpg}')
.pipe(webp())
.pipe(gulp.dest(gulp.src)) //something like this, but it doesn't work
}
);
This can be done with the help of gulp-rename.
To install gulp-rename run:
npm i -D gulp-rename
Then in your gulpfile.js add the import:
const rename = require('gulp-rename');
Then change your stream like this:
return gulp.src('./dev/img/**/*.{png,gif,jpg}')
.pipe(webp())
.pipe(rename({ prefix: 'webp/' }))
.pipe(gulp.dest('./dev/img'));
The prefix option inserts "webp/" before the bare filename after the dest target "./dev/img".
I have a task:
gulp.task('compile_scss, function() {
return gulp.src('/admin_app/scss/*.scss')
.pipe(sass())
.pipe(dest('/admin_app/css/'))
});
When I am adding new empty ".scss" file to '/admin_app/scss/' and running task from above, empty ".scss" files is copied to destination folder. If file is not empty everything is ok: a valid css file( with ".css" extension) is compiled and no ".scss" files are copied. The problem is when I add new ".scss" file to "/admin_app/scss/" directory, a "watch" task is triggered, and because file is empty, it is copied to destination directory. As a result, a lot of unneeded garbage is dest folder. Why this happens and how can I get rid of it?
UPDATED
My "watch" and "default" tasks:
gulp.task('watch', ['compile_scss'], function() {
apps.forEach(function(appName) {
gulp.watch('/admin_app/scss/*.scss', ['compile_scss']);
});
});
gulp.task('default', ['watch']);
One way to solve this problem would be to simply filter the empty files.
Try something like this:
var filter = require('gulp-filter'),
gulp.task('compile_scss, function() {
return gulp.src('/admin_app/scss/*.scss')
.pipe(filter(function(a){ return a.stat && a.stat.size }))
.pipe(sass())
.pipe(dest('/admin_app/css/'))
});
There's also a plugin specifically for this purpose. You can use it like this:
var clip = require('gulp-clip-empty-files'),
gulp.task('compile_scss, function() {
return gulp.src('/admin_app/scss/*.scss')
.pipe(clip())
.pipe(sass())
.pipe(dest('/admin_app/css/'))
});
In addition: there seem to have been several reports of problems in gulp-sass and underlying libraries when compiling empty files. There is a Github issue for gulp-sass, reporting this should be solved in the 2.x versions of the plugin. If you're already running 2.x, the problem you are facing might be an issue introduced by solving the original problem.
If you add empty scss files in your sass folder, prefix them with underscore: _empty.scss.
See "Partials" here: http://sass-lang.com/guide#topic-4
You can create partial Sass files that contain little snippets of CSS
that you can include in other Sass files. This is a great way to
modularize your CSS and help keep things easier to maintain. A partial
is simply a Sass file named with a leading underscore. You might name
it something like _partial.scss. The underscore lets Sass know that
the file is only a partial file and that it should not be generated
into a CSS file. Sass partials are used with the #import directive.
Say, for example, you are building a project on Backbone or whatever and you need to load scripts in a certain order, e.g. underscore.js needs to be loaded before backbone.js.
How do I get it to concat the scripts so that they’re in order?
// JS concat, strip debugging and minify
gulp.task('scripts', function() {
gulp.src(['./source/js/*.js', './source/js/**/*.js'])
.pipe(concat('script.js'))
.pipe(stripDebug())
.pipe(uglify())
.pipe(gulp.dest('./build/js/'));
});
I have the right order of scripts in my source/index.html, but since files are organized by alphabetic order, gulp will concat underscore.js after backbone.js, and the order of the scripts in my source/index.html does not matter, it looks at the files in the directory.
So does anyone have an idea on this?
Best idea I have is to rename the vendor scripts with 1, 2, 3 to give them the proper order, but I am not sure if I like this.
As I learned more I found Browserify is a great solution, it can be a pain at first but it’s great.
I had a similar problem recently with Grunt when building my AngularJS app. Here's a question I posted.
What I ended up doing is to explicitly list the files in order in the grunt config. The config file will then look like this:
[
'/path/to/app.js',
'/path/to/mymodule/mymodule.js',
'/path/to/mymodule/mymodule/*.js'
]
Grunt is able to figure out which files are duplicates and not include them. The same technique will work with Gulp as well.
Another thing that helps if you need some files to come after a blob of files, is to exclude specific files from your glob, like so:
[
'/src/**/!(foobar)*.js', // all files that end in .js EXCEPT foobar*.js
'/src/js/foobar.js',
]
You can combine this with specifying files that need to come first as explained in Chad Johnson's answer.
I have used the gulp-order plugin but it is not always successful as you can see by my stack overflow post gulp-order node module with merged streams. When browsing through the Gulp docs I came across the streamque module which has worked quite well for specifying order of in my case concatenation. https://github.com/gulpjs/gulp/blob/master/docs/recipes/using-multiple-sources-in-one-task.md
Example of how I used it is below
var gulp = require('gulp');
var concat = require('gulp-concat');
var handleErrors = require('../util/handleErrors');
var streamqueue = require('streamqueue');
gulp.task('scripts', function() {
return streamqueue({ objectMode: true },
gulp.src('./public/angular/config/*.js'),
gulp.src('./public/angular/services/**/*.js'),
gulp.src('./public/angular/modules/**/*.js'),
gulp.src('./public/angular/primitives/**/*.js'),
gulp.src('./public/js/**/*.js')
)
.pipe(concat('app.js'))
.pipe(gulp.dest('./public/build/js'))
.on('error', handleErrors);
});
With gulp-useref you can concatenate every script declared in your index file, in the order in which you declare it.
https://www.npmjs.com/package/gulp-useref
var $ = require('gulp-load-plugins')();
gulp.task('jsbuild', function () {
var assets = $.useref.assets({searchPath: '{.tmp,app}'});
return gulp.src('app/**/*.html')
.pipe(assets)
.pipe($.if('*.js', $.uglify({preserveComments: 'some'})))
.pipe(gulp.dest('dist'))
.pipe($.size({title: 'html'}));
});
And in the HTML you have to declare the name of the build file you want to generate, like this:
<!-- build:js js/main.min.js -->
<script src="js/vendor/vendor.js"></script>
<script src="js/modules/test.js"></script>
<script src="js/main.js"></script>
In your build directory you will have the reference to main.min.js which will contain vendor.js, test.js, and main.js
The sort-stream may also be used to ensure specific order of files with gulp.src. Sample code that puts the backbone.js always as the last file to process:
var gulp = require('gulp');
var sort = require('sort-stream');
gulp.task('scripts', function() {
gulp.src(['./source/js/*.js', './source/js/**/*.js'])
.pipe(sort(function(a, b){
aScore = a.path.match(/backbone.js$/) ? 1 : 0;
bScore = b.path.match(/backbone.js$/) ? 1 : 0;
return aScore - bScore;
}))
.pipe(concat('script.js'))
.pipe(stripDebug())
.pipe(uglify())
.pipe(gulp.dest('./build/js/'));
});
I just add numbers to the beginning of file name:
0_normalize.scss
1_tikitaka.scss
main.scss
It works in gulp without any problems.
I have my scripts organized in different folders for each package I pull in from bower, plus my own script for my app. Since you are going to list the order of these scripts somewhere, why not just list them in your gulp file? For new developers on your project, it's nice that all your script end-points are listed here. You can do this with gulp-add-src:
gulpfile.js
var gulp = require('gulp'),
less = require('gulp-less'),
minifyCSS = require('gulp-minify-css'),
uglify = require('gulp-uglify'),
concat = require('gulp-concat'),
addsrc = require('gulp-add-src'),
sourcemaps = require('gulp-sourcemaps');
// CSS & Less
gulp.task('css', function(){
gulp.src('less/all.less')
.pipe(sourcemaps.init())
.pipe(less())
.pipe(minifyCSS())
.pipe(sourcemaps.write('source-maps'))
.pipe(gulp.dest('public/css'));
});
// JS
gulp.task('js', function() {
gulp.src('resources/assets/bower/jquery/dist/jquery.js')
.pipe(addsrc.append('resources/assets/bower/bootstrap/dist/js/bootstrap.js'))
.pipe(addsrc.append('resources/assets/bower/blahblah/dist/js/blah.js'))
.pipe(addsrc.append('resources/assets/js/my-script.js'))
.pipe(sourcemaps.init())
.pipe(concat('all.js'))
.pipe(uglify())
.pipe(sourcemaps.write('source-maps'))
.pipe(gulp.dest('public/js'));
});
gulp.task('default',['css','js']);
Note: jQuery and Bootstrap added for demonstration purposes of order. Probably better to use CDNs for those since they are so widely used and browsers could have them cached from other sites already.
Try stream-series. It works like merge-stream/event-stream.merge() except that instead of interleaving, it appends to the end. It doesn't require you to specify the object mode like streamqueue, so your code comes out cleaner.
var series = require('stream-series');
gulp.task('minifyInOrder', function() {
return series(gulp.src('vendor/*'),gulp.src('extra'),gulp.src('house/*'))
.pipe(concat('a.js'))
.pipe(uglify())
.pipe(gulp.dest('dest'))
});
merge2 looks like the only working and maintained ordered stream merging tool at the moment.
Update 2020
The APIs are always changing, some libraries become unusable or contain vulnerabilities, or their dependencies contain vulnerabilities, that are not fixed for years. For text files manipulations you'd better use custom NodeJS scripts and popular libraries like globby and fs-extra along with other libraries without Gulp, Grunt, etc wrappers.
import globby from 'globby';
import fs from 'fs-extra';
async function bundleScripts() {
const rootPaths = await globby('./source/js/*.js');
const otherPaths = (await globby('./source/**/*.js'))
.filter(f => !rootFiles.includes(f));
const paths = rootPaths.concat(otherPaths);
const files = Promise.all(
paths.map(
// Returns a Promise
path => fs.readFile(path, {encoding: 'utf8'})
)
);
let bundle = files.join('\n');
bundle = uglify(bundle);
bundle = whatever(bundle);
bundle = bundle.replace(/\/\*.*?\*\//g, '');
await fs.outputFile('./build/js/script.js', bundle, {encoding: 'utf8'});
}
bundleScripts.then(() => console.log('done');
An alternative method is to use a Gulp plugin created specifically for this problem. https://www.npmjs.com/package/gulp-ng-module-sort
It allows you to sort your scripts by adding in a .pipe(ngModuleSort()) as such:
var ngModuleSort = require('gulp-ng-module-sort');
var concat = require('gulp-concat');
gulp.task('angular-scripts', function() {
return gulp.src('./src/app/**/*.js')
.pipe(ngModuleSort())
.pipe(concat('angularAppScripts.js))
.pipe(gulp.dest('./dist/));
});
Assuming a directory convention of:
|——— src/
| |——— app/
| |——— module1/
| |——— sub-module1/
| |——— sub-module1.js
| |——— module1.js
| |——— module2/
| |——— sub-module2/
| |——— sub-module2.js
| |——— sub-module3/
| |——— sub-module3.js
| |——— module2.js
| |——— app.js
Hope this helps!
For me I had natualSort() and angularFileSort() in pipe which was reordering the files. I removed it and now it works fine for me
$.inject( // app/**/*.js files
gulp.src(paths.jsFiles)
.pipe($.plumber()), // use plumber so watch can start despite js errors
//.pipe($.naturalSort())
//.pipe($.angularFilesort()),
{relative: true}))
I just use gulp-angular-filesort
function concatOrder() {
return gulp.src('./build/src/app/**/*.js')
.pipe(sort())
.pipe(plug.concat('concat.js'))
.pipe(gulp.dest('./output/'));
}
I'm in a module environnement where all are core-dependents using gulp.
So, the core module needs to be appended before the others.
What I did:
Move all the scripts to an src folder
Just gulp-rename your core directory to _core
gulp is keeping the order of your gulp.src, my concat src looks like this:
concat: ['./client/src/js/*.js', './client/src/js/**/*.js', './client/src/js/**/**/*.js']
It'll obviously take the _ as the first directory from the list (natural sort?).
Note (angularjs):
I then use gulp-angular-extender to dynamically add the modules to the core module.
Compiled it looks like this:
angular.module('Core', ["ui.router","mm.foundation",(...),"Admin","Products"])
Where Admin and Products are two modules.
if you would like to order third party libraries dependencies, try wiredep. This package basically checks each package dependency in bower.json then wire them up for you.
I tried several solutions from this page, but none worked. I had a series of numbered files which I simply wanted be ordered by alphabetical foldername so when piped to concat() they'd be in the same order. That is, preserve the order of the globbing input. Easy, right?
Here's my specific proof-of-concept code (print is just to see the order printed to the cli):
var order = require('gulp-order');
var gulp = require('gulp');
var print = require('gulp-print').default;
var options = {};
options.rootPath = {
inputDir: process.env.INIT_CWD + '/Draft',
inputGlob: '/**/*.md',
};
gulp.task('default', function(){
gulp.src(options.rootPath.inputDir + options.rootPath.inputGlob, {base: '.'})
.pipe(order([options.rootPath.inputDir + options.rootPath.inputGlob]))
.pipe(print());
});
The reason for the madness of gulp.src? I determined that gulp.src was running async when I was able to use a sleep() function (using a .map with sleeptime incremented by index) to order the stream output properly.
The upshot of the async of src mean dirs with more files in it came after dirs with fewer files, because they took longer to process.
In my gulp setup, I'm specifying the vendor files first and then specifying the (more general) everything, second. And it successfully puts the vendor js before the other custom stuff.
gulp.src([
// vendor folder first
path.join(folder, '/vendor/**/*.js'),
// custom js after vendor
path.join(folder, '/**/*.js')
])
Apparently you can pass in the "nosort" option to gulp.src gulp.src.
I have a folder layout such that:
/
-- css/
-- js/
-- apps/
-- -- myFirstApp/
-- -- mySecondApp/
-- -- ...
Each of these are git submodules, and have a corresponding Gruntfile, package.json, etc. What I want to do is the same sequence of commands, but differ depending on the respective package.json.
My command list is this:
npm install
grunt dist
copy app/css/[fileName].css (from package.json) to css/
copy app/js/[fileName].js to js/
copy app/js/[fileName].html to /
Is there a plugin or something I'm overlooking that I can use with grunt to do this? I don't want to do it statically if at all possible -- I'd like to only have to update the submodule list for this to work.
I don't know of any pre-built Grunt task that will do this for you, but writing the task isn't so difficult. You'll need to pull in the Node fs module to deal with the filesystem and obviously there will be some other things to do... here's a general structure for it with some code and some TODO's:
var fs = require("fs"),
path = require("path");
module.exports = function ( grunt ) {
grunt.initConfig({
... // all of your other Grunt config
// options for our new task
copymodulefiles: {
all: {
rootDir: "apps/"
}
}
});
// Here's our custom task definition
grunt.registerMultiTask("copymodulefiles", "Copies files from sub-projects", function() {
var done = this.async(), // tell Grunt this is an async task
root = grunt.config(this.name + "." + this.target + ".rootDir"),
modules = fs.readdirSync(root);
modules.forEach(function(dirName) {
var pkg = fs.readFileSync(root + dirName + path.sep + "package.json", "utf8");
pkgJson = JSON.parse(pkg);
// TODO: find what you need in the pkgJson variable
// copy files from wherever to wherever
// You can write a file like so:
fs.writeFile(theFilenameToWrite, "Contents of the new file", function (err) {
// (check for errors!)
// log it?
grunt.log.ok("file written!");
});
});
// Determine when all are complete and call "done()" to tell Grunt everything's complete
// call Grunt's "done" method to signal task completion
done();
});
};
Try with grunt-shell i found it perfect and did similar tasks like what you are trying to do.
Have a look at my Gruntfile.js configuration what i have written to run shell commands:
shell: {
multiple: {
command: ['bower install',
'mv bower_components/** public/',
'rm -rf bower_components'
].join('&&')
}
}
So here i am running bower, then i am copying its components to public folder and after that i am deleting the bower_components folder. So i guess from here onwards you can customize this script as per your usage.