New to Gulp, using v4 (alpha), learning as I go. I'm thinking of an optimized build process and am at a point where I want to concatenate files (gulp-concat) and delete (del) the source files in 1 step. (my build process copies files from the source location to the build destination, and then only works on files there, rather than directly on source files (as a practice). Once I concat the files, I don't want them in the build, and want to delete them afterwards. FYI, the dest variable just points to the build folder based on flags set when run (./build/ for development builds and ./dist/ for deployment builds.)
gulp.src(dest + 'assets/scripts/**/*.js')
.pipe(gulp.concat('combined.js')
// ??? (stuck here)
Correct me if I'm wrong, but at this point in the pipe, I've lost reference to my "source" files, and only the combined.js file is sitting in the pipe. I could gulp.dest() and save it, but as I understand it that creates a writeStream which I cannot continue to the next .pipe(), right? On top of that, the del plugin does not seem to return a stream, so I'm not sure if I could use it in a pipe, but I just want to ensure it only runs after I've done the concatenation.
Related
I have a fairly simple webpack set up with a bit of a twist. I have a few different ways I can think of to create my intended behavior, but I'm wondering if there are better approaches (I'm still fairly new to webpack).
I have a basic TypeScript/Scss application and all of the src files exist in a src directory. I also have a component library bundle that's dynamically generated through a separate build process (triggered via Node). This bundle also ends up in the src directory (it contains some sass variables and a few other assets that belong in src). This is src/custom-library-bundle. My current webpack set up moves the appropriate bundle files to a public (dist) directory via the CopyWebpackPlugin. My webpack dev server watches for changes and reloads as expected. This all works beautifully.
Now for the twist. I've set up a custom watcher that exists elsewhere to watch for the custom component library bundle changes. When a change occurs there, it triggers that custom build process mentioned above. This (as expected) modifies the src/custom-library-bundle directory and sets off several watches/reloads as the bundle is populated. Technically it works? And the behavior is expected, but ideally, I could tell it to wait until the custom installation work is "done" and only then have it trigger the compilation/reload.
Phew. This isn't an easy one to explain. Here's a webpack config that will (hopefully) help clarify:
devServer: {
contentBase: path.join(__dirname, 'public'),
port: 9000,
host: '127.0.0.1',
after: (app, server) => {
new MyCustomWatcherForOtherThings().watch(() => {
// invoked after the src/custom-library-bundle is done doing its thing (each time)
// now that I know it's done, I want to trigger the normal compilation/reload
})
},
watchOptions: {
ignored: [
/node_modules/
// I've been experimenting with the ignored feature a bit
// path.resolve(__dirname, 'src/custom-library-bundle/')
]
}
}
Ideal Approach: In my most ideal scenario, I want to just manually trigger webpack dev server to do its thing in my custom watch callback; have it ignore the src/custom-library-bundle until I tell it to pay attention. I can't seem to find a way to do this, however. Is this possible?
Alternate Approach #1: I could ignore the src/custom-library-bundle directory, move the updated files to public (not utilizing the webpack approach), and then trigger just a reload when I know that's complete. I don't love this because I want to utilize the same process whether I'm watching or just doing a one-off build (I want everything to end up in the public directory because webpack did that work, not becuase I wrote a script to put it there under specific scenarios). But, let's say I get over that, how can I trigger a browser reload for the dev server? Is this possible?
Alternate Approach #2 This is the one I'm leaning towards but it feels like extra, unnecessary work. I could have my custom build process output to a different directory (one that my webpack set up doesn't care about at all). Once the build process is complete, I could move all the files to src/custom-library-bundle where the watch would pick up 1 change and do a single complication/reload. This gets me so close, but feels like I'm adding a step I don't want to.
Alternate Approach #3? Can you think of a better way to solve this?
Update (including versions):
webpack 4.26.1
webpack-dev-server 3.1.14
Add the following server.sockWrite call to your after method:
devServer: {
after: (app, server) => {
new MyCustomWatcherForOtherThings().watch(() => {
// invoked after the src/custom-library-bundle is done doing its thing (each time)
// now that I know it's done, I want to trigger the normal compilation/reload
// calling this on `server` triggers a full page refresh
server.sockWrite(server.sockets, "content-changed");
});
};
}
I've never found this in the documentation, but one of webpack's core devs mentioned it in a comment on GitHub, so it's sort of sanctioned.
Useful things that webpack provides that come to mind are multi-compiler builds, creating a child compiler from a Compiler or Compilation instance, the DllPlugin and programmatically managing the compiler by calling webpack.watch() or webpack.compile().
the multi-compiler setup is useful when you want to build several compilations in a single run whose outputs are independent of each other
the child compilers allows you to set up dependencies between
compilers and using hooks they allow you to block the parent until,
say, the child compilation finished emitting the latest bundle into
the assets
the DllPlugin allows you to create a compilation and
it's manifest that could produce chunks that can contain modules that
can be used as dependents for yet unbuilt compilations (the manifest
needs to be produced and passed to them beforehand)
programmatically
managing you compiler lets you write a straightforward Node.js script
that could accomplish most of this manually.
If I understood correctly, your webpack compiler doesn't really have any dependencies on your bundle aside from expecting it to have something added to the output assets, so you might consider doing a multi-compiler build. If that doesn't work for you, you can write a simple plugin that creates a child compiler that watches all the component bundle dependencies and emits the built bundle into the main compilation assets on changes. Ultimately, as you mentioned yourself, you could write a simple script and programmatically weave everything together. All of these approaches offload tracking build dependencies to webpack, so if you put the compilers into watch mode webpack will keep track when something needs updating.
If you're interested in taking a closer look at how child compilers are created and used, I would heartily recommend reading through the sources of the html-webpack-plugin plugin. It doesn't seem like the plugin is handling the same kind of build setup as you have, but worth noting is that the HTML plugin works with files that aren't part of the build dependencies (nothing in the sources references or depends on the files/templates that are used for creating the HTML files that are added to the output).
hello and today i want to know if i can have only one node_modules folder for all my node js bots, note they are not in app form just single files that use ..... = require('module'); my point is that i have easily more than 7 bots and i go back and forth improving them and maintaining the node_modules folder can be tedius, is there a way where i can have only one folder with all my modules and just do .... = require('path/to/node_modules/module');
or .... = require('path/to/node_modules/'); could someone please point me in the right direction, because when i update my code on my various bots i have to make sure the module is installed, can i have only one node_modules folder???
I can think of these different options:
You can put all your bot files in the same directory and run them all from that directory so all the modules installed in that directory can be used by all the bot files.
You can install all your modules globally.
You could make one shared directory where you installed all the modules and then explicitly point to that every time you want to load a module by referencing that path. This seems less clean because every single user of a module has to know where to load it from.
For each module you want to use in a project, create symlinks to the place where the modules are actually installed. This doesn't seem like it saves any work over just installing the modules again for each bot though.
In another file say myLibs.js you could do something like
module.exports = {
lib1: require('lib1'),
...
libn: require('libn')
}
then in the original file you would do
var myLibs=require('./myLibs.js');
and reference them by myLibs.lib1...
A little clunky, I know, but it should work.
I am trying to get my gulp file working right and I am not sure whats wrong. Here is the file. https://gist.github.com/digilord/9265640 The section I am having issues with is the js:dev. When a coffeescript file is changed it drops the resulting js file into the app/js directory. The js:dev rule isn't picking up that change. Can anyone tell me what I am missing?
Immediate solution: gulp-watch needs the glob information to find new files. You can pass your glob to its glob option. Something like:
return gulp.src('app/js/*.coffee')
.pipe(watch({glob: 'app/js/*.coffee'}))
Meanwhile if you were to use gulp.watch() it wouldn't pick up new files either but only because of a bug in its dependency, which a fix is coming soon for.
I have to also mention that your gulpfile is needlessly complex. Your coffee and js operations could be in the same stream instead of creating intermediate js files. I also don't really know why some of your streams are repeated twice. Stay dry:
var coffeeGlob = 'app/js/*.coffee';
gulp.task('js:dev', ['clean:dev'], function() {
return gulp.src(coffeeGlob)
.pipe(watch({glob: coffeeGlob}))
.pipe(coffee({bare: true}).on('error', gutil.log))
.pipe(concat('app.js'))
.pipe(gulp.dest('build/js'))
.pipe(reload());
});
As I mentioned on IRC the boilerplate you based it on might not be up to date with best practices, and I recommend basing on something newer like Yeoman's gulp generator's gulpfile.
I have a node.js (v0.6.12) application that starts by evaluating a Javascript file, startup.js. It takes a long time to evaluate startup.js, and I'd like to 'bake it in' to a custom build of Node if possible.
The v8 source directory distributed with Node, node/deps/v8/src, contains a SconScript that can almost be used to do this. On line 302, we have
LIBRARY_FILES = '''
runtime.js
v8natives.js
array.js
string.js
uri.js
math.js
messages.js
apinatives.js
date.js
regexp.js
json.js
liveedit-debugger.js
mirror-debugger.js
debug-debugger.js
'''.split()
Those javascript files are present in the same directory. Something in the build process apparently evaluates them, takes a snapshot of state, and saves it as a byte string in node/out/Release/obj/release/snapshot.cc (on Mac OS). This file seems to be baked into Node.
Some customization of the startup snapshot is possible by altering the SconScript. For example, I can change the definition of the builtin Date.toString by altering date.js. I can even add new global variables by adding startup.js to the list of library files, with contents global.test = 1.
However, I can't put just any javascript code in startup.js. If it contains Date.toString = 1;, an error results even though the code is valid at the node repl:
Build failed: -> task failed (err #2):
{task: libv8.a SConstruct -> libv8.a}
make: *** [program] Error 1
And it obviously can't make use of code that depends on libraries Node adds to v8. global.underscore = require('underscore'); causes the same error.
I'd ideally like a tool, customSnapshot, where customSnapshot startup.js evaluates startup.js with Node and then dumps a snapshot to a file, snapshot.cc, which I can put into the Node source directory. I can then build node and tell it not to rebuild the snapshot.
I just added an option to the mksnapshot command (which runs while you are building V8). The new --extra-file=filename.js flag lets you specify a file that is to be loaded and run in the process and then put in the snapshot. It's on the trunk version of V8, not the 3.11 branch that is being used for node 0.8 so you will have to run node 0.8 with V8 version 3.11. As far as I know at the moment that works, but you will be somewhat on your own.
Please file bugs if you try this and it doesn't work for you.
I have been looking into integrating testing into my app based on RequireJS. I have found this example of how QUnit testing could be integrated into the RequireJS structure. Obviously you don't want the testing code to be lying around in the Production build. How can you keep testing out of the final production build in RequireJS?
There are lots of options you can set in the build file. See the full example on GitHub (https://github.com/jrburke/r.js/blob/master/build/example.build.js)
What you want to do is exclude certain items from your module:
//This module entry combines all the dependencies of foo/bar/bip into one file,
//but excludes foo/bar/bop and its dependencies from the built file. If you want
//to exclude a module that is also another module being optimized, it is more
//efficient if you define that module optimization entry before using it
//in an exclude array.
{
name: "foo/bar/bip",
exclude: [
"foo/bar/bop"
]
},
//This module entry shows how to specify a specific module be excluded
//from the built module file. excludeShallow means just exclude that
//specific module, but if that module has nested dependencies that are
//part of the built file, keep them in there. This is useful during
//development when you want to have a fast bundled set of modules, but
//just develop/debug one or two modules at a time.
{
name: "foo/bar/bin",
excludeShallow: [
"foo/bar/bot"
]
}
You can also exclude items with a regular expression, but this is probably overkill:
//When the optimizer copies files from the source location to the
//destination directory, it will skip directories and files that start
//with a ".". If you want to copy .directories or certain .files, for
//instance if you keep some packages in a .packages directory, or copy
//over .htaccess files, you can set this to null. If you want to change
//the exclusion rules, change it to a different regexp. If the regexp
//matches, it means the directory will be excluded. This used to be
//called dirExclusionRegExp before the 1.0.2 release.
//As of 1.0.3, this value can also be a string that is converted to a
//RegExp via new RegExp().
fileExclusionRegExp: /^\./,