Due to the governance constraints on my day-job project, I'm unable to use a lot of the nice new JS libraries and frameworks. So, for the MVP we're using vanilla JS loaded directly (non-minified) to the client (gross). While this is a great way to show management that's not the best approach, I still have to get it out of the door. However, we do have NPM and Node installed on our dev machines, and I'd like to use this to optimize our JS into a combined bundle, with a hash to break caching.
My question is how do I go about gathering a bunch of disjoint JS files and adding them to a new NPM project to be minified while avoiding having the expected variables, objects, and functions being mangled by webpack/prettify?
I tried just copying the js files into the src directory, and adding the normal import/export lines, but I was getting a bunch of "identifier not found" messages in the console, so I think a lot of stuff was getting mangled.
I have some experience using create-react-app for React side projects, but unfortunately that insulates me from the pain of getting a project setup the hard way, and now I'm paying for my inexperience.
EDIT: To be more succinct, I'm trying to package a bunch of existing js files into a single bundle, while maintaining the same interface. If this isn't possible, please let me know. Otherwise, a place to start would be ideal.
In the days before browserify, babel, webpack, grunt, gulp... we would just concatenate the files and minify them. Since the files expose their API as global objects everything keeps working as if the files where included with different script tags.
On Linux you can simply do this
cat a.js b.js c.js | uglifyjs -cm > bundle.js
The files would often start with a semicolon to make sure nothing breaks when concatenating them.
Once babel is configured integration is as easy as
cat a.js b.js c.js | babel | uglifyjs -cm > bundle.js
https://babeljs.io/docs/usage/cli/
Related
I am using Webstorm a start a angularjs project. I've created a lot of *.js files. I have the include them using the tag one by one in the html files. whenever I created a new js file, I have to create the script tag.
The things I needed is just like gulp-concat, but without minifying. minified code are hard to debug.
Can anyone help on this? Thanks!
WebStorm doesn't have any built-in functions for combining files... But there are plenty of different tools on the web - plus you can create your own batch files for this.
I can suggest using Grunt grunt-contrib-concat task (https://github.com/gruntjs/grunt-contrib-concat). It supports merging files. You can run the task using Node.js run configuration, or configure it as a file watcher, or use Grunt console.
Browserify (http://browserify.org/) is one more way to go - it allows using commonjs-style syntax when developing front-end applications, combining the files into a single file for production
Or, try Webpack (https://webpack.github.io/) - it's a modern powerful module bundler
I've been browsing many of the articles here about how and why one should combine JS/CSS files for performance, but none of those articles offered any real guideline as to when is the right time.
I'm developing a single-page microsite that uses seven Javascript files (a mixture of third-party plugins from CDNs and my own files), and eight different CSS files (basically one per plugin, and my own compiled SASS file).
The site loads slowly even on the intranet here; I'm concerned about the performance outside. While searching for several plugins yesterday, I found several CodePen and plugin articles that basically said "cool kids concatenate JS" (literally) which got me thinking about this whole thing.
At what point should I start concatenating and minifying my Javascript/CSS?
And should I paste the CDN scripts into my own JS files, or is it better in the long run to have another HTTP request but use the statically served plugin files?
Edit: Just to clarify - I'm not asking for tools/techniques, but wondering when it becomes important to combine and minify files - should it always been done as #RobG suggested?
You should deliver code to UAT that is as close to production code as possible, including all minification and combining of files. If you don't, then you aren't doing proper UAT
To be honest, it depends.
People are often, wrongly, obsessed with merge-min... That's not always the case. The need for merge-min depends on a few things:
Sometimes it's faster and BETTER practice to load 2 css files than one big one? Why? Because they'll load in parallel. That simple.
So don't go with the merge-min obsession. If your users are returning, daily users, do merge and rely on browser cache. If not, optimise parallel loads by not merging.
And ignore the simplistic: 'yes you must merge because that's what was best 10 years ago and I've never questioned it' :)
When Should I Combine my JS and CSS Files?
Every time you are finished with development. specifically when your code is going to User Acceptance Test (UAT), if not earlier. thanks #RobG for mentioning it.
Which tools do you suggest?
Browserify
Let's Start with your JS files. I think a great tool for bundling various JS files/modules is Browserify.
Browsers don't have the require method defined, but Node.js does. With Browserify you can write code that uses require in the same way that you would use it in Node.
Here is a tutorial on how to use Browserify on the command line to bundle up a simple file called main.js along with all of its dependencies:
var unique = require('uniq');
var data = [1, 2, 2, 3, 4, 5, 5, 5, 6];
console.log(unique(data));
Install the uniq module with npm:
npm install uniq
Now recursively bundle up all the required modules starting at main.js into a single file called bundle.js with the browserify command:
browserify main.js -o bundle.js
Browserify parses the AST for require() calls to traverse the entire dependency graph of your project.
Drop a single tag into your html and you're done!
<script src="bundle.js"></script>
Also there is a tool similer for CSS files called browserify-css.
Gulp
gulp is a toolkit that will help you automate painful or time-consuming tasks in your development workflow. For web development (if that's your thing) it can help you by doing CSS preprocessing, JS transpiling, minification, live reloading, and much more. Integrations are built into all major IDEs and people are loving gulp across PHP, .NET, Node.js, Java, and more. With over 1700 plugins (and plenty you can do without plugins), gulp lets you quit messing with build systems and get back to work.
Public CDN scripts
should I paste the CDN scripts into my own JS files, or is it better in the long run to have another HTTP request but use the statically served plugin files?
You can keep them in public CDN; To avoid needlessly overloading servers, browsers limit the number of connections that can be made simultaneously. Depending on which browser, this limit may be as low as two connections per hostname.
Using a public CDN (like Google AJAX Libraries CDN) eliminates one request to your site, allowing more of your local content to downloaded in parallel. Read more on this here
I really like using cdnjs to load up javascript on the client-side, it makes my project smaller and cleaner, and loads everything faster as well. I currently use require.js for module loading, which can load from cdnjs and shim traditional scripts to work with it easily. I've been looking more into browserify recently as an alternative, and while I did find browserify-shim, which can shim non-cjs modules much like require does, I'm curious if there is a way to load a script from a remote source with browserify, or if you have to install everything locally no matter what.
If the answer is that you would have to install everything locally through npm, this makes things a little weird. On one hand, you can add node_modules to the .gitignore file and not have to worry about keeping all the deps on version control if you are using a package.json, but on the other hand, you'd need to get the modules back in there on deploy, which means an additional post-deploy step that would run npm install and that node would need to be installed wherever you are deploying to, which also seems a little awkward to me for a static site especially.
Really, any ideas or discussion on this would be great : )
The way I think about it is this, you have three options: concat the JS files together locally (browserify) before deployment, load them in real-time (require.js), or a mix of both. To be fair, you can use require.js to concat your files with r.js too. For me at least, I like how browserify is designed to use the same syntax and mentality as npm modules. I think in the end the weirdness your experiencing doesn't really matter. If all the code is packaged together, you deploy, and there aren't any dependencies, seems like a win to me. Also, I think this is more in line with Java and similar compiled languages are doing, which is putting all the deps together in a deployable package. I know I mention Java but don't let that scare you because really we are all benefitting from the ideas of those around us even the languages we think we don't like. If I had to bet my money, I would bet on browserify since it's offering (what I consider) to be a more mature means of handling modules (organized by file based rather than syntax). The npm also gives us a great way to share our code so two thumbs up for them.
As the title says I'm trying to automatically compile the public static coffeescript files on page load, rather than having to compile them myself and use the .js files, how might I achieve this, I'm trying to maintain a full CoffeeScript stack, this is the only thing I'm having trouble figuring out.
tl;dr: Read the title of the post.
There are a number of ways to accomplish what you're trying to do. Two of the easiest that I know of:
Use the connect-assets module. The idea behind this is that you have an /assets folder in the root, and you instantiate it with express.static, as you normally would with your /public folder. In there, you have two more folders: /js and /css. Your CoffeeScript goes in your /js folder. Then, from within your view template, just call js('yourfile'). It's a wonderfully simple module, but isn't the most advocated asset pipeline.
Use asset-rack. While not as simple to grasp as connect-assets, it's very flexible and is easy to extend. It would be the closest comparator to Rails' asset pipeline and is used by most of the popular JS frameworks (like Sails.js).
However, I would really advise that you refrain from on-the-fly compilation of assets, as it can really drain server performance.
It would be way better to compile on file save using a build system - CoffeeScript ships with Cake, so you can define a watch/compile/build/concatenate step right in your Cakefile and all it would take for you to compile on file save is to type $ cake watch in to the terminal before you change any code.
Alternatively you could write code, then $ cake build. Whichever you prefer. I might also add that cake-flour takes all of the pain out of writing Cake tasks.
coffee --watch --compile .
watches for files changes in . and compiles them as they change. Since you are using expressjs I think you would also like to restart the server as soon as a recompile happens:
coffee --compile --watch . &; nodemon server.js
which uses https://github.com/remy/nodemon
If you only want to have access to compiled CoffeeScript files over HTTP, you could also give connect-coffee-script middleware a try. It has worked quite well for me.
It has the advantage over using coffee --watch --compile that you don't need to have a separate program running.
If I have a node.js application that is filled with many require statements, how can I compile this into a single .js file? I'd have to manually resolve the require statements and ensure that the classes are loaded in the correct order. Is there some tool that does this?
Let me clarify.
The code that is being run on node.js is not node specific. The only thing I'm doing that doesn't have a direct browser equivalent is using require, which is why I'm asking. It is not using any of the node libraries.
You can use webpack with target: 'node', it will inline all required modules and export everything as a single, standalone, one file, nodejs module
https://webpack.js.org/configuration/target/#root
2021 edit: There are now other solutions you could investigate, examples.
Namely:
https://esbuild.github.io
https://github.com/huozhi/bunchee
Try below:
npm i -g #vercel/ncc
ncc build app.ts -o dist
see detail here https://stackoverflow.com/a/65317389/1979406
If you want to send common code to the browser I would personally recommend something like brequire or requireJS which can "compile" your nodeJS source into asynchronously loading code whilst maintaining the order.
For an actual compiler into a single file you might get away with one for requireJS but I would not trust it with large projects with high complexity and edge-cases.
It shouldn't be too hard to write a file like package.json that npm uses to state in which order the files should occur in your packaging. This way it's your responsibility to make sure everything is compacted in the correct order, you can then write a simplistic node application to reads your package.json file and uses file IO to create your compiled script.
Automatically generating the order in which files should be packaged requires building up a dependency tree and doing lots of file parsing. It should be possible but it will probably crash on circular dependencies. I don't know of any libraries out there to do this for you.
Do NOT use requireJS if you value your sanity. I've seen it used in a largish project and it was an absolute disaster ... maybe the worst technical choice made at that company. RequireJS is designed to run in-browser and to asynchronously and recursively load JS dependencies. That is a TERRIBLE idea. Browsers suck at loading lots and lots of little files over the network; every single doc on web performance will tell you this. So you'll very very quickly end up needing a solution to smash your JS files together ... at which point, what's the point of having an in-browser dependency resolution mechanism? And even though your production site will be smashed into a single JS file, with requireJS, your code must constantly assume that any dependency might or might not be loaded yet; in a complex project, this leads to thousands of async load barriers wrapping every interaction point between modules. At my last company, we had some places where the closure stack was 12+ levels deep. All that "if loaded yet" logic makes your code more complex and harder to work with. It also bloats the code increasing the number of bytes sent to the client. Plus, the client has to load the requireJS library itself, which burns another 14.4k. The size alone should tell you something about the level of feature creep in the requireJS project. For comparison, the entire underscore.js toolkit is only 4k.
What you want is a compile-time step for smashing JS together, not a heavyweight framework that will run in the browser....
You should check out https://github.com/substack/node-browserify
Browserify does exactly what you are asking for .... combines multiple NPM modules into a single JS file for distribution to the browser. The consolidated code is functionally identical to the original code, and the overhead is low (approx 4k + 140 bytes per additional file, including the "require('file')" line). If you are picky, you can cut out most of that 4k, which provides wrappers to emulate common node.js globals in the browser (eg "process.nextTick()").