I'm trying to figure out how to best manage Javascript file dependencies and have that drive the packaging of a 100% front-end app. In short, I'm building an application using backbone.js along with some other libraries. I want an organized codebase and would like to be able to declare dependencies within each file. Ideally those declarations would a) drive the order in which files are loaded by the browser (while in development I want the files to load separately) and drive the order in which the packaging scripts load the scripts for concatenation (I'm aiming to serve a single file for the entire app).
I've been reading on requirejs and commonjs but I'm not convinced.
I have a simple shell script right now that uses cat <file> <file> <file> <file> > concatenated.file to do what I want but it's a pain to maintain that list of files up to date and in the right order. It'd be much easier to be able to declare the dependency at the begining of each javascript file and have the packager and loaders be smart about using that information to concatenate/load scripts.
Any suggestions?
Thanks you,
Luis
I am partial to stealjs myself. It's part of JavascriptMVC but no reason why you can't use it with Backbone.js
The nice part about this one is that it builds your app for production including minifying your css and js and neatly packing all of it into 2 files: production.css and production.js.
It can handle loading non JS files too so you can do things like steal('somefile.css').then(function() {...});
For files, its very much like you would do in other languages:
steal(dep1, dep2, dep3).then(function () {
// code
});
For complex frontend apps Asynchronous Module Definition (AMD) format is best choice. And it's alot of great loaders that supports AMD (curl.js, RequireJS).
I recomend this articles to learn about modern approaches in javascript dependecy management:
Writing Modular JavaScript With AMD, CommonJS & ES Harmony
Why AMD?
For packaging take into account CommonJS specifications, there are few implementations and it's a matter of taste, but in any case I recommend to choose tools, that is compliant with some of that specifications.
It'd be much easier to be able to declare the dependency at the begining of each javascript file and have the packager and loaders be smart about using that information to concatenate/load scripts.
I have had the same idea several months ago and are working on a dependency resolver for my Resource Builder which already makes it easier for me (including the need to distinuish between development and deployed version, with the optional debug parameter).
JsDoc Toolkit (and related efforts), which syntax is supported e. g. by Eclipse JSDT, provides a #requires tag, so you could use that. But resolving dependencies is still not a trivial task (as you can see in ResourceBuilder::resolveDeps()). (The ultimate goal is to use static code analysis to resolve dependencies automatically, without any tags.)
This would reduce the current
<script type="text/javascript" src="builder?src=object,types,dom,dom/css"></script>
to a mere
<script type="text/javascript" src="builder?src=dom/css"></script>
As for asynchronous loaders: The good thing about asynchronous loaders is that they are fast. The bad thing about asynchronous loaders is that – if they work; they are all based on a non-standard approach – they are so fast that you cannot be sure that the features the scripts provide are available in following scripts. So you have to have your code executed by their listeners. I recommend avoiding them unless you really have features in your application that are only needed on demand.
Related
This is vague - I apologize in advance, I am trying to be as succinct as I can with my limited understanding, while exposing my potentially incorrect assumptions.
I have a website that's literally one huge HTML file. It runs scripts defined in-line in a <scripts> tag.
My goal is to move all the scripts into individual .js files and pull them into index.html (or into one another where required). I am familiar with the usage of Node's require and I am familiar with import, as used in Angular. I stress usage because I don't really know how they work.
Assumption 1: I cannot use require - it is purely for Node.js. The reason I bring it up is that I am pretty sure I have seen require in AngularJS 1.5 code, so assuming makes me uncomfortable. I am guessing this was stitched together by WebPack, Gulp, or similar.
Assumption 2: I should use import, but import only works with a URL address, if I have my .js hosted on a server or CDN, it will be be pulled. BUT, I cannot give local pathing (on the server) here - index.html will NOT automatically pull in the dependencies while being served. I need npm/Webpack/other to pre-compile my index.html if I want the deps pulled in on the server.
Assumption 3: Pre-compiling into a single, ready-to-go, html file is the desired way to build things, because the file can be served quickly (assuming it's ready to go). I make the assumption with the recent trend of serving Markdown from CDNs, the appearance of the JAMstack, and the number of sites using Jekyll and such (though admittedly for traditional Jekyll sites).
Assumption 4: I also want to go to Typescript eventually, but I assume that changes nothing, since I will just pull in TS to compile it down to .js and then use whatever solution I used above
Question: If it's clear what I am trying to do and what confuses me, is a decent solution to look into npm/Webpack to stich together my .js files? What would prepare them for being stiched together, using import/export? If so, is there an example of how this is usually done?
As you mentioned, require cannot be used for your purposes, since it is a part of CommonJS and NodeJS's module system. More info on require can be found here: What is this Javascript "require"?
Import is a ES2015 standard JS syntax. But ES2015 standard and everything above it has very limited browser support. You can read more about it here: Import Reference
However, you can still code in the latest standard (thereby enabling the use of import/export etc.,) and then transpile the code to be able to run on the browser. Inorder to do this, you require a transpiler. You can refer Babel which is one of the most popular transpilers : https://babeljs.io/
For your exact purpose, you need to use a module bundler. Webpack/Rollup etc., are some popular module bundlers. They can automatically identify all the JS files referenced through import, combine them and then transpile code to be able to run on the browser (they also use a transpiler) and produce a single JS file (or multiple, based on your configurations).
You can refer the getting started guides of Webpack: https://webpack.js.org/guides/getting-started/
or RollupJS: https://rollupjs.org/guide/en#quick-start
In our AngularJS application we currently have a lot (400+) of files, which get included via <script>-tags. The order these files is something like this:
AngularJS script files
3rd party plugins / modules
business Logic files
modules
services
controllers / components / directives
We would like to move to a better approach utilizing a module bundler and TypeScript. New files are already written in TypeScript but don't make use of import/export. In order to make things easier, we could convert every JavaScript file into a TypeScript file and fix the resulting errors in a feasible time.
However, before we do this, we would like to have a viable strategy on how we could gradually make use of import/export. I'm thinking of something like rewriting one module from time to time, starting with modules deep down in the dependency tree.
However I was not able to achieve this, but I'm quiet sure that others already had to solve this before.
Once you decide on a module bundler, you'll need to learn about its facilities for (1) allowing other JavaScript on the page to access things defined in the bundle and (2) allowing the bundle to access things defined by other JavaScript on the page. (If you're able to migrate in strict dependency order, you might never need #2.) For example, for Webpack, #1 would be the library* output options and #2 would be externals. Then just move the code into modules little by little, adjusting the configuration as necessary so that each part of the code has access to the things it needs from the other part of the code. Since Webpack only supports a single library export module, during the transition, you may have to maintain a dummy module that just re-exports all modules that you need to access from code outside the bundle. This is a little tedious and represents extra work that you wouldn't have to do if you migrated all at once, but you may decide it's worth paying that cost in order to be able to migrate gradually.
If you have issues getting type information from TypeScript module files in non-module TypeScript files, see this answer for a workaround.
I'm currently preparing an evaluation JavaScript modularization approaches for my corp. We are in process of defining "JavaScript Best Practices" for our projects, modularization is one of the central questions.
From my research so far revealed two leading approaches:
amd
commonjs
With a huge number of loaders, plugins, libraries etc. around them.
Apart from that there's also goog.provide/goog.require from the Google Closure Library.
Are there further approaches to consider? Any important/relevant specs I missed?
Our requirements, briefly:
Structure JavaScript code in separate files.
Load relevant modules in the runtime.
...without having to include every single file as script tag.
Must not be necessary to maintain an index of JavaScript files.
Support aggregation and minification - ability to build and use a single minified/optimized JavaScript file.
Be able to use modules in different combinations - there are often different web pages/clients which need different subsets of modules.
Supporting documentation (with JSDoc?).
Suitable for testing.
Suitable for web, cross-browser.
Reasonable IDE support.
Potentially:
Aligned with ES6 modules.
Suitable for Node.js and mobile platforms (like PhoneGap/Cordova).
New suggestions from answers:
ecmascript-harmony plus extra compiler.
angularjs (see the note below).
extjs (see the note below).
Side notes:
The question is not about which approach is better.
I am not asking for specific libraries and tools, but rather for approaches and specifications.
I am not specifically asking for an off-site resource. (If there's no SO tag for this, it's probably not reasonable for us to consider it.)
A note on frameworks like angualjs or extjs. This is not really suitable in the frame of this quesion. If the project needs a framework (be it AngularJS or ExtJS) then there's mostly no modularization question as the framework must deliver modularization OOTB. If the project does not need a framework, it is an overkill to bring a framework because of the modularization. This is one of the reasons I am specifically not asking about libraries/tools.
How about ES Harmony?
quote from here: http://addyosmani.com/writing-modular-js/
Note: Although Harmony is still in the proposal phases, you can
already try out (partial) features of ES.next that address native
support for writing modular JavaScript thanks to Google's Traceur
compiler. To get up and running with Traceur in under a minute, read
this getting started guide. There's also a JSConf presentation about
it that's worth looking at if you're interested in learning more about
the project.
hopeThatHelps
Another option: the AngularJS module system, as described here. This is really only useful on the client side, however.
You wrote "I am not asking for specific libraries and tools, but rather for approaches and specifications." however you can look closer to ExtJS 5 environment which fulfils all your requirements.
If you are not interested in such commercial product you can just get to know patterns and solutions in it.
Relation to your requirements:
Structure JavaScript code in separate files.
It implement Object-Oriented Programming paradigm so you can create classes, subclasses, objects, mixins, plugins. It connect class-based programming and prototype-based programming.
Worth noting MVVM architecture (View, Controller, ViewModel), data binding, data session (records/entities client-side management).
Config system is also quite interesting. It's very handy. The config property is merged from parent class to subclasses and also during object creation you can pass config which will be merged too. It is very useful when we want have customizable and flexible components.
Load relevant modules in the runtime.
Each class may has requires or uses directive which are used when builing application to one file. You can also manually load files.
...without having to include every single file as script tag.
In dev env files are loaded dynamically (asynchronous or synchronous).
In prod env necessary files has been built to one minified file.
Support aggregation and minification - ability to build and use a single minified/optimized JavaScript file.
You can build application with Sencha cmd tool (and do a few other things).
You can use three predefined env (development, testing, production) or create your own (based on config files and ant).
Be able to use modules in different combinations - there are often different web pages/clients which need different subsets of modules.
You can use workspaces and packages.
Supporting documentation (with JSDoc?).
JS Duck, tutorial
Suitable for testing.
You can do unit tests (PhantomJS, JSLint, PhantomLint, Jasmine).
You can use dedicated framework like Siesta or other popular testing frameworks like Selenium.
Suitable for web, cross-browser.
From offical website:
Deliver apps on the widest selection of browsers and operating systems
with a single code base. Ext JS 5 leverages HTML5 features on modern
browsers while maintaining compatibility and functionality for legacy
browsers. Confidently deliver apps to your end users regardless of
what browser they are using.
Support:
Safari 6+, Firefox, IE 8+, Chrome, Opera 12+, Safari/iOS, Safari / iOS 6+, Chrome/Android, Chrome / Android 4.1+, IE 10+ / Win 8
Supports Cordova and PhoneGap application.
Reasonable IDE support.
I don't know very good IDE with dedicated support for ExtJS but I work on Webstorm and it's fine. Library sources are inside project so autocompletion works (but not 100% perfect).
Conclusion
I don't want to glorify ExtJS 5. Environment is quite mature and stable but latest version of framework (v5) has a couple bugs and not everything is great. However I could go deeper and get to know principles of that framework which are reasonable, driven in good direction but sometimes bad implemented ;)
RequireJS is a good approach since it enables dynamic javascript module loading as well as it keeps the code clean. You can also minify the whole modules and it actually performs really fast. It also allows something called as shimming where you can specify the dependencies for a library or any js file such that whenever you try to load it, all the dependencies also follow
Take a look at systemJS:
Spec-compliant universal module loader - loads ES6 modules, AMD,
CommonJS and global scripts.
Designed as a collection of small extensions to the ES6 specification
System loader, which can also be applied individually.
Loads any module format, by detecting the format automatically.
Modules can also specify their format with meta config. Provides
comprehensive and exact replications of AMD, CommonJS and ES6 circular
reference handling. Loads ES6 modules compiled into the
System.register form for production, maintaining full circular
references support. Supports RequireJS-style map, paths, bundles, shim
and plugins. Tracks package versions, and resolves semver-compatibile
requests through package version syntax - package#x.y.z,
package^#x.y.z. Loader plugins allow loading assets through the module
naming system such as CSS, JSON or images. Designed to work with the
ES6 Module Loader polyfill (9KB) for a combined total footprint of
16KB minified and gzipped. In future, with native implementations, the
ES6 Module Loader polyfill should no longer be necessary. As jQuery
provides for the DOM, this library can smooth over inconsistiencies
and missing practical functionality provided by the native System
loader.
Runs in IE8+ and NodeJS.
The creator of the lib -- Guy Bedford -- is a great presenter as well: systemJS presentation.
Take a look on a browserify. It implements interesting approach. With browserify you can write code that uses require in the same way that you would use it in Node.
There are various libraries available for modular development,
From which some full fill your criteria.
System.js
System.js is modular development library with some basic features to work with IE8+ and nodejs. It also provides feature to develop modules and you can include it in your main file. For more information about System.js follow https://github.com/systemjs/systemjs
Require.js
The best library for modular development. Provides various useful features and supports older browser too. It supports from IE 6+. Another useful feature of require.js is that it can be used with Rhinojs and nodejs. Implementation is simple as like you include modules in nodejs.
I've been using a grunt file to concatenate all my JS into a single file which is then sent to the client. What advantage do I have in using require calls then? The dependencies are inherent from the concatenation order and I don't have to muddy all my JS with extra code and another third-party library.
Further, backbone models (for example) clearly state their inheritance in their definitions. Not to mention that they simply wouldn't work if their dependencies weren't included anyway.
Also, wouldn't maintenance be easier if all comments related to dependencies were in one place (the grunt file) to prevent human error and having to open every JS file to understand its dependencies?
EDIT
My (ordered) file list looks something like:
....
files: [
"js/somelib.js",
"js/somelib2.js",
"js/somelib3.js",
"js/models.js",
"js/views.js",
"js/controllers.js",
"js/main.js"
], ...
So perhaps requireJS isn't worth it for small projects anyway.
Using require.js allow you to break down each part of your application into reusable modules (AMD) and to manage those dependencies easily. It is not easy to manage dependencies in a javascript application with 100 classes, for example.
Also, if you don't want all the overhead of require, check this out (developed by the same guy who created require.js): https://github.com/jrburke/almond
The answer depends on the size of your app and the end use case..
A single site.min.js payload for the front end (client) generally aims for small file sizes and simple architectures (1 single file generated from maybe 10).
back end based (server) apps are usually much bigger and complicated and therefore may warrant the use of another tool to help with managing large code libraries and dependencies (50 files for example).
In general, RequireJS is worthwhile but only if you have many files and dependencies. An alternative for use in the client would be almond. Again, using a tool like this must warrant the need (many files and dependencies).
The answer from orourkedd is also worth reading.
I would like to create a (require.js style) AMD module that can be used in the browser and in node. What is the best way to do this? I keep seeing references to r.js, but still not 100% sure on how to use it, or if it is necessary for my situation.
Also, when including this module in node, do I still run require('module-name'), or will this change as well?
First things first: AMD basics, What all you can do with them, How to optimize them
In extremely simple terms
AMD modules are reusable JS code. Think of them as functions kept in separate files.
AMD loaders are central functions which call all other functions (modules). Think of them as "main" method in C or Java.
RequireJS is a framework which pulls all this scattered code and stitches it in a usable form.
RequireJS works in a browser. As a result, all your code is "stitched" together in a web browser.
r.js works offline (in a web server or on your development machine) to "stitch" all the code offline so that when it reaches a web browser, it's already "stitched".
Use of RequireJS lib is a must no matter you want to "stitch" your code in browser or you want to serve your code "pre-stitched".
Use of r.js is optional. It's needed only if you want to increase performance and decrease HTTP calls.