Develop Reference

On-demand in-memory bundling of Javascript files at runtime

I'm wondering whether there is a nice way to continuously bundle complex javascript files in a single, customized, JS file that is then directly served to clients, sort of a "smart" CDN.
My use case is really simple: serve JS files to different web applications from a microservice, like RUM scripts (or also Google AdSense scripts, or anything else that works by importing a JS file from a service in your web application).
Problem is, at every request the JS file will be different, based on several parameters (for example, IP address of the request, query params in the URI, back-end configuration, and so on).
Solution can be trivial, for example I could manually bundle JS files by concatenating strings, and then serve the resulting file, with minimal logic or with the help of a standard template engine that handles placeholders/pre-processing within the source JS files.
I was wondering if there is a better way to do this, like using JS bundlers as Webpack or Parcel.js.
My experience in using JS bundlers is limited to their standard usage, so that use them in the build process (or at application startup), and make them write to disk the compiled JS file.
Is there anything better to use, or a best practice to suggest for my use case?
Programming language makes no difference, I added the node.js tag because ideally the backend would be developed using Express.js, but also Java or other compiled languages would be ok.
Thanks in advance,

Sure, that's definitely possible.
Esbuild (a new, fast bundler) has a well-documented Node.js API: https://esbuild.github.io/api/#js-specific-details
The two other bundlers you mention are similarly available:
Webpack has a well-documented Node.js API: https://webpack.js.org/api/node/
Parcel has a well-documented Node.js API: https://parceljs.org/api.html
Then it's just (always "just" ;-) ) a matter of hooking those up to the web framework of your choice and going to town.
I'd recommend thinking about caching in advance, though – even if e.g. Esbuild and Parcel are pretty fast, not doing work is always faster than doing work.

How is it possible that browserify can make server code run in the browser when servers need access to things like sockets?

Am I simply ignorant of the features that some browsers offer or does this library use some strange workaround to implement these features?

Actually, it's neither nor ;-)
Basically, the simple answer is: Browserify does not bring everything into the browser, only the things that actually make sense and are feasible from a technical point of view.
E.g., you can easily have
url.format(...)
in the browser, as this means only handling objects and strings in memory, but you can not have
http.createServer(...)
since this simply does not work. Additionally to this, Browserify provides shims for require and module.exports, hence you can use CommonJS modules in the browser as well, but only as long as they stick to the things that are available there. Again, e.g., any CommonJS module written in C++ will not work in the browser, with or without Browserify.
Finally, Browserify uses several modules that are basically API-compatible to Node.js, but have been rewritten for the browser, e.g. http-browserify.
So, in the end, it is a great tool for so-called "isomorphic JavaScript", or in other words: To use CommonJS modules on the server and in the browser without the need for special patterns on either side, such as AMD or UMD.
But, of course, it does not provide any magic :-)

Browserify doesn't add functionality to your browser. It compiles your Node.js code so that it is runnable inside of a browser.
In short, Browserify allows you to use Node's require in your browser-side code; it does not grant your browser access to network and filesystem abilities that it doesn't already have.
Instead, you'll need to use (or write) custom modules that simulate the server-side capabilities. For example, if you use a database module in your server-side code (e.g., Postgres or Mongo), you could write a new database module (with the same API as the server-side module) that uses browser-supported storage mechanisms like IndexedDB.

Modular JavaScript - are there any approaches apart CommonJS and AMD to consider?

I'm currently preparing an evaluation JavaScript modularization approaches for my corp. We are in process of defining "JavaScript Best Practices" for our projects, modularization is one of the central questions.
From my research so far revealed two leading approaches:
amd
commonjs
With a huge number of loaders, plugins, libraries etc. around them.
Apart from that there's also goog.provide/goog.require from the Google Closure Library.
Are there further approaches to consider? Any important/relevant specs I missed?
Our requirements, briefly:
Structure JavaScript code in separate files.
Load relevant modules in the runtime.
...without having to include every single file as script tag.
Must not be necessary to maintain an index of JavaScript files.
Support aggregation and minification - ability to build and use a single minified/optimized JavaScript file.
Be able to use modules in different combinations - there are often different web pages/clients which need different subsets of modules.
Supporting documentation (with JSDoc?).
Suitable for testing.
Suitable for web, cross-browser.
Reasonable IDE support.
Potentially:
Aligned with ES6 modules.
Suitable for Node.js and mobile platforms (like PhoneGap/Cordova).
New suggestions from answers:
ecmascript-harmony plus extra compiler.
angularjs (see the note below).
extjs (see the note below).
Side notes:
The question is not about which approach is better.
I am not asking for specific libraries and tools, but rather for approaches and specifications.
I am not specifically asking for an off-site resource. (If there's no SO tag for this, it's probably not reasonable for us to consider it.)
A note on frameworks like angualjs or extjs. This is not really suitable in the frame of this quesion. If the project needs a framework (be it AngularJS or ExtJS) then there's mostly no modularization question as the framework must deliver modularization OOTB. If the project does not need a framework, it is an overkill to bring a framework because of the modularization. This is one of the reasons I am specifically not asking about libraries/tools.

How about ES Harmony?
quote from here: http://addyosmani.com/writing-modular-js/
Note: Although Harmony is still in the proposal phases, you can
already try out (partial) features of ES.next that address native
support for writing modular JavaScript thanks to Google's Traceur
compiler. To get up and running with Traceur in under a minute, read
this getting started guide. There's also a JSConf presentation about
it that's worth looking at if you're interested in learning more about
the project.
hopeThatHelps

Another option: the AngularJS module system, as described here. This is really only useful on the client side, however.

You wrote "I am not asking for specific libraries and tools, but rather for approaches and specifications." however you can look closer to ExtJS 5 environment which fulfils all your requirements.
If you are not interested in such commercial product you can just get to know patterns and solutions in it.
Relation to your requirements:
Structure JavaScript code in separate files.
It implement Object-Oriented Programming paradigm so you can create classes, subclasses, objects, mixins, plugins. It connect class-based programming and prototype-based programming.
Worth noting MVVM architecture (View, Controller, ViewModel), data binding, data session (records/entities client-side management).
Config system is also quite interesting. It's very handy. The config property is merged from parent class to subclasses and also during object creation you can pass config which will be merged too. It is very useful when we want have customizable and flexible components.
Load relevant modules in the runtime.
Each class may has requires or uses directive which are used when builing application to one file. You can also manually load files.
...without having to include every single file as script tag.
In dev env files are loaded dynamically (asynchronous or synchronous).
In prod env necessary files has been built to one minified file.
Support aggregation and minification - ability to build and use a single minified/optimized JavaScript file.
You can build application with Sencha cmd tool (and do a few other things).
You can use three predefined env (development, testing, production) or create your own (based on config files and ant).
Be able to use modules in different combinations - there are often different web pages/clients which need different subsets of modules.
You can use workspaces and packages.
Supporting documentation (with JSDoc?).
JS Duck, tutorial
Suitable for testing.
You can do unit tests (PhantomJS, JSLint, PhantomLint, Jasmine).
You can use dedicated framework like Siesta or other popular testing frameworks like Selenium.
Suitable for web, cross-browser.
From offical website:
Deliver apps on the widest selection of browsers and operating systems
with a single code base. Ext JS 5 leverages HTML5 features on modern
browsers while maintaining compatibility and functionality for legacy
browsers. Confidently deliver apps to your end users regardless of
what browser they are using.
Support:
Safari 6+, Firefox, IE 8+, Chrome, Opera 12+, Safari/iOS, Safari / iOS 6+, Chrome/Android, Chrome / Android 4.1+, IE 10+ / Win 8
Supports Cordova and PhoneGap application.
Reasonable IDE support.
I don't know very good IDE with dedicated support for ExtJS but I work on Webstorm and it's fine. Library sources are inside project so autocompletion works (but not 100% perfect).
Conclusion
I don't want to glorify ExtJS 5. Environment is quite mature and stable but latest version of framework (v5) has a couple bugs and not everything is great. However I could go deeper and get to know principles of that framework which are reasonable, driven in good direction but sometimes bad implemented ;)

RequireJS is a good approach since it enables dynamic javascript module loading as well as it keeps the code clean. You can also minify the whole modules and it actually performs really fast. It also allows something called as shimming where you can specify the dependencies for a library or any js file such that whenever you try to load it, all the dependencies also follow

Take a look at systemJS:
Spec-compliant universal module loader - loads ES6 modules, AMD,
CommonJS and global scripts.
Designed as a collection of small extensions to the ES6 specification
System loader, which can also be applied individually.
Loads any module format, by detecting the format automatically.
Modules can also specify their format with meta config. Provides
comprehensive and exact replications of AMD, CommonJS and ES6 circular
reference handling. Loads ES6 modules compiled into the
System.register form for production, maintaining full circular
references support. Supports RequireJS-style map, paths, bundles, shim
and plugins. Tracks package versions, and resolves semver-compatibile
requests through package version syntax - package#x.y.z,
package^#x.y.z. Loader plugins allow loading assets through the module
naming system such as CSS, JSON or images. Designed to work with the
ES6 Module Loader polyfill (9KB) for a combined total footprint of
16KB minified and gzipped. In future, with native implementations, the
ES6 Module Loader polyfill should no longer be necessary. As jQuery
provides for the DOM, this library can smooth over inconsistiencies
and missing practical functionality provided by the native System
loader.
Runs in IE8+ and NodeJS.
The creator of the lib -- Guy Bedford -- is a great presenter as well: systemJS presentation.

Take a look on a browserify. It implements interesting approach. With browserify you can write code that uses require in the same way that you would use it in Node.

There are various libraries available for modular development,
From which some full fill your criteria.
System.js
System.js is modular development library with some basic features to work with IE8+ and nodejs. It also provides feature to develop modules and you can include it in your main file. For more information about System.js follow https://github.com/systemjs/systemjs
Require.js
The best library for modular development. Provides various useful features and supports older browser too. It supports from IE 6+. Another useful feature of require.js is that it can be used with Rhinojs and nodejs. Implementation is simple as like you include modules in nodejs.

Using require.js module in node AND browser

I would like to create a (require.js style) AMD module that can be used in the browser and in node. What is the best way to do this? I keep seeing references to r.js, but still not 100% sure on how to use it, or if it is necessary for my situation.
Also, when including this module in node, do I still run require('module-name'), or will this change as well?

First things first: AMD basics, What all you can do with them, How to optimize them
In extremely simple terms
AMD modules are reusable JS code. Think of them as functions kept in separate files.
AMD loaders are central functions which call all other functions (modules). Think of them as "main" method in C or Java.
RequireJS is a framework which pulls all this scattered code and stitches it in a usable form.
RequireJS works in a browser. As a result, all your code is "stitched" together in a web browser.
r.js works offline (in a web server or on your development machine) to "stitch" all the code offline so that when it reaches a web browser, it's already "stitched".
Use of RequireJS lib is a must no matter you want to "stitch" your code in browser or you want to serve your code "pre-stitched".
Use of r.js is optional. It's needed only if you want to increase performance and decrease HTTP calls.

Javascript for command line utilities

Given a need to write command line utilities to do common tasks like uploading files to a remote FTP site, downloading data from a remote MySQL database etc.
Is it practical to use JavaScript for this sort of thing? I know there are JavaScript interpreters that can be run from the command line, but are there libraries for things like FTP and database access the way there are for e.g. Java? If so, what's the best place to look for them? (Google searches with JavaScript in the keywords always seem to return many pages of browser specific things.)
And is there a way to package a JavaScript program up as a standalone executable on Windows?
Update: I've decided Python is a better tool for this kind of job, but the answers to the original question are still good ones.

Standalone executable?
By the way you ask the question, I'm not sure if you are aware, but the Windows Script Host - included in Windows - allows you to run .js files from the command-line. Your javascript will not be an executable, it will remain a script, a text file. The script runs within cscript.exe, which is provided by WSH. There's no compilation required. Maybe you knew all that.
I use Javascript this way for various utilities on Windows.
I think your instinct is right on the availability of libraries. You are sort of on your own to find all those things. Although, once you find them, it's not hard to package Javascript libraries as COM components and allow re-use from anywhere. See here for an example of packaging the Google Diff/Patch/Match Javascript library in COM.
Addendum: Once a bit of code is available within COM, it can be consumed by any Javascript running on the machine. Some examples of COM objects available to Javascript scripts running in WSH:
MSXML2.XMLHTTP object - used in AJAX, but can be used for any HTTP communication. There also an object for the XSLT engine so you can do transforms from script.
Excel.Application - allows you to open up Excel spreadsheets and automate them from Javascript.
Communicator.UIAutomation - automate MS Communicator (send IM's via script)
COM objects for Google Earth.
SlowAES - an all-Javascript implementation of AES encryption.

You can use Rhino to compile Javascript into Java byte code, and get access to all Java libraries.
Or you could use JScript.net, and get access to the .net libraries. .net includes a jsc.exe that produces exe-files.
Both of these requires the respective framework to be installed to be able to run.

Node.js is by far the best environment for running non-browser JS. I've used Rhino and SpiderMonkey, and there's a pretty huge difference in everything from the basics like how errors are handled to the size of the community using the tool. Node is pitched for "server-side" JS - building server apps in JS. It's great for this. But it works equally well for building command line tools.
The NPM package manager (bundled with Node) provides a nice global directory for finding and installing packages. It works much better than other language equivalents like PECL / Pear / CPAN / etc. Several high quality tools like JSHint, The Jade templating language, and the CoffeeScript compiler are all already available through NPM/Node:
npm install -g jshint, coffee-script, jade
jshint my_code.js
jade < my.jade > my.html
For args parsing, there are packages like commander.js. I currently use a heavily extended version of Commander in my underscore-cli command-line tool.
For messing with JSON or for doing command-line JS work (similar to "perl -pe"), check out underscore-cli - It's a really powerful tool for processing JSON data, processing underscore templates, and running JS expressions from the command-line. I use it for 1001 different things that would otherwise be really annoying to accomplish.

Rhino is bundled with JDK 1.6, jrunscript.exe in the bin directory will allow you to run any Javascript you want. Since it runs under Java you get access to any Java libraries that you may have.
We use it from the command line extensively. It's very good at that.

One way is to write these utilities as AIR applications - They can be written in JavaScript and need not have a UI. They have access to the command line, and there are existing ActionScript 3 libraries that can handle FTP etc. These ActionScript APIs can be called from JS, in AIR applications. AIR applications also have access to a sqlite database.

jslibs is a good standalone JavaScript runtime that support many 3rd party open source libraries like zlib, SQLite, NSPR, libiconv, libTomCrypt, OpenGL, ...