I'm wondering whether there is a nice way to continuously bundle complex javascript files in a single, customized, JS file that is then directly served to clients, sort of a "smart" CDN.
My use case is really simple: serve JS files to different web applications from a microservice, like RUM scripts (or also Google AdSense scripts, or anything else that works by importing a JS file from a service in your web application).
Problem is, at every request the JS file will be different, based on several parameters (for example, IP address of the request, query params in the URI, back-end configuration, and so on).
Solution can be trivial, for example I could manually bundle JS files by concatenating strings, and then serve the resulting file, with minimal logic or with the help of a standard template engine that handles placeholders/pre-processing within the source JS files.
I was wondering if there is a better way to do this, like using JS bundlers as Webpack or Parcel.js.
My experience in using JS bundlers is limited to their standard usage, so that use them in the build process (or at application startup), and make them write to disk the compiled JS file.
Is there anything better to use, or a best practice to suggest for my use case?
Programming language makes no difference, I added the node.js tag because ideally the backend would be developed using Express.js, but also Java or other compiled languages would be ok.
Thanks in advance,
Sure, that's definitely possible.
Esbuild (a new, fast bundler) has a well-documented Node.js API: https://esbuild.github.io/api/#js-specific-details
The two other bundlers you mention are similarly available:
Webpack has a well-documented Node.js API: https://webpack.js.org/api/node/
Parcel has a well-documented Node.js API: https://parceljs.org/api.html
Then it's just (always "just" ;-) ) a matter of hooking those up to the web framework of your choice and going to town.
I'd recommend thinking about caching in advance, though – even if e.g. Esbuild and Parcel are pretty fast, not doing work is always faster than doing work.
Related
I'm looking for a solution to have server settings in a config file of some sort separate from the JS app (which I could be running in dev or prod (built) mode) but allow for the server to replace the file with a new one at random point in time (when settings have changed) and allow the JS app to pick it up and refresh it's settings.
This is mainly for setting URLs to API endpoints.
My choice of JS framework is Angular 1 and 4 but answers geared generally towards various JS frameworks/libraries are applicable.
to have server settings in a config file of some sort separate from the JS app
If you are thinking about having your server settings exposed to the front you are gonna have a bad time ;)
I will assume that you think you will keep API URLs and maybe some config along with it in a separate file while you develop. Then I can advise two approaches:
1.) SIMPLE - Have a single JS file with a single object with all the necessary API URLs and settings in a tree like structure for organisational purposes. So you have a dev, a test and production property with everything in them. Once you load app 1st thing that should the app do is load the correct settings. This is fast and doesn't require any additional knowledge. Everywhere you had something configurable hard coded you replace it with a variable that derives from that file. This should suffice for simple applications.
2.) POWERFULL - For bigger applications with loads of different dependencies and more formal and professional approach do what most people do, use NodeJs, leverage Grunt or Gulp to help you version dependencies and write scripts for different builds. It requires you to install and learn to use new stuff but offers you way more power.
Even better combine this two approaches, link the external dependencies dynamically with the build and internal via including the correct JS file with all the API information.
You can also use other npm modules to assist you with your development like precompiling CSS, lint your code, rearrange and organise files by types into folders, auto-name files or folders.
Basically, MOST of the REPETITIVE tasks you do while developing can be automated this way. This truly shines on bigger projects when all this is handled by someone who is proficient and experienced with the 2nd approach, it can save months of repetitive tasks across the team.
You could use gulp (or even npm) as a task runner to do this kind of work. Maybe this article would help? https://www.keithcirkel.co.uk/how-to-use-npm-as-a-build-tool/
The main point of a task runner is that you can set a specific variable ie prod or development, and the task runner can then switch out your config files with the present definitions for prod or dev config.
In order to build static web pages to serve interactive news applications (mostly data visualizations scripted with something like D3 or lightweight apps built with Bootstrap.js), is it worth the effort to install and configure a build system like Middleman or Grunt.js?
The reason why I am asking is because I am not familiar with Ruby and Ruby Gems at all, and do not have the time and willingness at the moment to dwelve deeper into it, as I already know Python & Javascript pretty well.
Do I need to know Ruby in order to use Middleman productively? Or should I just use Grunt? I am asking because I want to reuse the same templates over and over again and heard that this is easy with Middleman.
There is a distinct difference between Middleman and Grunt.
Grunt is a generic task runner while Middleman is a static site generator.
Middleman for example would be used to create a site with multiple URLs. You put some files in a folder and Middleman takes care of arranging them, creating subfolders for URLs, rendering layouts, compressing assets and in the end you have a bunch of files that represent the site. All this is built in.
You don’t necessarily need to know a lot of Ruby to get started with Middleman, but to get the most out of it, understand how it works and change configurations to your needs, basic Ruby knowledge is key.
Grunt on the other hand works by using plugins and telling them exactly what to do. By default it does nothing. You could think of Grunt as "framework", ready for you to add tasks. Compress these two css files. Combine these JS files and move theme over here. Things like that.
For building a complete website I recommend Middleman. In order to be able to use the templates you create in another language like JS, you could use Jade as a templating language. By default Middleman uses ERB (part of the Ruby standard library). Another popular choice is HAML, but I’m not sure if there is a decent JS compiler for it.
If you want to have really fine grained control over the output of your JS files (you’re talking about JS client side apps in your question) you can even use Grunt and Middleman together. Grunt would take care of the assets (you can have really detailed configurations there) and Middleman would handle templating, URL generation and all the other "default website related" stuff.
If you're keen to use Grunt as a base for this sort of build, you may like to look at Assemble.io.
This is a static-site generator like Middleman, but one that is completely (currently) dependant upon Grunt tasks in order to do its thing.
I believe the learning curve for Assemble.io is rather higher than Middleman. It is based on Node, rather than Ruby. However I don't believe it's necessary to know Ruby in order to get a lot from Middleman.
You may have a look to http://wintersmith.io/ a static page generator like Jekill or MiddleMan (I don't really know this one)
With Grunt you have something at a lower level: Grunt provides you tasks to automate things so you'll have to find or write the task that fits your needs.
I am in the process of writing a heavy Javascript app, which will ultimately be used by injecting one script into clients websites.
As of now I am writing all the modules in one JS file, however I am quickly finding that to be ineffective, as it feels very messy and cluttered, and I feel like the modules should all be in separate files.
My question is, what is a good approach to managing this. Should I write all the apps modules in separate files, and than compile them into one on the Server?
If it matters, I am using Node.js for my server.
First point : don't try to code everything in one file. Most big javascript applications contain dozens of files.
Use some kind of makefile to concatenate your js (and css) files. And after that use a minifier (I use Google Closure Compiler). To help debug, my deployement scripts always make two versions in parallel : one non concatenated/minified and one concatenated/minified. The uncompressed version enables the development/test onsite without any deployement operation.
This means that, as for all big application development, you need some kind of deployement toolchain to orchestrate the operations. This may be based on shell scripts, maven, ant, etc.
Secondly : use classes (without abuse, javascript isn't really OOP) and namespaces to clearly isolate your functions.
Yes, keep all your files logically separate and then minify and combine them as a publish step or on the fly when serving them. Scott Hansleman wrote a very good blog post on why you should do this here.
I'm trying to figure out how to best manage Javascript file dependencies and have that drive the packaging of a 100% front-end app. In short, I'm building an application using backbone.js along with some other libraries. I want an organized codebase and would like to be able to declare dependencies within each file. Ideally those declarations would a) drive the order in which files are loaded by the browser (while in development I want the files to load separately) and drive the order in which the packaging scripts load the scripts for concatenation (I'm aiming to serve a single file for the entire app).
I've been reading on requirejs and commonjs but I'm not convinced.
I have a simple shell script right now that uses cat <file> <file> <file> <file> > concatenated.file to do what I want but it's a pain to maintain that list of files up to date and in the right order. It'd be much easier to be able to declare the dependency at the begining of each javascript file and have the packager and loaders be smart about using that information to concatenate/load scripts.
Any suggestions?
Thanks you,
Luis
I am partial to stealjs myself. It's part of JavascriptMVC but no reason why you can't use it with Backbone.js
The nice part about this one is that it builds your app for production including minifying your css and js and neatly packing all of it into 2 files: production.css and production.js.
It can handle loading non JS files too so you can do things like steal('somefile.css').then(function() {...});
For files, its very much like you would do in other languages:
steal(dep1, dep2, dep3).then(function () {
// code
});
For complex frontend apps Asynchronous Module Definition (AMD) format is best choice. And it's alot of great loaders that supports AMD (curl.js, RequireJS).
I recomend this articles to learn about modern approaches in javascript dependecy management:
Writing Modular JavaScript With AMD, CommonJS & ES Harmony
Why AMD?
For packaging take into account CommonJS specifications, there are few implementations and it's a matter of taste, but in any case I recommend to choose tools, that is compliant with some of that specifications.
It'd be much easier to be able to declare the dependency at the begining of each javascript file and have the packager and loaders be smart about using that information to concatenate/load scripts.
I have had the same idea several months ago and are working on a dependency resolver for my Resource Builder which already makes it easier for me (including the need to distinuish between development and deployed version, with the optional debug parameter).
JsDoc Toolkit (and related efforts), which syntax is supported e. g. by Eclipse JSDT, provides a #requires tag, so you could use that. But resolving dependencies is still not a trivial task (as you can see in ResourceBuilder::resolveDeps()). (The ultimate goal is to use static code analysis to resolve dependencies automatically, without any tags.)
This would reduce the current
<script type="text/javascript" src="builder?src=object,types,dom,dom/css"></script>
to a mere
<script type="text/javascript" src="builder?src=dom/css"></script>
As for asynchronous loaders: The good thing about asynchronous loaders is that they are fast. The bad thing about asynchronous loaders is that – if they work; they are all based on a non-standard approach – they are so fast that you cannot be sure that the features the scripts provide are available in following scripts. So you have to have your code executed by their listeners. I recommend avoiding them unless you really have features in your application that are only needed on demand.
So we're writing a full-text search framework MongoDb. MongoDB is pretty much javascript-native, so we wrote the javascript library first, and it works.
Now I'm trying to write a python framework for it, which will be partially in python, but partially use those same stored javascript functions - the javascript functions are an intrinsic part of the library. On the other hand, the javascript framework does not depend on python. since they are pretty intertwined it seems like it's worthwhile keeping them in the same repository.
I'm trying to work out a way of structuring the whole project to give the javascript and python frameworks equal status (maybe a ruby driver or whatever in the future?), but still allow the python library to install nicely.
Currently it looks like this: (simplified a little)
javascript/jstest/test1.js
javascript/mongo-fulltext/search.js
javascript/mongo-fulltext/util.js
python/docs/indext.rst
python/tests/search_test.py
python/tests/__init__.py
python/mongofulltextsearch/__init__.py
python/mongofulltextsearch/mongo_search.py
python/mongofulltextsearch/util.py
python/setup.py
I've skipped out a few files for simplicity, but you get the general idea; it' a pretty much standard python project... except that it depends critcally ona whole bunch of javascript which is stored in a sibling directory tree.
What's the preferred setup for dealing with this kind of thing when it comes to setuptools? I can work out how to use package_data etc to install data files that live inside my python project as per the setuptools docs.
The problem is if i want to use setuptools to install stuff, including the javascript files from outside the python code tree, and then also access them in a consistent way when I'm developing the python code and when it is easy_installed to someone's site.
Is that supported behaviour for setuptools? Should i be using paver or distutils2 or Distribute or something? (basic distutils is not an option; the whole reason I'm doing this is to enable requirements tracking) How should i be reading the contents of those files into python scripts?
The short answer is that none of the Python distribution tools is going to do what you want, the exact way you want it. Even if you use distutils' data_files feature, you're still going to have to have your javascript files copied into your Python project directory (i.e., somewhere under the same directory as your setup.py.)
Given that, you might as well just copy the .js files to your package (i.e. alongside mongofulltextsearch/init.py) as part of your build process, and use package_data or include_package_data=True.
(Or alternatively, you could possibly use symlinks, externals, or some such, if your revision control system supports those. I believe that when building source distributions, the Python distribution tools convert symlinks to real files. At least, you could give that a try.)