Develop Reference

Django: render staticfiles through template engine at deploy-time

I want to render some static files (*.js in particularly) using Django template variables. I believe this is a common use-case, especially when doing anything AJAX-y; I don't want to hardcode AJAX urls in my .js files any more than I want to hardcode them in my .html files. Buuuut of course I don't want those static files to have to run through the template engine at every client request because this will be slow. I am referring to things like URLs (which do not change after compile/deploy) or static (non-db) model attributes. (I suppose there are use cases where these things might be changed at run-time - this is Python, after all- but I think they are uncommon). For some possible template variables (e.g. model fields), of course the file must be rendered at the time of the client request, but this is not what I'm talking about.
So wouldn't it make sense to render some of my static files through the template engine, for a subset of possible template variables, perhaps at the same time as collectstatic?
As far as I can tell this is not currently the case. To be clear, what I am looking for is a solution to render static files through the template engine at compile/deploy-time so that at "client-request-time" they are in fact plain old static files.
Such an approach would avoid these hacks:
DRY URLs in Django Javascript
Using the Django URL Tag in an AJAX Call
Disclaimers:
Yes I know there are template engines out there for javascript (mustache, handlebars, prototype, etc). Why should I add another template engine to the stack when Django already has one? Plus the syntax collides! That seems silly.
This looks like it takes a crack at it, but it's complicated and not fully implemented.
So:
Is there a solution out there that I am missing?
If not, is there a way to hook into collectstatic (like a pre-collectstatic hook) that would allow one to render certain static files through the template engine before "collecting" them?
EDIT:
No responses yet...is this a really dumb question, and I'm missing something obvious? If so...go ahead and let me know...

There are several frameworks for Django for same purpose: django-pipeline, django-assets, and etc. which integrates different static files processing strategies, with varying degrees of difficulty configuring.
I use an external tool - Grunt (it requires node.js) - for asset post-processing after collectstatic. It is easier and has a lots of plugins for any purpose (source validation, css/js/images minification, merging, testing and etc.).
It is possible to hook in collectstatic by a custom static files storage with overrided post_process method.
example/settings.py
STATIC_ROOT = 'assets'
STATICFILES_STORAGE = 'example.storage.MyStaticFilesStorage'
example/storage.py
import os
from django.contrib.staticfiles.storage import StaticFilesStorage
from django.core.files.base import ContentFile
from django.template import Template, Context
class MyStaticFilesStorage(StaticFilesStorage):
def post_process(self, paths, dry_run=False, **options):
# don't even dare to process the files if we're in dry run mode
if dry_run:
return
js_template_data = {'foo': 'bar'} # template variables
js_template_extension = '.jst'
js_extension = '.js'
for original_name, (storage, path) in paths.items():
processed = False
saved_name = original_name
original_path, original_extension = os.path.splitext(original_name)
if original_extension == js_template_extension:
with storage.open(path) as original_file:
saved_name = original_path + js_extension
if hasattr(original_file, 'seek'):
original_file.seek(0)
original_source = original_file.read()
c = Context(js_template_data)
saved_source = Template(original_source).render(c)
self.delete(saved_name)
self.delete(original_name)
self._save(saved_name, ContentFile(saved_source))
processed = True
yield original_name, saved_name, processed

A completely different way to approach the problem would be to ask if you really need to get those URLs in javascript--instead, can the Javascript get the URLs from things like data attributes in your HTML?
In other words, you might have wanted:
homepage.html:
<div id="pop-up-modal">pop me up</div>
homepage.js:
$("#pop-up-modal").click(function {
$.ajax("{% url 'some-class-name %}")
...
});
When it can often be more straightforward to do something like:
homagepage.html:
<div id="pop-up-modal" data-popurl="{% url 'some-class-name' %}">pop me up</div>
homepage.js:
$("#pop-up-modal").click(function {
$.ajax($(this).data('popurl'))
...
});

I think that django-medusa would suit your needs.
By setting up a renderer and using the disk based backend, generating the static files would be as easy as:
django-admin.py staticsitegen

You aren't crazy. I was frustrated by this as well and found myself hacking something together for each new Django project I tackled. I think the reason for the lack of direct solutions is that this is mega-drought bone DRY. Its super easy to just hard code these things and call it day. This and the two most common use cases for this involve generating code in one language from code in another which tends to be viewed as suspect.
I've recently published a new Django package django-render-static that solves this problem generically. It piggy-backs on Django's existing template engine infrastructure. A management command called render_static should be run before collectstatic. This command will look for templates registered in settings (or passed as arguments) and render them to your static file location on disk. They're then available for whatever normal static file packaging pipeline you have configured.
I'm sure there are more use cases, but the two most common I've found are providing a reverse utility in static JavaScript thats equivalent to Django's and auto-translating define-like python structures (i.e. choice fields) into JavaScript. The app provides template tags that do both.
The JavaScript url reversal code is guaranteed to be functionally equivalent to Django's reverse function. I won't bother plopping example code down here because it's been well documented.

Using require.js for client side dependancies in Adobe CQ5

I was wondering if anyone had experience using require.js with the Adobe CQ5 platform. I'm writing a Chaplin.js(backbone-based) single page app that will be integrated into the rest of CQ5-based site we're working on. Chaplin requires the use of a module system like AMD/Common.js and I want to make sure my compiled javascript file will usable within CQ5's clientlibs. Is it as simple as adding require.js as a dependency in clientlibs prior to loading in my javascript modules? Someone's insight who has experience in doing this would be greatly appreciated.

I've implemented this as a solution of organize in a better way all the modules such as:
//public/js/modules/myModule.js
define('myModule',[ /* dependencies */] ,function( /* stuff here */ ))
and in the components such:
<% //components/component.jsp %>
<div>
<!-- stuff here -->
</div>
componentJS:
//components/component/clientlibs/js/component.js
require(['modules/myModule']);
and finally I've configured require (config.js) and I've stored the JSs modules in a new different design folder. Located the compiled JS after close </body> to guarantee the JS is always located at the bottom after the HTML.
<!-- page/body.jsp -->
...
<cq:includeClientLib js="specialclientlibs.footer"/>
</body>
solving with this the issue of have "ready" all the content before the JS is executed. I've had some problems to resolve with this async stuff managed for the clienlibs compilation tool, in production the problem was different, however, in development, the order in what CQ compiles the JS has produced me some lacks in terms of order of the JS. The problem really was a little bit more complex than the explanation because the number of JS was really big and the team too, but in simple terms it was the best way I've discovered so far..

The Idea
I think you can compile your Chaplin.js with one of the AMDShims to make it self contained, wraps every dependencies it needs inside a function scope, expose an entry point as global variable (which is a bad practise, but CQ do it all the time...) and then use the compiled.js inside a normal clientlib:
AMD Shims
https://github.com/jrburke/requirejs/wiki/AMD-API-Shims
Example
Here is an example of how we make the one of our libs self-contained:
https://github.com/normanzb/chuanr/blob/master/gruntfile.js
Basically, in source code level the lib "require"s the other modules just as usual. However after compiled, the generated chuanr.js file contains everything its needs, even wrapped a piece of lightweight AMD compatible implementation.
check out compiled file here: https://github.com/normanzb/chuanr/blob/master/Chuanr.js
and the source code: https://github.com/normanzb/chuanr/tree/master/src
Alternative
Alternatively rather than compile every lib you are using to be independent/self-contained, what we do in our project is simply use below amdshim instead of the real require.js. so on cq component level you can call into require() function as usual:
require(['foo/bar'], function(){});
The amd shim will not send the http request to the module immediately, instead it will wait until someone else loads the module.
and then at the bottom of the page, we collect all the dependencies, send the requirements to server side handler (scriptmananger) for dynamic merging (by internally calling into r.js):
var paths = [];
for (var path in amdShim.waiting){
paths.push(path);
}
document.write('/scriptmananger?' + paths.join(','));
The amdShim we are using: https://github.com/normanzb/amdshim/tree/exp/defer

Using multiple pre-compiled templates with Handlebars .js (multiple HTTP requests)?

I think this question will give mine a little more context:
Using pre-compiled templates with Handlebars.js (jQuery Mobile environment)
Basically I'm trying to learn the precompiling stuff, so I can save load time and keep my html documents neat. I haven't started it yet, but based on the above link, every template needs to have it's own file. Isn't that going to be a lot of links to load? I don't want to be making multiple HTTP requests if I don't have to.
So if someone could shed some light, perhaps offer an alternative where I can get the templates out of my html, but not have to load up 100 different template files.

Tools like Grunt.js allow you to have your templates and consume them too. For example, this file compiles the templates and then concatenates them into a single file:
module.exports = function(grunt) {
grunt.loadNpmTasks("grunt-contrib-handlebars");
// Project configuration.
grunt.initConfig({
// Project metadata, used by the <banner> directive.
meta: {},
handlebars: {
dist: {
options: {
namespace: "JST",
wrapped: "true"
},
files: {
"templates.js": [
"./fileA.tmpl",
"./fileB.tmpl"
]
}
}
}
});
// Default task.
grunt.registerTask("default", "handlebars");
};
What I've yet to work out since I'm just getting started with pre-compiled templates is the workflow. I want to be able to have compiled templates when we're running a deployed version of the app but when doing development and debugging I'd much rather have my original individual files in uncompiled form so I can just edit them and reload the page.
Follow Up:
I wanted to come back to this after having worked out some of how to both have my pre-compiled templates when available and use individual templates which can be edited on the fly when people are doing development and debugging work and would like to have quick edit-reload-test cycles without doing Grunt builds.
The answer I came up with was to check for the existence of the JST[] data structure and then further to test and see whether a particular pre-compiled template is present in that structure. If it is then nothing further need be done. If it's not there then the template is loaded (we use RequireJS to do that) and compiled and put into the same JST[] array under the same name it would have if pre-compiled templates had been loaded.
That way when it comes time to actually use the template, the code only looks for it in one place and it's always the same.
In the near future I think we'll likely have RequireJS plugins to perform the test and load/compile code while keeping it simple for developers.

Django-pipeline and javascript dependencies

I'm working on a Django project that uses Django-pipeline for assets, and I keep having issues where I define something in one javascript file that is required by another file, but the second file gets loaded before the first and thus the second file fails to load properly. I can mess with the order things get included into PIPELINE_JS but this is pretty awkward to deal with. In most languages you can do things like require foo to make sure that foo is defined but it seems like with javascript and django-pipeline this isn't possible. I've looked into RequireJS a little but I'm not sure how whether I can use it with django-pipeline. What should I do in this case? What do others who use django-pipeline or django in general do for javascript dependency management?
As a side note, I'm actually using Coffeescript, not straight Javascript, but that doesn't seem to help things any. In rails I could do #= require 'foo' to require another coffeescript file but that seems to be linked to the rails asset pipeline.

The only way to do this is to order 'source_filenames' list accordingly, also remember that those file will be concatenated in this order when running collectstatic.
Pipeline will respect this order, it will also avoid duplicate so that your are safe when doing this :
'base.coffee',
'*.coffee',
There is no "require" syntax for now in django-pipeline.
Hope this helps.

How to efficiently implement clientside javascript templating partials?

I'd like to build my own, but I'm not sure about the best way to do it. A partial is a template that is only a part of another bigger template and which can be inserted into multiple other templates at will.
Templating itself is fairly basic, just string exctraction and concatenation, but clientside partials have me a little stumped.
Here are a few methods I thought about:
1,
I could write a javascript helper function that loads partials through ajax into some form of local storage I suppose, and all subsequent templates that require that particular partial would first look inside local storage. I think this method isn't very safe because local storage isn't always guaranteed. And if I can't save them into local storage, partials would result in too many ajax calls.
2, I could put them all into script tags inside my main html file. This would work reasonably well, especially with head.js (to enable parallel loading of script tags), but still - I think each script tag is a separate call to the server right? That doesn't exactly improve the situation.
3, I could put all templates into a single script tag (or html I guess) and manually filter through some kind of delimiter...like: "#template1(blabla template1 string) #template2(blablabla template2 string) and put those strings into globals.
This would result only into a single call to the server, all the rest is done on the client.
Suggestions? I have looked at existing templating engines, but I can't really determine how they do it. The code is pretty complicated

The approach I took to spec out template calls and on-demand loads for the spec/rewrite of jQuery templates is to pipeline it.
See section 9 of the (early) draft spec, and see the conformance suite tests at the bottom for an example of custom on-demand template loading (Testcase "Main calls and Loaded just in time!" is the relevant one).
The basic gist is that plugin loaders (written in JS) get to hook in between-parsing and compiling to inspect the parse tree. A plugin pass gets an object mapping template names to parse trees. If they see any partial template selectors (to use your parlance) they can try and load any unresolved templates using AJAX calls or file I/O on Node.js, and add the partials to the input object to cause the compiler to compile the just loaded partials along with the public templates.
Efficiency-wise, see the benchmarks. I'm in the process of migrating the code to github : https://github.com/mikesamuel/jquery-jquery-tmpl-proposal in case you want to collaborate.