I would like to compress the CSS and JS files on my server to minimise load times, problem.
My hosting is with Streamline.net (big mistake, never go there) who will not activate mod_gzip and mod_deflate due to security issues.
Does anyone have another way to compress these types of files (and image files too if poss) without going the way of mod_gzip and mod_deflate.
Answers would be hugely welcome.
Yes, the answer is Minification.
Obviously, it will not compress as much as gzip or deflate. But it helps, and it is very easy to do with the right tools.
You can run your files through a script which would gzip them for you and add appropriate expiration headers.
Either set up an URL rewrite, or rewrite the URLs manually:
<script src="js/somescript.js"></script>
becomes
<script src="compress.php?somescript.js"></script>
and in compress.php, you can do something like
<?php
$file = 'js/' . basename($_SERVER['QUERY_STRING']);
if (file_exists($file)) {
header ('Last-Modified: ' . date('r',filemtime($file));
header ('Content-Type: text/javascript'); // otherwise PHP sends text/html, which could confuse browsers
ob_start('ob_gzhandler');
readfile($file);
} else {
header('HTTP/1.1 404 Not Found');
}
Obviously this can be extended to also provide HTTP caching, and/or on-the-fly minification, further speeding up your visitors' browsing.
Instead of getting mod_gzip to gzip your CSS and JavaScript files dynamically, you can gzip them yourself, then upload them.
This does introduce another step before you upload CSS and JavaScript, but it works, and maybe even saves a tiny bit of server processing time for each request compared to mod_gzip.
On Mac OS X, gzipping a file on the command line is as easy as, e.g.:
gzip -c styles.css > styles-gzip.css
Make sure these files get served with the right content-type header though.
Just as a sidenote: Compressing images would not be beneficial if these are already saved in a compressed format with the maximum compression that still looks good to the user.
Most programming languages support some data or file compression formats like ZLIB or GZIP. So you could use a programming language to compress your files with one of these formats.
Related
I deployed a .war file through IntelliJ Idea on a Tomcat server.
I noticed a character "ä" was not properly displayed while at other places the same character was displayed correctly. I found out that only the special characters that I hard-coded in my .js files were affected.
I tried to set all my .js files to UTF-8 in IntelliJ, I also changed all standard encoding settings to UTF-8 but the error didn't go away.
All my js files are mapped into one index.js file using webpack, but how exactly I don't know because this is a project initially set up by someone else.
I recently made a new interesting observation:
When I first open up a browser (tested with Firefox and Chrome) it's displayed incorrectly:
On regular reload (F5) nothing changes, but when reloading with CTRL + F5 it's suddenly correct:
This really confused me...does anyone have an idea what might be going on here?
I used to have the same problems with my Java files, but after changing the encoding in my gradle build file that worked.
Ultimately my question is:
What do you think should I change in order for the special characters to always be displayed correctly?
I add a similar problem after a tomcat update on a Windows server: the javascripts content corrupted characters at the browser side.
The http headers were corrects so I investigated a bit further.
On the server, the javascript files were saved in utf-8 without BOM.
With Wireshark, I saw that the character 'é' (C3-A9 in the file UTF-8 encoded ) was transmitted as (C3-83-C2-A9). It means that Tomcat was reading an ANSI file and gently converted it to UTF8!
So I just added the BOM to the saved the files and it fixed the bug. (REM: it is easy to add the BOM with notepad++).
But I didn't want to update all the server files, I wanted tomcat to read UTF-8 correctly.
The easy fix is to define the file encoding in the tomcat web.xml like this:
<servlet>
<servlet-name>default</servlet-name>
<servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-class>
<init-param>
<param-name>debug</param-name>
<param-value>0</param-value>
</init-param>
<init-param>
<param-name>listings</param-name>
<param-value>false</param-value>
</init-param>
<!------------------- add the settings here ------------->
<init-param>
<param-name>fileEncoding</param-name>
<param-value>utf-8</param-value>
</init-param>
<!------------------- end of the added settings ------------->
<load-on-startup>1</load-on-startup>
</servlet>
This really confused me...does anyone have an idea what might be going on here?
Caching. Ctrl+F5 tells the browser to reload the resource even if it has it cached. F5 will reuse the resource from cache if it's in cache.
What do you think should I change in order for the special characters to always be displayed correctly?
You may have already done it given the F5/Ctrl+F5 thing above.
Basically, ensure that:
The files (.js, .html, etc.) are stored in the correct encoding and, when viewed with that encoding, show the characters correctly. Strongly recommend using the same encoding for each type of file, although theoretically it's possible to use UTF-8 for JavaScript files and (say) Windows-1252 for HTML files. But that's just asking for complexity and hassle.
Ensure that every step in the pipeline correctly identifies the encoding being used for the files. That means (for instance) that Tomcat needs to include the header Content-Type: application/javascript; charset=utf-8 or similar for your .js files. (text/javascript; charset=utf-8 will also work, but is obsolete.) For HTML files, though, the W3C recommends including the meta header and omitting the charset from Content-Type.
Ensure that your HTML files identify the encoding in a meta tag near the top of head (within the first 1024 bytes) as well: <meta charset="UTF-8"> The W3C provide several reasons (same link as the bullet above) for doing this, such as saving the file locally and opening it (thus not having an HTTP header), making it clear to human and machine readers, etc.
I read this post
securing the source code in a node-webkit desktop application
I would like to secure my font files and I was thinking this snapshot approach might be a way. So instead of running this
nwsnapshot --extra-code application.js application.bin
Could I run
nwsnapshot --extra-code font_file font_file.bin
Then in package.json add this?
snapshot: 'font_file.bin'
Or would there be an alternative mechanism to reference the binary font? Would it be possible to convert the CSS file referencing the font into binary? Can anything else other than javascript be converted to binary?
One dumb thing you can do is to add your assets to the exe file as stated here:
https://github.com/nwjs/nw.js/wiki/How-to-package-and-distribute-your-apps#step-2a-put-your-app-with-nw-executable
Basically you have to create a zip of your content (included your package.json) and rename it to "package.nw" then you can "merge it" into the exe file by typing this if you're in windows (the link explains how to do this in other OS's):
`copy /b nw.exe+app.nw app.exe `
This is not a great security measure (beacuse it can be opened as a zip file) but is one step further.
Another thing that could add security to your files is to encrypt them and then add them dinamically through js (while decrypting them) for this you could use the encrypt and decrypt methods available in node.
http://lollyrock.com/articles/nodejs-encryption/
However the weakest point in the application is still the packages.json file and for this nw provides nothing.
Cheers!
I know you can include css and images, among other file types, which have been stored in base64 form within a javascript file. However, those are decently huge... and gzipped, they shrink down a LOT, even with the ~33% overhead from base64 encoding.
Non-gzipped, images are data:image/gif;base64, data:image/jpeg, data:image/png, and css is data:text/css;base64. What mime type can/should I be using, then, to include css or image data URIs which are gzipped? (Or if gzip+base64 can't work, is there any other compression I can do to bring down the string's size, while still keeping the data stored within the javascript?)
..edit..
I think the question is being misunderstood. I am not asking if I should include gzipped base64 strings within javascript. Yes, I know it's best, in most cases, to gzip the javascript and other files on the server end. But that is not applicable for a userscript; a userscript has no server, and consists of only a single file. Firefox allows a #require directive, but Opera and Chrome do not, and local file security issues come into play with loading any local files. Thus anything needed by the script has to be either: 1) on the web (slow) or 2) embedded in the userscript (big).
Now this question assumes that big is preferable to slow, but that big does not have to mean we totally ignore just how big; if it can be smaller, that's an improvement.
So assuming that a base64 string is embedded in javascript, the question is how to make it into something meaningful.
Either:
1) atob() can convert raw base64-encoded gzip to raw gzip within javascript. (atob does not need to know the mediatype). The question then would be how to decompress that raw gzipped css or image file so that the resulting output can be fed into the document.
or 2) given the proper mediatype, browsers at least theoretically (per the datauri RFC) should be able to load any file directly from a datauri. "" is sufficient to load a non-gzipped css stylesheet. The question here would be what link type attribute and datauri mediatype combination should work (and which browsers would it work for)? Preferably, for a userscript, this would be a combination that works in Opera, FF, and Chrome.
In HTTP, compression is most often only applied for transmission to reduce the payload that is to be transmitted. This is done by the Content-Encoding header field.
But the data URL scheme is very limited and you can only specify the media type:
dataurl := "data:" [ mediatype ] [ ";base64" ] "," data
Although you could use a multipart message, most user agents don’t support them in data URLs. It would also be questionable whether the additional data to describe such a multipart message wouldn’t be more than the data you safe by compressing the actual payload.
So compressing the data in a data URL is possible in theory but impracticable. It is better to simply compress the whole document the data URL is embedded in.
I was wondering if there was a way to compress javascript in realtime, much like gzip works for HTML (and CSS apparently)?
I don't want to have to compress my file manually before upload everytime, I want the server to do it for me without any added work from lazy coders.
gzip works on all text, including JavaScript
If you want to do more compression (e.g. by using YUI compressor before gzipping) then you could get your server to do it, but it would be easier to do it before uploading.
The trick is to automate the build and publish process — so you run a script that does the compression then uploads the result, rather then drag'n'dropping files manually or similar.
I suggest that you write a small script that uploads the file automatically; then you can compress it before the upload.
The other option is to tell your web server to compress files before transfer (but it should do that automatically).
Another option is a cron job (see cron(1) or the Windows scheduler) on the server which checks the file periodically (say once a day) and compresses it when a new version has been uploaded.
On Apache, this compresses everything except images:
<IfModule mod_deflate.c>
# Insert filter
SetOutputFilter DEFLATE
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</IfModule>
If you mean "compress" as in "maintain the code in exactly the same format but push it down the pipe compressed", then enable gzip compression as per the instructions for your server, which you haven't specified in the tags/question.
If you mean "compress" as in "minify the code", then there are options that you can try, like including a minification module/process in the pipeline. I've, very recently, created a solution in ASP.net that uses the Microsoft Ajax Minifier 4.0 to minify javascript files requested, on the fly basically by adding script tags to the page with a src tag that is similar to minifier.ashx?source=my/javascript/file/and/path/here.js and using the Minifier class in AjaxMin.dll to minify the code on demand. This has several advantages over pre-minification:
You can pass a flag into your live site, via a cookie, a querystring value, a given username, to disable minification which makes debugging that little bit easier if you need to resolve the issue "on live"
You can make emergency changes to your live code without having to change your unminified code, minify it, upload it (obviously not something you should do regularly, but it's nice that the ability is there, if needs be).
There's one less set of build artifacts (the minified js files) to worry about keeping track of, source-controlling, etc,..
Dis-advantages:
The minification adds some overhead to script requests. This can be mitigated by either caching the minification result in memory or on disk
I'm not sure if the license for AjaxMin.dll permits this kind of use
If you're using Visual Studio / another tool that provides intellisense for script, it may not now be able to give you that from your script tags
I've got a site where individual pages might require some javascript or CSS files hooked into their heads. I'm trying to keep everything client side when it comes to managing this process, rather than getting on the FTP and sorting everything out in the code so I need to be able to upload css and js files.
I've got CCK filefield up and running, and it works with css files, but it refuses to upload .js files. It instead seems to view every .js as ".js.txt" and then the file appears on the server as thisismyfile.js.txt
Not ideal...
Does anyone know how to work around this. Is it a mime type problem with Drupal or the server, or is Drupal set up to avoid script uploads and n00b hack attacks.
Once the files are uploaded I intend to use PHP mode on the page or node to call drupal_add_css and drupal_add_js.
Looking at the field_file_save_file() function in field_file.inc from filefield module, you can find the following snippet
// Rename potentially executable files, to help prevent exploits.
if (preg_match('/\.(php|pl|py|cgi|asp|js)$/i', $file->filename) && (substr($file->filename, -4) != '.txt')) {
$file->filemime = 'text/plain';
$file->filepath .= '.txt';
$file->filename .= '.txt';
}
So yes, it's a 'security thing', as Jeremy guessed.
You could patch that RegEx for an immediate 'fix', but that would remove this otherwise useful security check completely for all filefields used on the site.
So a more specific workaround might be a better approach. Since you want to add the files via drupal_add_js() calls from code anyways, you might as well do the renaming there, adding some kind of verification to make sure you can 'trust' the file (e.g. who uploaded it, whatever).
Edit: Concerning options to rename (and alternatives) when calling drupal_add_js():
For renaming the file, look into the file_move() function. A problem with this would be that it won't update the corresponding entry in the files table, so you would have to do that also, if the move operation succeeded. (The filefield just stores the 'fid' of the corresponding entry in the files table, so you'd need to find it there by 'fid' and change the 'filename', 'filepath' and 'filemime' entries according to your rename/move)
Alternatively, you could just load the content of the *.js.txt file and add that string with the 'inline' option of drupal_add_js(). This would be less 'elegant' and could be a performance hit, but if those are not important criteria in your specific case, it is less trouble.
Yet another option would be just passing the *.js.txt file as is to drupal_add_js(), ignoring the 'wrong' extension. A short local test showed that this works (at least in firefox). This might be the 'least effort' solution, but would need some additional testing concerning different browser behavior concerning usage of 'misnamed' js files.
Allowing Drupal to upload javascript files would be a security risk, which is also why it doesn't allow you to do it, but instead appends the .txt extension. The reason is that js files are executable along with php, pl, py, cgi, asp. So if Drupal could upload those files to the server, it would be possible for evil doers to upload a file and run it doing all kinds of nasty things on your server, basically anything is possible. Best thing would be to find a different way of uploading files which are secure.
I had a similar need, and found a way to get around the security by first changing the 'allow_insecure_uploads' variable value by running this line of code in your hook_install:
variable_set('allow_insecure_uploads', 1);
Then in a module add this function
/**
* Implementation of FileField's hook_file_insert().
*/
function MODULE_NAME_file_insert(&$file) {
//look for files with the extenstion .js.txt and rename them to just .js
if(substr($file->filename, -7) == '.js.txt'){
$file_path = $file->filepath;
$new_file_path = substr($file_path, 0, strlen($file_path)-4);
file_move($file_path, $new_file_path);
$file->filepath = $file_path;
$file->filename = substr($file->filename, 0, strlen($file->filename)-4);
$file->filemime = file_get_mimetype($file->filename);
$file->destination = $file->filepath;
$file->status = FILE_STATUS_TEMPORARY;
drupal_write_record('files', $file);
}
What this does is in the hook_insert call it checks if a file has the extension ".js.txt". If it does it copies it to a new location and renames it. This is after the security check so its ok. I don't think you need to worry about the cache clear deleting your js files as long as you don't put them in the files/js directory. Create your own directory for you module and you should be ok.
I faced this situation when I wanted to allow .js file to be upload as is (without .txt and with 'application/javascript' mimetype) for a specific field. Also, I didn't wanted to alter Drupal core... of course.
So I needed to create a module implementing hook_file_presave(). This also work for Multiupload File Widget, since its hook is on file_save().
Note that you would have to replace MYMODULE_NAME and MYFIELD_NAME by your own values.
function MYMODULE_NAME_file_presave($file) {
// Bypass secure file extension for .js for field_additional_js field only
if((isset($file->source) && strpos($file->source, "MYFIELD_NAME") !== FALSE) && substr($file->filename, strlen($file->filename) - 7) == ".js.txt") {
// Define new uri and save previous
$original_uri = $file->uri;
$new_uri = substr($file->destination, null, -4);
// Alter file object
$file->filemime = 'application/javascript';
$file->filename = substr($file->filename, null, -4);
$file->destination = file_destination($new_uri, FILE_EXISTS_RENAME);
$file->uri = $file->destination;
// Move fil (to remove .txt)
file_unmanaged_move($original_uri, $file->destination);
// Display message that says that
drupal_set_message(t('Security bypassed for .js for this specific field (%f).', array('%f' => $file->filename)));
}
}
Drupal also "munges" javascript files. To prevent Drupal from automatically adding underscores to the filename there is a hidden variable that is checked before the filename is "munged".
Setting the variable to 1 solves the issue for me (along with altering the REGEX in includes/file.inc).
I hate hacking core, but this seems like a poor design to me. Javascript files are not server side scripts like php, py, pl, cgi, and asp.
You can use the allowed file extensions settings to prevent php and other server side scripts from being uploaded.
eg:
variable_set('allow_insecure_uploads', 1);
See:
http://api.drupal.org/api/function/file_munge_filename/6
So uploading .js files to the files directory is pretty much impossible.
Even if you manage to get .js files uploaded cleanly, these files will get deleted when the cache is cleared.
Any js files that live inside the files directory will be deleted whenever the drupal_clear_js_cache() function is executed.
http://api.drupal.org/api/function/drupal_clear_js_cache/6
So Drupal sees .js files living in the file uploads directory as temporary.
Now I understand why they are appending ".txt", it is to prevent them from being removed when the cache is cleared.
So as a compromise I guess I will just be uploading .js files manually (via FTP) to the /misc folder. :(