I know you can include css and images, among other file types, which have been stored in base64 form within a javascript file. However, those are decently huge... and gzipped, they shrink down a LOT, even with the ~33% overhead from base64 encoding.
Non-gzipped, images are data:image/gif;base64, data:image/jpeg, data:image/png, and css is data:text/css;base64. What mime type can/should I be using, then, to include css or image data URIs which are gzipped? (Or if gzip+base64 can't work, is there any other compression I can do to bring down the string's size, while still keeping the data stored within the javascript?)
..edit..
I think the question is being misunderstood. I am not asking if I should include gzipped base64 strings within javascript. Yes, I know it's best, in most cases, to gzip the javascript and other files on the server end. But that is not applicable for a userscript; a userscript has no server, and consists of only a single file. Firefox allows a #require directive, but Opera and Chrome do not, and local file security issues come into play with loading any local files. Thus anything needed by the script has to be either: 1) on the web (slow) or 2) embedded in the userscript (big).
Now this question assumes that big is preferable to slow, but that big does not have to mean we totally ignore just how big; if it can be smaller, that's an improvement.
So assuming that a base64 string is embedded in javascript, the question is how to make it into something meaningful.
Either:
1) atob() can convert raw base64-encoded gzip to raw gzip within javascript. (atob does not need to know the mediatype). The question then would be how to decompress that raw gzipped css or image file so that the resulting output can be fed into the document.
or 2) given the proper mediatype, browsers at least theoretically (per the datauri RFC) should be able to load any file directly from a datauri. "" is sufficient to load a non-gzipped css stylesheet. The question here would be what link type attribute and datauri mediatype combination should work (and which browsers would it work for)? Preferably, for a userscript, this would be a combination that works in Opera, FF, and Chrome.
In HTTP, compression is most often only applied for transmission to reduce the payload that is to be transmitted. This is done by the Content-Encoding header field.
But the data URL scheme is very limited and you can only specify the media type:
dataurl := "data:" [ mediatype ] [ ";base64" ] "," data
Although you could use a multipart message, most user agents don’t support them in data URLs. It would also be questionable whether the additional data to describe such a multipart message wouldn’t be more than the data you safe by compressing the actual payload.
So compressing the data in a data URL is possible in theory but impracticable. It is better to simply compress the whole document the data URL is embedded in.
Related
We have a scenario where we are sending multiple json files in a single response. These json files are stored as separate blobs in the backend (aerospike blobstore) and are fetched dynamically in response to a single request.
As long as we send these blobs uncompressed its working fine. i.e. we add a separator after each blob and use this separator to isolate each json blob something like this -
{
// first json here
}
-- JSONEND--blobid1
{
// second json here
}
-- JSONEND--blobid2
and so on.
As long as the blobs are uncompressed from the source i.e. blob store it works fine and we are able to isolate each json in javascript into a separate variable after parsing.
But our challenge is - these blobs are precompressed and saved into the blobstore for various reasons (performance / reduced diskspace ) and we want to simply send these compressed blobs in one response to the client. Scripts on client side use these blobs and parse them into separate json object trees.
Is this possible ? how ? We need to support only chrome and possibly firefox.
if you using Gzip and you have a nodejs server
see here, how to use gzjoin
gzjoin.c
join gzip files without recalculating the crc or recompressing
- illustrates the use of the Z_BLOCK flush parameter for inflate()
- illustrates the use of crc32_combine()
https://github.com/nodejs/node/tree/master/deps/zlib/examples
Zlib documentation:
https://nodejs.org/api/zlib.html#zlib_zlib
this is correct answer:
gzlog.c
gzlog.h
efficiently and robustly maintain a message log file in gzip format
- illustrates use of raw deflate, Z_PARTIAL_FLUSH, deflatePrime(),
and deflateSetDictionary()
- illustrates use of a gzip header extra field
OR
zpipe.c
reads and writes zlib streams from stdin to stdout
- illustrates the proper use of deflate() and inflate()
- deeply commented in zlib_how.html (see above)
OR
zran.c
index a zlib or gzip stream and randomly access it
- illustrates the use of Z_BLOCK, inflatePrime(), and
inflateSetDictionary() to provide random access
My program Precomp can be used to further compress already compressed file formats like GIF, PNG, PDF, ZIP and more. Roughly summarized, it does this by decompressing the compressed streams, recompressing them and storing the differences between the expected compressed stream and the actual compressed stream. As an example, this rotating earth picture from Wikipedia is compressed from 1429 KB to 755 KB. The process is lossless, so the original GIF file can be restored.
The algorithm for the GIF file format can be isolated and implemented relatively easy, so I was thinking about a proof-of-concept implementation in JavaScript. This would lead to the web server sending a compressed version of the GIF file (.pcf ending, essentially a bzip2 compressed file of the
GIF image contents) and the client decompressing the data, recompressing to GIF and displaying it. The following things would've to be done:
The web site author would've to compress his GIF files using the standard version of Precomp and serve these instead of the GIF files together with a JavaScript for the client side recompression.
The client would decompress the bzip2 compressed file, this could be done using one of the existing bzip2 Javascript implementations.
The client would recompress the image content into the original GIF file.
The process is trade of bandwidth against CPU usage on the client side.
Now my questions are the following:
Are there any general problems with the process of loading a different file and "converting" it to GIF?
What would you recommend to display before the client side finishes (image placeholder)?
What do I have to do to make sure the .pcf file is cached? Bandwidth savings were useless if doesn't get cached.
Is there a way to display the original GIF if JavaScript is deactivated, but avoid loading the GIF if JavaScript is activated?
Can I give the users a way to configure the behaviour? E.g. on mobile devices, some might avoid bandwidth, but others might want less CPU usage.
Would it be possible to display interlaced GIFs as supposed (going from a rough version to the final image)? This would require updating the image content multiple times at different stages of recompression.
Let's begin by answering your specific questions. Code example below.
Q&A
Are there any general problems with the process of loading a different file and "converting" it to GIF?
The main problem is complication. You are effectively writing a browser addon, like those for JPEG2000.
If you are writing real browser addons, each major browsers do it differently, and change addon formats occasionally, so you have to actively maintain them.
If you are writing a JS library, it will be easier to write and maintain, but it will be unprivileged and suffer from limitations such as cross original restriction.
What would you recommend to display before the client side finishes (image placeholder)?
Depends on what your format can offer.
If you encode the image dimension and a small thumbnail early, you can display an accurate place-holder pretty early.
It is your format, afterall.
What do I have to do to make sure the .pcf file is cached? Bandwidth savings were useless if doesn't get cached.
Nothing different from other files.
Configure the Expires and Cache-Control header on server side and they will be cached.
Manifest and prefetch can also be used.
Is there a way to display the original GIF if JavaScript is deactivated, but avoid loading the GIF if JavaScript is activated?
This is tricky. When JavaScript is disabled, you can only replace elements, not attributes.
This means you cannot create an image somewhere that points to the .pcf files, and ask browser to rewrite the src attribute when JS is unavailable.
I think the best solution to support no JS is outputting the images with document.write, using noscript as fall back:
<noscript>
<img src=demo.gif width=90>
</noscript><script>
loadPcf("demo.pcf","width=90")
</script>
(Some library or framework may make you consider <img src=demo.gif data-pcf=demo.pcf>.
This will not work for you, because browsers will preload 'demo.gif' before your script kicks in, causing additional data transfer.)
Alternatively, browser addons are unaffected by "disable JS" settings, so if you make addons instead then you don't need to worry about it.
Can I give the users a way to configure the behaviour? E.g. on mobile devices, some might avoid bandwidth, but others might want less CPU usage.
Perhaps. You can code a user interface and store the preference in cookie or in localStorage.
Then you can detect preference and switch the logic in server code (if cookie) or in client code.
If you are doing addons, all browsers provide reliable preference mechanism.
The problem is that, again, every browser do it differently.
Would it be possible to display interlaced GIFs as supposed (going from a rough version to the final image)? This would require updating the image content multiple times at different stages of recompression.
If you hands browsers a partial image, they may think the image is corrupted and refuse to show it.
In this case you have to implement your own GIF decoder AND encoder so that you can hands browser a complete placeholder image, just to be safe.
(new) Can I decode image loaded from another site?
I must also repeat the warning that non-addon JS image decoding does not work with cross origin images.
This means, all .pcf files must be on the same server, same port, and same protocol with the site using it.
For example you cannot share images for multiple sites or do optimisations like domain sharding.
Code Example
Here is a minimal example that creates an <img>, loads a gif, half its width, and put it back to the <img>.
To support placeholder or progressive loading, listen to onprogress instead of/in addition to onload.
<!DOCTYPE html><html><head><meta charset="UTF-8"><script>
function loadPcf( file, attr ) {
var atr = attr || '', id = loadPcf.autoid = 1 + ~~loadPcf.autoid;
document.write( '<img id=pcf'+id+' ' + atr + ' />' );
var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer'; // IE 10+ only, sorry.
xhr.onload = function() { // I am loading gif for demo, you can load anything.
var data = xhr.response, img = document.querySelector( '#pcf' + id );
if ( ! img || ! data instanceof ArrayBuffer ) return;
var buf = new DataView( data ), head = buf.getUint32(0), width = buf.getUint16(6,1);
if ( head !== 1195984440 ) return console.log( 'Not a GIF: ' + file ); // 'GIF8' = 1195984440
// Modify data, image width in this case, and push it to the <img> as gif.
buf.setInt16( 6, ~~(width/2), 1 );
img.src = URL.createObjectURL( new Blob( [ buf.buffer ], { type: "image/gif" } ) );
};
xhr.open( 'GET', file );
xhr.send();
}
</script>
<h1>Foo<noscript><img src=a.gif width=90></noscript><script>loadPcf("a.gif","width=90")</script>Bar</h1>
If you don't need <noscript> compatibility (and thus prevent facebook/google+ from seeing the images when the page is shared), you can put the pcf file in <img> src and use JS to handle them en mass, so that you don't need to call loadPcf for each image and will make the html much simpler.
How about <video>?
What you envisioned is mostly doable, in theory, but perhaps you should reconsider.
Judging from the questions you ask, it will be quite difficult for you to define and pull off your vision smoothly.
It is perhaps better to encode your animation in WebM and use <video> instead.
Better browser support back to IE 9 - just add H.264 to make it a dual format video. You need IE 10+ to modify binary data.
Size: Movies has many, many options and tricks to minimise size, and what you learned can be reused in the future.
Progressive: <video> have had some techs for adaptive video, and hopefully they will stabilise soon.
JavaScript: <video> does not depend on JavaScript.
Future-proof: Both WebM and H.264 will be supported by many programs, long after you stopped working on your special format.
Cost-effective: Create a low-bandwith option using smaller or lower quality media is easier and more reliable than a custom format. This is why wikipedia and youtube offers their media in different resolutions.
For non-animations, PNG can also be colour indexed and 7z optimised (keeping the PNG format).
Icon size indexed PNG is often smaller than the same GIF.
Or perhaps your vision (as described in the pcf website) is the capability to encode many different files, not only GIF.
This will be more like supporting a new network protocol, and is likely beyond the scope of humble JavaScript. (e.g. how are you going to handle pdf download or streaming?)
I have an SQL Server table that contains two columns - an integer ID and a varbinary(max) ImageData, which is a JPEG image. In Javascript, I have an Ajax call to get the ImageData for a particular ID and return it as a byte array. How do I generate an XHTML IMG tag that will display the image?
My thought was to use createObjectURL on the returned array, but (a) this needs to be able to run on IE 8, which doesn't seem to support the method, and (b) Firefox appears to be expecting an actual Blob type rather than an array of bytes (and the Blob() constructor doesn't appear to exist).
I do have an alternative - return the data as a Base64 string, then use the tag, but that does not work in IE8, and there may be a size limitation on other browsers (some of the images are 600K, which means that the tag could have over 1 million characters).
You are over-engineering a problem where a simpler solution already exists.
Insert the following HTML using AJAX and let the browser fetch the image itself via HTTP. Sending out image data via AJAX would require a pointless base64 encode/decode and ends up taking up more bandwith and defeats the browser cache.
<img src="imageServer.php?imageID=3224" />
Then imageServer.php (you build this or grab it somewhere) will grab the appropriate image record, spit out an image MIME type, then stream the binary data.
Backstory
I wrote a specialized image inliner script that is intended to be used with both GreaseMonkey and Google Chrome. It is supposed to download PNG files and store them in data: urls in image src attributes. This may sound ridiculous, but a certain website sets Content-Disposition to attachment for images, and I don't want the «Save As» dialog to pop up every time.
The actual question
The script fetches data with an XMLHttpRequest, encodes it into base64 and stores it in a proper place. So far, good. But it only works when I run it through both Firebug and Chrome dev consoles, and does not when I use it as a proper userscript. As far as I understand, this is because Greasemonkey scripts cannot use XMLHttpRequest objects directly and should rely on calls to GM_xmlhttpRequest instead. However, I cannot set responseType to "blob" or "arraybuffer" that way, and the binary parameter seems to only work for sending data through POST requests. I only get Unicode strings.
Just in case, the images are served from the same domain as the page that links to them. I believe it satisfies the «same origin» thingy.
http://wiki.greasespot.net/GM_xmlhttpRequest here are the GM_xmlhttpRequest docs.
Is there a way to fetch an arraybuffer from within a Greasemonkey userscript?
If it is same-domain, then you can use XMLHttpRequest, with no problems. The only reason to use GM_xmlhttpRequest (which currently has a crippled subset of functionality) is if the images/files are cross domain.
For same-domain, you can use XHR2 as shown in this answer.
For cross-domain, you must: use GM_xmlhttpRequest, override the mime-type, and use a custom encoder algorithm. Again, this is all shown in that same answer.
However, it sounds like you are just trying to make it easier to download images? If that is so, then you might be better off just using the excellent DownThemAll extension.
overrideMimeType String (Compatibility: 0.6.8+) Optional. A MIME type
to specify with the request (E.G. "text/html; charset=ISO-8859-1").
You can set this to plain/text; charset=x-user-defined (the type doesn't matter but the charset does), bitwise AND through the response string and add the values to a typed array and get the buffer:
var text = xhr.responseText,
len = text.length,
arr = new Uint8Array(len),
i = 0;
for( i = 0; i < len; ++i ) {
arr[i] = text.charCodeAt(i) & 0xFF;
}
arr.buffer //The arraybuffer
Note: this is for raw binary responses, not base64.
What is the best way to generate image data from the contents of an HTML canvas element?
I'd like to create the image data such that it can be transmitted to a server (it's not necessary for the user to be able to directly save to a file). The image data should be in a common format such as PNG or JPEG.
Solutions that work correctly in multiple browsers are preferred, but if every solution depends on the browser, recent versions of Firefox should be targeted.
Firefox and Opera have a toDataURL() method that returns a data-URL formatted PNG. You can assign the result to a form field to submit it to the server.
The data URL is base-64 encoded, so you will have to decode it on the server side. You would also need to strip off the "data:image/png;" part of course.
I think a lib you can use is Canvas2Image, it uses native features from Canvas, but it won't work on any browser. I have an optimized version of this lib, if you want to, I'll share it with you.
Then you could get the generated Data URI and send it using Ajax to the server.