Develop Reference

Large blob file in Javascript

I have an XHR object that downloads 1GB file.
function getFile(callback)
{
var xhr = new XMLHttpRequest();
xhr.onload = function () {
if (xhr.status == 200) {
callback.apply(xhr);
}else{
console.log("Request error: " + xhr.statusText);
}
};
xhr.open('GET', 'download', true);
xhr.onprogress = updateProgress;
xhr.responseType = "arraybuffer";
xhr.send();
}
But the File API can't load all that into memory even from a worker
it throws out of memory...
btn.addEventListener('click', function() {
getFile(function() {
var worker = new Worker("js/saving.worker.js");
worker.onmessage = function(e) {
saveAs(e.data); // FileSaver.js it creates URL from blob... but its too large
};
worker.postMessage(this.response);
});
});
Web Worker
onmessage = function (e) {
var view = new DataView(e.data, 0);
var file = new File([view], 'file.zip', {type: "application/zip"});
postMessage('file');
};
I'm not trying to compress the file, this file is already compressed from server.
I thought storing it first on indexedDB but i i'll have to load blob or file anyway, even if i do request by range bytes, soon or late i will have to build this giant blob..
I want to create blob: url and send it to user after been downloaded by browser
I'll use FileSystem API for Google Chrome, but i want make something for firefox, i looked into File Handle Api but nothing...
Do i have to build an extension for firefox, in order to do the same thing as FileSystem does for google chrome?
Ubuntu 32 bits

Loading 1gb+ with ajax isn't convenient just for monitoring download progress and filling up the memory.
Instead I would just send the file with a Content-Disposition header to save the file.
There are however ways to go around it to monitor the progress. Option one is to have a second websocket that signals how much you have downloaded while you are downloading normally with a get request. the other option will be described later in the bottom
I know you talked about using Blinks sandboxed filesystem in the conversation. but it has some drawbacks. It may need permission if using persistent storage. It only allows 20% of the available disk that are left. And if chrome needs to free some space then it will throw away any others domains temporary storage that was last used for the most recent file. Beside it doesn't work in private mode.
Not to mention that it has been dropping support for it and may never end up in other browsers - but they will most likely not remove it since many sites still depend on it
The only way to process this large file is with streams. That is why I have created a StreamSaver. This is only going to work in Blink (chrome & opera) ATM but it will eventually be supported by other browsers with the whatwg spec to back it up as a standard.
fetch(url).then(res => {
// One idea is to get the filename from Content-Disposition header...
const size = ~~res.headers.get('Content-Length')
const fileStream = streamSaver.createWriteStream('filename.zip', size)
const writeStream = fileStream.getWriter()
// Later you will be able to just simply do
// res.body.pipeTo(fileStream)
// instead of pumping
const reader = res.body.getReader()
const pump = () => reader.read()
.then(({ value, done }) => {
// here you know how large the value (chunk) is and you can
// figure out the download speed/progress when comparing it to the size
return done
? writeStream.close()
: writeStream.write(value).then(pump)
)
// Start the reader
pump().then(() =>
console.log('Closed the stream, Done writing')
)
})
This will not take up any memory

I have a theory that is if you split the file into chunks and store them in the indexedDB and then later merge them together it will work
A blob isn't made of data... it's more like pointers to where a file can be read from
Meaning if you store them in indexedDB and then do something like this (using FileSaver or alternative)
finalBlob = new Blob([blob_A_fromDB, blob_B_fromDB])
saveAs(finalBlob, 'filename.zip')
But i can't confirm this since i haven't tested it, would be good if someone else could

Blob is cool until you want to download a large file, there is a 600MB limit(chrome) for blob since it stores everything in memory.

Is there anything similar to Chrome's fileSystem API for Firefox addons?

I have made a Chrome app that relies heavily on Chrome's fileSystem API to record and save video streams from various websites. Since the stream data is processed in javascript before being saved, simply downloading the streams doesn't work.
Now I am considering making a Firefox version...
I know that Firefox has a sandboxed file system API, but as far as I know, it is not possible to save the files to the physical file system.
Only option I can see is creating a blob from the sandboxed file system and download that blob.
I have actually two questions:
Are there any options I have missed to create and save files directly in the physical file system from Firefox addons?
Even if I have to rely on the sandboxed file system, is it possible to open files in append mode, ie. to append data to existing files?

Yes to your first question: there is the io/file API. Opening a file returns a stream (io/bytestream). Examples from the docs
function readBinaryDataFromFile (filename) {
var fileIO = require("sdk/io/file");
var data = null;
if (fileIO.exists(filename)) {
var ByteReader = fileIO.open(filename, "rb");
if (!ByteReader.closed) {
data = ByteReader.read();
ByteReader.close();
}
}
return data;
}
function writeBinaryDataToFile(data, filename) {
var fileIO = require("sdk/io/file");
var ByteWriter = fileIO.open(filename, "wb");
if (!ByteWriter.closed) {
ByteWriter.write(data);
ByteWriter.close();
}
}

Download blobs locally using Safari

I'm trying to find a cross browser way to store data locally in HTML5. I have generated a chunk of data in a Blob (see MDN). Now I want to move this Blob to the actual filesystem and save it locally. I've found the following ways to achieve this;
Use the <a download> attribute. This works only in Chrome currently.
Microsoft introduces a saveAs function in IE 10 which will achieve this.
Open the Blob URL in the browser and save it that way.
None of these seems to work in Safari though. While (1) works in Chrome, (2) in IE and (3) in Firefox no one works in Safari 6. The download attribute is not yet implemented and when trying to open a blob using the URL Safari complains that URLs starting with blob: are not valid URLs.
There is a good script that encapsulates (1) and (3) called FileSaver.js but that does not work using the latest Safari version.
Is there a way to save Blobs locally in a cross browser fashion?

FileSaver.js has beed updated recently and it works on IE10, Safari5+ etc.
See: https://github.com/eligrey/FileSaver.js/#supported-browsers

The file name sucks, but this works for me in Safari 8:
window.open('data:attachment/csv;charset=utf-8,' + encodeURI(csvString));
UPDATE: No longer working in Safari 9.x

The only solution that I have come up with is making a data: url instead. For me this looks like:
window.open("data:image/svg+xml," + encodeURIComponent(currentSVGString));

Here data is the array buffer data coming from response while making http rest call in js. This works in safari, however there might me some issue in filename as it comes to be untitled.
var binary = '';
var bytes = new Uint8Array(data);
var len = bytes.byteLength;
for (var i = 0; i < len; i++) {
binary += String.fromCharCode(bytes[i]);
}
var base64 = 'data:' + contentType + ';base64,' + window.btoa(binary);
var uri = encodeURI(base64);
var anchor = document.createElement('a');
document.body.appendChild(anchor);
anchor.href = uri;
anchor.download = fileName;
anchor.click();
document.body.removeChild(anchor);

Have you read this article? http://updates.html5rocks.com/2012/06/Don-t-Build-Blobs-Construct-Them
Relating to http://caniuse.com/#search=blob, blobs are possible to use in safari.
You should consturct a servlet which delivers the blob via standard http:// url, so you can avoid using blob: url. Just make a request to that url and build your blob.
Afterwards you can save it in your filesystem or local storage.

The download attribute is supported since ~safari 10.1, so currently this is the way to go.

This is the only thing that worked for me on safari.
var newWindow = window.open();
const blobPDF = await renderMapPDF(); // Your async stuff goes here
if (!newWindow) throw new Error('Window could not be opened.');
newWindow.location = URL.createObjectURL(blobPDF);

Post binary data cross domain in javascript

I'm writing a Chrome browser extension that takes a snapshot of the current tab's view and uploads it to a web service API that I don't control. The Chrome extension library has a function (chrome.tabs.captureVisibleTab. see http://code.google.com/chrome/extensions/tabs.html) that takes a snapshot and returns the data in a data url. I'm at an impasse as to how to get that data uploaded.
I've tried to write my own multipart-form request and use an ajax request to POST the data. But, ajax insists on UTF-8 encoding the data and the API insists on 8-bit encoded binary. I thought maybe using a file uploader plugin like http://malsup.com/jquery/form/ would work, but I can't seem to get the data from the JS variable into a form the uploader will take.
Any ideas for at least a new path of investigation would be highly appreciated.

Turns out that you can do this.
Chrome has a way to send a blob via XMLHTTPRequest.
Here's a link to example code from the Chromium issue tracker:
http://code.google.com/p/chromium/issues/detail?id=35705#c34
XMLHttpRequest.prototype.sendAsBinary = function(datastr,contentType) {
var bb = new BlobBuilder();
var len = datastr.length;
var data = new Uint8Array(len);
for (var i=0; i<len; i++) {
data[i] = datastr.charCodeAt(i);
}
bb.append(data.buffer);
this.send(bb.getBlob(contentType));
}

How can I get a file's upload size using simple Javascript?

I have upload file functionality on one of the page. I check for the extension of the file using JavaScript. Now i want to restrict the user from uploading file greater than 1 MB. Is there any way i can check the file upload size using JavaScript.
My code currently look like this:
<script language="JavaScript">
function validate() {
var filename = document.getElementById("txtChooseFile").value;
var ext = getExt(filename);
if(ext == "txt" || ext == "csv")
return true;
alert("Please upload Text files only.");
return false;
}
function getExt(filename) {
var dot_pos = filename.lastIndexOf(".");
if(dot_pos == -1)
return "";
return filename.substr(dot_pos+1).toLowerCase();
}
</script>

See http://www.w3.org/TR/FileAPI/. It is supported by Firefox 3.6; I don't know about any other browsers.
Within the onchange event of a <input id="fileInput" type="file" /> simply:
var fi = document.getElementById('fileInput');
alert(fi.files[0].size); // maybe fileSize, I forget
You can also return the contents of the file as a string, and so forth. But again, this may only work with Firefox 3.6.

Now it is possible to get file size using pure JavaScript. Nearly all browser support FileReader, which you can use to read file size as well as you can show image without uploading file to server. link
Code:
var oFile = document.getElementById("file-input").files[0]; // input box with type file;
var img = document.getElementById("imgtag");
var reader = new FileReader();
reader.onload = function (e) {
console.log(e.total); // file size
img.src = e.target.result; // putting file in dom without server upload.
};
reader.readAsDataURL(oFile );
You can get file size directly from file object using following code.
var fileSize = oFile.size;

Other that aquiring the filename there is no way for you to find out any other details about the file in javascript including its size.
Instead you should configure server-side script to block an oversized upload.

Most of these answers are way out-of-date. It is currently possible to determine file size client-side in any browser that supports the File API. This includes, pretty much, all browsers other than IE9 and older.

It might be possible using a lot of browser-specific code. Take a look at the source of TiddlyWiki, which manages to save itself on the user's hard drive by hooking into Windows Scripting Host (IE), XPCOM (Mozilla), etc.

I don't think there is any way of doing that with plain JS from a web page.
With a browser extension maybe, but from a page javascript cannot access the filesystem for security reasons.
Flash and Java should have similar restrictions, but maybe they are a bit less strict.

not possible. would be a major security concern to allow client side scripts to run that can read file info from and end users hard drive.

See here:
http://www.kavoir.com/2009/01/check-for-file-size-with-javascript-before-uploading.html
As to all the people saying this has to be done server side, they are absolutely spot on it does.
In my case though the maximum size I will except is 128Mb, if a user tries to upload something that is 130Mb they should not have to wait the 5 minute upload time to find out it is too big so I need to do an additional check before they submit the page for usability sake.

I had the same issue, Here's a simple JavaScript snippet worked for me. Adding for future googlers.
HTML
<input type="file" name="photo" id="photo" accept="image/*">
JS
const file = document.getElementById('photo');
// Show KB (add one more /1024 for MB)
const filesize = file.files[0].size / 1024;
if (filesize > 500) { // Alert greater than 500kb
console.log(filesize);
alert('Please upload image less than 500 KB');
return;
}