I need to store a lot of text in WebSQl, so I decided to compress the text with zip.js
and store the compressed Blobs.
From the documentation you can compress a blob as follows:
function zipBlob(filename, blob, callback) {
// use a zip.BlobWriter object to write zipped data into a Blob object
zip.createWriter(new zip.BlobWriter("application/zip"), function(zipWriter) {
// use a BlobReader object to read the data stored into blob variable
zipWriter.add(filename, new zip.BlobReader(blob), function() {
// close the writer and calls callback function
zipWriter.close(callback);
});
}, onerror);
}
Although this works, I don't understand why you need to specify a filename. Is this really necessary? And, is this file always removed after compression?
Check this answer here - it does not require a filename, and I'll bet its much easier to use. I've tried quite a few javascript compression/decompression implementations and have been stung by problems such as limits on size of original data, overall speed, efficiency, and so forth. It is oddly difficult to find a good compression/decompression implementation in javascript, but thankfully this one hasn't failed me yet (and I've used it quite a bit):
Compressing a blob in javascript
The implementation you have currently requires the filename because it is trying to be consistent with the zip, so that you can save it for example to a desktop and open it with your favorite zip utility. It sounds like your challenge is very similar to mine, I needed to save and restore compressed items out of local storage in the browser as well as on the server.
The filename is necessary according to this implementation. It wouldn't be necessary if you were only compressing the data, but zip.js builds zip files, which store files, which must have filenames.
In your original example, zipWriter.add() effectively converts your blob into a new file and adds it to a zip – and the "filename" parameter is the name you want that new file to have.
Here's an example that uses zip.js to add multiple blobs to a zip and then downloads it with FileSaver.js:
function zipBlob() {
zip.createWriter(new zip.BlobWriter("application/zip"), function(writer) {
files = ["abcd", "123"];
var f = 0;
function nextFile(f) {
fblob = new Blob([files[f]], { type: "text/plain" });
writer.add("file"+f, new zip.BlobReader(fblob), function() {
// callback
f++;
if (f < files.length) {
nextFile(f);
} else close();
});
}
function close() {
// close the writer
writer.close(function(blob) {
// save with FileSaver.js
saveAs(blob, "example.zip");
});
}
nextFile(f);
}, onerror);
}
Related
I am creating an App for Android using Cordova, and I would like to open and display a file (PDF or image) that is served from the server as Base64-encoded binary data.
Of course I have read the multiple other posts on the subject that already exist on this website, but none of the proposed solutions have worked for me, more details below.
To be more precise, the server sends a JSON-file to the app, which among many other things contains a string consisting of the base64-encoded contents of a PDF file. I want to convert this data back into the represented PDF and display it to the user.
If this were a pure browser page, I would simply package my base64 data into a data-URL, attach this as the href of some anchor, and add a download-attribute. Optionally I could wrap all of my data into a blob and create an object url for that first.
In Cordova, this does not work. Clicking the <a> does nothing. Here is what I have attempted so far:
Using the file plugin, I can write the binary data to a file on the device. This works, and using a terminal I can see that the file was downloaded correctly, but into an app-private directory which I cannot access normally (e.g. through the file explorer).
Accessing the user's "downloads" folder is blocked by the file system
Using window.open with the file path as the first argument and "_system" as the target does nothing. There is no error but also nothing happens. Setting the target to "_blank" instead, I get an error saying ACCESS_DENIED.
Using cordova.InAppBrowser behaves the same was as window.open
With the plugin file-opener2 installed, the app will not compile, because the plugin is looking for an android4 toolchain, and I am building for android 9 and up
The plugin document-viewer (restricting to PDFs for the time being) suffers the same problem and does not compile.
Passing the data-URI to window.open (or cordova.InAppBrowser) directly loads for a very long time and eventually tells me that the desired page could not be loaded.
The PDF file I am using for testing is roughly 17kb after converting to base64. I know this is technically above the spec for how long data-URIs can be, but Chrome in the browser has no trouble with it whatsoever, and using a much shorter URI (only a few dozen bytes) produces the same behavior.
Ideally, what I would like to do, is download the file and then trigger the user's standard browser to open the file itself. That was, I would not have to deal with MIME types and also it would look exactly how the user expected from their own device.
Alternatively, if that doesn't work, I would be ok with downloading the file into a system-wide directory and prompting the user to open it themselves. This is not optimal, but I would be able to swallow that pill.
And lastly, if there is a plugin or some other solution that solves the problem amazingly, but for PDFs only, then I can also work out something else for images (e.g. embedding a new into my app and assigning the URI to that).
I would be thankful for any suggestion you might have on how to solve this problem. The code I use to download the file currently is shown below.
Thank you for your time.
var filePath = cordova.file.externalDataDirectory; // Note: documentsDirectory is set to "" by Cordova, so I cannot use that
var fileName = "someFileName.pdf";
var mime = "application/pdf";
var dataBlob = /* some blob containing the binary data for a PDF */
function writeFile(fileEntry, dataBlob) {
// Create a FileWriter object for our FileEntry.
// This code is taken directly from the cordova-plugin-file documentation
fileEntry.createWriter(function (fileWriter) {
fileWriter.onwriteend = function() {
console.log("Successful file write...");
readFile(fileEntry);
};
fileWriter.onerror = function (e) {
console.log("Failed file write: " + e.toString());
};
fileWriter.write(dataBlob);
});
}
window.resolveLocalFileSystemURL(
filePath,
function onResolveSuccess (dirEntry) {
dirEntry.getFile(
fileName,
{ create: true },
function onGetFileSuccess (file) (
writeFile(file, dataBlob);
// At this point, the file has been downloaded successfully
window.open(file.toURL(), "_system"); // This line does nothing, and I don't understand why.
}
);
}
);
I managed to solve the problem.
As per the documentation of the file-opener2 plugin, you need to also add the androidx-adapter plugin to correct for the outdated (android 4) packages. With the plugins file, file-opener2 and androidx-adapter installed, the complete code is the following:
var filePath = cordova.file.externalDataDirectory; // Note: documentsDirectory is set to "" by Cordova, so I cannot use that
var fileName = "someFileName.pdf";
var mime = "application/pdf";
var dataBlob = /* some blob containing the binary data for a PDF */
function writeFile(fileEntry, dataBlob) {
// Create a FileWriter object for our FileEntry.
// This code is taken directly from the cordova-plugin-file documentation
fileEntry.createWriter(function (fileWriter) {
fileWriter.onwriteend = function() {
console.log("Successful file write...");
readFile(fileEntry);
};
fileWriter.onerror = function (e) {
console.log("Failed file write: " + e.toString());
};
fileWriter.write(dataBlob);
});
}
window.resolveLocalFileSystemURL(
filePath,
function onResolveSuccess (dirEntry) {
dirEntry.getFile(
fileName,
{ create: true },
function onGetFileSuccess (file) (
writeFile(file, dataBlob);
// At this point, the file has been downloaded successfully
cordova.plugins.fileOpener2.open(
filepath + filename,
mime,
{
error : function(){ },
success : function(){ }
}
);
}
);
}
);
Can somebody explain, how is it possible, that jszip is not compressing files? I am trying to use compression and the compressed zip file size is bigger than file size of uncompressed files inside. Am I doing something wrong?
var fs = require("fs");
var JSZip = require("jszip");
var zip = new JSZip();
zip.file('try.txt', 'Hello World ');
zip.generateNodeStream({type:'nodebuffer',streamFiles:true}).pipe(fs.createWriteStream('out.zip')).on('finish', function () {
// JSZip generates a readable stream with a "end" event,
// but is piped here in a writable stream which emits a "finish" event.
console.log("out.zip written.");
});
I see the issue also on the official page when downloading example. https://stuk.github.io/jszip/
As I can see, the file is the same usually as uncompressed file and it is doing no compression at all.
Ok, I took a look at the example on their website as you suggested. I made an obviously compressable file like I suggested in my comment (and you followed up on), download a 100KB file, obviously uncompressed. I then unzipped that and rezipped it using both windows and 7zip and resulted in a ~1KB file instead. You're definitely right, jszip's own example is creating an uncompressed zip.
For the case on their website, generate_async() doesn't compress by default, you have to pass compression options like so, and adjusting the call on their website does work, like so:
var zip = new JSZip();
zip.file("Hello.txt", "11111111 I had a bunch more 1s that I removed\n");
var img = zip.folder("images");
img.file("smile.gif", imgData, {base64: true});
zip.generateAsync({type:"blob",
/* NOTE THESE ADDED COMPRESSION OPTIONS */
/* deflate is the name of the compression algorithm used */
compression: "DEFLATE",
compressionOptions: {
/* compression level ranges from 1 (best speed) to 9 (best compression) */
level: 9
}})
.then(function(content) {
// see FileSaver.js
saveAs(content, "example.zip");
});
Similarly, for your call I believe adding similar options should resolve it based on the generateNodeStream() documentation:
zip.generateNodeStream(
{type:'nodebuffer',streamFiles:true,compression: "DEFLATE", compressionOptions: {level: 9}}
).pipe(fs.createWriteStream('out.zip')).on('finish', function () {
// JSZip generates a readable stream with a "end" event,
// but is piped here in a writable stream which emits a "finish" event.
console.log("out.zip written.");
});
Also, you're not the first to have this issue. Here's another user on their github who fell into the same trap: https://github.com/Stuk/jszip/issues/503 I commented on that issue suggesting they change the examples to default to using compression to help avoid this problem. I didn't create a separate issue for it, but if you were passionate about it you could do so.
In my app there is the ability to generate PDFs. The library I am using doesn't dispose of them after they are created and download. Eventually, after generating enough PDFs, the stored blob data becomes so large that it crashes the browser.
One solution would be to refresh the browser after print. That would really screw up the user experience though because they would lose their current print parameters.
The solution to that would be to store the print parameters in sessionStorage or something so that they are restored when a refresh is done.
At any rate, the easiest solution to implement would be just to clear out the specific blob data, or clear all blob data, once the print is done.
This is what I am seeing chrome://blob-internals. The first item is what I want to delete; the two items below the line are just URL references to the blob.
How do I clear this out either specifically or just clearing out all blobs in Javascipt?
Tried to do the following:
handleDownload(filename, blob, url, e) {
e.preventDefault();
saveAs(blob, filename);
this.props.pdfPrint(false);
delete blob;
}
But this gives a Deleting local variable in strict mode error. Tried to:
handleDownload(filename, blob, url, e) {
e.preventDefault();
saveAs(blob, filename);
this.props.pdfPrint(false);
URL.revokeObjectURL(url);
var test = {};
test = blob
delete test.blob;
}
Which just deletes one of the references to the blob, but not the blob it self.
In reading this, all references need to be removed first:
https://stackoverflow.com/a/22899690/3123109
I'm trying to read and write raw data from files using Mozilla's add-on SDK. Currently I'm reading data with something like:
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var data = NetUtil.readInputStreamToString(inputStream, inputStream.available(),{charset:"UTF-8"});
callback(data, status, nsiFile);
});
}
This works for text files, but when I start messing with raw bytes outside of Unicode's normal range, it doesn't work. For example, if a file contains the byte 0xff, then that byte and anything past that byte isn't read at all. Is there any way to read (and write) raw data using the SDK?
You've specified an explicit charset in the options to NetUtil.readInputStream.
When you omit the charset option, then the data will be read as raw bytes. (Source)
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
// Do not specify a charset at all!
var data = NetUtil.readInputStreamToString(inputStream, inputStream.available());
callback(data, status, nsiFile);
});
}
The suggestion to use io/byte-streams is OK as well, but keep in mind that that SDK module is still marked experimental, and that using ByteReader via io/file as the example suggests is not a good idea because this would be sync I/O on the main thread.
I don't really see the upside, as you'd use NetUtil anyway.
Anyway, this should work:
const {ByteReader} = require("sdk/io/byte-streams");
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var reader = new ByteReader(inputStream);
var data = reader.read(inputStream);
reader.close();
callback(data, status, nsiFile);
});
}
Also, please keep in mind that reading large files like this is problematic. Not only will the whole file buffered in memory, obviously, but:
The file is read as a char (byte) array first, so there will be a temporary buffer in the stream of at least file.size length (via asyncFetch).
Both NetUtil.readInputStreamToString and ByteReader will use another char (byte) array to read the result into from the inputStream, but ByteReader will do that in 32K chunks, while NetUtil.readInputStreamToString, will use a big buffer of file.length.
The data is then read into the resulting jschar/wchar_t (word) array aka. Javascript string, i.e. you need file.size * 2 bytes in memory at least.
E.g., reading a 1MB file would require more than fileSize * 4 = 4MB memory (NetUtil.readInputStreamToString) and/or more than fileSize * 3 = 3MB memory (ByteReader) during the read operation. After the operation, 2MB of that memory will be still alive to store the resulting data in a Javascript string.
Reading a 1MB file might be OK, but a 10MB file might be already problematic on mobile (Firefox for Android, Firefox OS) and a 100MB would be problematic even on desktop.
You can also read the data directly into an ArrayBuffer (or Uint8Array), which has more efficient storage for byte arrays than a Javascript string and avoid the temporary buffers of NetUtil.readInputStreamToString and/or ByteReader.
function readnsIFile(fileName, callback){
var nsiFile = new FileUtils.File(fileName);
NetUtil.asyncFetch(nsiFile, function (inputStream, status) {
var bs = Cc["#mozilla.org/binaryinputstream;1"].
createInstance(Ci.nsIBinaryInputStream);
bs.setInputStream(inputStream);
var len = inputStream.available();
var data = new Uint8Array(len);
reader.readArrayBuffer(len, data.buffer);
bs.close();
callback(data, status, nsiFile);
});
}
PS: The MDN documentation might state something about "iso-8859-1" being the default if the charset option is omitted in the NetUtil.readInputStreamToString call, but the documentation is wrong. I'll fix it.
I'm writing a web browser app (client-side) that downloads a huge amount of chunks from many locations and joins them to build a blob. Then that blob is saved to local filesystem as a common file. The way I'm doing this is by mean of ArrayBuffer objects and a blob.
var blob = new Blob([ArrayBuffer1, ArrayBuffer2, ArrayBuffer3, ...], {type: mimetype})
This works ok for small and medium-sized files (until 700 MB aprox), but browser crashes with larger files. I understand that RAM memory has its limits. The case is that I need to build the blob in order to generate a file, but I wanna allow users to download files much larger than that size (imagine, for instance, files about 8GB).
¿How can I build the blob avoiding size limits? LocalStorage is more limited than RAM, so I do not know what to use or how to do it.
It looks like you are just concatenating arrays of data together? Why not go about appending the array-buffers together in a giant blob. You'd have to iterate and append each arrayBuffer one at a time. You would seek to the end of the filewriter to append arrays. And for reading only portions of your giant blob back you get a slice of the blob to avoid the browser crashing.
Appending Function
function appendToFile(fPath,data,callback){
fs.root.getFile(fPath, {
create: false
}, function(fileEntry) {
fileEntry.createWriter(function(writer) {
writer.onwriteend = function(e) {
callback();
};
writer.seek(writer.length);
var blob = new Blob([data]);
writer.write(blob);
}, errorHandler);
}, errorHandler);
}
Again to avoid reading the entire blob back, only read portions/chunks of your giant blob when generating the file you mention.
Partial Read Function
function getPartialBlobFromFile(fPath,start,stop,callback){
fs.root.getFile(fPath, {
creation:false
}, function(fileEntry){
fileEntry.file(function(file){
var reader = new FileReader();
reader.onloadend = function(evt){
if(evt.target.readyState == FileReader.DONE){
callback(evt.target.result);
}
};
stop = Math.min(stop,file.size);
reader.readAsArrayBuffer(file.slice(start,stop));
}, errorHandler)
}, errorHandler);
}
You may have to keep indexes, perhaps in a header section of your giant BLOB - I would need to know more before I could give more precise feedback.
Update - avoiding quota limits, Temporary vs Persistent
in response to your comments below
It appears that you are running into issues with storage quota because you are using temporary storage. The following is a snippet borrowed from google found here
Temporary storage is shared among all web apps running in the browser. The shared pool can be up to half of the of available disk space. Storage already used by apps is included in the calculation of the shared pool; that is to say, the calculation is based on (available storage space + storage being used by apps) * .5 .
Each app can have up to 20% of the shared pool. As an example, if the total available disk space is 50 GB, the shared pool is 25 GB, and the app can have up to 5 GB. This is calculated from 20% (up to 5 GB) of half (up to 25 GB) of the available disk space (50 GB).
To avoid this limit you'll have to switch to persistent, it will allow you to quota up to the available free space on the disk. To do this use the following to initialize the File-system instead of the temporary storage request.
navigator.webkitPersistentStorage.requestQuota(1024*1024*5,
function(gB){
window.requestFileSystem(PERSISTENT, gB, onInitFs, errorHandler);
}, function(e){
console.log('Error', e);
})