Javascript previews with new FileReader API and DataURLs seem inefficient - javascript

I am using the new FileReader API to preview images before upload. This is done using DataURLs. But DataURLs can be massive if the images are large. This is especially a problem for me as the user may upload multiple images at a time and previewing the bunch has actually slowed my browser considerably and actually crashed chrome a few times.
Is there any alternative to using DataURLs for previewing images on the client before upload?

You can also store data on the client's disk (in another location so that you can access the file using JavaScript). This article is quite extensive when it comes to this subject:
http://www.html5rocks.com/en/tutorials/file/filesystem/
It's not supported on all browsers though.
You have to request storage space (the file system), then create a file, write data to it, and finally fetch the URL:
window.requestFileSystem(window.PERSISTENT, 5*1024*1024, function(fs) {
fs.root.getFile(filename, {create: true}, function(fileEntry) {
fileEntry.createWriter(function(fileWriter) {
var arr = new Uint8Array(data.length);
// fill arr with image byte data here
var builder = new BlobBuilder();
builder.append(arr.buffer);
var blob = builder.getBlob();
fileWriter.write(blob);
location.href = fileEntry.toURL(); // navigate to file. The URL does not contain the data but only the path and filename.
});
});
}, function() {});

Related

How do I open and display a base64 pdf from inside my Cordova App?

I am creating an App for Android using Cordova, and I would like to open and display a file (PDF or image) that is served from the server as Base64-encoded binary data.
Of course I have read the multiple other posts on the subject that already exist on this website, but none of the proposed solutions have worked for me, more details below.
To be more precise, the server sends a JSON-file to the app, which among many other things contains a string consisting of the base64-encoded contents of a PDF file. I want to convert this data back into the represented PDF and display it to the user.
If this were a pure browser page, I would simply package my base64 data into a data-URL, attach this as the href of some anchor, and add a download-attribute. Optionally I could wrap all of my data into a blob and create an object url for that first.
In Cordova, this does not work. Clicking the <a> does nothing. Here is what I have attempted so far:
Using the file plugin, I can write the binary data to a file on the device. This works, and using a terminal I can see that the file was downloaded correctly, but into an app-private directory which I cannot access normally (e.g. through the file explorer).
Accessing the user's "downloads" folder is blocked by the file system
Using window.open with the file path as the first argument and "_system" as the target does nothing. There is no error but also nothing happens. Setting the target to "_blank" instead, I get an error saying ACCESS_DENIED.
Using cordova.InAppBrowser behaves the same was as window.open
With the plugin file-opener2 installed, the app will not compile, because the plugin is looking for an android4 toolchain, and I am building for android 9 and up
The plugin document-viewer (restricting to PDFs for the time being) suffers the same problem and does not compile.
Passing the data-URI to window.open (or cordova.InAppBrowser) directly loads for a very long time and eventually tells me that the desired page could not be loaded.
The PDF file I am using for testing is roughly 17kb after converting to base64. I know this is technically above the spec for how long data-URIs can be, but Chrome in the browser has no trouble with it whatsoever, and using a much shorter URI (only a few dozen bytes) produces the same behavior.
Ideally, what I would like to do, is download the file and then trigger the user's standard browser to open the file itself. That was, I would not have to deal with MIME types and also it would look exactly how the user expected from their own device.
Alternatively, if that doesn't work, I would be ok with downloading the file into a system-wide directory and prompting the user to open it themselves. This is not optimal, but I would be able to swallow that pill.
And lastly, if there is a plugin or some other solution that solves the problem amazingly, but for PDFs only, then I can also work out something else for images (e.g. embedding a new into my app and assigning the URI to that).
I would be thankful for any suggestion you might have on how to solve this problem. The code I use to download the file currently is shown below.
Thank you for your time.
var filePath = cordova.file.externalDataDirectory; // Note: documentsDirectory is set to "" by Cordova, so I cannot use that
var fileName = "someFileName.pdf";
var mime = "application/pdf";
var dataBlob = /* some blob containing the binary data for a PDF */
function writeFile(fileEntry, dataBlob) {
// Create a FileWriter object for our FileEntry.
// This code is taken directly from the cordova-plugin-file documentation
fileEntry.createWriter(function (fileWriter) {
fileWriter.onwriteend = function() {
console.log("Successful file write...");
readFile(fileEntry);
};
fileWriter.onerror = function (e) {
console.log("Failed file write: " + e.toString());
};
fileWriter.write(dataBlob);
});
}
window.resolveLocalFileSystemURL(
filePath,
function onResolveSuccess (dirEntry) {
dirEntry.getFile(
fileName,
{ create: true },
function onGetFileSuccess (file) (
writeFile(file, dataBlob);
// At this point, the file has been downloaded successfully
window.open(file.toURL(), "_system"); // This line does nothing, and I don't understand why.
}
);
}
);
I managed to solve the problem.
As per the documentation of the file-opener2 plugin, you need to also add the androidx-adapter plugin to correct for the outdated (android 4) packages. With the plugins file, file-opener2 and androidx-adapter installed, the complete code is the following:
var filePath = cordova.file.externalDataDirectory; // Note: documentsDirectory is set to "" by Cordova, so I cannot use that
var fileName = "someFileName.pdf";
var mime = "application/pdf";
var dataBlob = /* some blob containing the binary data for a PDF */
function writeFile(fileEntry, dataBlob) {
// Create a FileWriter object for our FileEntry.
// This code is taken directly from the cordova-plugin-file documentation
fileEntry.createWriter(function (fileWriter) {
fileWriter.onwriteend = function() {
console.log("Successful file write...");
readFile(fileEntry);
};
fileWriter.onerror = function (e) {
console.log("Failed file write: " + e.toString());
};
fileWriter.write(dataBlob);
});
}
window.resolveLocalFileSystemURL(
filePath,
function onResolveSuccess (dirEntry) {
dirEntry.getFile(
fileName,
{ create: true },
function onGetFileSuccess (file) (
writeFile(file, dataBlob);
// At this point, the file has been downloaded successfully
cordova.plugins.fileOpener2.open(
    filepath + filename,
    mime,
    {
     error : function(){ },
success : function(){ }
    }
);
}
);
}
);

How to display PDFs fetched from S3 using JavaScript?

I need to fetch a PDF file from s3.amazonaws.com and when I query it using Postman (or paste directly into the browser), it loads fine. However when I try to generate the file path for it (to pass to a viewer later), it didn't work:
fetch(<S3URL>).then(res => res.blob()).then(blob => {
// THIS STEP DOES NOT WORK
let myBlob = new Blob(blob, {type: 'application/pdf'});
// expect something like 'www.mysite.com/my-file.pdf'
let PDFLink = window.URL.createObjectURL(myBlob);
return PDFLink;
}
I'm using Autodesk's Forge PDF viewer and it works perfectly fine for local PDF files:
let myPDFLink = 'public/my-file.pdf';
Autodesk.Viewing.Initializer(options, () => {
viewer = new Autodesk.Viewing.Private.GuiViewer3D(document.getElementById('forgeViewer'));
viewer.start();
viewer.loadExtension('Autodesk.PDF').then( () => {
viewer.loadModel(myPDFLink, viewer); // <-- works fine here
});
});
// from https://github.com/wallabyway/offline-pdf-markup
So, how do I go from the S3 URL (e.g. s3.amazonaws.com/com.autodesk.oss-persistent/0d/ff/c4/2dfd1860d1...) to something the PDF viewer can understand (i.e. has .pdf extension in the URL)?
I know for JSON files I need to do res.json() to extract the JSON content, but for PDFs, what should I do with the res object?
Note: I don't have control over the S3 URL. Autodesk generates a temporary S3 link whenever I want to download documents from their BIM360 portal.
I tried a lot of options and the only way I could display a PDF fetched via API calls is by using an object element:
<object data='<PDF link>' type='application/pdf'>
Converting the downloaded blob to base64 doesn't work. Putting the PDF link in an iframe doesn't work either (it still downloads instead of displaying). All the options I have read only work if the PDFs are part of the frontend application (i.e. local files, not something fetched from a remote server).

File Uploading ReadAsDataUrl

I have a question about the File API and uploading files in JavaScript and how I should do this.
I have already utilized a file uploader that was quite simple, it simply took the files from an input and made a request to the server, the server then handled the files and uploaded a copy file on the server in an uploads directory.
However, I am trying to give people to option to preview a file before uploading it. So I took advantage of the File API, specifically the new FileReader() and the following readAsDataURL().
The file object has a list of properties such as .size and .lastModifiedDate and I added the readAsDataURL() output to my file object as a property for easy access in my Angular ng-repeat().
My question is, it occurred to me as I was doing this that I could store the dataurl in a database rather than upload the actual file? I was unsure if modifying the File data directly with it's dataurl as a property would affect its transfer.
What is the best practice? Is it better to upload a file or can you just store the dataurl and then output that, since that is essentially the file itself? Should I not modify the file object directly?
Thank you.
Edit: I should also note that this is a project for a customer that wants it to be hard for users to simply take uploaded content from the application and save it and then redistribute it. Would saving the files are urls in a database mitigate against right-click-save-as behavior or not really?
There is more then one way to preview a file. first is dataURL with filereader as you mention. but there is also the URL.createObjectURL which is faster
Decoding and encoding to and from base64 will take longer, it needs more calculations, more cpu/memory then if it would be in binary format.
Which i can demonstrate below
var url = 'https://upload.wikimedia.org/wikipedia/commons/c/cc/ESC_large_ISS022_ISS022-E-11387-edit_01.JPG'
fetch(url).then(res => res.blob()).then(blob => {
// Simulates a file as if you where to upload it throght a file input and listen for on change
var files = [blob]
var img = new Image
var t = performance.now()
var fr = new FileReader
img.onload = () => {
// show it...
// $('body').append(img)
var ms = performance.now() - t
document.body.innerHTML = `it took ${ms.toFixed(0)}ms to load the image with FileReader<br>`
// Now create a Object url instead of using base64 that takes time to
// 1 encode blob to base64
// 2 decode it back again from base64 to binary
var t2 = performance.now()
var img2 = new Image
img2.onload = () => {
// show it...
// $('body').append(img)
var ms2 = performance.now() - t2
document.body.innerHTML += `it took ${ms2.toFixed(0)}ms to load the image with URL.createObjectURL<br><br>`
document.body.innerHTML += `URL.createObjectURL was ${(ms - ms2).toFixed(0)}ms faster`
}
img2.src = URL.createObjectURL(files[0])
}
fr.onload = () => (img.src = fr.result)
fr.readAsDataURL(files[0])
})
The base64 will be ~3x larger. For mobile devices I think you would want to save bandwidth and battery.
But then there is also the latency of doing a extra request but that's where http 2 comes to rescue

Large blob file in Javascript

I have an XHR object that downloads 1GB file.
function getFile(callback)
{
var xhr = new XMLHttpRequest();
xhr.onload = function () {
if (xhr.status == 200) {
callback.apply(xhr);
}else{
console.log("Request error: " + xhr.statusText);
}
};
xhr.open('GET', 'download', true);
xhr.onprogress = updateProgress;
xhr.responseType = "arraybuffer";
xhr.send();
}
But the File API can't load all that into memory even from a worker
it throws out of memory...
btn.addEventListener('click', function() {
getFile(function() {
var worker = new Worker("js/saving.worker.js");
worker.onmessage = function(e) {
saveAs(e.data); // FileSaver.js it creates URL from blob... but its too large
};
worker.postMessage(this.response);
});
});
Web Worker
onmessage = function (e) {
var view = new DataView(e.data, 0);
var file = new File([view], 'file.zip', {type: "application/zip"});
postMessage('file');
};
I'm not trying to compress the file, this file is already compressed from server.
I thought storing it first on indexedDB but i i'll have to load blob or file anyway, even if i do request by range bytes, soon or late i will have to build this giant blob..
I want to create blob: url and send it to user after been downloaded by browser
I'll use FileSystem API for Google Chrome, but i want make something for firefox, i looked into File Handle Api but nothing...
Do i have to build an extension for firefox, in order to do the same thing as FileSystem does for google chrome?
Ubuntu 32 bits
Loading 1gb+ with ajax isn't convenient just for monitoring download progress and filling up the memory.
Instead I would just send the file with a Content-Disposition header to save the file.
There are however ways to go around it to monitor the progress. Option one is to have a second websocket that signals how much you have downloaded while you are downloading normally with a get request. the other option will be described later in the bottom
I know you talked about using Blinks sandboxed filesystem in the conversation. but it has some drawbacks. It may need permission if using persistent storage. It only allows 20% of the available disk that are left. And if chrome needs to free some space then it will throw away any others domains temporary storage that was last used for the most recent file. Beside it doesn't work in private mode.
Not to mention that it has been dropping support for it and may never end up in other browsers - but they will most likely not remove it since many sites still depend on it
The only way to process this large file is with streams. That is why I have created a StreamSaver. This is only going to work in Blink (chrome & opera) ATM but it will eventually be supported by other browsers with the whatwg spec to back it up as a standard.
fetch(url).then(res => {
// One idea is to get the filename from Content-Disposition header...
const size = ~~res.headers.get('Content-Length')
const fileStream = streamSaver.createWriteStream('filename.zip', size)
const writeStream = fileStream.getWriter()
// Later you will be able to just simply do
// res.body.pipeTo(fileStream)
// instead of pumping
const reader = res.body.getReader()
const pump = () => reader.read()
.then(({ value, done }) => {
// here you know how large the value (chunk) is and you can
// figure out the download speed/progress when comparing it to the size
return done
? writeStream.close()
: writeStream.write(value).then(pump)
)
// Start the reader
pump().then(() =>
console.log('Closed the stream, Done writing')
)
})
This will not take up any memory
I have a theory that is if you split the file into chunks and store them in the indexedDB and then later merge them together it will work
A blob isn't made of data... it's more like pointers to where a file can be read from
Meaning if you store them in indexedDB and then do something like this (using FileSaver or alternative)
finalBlob = new Blob([blob_A_fromDB, blob_B_fromDB])
saveAs(finalBlob, 'filename.zip')
But i can't confirm this since i haven't tested it, would be good if someone else could
Blob is cool until you want to download a large file, there is a 600MB limit(chrome) for blob since it stores everything in memory.

Filesystem API - Upload from local drive to local filesystem

Ive read a lot about the filesystem API and HTML5, but i just couldn't find a working solution so i ask you guys:
I want to have a file upload form, drag drop or regular input box doesnt matter, however i want to select a file, and after uploading it should take the file or a whole folder and "upload" it to the filesystem located on the clients computer. The upload is in brackets because i actually want to copy the file/folder to the clients local file system.
Is it even possible? Because i want to make an application, where a user can upload his files such as music or large videos and movies to his local filesystem and edit/watch etc them in my application. I know i have to upload those big files i have to cut them into pieces and load them stacked up, but i just want to start little :)
Thanks in advance
There's indeed little information on this subject at the moment, so I put together an example that combines:
Using the webkitdirectory attribute on <input type="file">.
This allows the user to select a directory using an appropriate dialog box.
Using the Filesystem API.
This is about the sandboxed filesystem which allows you to store files on the client's machine.
Using the File API.
This is the API that allows you to read files. The files are accessible through an <input type="file"> element, through a transfer using drag and drop, or through the Filesystem API.
As these are currently only working nicely in Chrome, I used the webkit prefix where necessary.
http://jsfiddle.net/zLna6/3/
The code itself has comments which I hope are clear:
var fs,
err = function(e) {
throw e;
};
// request the sandboxed filesystem
webkitRequestFileSystem(
window.TEMPORARY,
5 * 1024 * 1024,
function(_fs) {
fs = _fs;
},
err
);
// when a directory is selected
$(":file").on("change", function() {
$("ul").empty();
// the selected files
var files = this.files;
if(!files) return;
// this function copies the file into the sandboxed filesystem
function save(i) {
var file = files[i];
var text = file ? file.name : "Done!";
// show the filename in the list
$("<li>").text(text).appendTo("ul");
if(!file) return;
// create a sandboxed file
fs.root.getFile(
file.name,
{ create: true },
function(fileEntry) {
// create a writer that can put data in the file
fileEntry.createWriter(function(writer) {
writer.onwriteend = function() {
// when done, continue to the next file
save(i + 1);
};
writer.onerror = err;
// this will read the contents of the current file
var fr = new FileReader;
fr.onloadend = function() {
// create a blob as that's what the
// file writer wants
var builder = new WebKitBlobBuilder;
builder.append(fr.result);
writer.write(builder.getBlob());
};
fr.onerror = err;
fr.readAsArrayBuffer(file);
}, err);
},
err
);
}
save(0);
});
$("ul").on("click", "li:not(:last)", function() {
// get the entry with this filename from the sandboxed filesystem
fs.root.getFile($(this).text(), {}, function(fileEntry) {
// get the file from the entry
fileEntry.file(function(file) {
// this will read the contents of the sandboxed file
var fr = new FileReader;
fr.onloadend = function() {
// log part of it
console.log(fr.result.slice(0, 100));
};
fr.readAsBinaryString(file);
});
}, err);
});
That is not possible, exactly, but your app can still probably work. Reading the file is possible through a file input form element, but writing the file back to disk is where you'll run into trouble.
The two ways your browser can write to disk are 1) downloading a file and 2) the HTML5 filesystem API. Option #1 obviously doesn't let your application choose the destination and option #2 only works with browser-created sandbox filesystems. That restriction might not be a deal-breaker for you -- it just means that the folders that your app uses will be buried somewhere in your browser's data files.
Also, the Filesystem API is currently Chrome-only (but it is an open standard). If you want cross-platform support, maybe you can use IndexedDB. You could use localStorage, but Chrome has a hard 5MB limit, which would be terrible for a media application.

Categories

Resources