Read large (1G) binary file as it loads in Javascript [duplicate] - javascript

This question already has answers here:
filereader api on big files
(2 answers)
Closed 5 years ago.
I have always fully loaded binary files and then gone on to scan the content blocks within. This was fine for anything up to say 500K. I'm now looking at scanning files that are as large as 1G or higher. (Client side)
Loading files greater then 5mb (as large as 1G) is not great, and I would like to move to a process where as the file is loading I can grab the blocks of code and process it. The file format is made up of blocks that have a size in it. So I would be able to grab the header, the first block and do parsing as the file loads.
If anyone knows where i can find some good examples of code like this working, or useful texts I can read, I would be very grateful.
My current code for loading is as follows. Jquery change on an input box, calls another function to load the file into memory, then processes it. The scanMyFile(buffer) is the function where the arraybuffer is then been sent to do its work identifying everything in the file.
$("#myfile").change(function (e) {
try {
readMyFile(e);
} catch (error) {
alert("Found this error : " + error);
}
});
function readMyFile(evt) {
var f = evt.target.files[0];
if (f) {
var r = new FileReader();
r.onload = function (e) {
var buffer = r.result;
scanMyFile(buffer);
};
r.readAsArrayBuffer(f);
} else {
alert("Failed to load file");
}
}

A File objects is also an instance of a Blob, which offers the .slice method to create a smaller view of the file.
Might want to check out this question:
filereader api on big files

Related

How do I open and display a base64 pdf from inside my Cordova App?

I am creating an App for Android using Cordova, and I would like to open and display a file (PDF or image) that is served from the server as Base64-encoded binary data.
Of course I have read the multiple other posts on the subject that already exist on this website, but none of the proposed solutions have worked for me, more details below.
To be more precise, the server sends a JSON-file to the app, which among many other things contains a string consisting of the base64-encoded contents of a PDF file. I want to convert this data back into the represented PDF and display it to the user.
If this were a pure browser page, I would simply package my base64 data into a data-URL, attach this as the href of some anchor, and add a download-attribute. Optionally I could wrap all of my data into a blob and create an object url for that first.
In Cordova, this does not work. Clicking the <a> does nothing. Here is what I have attempted so far:
Using the file plugin, I can write the binary data to a file on the device. This works, and using a terminal I can see that the file was downloaded correctly, but into an app-private directory which I cannot access normally (e.g. through the file explorer).
Accessing the user's "downloads" folder is blocked by the file system
Using window.open with the file path as the first argument and "_system" as the target does nothing. There is no error but also nothing happens. Setting the target to "_blank" instead, I get an error saying ACCESS_DENIED.
Using cordova.InAppBrowser behaves the same was as window.open
With the plugin file-opener2 installed, the app will not compile, because the plugin is looking for an android4 toolchain, and I am building for android 9 and up
The plugin document-viewer (restricting to PDFs for the time being) suffers the same problem and does not compile.
Passing the data-URI to window.open (or cordova.InAppBrowser) directly loads for a very long time and eventually tells me that the desired page could not be loaded.
The PDF file I am using for testing is roughly 17kb after converting to base64. I know this is technically above the spec for how long data-URIs can be, but Chrome in the browser has no trouble with it whatsoever, and using a much shorter URI (only a few dozen bytes) produces the same behavior.
Ideally, what I would like to do, is download the file and then trigger the user's standard browser to open the file itself. That was, I would not have to deal with MIME types and also it would look exactly how the user expected from their own device.
Alternatively, if that doesn't work, I would be ok with downloading the file into a system-wide directory and prompting the user to open it themselves. This is not optimal, but I would be able to swallow that pill.
And lastly, if there is a plugin or some other solution that solves the problem amazingly, but for PDFs only, then I can also work out something else for images (e.g. embedding a new into my app and assigning the URI to that).
I would be thankful for any suggestion you might have on how to solve this problem. The code I use to download the file currently is shown below.
Thank you for your time.
var filePath = cordova.file.externalDataDirectory; // Note: documentsDirectory is set to "" by Cordova, so I cannot use that
var fileName = "someFileName.pdf";
var mime = "application/pdf";
var dataBlob = /* some blob containing the binary data for a PDF */
function writeFile(fileEntry, dataBlob) {
// Create a FileWriter object for our FileEntry.
// This code is taken directly from the cordova-plugin-file documentation
fileEntry.createWriter(function (fileWriter) {
fileWriter.onwriteend = function() {
console.log("Successful file write...");
readFile(fileEntry);
};
fileWriter.onerror = function (e) {
console.log("Failed file write: " + e.toString());
};
fileWriter.write(dataBlob);
});
}
window.resolveLocalFileSystemURL(
filePath,
function onResolveSuccess (dirEntry) {
dirEntry.getFile(
fileName,
{ create: true },
function onGetFileSuccess (file) (
writeFile(file, dataBlob);
// At this point, the file has been downloaded successfully
window.open(file.toURL(), "_system"); // This line does nothing, and I don't understand why.
}
);
}
);
I managed to solve the problem.
As per the documentation of the file-opener2 plugin, you need to also add the androidx-adapter plugin to correct for the outdated (android 4) packages. With the plugins file, file-opener2 and androidx-adapter installed, the complete code is the following:
var filePath = cordova.file.externalDataDirectory; // Note: documentsDirectory is set to "" by Cordova, so I cannot use that
var fileName = "someFileName.pdf";
var mime = "application/pdf";
var dataBlob = /* some blob containing the binary data for a PDF */
function writeFile(fileEntry, dataBlob) {
// Create a FileWriter object for our FileEntry.
// This code is taken directly from the cordova-plugin-file documentation
fileEntry.createWriter(function (fileWriter) {
fileWriter.onwriteend = function() {
console.log("Successful file write...");
readFile(fileEntry);
};
fileWriter.onerror = function (e) {
console.log("Failed file write: " + e.toString());
};
fileWriter.write(dataBlob);
});
}
window.resolveLocalFileSystemURL(
filePath,
function onResolveSuccess (dirEntry) {
dirEntry.getFile(
fileName,
{ create: true },
function onGetFileSuccess (file) (
writeFile(file, dataBlob);
// At this point, the file has been downloaded successfully
cordova.plugins.fileOpener2.open(
    filepath + filename,
    mime,
    {
     error : function(){ },
success : function(){ }
    }
);
}
);
}
);

Javascript not saving all images at once

So, this is my problem. I am trying to save all the images in the local drive from Javascript and the code that does that looks like this:
window.onload = function () {
console.log(" IN ON LOAD FUNCTION ")
for (filename=1; filename<=80; filename++)
{
console.log(filename)
html2canvas(document.querySelector(`.lead${filename}`)).then(canvas => {
canvas.toBlob(function(blob) {
saveAs(blob, `${filename}.png`);
});
});
}
}
So the code is quite clear. I have about 80 divs with different class name and I am extracting the text from it and saving it as canvas and downloading that canvas as png format, but the problem is sometimes it downloads about 10 images, and sometimes it doesn't download at all. Any help regarding this issue? Any other way around?

Reading files from CD using HTML5, webkitdirectory takes more time compared to reading local files

I have a web application which allows the user to upload DICOM and Non-DICOM files to their account. I am using JavaScript, HTML5, Webkitdirectory, Chrome and Datatable to populate selected files on UI. The issue i am facing is -
While selecting files from their local machine the following code seems to work pretty fast and the selected files are populated immediately on the UI, but while selecting same amount of files from a CD it takes time to render on UI. Here is an example -
For a CD with 20 DICOMs + 2 Non DICOMs studies, and about 2241 images, it takes about 5-6 min to populate the list the first time
on UI. If I try to select same CD folder, the list will populate in
roughly 60 sec if it’s been populated once before during the same
session.
But if i use the same set of files from the local machine then it roughly takes 6 -7 sec to populate on UI.
Here is my code which is executed for each and every DICOM file -
var fileReader = new FileReader();
fileReader.onload = function(evt){
console.log("Completed Reading");
var arrayBuffer = fileReader.result;
var byteArray = new Uint8Array(arrayBuffer);
_parseDicom(byteArray);
try {
if (fileReader.readyState !== 2) {
fileReader.abort();
}
}
catch (err) {
console.log('error occured: '+err);
}
}
var blob = f.slice(0, 50000);
console.log("Starting to Read");
fileReader.readAsArrayBuffer(blob);
After analyzing the issue i came up with,
The basic thing which i guess is, OS takes time to mount the CD to its memory as it is an external drive. It takes less time if we access the second time, because the CD content is already mounted.
The time between "Starting to Read" and "Completed Reading" is relatively more while reading files from CD than from local machine.
I also tried looking for DICOMDIR file, which is an index of all study files contained on the disc, and is included for exactly this reason: to avoid lengthy scans of the disc. But I didn't find any standard or way to parse the DICOMDIR file in JavaScript
Is there any way to reduce the amount of time it takes to read files from CD ??
UPDATE -
I am able to get DICOMDIR file structure now into JavaScript using dicomParser -
https://github.com/chafey/dicomParser
var fr = new FileReader();
fr.onload = function(evt){
var byteArray = new Uint8Array(fr.result);
try {
var dataSet = dicomParser.parseDicom(byteArray);
_searchDicom(dataSet, f);
} catch (err) {
if (typeof err.dataSet != 'undefined') {
_searchDicom(err.dataSet, f);
}
}
}
var blob = f.slice(0, 1000000);
fr.readAsArrayBuffer(blob);
function _searchDicom(dataset,f) {
var data = dataset.elements.x00041220.items;
if(data) {
data.forEach(function (e) {
if (e.dataSet.string('x00041430') === 'PATIENT') {
console.log("Patient Name - "+e.dataSet.string('x00100010'));
}
else if (e.dataSet.string('x00041430') === 'STUDY') {}
else if (e.dataSet.string('x00041430') === 'SERIES') {}
else if (e.dataSet.string('x00041430') === 'IMAGE') {}
});
}
}
The structure of object is similar to what is displayed on -
https://rawgit.com/chafey/dicomParser/master/examples/dumpWithDataDictionary/index.html
when we upload any DICOMDIR file.
The problem here is I am not able to collect all patients,studies,series or images at once. The only solution I found is to iterate and check whether it's a patient,study,series or image object
Is there any method/standard to retrieve in a better way ??

How to get full path to user's type="file" using jQuery Fileupload

Im trying to build file preview feature before file being uploaded to the server.
Im stucked with the problem of reading file from the user's computer, not a server
Im usuing jquery Fileupload which fires the file processing on the change function
return $('#new_order_file').fileupload({
change: function(e, data) {
return $.each(data.files, function(index, file) {
read_file(file);
console.log(file)
});
},
});
Log gives File - name, size, type, proto and LastmodifiedDate
Than, this part is tricky for me, I know that I have to use FileReader() but Im not sure how to
function read_file(f){
var reader = new FileReader();
reader.onload = function(e) {
$('#CO-show-model').css('display', 'block');
loaded(f.name, e.target.result);
};
reader.readAsArrayBuffer(f);
}
Next, loaded function is for displaying the file model using three.js
function loaded(file) {
var uploaded_file = file // gives 404
// var uploaded_file = './stl/test-file.stl' // ASCII test
}
The fucntion itself is good, I tested it with an hardcoded path to some test file. But the problem is that it was on a server. Now, script gives an 404 for http://example.com/filename.stl which is true, because I pass file.name there. If I just pass file, it gives 404 for http://example.com/[object%20File]
Edit 1:
I tried suggest ReadAsDataUrl, but seems to be that it's not supported by three.js. Also, the FileReader() seems to be ok
Your file is an object so you have to get the right attribute from it.
Take a look here Filereader Mozilla documentation
It's a security feature. Browsers will not give you the actual local url to the file you are uploading.
See How to get full path of selected file on change of <input type=‘file’> using javascript, jquery-ajax?
Edit: Maybe what you are looking for is this question instead? What is the preferred method for loading STL files in Three.js

Parse a local file to an array using JavaScript

I am trying to read a local file and have each line as an index in an array using JavaScript. I have been searching for the past 20 minutes and either I'm stupid or there really isn't an answer that pertains to my problem (...but it's probably the former :P). I am really new to JavaScript so if you have an answer could you please comment the code just to I know what's going on?
Also, from the searching I've done on the internet some people said JavaScript can't read local file for security reasons so if that is correct is there another language I can use? I'm a bit familiar with PHP if that is an option, which I doubt it is.
EDIT
As per thg435's question, I'll explain what I am trying to accomplish.
My project is to analyze a BUNCH of water quality data that has been collected by the Ontario gov't (which I've done) and display it in some way. I have chosen to display it on a webpage using the Google Maps API. I currently have a file of chemicals that were found. Each line is a different chemical. I would like to read the file in an array then create an option menu displaying the chemicals in the array.
Also, the local file I would like to read will the be the same name and location all the time. I have seen people have boxes where the user clicks and chooses their file or to drag and drop but that's not what I'm looking for.
I don't think I explained this properly. I have a file in the same directory as my HTML and JavaScript files that contains words. Example:
Line 1: "Iron"
Line 2: "Aluminum"
Line 3: "Steel"
etc...
I would like to read the file and parse each line into a different index in an array. I don't want the user to be able to choose which file to read using the <input ... /> thing.
You're going to want to take a look at the FileReader API. This should allow you to read the text of a local file via readAsText(). This won't work in every browser but should work in all modern browser. You can see which browsers support it here.
Example:
<input id="file" type="file" />
var filesInput = document.getElementById("file");
filesInput.addEventListener("change", function (event) {
var files = event.target.files;
var file = files[0];
var reader = new FileReader();
reader.addEventListener("load", function (event) {
var textFile = event.target;
alert(textFile.result);
});
reader.readAsText(file);
});
It's not possible to invoke the FileReader API without user interaction. Consequently, your user would have to select whatever file to load in order for it to be read in pure JS. Since I'm assuming this will be up on a server, why not just put the list of chemicals also up on the server and GET the JSON encoded array of the results. Then you can decode them with Javascript.
You can access local files in 2 ways that I know of. The first way is making the user drag-and-drop the files onto the page, and using an <input type="file"> tag.
For the former, you would need to do the following:
addEventListener('dragover', function(e){e.preventDefault();});
addEventListener('drop', function(e) {
eventHandler.call(e.dataTransfer||e.clipboardData);
e.preventDefault();
});
For the latter, you'd need to add an event listener for the change event on the input:
document.getElementById('upload').addEventListener('change', eventHandler)
And for both, you'd need to have this as a basic callback function:
function eventHandler() {
var file = this.files[0]; //get the files
var reader = new FileReader(); //initiate reader
reader.onloadend = callbackFn; //set event handler
reader.readAsText(file); //initiate reading of files
if (this.id) { //only run if this is the input
var id = this.id;
this.outerHTML = this.outerHTML; //this resets the input
document.getElementById(id).addEventListener('change', eventHandler); //reattach event handler
}
function callbackFn(e) {
document.getElementById('output').value = e.target.result; //output it to a textarea
}
}
Here is a demo where the text contents (that what you see when opening it in notepad) of any file you drop in it, or any file you select from the input, is put in the textarea.
For more information, see https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications.

Categories

Resources