Me and my team facing one issue of memory in our application. The issue is whenever we trigger jsPDF instance its works but after that it's holding lot of memory and we facing performance issue because its never release that memory after completion of task. For reference i prepare one example and you can check with that.
https://stackblitz.com/edit/web-platform-vkewvw?file=index.html
So in this example you see my memory footprint its some around 36 MB you can see on first screen shotenter image description here
and after run the code its goes around 56MB and its not releasing the memory you see in the next screen shot.
enter image description here
Can any one help on that how we overcome that problem we tried with iframe itself by not working properly.
Your help is appreciable for us.
As raised in comment, altering the timing of revoking blob urls may improve memory handling in an application using JSPDF, but does not come with a guarantee to do so ...
The Blob Store
User agents maintain a blob ULR store that keeps a reference to Blob objects keyed by the urls for them returned from URL.createObjectURL(blob). Holding a reference in the store stops the blob object from being garbage collected from memory even if JavaScript code has not kept a reference to the blob object itself.
The blob object can be removed from the URL store by calling URL.revokeObjectURL(blobURL), after which the blob is eligible for garbage collection from memory provided no reference to it is reachable in JS.
Now jsPDF sets its global object to window in browsers' main JavaScript thread in globalObject.js, and imports the global object as _global in FileSaver.js.
Lines 85 and 86 of FileSaver.js define the module's saveAs export as
var saveAs =
_global.saveAs ||
... code for saving file
which implies you should be able to shim the saveAs function for experimental purposes by declaring a shim (using function saveAs or window.saveAs =) in file level script before including jsPDF.js in the HTML document.
Initially you could use the original saveAs code with console logs to demonstrate the shimming process works and that jsPDF still works. Things I would want to look at include
is jsPDF synchronous - meaning does it only return to the caller after clicking the link in saveAs to save the PDF file produced?
If it's potentially asynchronous, how to arrange a callback or promise resolution from the shim after clicking the link to prevent sequentially produced PDF files being processed in parallel.
Does reducing the time before revoking the blob's URL, currently set to 40 seconds for most browsers in line 188 of FileSave.js (linked above), materially affect performance of the application?
How well does the application run in Safari and ChromeIOS, which receive exceptional support in FileSave.js?
Synchronous Revocation of Blob URLs
There is some possibility that Blob URLs can be revoked synchronously after use. The following code (which doesn't work as a code snippet) creates a blob and downloads it:
function blobURL(string) {
const blob = new Blob(Array.from(string));
return URL.createObjectURL(blob); // see NOTE 1
}
const url = blobURL("hello folks and world");
console.log("2 seconds...");
setTimeout( saveBlob, 2000, url); // see NOTE 2
function saveBlob(url) {
const link = document.createElement('a');
link.setAttribute("download", "hello.txt");
link.href= url;
document.body.appendChild(link);
link.click();
URL.revokeObjectURL(url); // SEE NOTE 3
console.log("Call to link.click() has returned");
}
Note
The script does not keep a reference to the blob created
Memory garbage collection could (in theory) run during a timeout period.
The blob's url is revoked synchronously, after link.click(), before returning to the event loop.
Calling URL.revokeObjectURL() immediately after programatially clicking the link to download the blob did not affect the success of downloading in Firefox or Edge/webkit when tested. This implies that these browsers synchronously obtains a reference to the Blob instance (using Blob Store lookup) before returning from link.click().
This is consistent with the behavior of events programmatically dispatched on an element being processed synchronously (which I looked at recently in answer to "Do events bubble in microtasks?"). How safe it is to make use of this in production, however, is not something I am personally in a position to guarantee across all browsers.
Related
I know crypto.subtle.digest could be used to generate a digest of a given ArrayBuffer.
However, when the file is large, e.g. 5GB, I always get this error
Uncaught (in promise) DOMException: The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired.
click https://jsfiddle.net/kq4s2utf/ to see the full version.
How do I solve this?
I believe the right answer would be to stream the file content instead of reading whole file in memory at once.
Blob allows to read a file as a stream: https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream
Now the problem is that Web Cryptography API you're using doesn't support streams or incremental hashing. There is a long (and quite old) discussion about that with no clear outcome: https://github.com/w3c/webcrypto/issues/73 .
I would suggest using some 3rd party library that supports incremental hashing. E.g. https://github.com/Caligatio/jsSHA
The resulting code could look like
async function calcDigest() {
const reader = finput.files[0].stream().getReader()
const shaObj = new jsSHA("SHA-256", "ARRAYBUFFER")
while (true) {
const {done, value} = await reader.read()
if (done) break
shaObj.update(value)
}
console.log(shaObj.getHash("HEX"))
}
One possible solution could be compressing the ArrayBuffer through LZMA (LZMA-JS) or something and creating a digest of that instead of the complete data buffer.
I haven't had the time to test this out because I didn't have any files larger than 4GB, but it could work.
The Problem
When creating audio buffers using the Web Audio API, there are buffers created by the decodeAudioData method, which reside in memory and are apparently not accessible through JavaScript. They seem to hang around for the entire life of a browser tab, and never get garbage collected.
Possible Reason For the Problem
I know that these buffers are separated from the main thread and set on another thread for asynchronous decoding. I also know that the API spec says that decodeAudioData should not be allowed to decode the same input buffer twice, which I assume is why a copy of the decoded buffer and/or the encoded input buffer are kept around. However, on memory limited devices like Chromecast, this causes huge amounts of memory to accumulate and Chromecast crashes.
Reproducibility
In my example code, I fetch an mp3 using Ajax and then pass the arraybuffer into the decodeAudioData function. Normally within that function there is a onsuccess callback which can take the decoded AudioBuffer as a parameter. But here in my code, I don't even pass that in. Therefore I also don't do anything with the decoded buffer after decoding it. It is not referenced anywhere within my code. It is entirely left in the native code. However, every call to this function increases the memory allocation and it is never released. For example, in Firefox about:memory shows the audiobuffers there for the life of the Tab. Non-reference should be sufficient for the garbage collector to get rid of these buffers.
My main question then is, is there any reference to these decoded audio buffers, say within the audiocontext object, or somewhere else that I can try to remove them from memory? Or is there any other way that I can cause these stored and unreachable buffers to disappear?
My question differs from all the others currently on SO regarding decodeAudioData because I show that the memory leak happens even without the user storing any reference or even using the returned decoded audio buffer.
Code To Reproduce
function loadBuffer() {
// create an audio context
var context = new (window.AudioContext || window.webkitAudioContext)();
// fetch mp3 as an arraybuffer async
var url = "beep.mp3";
var request = new XMLHttpRequest();
request.open("GET", url, true);
request.responseType = "arraybuffer";
request.onload = function () {
context.decodeAudioData(
request.response,
function () {// not even passing buffer into this function as a parameter
console.log("just got tiny beep file and did nothing with it, and yet there are audio buffers in memory that never seem to be released or gc'd");
},
function (error) {
console.error('decodeAudioData error', error);
}
);
};
request.onerror = function () {
console.log('error loading mp3');
}
request.send();
}
To anticipate some possible responses.
I must use Web Audio API because I am playing four part harmony from four audio files on Chromecast and the html audio element does not support multiple simultaneous playback on Chromecast.
Probably any JS library you may reference [e.g. Howler.js, Tone.js, Amplitude.js etc.] is built upon the Web Audio API, and so they will all share this memory leak problem.
I know that the WAA is implementation dependent on a per browser basis. My primary concern at the moment is Chromecast, but the problem exists for every browser I've tried.
Therefore, I think it is a spec related issue where the spec requires the non-dupe encoding rule, and so implementers keep copies of the buffer around on a browser level thread so they can check them against new xhr inputs. If the spec writer's happen to read my question, is there not a way that the user can have the option for this behavior, and opt out of it if they wish in order to prevent the internal buffer storage on mobile and thin memory platforms?
I have not been able to find any reference to these buffers in any JS object.
I know that I can audio_context.close() and then hope for garbage collection of all the resources held by the audio_context, and then hope that I can reinstantiate the audio_context with a new one, but that has not empirically been timely enough for my application. Chromecast crashes before GC takes out the trash.
Pragmatic Workaround
I have found a method to solve the problem of the Web Audio API audiobuffers handing around indefinitely and crashing Chromecast and other mobile platforms. [[ I have not tested this on all browsers - your mileage may vary. ]]
LOADING STAGE
Load the document using Web Audio API inside an iFrame.
Load your audio buffers and do whatever you do to play them.
CLEARING STAGE
Call sourceNode.stop on all of the playing nodes you have reference to.
Call source.disconnect(); on all source nodes.
Call gainNode.disconnect(); on all gain nodes those source nodes are associated with (and whatever other kind of WAA nodes you might be using that have a disconnect method)
Set all referenced gainNodes and sourceNodes to null;
Null out any buffers you have referenced both decoded and your xhr fetched encoded audiobuffers;
KEY: Within the WAA page call audio_context.close(); then set audio_context=null; (this can be done from the parent of the iFrame using contentWindow).
Note: Some of these nulling steps may not be absolutely necessary, however this approach has worked for me.
RE-LOADING STAGE
Reload the iframe from the parent page. This will cause all of the audiobuffers to be garbage collected ON THE NEXT GC ROUND, including the ones in the hidden (non JS) areas of memory.
Your iframe will have to reinstantiate the web audio context and load its buffers and create nodes etc. just as you did when you first loaded it.
Notes: You must decide when you are going to use this clearing method (e.g. after so many buffers have been loaded and played). You can do it without an iframe, but you may have to reload the page once or twice to get garbage collection to fire. This is a pragmatic workaround for those who need to load lots of Web Audio API audio buffers on memory thin platforms like Chromecast or other mobile devices.
FROM PARENT
function hack_memory_management() {
var frame_player = document.getElementById("castFrame");
//sample is the object which holds an audio_context
frame_player.contentWindow.sample.clearBuffers();
setTimeout(function () {
frame_player.contentWindow.location.reload();
}, 1000);
}
INSIDE WAA IFRAME
CrossfadeSample.prototype.clearBuffers = function () {
console.log("CLEARING ALL BUFFERS -IT'S UP TO GC NOW'");
// I have four of each thing because I am doing four part harmony
// these are the decoded audiobuffers used to be passed to the source nodes
this.soprano = null;
this.alto = null;
this.tenor = null;
this.bass = null;
if (this.ctl1) {
//these are the control handles which hold a source node and gain node
var offName = 'stop';
this.ctl1.source[offName](0);
this.ctl2.source[offName](0);
this.ctl3.source[offName](0);
this.ctl4.source[offName](0);
// MAX GARGABE COLLECTION PARANOIA
//disconnect all source nodes
this.ctl1.source.disconnect();
this.ctl2.source.disconnect();
this.ctl3.source.disconnect();
this.ctl4.source.disconnect();
//disconnect all gain nodes
this.ctl1.gainNode.disconnect();
this.ctl2.gainNode.disconnect();
this.ctl3.gainNode.disconnect();
this.ctl4.gainNode.disconnect();
// null out all source and gain nodes
this.ctl1.source = null;
this.ctl2.source = null;
this.ctl3.source = null;
this.ctl4.source = null;
this.ctl1.gainNode = null;
this.ctl2.gainNode = null;
this.ctl3.gainNode = null;
this.ctl4.gainNode = null;
}
// null out the controls
this.ctl1 = null;
this.ctl2 = null;
this.ctl3 = null;
this.ctl4 = null;
// close the audio context
if (this.audio_context) {
this.audio_context.close();
}
// null the audio context
this.audio_context = null;
};
Update:
Sadly, even this does not reliably work and Chromecast can still crash given a few clear and loads of new mp3s. See "My present solution" elsewhere on this page.
Can you maybe use multiple audio-tags on Chromecast when you route each of them into the Web Audio graph (by using a MediaElementAudioSourceNode)?
My present solution
I could not find a final satisfactory solution for Chromecast using the Web Audio API and simultaneous playback of four mp3s - used for four part harmony. The 2nd Gen seems to simply not have enough resources to hold the audiobuffers and simultaneously decode four mp3 files using decodeAudioData without leaving too much garbage around and eventually crashing. I decided to go with surikov's webaudiofont which is built on top of the Web Audio API, and to use midi files. I never had a problem on desktop browsers or other devices with more resources, but I have to have it work on Chromecast. I have no problems at all now using webaudiofont.
I was facing the same problem. What eventually worked for me was to disconnected and delete all connected resources:
if (this.source) {
this.source.disconnect()
delete this.source
}
if (this.gain) {
this.gain.disconnect()
delete this.gain
}
await this.audioContext.close()
delete this.audioContext
delete this.audioBuffer
Just closing the audioContext is not enough. It seems that references will continue to exist preventing garbage collection.
A lot of answers I have seen seem to overcomplicate this. I have run into this same issue while rebuilding an audio system for an application I'm building, but then I realised it previously was not an issue, this was because everytime I played a new audio I closed the previous AudioContext and used the variable it was referenced in for a new AudioContext.
This means that the only two things one has to do to clear this overly memory usage is to use AudioContext.close(), and remove references to it, disconnecting nodes and such is not required.
I am trying to use FineUploader to upload a large amount of files. I also need to manipulate the files before uploading them - namely I need to anonymize some identifying information. In another answer, Ray Nicholus suggested to reject the original file in the onSubmit handler and then re-add the manipulated file. So my onSubmit handler looks like so:
onSubmit: function (id, name)
{
var file = this.getFile(id)
if (file.isAnonymized)
{
return;
}
var reader = new FileReader()
reader.onload = function()
{
var arrayBuffer = this.result
var byteArray = new Uint8Array(arrayBuffer)
// Manipulate the byteArray in some way...
var blob = new window.Blob([byteArray])
blob.isAnonymized = true
// add the anonymized file instead
uploader.addFiles({blob: blob, name: name})
}
reader.readAsArrayBuffer(file)
// cancel the original file
return false
},
This works fine for a small amount of files. In a concrete example, a customer tried to upload ~1.500 files of 3MB each in Firefox, and saw Firefox's memory usage spike through the roof before the tab eventually crashed. Other browsers (Chrome, Edge) exhibit similar behavior. Using the browser's developer tools doesn't seem to show any large memory allocations. There are no problems when simply uploading the files as-is, but that's not an option.
I cleaned up the example at https://github.com/sgarcialaguna-mms/file-upload-memory/ somewhat and am now confident that the error is due to the fineuploader library holding on to blobs longer than needed.
The example now loads one file into memory at a time, then passes the blob to the upload library. I also now use an actual server (the Django example from the fineuploader server-examples repository).
With Firefox, when I drag in ~1GB of files, Firefox's memory usage steadily rises during the upload and stays high even after the upload is completed. I can open about:memory, click "Minimize memory usage" to trigger a garbage collection, press on "Measure" and the file data shows up under "memory-file-data/large". Call uploader.reset(), trigger a garbage collection again and Firefox's memory usage drops sharply. Measuring again shows the "memory-file-data/large" objects are no longer present in memory. As per https://github.com/FineUploader/fine-uploader/issues/1540#issuecomment-194201646, calling this._handler.expunge(id) after every upload works as well.
Chrome behaves a bit differently, due to a long-standing bug it eventually starts throwing ERR_FILE_NOT_FOUND errors once more than 500 MB of blob data accumulates. The chrome://blob-internals page shows which blobs are being held as well as their refcount.
I don't know if there is an easy way to tell which variable / closure / whatever is holding on to these objects, but it would help immensely.
I am fumbling around with the free Chrome Dev Editor on my Chromebook. I am trying to use the fileSystem to read and write .txt files. It is all very wrapped up, not at all like in C. I can no more tell if I am even allowed to do something, let alone where the proper place is to find out how.
I think the files I can see using the Files thingy are in the sandbox that I am allowed to play in (meaning, folders that are accessible by the app?) The root is called Downloads. Sure enough, if I use all the dot calls and callback arguments for the read, as in the examples at developer.chrome.com/apps/filesystem, it works. But I have to have a prompt
every time for both reads and writes.
A little more Googling came up with this trick: (I think it was here in stackoverflow, in fact) a chrome.runtime call, getPackagedDirectoryEntry, that seems to give me a handle to the folder of my app. Great! That's all I need to not have to go through the prompting. For the readfile, anyway.
But then trying to apply the same trick to the writefile did not work. In fact, it did nothing discernible. No errors, no complaints. Nothing. Even though the write file with prompting works fine (so presumably I have the permissions and Blob construction right.) What to do?
Here is my code:
function test(){
// Samsung 303C Chromebook - Chrome Dev Editor - /Downloads/Daily/main.js
// prompted write
chrome.fileSystem.chooseEntry({type:'saveFile'},function(a){
a.createWriter(function(b){
b.write(new Blob(["Programming fun"],{type:'text/plain'}));
},function(e){trace.innerText = 'error is ' + e;});
});
// unprompted read
chrome.runtime.getPackageDirectoryEntry(function(a){
a.getFile('text.txt',{},function(b){
b.file(function(c){
var d = new FileReader();
d.onloadend = function(){trace.innerText = this.result;};
d.readAsText(c);
});
});
});
// unprompted write - why not?
chrome.runtime.getPackageDirectoryEntry(function(a){
a.getFile('new.txt',{create:true},function(b){
b.createWriter(function(c){
c.write(new Blob(["Miss Manners fan"],{type:'text/plain'}));
},function(e){trace.innerText = 'error is ' + e;});
});
});
}
To be fair, Filesystem API is a big mess of callbacks and it's not unreasonable to get drowned in it.
It's not currently documented, but chrome.runtime.getPackageDirectoryEntry returns a read-only DirectoryEntry, and there is no way to make it writable (it's specifically blacklisted).
You probably don't see an error, because it fails at the getFile stage, for which you don't have an error handler.
Unfortunately, for a Chrome App the only option to write out to a real filesystem is to prompt the user. However, you can retain the entry and ask only once.
If you don't need to write out to the real filesystem but need only internal storage, HTML Filesystem API can help you (yes, it's marked as abandoned, but Chrome maintains it since chrome.fileSystem is built on it).
Extensions additionally have access to chrome.downloads API that enables writing to (but not reading) the Downloads folder.
P.S. What you see in Files app is your "real" local filesystem in ChromeOS + mounted cloud filesystems (e.g. Google Drive)
You can use the basic web Filesystem API. First, add the "unlimitedStorage" permission. Then, copy the packaged files to the sandboxed filesystem, like this:
chrome.runtime.getPackageDirectoryEntry(function(package) {
package.getMetadata(function(metadata) {
webkitRequestFileSystem(PERSISTENT, metadata.size, function(filesystem) {
package.copyTo(filesystem.root)
})
})
})
I am building a file storage for HTML5, and I am using indexedDB as the storage, I ask the files from the server via xmlHttpRequest with the response type as arrayBuffer (for chrome) and blob (for other browsers).
Everything is fine even if the files-collection size is 500MB or more, (hey, it can even reach GB). But I noticed something strange when I add the file to the indexedDB, it will trigger error when the single file exceeds ~120MB, so it is not stored. But when the file is less than 120MB, it will store it.
Notice that it will only have this error when storing a single file > 120MB, for example, an .mp4 file of 200MB will trigger an error, but if I have 5 videos with each of them have a size of 100MB (so the total will be 500MB) it will be all fine.
I would like to know whether this is a limit-rule or some glitch and the two have the same error. I didn't find any documentation about it. I tested it in IE and Chrome.
EDIT:
Ok, I got this error apparently in the add or put function of indexedDB when storing the file:
inside the e.target.error.message:
The serialized value is too large (size=140989466 bytes, max=133169152 bytes)
At the time this question was asked, Chrome still didn't support saving Blobs to IndexedDB, it only came the next month.
For anyone facing the same issue nowadays, store Blobs or Files directly, not ArrayBuffers.
Contrarily to ArrayBuffers, saving a Blob to IDB doesn't require to serialize its data, the serialization steps of a Blob are just to make a snapshot of the js object and keep the link to the same underlying byte sequence, which itself is not cloned.
The only limit you should face would be the one of the db itself.
Code taken from Chrome's announcement:
var store = db.transaction(['entries'], 'readwrite').objectStore('entries');
// Store the object
var req = store.put(blob, 'blob');
req.onerror = function(e) {
console.log(e);
};
req.onsuccess = function(event) {
console.log('Successfully stored a blob as Blob.');
};
I think this is an issue with your browser's implementation of IndexedDB. I ran into this same error myself in Firefox, when I tried to store a 100 MB file into a IndexedDB record, but the identical code worked fine in Chrome. It seems different browsers have different implementation quirks and limits.
Personally, I suspect this is a bug in Firefox, since Firefox grants the requested size, but then prevents single-record usage of that entire size, whereas Chrome is more forgiving.