Data URI leak in Safari (was: Memory Leak with HTML5 canvas) - javascript

I have created a webpage that receives base64 encoded bitmaps over a Websocket and then draws them to a canvas. It works perfectly. Except, the browser's (whether Firefox, Chrome, or Safari) memory usage increases with each image and never goes down. So, there must be a memory leak in my code or some other bug. If I comment out the call to context.drawImage, the memory leak does not occur (but then of course the image is never drawn). Below are snippets from my webpage. Any help is appreciated. Thanks!
// global variables
var canvas;
var context;
...
ws.onmessage = function(evt)
{
var received_msg = evt.data;
var display_image = new Image();
display_image.onload = function ()
{
context.drawImage(this, 0, 0);
}
display_image.src = 'data:image/bmp;base64,'+received_msg;
}
...
canvas=document.getElementById('ImageCanvas');
context=canvas.getContext('2d');
...
<canvas id="ImageCanvas" width="430" height="330"></canvas>
UPDATE 12/19/2011
I can work around this problem by dynamically creating/destroying the canvas every 100 images or so with createElement/appendChild and removeChild. After that, I have no more memory problems with Firefox and Chrome.
However, Safari still has a memory usage problem, but I think it is a different problem, unrelated to Canvas. There seems to be an issue with repeatedly changing the "src" of the image in Safari, as if it will never free this memory.
display_image.src = 'data:image/bmp;base64,'+received_msg;
This is the same problem described on the following site: http://waldheinz.de/2010/06/webkit-leaks-data-uris/
UPDATE 12/21/2011
I was hoping to get around this Safari problem by converting my received base64 string to a blob (with a "dataURItoBlob" function that I found on this site) and back to a URL with window.URL.createObjectURL, setting my image src to this URL, and then later freeing the memory by calling window.URL.revokeObjectURL. I got this all working, and Chrome and Firefox display the images correctly. Unfortunately, Safari does not appear to have support for BlobBuilder, so it is not a solution I can use. This is strange, since many places including the O'Reilly "Programming HTML5 Applications" book state that BlobBuilder is supported in Safari/WebKit Nightly Builds. I downloaded the latest Windows nightly build from http://nightly.webkit.org/ and ran WebKit.exe but BlobBuilder and WebKitBlobBuilder are still undefined.
UPDATE 01/03/2012
Ok, I finally fixed this by decoding the base64-encoded data URI string with atob() and then creating a pixel data array and writing it to the canvas with putImageData (see http://beej.us/blog/2010/02/html5s-canvas-part-ii-pixel-manipulation/). Doing it this way (as opposed to constantly modifying an image's "src" and calling drawImage in the onload function), I no longer see a memory leak in Safari or any browser.

Without actual working code we can only speculate as to why.
If you're sending the same image over and over you're making a new image every time. This is bad. You'd want to do something like this:
var images = {}; // a map of all the images
ws.onmessage = function(evt)
{
var received_msg = evt.data;
var display_image;
var src = 'data:image/bmp;base64,'+received_msg;
// We've got two distinct scenarios here for images coming over the line:
if (images[src] !== undefined) {
// Image has come over before and therefore already been created,
// so don't make a new one!
display_image = images[src];
display_image.onload = function () {
context.drawImage(this, 0, 0);
}
} else {
// Never before seen image, make a new Image()
display_image = new Image();
display_image.onload = function () {
context.drawImage(this, 0, 0);
}
display_image.src = src;
images[src] = display_image; // save it for reuse
}
}
There are more efficient ways to write that (I'm duplicating onload code for instance, and I am not checking to see if an image is already complete). I'll leave those parts up to you though, you get the idea.

you're probably drawing the image a lot more times than you are expecting to. try adding a counter and output the number to an alert or to a div in the page to see how many times the image is being drawn.

That's very interesting. This is worth reporting as a bug to the various browser vendors (my feeling is that it shouldn't happen). You might responses along the lines of "Don't do that, instead do such and such" but at least then you'll know the right answer and have an interesting thing to write up for a blog post (more people will definitely run into this issue).
One thing to try is unsetting the image src (and onload handler) right after the call to drawImage. It might not free up all the memory but it might get most of it back.
If that doesn't work, you could always create a pool of image objects and re-use them once they have drawn to the canvas. That's a hassle because you'll have to track the state of those objects and also set your pool to an appropriate size (or make it grow/shrink based on traffic).
Please report back your results. I'm very interested because I use a similar technique for one of the tightPNG encoding in noVNC (and I'm sure others will be interested too).

I don't believe this is a bug. The problem seems to be that the images are stacked on top of each other. So to clear up the memory, you need to use clearRect() to clear your canvas before drawing the new image in it.
ctx.clearRect(0, 0, canvas.width, canvas.height);
How to clear your canvas matters

Related

Adding img.crossOrigin = "*" interferes with img.complete

I have a function that reads map tile images. I want to keep track of whether or not a certain image has already been cached. I'm using this function from this thread:
function is_cached(src) {
var image = new Image();
image.src = src;
return image.complete;
}
This was working great. But then I needed to do some image processing. In order to copy the image data to a canvas and process it pixel by pixel, I need to use CanvasRenderingContext2D.drawImage(image, 0, 0). But it bugs me with a cross-origin error. So I can add a image.crossOrigin = "*", which solves that problem, and I can write to a canvas and do the image processing I need. That bit looks like this:
imageOutput.crossOrigin = "*"
var demCtx;
imageOutput.onload = function(){
var c = document.createElement('canvas')
c.width = c.height = 256
demCtx = c.getContext('2d')
demCtx.drawImage(imageOutput, 0, 0)
var imageData = demCtx.getImageData(0, 0, 256, 256)
}
The issue that arises is that every time I run the larger function which contains these two bits of code, the is_cached function returns false every time, except the first time. But I know that even though is_cached is returning false, the images are indeed cached, as they are loading with 0 lag (as opposed to when a novel image is called and it takes a moment to grab it from the server).
Why might .crossOrigin = "*" be interfering with the .complete status of an image?
This is happening within an ObservableHQ notebook. Might that have something to do with it? ObservaleHQ gets weird sometimes.
ObservableHQ Notebook with the problem
You can find this code in the getTileUrl cell at the bottom. This notebook is not yet finished. You can see the cached status at the Tile Previously Cached line after you click around the map of submit changes to the inputs.
Thanks for reading.
Maybe fetch api can enforce cache using the param {cache:"force-cache"}, however images should be cached as expected. You can fetch the image and pass its blob as an image source.
replace your imageOutput.src with
imageOutput.src = URL.createObjectURL(await fetch(imageUrl, {cache:"force-cache"}).then(r => r.blob()));
make your getTileURL function async as we have to await fetch and blob to be ready to be passed as image source
async function getTileURL(latArg, lngArg, zoomArg) {
Use devtools to inspect network and see tile images coming from disk cache
edit:
just try your original code and inspect network via devtools. The tiles images are cache as expected. So no need to hack into fetch blob src.

I need a way to copy parts of a canvas somewhere else on the same canvas

I know this has already been asked, but many of the answers I found were inconclusive and the questions weren't exactly depictive of my specific problem.
All I need to do is a "select-copy-paste" on a canvas, something that would take very little time on Microsoft Paint, but is turning out to be extremely difficult in JS. The point is, I actually need to perform a couple thousand of those on images that are up to 12 thousand pixels in width/height, so doing it on Paint is not an option.
Here's the most basic example I could write, just to focus on the issue I'm having:
Here's the HTML body:
<canvas id="canvas" width="200" height="100"></canvas>
<img id="house" class="hidden" type="img/png" src="house.png"></img>
Here's the JS:
window.onload = function () {
var ctx = document.getElementById("canvas").getContext("2d");
var house = document.getElementById("house");
ctx.drawImage(house, 10, 10);
var data = ctx.getImageData(10, 10, 80, 80); // ERROR
ctx.putImageData(data, 110, 10);
}
And (although I think it's irrelevant) here's the CSS:
.hidden {
display: none;
}
The line I commented with "ERROR" throws the following:
Failed to execute 'getImageData' on 'CanvasRenderingContext2D': The canvas has been tainted by cross-origin data.
A few important things to note are that all of these files are in my local directories (in a folder on my desktop, to be exact), so I'm not involved with any website or host or anything internet-dependent. Also, I'm not running any servers nor do I have any, therefore I cannot change any authorizations or CORS settings; after all, why would I have to? All I'm trying to do is copy and paste pixels.
I also tried to create 12k by 12k matrix of strings, where each string represents the pixel value of the image. This way I wouldn't have to use getImageData on the canvas, but I could work on the strings and change their values, except this doesn't work because I can't use getImageData on an image.
If anybody could enlighten me with a solution to this problem I'd be extremely thankful.
EDIT
I feel silly for not having tried it earlier, but I found an extremely simple solution: use Explorer instead of Chrome. Explorer doesn't throw any errors when calling getImageData.

efficient way of streaming a html5 canvas content?

I'm trying to stream the content of a html5 canvas on a live basis using websockets and nodejs.
The content of the html5 canvas is just a video.
What I have done so far is:
I convert the canvas to blob and then get the blob URL and send that URL to my nodejs server using websockets.
I get the blob URL like this:
canvas.toBlob(function(blob) {
url = window.URL.createObjectURL(blob);
});
The blob URLs are generated per video frame (20 frames per second to be exact) and they look something like this:
blob:null/e3e8888e-98da-41aa-a3c0-8fe3f44frt53
I then get that blob URL back from the the server via websockets so I can use it to DRAW it onto another canvas for other users to see.
I did search how to draw onto canvas from blob URL but I couldn't find anything close to what i am trying to do.
So the questions I have are:
Is this the correct way of doing what i am trying to achieve? any
pros and cons would be appreciated.
Is there any other more efficient way of doing this or I'm on a right
path?
Thanks in advance.
EDIT:
I should have mentioned that I cannot use WebRTC in this project and I have to do it all with what I have.
to make it easier for everyone where I am at right now, this how I tried to display the blob URLs that I mentioned above in my canvas using websockets:
websocket.onopen = function(event) {
websocket.onmessage = function(evt) {
var val = evt.data;
console.log("new data "+val);
var canvas2 = document.querySelector('.canvMotion2');
var ctx2 = canvas2.getContext('2d');
var img = new Image();
img.onload = function(){
ctx2.drawImage(img, 0, 0)
}
img.src = val;
};
// Listen for socket closes
websocket.onclose = function(event) {
};
websocket.onerror = function(evt) {
};
};
The issue is that when I run that code in FireFox, the canvas is always empty/blank but I see the blob URLs in my console so that makes me think that what I am doing is wrong.
and in Google chrome, i get Not allowed to load local resource: blob: error.
SECOND EDIT:
This is where I am at the moment.
First option
I tried to send the whole blob(s) via websockets and I managed that successfully. However, I couldn't read it back on the client side for some strange reason!
when I looked on my nodejs server's console, I could see something like this for each blob that I was sending to the server:
<buffer fd67676 hdsjuhsd8 sjhjs....
Second option:
So the option above failed and I thought of something else which is turning each canvas frame to base64(jpeg) and send that to the server via websockets and then display/draw those base64 image onto the canvas on the client side.
I'm sending 24 frames per second to the server.
This worked. BUT the client side canvas where these base64 images are being displayed again is very slow and and its like its drawing 1 frame per second. and this is the issue that i have at the moment.
Third option:
I also tried to use a video without a canvas. So, using WebRTC, I got the video Stream as a single Blob. but I'm not entiely sure how to use that and send it to the client side so people can see it.
IMPORTANT: this system that I am working on is not a peer to peer connection. its just a one way streaming that I am trying to achieve.
The most natural way to stream a canvas content: WebRTC
OP made it clear that they can't use it, and it may be the case for many because,
Browser support is still not that great.
It implies to have a MediaServer running (at least ICE+STUN/TURN, and maybe a gateway if you want to stream to more than one peer).
But still, if you can afford it, all you need then to get a MediaStream from your canvas element is
const canvas_stream = canvas.captureStream(minimumFrameRate);
and then you'd just have to add it to your RTCPeerConnection:
pc.addTrack(stream.getVideoTracks()[0], stream);
Example below will just display the MediaStream to a <video> element.
let x = 0;
const ctx = canvas.getContext('2d');
draw();
startStream();
function startStream() {
// grab our MediaStream
const stream = canvas.captureStream(30);
// feed the <video>
vid.srcObject = stream;
vid.play();
}
function draw() {
x = (x + 1) % (canvas.width + 50);
ctx.fillStyle = 'white';
ctx.fillRect(0,0,canvas.width,canvas.height);
ctx.fillStyle = 'red';
ctx.beginPath();
ctx.arc(x - 25, 75, 25, 0, Math.PI*2);
ctx.fill();
requestAnimationFrame(draw);
}
video,canvas{border:1px solid}
<canvas id="canvas">75</canvas>
<video id="vid" controls></video>
The most efficient way to stream a live canvas drawing: stream the drawing operations.
Once again, OP said they didn't want this solution because their set-up doesn't match, but might be helpful for many readers:
Instead of sending the result of the canvas, simply send the drawing commands to your peers, which will then execute these on their side.
But this approach has its own caveats:
You will have to write your own encoder/decoder to pass the commands.
Some cases might get hard to share (e.g external media would have to be shared and preloaded the same way on all peers, and the worse case being drawing an other canvas, where you'd have to also have shared its own drawing process).
You may want to avoid intensive image processing (e.g ImageData manipulation) to be done on all peers.
So a third, definitely less performant way to do it, is like OP tried to do:
Upload frames at regular interval.
I won't go in details in here, but keep in mind that you are sending standalone image files, and hence a whole lot more data than if it had been encoded as a video.
Instead, I'll focus on why OP's code didn't work?
First it may be good to have a small reminder of what is a Blob (the thing that is provided in the callback of canvas.toBlob(callback)).
A Blob is a special JavaScript object, which represents binary data, generally stored either in browser's memory, or at least on user's disk, accessible by the browser.
This binary data is not directly available to JavaScript though. To be able to access it, we need to either read this Blob (through a FileReader or a Response object), or to create a BlobURI, which is a fake URI, allowing most APIs to point at the binary data just like if it was stored on a real server, even though the binary data is still just in the browser's allocated memory.
But this BlobURI being just a fake, temporary, and domain restricted path to the browser's memory, can not be shared to any other cross-domain document, application, and even less computer.
All this to say that what should have been sent to the WebSocket, are the Blobs directly, and not the BlobURIs.
You'd create the BlobURIs only on the consumers' side, so that they can load these images from the Blob's binary data that is now in their allocated memory.
Emitter side:
canvas.toBlob(blob=>ws.send(blob));
Consumer side:
ws.onmessage = function(evt) {
const blob = evt.data;
const url = URL.createObjectURL(blob);
img.src = url;
};
But actually, to even better answer OP's problem, a final solution, which is probably the best in this scenario,
Share the video stream that is painted on the canvas.

Which way to create a canvas pattern from a dataToURL-image string as directly as possible?

I'm using an image that I much previously had made by
var patternImageAsDataURL= canvasObject.toDataURL('image/png');
In a later stage I want to make a canvas pattern object. The following code doesn't work - I assume the image is simply not loaded when going to the last line, where it is needed in the createPattern function.
var img = document.createElement('img');
img.src = patternImageAsDataURL;
// canvasctx was created somewhere else in the program
pattern = canvasctx.createPattern(img,'repeat');
I get the error: NS_ERROR_NOT_AVAILABLE: on the last line. (And when using console.log on width and heigth of img between the two last lines, I see when it's not working the dimensions are 0.)
When later on the same operation is done with the same dataURL, it does work. Though the image (img) should always be created anew. (Only reason I can see it's because of some internal optimization in Firefox. But that's offtopic here, unless someone does know the answer.) The width and height when printing them out to the console are correct then.
While I will quite soon program some pattern handling service, that should solve this, my question is in general and for speed concerns and for simplicity. (If I use some code with like 20 to 50 objects with patterns, I would prefer a lean solution over a memory or time saving function.)
Could I somehow use the dataURL more directly (and faster) for the
createPattern function?
And:
Could I force the program to wait after the img.src = patternImageAsDataURL; command until the image is loaded, and then to go on processing the code? (Like in the synchronous mode of the XMLrequests.)
(Using the onload event of the image isn't feasible in the current program flow.)
This is running on Firefox 32, Win 7.
A faster, more direct way to create a pattern
You can use a second canvas element as the source for a pattern.
This allows you to completely skip the interim step of creating an ImageURL and Image from your source canvas so your pattern creation will be faster.
var canvas=document.getElementById("canvas");
var ctx=canvas.getContext("2d");
var cw=canvas.width;
var ch=canvas.height;
// Make a temporary canvas to be the template for a pattern
var pc=document.createElement('canvas');
var px=pc.getContext('2d');
pc.width=4;
pc.height=4;
px.fillStyle='palegreen';
px.fillRect(0,0,2,2);
px.fillRect(2,2,2,2);
// Use the temporary canvas as the image source for "createPattern"
var pattern=ctx.createPattern(pc,'repeat');
ctx.fillStyle=pattern;
ctx.fillRect(50,50,100,75);
ctx.strokeRect(50,50,100,75);
body{ background-color: ivory; }
#canvas{border:1px solid red;}
<h4>Using a temporary canvas as source for a Pattern.</h4>
<canvas id="canvas" width=300 height=300></canvas>
Option 1 - Canvas as image source
The obvious is of course to use the canvas itself as image source for the pattern.
createPattern() can take image, canvas, context (although not all browsers allow this) or even video as source.
CanvasPattern createPattern(CanvasImageSource image,
[TreatNullAs=EmptyString] DOMString repetition);
where CanvasImageSource is defined as:
typedef (HTMLImageElement or
HTMLVideoElement or
HTMLCanvasElement or
CanvasRenderingContext2D or
ImageBitmap) CanvasImageSource;
This is also the only way that will allow you to not use onload at some point later (provided the pattern is generated and not drawn in from an image/video source).
You cannot deal with asynchronous behavior without using callbacks (or promises), and expect the program to work properly. Period.
Option 2 - Data-URIs
If you for some reason cannot use the original canvas as source, you have to deal with the image asynchronously. Add a onload handler for it and continue from inside it:
var img = document.createElement('img');
img.onload = function() {
pattern = canvasctx.createPattern(this, 'repeat');
// continue from here..
};
img.src = patternImageAsDataURL;
Note that the process of this is relative slow due to the additional encoding/decoding process on top of the image handling itself. You can find more details about this in this answer.
Option 3 - Blob and object-URL
A Blob lets you store the data in binary form. This is preferred over storing the binary data as encoded string as with data-URIs. This will be faster to embed as well as extract compared to data-URIs.
You can use URL form with the Blob and use that as image source.
First create the Blob directly from canvas:
var patternImageAsBlob = canvas.toBlob(...); //IE: msToBlob()
This is also an asynchronous call so you need to take that into account.
For example:
var patternAsBlob;
canvas.toBlob(function(blob) {
patternAsBlob = blob;
// continue from here
}
Then when you need it as an image, generate an Object-URL for it like this:
var img = new Image(),
url = URL.createObjectURL(patternAsBlob);
img.onload = function() {
URL.revokeObjectURL(url); // clean up by removing the url object
pattern = canvasctx.createPattern(this, 'repeat');
// continue from here..
};
img.src = url;
Tips
If you have several images to load and set, it would be better to make an image loader to load in all resources to an array, when done create the patterns.
This will simplify the asynchronous chain-calling (optionally use promises, but this is not yet supported in IE without a polyfill).
You may need a polyfill for toBlob in older browser. One can be found here.
You may need to "unprefix" the createObjectURL(), here is one way:
var domURL = self.URL || self.webkitURL || self;
var url = domURL.createObjectURL( ... );

Access Kinect RGB image data from ZigJS

I have ZigJS running in the browser and everything is working well, but I want to record the Kinect webcam images in order to play them back as a recorded video. I've looked through the documentation at http://zigfu.com/apidoc/ but cannot find anything related to the RGB information.
However, this SO answer leads me to believe this is possible:
We also support serialization of the depth and RGB image into canvas objects in the browser
Is it possible to capture the RGB image data from ZigJS and if so how?
Assuming you have plugin version 0.9.7, something along the lines of:
var plugin = document.getElementById("ZigPlugin"); // the <object> element
plugin.requestStreams(false, true, false); // tell the plugin to update the RGB image
plugin.addEventListener("NewFrame", function() { // triggered every new kinect frame
var rgbImage = Base64.decode(plugin.imageMap);
// plugin.imageMapResolution stores the resolution, right now hard-coded
// to QQVGA (160x120 for CPU-usage reasons)
// do stuff with the image
}
Also, I recommend you take the base64 decoder I wrote, from, say, http://motionos.com/webgl because it's an order of magnitude faster than the random javascript decoders I found via Google.
If you have version 0.9.8 of the plugin, there was an API change, so you should call:
plugin.requestStreams({updateImage:true});

Categories

Resources