Develop Reference

efficient way of streaming a html5 canvas content?

I'm trying to stream the content of a html5 canvas on a live basis using websockets and nodejs.
The content of the html5 canvas is just a video.
What I have done so far is:
I convert the canvas to blob and then get the blob URL and send that URL to my nodejs server using websockets.
I get the blob URL like this:
canvas.toBlob(function(blob) {
url = window.URL.createObjectURL(blob);
});
The blob URLs are generated per video frame (20 frames per second to be exact) and they look something like this:
blob:null/e3e8888e-98da-41aa-a3c0-8fe3f44frt53
I then get that blob URL back from the the server via websockets so I can use it to DRAW it onto another canvas for other users to see.
I did search how to draw onto canvas from blob URL but I couldn't find anything close to what i am trying to do.
So the questions I have are:
Is this the correct way of doing what i am trying to achieve? any
pros and cons would be appreciated.
Is there any other more efficient way of doing this or I'm on a right
path?
Thanks in advance.
EDIT:
I should have mentioned that I cannot use WebRTC in this project and I have to do it all with what I have.
to make it easier for everyone where I am at right now, this how I tried to display the blob URLs that I mentioned above in my canvas using websockets:
websocket.onopen = function(event) {
websocket.onmessage = function(evt) {
var val = evt.data;
console.log("new data "+val);
var canvas2 = document.querySelector('.canvMotion2');
var ctx2 = canvas2.getContext('2d');
var img = new Image();
img.onload = function(){
ctx2.drawImage(img, 0, 0)
}
img.src = val;
};
// Listen for socket closes
websocket.onclose = function(event) {
};
websocket.onerror = function(evt) {
};
};
The issue is that when I run that code in FireFox, the canvas is always empty/blank but I see the blob URLs in my console so that makes me think that what I am doing is wrong.
and in Google chrome, i get Not allowed to load local resource: blob: error.
SECOND EDIT:
This is where I am at the moment.
First option
I tried to send the whole blob(s) via websockets and I managed that successfully. However, I couldn't read it back on the client side for some strange reason!
when I looked on my nodejs server's console, I could see something like this for each blob that I was sending to the server:
<buffer fd67676 hdsjuhsd8 sjhjs....
Second option:
So the option above failed and I thought of something else which is turning each canvas frame to base64(jpeg) and send that to the server via websockets and then display/draw those base64 image onto the canvas on the client side.
I'm sending 24 frames per second to the server.
This worked. BUT the client side canvas where these base64 images are being displayed again is very slow and and its like its drawing 1 frame per second. and this is the issue that i have at the moment.
Third option:
I also tried to use a video without a canvas. So, using WebRTC, I got the video Stream as a single Blob. but I'm not entiely sure how to use that and send it to the client side so people can see it.
IMPORTANT: this system that I am working on is not a peer to peer connection. its just a one way streaming that I am trying to achieve.

The most natural way to stream a canvas content: WebRTC
OP made it clear that they can't use it, and it may be the case for many because,
Browser support is still not that great.
It implies to have a MediaServer running (at least ICE+STUN/TURN, and maybe a gateway if you want to stream to more than one peer).
But still, if you can afford it, all you need then to get a MediaStream from your canvas element is
const canvas_stream = canvas.captureStream(minimumFrameRate);
and then you'd just have to add it to your RTCPeerConnection:
pc.addTrack(stream.getVideoTracks()[0], stream);
Example below will just display the MediaStream to a <video> element.
let x = 0;
const ctx = canvas.getContext('2d');
draw();
startStream();
function startStream() {
// grab our MediaStream
const stream = canvas.captureStream(30);
// feed the <video>
vid.srcObject = stream;
vid.play();
}
function draw() {
x = (x + 1) % (canvas.width + 50);
ctx.fillStyle = 'white';
ctx.fillRect(0,0,canvas.width,canvas.height);
ctx.fillStyle = 'red';
ctx.beginPath();
ctx.arc(x - 25, 75, 25, 0, Math.PI*2);
ctx.fill();
requestAnimationFrame(draw);
}
video,canvas{border:1px solid}
<canvas id="canvas">75</canvas>
<video id="vid" controls></video>
The most efficient way to stream a live canvas drawing: stream the drawing operations.
Once again, OP said they didn't want this solution because their set-up doesn't match, but might be helpful for many readers:
Instead of sending the result of the canvas, simply send the drawing commands to your peers, which will then execute these on their side.
But this approach has its own caveats:
You will have to write your own encoder/decoder to pass the commands.
Some cases might get hard to share (e.g external media would have to be shared and preloaded the same way on all peers, and the worse case being drawing an other canvas, where you'd have to also have shared its own drawing process).
You may want to avoid intensive image processing (e.g ImageData manipulation) to be done on all peers.
So a third, definitely less performant way to do it, is like OP tried to do:
Upload frames at regular interval.
I won't go in details in here, but keep in mind that you are sending standalone image files, and hence a whole lot more data than if it had been encoded as a video.
Instead, I'll focus on why OP's code didn't work?
First it may be good to have a small reminder of what is a Blob (the thing that is provided in the callback of canvas.toBlob(callback)).
A Blob is a special JavaScript object, which represents binary data, generally stored either in browser's memory, or at least on user's disk, accessible by the browser.
This binary data is not directly available to JavaScript though. To be able to access it, we need to either read this Blob (through a FileReader or a Response object), or to create a BlobURI, which is a fake URI, allowing most APIs to point at the binary data just like if it was stored on a real server, even though the binary data is still just in the browser's allocated memory.
But this BlobURI being just a fake, temporary, and domain restricted path to the browser's memory, can not be shared to any other cross-domain document, application, and even less computer.
All this to say that what should have been sent to the WebSocket, are the Blobs directly, and not the BlobURIs.
You'd create the BlobURIs only on the consumers' side, so that they can load these images from the Blob's binary data that is now in their allocated memory.
Emitter side:
canvas.toBlob(blob=>ws.send(blob));
Consumer side:
ws.onmessage = function(evt) {
const blob = evt.data;
const url = URL.createObjectURL(blob);
img.src = url;
};
But actually, to even better answer OP's problem, a final solution, which is probably the best in this scenario,
Share the video stream that is painted on the canvas.

Javascript - Draw specific part of image on canvas

I'm trying to send images from desk to webapp, on the desk part I have a button that takes a screen-shot then compares it with the one taken before, it calculates the difference between them then generate a new Bitmap based on this difference, then it sends this new Bitmap to the webapp which draws it on a canvas, this all happens in real-time using socket.io.
My problem is, let's assume I use Google Chrome on my screen and took a screen-shot, then I opened cmd.exe & took another screenshot, assuming that cmd.exe size is 300 * 100, and starts at half of the screen, the new generated bitmap will contains the cmd.exe screen-shots but rest of the screen (the uncover parts of Google Chrome) will be black, which is perfect for my case since i want to reduce bandwidth usage, now what I want to do, is take the difference bitmap (blob) on the JavaScript side and draw the difference on the previous screen-shot, which will make it look like I transferred the whole screen-shot, If i simply parse the blobs on the canvas, It will make a black screen with the cmd.exe on the center of it, here is my current code :
socket.on("image up", (bin) => {
var ctx = canvas[0].getContext('2d');
var img = new Image();
img.onload = function() {
ctx.drawImage(img, 0, 0);
}
var urlCreator = window.URL || window.webkitURL;
var binaryData = [];
binaryData.push(bin);
img.src = urlCreator.createObjectURL(new Blob(binaryData));
});
Any suggestions ? Thanks in advance

LocalStorage phonegap camera image

i'm trying to use localStorage to save an image, or multiple images for retrieval at a later date to upload to a server.
The current camera code is as follows:
function capturePhoto() {
navigator.camera.getPicture(onCameraSuccess, onCameraFail, {quality: 70, destinationType : Camera.DestinationType.DATA_URL});
}
function onCameraSuccess(imageData) {
//In our success call we want to first process the image to save in our image box on the screen.
var image = document.getElementById('image');
image.src = "data:image/jpeg;base64," + imageData;
//Create a new canvas for our image holder
var imgCanvas = document.createElement("canvas"),
imgContext = imgCanvas.getContext("2d");
// Make sure canvas is as big as the picture
imgCanvas.width = image.width;
imgCanvas.height = image.height;
// Draw image into canvas element
imgContext.drawImage(image, 0, 0, image.width, image.height);
// Get canvas contents as a data URL
var imgAsDataURL = imgCanvas.toDataURL("image/png");
// Save image into localStorage
try {
// localStorage.setItem(“savedImage”, imgAsDataURL);
localStorage.setItem("savedImage", imageData);
alert('Image Saved');
}
catch (e) {
alert("Storage failed: " + e);
}
var imageStorage = localStorage.getItem("savedImage");
// myCardHolder= document.getElementById(“m1-cardStorage-image1″);
// Reuse existing Data URL from localStorage
var imageInfo = document.getElementById('image');
imageInfo.src = "data:image/jpeg;base64," + imageStorage;
}
This triggers the camera, and the image captured is displayed into
<img id="image" src=""></img>
It also draws a canvas to output the image into. What i'm really trying to achieve is to capture the images base64 data to be able to store it into an array so that it may be uploaded/downloaded from a server.
Ideally i'd like to completely avoid having to display the image to the user, and simply store the images data
I may have misunderstood the localStorage/camera api a little, so any pointers would be great.
Does the image HAVE to be output into an element before the data can be stored? If i could just output it into the canvas that may never have to be shown, and extract the data from the canvas element?

Does the image HAVE to be output into an element before the data can be stored?
Not at all, in this case anyways. You are already receiving the image as base64 data so just store that directly.
Problems:
datauris can be chopped by the browser if too long
if not chopped by browser on string level, the data can be chopped by localstorage itself which has a size limit (i think it's currently around 5 mb for most browsers but there is no standard here)
a string uses two bytes per char so the storage is in effect the half
A better local storage is to use indexedDB.
When you read the base64 data, then you have to use an Image to show the data. Just prefix as you do with data:... etc. and remember to use correct file type.

Last year I was trying to solve the same problem, I don't have the code right now but I followed kind of the approach taken on this answer:
How to convert image into base64 string using javascript
Remember that localStorage has a limit of 5 MB, so if you save a lot of images in b64 you can reach that limit easily. (which was my case), so I had to move my storage to somewhere else, like a sqlite or something like that.

Why does canvas.toDataURL() not produce the same base64 as in Ruby for an image?

I'm trying to produce the same base64 data for an image file in both JavaScript and in Ruby. Unfortunately both are outputting two very different values.
In Ruby I do this:
Base64.encode64(File.binread('test.png'));
And then in JavaScript:
var image = new Image();
image.src = 'http://localhost:8000/test.png';
$(image).load(function() {
var canvas, context, base64ImageData;
canvas = document.createElement('canvas');
context = canvas.getContext('2d');
canvas.width = this.width;
canvas.height = this.height;
context.drawImage(this, 0, 0);
imageData = canvas.toDataURL('image/png').replace(/data:image\/[a-z]+;base64,/, '');
console.log(imageData);
});
Any idea why these outputs are different?

When you load the image in Ruby the binary file without any modifications will be encoded directly to base-64.
When you load an image in the browser it will apply some processing to the image before you will be able to use it with canvas:
ICC profile will be applied (if the image file contains that)
Gamma correction (where supported)
By the time you draw the image to canvas, the bitmap values has already been changed and won't necessarily be identical to the bitmap that was encoded before loading it as image (if you have an alpha channel in the file this may affect the color values when drawn to canvas - canvas is a little peculiar at this..).
As the color values are changed the resulting string from canvas will naturally also be different, before you even get to the stage of re-encoding the bitmap (as PNG is loss-less the encoding/compressing should be fairly identical, but factors may exist depending on the browser implementation that will influence that as well. to test, save out a black unprocessed canvas as PNG and compare with a similar image from your application - all values should be 0 incl. alpha and at the same size of course).
The only way to avoid this is to deal with the binary data directly. This is of course a bit overkill (in general at least) and a relative slow process in a browser.
A possible solution that works in some cases, is to remove any ICC profile from the image file. To save an image from Photoshop without ICC choose "Save for web.." in the file menu.

The browser is re-encoding the image as you save the canvas.
It does not generate an identical encoding to the file you rendered.

So I actually ended up solving this...
Fortunately I am using imgcache.js to cache images in the local filesystem using the FileSystem API. My solution is to use this API (and imgcache.js makes it easy) to get the base64 data from the actual cached copy of the file. The code looks like this:
var imageUrl = 'http://localhost:8000/test.png';
ImgCache.init(function() {
ImgCache.cacheFile(imageUrl, function() {
ImgCache.getCachedFile(imageUrl, function(url, fileEntry) {
fileEntry.file(function(file) {
var reader = new FileReader();
reader.onloadend = function(e) {
console.log($.md5(this.result.replace(/data:image\/[a-z]+;base64,/, '')));
};
reader.readAsDataURL(file);
});
});
});
});
Also, and very importantly, I had to remove line breaks from the base64 in Ruby:
Base64.encode64(File.binread('test.png')).gsub("\n", '');

build a chrome extension in order to upload images (from clipboard)

I wanted to write a simple chrome extension in order to substitute the following sequence of steps which i have to do very often for university:
make screenshot of something
edit screenshot in Paint
save unnamend.png to harddrive
upload unnamed.png to imageshack.us/pic-upload.de or any
other website
share link of image with others.
I don't care which image upload service to use, i just want automize this use-case in order to save time (I already red and did a getting-started chrome extension and checked out their API, but that's it, this page: http://farter.users.sourceforge.net/blog/2010/11/20/accessing-operating-system-clipboard-in-chromium-chrome-extensions/ seemed useful, but i couldn't make it overwrite my systems clipboard - moreover i can't find a tutorial which helps me further).

To answer your questions, I will give you some hints and resources to do what you want:
1 - Screenshot using Chrome Extensions API
You can use chrome.tabs.captureVisibleTab to screenshot what you see.
chrome.tabs.captureVisibleTab(null, {format:'png'}, function(dataURL) {
// Your image is in the dataURL
});
2 - Edit Screenshot using HTML5
Well, here is a tricky one, why do you want to use Paint while you can use HTML5 or a web service? If you want to use paint, then the only way doing this is natively through NPAPI (C++). Exposing something natively is really discouraged since it poses additional security risks to users. And it requires manual review for submission and a deadly warning when installing.
Why can't you use HTML5 Canvas to modify the screenshot? Or even, use Picnik online photo editing http://www.picnik.com/
var image_buffer = document.createElement('img');
image_buffer.src = dataURL; // From #1 capture tab
image_buffer.onload = function() {
var canvas = document.createElement('canvas');
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;
var ctx = canvas.getContext('2d');
ctx.drawImage(image_buffer, 0, 0);
// Draw something extra ontop of it such as a circle.
ctx.beginPath();
ctx.arc(0, 0, 10, 0, Math.PI*2, true);
ctx.closePath();
ctx.fill();
// Convert that back to a dataURL
var dataURL = canvas.toDataURL('image/png');
// Base64 image only.
var image64 = dataURL.replace(/data:image\/png;base64,/, '');
};
3 - Save Image to Hard drive
This is another tricky one, right now, if you use a "service" like Picnick, then you can use their saving utility to save to your harddrive, otherwise you can use HTML5 FileWriter API which is currently being developed.
If you really want to with your MSPaint route, then you would need to use NPAPI as mentioned above.
But when you use HTML5, you can do the following, but it is still early in spec:
var bb = new BlobBuilder();
bb.append(image64); // From #2 when done editing.
var blob = bb.getBlob();
location.href = createObjectURL(blob);
4 - Upload image to an Online Image Service
You can use http://imgurl.com to upload too, it has a neat API that you can use. All you need to know is plain old javascript, just send an XHR request to request the service and communicate with it.
To do that, just use their API:
var xhr = new XMLHttpRequest();
xhr.open('POST', 'http://api.imgur.com/2/upload.json?key=' + apikey, true);
xhr.setRequestHeader('Cache-Control', 'no-cache');
xhr.onreadystatechange = function() {
if (this.readyState == 4) {
var response = JSON.parse(xhr.response);
if (response.error) {
alert('Error: ' + response.error.message);
return;
}
var image_url = response.upload.links.original;
}
};
xhr.send(image64); // From #2 where we edit the screenshot.
5 - Share link of image with others.
Same as above, this is just plain old javascript, depends on the service (Facebook, Twitter, Buzz), you use their API or another service does that already for you to share the images.
Note:
I would say the best way to do this extension is using HTML5. You can use XHR to communicate to external websites, Canvas to edit the photos, and FileWriter to persist that to disk.
I would highly discourage the NPAPI approach for such extension since it isn't needed. Plus, if you go through NPAPI, you would need to make it cross platform and develop plugins for each browser.