Develop Reference

Detect differences between two video frames displayed in HTML5 canvas

I'm displaying a live video stream in a HTML5 canvas which works fine.
Now, what I need to do is to check for any "motion" in the camera which is being displayed in the HTML5 Canvas.
During my research, I found out that this can be done by checking the previous frame with the current frame that's being displayed in the canvas.
So I tried this code within a setInterval function:
var c = document.querySelector('.mycanv');
var ctx = c.getContext("2d");
var imageData = ctx.getImageData(0, 0, 200, 200);
var data = imageData.data.length;
console.log(data);
However, when I look in the console, the Number that the variable data outputs is always the same which is 16000! and it wont change even if there is a movemnet infront of the camera.
I'm not entirely sure if I am on a right track.
Could someone please advice on this issue and point me in a right direction please?
Thanks in advance.
Here is a minified and simple example:
https://jsfiddle.net/2648xwgz/
Basically, the code draws a frame of the video in the canvas.
Now, what I need to do is to check if the previous frame/image in the canvas is the same as the current frame/image.
Second EDIT:
Okay, so I've taken all the advice in the comments on board and tried to come up with something that actually doesn't check for all the random pixels in the images as that is heavy and not a good practic...
So, I tried something like this:
https://jsfiddle.net/qpxjcv3a/6/
The above code will run fine in Firefox and with a local Video file. but on JSFIDDLE, you will get a cross origion error.
Anyway, The key point in the code above is using the ctx.globalCompositeOperation = 'difference'; i guess.
And then doing a "scrore" calculation taken from here to detect some changes.
However, when I run my code, I always get: console.log('we have motion'); in the console! Even when the video is paused and there is no more new frames.
So I did console.log(imageScore); and it is always adding up by 10000 even when the video is paused or ended! I'm not sure why that is and whether this calculation is correct at all. But that is where I am at the moment.
Any pointers and help appreciated.

As Daniel said you need to check the pixels, the legnth would be the same for all iterations.
You should look into image hashing algorithms. At every interval you can calculate a hash, and store that in a global variable to compare at the next interval. This would also give you the option to set a threshold, so minor changes would not trigger motion detection.
This page explains image hashing in more details: https://jenssegers.com/61/perceptual-image-hashes
You can start by implementing average hash. It is quite simple. You would reduce your canvas size to 8x8 pixels.
ctx.drawImage(video, 0, 0, video.width, video.height, 0, 0, 8, 8);
var imageData = ctx.getImageData(0, 0, 8, 8);
var data = imageData.data;
Then you iterate over the image data and calculate the brightness.
var brightnessdata = []
for(var i = 0; i<data.length; i+=4){
brightnessdata.push((data[i]+data[i+1]+data[i+2])/3);
}
The rest is simply calculating the average brightness and comparing each pixel brightness to the average brightness to calculate the hash.

HTML5 Canvas FPS drops when rendering another canvas as an image using drawImage() at a specific width & height

So I have run into a very strange problem while using the HTML5 Canvas API. I'm attempting to create a game and the problem occurs because I'm drawing different canvases (using drawImage()) onto the main canvas. The other canvases has had graphics drawn on them and then I am simply drawing those canvases onto the main one. The problem is that at very specific widths and heights (coming close to the width and height of the main canvas) the fps suddenly drops by about 20-30. And this happens when only drawing one of those big canvases onto the main one. I thought the performance drop might've been attributed to drawing such a big canvas with graphics on them, so I emptied the canvases and was basically drawing an "empty" canvas onto the main canvas. Even so, the performance dropped. What's even more stange is that when I subtract just ONE pixel from either the width or height of the big canvas the fps goes back to 60! The widths and heights that this has occurred (there are probably more sets) are:
W: 1797
H: 891
W: 2026
H: 790
So for example for the first set, if you were to draw a canvas with those measurements (EVEN AN EMPTY ONE) you would get 30-40 fps. Yet if you were to draw a canvas with those one of the measurements subtracted by one (i.e. W: 1796, H: 891) then it would go back to 60 fps.
What I find even more strange is that this happens on only Chrome. I have tried it on Internet Explorer and Safari and I can draw significantly bigger canvases (i.e. far bigger than even the main canvas) and still get 60 fps. I'm sorry for not being able to list the code because it would require me to post a significantly large piece of code (the code is intertwined in multiple files). Could somebody elaborate on why this is happening? Thank you!
EDIT: This also happens if I draw only a clipped version of the big canvas using the other version of drawImage(). So it doesn't even have to do with rendering the actual number of pixels...which I find extremely strange.
EDIT 2: So I have run some tests and turns out it has nothing to do with the rendering but rather the memory usage because of the fact that I have an array that's holding 25 of these big canvases. I created a fiddle to benchmark:
https://jsfiddle.net/eu3zoc4f/3/
HTML:
<body>
<canvas id="canvas" style="display: block;"></canvas>
</body>
JS:
var timer = {
startedAt: null,
stoppedAt: null,
start: function() {
this.stoppedAt = null;
this.startedAt = new Date();
},
stop: function() {
this.stoppedAt = new Date();
},
getTime: function() {
if (!this.stoppedAt) this.stop();
return this.stoppedAt.getTime() - this.startedAt.getTime();
}
};
var body = document.getElementsByTagName("body");
body[0].style.width = screen.availWidth + "px";
document.getElementById("canvas").width = window.innerWidth;
document.getElementById("canvas").height = window.innerHeight;
var context = document.getElementById("canvas").getContext("2d");
var TESTING_1 = [];
for (var c = 0; c < 25; c++) {
TESTING_1[c] = document.createElement('canvas');
TESTING_1[c].width = 2000;
TESTING_1[c].height = 1200;
TESTING_1[c].getContext('2d').fillStyle = 'rgb(255, 0, 0)';
TESTING_1[c].getContext('2d').fillRect(0, 0, 1900, 897);
}
function main() {
timer.start();
context.drawImage(TESTING_1[0], 0, 0);
context.fillStyle = "blue";
context.font = "20px Arial";
context.fillText(timer.getTime(), 100, 100);
}
main();
If you change the "25" number that's in the for loop to a lower number you will get much faster results. So the problem now is why is this happening only in Chrome and is there a way to fix it?

Image Flickering In Canvas Game

For a university project I have been tasked with creating a Flappy Bird clone. It's being done using the HTML5 canvas.
The issue doesn't happen very often, but it seems that every 6 or so seconds, the grass will flicker. I'm not sure what's causing this, it could be a performance issue.
Here is a link so you may see the issue: http://canvas.pixcelstudios.uk
Here is the function I'm using to the draw the grass:
var drawGrass = function(cWidth, ctx, minusX)
{
var x = bg_grass.x;
var y = bg_grass.y;
var w = bg_grass.w;
var h = bg_grass.h;
var img = bg_grass.img;
if (minusX[0] >= cWidth)
{
bg_grass.x = 0;
minusX[0] = 0;
}
ctx.drawImage(img, x, y, w, h);
if (minusX[0] > 0)
{
ctx.drawImage(img, w-minusX[0], y, w, h);
}
};
Basically, I'm drawing two grass sprites, each taking up a canvas width. One starts with an X of 0 and the other starts at the end of the canvas. Both are decremented each frame, then one is completely off the screen, it's completely reset to keep it looping.
I don't think it's anything to do with my update loop which is as follows:
this.update = function()
{
clearScreen();
updateBackground();
updatePositions();
checkCollisions();
render();
requestAnimFrame(gameSpace.update);
};
I've done a little bit of reading and I've read about having a second canvas to act as a buffer. Apparently this can stop flickering and improve performance? But all of the examples I've seen show the parts being drawn into the canvas out of a loop and I can't really see how doing it within a game loop (moving parts and all) would increase performance rather than decrease it. Surely the same operations are being performed, except now you also have to draw the second canvas onto the first?
Please let me know if you need any more information (although you should be able to see the whole source from the web link).
Thanks!

Okay I found the issue! Was just a simple mistake in my drawGrass function.
Due to the ordering, there'd be just a single frame where I'd set my shorthand X variable to bg_grass.x and THEN set bg_grass.x to something else, therefore drawing the wrong value.
I've now set my shorthand variables after the first if-statement.
However, if anyone could provide any insight into the second part of the question regarding a buffer canvas, I'd still much appreciate that.

Analyser not updating canvas draws as quickly as audio is coming in

Whenever I have a new buffer that come into my client I want to redraw that instance of audio onto my canvas. I took the sample code from http://webaudioapi.com/samples/visualizer/ and tried to alter it to fit my needs in a live environment. I seem to have something working because I do see the canvas updating when I call .draw() but it's not nearly as fast as it should be. I'm probably seeing about 1 fps as it is. How do I speed up my fps and still call draw for each instance of a new buffer?
Entire code:
https://github.com/grkblood13/web-audio-stream/tree/master/visualizer
Here's the portion calling .draw() for every buffer:
function playBuffer(audio, sampleRate) {
var source = context.createBufferSource();
var audioBuffer = context.createBuffer(1, audio.length , sampleRate);
source.buffer = audioBuffer;
audioBuffer.getChannelData(0).set(audio);
source.connect(analyser);
var visualizer = new Visualizer(analyser);
visualizer.analyser.connect(context.destination);
visualizer.draw(); // Draw new canvas for every new set of data
if (nextTime == 0) {
nextTime = context.currentTime + 0.05; /// add 50ms latency to work well across systems - tune this if you like
}
source.start(nextTime);
nextTime+=source.buffer.duration; // Make the next buffer wait the length of the last buffer before being played
}
And here's the .draw() method:
Visualizer.prototype.draw = function() {
function myDraw() {
this.analyser.smoothingTimeConstant = SMOOTHING;
this.analyser.fftSize = FFT_SIZE;
// Get the frequency data from the currently playing music
this.analyser.getByteFrequencyData(this.freqs);
this.analyser.getByteTimeDomainData(this.times);
var width = Math.floor(1/this.freqs.length, 10);
// Draw the time domain chart.
this.drawContext.fillStyle = 'black';
this.drawContext.fillRect(0, 0, WIDTH, HEIGHT);
for (var i = 0; i < this.analyser.frequencyBinCount; i++) {
var value = this.times[i];
var percent = value / 256;
var height = HEIGHT * percent;
var offset = HEIGHT - height - 1;
var barWidth = WIDTH/this.analyser.frequencyBinCount;
this.drawContext.fillStyle = 'green';
this.drawContext.fillRect(i * barWidth, offset, 1, 2);
}
}
requestAnimFrame(myDraw.bind(this));
}

Do you have a working demo? as you can easily debug this using the Timeline in chrome. You can find out what process takes long. Also please take unnecessary math out. Most of your code doesn't need to be executed every frame. Also, how many times is the draw function called from playBuffer? When you call play, on the end of that function it requests a new animation frame. If you call play every time you get a buffer, you get much more cycles of math->drawing->request frame. This also makes it very slow. If you are already using the requestanimationframe, you shall only call the play function once.
To fix up the multi frame issue:
window.animframe = requestAnimFrame(myDraw.bind(this));
And on your playBuffer:
if(!animframe) visualizer.draw();
This makes sure it only executes the play function when there is no request.

Do you have a live example demo? I'd like to run it through some profiling. You're trying to get updates for the playing audio, not just once per chunk, right?
I see a number of inefficiencies:
you're copying the data at least once more than necessary - you should have your scheduleBuffers() method create an AudioBuffer of the appropriate length, rather than an array that then needs to be converted.
If I understand your code logic, it's going to create a new Visualizer for every incoming chunk, although they use the same Analyser. I'm not sure you really want a new Visualizer every time - in fact, I think you probably don't.
-You're using a pretty big fftSize, which might be desirable for a frequency analysis, but at 2048/44100 you're sampling more than you need. Minor point, though.
-I'm not sure why you're doing a getByteFrequencyData at all.
-I think the extra closure may be causing memory leakage. This is one of the reasons I'd like to run it through the dev tools.
-You should move the barWidth definition outside of the loop, along with the snippet length:
var snippetLength = this.analyser.frequencyBinCount;
var barWidth = WIDTH/snippetLength;
if you can post a live demo that shows the 1fps behavior, or send an URL to me privately (cwilso at google or gmail), I'd be happy to take a look.

HTML 5 canvas clip very costly

I have serious performance problems using canvas clip() with chrome.
I have made a test case to illustrate.
Even in a simple case like this the red rectangle blinks as if it takes too much time to redraw, and a CPU profiling shows the clip() method takes about 10% of the CPU.
In my real program, it get to 16% and keep increasing each frame until the canvas almost freezes the browser..
Is there something wrong in my use of clip ?
Thank you for any suggestions,
Regards.

Cause
Insert a beginPath() as rect() adds to the path unlike fillRect()/strokeRect(). What happens here is that the rectangles are accumulating eventually slowing the clipping down over time.
This in combination with using setInterval, which is not able to synchronize with monitor/screen refreshes, worsens the problem.
To elaborate on the latter:
Using setInterval()/setTimeout() can cause tearing which happens when the draw is in the "middle" of its operation not fully completed and a screen refresh occur.
setInterval can only take integer values and you would need for 16.67 ms to synchronize frame-wise (#60Hz). Even if setInterval could take floats it would not synchronize the timing with the monitor timing as the timer mechanism isn't bound to monitor at all.
To solve this always use requestAnimationFrame to synchronize drawings with screen updates. This is directly linked to monitor refreshes and is a more low-level and efficient implementation than the other, and is made for this purpose, hence the name.
Solution embedding both fixes above
See modified bin here.
The code for future visitors:
function draw() {
context.fillStyle = '#000';
context.fillRect(0, 0, width, height);
context.save();
context.beginPath(); /// add a beginPath here
context.rect(0, 0, 100, 100);
context.clip();
context.fillStyle = '#ff0000';
context.fillRect(0, 0, 200, 200);
context.restore();
requestAnimationFrame(draw); /// use rAF here
}
canvas.width = width;
canvas.height = height;
canvas.style.width = width+'px';
canvas.style.height = height+'px';
requestAnimationFrame(draw); /// start loop
PS: If you need to stop the loop inject a condition to run rAF all inside the loop, ie:
if (isPlaying) requestAnimationFrame(draw);
There is BTW no need for closePath() as this will be done implicit for you as by the specs. You can add it if you want, but calling clip() (or fill() etc.) will do this for you (specs is addressing browser implementers):
Open subpaths must be implicitly closed when computing the clipping region