Develop Reference

Why does switching from Canvas to OffScreenCanvas improve performance of drawImage and getImageData?

I am running a real-time process from the live media stream of the webcam, which requires the FPS to stay around 30FPS. getImageData was one of my main bottlenecks, it was requiring around 22ms to extract the ImageData object from the canvas, leaving me only 11ms to do my other operations.
Here is a simple snippet of what I had before
video = document.createElement("video");
inputCanvas = document.createElement("canvas");
inputCanvas.width = width;
inputCanvas.height = height;
inputCanvasCtx = inputCanvas.getContext("2d");
video.width = width;
video.height = height;
video.srcObject = stream;
// And using the requestVideoFrameCallback API, whenever a frame is presented, this callback is triggered
function onFrame(now, metadata){
inputCanvasCtx.drawImage(video, 0, 0);;
// Get frame from our canvas.
frame = inputCanvasCtx.getImageData(0,0,inputCanvas.width, inputCanvas.height);
// Other computations
}
After doing this simple change to my setup, without changing anything else I managed to extract the frame in 11ms, half the time it took before, and I cannot understand why. I only converted the regular canvas to an offscreencanvas in two lines.
video = document.createElement("video");
inputCanvas = document.createElement("canvas");
inputCanvas.width = width;
inputCanvas.height = height;
offscreen = inputCanvas.transferControlToOffscreen(); // Changed line
inputCanvasCtx = offscreen.getContext("2d"); // Changed line
video.width = width;
video.height = height;
video.srcObject = stream;
I would appreciate any insight on this matter.

Offset playback of two videos when drawing canvas

It's my first post on Stackoverflow! I hope this is clear, and thanks in advance for your help.
I am drawing two videos to canvas, and recording the canvas for a single video for playback and review.
video1 is a video from our server.
video2 is the user's webcam feed.
The issue:
The two videos are not entirely synced due to some delay in the webcam capture.
I am trying to figure out how to delay the ctx.DrawImage of video1 by a "x" milliseconds so that I can get the two videos to appear on the canvas more synchronized, making the final recording look more synchronized.
Here is my code for drawing to the canvas.
// Draw two videos to one canvas
$(function() {
var canvas = document.getElementById('canvas');
var ctx = canvas.getContext('2d');
var video1 = document.getElementById('teachervid');
var video2 = document.getElementById('studentvideo');
video1.addEventListener('play', function() {
var $this = this; //cache
(function loop() {
if (!$this.paused && !$this.ended) {
ctx.drawImage($this, 0, 0, 960, 540);
setTimeout(loop, 1000 / 30); // drawing at 30fps
}
})();
}, 0);
video2.addEventListener('play', function() {
var $this = this; //cache
(function loop() {
if (!$this.paused && !$this.ended) {
ctx.drawImage($this, 960, 0, 960, 540);
setTimeout(loop, 1000 / 30); // drawing at 30fps
}
})();
}, 0);
});
To be clear, I am not trying to change the frame rate, simply offset the start points of the two videos so that video 1 is drawn X milliseconds after video2 is drawn. I assume I will need to tweak and test what X should be once I figure out how.
Note, Keep in mind that I am new to JS and programming when responding.
Cheers!

Safari is open new window when I play the video

I create a video tag in javascript and using the canvas to display the video:
const canvas = document.getElementById('canvas');
const ctx2d = canvas.getContext('2d');
const videoElement = document.createElement('video');
videoElement.height = canvas.height;
videoElement.width = canvas.width;
videoElement.playsinline = "playsinline";
videoElement.setAttribute('webkit-playsinline', 'webkit-playsinline');
videoElement.src = '.....';
videoElement.autoplay = false;
videoElement.loop = false;
videoElement.muted = true;
videoElement.addEventListener('play', () => {
(function loop() {
if (videoElement && !videoElement.paused && !videoElement.ended) {
ctx2d.drawImage(videoElement, 0, 0, canvas.width, canvas.height);
setTimeout(loop, 1000 / 30); // drawing at 30fps
}
})();
});
The problem is in my ios device (iphone 11 - safari), when I play the video, ios play the video in new window (fullscreen).
What I need to do to solve that problem? any safari developers here?
This is what I get when I play the video. the fullscreen mode of safari.

javascript and HTML5 draw video into canvas

I'm trying to draw a video into the canvas in javascript and HTML5 (like an intro video for a game) which is working fine on the web browser but is not working on android, so I'm not sure if I making a mistake somewhere or this is not supported on mobile devices.
Here is the code:
//create canvas and get 2d context
var canvas = document.getElementById('introVid');
var ctx = canvas.getContext('2d');
// allow fullscreen
function fullscreen() {
var el = canvas;
if (el.webkitRequestFullScreen) {
el.webkitRequestFullScreen(); //chrome
} else {
el.mozRequestFullScreen(); //firefox
}
}
canvas.addEventListener("click", fullscreen); //enable fullscreen onClick
//draw the video into canvas
var video = document.getElementById("video");
video.addEventListener('loadeddata', function() {
video.play(); // start playing
update(); //start rendering
});
function update() {
ctx.drawImage(video, 0, 0, 300, 150);
requestAnimationFrame(update); //wait the browser be ready for next frame
};
Here a working demo on fiddle:
http://jsfiddle.net/h1hjp0Lp/122/

Recording speed javascript

I am currently creating a project that supports video recording through my website.
I create a canvas and then push the recorded frames to it. The problem is, when I play the video after its recorded, it plays too fast. A 10 second long video plays in like 2 seconds. I have checked the playbackRate is set to 1. I save the recording to a database and its speeded up there aswell, so it has nothing to do with the browsers videoplayer.
I am relative new to AngularJS and javascript so im sorry if I left something important out.
I have tried changing alot of the values back and forth but I cant seem to find the cause for the problem. Any ideas?
Here is the code for the video recording:
scope.startRecording = function () {
if (mediaStream) {
var video = $('.video-capture')[0];
var canvas = document.createElement('canvas');
canvas.height = video.videoHeight;
canvas.width = video.videoWidth;
ctx = canvas.getContext('2d');
var CANVAS_WIDTH = canvas.width;
var CANVAS_HEIGHT = canvas.height;
function drawVideoFrame(time) {
videoRecorder = requestAnimationFrame(drawVideoFrame);
ctx.drawImage(video, 0, 0, CANVAS_WIDTH, CANVAS_HEIGHT);
recordedFrames.push(canvas.toDataURL('image/webp', 1));
}
videoRecorder = requestAnimationFrame(drawVideoFrame); // Note: not using vendor prefixes!
scope.recording = true;
}
};
scope.stopRecording = function () {
cancelAnimationFrame(videoRecorder); // Note: not using vendor prefixes!
// 2nd param: framerate for the video file.
scope.video.files = Whammy.fromImageArray(recordedFrames, 1000 / 30);
recordedVideoBlob = Whammy.fromImageArray(recordedFrames, 1000 / 30);
scope.videoMode = 'viewRecording';
scope.recording = false;
};

I am guess the culprit is requestAnimationFrame, left on it's own, you cannot tell at what intervals it keeps calling the callback, it can be as high as 60fps.
also looking at your code, I cannot tell how you came to the conclusion that frame rate = 1000/30
my advice( at least for your case) would be to go with $interval,
you can do something like:
scope.frameRate = 10, videoInterval; // the amount I consider ideal for client-side video recording.
scope.startRecording = function () {
if (mediaStream) {
var video = $('.video-capture')[0];
var canvas = document.createElement('canvas');
canvas.height = video.videoHeight;
canvas.width = video.videoWidth;
ctx = canvas.getContext('2d');
var CANVAS_WIDTH = canvas.width;
var CANVAS_HEIGHT = canvas.height;
function drawVideoFrame() {
ctx.drawImage(video, 0, 0, CANVAS_WIDTH, CANVAS_HEIGHT);
recordedFrames.push(canvas.toDataURL('image/webp', 1));
}
videoInterval = $interval(drawVideoFrame, 1000/scope.frameRate);
scope.recording = true;
}
};
scope.stopRecording = function () {
$interval.cancel(videoInterval);
// 2nd param: framerate for the video file.
scope.video.files = Whammy.fromImageArray(recordedFrames, scope.frameRate);
recordedVideoBlob = Whammy.fromImageArray(recordedFrames, scope.frameRate); // you can chage this to some file copy method, so leave out the duplicate processing of images into video.
scope.videoMode = 'viewRecording';
scope.recording = false;
};