I want to create a browser extension that would allow users to add effects to their video/audio streams, without special plugins, on any site that uses the javascript web apis.
Google searching has not been particularly helpful so I'm starting to wonder if this is even possible.
I have 2 primary questions here:
Is this possible with javascript+chrome?
Any links to additional resources are greatly appreciated.
I am not really into web-extensions, so there may even be a simpler API available and I won't go into details about the implementation but theoretically you can indeed do it.
All it takes is to override the methods from where you'd get your MediaStream, to draw the original MediaStream to an HTML canvas where you'd be able to apply your filter, and then simply to return a new MediaStream made of the VideoTrack of a MediaStream from the canvas element's captureStream(), and possibly other tracks from the original MediaStream.
A very basic proof of concept implementation for gUM could look like:
// overrides getUserMedia so it applies an invert filter on the videoTrack
{
const mediaDevices = navigator.mediaDevices;
const original_gUM = mediaDevices.getUserMedia.bind(mediaDevices);
mediaDevices.getUserMedia = async (...args) => {
const original_stream = await original_gUM(...args);
// no video track, no filter
if( !original_stream.getVideoTracks().length ) {
return original_stream;
}
// prepare our DOM elements
const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');
const video = document.createElement('video');
// a flag to know if we should keep drawing on the canvas or not
let should_draw = true;
// no need for audio there
video.muted = true;
// gUM video tracks can change size
video.onresize = (evt) => {
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
};
// in case users blocks the camera?
video.onpause = (evt) => {
should_draw = false;
};
video.onplaying = (evt) => {
should_draw = true;
drawVideoToCanvas();
};
video.srcObject = original_stream;
await video.play();
const canvas_track = canvas.captureStream().getVideoTracks()[0];
const originalStop = canvas_track.stop.bind(canvas_track);
// override the #stop method so we can revoke the camera stream
canvas_track.stop = () => {
originalStop();
should_draw = false;
original_stream.getVideoTracks()[0].stop();
};
// merge with audio tracks
return new MediaStream( original_stream.getAudioTracks().concat( canvas_track ) );
// the drawing loop
function drawVideoToCanvas() {
if(!should_draw) {
return;
}
ctx.filter = "none";
ctx.clearRect(0, 0, canvas.width, canvas.height);
ctx.filter = "invert(100%)";
ctx.drawImage(video,0,0);
requestAnimationFrame( drawVideoToCanvas );
}
};
}
And then every scripts that would call this method would receive a filtered videoTrack.
Outsourced example since gUM is not friend with StackSnippets.
Now I'm not sure how to override methods from web-extensions, you'll have to learn that by yourself, and beware this script is really just a proof of concept and not ready for production. I didn't put any though in handling anything than the demo case.
Related
I'd like to build an event handler to deal with each new frame of an HTML 5 Video element. Unfortunately, there's no built in event that fires for each new video frame (the timeupdate event is the closest but fires for each time change rather than each video frame).
Has anyone else run into this same issue? Is there a good way around it?
There is an HTMLVideoElement.requestVideoFrameCallback() method that is still being drafted, and thus neither stable, nor widely implemented (it is only in Chromium based browsers), but which does what you want, along with giving many other details about that frame.
For your Firefox users, this browser has a non standard seekToNextFrame() method, which, depending on what you want to do you could use. This won't exactly work as an event though, it more of a way to, well... seek to the next frame. So this will greatly affect the playing of the video, since it won't respect the duration of each frames.
And for Safari users, the closest is indeed the timeupdate event, but as you know, this doesn't really match the displayed frame.
(async() => {
const log = document.querySelector("pre");
const vid = document.querySelector("video");
const canvas = document.querySelector("canvas");
const ctx = canvas.getContext("2d");
if( vid.requestVideoFrameCallback ) {
await vid.play();
canvas.width = vid.videoWidth;
canvas.height = vid.videoHeight;
ctx.filter = "invert(1)";
const drawingLoop = (timestamp, frame) => {
log.textContent = `timestamp: ${ timestamp }
frame: ${ JSON.stringify( frame, null, 4 ) }`;
ctx.drawImage( vid, 0, 0 );
vid.requestVideoFrameCallback( drawingLoop );
};
vid.requestVideoFrameCallback( drawingLoop );
}
else if( vid.seekToNextFrame ) {
const requestNextFrame = (callback) => {
vid.addEventListener( "seeked", () => callback( vid.currentTime ), { once: true } );
vid.seekToNextFrame();
};
await vid.play();
await vid.pause();
canvas.width = vid.videoWidth;
canvas.height = vid.videoHeight;
ctx.filter = "invert(1)";
const drawingLoop = (timestamp) => {
log.textContent = "timestamp: " + timestamp;
ctx.drawImage( vid, 0, 0 );
requestNextFrame( drawingLoop );
};
requestNextFrame( drawingLoop );
}
else {
console.error("Your browser doesn't support any of these methods, we should fallback to timeupdate");
}
})();
video, canvas {
width: 260px;
}
<pre></pre>
<video src="https://upload.wikimedia.org/wikipedia/commons/2/22/Volcano_Lava_Sample.webm" muted controls></video>
<canvas></canvas>
Note that the encoded frames and the displayed ones are not necessarily the same thing anyway and that browser may not respect the encoded frame rate at all. So based on what you are willing to do, maybe a simple requestAnimationFrame loop, which would fire at every update of the monitor might be better.
I am a student studying programming.
I am not good at English, so I wrote it using a translator.
I'm studying the mediapipe.
https://google.github.io/mediapipe/solutions/face_mesh
Do you know how to use local video instead of webcam?
let videoElement = document.querySelector(".input_video")
//#mediapipe/camera_utils/camera_utils.js"
const camera = new Camera(videoElement, {
onFrame: async () => {
await holistic.send({ image: videoElement });
},
width: 640,
height: 480,
});
camera.start();
This is the code to get the webcam.
I think I need to change this code but I don't know how to work it.
so I tried to find out about '#mediapipe/camera_utils/camera_utils.js', I couldn't find any data.
And I found using the local video in the codepen demo.
https://codepen.io/mediapipe/details/KKgVaPJ
But I don't know which part of the code to use.
Please teach me the way.
Rather than create a new Camera, you need to send the frames using requestAnimationFrame(). However as the send needs to be in an async function
the requestAnimationFrame needs to be within a Promise.
You have the standard mediapipe setup
let videoElement = document.querySelector(".input_video");
const config = {
locateFile: (file) => {
return 'https://cdn.jsdelivr.net/npm/#mediapipe/face_mesh#' +
`${mpFaceMesh.VERSION}/${file}`;
}
};
const solutionOptions = {
selfieMode: false,
enableFaceGeometry: false,
maxNumFaces: 1,
refineLandmarks: true, //false,
minDetectionConfidence: 0.5,
minTrackingConfidence: 0.5
};
const faceMesh = new mpFaceMesh.FaceMesh(config);
faceMesh.setOptions(solutionOptions);
faceMesh.onResults(onResults);
but rather than the new Camera() or SourcePicker() you need the an animation frame loop
async function onFrame() {
if (!videoElement.paused && !videoElement.ended) {
await faceMesh.send({
image: videoElement
});
// https://stackoverflow.com/questions/65144038/how-to-use-requestanimationframe-with-promise
await new Promise(requestAnimationFrame);
onFrame();
} else
setTimeout(onFrame, 500);
}
load the video
// must be same domain otherwise it will taint the canvas!
videoElement.src = "./mylocalvideo.mp4";
videoElement.onloadeddata = (evt) => {
let video = evt.target;
canvasElement.width = video.videoWidth;
canvasElement.height = video.videoHeight;
videoElement.play();
onFrame();
}
My product has a tool that allows you to share a video via WebRTC. When we first deployed it, we tried using a code like the following:
this.videoEl = document.createElement("video");
this.videoEl.src = url;
this.videoEl.oncanplay = function() {
this.oncanplay = undefined;
this.mediaStream = this.videoEl.captureStream();
};
The issue is that when sending this mediaStream, the result is a pitch green video, but with working audio:
The solution we came up with is to create a canvas and draw to our canvas the video contents, something like this:
this.canvas = document.createElement("canvas");
this.videoEl = document.createElement("video");
this.ctx = this.canvas.getContext("2d");
this.videoEl.src = url;
this.videoEl.oncanplay = function() {
this.oncanplay = undefined;
// Some code (stripping a lot of unnecessary stuff)
// Canvas drawing loop
this.canvas.width = this.videoEl.videoWidth;
this.canvas.height = this.videoEl.videoHeight;
this.ctx.drawImage(this.videoEl, 0, 0, this.videoEl.videoWidth, this.videoEl.videoHeight);
// Loop ends and more code
// Media stream element
this.mediaStream = this.canvas.captureStream(25);
// Attached audio track to Media Stream
try {
var audioContext = new AudioContext();
this.gainNode = audioContext.createGain();
audioSource = audioContext.createMediaStreamSource(this.videoEl.captureStream(25));
audioDestination = audioContext.createMediaStreamDestination();
audioSource.connect(this.gainNode);
this.gainNode.connect(audioDestination);
this.gainNode.gain.value = 1;
this.mediaStream.addTrack(audioDestination.stream.getAudioTracks()[0]);
} catch (e) {
// No audio tracks found
this.noAudio = true;
}
};
The solution works, however it consumes a lot of CPU and it would be great to avoid having to write all of that code. We also have customers complaining that the audio gets out of sync sometimes (which is understandable since I'm using a captureStream for audio and not for video.
At first I thought it was green because it was tainting the MediaStream, but that's not the case since I can normally draw the video to a canvas and capturing a MediaStream from it. PS: We are using a URL.createObjectURL(file) call to get the video url.
Do you know why the video is green?
Thanks.
It turns out it's a Google Chrome Bug.
Thanks to Philipp Hancke.
I am implementing Screen Capture API and I want to combine both primary and extended monitor for streaming and taking screenshots. Here is how I can capture one screen and save the screenshot and I want to enable the same for extended display as well.
screenshot = async() => {
const mediaDevices = navigator.mediaDevices as any;
let displayMediaOptions = {
video: {
mediaSource: 'screen'
}
}
const stream = await mediaDevices.getDisplayMedia(displayMediaOptions);
console.log('stream', stream.getTracks())
const track = stream.getVideoTracks()[0]
// init Image Capture and not Video stream
console.log(track)
const imageCapture = new ImageCapture(track)
const bitmap = await imageCapture.grabFrame()
// destory video track to prevent more recording / mem leak
track.stop()
const canvas = document.getElementById('fake')
// this could be a document.createElement('canvas') if you want
// draw weird image type to canvas so we can get a useful image
canvas.width = bitmap.width
canvas.height = bitmap.height
const context = canvas.getContext('2d')
context.drawImage(bitmap, 0, 0, bitmap.width, bitmap.height)
const image = canvas.toDataURL()
// this turns the base 64 string to a [File] object
const res = await fetch(image)
const buff = await res.arrayBuffer()
// clone so we can rename, and put into array for easy proccessing
const file = [
new File([buff], `photo_${new Date()}.jpg`, {
type: 'image/jpeg',
}),
]
return file
}
I can get the screenshot from single screen and attach it to a canvas. How can I do for extended display as well ?
I'm making an audio player with JavaScript, everything works fine until I add a sound visualizer. When I pause the song and then play it again, the sound gets more louder every time I do it, until it gets distorsionated.
I'm newbie with the HTML5 Audio API, I've tried to set the volume as a fixed value, but not works.
The code of the visualizer it's:
function visualizer(audio) {
let context = new AudioContext();
const gainNode = context.createGain();
gainNode.gain.value = 1; // setting it to 100%
gainNode.connect(context.destination);
let src = context.createMediaElementSource(audio);
let analyser = context.createAnalyser();
let canvas = document.getElementById("canvas");
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;
let ctx = canvas.getContext("2d");
src.connect(analyser);
analyser.connect(context.destination);
analyser.fftSize = 2048;
let bufferLength = analyser.frequencyBinCount;
let dataArray = new Uint8Array(bufferLength);
let WIDTH = ctx.canvas.width;
let HEIGHT = ctx.canvas.height;
let barWidth = (WIDTH / bufferLength) * 1.5;
let barHeight;
let x = 0;
let color = randomColor();
function renderFrame() {
requestAnimationFrame(renderFrame);
x = 0;
analyser.getByteFrequencyData(dataArray);
ctx.clearRect(0, 0, WIDTH, HEIGHT);
for (let i = 0; i < bufferLength; i++) {
barHeight = dataArray[i];
ctx.fillStyle = color;
ctx.fillRect(x, HEIGHT - barHeight, barWidth, barHeight);
x += barWidth + 1;
}
}
musicPlay();
renderFrame();
}
And:
function musicPlay() {
status = 'playing';
audio.play();
}
So, I don't know if I'm doing something wrong on the audio analyzer, I've tried to make a global context and don't do the new AudioContext(); every time I enter on the function, also I've tried to specify a fixed volume with:
audio.volume = 1;
or with the GainNode as you can see on the function, but it's not working.
Where is my mistake and why the sound gets louder?
Regards!
--- Update 1 ---
The audio it's loaded from an URL:
function loadAudioElement(url) {
return new Promise(function (resolve, reject) {
let audio = new Audio();
audio.addEventListener('canplay', function () {
/* Resolve the promise, passing through the element. */
resolve(audio);
});
/* Reject the promise on an error. */
audio.addEventListener('error', reject);
audio.src = url;
});
}
And on my player I have:
let playButtonFunction = function () {
if (playstatus === 'pause') {
loadAudioElement(audio.src).then(
visualizer(audio)
);
} else if (playstatus === 'playing') {
musicPause();
}
};
I had a similar issue, did you try to set the audio context to a global object?
This is what I found here:
https://developer.mozilla.org/en-US/docs/Web/API/AudioContext
It's recommended to create one AudioContext and reuse it instead of initializing a new one each time
The AudioContext interface represents an audio-processing graph built from audio modules linked together, each represented by an AudioNode.
An audio context controls both the creation of the nodes it contains and the execution of the audio processing, or decoding. You need to create an AudioContext before you do anything else, as everything happens inside a context. It's recommended to create one AudioContext and reuse it instead of initializing a new one each time, and it's OK to use a single AudioContext for several different audio sources and pipeline concurrently.
Well, as Get Off My Lawn pointed, I was adding by mistake multiple audio elements.
The solution was taking the code of load the song outside the playButtonFunction and only do:
let playButtonFunction = function () {
if (playstatus === 'pause') {
musicPlay();
} else if (playstatus === 'playing') {
musicPause();
}
};
But I still had one problem, with the next/previous functions. In these cases I need call the loadAudioElement function because the song is changing (when you press play/pause no, it's the same song) but with this I have the same problem again.
Well, after a bit of digging, I found that if you want to play a playlist and visualize the music all the time, YOU HAVE TO RELEASE THE OLD CONTEXT BEFORE LOAD THE NEW SONG. Not only to avoid the increase of the song volume, the cpu and memory will also get increased after 3 - 4 songs and the browser will start to run slowly depending on the machine. So:
1 - I made a global variable called clearContextAudio = false;
2 - On my next/previous functions I added this code:
if (closeAudioContext) { //MANDATORY RELEASE THE PREVIOUS RESOURCES TO AVOID OBJECT OVERLAPPING AND CPU-MEMORY USE
context.close();
context = new AudioContext();
}
loadAudioElement(audio.src).then(
visualizer(audio)
);
3 - On my visualizer(audio) function I changed:
let context = new AudioContext();
to
closeAudioContext = true; //MANDATORY RELEASE THE PREVIOUS RESOURCES TO AVOID OBJECT OVERLAPPING AND CPU-MEMORY USE
The value it's initialized to false because the first time there is no song playing, and after play a song you will always need to release the old resources, so the variable will always set to true. Now, you can skip all the times you want a song and not concern about the memory and the overlapping issues.
Hope this helps someone else trying to achieve the same thing! Regards!