My research suggests that creating an async class method is possible but I've found myself going in loops trying to get the values from a promise in my camera config class for a three.js project. So far the geometry is all located on the server and is sent to the browser on a different API without issue. I also need to send some site parameters from the server for the camera animation to work correctly and that is where I'm having a problem.
On the client's browser I set up the camera config class which is then used by multiple functions to center and move the camera in relation to the overall geometry. I thought it would be helpful to reuse the offset2middle value since it's already precalculated on the node server. However, I can't seem to find a convenient way to get the values directly into this.cameracenter. I need the values directly because I plan to do further manipulation of the values in the various camera animation functions to make sure the motion looks polished.
Does anyone have any suggestions about how this can be accomplished?
//All in the render.js file for the client
async function get_field_parameters() {
// gets field parameters from server using standard node js api
const response = await fetch(`/get-field-parameters/`);
const data = await response.json();
return data;
}
// Camera Config Class
class SCREEN_DATA {
constructor(window) {
this.height = window.innerHeight;
this.width = window.innerWidth;
this.aspect = this.width / this.height;
this.fov = 80;
this.camerafocus = this.find_center();
}
//Method
async find_center() {
let p = await get_field_parameters();
return p.offset2middle;
}
}
camera_config = new SCREEN_DATA(window); // screen Configuration
// Example of how I'd like to use the values
function main_camera(config) {
console.log(config.camerafocus); // return [[PromiseResult]]: Object x: 50 y: 0 z: 50
// Animated Camera
const camera = new THREE.PerspectiveCamera(config.fov, config.aspect, 1, 400);
camera.position.set(200, 100, 100); // start
camera.lookAt(50, 0, 50); // Want to use camera focus values here
return camera;
}
Related
I am attempting to create an app to detect a ball and another object while taking a video, say a racket, and these objects must be in FOV of the camera all the time to activate a function that then takes the average speed of the ball. I have established a site that does request permission to access webcam and displays it on screen, while making a recording button and all. However, my issues start turning up in my Javascript.EDIT:Start Recording contains id="togglerecordingmodeEl" onclick="toggleRecording()"
HTML:
Start Recording
Preview
JavaScript:
let video = null; // video element
let detector = null; // detector object
let detections = []; // store detection result
let videoVisibility = true;
let detecting = false;
const toggleRecordingEl = document.getElementById('toggleRecordingEl');
function toggleRecording() {
if (!video || !detector) return;
if (!detecting) {
detect();
toggleDetectingEl.innerText = 'Stop Detecting';
} else {
toggleDetectingEl.innerText = 'Start Detecting';
}
detecting = !detecting;
}
function detect() {
// instruct "detector" object to start detect object from video element
// and "onDetected" function is called when object is detected
detector.detect(video, onDetected);
}
// callback function. it is called when object is detected
function onDetected(error, results) {
if (error) {
console.error(error);
}
detections = results;
}
In summary, I want it to first recognise two objects in the FOV, and only if these two objects are in the FOV, then I can work on a new function. I have used https://medium.com/the-web-tub/object-detection-with-javascript-the-easy-way-74fbe98741cf & https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API/Recording_a_media_element to construct this app so far.
I am trying to implement a in browser raster drawing plugin for the leaflet library that that extends the leaflets GridLayer api. Essentially for every tile there is function createTile that returns a canvas with some drawing on it. and leaflet shows the tile in correct position.
initialize: function(raster_data){
this.raster_data = raster_data;
},
createTile: function (tile_coords) {
let _tile = document.createElement('canvas');
let _tile_ctx = _tile.getContext('2d');
// do some drawing here with values from this.raster_data
return _tile;
}
This implementation is so far working fine. Than I thought of offloading drawing with offscreen-canvas in a webworker. so I restructured the code like this
initialize: function(raster_data){
this.raster_data = raster_data;
this.tile_worker = new Worker('tile_renderer.js')
},
createTile: function (tile_coords) {
let _tile = document.createElement('canvas').transferControlToOffscreen();
this.tile_worker.postMessage({
_tile: _tile,
_raster_data: this.raster_data
},[_tile])
return _tile;
}
This works but every now and then i see a canvas that is just blank. That thing is quite random I don't know start from where and how should I debug this. can this be a problem that I am using a single worker for rendering every tile? any help is appreciated. Here is an example of a blank canvas.
This a known bug: https://crbug.com/1202481
The issue appears when too many OffscreenCanvases are sent to the Worker serially.
The workaround is then to batch send all these OffscreenCanvases in a single call to postMessage().
In your code you could achieve this by storing all the objects to be sent and use a simple debouncing strategy using a 0 timeout to send them all at once:
createTile: function (tile_coords) {
let _tile = document.createElement('canvas');
_tile.setAttribute('width', 512);
_tile.setAttribute('height', 512);
let _off_tile = _tile.transferControlToOffscreen();
this.tiles_to_add.push( _off_tile ); // store for later
clearTimeout( this.batch_timeout_id ); // so that the callback is called only once
this.batch_timeout_id = setTimeout( () => {
// send them all at once
this.tileWorker.postMessage( { tiles: this.tiles_to_add }, this.tiles_to_add );
this.tiles_to_add.length = 0;
});
return _tile;
}
Live example: https://artistic-quill-tote.glitch.me/
I want to create a browser extension that would allow users to add effects to their video/audio streams, without special plugins, on any site that uses the javascript web apis.
Google searching has not been particularly helpful so I'm starting to wonder if this is even possible.
I have 2 primary questions here:
Is this possible with javascript+chrome?
Any links to additional resources are greatly appreciated.
I am not really into web-extensions, so there may even be a simpler API available and I won't go into details about the implementation but theoretically you can indeed do it.
All it takes is to override the methods from where you'd get your MediaStream, to draw the original MediaStream to an HTML canvas where you'd be able to apply your filter, and then simply to return a new MediaStream made of the VideoTrack of a MediaStream from the canvas element's captureStream(), and possibly other tracks from the original MediaStream.
A very basic proof of concept implementation for gUM could look like:
// overrides getUserMedia so it applies an invert filter on the videoTrack
{
const mediaDevices = navigator.mediaDevices;
const original_gUM = mediaDevices.getUserMedia.bind(mediaDevices);
mediaDevices.getUserMedia = async (...args) => {
const original_stream = await original_gUM(...args);
// no video track, no filter
if( !original_stream.getVideoTracks().length ) {
return original_stream;
}
// prepare our DOM elements
const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');
const video = document.createElement('video');
// a flag to know if we should keep drawing on the canvas or not
let should_draw = true;
// no need for audio there
video.muted = true;
// gUM video tracks can change size
video.onresize = (evt) => {
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
};
// in case users blocks the camera?
video.onpause = (evt) => {
should_draw = false;
};
video.onplaying = (evt) => {
should_draw = true;
drawVideoToCanvas();
};
video.srcObject = original_stream;
await video.play();
const canvas_track = canvas.captureStream().getVideoTracks()[0];
const originalStop = canvas_track.stop.bind(canvas_track);
// override the #stop method so we can revoke the camera stream
canvas_track.stop = () => {
originalStop();
should_draw = false;
original_stream.getVideoTracks()[0].stop();
};
// merge with audio tracks
return new MediaStream( original_stream.getAudioTracks().concat( canvas_track ) );
// the drawing loop
function drawVideoToCanvas() {
if(!should_draw) {
return;
}
ctx.filter = "none";
ctx.clearRect(0, 0, canvas.width, canvas.height);
ctx.filter = "invert(100%)";
ctx.drawImage(video,0,0);
requestAnimationFrame( drawVideoToCanvas );
}
};
}
And then every scripts that would call this method would receive a filtered videoTrack.
Outsourced example since gUM is not friend with StackSnippets.
Now I'm not sure how to override methods from web-extensions, you'll have to learn that by yourself, and beware this script is really just a proof of concept and not ready for production. I didn't put any though in handling anything than the demo case.
Here's my code to capture an image from a Canvas playing video:
let drawImage = function(time) {
prevCtx.drawImage(videoPlayer, 0, 0, w, h);
requestAnimationFrame(drawImage);
}
requestAnimationFrame(drawImage);
let currIndex = 0;
setInterval(function () {
if(currIndex === 30) {
currIndex = 0;
console.log("Finishing video...");
videoWorker.postMessage({action : "finish"});
} else {
console.log("Adding frame...");
// w/o this `toDataURL` this loop runs at 30 cycle / second
// so that means, this is the hot-spot and needs optimization:
const base64img = preview.toDataURL(mimeType, 0.9);
videoWorker.postMessage({ action: "addFrame", data: base64img});
currIndex++;
}
}, 1000 / 30)
The goal is at each 30 frames (which should be at 1 second) it would trigger to transcode the frames added.
The problem here is that the preview.toDataURL(mimeType, 0.9); adds at least 1 second, without it the log shows the currIndex === 30 gets triggered every second. What would be the best approach to be able to capture at least about 30 FPS image. What is the fastest way to capture image from a HTML Canvas that it will not be the bottleneck of real-time video transcoding process?
You should probably revise your project, because saving the whole video as still images will blow out the memory of most devices in no time. Instead have a look at MediaStreams and MediaRecorder APIs, which are able to do the transcoding and compression in real time. You can request a MediaStream from a canvas through its captureStream() method.
The fastest is probably to send an ImageBitmap to your Worker thread, these are really fast to generate from a canvas (simple copy of the pixel buffer), and can be transferred to your worker script, from where you should be able to draw it on a an OffscreenCanvas.
Main drawback: it's currently only supported in latest Chrome and Firefox (through webgl), and this can't be polyfilled...
main.js
else {
console.log("Adding frame...");
const bitmap = await createImageBitmap(preview);
videoWorker.postMessage({ action: "addFrame", data: bitmap }, [bitmap]);
currIndex++;
}
worker.js
const canvas = new OffscreenCanvas(width,height);
const ctx = canvas.getContext('2d'); // Chrome only
onmessage = async (evt) => {
// ...
ctx.drawImage( evt.data.data, 0, 0 );
const image = await canvas.convertToBlob();
storeImage(image);
};
An other option is to transfer an ImageData data. Not as fast as an ImageBitmap, it still has the advantage of not stopping your main thread with the compression part and since it can be transferred, the message to the Worker isn't computation heavy either.
If you go this road, you may want to compress the data using something like pako (which uses the compression algorithm used by PNG images) from your Worker thread.
main.js
else {
console.log("Adding frame...");
const img_data = prevCtx.getImageData(0,0,width,height);
videoWorker.postMessage({ action: "addFrame", data: img_data }, [img_data.data]);
currIndex++;
}
worker.js
onmessage = (evt) => {
// ...
const image = pako.deflate(evt.data.data); // compress to store
storeImage(image);
};
Backgroud
I'm now processing on the client select image.
I want to do two actions on that image, and outputs the base64-encoded string.
If the image size has a width or height larger than 1000, resize it.
Compress the image with jpeg of quality 0.5.
So now I will do the below in the main script:
$(function() {
$('#upload').change(function() {
var URL = window.URL || window.webkitURL;
var imgURL = URL.createObjectURL(this.files[0]);
var img = new Image();
img.onload = function() {
var canvas = document.createElement('canvas');
var ctx = canvas.getContext('2d');
var w0 = img.width;
var h0 = img.height;
var w1 = Math.min(w0, 1000);
var h1 = h0 / w0 * w1;
canvas.width = w1;
canvas.height = h1;
ctx.drawImage(img, 0, 0, w0, h0,
0, 0, canvas.width, canvas.height);
// Here, the result is ready,
var data_src = canvas.toDataURL('image/jpeg', 0.5);
$('#img').attr('src', data_src);
URL.revokeObjectURL(imgURL);
};
img.src = imgURL;
});
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<input id="upload" type="file" accept="image/*" />
<img id="img" />
The Problem
But still, my code will work on a mobile, where the above procedure(resize and compress) can not work out soon. It causes the GUI stop response for a moment.
I want the procedure work in another thread, using web worker. So it won't block the UI, so the user experience would be better.
Now comes the problem, it seemed the web worker cannot deal with a canvas, how can I work around this?
Some Event driven coding
Saddly Web workers are not yet ready with browser support.
Limited support for toDataURL in web workers means another solution is needed. See MDN web worker APIs (ImageData) mid way down the page, only for Firefox at the moment.
Looking at your onload you have all the heavy duty work done in one blocking call to onload. You are blocking the UI during the process of creating the new canvas, getting its context, scaling, and toDataURL (don't know what revokeObjectURL does). You need to let the UI get a few calls in while this is happening. So a little event driven processing will help reduce the glitch if not make it unnoticeable.
Try rewriting the onload function as follows.
// have added some debugging code that would be useful to know if
// this does not solve the problem. Uncomment it and use it to see where
// the big delay is.
img.onload = function () {
var canvas, ctx, w, h, dataSrc, delay; // hosit vars just for readability as the following functions will close over them
// Just for the uninitiated in closure.
// var now, CPUProfile = []; // debug code
delay = 10; // 0 could work just as well and save you 20-30ms
function revokeObject() { // not sure what this does but playing it safe
// as an event.
// now = performance.now(); // debug code
URL.revokeObjectURL(imgURL);
//CPUProfile.push(performance.now()-now); // debug code
// setTimeout( function () { CPUProfile.forEach ( time => console.log(time)), 0);
}
function decodeImage() {
// now = performance.now(); // debug code
$('#img').attr('src', dataSrc);
setTimeout(revokeObject, delay); // gives the UI a second chance to get something done.
//CPUProfile.push(performance.now()-now); // debug code
}
function encodeImage() {
// now = performance.now(); // debug code
dataSrc = canvas.toDataURL('image/jpeg', 0.5);
setTimeout(decodeImage, delay); // gives the UI a second chance to get something done.
//CPUProfile.push(performance.now()-now); // debug code
}
function scaleImage() {
// now = performance.now(); // debug code
ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
setTimeout(encodeImage, delay); // gives the UI a second chance to get something done.
//CPUProfile.push(performance.now()-now); // debug code
}
// now = performance.now(); // debug code
canvas = document.createElement('canvas');
ctx = canvas.getContext('2d');
w = Math.min(img.width, 1000);
h = img.height / img.width * w;
canvas.width = w;
canvas.height = h;
setTimeout(scaleImage, delay); // gives the UI a chance to get something done.
//CPUProfile.push(performance.now()-now); // debug code
};
setTimeout allows the current call to exit, freeing up the call stack and allowing the UI to get its mitts on the DOM. I have given 10ms, personally I would start with 0ms as call stack access is not blocked, but I am playing it safe
With luck the problem will be greatly reduced. If it still remains an unacceptable delay then I can have a look at the CPU profile and see if a solution can not be found by targeting the bottle neck. My guess is the toDataURL is where the load is. If it is, a possible solution is to find a JPG encoder written in javascript that can be converted to an event driven encoder.
The problem is not how long it takes to process the data, but how long you block the UI.