I've made an app to run simple tfjs model on live camera in browser. The problem is that it kills the web page performance completely. The refresh rate is around 1-2fps and whole browser is laggy (other tabs, youtube movies, even system gui). What is strange, after the first model execution that is slow (model is compiled so it is understandable), model execution time is in range of 1.5-4ms (measured as in scrip below). So it should not be a problem for the browser.
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia({ video: true })
.then(function (stream) {
video!.srcObject = stream;
})
.catch(function (_) {
console.log("Something went wrong!");
});
}
let model = await tf.loadGraphModel('model/model.json')
let video = document.querySelector("#videoElement") as HTMLVideoElement;
async function do_inference()
{
let startTime = performance.now();
const image = tf.expandDims(tf.browser.fromPixels(video));
const image_fl32 = tf.cast(image, 'float32');
let final_pred = tf.tidy(() =>
{
return model.predict(image_fl32) as tf.Tensor4D;
});
image.dispose();
image_fl32.dispose();
final_pred.dispose();
let total = performance.now() - startTime;
console.log(total);
if (running)
{
requestAnimationFrame(do_inference);
}
}
Additional notes:
model is a very simple fully convolution model. Basically few layers of 3x3 and 5x5 convolutions. I also made a test with even smaller model (2 layers) and it has same problems.
the tfjs runs using webgl backend, changing it to wasm result in much greater execution time (around 150-200ms) but video is smooth. On production I would like to use even bigger model, so I cannot accept such execution time.
I've tested in on RTX 3060, 32 threads Ryzen, 128GB RAM. So PC should not be a problem
It seems that there are are no memory leaks on the tfjs side
the model is converted from tensorflow.keras.Model model using Model.save method and then using tensorflowjs_converter#3.9 with --output_format=tfjs_graph_model
Am I doing something wrong? Is there a way to make video smooth?
Related
I am currently trying to figure how to play chunked audio with the web audio API, right off the bat everything does work.. however most transitions between chunks aren't as smooth as I want them to be, there's a very very brief moment of silence between most of them.
My current loading and playback code:
const response = await fetch(`${this.src}`)
const reader = response.body.getReader()
let timestamptowaituntil = 0
let tolog = []
let tolog2 = []
while (true) {
const { done, value } = await reader.read()
if (done) {
console.log(tolog)
console.log(tolog2)
console.log(this.ctx)
break
} else {
let audiodata = await this.ctx.decodeAudioData(value.buffer)
let source = this.ctx.createBufferSource()
source.buffer = audiodata
source.connect(this.ctx.destination)
source.start(timestamptowaituntil, 0, audiodata.duration)
timestamptowaituntil +=audiodata.duration
tolog.push(audiodata)
tolog2.push(source)
}
}
How could I go about eliminating these little moments of silence (or overlap)?
Edit: So far I've tried the following
Removing some milliseconds off the waiting time.
Removing the amount of time that is in the latency properties of the AudioContext.
Making a function to get the playback length of the UInt8Array form data using its bitrate (this indeed got me a slightly different result than the .duration property of an audioBuffer, but there still is tiny gaps)
After trying a ton of different approaches, I finally got a thought that solved the issue in the end.
My new idea was to simply play the first chunk when it arrives, and meanwhile collect as many chunks as possible, whenever a chunk is collected, its chained with the previous chunk to make one bigger chunk (this way also makes it works in firefox which requires the chunk to have a header for decoding). The playback of the first chunk is stopped 0.5-1 second before the .duration property claims it would end, this way any anomalies in detecting length are avoided. At that same time, the next chunk is played.
A few things I added to my code for this is the following:
A function to concat two chunks:
const concat = (arrayOne, arrayTwo) => {
let mergedArray = new Uint8Array(arrayOne.length + arrayTwo.length)
mergedArray.set([...arrayOne, ...arrayTwo])
return mergedArray
}
Extra offset when timing:
source.start(timestamptowaituntil, 0, audiodata.duration - .75)
timestamptowaituntil += (audiodata.duration - .75 + this.ctx.currentTime)
This along with some more minor edits has brought me to a solution that makes the chunk-swap impossible to hear (every now and then it is when the cpu is overloaded and the timing slowed).
complete noob picking up NodeJS over the last few days here, and I've gotten myself in big trouble, it looks like. I've currently got a working Node JS+Express server instance, running on a Raspberry Pi, acting as a web interface for a local data acquisition script ("the DAQ"). When executed, the script writes out data to a local file on the Pi, in .csv format, writing out in real-time every second.
My Node app is a simple web interface to start (on-click) the data acquisition script, as well as to plot previously acquired data logs, and visualize the actively being collected data in real time. Plotting of old logs was simple, and I wrote a JS function (using Plotly + d3) to read a local csv file via AJAX call, and plot it - using this script as a starting point, but using the logs served by express rather than an external file.
When I went to translate this into a real-time plot, I started out using the setInterval() method to update the graph periodically, based on other examples. After dealing with a few unwanted recursion issues, and adjusting the interval to a more reasonable setting, I eliminated the memory/traffic issues which were crashing the browser after a minute or two, and things are mostly stable.
However, I need help with one thing primarily:
Improving the efficiency of my first attempt approach: This acquisition script absolutely needs to be written to file every second, but considering that a typical run might last 1-2 weeks, the file size being requested on every Interval loop will quickly start to balloon. I'm completely new to Node/Express, so I'm sure there's a much better way of doing the real-time rendering aspect of this - that's the real issue here. Any pointers of a better way to go about doing this would be massively helpful!
Right now, the killDAQ() call issued by the "Stop" button kills the underlying python process writing out the data to disk. Is there a way to hook into using that same button click to also terminate the setInterval() loop updating the graph? There's no need for it to be updated any longer after the data acquisition has been stopped so having the single click do double duty would be ideal. I think that setting up a listener or res/req approach would be an option, but pointers in the right direction would be massively helpful.
(Edit: I solved #2, using global window. variables. It's a hack, but it seems to work:
window.refreshIntervalId = setInterval(foo);
...
clearInterval(window.refreshIntervalId);
)
Thanks for much for the help!
MWE:
html (using Pug as a template engine):
doctype html
html
body.default
.container-fluid
.row
.col-md-5
.row.text-center
.col-md-6
button#start_button(type="button", onclick="makeCallToDAQ()") Start Acquisition
.col-md-6
button#stop_button(type="button", onclick="killDAQ()") Stop Acquisition
.col-md-7
#myDAQDiv(style='width: 980px; height: 500px;')
javascript (start/stop acquisition):
function makeCallToDAQ() {
fetch('/start_daq', {
// call to app to start the acquisition script
})
.then(console.log(dateTime))
.then(function(response) {
console.log(response)
setInterval(function(){ callPlotly(dateTime.concat('.csv')); }, 5000);
});
}
function killDAQ() {
fetch('/stop_daq')
// kills the process
.then(function(response) {
// Use the response sent here
alert('DAQ has stopped!')
})
}
javascript (call to Plotly for plotting):
function callPlotly(filename) {
var csv_filename = filename;
console.log(csv_filename)
function makeplot(csv_filename) {
// Read data via AJAX call and grab header names
var headerNames = [];
d3.csv(csv_filename, function(error, data) {
headerNames = d3.keys(data[0]);
processData(data, headerNames)
});
};
function processData(allRows, headerNames) {
// Plot data from relevant columns
var plotDiv = document.getElementById("plot");
var traces = [{
x: x,
y: y
}];
Plotly.newPlot('myDAQDiv', traces, plotting_options);
};
makeplot(filename);
}
node.js (the actual Node app):
// Start the DAQ
app.use(express.json());
var isDaqRunning = true;
var pythonPID = 0;
const { spawn } = require('child_process')
var process;
app.post('/start_daq', function(req, res) {
isDaqRunning = true;
// Call the python script here.
const process = spawn('python', ['../private/BIC_script.py', arg1, arg2])
pythonPID = process.pid;
process.stdout.on('data', (myData) => {
res.send("Done!")
})
process.stderr.on('data', (myErr) => {
// If anything gets written to stderr, it'll be in the myErr variable
})
res.status(200).send(); //.json(result);
})
// Stop the DAQ
app.get('/stop_daq', function(req, res) {
isDaqRunning = false;
process.on('close', (code, signal) => {
console.log(
`child process terminated due to receipt of signal ${signal}`);
});
// Send SIGTERM to process
process.kill('SIGTERM');
res.status(200).send();
})
Similar to this question:
Fetch API leaks memory in Chrome
When using fetch to regularly poll data, Chrome's memory usage continually increases without ever releasing the memory, which eventually causes a crash.
https://jsfiddle.net/abfinz/ahc65y3s/13/
const state = {
response: {},
count: 0
}
function poll(){
fetch('https://upload.wikimedia.org/wikipedia/commons/3/3d/LARGE_elevation.jpg')
.then(response => {
state.response = response;
state.count = state.count + 1;
if (state.count < 20){
console.log(state.count);
setTimeout(poll, 3000);
} else {
console.log("Will restart in 1 minute");
state.count = 0;
setTimeout(poll, 60000);
}
});
}
poll();
This JSFiddle demonstrates the issue fairly well. By polling for data every 3 seconds, it seems that something is causing Chrome to continually hold onto the memory. If I let it stop and wait for a few minutes, it usually will release the memory, but if polling continues, it usually holds onto it. Also, if I let it run for a few full cycles, even forcing garbage collection from the perfomance tab of the dev tools doesn't always release all of the memory.
The memory doesn't show up in the JS Heap. I have to use the Task Manager to see it.
Occasionally, the memory will clear while actively polling, but inevitably builds to extreme levels.
Edge also shows the issue, but seems to be more proactive in clearing the memory. Though it still eventually builds to 1GB+ of extra memory usage.
Am I doing something wrong, or is this a bug? Any ideas on how I can get this kind of polling to work long-term without the memory leak?
I played around a bit with it and it seems to be a bug with the handling of the response so that it won't free the allocated memory if you are not calling any of the response functions.
The chrome task manager and the windows task manager report the same size of 30 MB constantly if i start the code snippet here using this order of execution. Meanwhile it runs on jsfiddle too with 30 MB on request #120.
const state = {
response: {},
count: 0
},
uri = 'https://upload.wikimedia.org/wikipedia/commons/3/3d/LARGE_elevation.jpg';
!function poll() {
const controller = new AbortController(),
signal = controller.signal;
// using this you can cancel it and destroy it completly.
fetch(uri, { signal })
// this is triggered as soon as the header data is transfered
.then(response => {
/**
* Continung without doing anything on response
* fills the memory.
*
* Chrome downloads the file in the background and
* seems to wait for the use of a call on the
* response.fx() or an abort signal.
*
* This seems to be a bug or a small design mistake
* if response is never used.
*
* If response.json(), .blob(), .body() or .text() is
* called the memory will be free'd.
*/
return response.blob();
}).then((binary) => {
// store data to a var
return state.response = binary;
}).catch((err) => {
console.error(err);
}).finally(() => {
// and start the next poll
console.log(++state.count, state.response.type, (state.response.size / 1024 / 1024).toFixed(2)+' MB');
requestAnimationFrame(poll);
// console.dir(state.response);
// reduces memory a bit more
controller.abort();
})
}()
Safari on iOS puts a scrubber on its lock screen for simple HTMLAudioElements. For example:
const a = new Audio();
a.src = 'https://example.com/audio.m4a'
a.play();
JSFiddle: https://jsfiddle.net/0seckLfd/
The lock screen will allow me to choose a position in the currently playing audio file.
How can I disable the ability for the user to scrub the file on the lock screen? The metadata showing is fine, and being able to pause/play is also acceptable, but I'm also fine with disabling it all if I need to.
DISABLE Player on lock screen completely
if you want to completely remove the lock screen player you could do something like
const a = new Audio();
document.querySelector('button').addEventListener('click', (e) => {
a.src = 'http://sprott.physics.wisc.edu/wop/sounds/Bicycle%20Race-Full.m4a'
a.play();
});
document.addEventListener('visibilitychange', () => {
if (document.hidden) a.src = undefined
})
https://jsfiddle.net/5s8c9eL0/3/
that is stoping the player when changing tab or locking screen
(code to be cleaned improved depending on your needs)
From my understanding you can't block/hide the scrubbing commands unless you can tag the audio as a live stream. That being said, you can use js to refuse scrubbing server-side. Reference the answer here. Although that answer speaks of video, it also works with audio.
The lock screen / control center scrubber can also be avoided by using Web Audio API.
This is an example of preloading a sound and playing it, with commentary and error handling:
try {
// <audio> element is simpler for sound effects,
// but in iOS/iPad it shows up in the Control Center, as if it's music you'd want to play/pause/etc.
// Also, on subsequent plays, it only plays part of the sound.
// And Web Audio API is better for playing sound effects anyway because it can play a sound overlapping with itself, without maintaining a pool of <audio> elements.
window.audioContext = window.audioContext || new AudioContext(); // Interoperate with other things using Web Audio API, assuming they use the same global & pattern.
const audio_buffer_promise =
fetch("audio/sound.wav")
.then(response => response.arrayBuffer())
.then(array_buffer => audioContext.decodeAudioData(array_buffer))
var play_sound = async function () {
audioContext.resume(); // in case it was not allowed to start until a user interaction
// Note that this should be before waiting for the audio buffer,
// so that it works the first time (it would no longer be "within a user gesture")
// This only works if play_sound is called during a user gesture (at least once), otherwise audioContext.resume(); needs to be called externally.
const audio_buffer = await audio_buffer_promise; // Promises can be awaited any number of times. This waits for the fetch the first time, and is instant the next time.
// Note that if the fetch failed, it will not retry. One could instead rely on HTTP caching and just fetch() each time, but that would be a little less efficient as it would need to decode the audio file each time, so the best option might be custom caching with request error handling.
const source = audioContext.createBufferSource();
source.buffer = audio_buffer;
source.connect(audioContext.destination);
source.start();
};
} catch (error) {
console.log("AudioContext not supported", error);
play_sound = function() {
// no-op
// console.log("SFX disabled because AudioContext setup failed.");
};
}
I did a search, in search of a way to help you, but I did not find an effective way to disable the commands, however, I found a way to customize them, it may help you, follow the apple tutorial link
I think what's left to do now is wait, see if ios 13 will bring some option that will do what you want.
Safari on iOS puts a scrubber on its lock screen for simple HTMLAudioElements. For example:
const a = new Audio();
a.src = 'https://example.com/audio.m4a'
a.play();
JSFiddle: https://jsfiddle.net/0seckLfd/
The lock screen will allow me to choose a position in the currently playing audio file.
How can I disable the ability for the user to scrub the file on the lock screen? The metadata showing is fine, and being able to pause/play is also acceptable, but I'm also fine with disabling it all if I need to.
DISABLE Player on lock screen completely
if you want to completely remove the lock screen player you could do something like
const a = new Audio();
document.querySelector('button').addEventListener('click', (e) => {
a.src = 'http://sprott.physics.wisc.edu/wop/sounds/Bicycle%20Race-Full.m4a'
a.play();
});
document.addEventListener('visibilitychange', () => {
if (document.hidden) a.src = undefined
})
https://jsfiddle.net/5s8c9eL0/3/
that is stoping the player when changing tab or locking screen
(code to be cleaned improved depending on your needs)
From my understanding you can't block/hide the scrubbing commands unless you can tag the audio as a live stream. That being said, you can use js to refuse scrubbing server-side. Reference the answer here. Although that answer speaks of video, it also works with audio.
The lock screen / control center scrubber can also be avoided by using Web Audio API.
This is an example of preloading a sound and playing it, with commentary and error handling:
try {
// <audio> element is simpler for sound effects,
// but in iOS/iPad it shows up in the Control Center, as if it's music you'd want to play/pause/etc.
// Also, on subsequent plays, it only plays part of the sound.
// And Web Audio API is better for playing sound effects anyway because it can play a sound overlapping with itself, without maintaining a pool of <audio> elements.
window.audioContext = window.audioContext || new AudioContext(); // Interoperate with other things using Web Audio API, assuming they use the same global & pattern.
const audio_buffer_promise =
fetch("audio/sound.wav")
.then(response => response.arrayBuffer())
.then(array_buffer => audioContext.decodeAudioData(array_buffer))
var play_sound = async function () {
audioContext.resume(); // in case it was not allowed to start until a user interaction
// Note that this should be before waiting for the audio buffer,
// so that it works the first time (it would no longer be "within a user gesture")
// This only works if play_sound is called during a user gesture (at least once), otherwise audioContext.resume(); needs to be called externally.
const audio_buffer = await audio_buffer_promise; // Promises can be awaited any number of times. This waits for the fetch the first time, and is instant the next time.
// Note that if the fetch failed, it will not retry. One could instead rely on HTTP caching and just fetch() each time, but that would be a little less efficient as it would need to decode the audio file each time, so the best option might be custom caching with request error handling.
const source = audioContext.createBufferSource();
source.buffer = audio_buffer;
source.connect(audioContext.destination);
source.start();
};
} catch (error) {
console.log("AudioContext not supported", error);
play_sound = function() {
// no-op
// console.log("SFX disabled because AudioContext setup failed.");
};
}
I did a search, in search of a way to help you, but I did not find an effective way to disable the commands, however, I found a way to customize them, it may help you, follow the apple tutorial link
I think what's left to do now is wait, see if ios 13 will bring some option that will do what you want.