Develop Reference

How to detect when mjpeg stream stops with javascript

I have a html/javascript client that is listening to a mjpeg video stream:
myImg = document.getElementById('my-image');
myImg.src = 'http://myserver.com/camera.mjpeg';
Works fine but if the video stream dies for whatever reason the video feed "freezes" on the last received image and I have no opportunity to display an error to the user. I've see this post that offers a solution (creating a long running ajax request alongside the stream) that only works some of the time. I was hoping there would be a more supported method like through a disconnect event or something.
Even an event for when data is received would be better than nothing. At least that way I could tell if it's been a while since a frame came through. Using addEventListener('load') only works on the very first frame.
Any ideas?
Update:
Based on comments I have tried the following approaches, none of which has worked:
myImg.addEventListener('error', event => { ... });
myImg.addEventListener('stalled', event => { ... });
myImg.addEventListener('suspend', event => { ... });

This is common with a normal implementation of a mjpeg, for example
<video src="http://myserver.com/camera.mjpeg" controls>
Your browser does not support the <code>video</code> element.
</video>
the mjpeg is a series of images and eventually it will not get the next one for whatever reason, breaking the connection. (this is sometimes because the source is cached, causing the browser to use the last image every time). I don't consider this an error, more something to program around with mjpeg streams.
A simple solution you can do, set a refresh rate and set the src continuously refreshing the connection every ~500ms (or less depending on your network connection/resources).
setInterval(function() {
var myImg = document.getElementById('myImg');
myImg.src = 'http://myserver.com/camera.mjpeg?rand=' + Math.random();
}, 5000);
The random number is added to prevent browser side caching in the event the server sends those headers.
Or you can create a ReadableStream, and keep reading a blob of bytes directly into the source of the image. There is a robust example in this repo, from this other question.

In Safari document.readyState will change from interactive to complete.
For example put this before the image loads:
<script>
console.log('Initial ready state', document.readyState);
document.onreadystatechange = function() {
console.log('Ready state changed to:', document.readyState);
}
</script>
And the output will be:
Initial ready state – "loading"
Ready state changed to: – "interactive"
// When the connection disconnects:
Ready state changed to: – "complete"
In google chrome the readyState doesn't stay on interactive, but it looks like chrome is better at reconnecting, so might not be an issue for you.
Edit: One way to make use of this is to drop the image in an iframe, you'll continually get load events in safari (this does not work in chrome).
iframe = document.createElement('iframe')
iframe.onload = console.log
iframe.src = "http://10.0.0.119:8080/stream"
document.body.append(iframe)
Edit2: Another technique -- use image.decode to detect when the connection is down and reload the image:
<img id="stream" src="http://10.0.0.119:8080/stream"></img>
<script>
let image = document.getElementById('stream');
async function check() {
while (true) {
try {
await image.decode();
} catch {
let src = image.src;
image.src = "";
image.src = src;
}
await new Promise((resolve) => setTimeout(resolve, 5000));
}
}
check();
</script>

Would something like this work?
function hasLoaded(myImg) {
return myImg.complete && myImg.naturalHeight !== 0;
}

Following Beau Bouchard answers.
The setInterval timer, works fine but it tends to max out active client listening. ( if ur mjpeg stream are coming directly from an IP Camera). Could possibly create a restreaming server the mjpeg server to allow more clients to be able to be listening to it) Short pooling though does tend to be very resource heavy.
Tried the Restream Api as well. When loading the image back into the img tag, you do get a jittery effect, most likely because the chunks are coming in randomly and not smooth out via time?
In the end, i use the onload img tag event. This triggers whenever an img is loaded. Then a time interval to check if the img tag has stop loading to determine if the mjpeg stream has stop.

Met the same requirement, test on Image onload event and works!!
If FFMPEG feed FFSERVER stop, although the MJPEG in a still image,
After couple times error count, mjpeg FAIL! detected.
<!DOCTYPE HTML>
<HTML>
<HEAD>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<TITLE> mjpeg detect </TITLE>
<script type="text/javascript" language="JavaScript" src="/js/jquery.js"></script>
<script type="text/javascript" language="JavaScript">
//------------------------------------------------------------------------------
// localhost/tool/mjpeg.htm
// document.ready
$(function(){
setTimeout("mjpegRefresh()", 10000);
});
var mTmjpegRefresh, mBmjpegStatus=0, mNmjpegError=0;
var mjpegRefresh = function()
{
clearTimeout(mTmjpegRefresh);
mBmjpegStatus=0;
$('#myMJPEG').attr('src', "http://192.168.1.17:8090/live.mjpeg?rand=" + Math.random());
console.log("mjpeg refresh: ", Math.round( (new Date()).getTime()/1000)) ;
mTmjpegRefresh = setTimeout("mjpegRefresh()", 10000);
mTmjpegStatusCheck = setTimeout("mjpegStatusCheck()", 5000);
}
var mjpegOnload = function()
{
console.log("mjpeg Onload");
mBmjpegStatus=1;
}
var mTmjpegStatusCheck;
var mjpegStatusCheck = function()
{
clearTimeout(mTmjpegStatusCheck);
if(mBmjpegStatus>0)
{
mNmjpegError=0;
}
else
{
mNmjpegError++;
}
if(mNmjpegError>5)
{
console.log("mjpeg FAIL!");
}
mTmjpegStatusCheck = setTimeout("mjpegStatusCheck()", 5000);
}
//------------------------------------------------------------------------------
</script>
</HEAD>
<BODY>
<img src="http://192.168.1.17:8090/live.mjpeg" width="720" height="404" id="myMJPEG" onload="mjpegOnload()">
</BODY>
</HTML>

Javascript Workers - why is the worker message treated so lately and can I do something against it?

I have a Worker that shares a SharedArrayBuffer with the "main thread". To work correctly, I have to make sure that the worker has access to the SAB before the main thread accesses to it. (EDIT: The code creating the worker has to be in a seperate function (EDIT2: which returns an array pointing to the SAB).) (Maybe, already this is not possible, you'll tell me).
The initial code looks like this:
function init() {
var code = `onmessage = function(event) {
console.log('starting');
var buffer=event.data;
var arr = new Uint32Array(buffer);// I need to have this done before accessing the buffer again from the main
//some other code, manipulating the array
}`
var buffer = new SharedArrayBuffer(BUFFER_ELEMENT_SIZE);
var blob = new Blob([code], { "type": 'application/javascript' });
var url = window.URL || window.webkitURL;
var blobUrl = url.createObjectURL(blob);
var counter = new Worker(blobUrl);
counter.postMessage(buffer);
let res = new Uint32Array(buffer);
return res;
}
function test (){
let array = init();
console.log('main');
//accessing the SAB again
};
The worker code is always executed after test(), the console shows always main, then starting.
Using timeouts does not help. Consider the following code for test:
function test (){
let array = [];
console.log('main');
setTimeout(function(){
array = initSAB();
},0);
setTimeout(function(){
console.log('main');
//accessing the SAB again
},0);
console.log('end');
};
The console shows end first, followed by main, followed by starting.
However, assigning the buffer to a global array outside the test() function does the job, even without timeouts.
My questions are the following:
why does the worker does not start directly after the message was send (= received?). AFAIK, workers have their own event queue, so they should not rely on the main stack becoming empty?
Is there a specification detailing when a worker starts working after sending a message?
Is there a way to make sure the worker has started before accessing the SAB again without using global variables? (One could use busy waiting, but I beware...) There is probably no way, but I want to be sure.
Edit
To be more precise:
In a completly parallel running scenario, the Worker would be able to
handle the message immediately after it was posted. This is obviously
not the case.
Most Browser API (and Worker is such an API) use a callback queue to handle calls to the API. But if this applied, the message would be
posted/handled before the timeout calbacks were executed.
To go even further: If I try busy waiting after postMessage by reading from the SAB until it changes one value will block the
program infinitely. For me, it means that the Browser does
not posts the message until the call stack is empty As far as
I know, this behaviour is not documentated and I cannot explain it.
To summerize: I want to know how the browser determines when to post the message and to handle it by the worker, if the call of postMessage is inside a function. I already found a workaround (global variables), so I'm more interested in how it works behind the scenes. But if someone can show me a working example, I'll take it.
EDIT 2:
The code using the global variable (the code that works fine) looks like this
function init() {
//Unchanged
}
var array = init(); //global
function test (){
console.log('main');
//accessing the SAB again
};
It prints starting, then main to the console.
What is also worth noticing : If I debug the code with the Firefox Browser (Chrome not tested) I get the result I want without the global variable (starting before main) Can someone explain?

why does the worker does not start directly after the message was sen[t] (= received?). AFAIK, workers have their own event queue, so they should not rely on the main stack becoming empty?
First, even though your Worker object is available in main thread synchronously, in the actual worker thread there are a lot of things to do before being able to handle your message:
it has to perform a network request to retrieve the script content. Even with a blobURI, it's an async operation.
it has to initialize the whole js context, so even if the network request was lightning fast, this would add up on parallel execution time.
it has to wait the event loop frame following the main script execution to handle your message. Even if the initialization was lightning fast, it will anyway wait some time.
So in normal circumstances, there is very little chances that your Worker could execute your code at the time you require the data.
Now you talked about blocking the main thread.
If I try busy waiting after postMessage by reading from the SAB until it changes one value will block the program infinitely
During the initialization of your Worker, the message are temporarily being kept on the main thread, in what is called the outside port. It's only after the fetching of the script is done that this outside port is entangled with the inside port, and that the messages actually pass to that parallel thread.
So if you do block the main thread before the ports have been entangled it won't be able to pass it to the worker's thread.
Is there a specification detailing when a worker starts working after sending a message?
Sure, and more specifically, the port message queue is enabled at the step 26, and the Event loop is actually started at the step 29.
Is there a way to make sure the worker has started before accessing the SAB again without using global variables? [...]
Sure, make your Worker post a message to the main thread when it did.
// some precautions because all browsers still haven't reenabled SharedArrayBuffers
const has_shared_array_buffer = window.SharedArrayBuffer;
function init() {
// since our worker will do only a single operation
// we can Promisify it
// if we were to use it for more than a single task,
// we could promisify each task by using a MessagePort
return new Promise((resolve, reject) => {
const code = `
onmessage = function(event) {
console.log('hi');
var buffer= event.data;
var arr = new Uint32Array(buffer);
arr.fill(255);
if(self.SharedArrayBuffer) {
postMessage("done");
}
else {
postMessage(buffer, [buffer]);
}
}`
let buffer = has_shared_array_buffer ? new SharedArrayBuffer(16) : new ArrayBuffer(16);
const blob = new Blob([code], { "type": 'application/javascript' });
const blobUrl = URL.createObjectURL(blob);
const counter = new Worker(blobUrl);
counter.onmessage = e => {
if(!has_shared_array_buffer) {
buffer = e.data;
}
const res = new Uint32Array(buffer);
resolve(res);
};
counter.onerror = reject;
if(has_shared_array_buffer) {
counter.postMessage(buffer);
}
else {
counter.postMessage(buffer, [buffer]);
}
});
};
async function test (){
let array = await init();
//accessing the SAB again
console.log(array);
};
test().catch(console.error);

According to MDN:
Data passed between the main page and workers is copied, not shared. Objects are serialized as they're handed to the worker, and subsequently, de-serialized on the other end. The page and worker do not share the same instance, so the end result is that a duplicate is created on each end. Most browsers implement this feature as structured cloning.
Read more about transferring data to and from workers
Here's a basic code that shares a buffer with a worker. It creates an array with even values (i*2) and it sends it to the worker. It uses Atomic operations to change the buffer values.
To make sure the worker has started you can just use different messages.
var code = document.querySelector('[type="javascript/worker"]').textContent;
var blob = new Blob([code], { "type": 'application/javascript' });
var blobUrl = URL.createObjectURL(blob);
var counter = new Worker(blobUrl);
var sab;
var initBuffer = function (msg) {
sab = new SharedArrayBuffer(16);
counter.postMessage({
init: true,
msg: msg,
buffer: sab
});
};
var editArray = function () {
var res = new Int32Array(sab);
for (let i = 0; i < 4; i++) {
Atomics.store(res, i, i*2);
}
console.log('Array edited', res);
};
initBuffer('Init buffer and start worker');
counter.onmessage = function(event) {
console.log(event.data.msg);
if (event.data.edit) {
editArray();
// share new buffer with worker
counter.postMessage({buffer: sab});
// end worker
counter.postMessage({end: true});
}
};
<script type="javascript/worker">
var sab;
self['onmessage'] = function(event) {
if (event.data.init) {
postMessage({msg: event.data.msg, edit: true});
}
if (event.data.buffer) {
sab = event.data.buffer;
var sharedArray = new Int32Array(sab);
postMessage({msg: 'Shared Array: '+sharedArray});
}
if (event.data.end) {
postMessage({msg: 'Time to rest'});
}
};
</script>

Javascript: Alternative to setTimeOut for FAST Timer in MIDI Sequencer App

I'm working on a Javascript Music App that includes a Sequencer. For those who are not familiar, MIDI sequencers work pretty much like this: There is something called PPQ: pulses per quarter note. Each pulse is called "Tick". It depicts how may "subdivisions" there are per quarter note, like resolution. So Sequencers "play" the Events that are in the tracks one Tick at a time: Play Tick1, wait Tick Duration, Play tick2, Tick Duration, and so on.
Now, let's say we have a BPM (Beats per Min) of 120 with PPQ=96 (standard). That means that each Quarter Note Duration is 500ms, and each Tick Duration is 5.20833ms.
What Timer Alternatives we have in Javascript?
1) We have the old setTimeOut. It has several problems: the min. wait time is 4ms. (https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout#Minimum_delay_and_timeout_nesting)
It is also subject to JITTER/time Variations. It is not precise and it is demanding, as call backs are stacked in the even loop.
2) There is an alternative to setTimeOut/setInterval which involves using requestAnimationFrame(). It is VERY precise and CPU efficient. However, the minimum time it can be set is around 16.7ms (the duration of a Frame in a typical 60FPS monitor)
Is there any other Alternative? To to precisely schedule an event every 2-5ms?
Note: the function done in side the loop, playEventsAtTick() is NOT demanding at all, so it would never take more time to execute than Tick Duration.
Thanks!
Danny Bullo

To maintain any sanity in doing this kind of thing, you're going to want to do the audio processing on a devoted thread. Better yet, use the Web Audio API and let people who have been thinking about these problems for a long time do the hard work of sample-accuracy.
Also check out Web MIDI (chrome only).

Thanks nvioli. I'm aware of Web Audio API. However, I don't think that can help here.
I'm not triggering AUDIO directly: I have MIDI events (or let's say just "EVENTS") stored in the TRACKS. And those events happen at any TICK. So the Sequencer needs to loop every Tick Duration to scan what to play at that particular tick.
Regards,
Danny Bullo

In a separate thread, such as a web worker, you can create an endless loop. In this loop, all you need to do is calculate the time between beats. After the time is valid, you can then send a message to the main process, to do some visuals, play a sound or what ever you would like to do.
Here is a Working example
class MyWorker {
constructor() {
// Keeps the loop running
this.run = true
// Beats per minute
this.bpm = 120
// Time last beat was called
this.lastLoopTime = this.milliseconds
}
get milliseconds() {
return new Date().getTime()
}
start() {
while (this.run) {
// Get the current time
let now = this.milliseconds
// Get the elapsed time between now and the last beat
let updateLength = now - this.lastLoopTime
// If not enough time has passed restart from the beginning of the loop
if (updateLength < (1000 * 60) / this.bpm) continue;
// Enough time has passed update the last time
this.lastLoopTime = now
// Do any processing that you would like here
// Send a message back to the main thread
postMessage({ msg: 'beat', time: now })
}
}
}
new MyWorker().start()
Next we can create the index page, which will run the worker, and flash a square everytime a message comes back from the worker.
<!DOCTYPE html>
<html lang="en">
<head>
<script>
// Start the worker
var myWorker = new Worker('worker.js')
// Listen for messages from the worker
myWorker.onmessage = function (e) {
var msg = e.data
switch (msg.msg) {
// If the message is a `beat` message, flash the square
case 'beat':
let div = document.querySelector('div')
div.classList.add('red')
setTimeout(() => div.classList.remove('red'), 100)
break;
}
}
</script>
<style>
div { width: 100px; height: 100px; border: solid 1px; }
.red { background: red; }
</style>
</head>
<body>
<div></div>
</body>
</html>

Get Off My Lawn: The approach you suggested does not completely work. Let's say I add a method to the web worker to STOP the Sequencer:
stop() {
this.run = false;
}
The problem is that the method myWorker.onmessage = function (e) {...} never get's triggered. I suspect it is because the Web Worker Thread is "TOO BUSY" with the endless loop. any way to solve that?
Also, while playing, it works.....but the CPU goes up considerably..... The only possible Solution would be a Sleep() method, but Real SLEEP that does not exist in Javascript...
Thanks

The JavaScript Event Loop and Web Workers

So I've been having a long talk with a colleague regarding the JavaScript event loop and the use of Web Workers. In a single Web page, different Web Workers have different stacks, heaps, and message queues, form here, specifically:
A web worker or a cross-origin iframe has its own stack, heap, and message
queue. Two distinct runtimes can only communicate through sending messages via
the postMessage method. This method adds a message to the other runtime if the
latter listens to message events.
but are all the messages executed inside the same event loop, or does each Web Worker have its own event loop?
I'm asking this because I have two Web Workers in a page, one executes a very computationally-heavy operation in sequence, while the other just handles a WebRTC connection.
I will not go into details but it seems to me that the computationally-heavy Web Worker is taking away so much computational time out of the JavaScript event loop that the other Worker, that only has to keep the connection alive (through heartbeat I suppose) isn't able to do so, and the connection is eventually lost.
This is what I believe. If that is not the case, and the two Web Workers work on different event loops then I cannot explain why the connection is lost when the load on the computing Web Worker is heavy (when the load is light then the connection is not lost).

Each worker has its own event loop. From the specification:
Each WorkerGlobalScope object has a distinct event loop, separate from those used by units of related similar-origin browsing contexts.
and then here:
The global scope is the "inside" of a worker.
...which is followed by the definition of the WorkerGlobalScope interface referenced in the earlier quote.
Your computation-heavy worker might be dominating the available processing time, but it isn't blocking the other worker's event loop.
We can also readily check this with a quick test:
page.html:
<!DOCTYPE HTML "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta charset="UTF-8">
<title>Two Workers</title>
<style>
body {
font-family: sans-serif;
}
pre {
margin: 0;
padding: 0;
}
</style>
</head>
<body>
<div>Fast: <span id="fast"></span></div>
<div>Slow: <span id="slow"></span></div>
<script>
(function() {
var fastWorker = new Worker("fastworker.js");
var fast = document.getElementById("fast");
var slowWorker = new Worker("slowworker.js");
var slow = document.getElementById("slow");
fastWorker.addEventListener("message", function(e) {
fast.innerHTML = e.data || "??";
fastWorker.postMessage("ping");
});
slowWorker.addEventListener("message", function(e) {
slow.innerHTML = e.data || "??";
slowWorker.postMessage("ping");
});
fastWorker.postMessage("start");
slowWorker.postMessage("start");
})();
</script>
</body>
</html>
slowworker.js:
var counter = 0;
self.addEventListener("message", function(e) {
var done = Date.now() + 1000; // 1 second
while (Date.now() < done) {
// Busy wait (boo!)
}
++counter;
self.postMessage(counter);
});
fastworker.js:
var counter = 0;
self.addEventListener("message", function(e) {
var done = Date.now() + 100; // 100ms
while (Date.now() < done) {
// Busy wait (boo!)
}
++counter;
self.postMessage(counter);
});
As you can see, "fast"'s number goes up much more quickly than "slow", showing it's processing its messages.
(I could have made one worker file and sent the delay in the start command, but...)

HTML5 Voice Recognition, wait til user answered

I'm playing arround with the HTML5 voice recognition.
Currently I have a function like this:
doSomething() {
listen("name");
console.log("done");
}
The "listen" Function works currently like this:
recognition = new webkitSpeechRecognition();
recognition.lang = "de-DE";
recognition.continuous = false;
//recognition.interimResults = true;
recognition.onresult = function(event) {
result = event.results[event.resultIndex];
confidence = result[0].confidence;
result = result[0].transcript.trim();
};
//TODO: remove old results, work with results
recognition.start();
What is happening is that Chrome asks for the microphone access and directly does the console.log.
What I want is for the console.log to wait until the speech recognition is done. Like this:
Chrome asks for mic access
User says something
Something is done with what the user said
the console.log and everything that follows will be executed.
How can I do that?
Thank you!

Javascript programming is event-driven. The code is not a sequence of statements to execute, but just a description of events to handle and reactions on them.
If you want to perform some action on speech recognized, you need to put it into even handler, in your case:
recognition.onresult = function(event) {
result = event.results[event.resultIndex];
confidence = result[0].confidence;
result = result[0].transcript.trim();
console.log("done")
};
You can access variables inside handler function and do more complex things.
There are many explanations of event-driven programming on the web, but the most complete one is Chapter 17 Handling Events of JavaScript: The Definitive Guide, 6th Edition