Develop Reference

Is there a way to reduce latency using getUserMedia?

While trying to reduce the video latency for a WebRTC communication, I measured the delay between the video capture and the video display.
To prevent measuring latency involved by WebRTC, I just used getUserMedia and an HTML video that displayed the stream.
I did it by displaying a timestamp every frame (using requestAnimationFrame), recording my screen with a USB camera and taking screenshots where both the video display and the displayed timestamp where visible.
On average, I measured a delay of ~150ms.
This must be an overestimation (due to requestAnimationFrame time between calls), however the minimum measure I made is around 120ms, that still a lot.
Now, is there a way to reduce this delay between the video capture and the video display ?
Note:
I tried using another video lector (window's built-in lector), and the measure were really close (average delay about 145ms)
I tried another video device (my laptop webcam with a mirror), and the results are less close but still elevated, on my opinion (average delay about 120ms)

In general this is something you can only fix in the browser itself.
The requestVideoFrameCallback API is gathering some numbers such as captureTime and renderTime. https://web.dev/requestvideoframecallback-rvfc/ has a pretty good description, https://webrtc.github.io/samples/src/content/peerconnection/per-frame-callback/ visualizes them.

HTML5 video: determine the intended framerate (fps) of a video object?

If I have a HTML5 video object, I can get its properties such as duration. Is there also a way to get its intended framerate? (i.e. frames per second)
Note that I don't mean the actual playback rate that is attained by the browser when playing it, but the video's own native target framerate.
Or alternatively, is there a way to get the total framecount in the entire video? The number of frames divided by the duration would be what I need.

Syncronizing Web Audio API and HTML5 video

I'm trying to synchronize an audio track being played via Web Audio API with a video being played in an HTML5 video element. Using a fibre optic synchronization device and audacity we can detect the drift between audio and video signals to a very high degree of accuracy.
I've tried detection the drift between the two sources and correcting it by either accelerating or decelerating the audio and as below just simply setting the video to the same position as the audio.
Play(){
//...Play logic
//Used to calculate when the track started playing, set when play is triggered.
startTime = audioContext.currentTime;
}
Loop(){
let audioCurrentTime = startTime - audioContext.currentTime;
if(videoElement.nativeElement.currentTime - audioCurrentTime > 0.1 ){
videoElement.nativeElement.currentTime = audioCurrentTime;
}
requestAnimationFrame(Loop);
}
With all of this, we still notice a variable drift between the two sources of around 40ms. I've come to believe that audioContext.currentTime does not report back accurately since when stepping through the code multiple loops will report back the same time even though quite obviously time has passed. My guess is the time being reported is the amount of the track that has been passed to some internal buffer. Is there another way to get a more accurate playback position from an audio source being played?
Edit: I've updated the code to be a little closer to the actual source. I set the time at which the play was initialized and compare that to the current time to see the track position. This still reports a time that is not an accurate playback position.

Decode audio and play intro then looped part

I have a song, consisting of intro (I), a part to be looped (L), and ending (E). I don't want the ending to be played at all, i.e. audio file song = I + L + E, audio played = I + L + L + ...
I succeeded to do that by separating the intro and the loop into two files, but I want to do that "on-the-fly" on the client side.
How do I do that?

Web Audio API provides AudioBufferSourceNode.loopStart and AudioBufferSourceNode.loopEnd (source) precisely for that. You have to remember to set AudioBufferSourceNode.loop = true also.

If you want to play the file using Web Audio (e.g. decode into an AudioBuffer using decodeAudioData() and then play the sound with BufferSourceNodes), then it's easy to do by pointing two BufferSource Nodes at the buffer with different offsets, and looping the second one.
Web Audio uses doubles (not floats) - so the statement "this is much more accurate than float seconds" isn't generally true (doubles have around 15 decimal places of accuracy). (Depends on the absolute value, of course.) It's more than accurate enough to play with sample-accuracy (i.e. to not glitch between the values); if you're concerned, though, just cross-fade between them.
If your goal is really just to produce an audio file with that part looped, that's a little different; but it sounds like you want playback.

Video does not play through even if enough content has been appended

I have a setup where I send a 10min long video (Elephants Dream) using the websockets protocol chunked in short segments of 4s each.
I use the browser as client, with the Websocket API to receive the content and the HTML5 Video Tag as player, to which I append the chunks as they come to the video using Media Source Extensions.
The thing is that there seems to be a limit somewhere (max receive buffer size, max mediasource sourcebuffer buffer size, max buffered content on video element, etc) so that the video does not play correctly to the end but stops earlier even if there is enough data.
All of the segments are arriving correctly and get appended in time. At the same time, the video starts playing back from the beginning.
You can see the grey line on the player showing buffered video grow until at some point in time where it stops growing and the video stops playing when getting to this position.
However, the full video has been appended to the mediasource element, regarding to the output messages, and which can also be tested by manually jumping to another position in future or past. It looks like there is always just a fraction of the content "loaded".
Since I'm testing it on localhost the throughput is very high so I tried lowering this to more common values (still good over video bitrate) to see if I'm overloading the client but this did not change anything.
Also tried different segment sizes, with exact same results, except for that the time in point where it stops is a different one.
Any idea on where this limitation can be or what may be happening?

I think you have a gap in the buffered data. Browsers have a limited buffer size to which you can append. When that limit is reached, if you append additional data, the browser will silently free some space by discarding some frames it does not need from the buffer. In my experience, if you happen too fast, you may end up with gaps in your buffer. You should monitor the status of the buffered attribute when appending to see if there is any gap.

Are you changing representations right before it stops? When you change representations, you need to append the init segment for the new representation before you append the next segment of the new representation.