Develop Reference

How to use MediaRecorder as MediaSource

As an exercise in learning WebRTC I am trying to to show the local webcam and side by side with a delayed playback of the webcam. In order to achieve this I am trying to pass recorded blobs to a BufferSource and use the corresponding MediaSource as source for a video element.
// the ondataavailable callback for the MediaRecorder
async function handleDataAvailable(event) {
// console.log("handleDataAvailable", event);
if (event.data && event.data.size > 0) {
recordedBlobs.push(event.data);
}
if (recordedBlobs.length > 5) {
if (recordedBlobs.length === 5)
console.log("buffered enough for delayed playback");
if (!updatingBuffer) {
updatingBuffer = true;
const bufferedBlob = recordedBlobs.shift();
const bufferedAsArrayBuffer = await bufferedBlob.arrayBuffer();
if (!sourceBuffer.updating) {
console.log("appending to buffer");
sourceBuffer.appendBuffer(bufferedAsArrayBuffer);
} else {
console.warn("Buffer still updating... ");
recordedBlobs.unshift(bufferedBlob);
}
}
}
}
// connecting the media source to the video element
recordedVideo.src = null;
recordedVideo.srcObject = null;
recordedVideo.src = window.URL.createObjectURL(mediaSource);
recordedVideo.controls = true;
try {
await recordedVideo.play();
} catch (e) {
console.error(`Play failed: ${e}`);
}
All code: https://jsfiddle.net/43rm7258/1/
When I run this in Chromium 78 I get an NotSupportedError: Failed to load because no supported source was found. from the play element of the video element.
I have no clue what I am doing wrong or how to proceed at this point.
This is about something similar, but does not help me: MediaSource randomly stops video
This example was my starting point: https://webrtc.github.io/samples/src/content/getusermedia/record/

In Summary
Getting it to work in Firefox and Chrome is easy: you just need to add an audio codec to your codecs list! video/webm;codecs=opus,vp8
Getting it to work in Safari is significantly more complicated. MediaRecorder is an "experimental" feature that has to be manually enabled under the developer options. Once enabled, Safari lacks an isTypeSupported method, so you need to handle that. Finally, no matter what you request from the MediaRecorder, Safari will always hand you an MP4 file - which cannot be streamed the way WEBM can. This means you need to perform transmuxing in JavaScript to convert the video container format on the fly
Android should work if Chrome works
iOS does not support Media Source Extensions, so SourceBuffer is not defined on iOS and the whole solution will not work
Original Post
Looking at the JSFiddle you posted, one quick fix before we get started:
You reference a variable errorMsgElement which is never defined. You should add a <div> to the page with an appropriate ID and then create a const errorMsgElement = document.querySelector(...) line to capture it
Now something to note when working with Media Source Extensions and MediaRecorder is that support is going to be very different per browser. Even though this is a "standardized" part of the HTML5 spec, it's not very consistent across platforms. In my experience getting MediaRecorder to work in Firefox doesn't take too much effort, getting it to work in Chrome is a bit harder, getting it to work in Safari is damned-near impossible, and getting it to work on iOS is literally not something you can do.
I went through and debugged this on a per-browser basis and recorded my steps, so that you can understand some of the tools available to you when debugging media issues
Firefox
When I checked out your JSFiddle in Firefox, I saw the following error in the console:
NotSupportedError: An audio track cannot be recorded: video/webm;codecs=vp8 indicates an unsupported codec
I recall that VP8 / VP9 were big pushes by Google and as such may not work in Firefox, so I tried making one small tweak to your code. I removed the , options) parameter from your call to new MediaRecorder(). This tells the browser to use whatever codec it wants, so you will likely get different output in every browser (but it should at least work in every browser)
This worked in Firefox, so I checked out Chrome.
Chrome
This time I got a new error:
(index):409 Uncaught (in promise) DOMException: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
at MediaRecorder.handleDataAvailable (https://fiddle.jshell.net/43rm7258/1/show/:409:22)
So I headed over to chrome://media-internals/ in my browser and saw this:
Audio stream codec opus doesn't match SourceBuffer codecs.
In your code you're specifying a video codec (VP9 or VP8) but not an audio codec, so the MediaRecorder is letting the browser choose any audio codec it wants. It looks like in Chrome's MediaRecorder by default chooses "opus" as the audio codec, but Chrome's SourceBuffer by default chooses something else. This was trivially fixed. I updated your two lines that set the options.mimeType like so:
options = { mimeType: "video/webm;codecs=opus, vp9" };
options = { mimeType: "video/webm;codecs=opus, vp8" };
Since you use the same options object for declaring the MediaRecorder and the SourceBuffer, adding the audio codec to the list means the SourceBuffer is now declared with a valid audio codec and the video plays
For good measure I tested the new code (with an audio codec) on Firefox. This worked! So we're 2 for 2 just by adding the audio codec to the options list (and leaving it in the parameters for declaring the MediaRecorder)
It looks like VP8 and opus work in Firefox, but aren't the defaults (although unlike Chrome, the default for MediaRecorder and SourceBuffer are the same, which is why removing the options parameter entirely worked)
Safari
This time we got an error that we may not be able to work through:
Unhandled Promise Rejection: ReferenceError: Can't find variable: MediaRecorder
The first thing I did was Google "Safari MediaRecorder", which turned up this article. I thought I'd give it a try, so I took a look. Sure enough:
I clicked on this to enable MediaRecorder and was met with the following in the console:
Unhandled Promise Rejection: TypeError: MediaRecorder.isTypeSupported is not a function. (In 'MediaRecorder.isTypeSupported(options.mimeType)', 'MediaRecorder.isTypeSupported' is undefined)
So Safari doesn't have the isTypeSupported method. Not to worry, we'll just say "if this method doesn't exist, assume it's Safari and set the type accordingly"
if (MediaRecorder.isTypeSupported) {
options = { mimeType: "video/webm;codecs=vp9" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.error(`${options.mimeType} is not Supported`);
errorMsgElement.innerHTML = `${options.mimeType} is not Supported`;
options = { mimeType: "video/webm;codecs=vp8" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.error(`${options.mimeType} is not Supported`);
errorMsgElement.innerHTML = `${options.mimeType} is not Supported`;
options = { mimeType: "video/webm" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.error(`${options.mimeType} is not Supported`);
errorMsgElement.innerHTML = `${options.mimeType} is not Supported`;
options = { mimeType: "" };
}
}
}
} else {
options = { mimeType: "" };
}
Now I just had to find a mimeType that Safari supported. Some light Googling suggests that H.264 is supported, so I tried:
options = { mimeType: "video/webm;codecs=h264" };
This successfully gave me MediaRecorder started, but failed at the line addSourceBuffer with the new error:
NotSupportedError: The operation is not supported.
I will continue to try and diagnose how to get this working in Safari, but for now I've at least addresses Firefox and Chrome
Update 1
I've continued to work on Safari. Unfortunately Safari lacks the tooling of Chrome and Firefox to dig deep into media internals, so there's a lot of guesswork involved.
I had previously figured out that we were getting an error "The operation is not supported" when trying to call addSourceBuffer. So I created a one-off page to try and call just this method under different circumstances:
Maybe add a source buffer before play is called on the video
Maybe add a source buffer before the media source has been attached to a video element
Maybe add a source buffer with different codecs
etc
I found that the issue was the codec still, and that the error messaging about the "operation" not being permitted was slightly misleading. It was the parameters that were not permitted. Simply supplying "h264" worked for the MediaRecorder, but the SourceBuffer needed me to pass along the codec parameters.
One of the first things I tried was heading to the MDN sample page and copying the codecs they used there: 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"'. This gave the same "operation not permitted" error. Digging into the meaning of these codec parameters (like what the heck does 42E01E even mean?). While I wish I had a better answer, while Googling it I stumbled upon this StackOverflow post which mentioned using 'video/mp4; codecs="avc1.64000d,mp4a.40.2"' on Safari. I gave it a try and the console errors were gone!
Although the console errors are gone now, I'm still not seeing any video. So there is still work to do.
Update 2
Further investigation in the Debugger in Safari (placing multiple breakpoints and inspecting variables at each step of the process) found that handleDataAvailable was never being called in Safari. It looks like in Firefox and Chrome mediaRecorder.start(100) will properly follow the spec and call ondatavailable every 100 milliseconds, but Safari ignores the parameter and buffers everything into one massive Blob. Calling mediaRecorder.stop() manually caused ondataavailable to be called with everything that had been recorded up until that point
I tried using setInterval to call mediaRecorder.requestData() every 100 milliseconds, but requestData was not defined in Safari (much like how isTypeSupported was not defined). This put me in a bit of a bind.
Next I tried cleaning up the whole MediaRecorder object and creating a new one every 100 milliseconds, but this threw an error on the line await bufferedBlob.arrayBuffer(). I'm still investigating why that one failed
Update 3
One thing I recall about the MP4 format is that the "moov" atom is required in order to play back any content. This is why you can't download the middle half of an MP4 file and play it. You need to download the WHOLE file. So I wondered if the fact that I selected MP4 was the reason I was not getting regular updates.
I tried changing video/mp4 to a few different values and got varying results:
video/webm -- Operation is not supported
video/x-m4v -- Behaved like MP4, I only got data when .stop() was called
video/3gpp -- Behaved like MP4
video/flv -- Operation is not supported
video/mpeg -- Behaved like MP4
Everything behaving like MP4 led me to inspect the data that was actually being passed to handleDataAvailable. That's when I noticed this:
No matter what I selected for the video format, Safari was always giving me an MP4!
Suddenly I remembered why Safari was such a nightmare, and why I had mentally classified it as "damned-near impossible". In order to stitch together several MP4s would require a JavaScript transmuxer
That's when I remembered, that's exactly what I had done before. I worked with MediaRecorder and SourceBuffer just over a year ago to try and create a JavaScript RTMP player. Once the player was done, I wanted to add support for DVR (seeking back to parts of the video that had already been streamed), which I did by using MediaRecorder and keeping a ring buffer in memory of 1-second video blobs. On Safari I ran these video blobs through the transmuxer I had coded to convert them from MP4 to ISO-BMFF so I could concatenate them together.
I wish I could share the code with you, but it's all owned by my old employer - so at this point the solution has been lost to me. I know that someone went through the trouble of compiling FFMPEG to JavaScript using emscripten, so you may be able to take advantage of that.

Additionally, I suffered issues with MediaRecorder. At the time of Audio Record, Mime Types are different like
Mac chrome - Mime Type:audio/webm;codecs=opus
Mac Safari - Mime Type:audio/mp4
Windows/Android - Mime Type:audio/webm;codecs=opus
Iphone Chrome - Mime Type:audio/mp4
In PC, I was saving the file as M4a but Audio was not running in IOS. After some analysis and Testing. I decided to convert the file after Upload in Server and used ffmpeg and It worked like a charm.
<!-- https://mvnrepository.com/artifact/org.bytedeco/ffmpeg-platform -->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg-platform</artifactId>
<version>4.3.2-1.5.5</version>
</dependency>
/**
* Convert the file into MP4 using H264 Codac in order to make it work in IOS Mobile Device
* #param file
* #param outputFile
*/
private void convertToM4A(File file, File outputFile) {
try {
String ffmpeg = Loader.load(org.bytedeco.ffmpeg.ffmpeg.class);
ProcessBuilder pb = new ProcessBuilder(ffmpeg, "-i", file.getPath(), "-vcodec", "h264", outputFile.getPath());
pb.inheritIO().start().waitFor();
}catch (Exception e ){
e.printStackTrace();
}
}

Displaying mp4 streamed video in browser

I'm trying to display a continuous video stream (live-stream) in a browser.
Description:
My client reported a video stream doesn't work in the Chrome browser. I thought it will be an easy case, I even tried to write a demo, to prove streaming should be available with just HTML5 native video tag:
https://github.com/mishaszu/streaming-video-demo
No problems with random video but:
the particular video stream on client-side doesn't work.
With html code:
<video id="video-block" width="320" height="200" autoplay>
<source src="url/to/my/video" type="video/mp4">
</video>
it shows loader for a while and dies.
What I know about the stream:
1. Codec used: H264-MPEG-4 AVC (part 10) (avc1)
2. It's a live stream, not a file, so I can't use command like MP4Box from a terminal with it
3. Because it's live stream it probably doesn't have "end of file"
4. I know it's not broken because VLC is able to display it
5. I tried native HTML 5 video tag with all Media type strings (just in case to check all codecs for mp4)
As I mentioned trying different mime types didn't help, I also tried to use MediaSource but I am really not sure how to use it with a live stream, as all information I found made assumptions:
a) waiting for resolve promise and then appends buffer
b) adding the event listener for updateend to appends buffer
I think in the case of a live stream it won't work.
Conclusion:
I found a lot of information about how a streamed file might contain metadata (at the beginning of the file or at the end)... and I ended up with a conclusion that maybe I do not fully understand what's going on.
Questions:
What's the proper way to handle the mp4 live stream?
If a native HTML video tag should support the live stream, how to debug it?
I thought that maybe I should look for something like HLS but for mp4 format?

I've went through the same - I needed to mux an incoming live stream from rtsp to HTML5 video, and sadly this may become non-trivial.
So, for a live stream you need a fragmented mp4 (check this SO question if you do not know what that is:). The is the isobmff specification, which sets rules on what boxes should be present in the stream. From my experience though browsers have their own quirks (had to debug chrome/firefox to find them) when it comes to a live stream. Chrome has chrome://media-internals/ tab, which shows the errors for all loaded players - this can help debugging as well.
So my shortlist to solve this would be:
1) If you say that VLC plays the stream, open the Messages window in VLC ( Tools -> Messages ), set severity to debug, and you should see the mp4 box info in there as the stream comes in, verify that moof boxes are present
2a) Load the stream in chrome, open chrome://media-internals/ in a new tab and inspect errors
2b) Chrome uses ffmpeg underneath, so you could try playing the stream with ffplay as well and check for any errors.
2c) You are actually incorrect about mp4box - you could simply load a number of starting bytes from the stream, save to a file and use mp4box or other tools on that (at worst it should complain about some corrupted boxes at the end if you cut a box short)
If none of 2a/2b/2c provide any relevant error info that you can fix yourself, update the question with the outputs from these, so that others have more info.

Asp Net Core FileContentResult Audio File to Safari

In Safari (11), a static audio file loaded into a src via html or javascript works, albeit with the limitations of requiring user input before playing.
ex.
<audio src="static.mp3">
or
var audio = new Audio('static.mp3');
audio.play();
work fine.
However, I need to load audio files from the database, so I was using a controller action like so:
public FileContentResult GetAudio(int audioId)
{
if (!DbContext.Audios.Any(a => a.Id == audioId))
{
return null;
}
var audio = DbContext.Audios.Single(a => a.Id == audioId);
return File(audio.File, "audio/mp3");
}
and set like
<audio src="getaudio?audioId=1">
or
var audio = new Audio('getaudio?audioId=1');
it will not play in MacOS (Safari) or iOS, but works fine in Chrome and Edge (except on iOS). Depending on how I configure things, I get some form of Unhandled Promise error. I've also tried loading into a Web Audio buffer, with the same exact success and failures.
Does anyone have a solution or workaround to load my audio files on Safari?
EDIT
Ok, so on further testing, I discovered that it's not so much whether the files were sent via action or static file, it's how they were saved to the database in the first place. I'm now working to figure out why files I save (as byte[]) and then reload are not recognized by Safari.

OK, so it turns out, I was making the recordings with MediaRecorder, which is a fairly new feature in Chrome and a few other browsers. It didn't matter what format I told it to save as, because only webm is supported. And guess who doesn't support webm format? Safari. Any other browser was picking it up fine, regardless of what incorrect extension I put on the file.
When I find a webm to m4a conversion, I will add it here. I'm sure there are some solutions out there.

Websocket saturation in Chrome, blob points to data which does not exist

I've got one of this very difficult to debug problems in my app.
I'm using websockets to receive images from my server. I'm getting around 50 images per second in binary and showing them in a canvas as image.
Everything works ok, but sometimes I'm getting an error:
FileReader error. Name: NotFoundError Message: A requested file or directory could not be found at the time an operation was processed.
My code looks like this:
wsVisualizerManager.onMessage = function(e)
{
if (e.data instanceof Blob)
{
var blob = e.data;
this.reader.readAsArrayBuffer(blob);
}
}
Basically I get a blob and I read it with a FileReader object (this.reader) to be able to visualize it as an image.
As I said everything works, but sometimes, seems that due to receive more images that the ones it can handle I start getting this error.
When I start getting this error I don't get it once, I get it many times, so my app is blocked during a lot of time. If I launch chrome development tools the error stops immediately and my app works again (so maybe this developer tool is cleaning some buffers?).
So, I guess that It is some kind of buffer which get completely full, but the think is that If I have to lose old messages I'm ok with that, but why calling onMessage with a bad formated or inexistent blob?
I don't know exactly how to debug this error, because websocket API has just a few methods and no one allows me to access a buffer or clean it.
A way of generate this error manually (otherwise I get it just twice a day) is putting one breakpoint just at the beginning of onMessage and wait some time. What happens then is that when I press play I start getting this error and after some seconds I start receiving retarded images. So seems that it's not removing the received images or at least not all of them.
I'm developing the server part too. And I'm using C++ with the library websocket++. And my chrome version is: 43.0.2357.125 (64-bit)
Any clue of how to proceed debugging this error or trying to solve the problem?

Html5 Audio plays only once in my Javascript code

I have a dashboard web-app that I want to play an alert sound if its having problems connecting. The site's ajax code will poll for data and throttle down its refresh rate if it can't connect. Once the server comes back up, the site will continue working.
In the mean time I would like a sound to play each time it can't connect (so I know to check the server). Here is that code. This code works.
var error_audio = new Audio("audio/"+settings.refresh.error_audio);
error_audio.load();
//this gets called when there is a connection error.
function onConnectionError() {
error_audio.play();
}
However the 2nd time through the function the audio doesn't play. Digging around in Chrome's debugger the 'played' attribute in the audio element gets set to true. Setting it to false has no results. Any ideas?

I encountered this just today, after more searching I found that you must set the source property on the audio element again to get it to restart. Don't worry, no network activity occurs, and the operation is heavily optimized.
var error_audio = new Audio("audio/"+settings.refresh.error_audio);
error_audio.load();
//this gets called when there is a connection error.
function onConnectionError() {
error_audio.src = "audio/"+settings.refresh.error_audio;
error_audio.play();
}
This behavior is expressed in chrome 21. FF doesn't seem to mind setting the src twice either!

Try setting error_audio.currentTime to 0 before playing it. Maybe it doesn't automatically go back to the beginning

You need to implement the Content-Range response headers, since Chrome requests the file in multiple parts via the Range HTTP header.
See here: HTML5 <audio> Safari live broadcast vs not
Once that has been implemented, both the play() function and setting the currentTime property should work.

Q: I’VE GOT AN AUDIOBUFFERSOURCENODE, THAT I JUST PLAYED BACK WITH NOTEON(), AND I WANT TO PLAY IT AGAIN, BUT NOTEON() DOESN’T DO ANYTHING! HELP!
A: Once a source node has finished playing back, it can’t play back more. To play back the underlying buffer again, you should create a new AudioBufferSourceNode and call noteOn().
Though re-creating the source node may feel inefficient, source nodes are heavily optimized for this pattern. Plus, if you keep a handle to the AudioBuffer, you don't need to make another request to the asset to play the same sound again. If you find yourself needing to repeat this pattern, encapsulate playback with a simple helper function like playSound(buffer).
Q: WHEN PLAYING BACK A SOUND, WHY DO YOU NEED TO MAKE A NEW SOURCE NODE EVERY TIME?
A: The idea of this architecture is to decouple audio asset from playback state. Taking a record player analogy, buffers are analogous to records and sources to play-heads. Because many applications involve multiple versions of the same buffer playing simultaneously, this pattern is essential.
source:
http://updates.html5rocks.com/2012/01/Web-Audio-FAQ

You need to pause the audio just before its end and change the current playing time to zero, then play it.
Javascript/Jquery to control HTML5 audio elements - check this link - explains How to handle/control the HTML5 audio elements?. It may help you!

Chrome/Safari have fixed this issue in newer versions of the browser and the above code now works as expected. I am not sure the precise version it was fixed in.