Develop Reference

WebAudioAPI decodeAudioData() giving null error on iOS 14 Safari

I have an mp3 audio stream player that works well in every desktop browser, using MediaSourceExtensions with a fallback to WebAudioAPI for those browsers that do not support MSE. iOS Safari is one such browser, and should theoretically support mp3 decoding via the Web Audio API without issues.
I've been struggling to get iOS Safari to properly play the mp3 audio chunks that are being returned from the stream. So far, it's the only browser that seems to have issues and I can't for the life of me figure out what's going on. Sadly, there isn't a whole lot of information on corner cases like this and the other questions here on StackOverflow haven't been any help.
Here's the relevant part of my js code where things are getting hung up. It's a callback function for an async fetch() process that's grabbing the mp3 data from the stream.
async function pushStream(value) {
// Web Audio streaming for browsers that don't support MSE
if (usingWebAudio) {
// convert the stream UInt8Array to an ArrayBuffer
var dataBuffer = value.stream.buffer;
// decode the raw mp3 chunks
audCtx.decodeAudioData(dataBuffer, function(newData) {
// add the decoded data to the buffer
console.log("pushing new audio data to buffer");
webAudioBuffer.push(newData);
// if we have audio in the buffer, play it
if (webAudioBuffer.length) {
scheduleWebAudio();
}
}, function(e) {
console.error(e);
});
What I'm seeing is the error callback being fired and printing null: null as its error message (very helpful). Every so often, I will see the console print pushing new audio data to buffer, but this seems to only happen about once every few minutes while the stream is playing. Almost all the stream data is erroring out during the decode and the lack of useful error messages is preventing me from figuring out why.
As far as I can tell, iOS safari should support mp3 streams without any issues. It also should support the decodeAudioData() function. Most of the other answers I was able to find related to trying to play audio before the user interacts with the screen. In my case, I start the audio using a play button on the page so I don't believe that's the problem either.
One final thing, I'm developing on Windows and using the remotedebug iOS adapter. This could possibly be the reason why I'm not getting useful debug messages, however all other debug and error prints seem to work fine so I don't believe that's the case.
Thanks in advance for any help!

Unfortunately there is a bug in Safari which causes it to reject the decodeAudioData() promise with null. From my experience this happens in cases where it should actually reject the promise with an EncodingError instead.
The bug can be reproduced by asking Safari do decode an image. https://github.com/chrisguttandin/standardized-audio-context/blob/9c705bd2e5d8a774b93b07c3b203c8f343737988/test/expectation/safari/any/offline-audio-context-constructor.js#L648-L663
In general decodeAudioData() can only handle full files. It isn't capable of decoding a file in chunks. The WebCodecs API is meant to solve that but I guess it won't be available on iOS anytime soon.
However there is one trick that works with MP3s because of their internal structure. MP3s are built out of chunks themselves and any number of those chunks form a technically valid MP3. That means you can pre-process your data by making sure that each of the buffers that you pass on to decodeAudioData() begins and ends exactly at those internal chunk boundaries. The phonograph library does for example follow that principle.

How to use MediaRecorder as MediaSource

As an exercise in learning WebRTC I am trying to to show the local webcam and side by side with a delayed playback of the webcam. In order to achieve this I am trying to pass recorded blobs to a BufferSource and use the corresponding MediaSource as source for a video element.
// the ondataavailable callback for the MediaRecorder
async function handleDataAvailable(event) {
// console.log("handleDataAvailable", event);
if (event.data && event.data.size > 0) {
recordedBlobs.push(event.data);
}
if (recordedBlobs.length > 5) {
if (recordedBlobs.length === 5)
console.log("buffered enough for delayed playback");
if (!updatingBuffer) {
updatingBuffer = true;
const bufferedBlob = recordedBlobs.shift();
const bufferedAsArrayBuffer = await bufferedBlob.arrayBuffer();
if (!sourceBuffer.updating) {
console.log("appending to buffer");
sourceBuffer.appendBuffer(bufferedAsArrayBuffer);
} else {
console.warn("Buffer still updating... ");
recordedBlobs.unshift(bufferedBlob);
}
}
}
}
// connecting the media source to the video element
recordedVideo.src = null;
recordedVideo.srcObject = null;
recordedVideo.src = window.URL.createObjectURL(mediaSource);
recordedVideo.controls = true;
try {
await recordedVideo.play();
} catch (e) {
console.error(`Play failed: ${e}`);
}
All code: https://jsfiddle.net/43rm7258/1/
When I run this in Chromium 78 I get an NotSupportedError: Failed to load because no supported source was found. from the play element of the video element.
I have no clue what I am doing wrong or how to proceed at this point.
This is about something similar, but does not help me: MediaSource randomly stops video
This example was my starting point: https://webrtc.github.io/samples/src/content/getusermedia/record/

In Summary
Getting it to work in Firefox and Chrome is easy: you just need to add an audio codec to your codecs list! video/webm;codecs=opus,vp8
Getting it to work in Safari is significantly more complicated. MediaRecorder is an "experimental" feature that has to be manually enabled under the developer options. Once enabled, Safari lacks an isTypeSupported method, so you need to handle that. Finally, no matter what you request from the MediaRecorder, Safari will always hand you an MP4 file - which cannot be streamed the way WEBM can. This means you need to perform transmuxing in JavaScript to convert the video container format on the fly
Android should work if Chrome works
iOS does not support Media Source Extensions, so SourceBuffer is not defined on iOS and the whole solution will not work
Original Post
Looking at the JSFiddle you posted, one quick fix before we get started:
You reference a variable errorMsgElement which is never defined. You should add a <div> to the page with an appropriate ID and then create a const errorMsgElement = document.querySelector(...) line to capture it
Now something to note when working with Media Source Extensions and MediaRecorder is that support is going to be very different per browser. Even though this is a "standardized" part of the HTML5 spec, it's not very consistent across platforms. In my experience getting MediaRecorder to work in Firefox doesn't take too much effort, getting it to work in Chrome is a bit harder, getting it to work in Safari is damned-near impossible, and getting it to work on iOS is literally not something you can do.
I went through and debugged this on a per-browser basis and recorded my steps, so that you can understand some of the tools available to you when debugging media issues
Firefox
When I checked out your JSFiddle in Firefox, I saw the following error in the console:
NotSupportedError: An audio track cannot be recorded: video/webm;codecs=vp8 indicates an unsupported codec
I recall that VP8 / VP9 were big pushes by Google and as such may not work in Firefox, so I tried making one small tweak to your code. I removed the , options) parameter from your call to new MediaRecorder(). This tells the browser to use whatever codec it wants, so you will likely get different output in every browser (but it should at least work in every browser)
This worked in Firefox, so I checked out Chrome.
Chrome
This time I got a new error:
(index):409 Uncaught (in promise) DOMException: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
at MediaRecorder.handleDataAvailable (https://fiddle.jshell.net/43rm7258/1/show/:409:22)
So I headed over to chrome://media-internals/ in my browser and saw this:
Audio stream codec opus doesn't match SourceBuffer codecs.
In your code you're specifying a video codec (VP9 or VP8) but not an audio codec, so the MediaRecorder is letting the browser choose any audio codec it wants. It looks like in Chrome's MediaRecorder by default chooses "opus" as the audio codec, but Chrome's SourceBuffer by default chooses something else. This was trivially fixed. I updated your two lines that set the options.mimeType like so:
options = { mimeType: "video/webm;codecs=opus, vp9" };
options = { mimeType: "video/webm;codecs=opus, vp8" };
Since you use the same options object for declaring the MediaRecorder and the SourceBuffer, adding the audio codec to the list means the SourceBuffer is now declared with a valid audio codec and the video plays
For good measure I tested the new code (with an audio codec) on Firefox. This worked! So we're 2 for 2 just by adding the audio codec to the options list (and leaving it in the parameters for declaring the MediaRecorder)
It looks like VP8 and opus work in Firefox, but aren't the defaults (although unlike Chrome, the default for MediaRecorder and SourceBuffer are the same, which is why removing the options parameter entirely worked)
Safari
This time we got an error that we may not be able to work through:
Unhandled Promise Rejection: ReferenceError: Can't find variable: MediaRecorder
The first thing I did was Google "Safari MediaRecorder", which turned up this article. I thought I'd give it a try, so I took a look. Sure enough:
I clicked on this to enable MediaRecorder and was met with the following in the console:
Unhandled Promise Rejection: TypeError: MediaRecorder.isTypeSupported is not a function. (In 'MediaRecorder.isTypeSupported(options.mimeType)', 'MediaRecorder.isTypeSupported' is undefined)
So Safari doesn't have the isTypeSupported method. Not to worry, we'll just say "if this method doesn't exist, assume it's Safari and set the type accordingly"
if (MediaRecorder.isTypeSupported) {
options = { mimeType: "video/webm;codecs=vp9" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.error(`${options.mimeType} is not Supported`);
errorMsgElement.innerHTML = `${options.mimeType} is not Supported`;
options = { mimeType: "video/webm;codecs=vp8" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.error(`${options.mimeType} is not Supported`);
errorMsgElement.innerHTML = `${options.mimeType} is not Supported`;
options = { mimeType: "video/webm" };
if (!MediaRecorder.isTypeSupported(options.mimeType)) {
console.error(`${options.mimeType} is not Supported`);
errorMsgElement.innerHTML = `${options.mimeType} is not Supported`;
options = { mimeType: "" };
}
}
}
} else {
options = { mimeType: "" };
}
Now I just had to find a mimeType that Safari supported. Some light Googling suggests that H.264 is supported, so I tried:
options = { mimeType: "video/webm;codecs=h264" };
This successfully gave me MediaRecorder started, but failed at the line addSourceBuffer with the new error:
NotSupportedError: The operation is not supported.
I will continue to try and diagnose how to get this working in Safari, but for now I've at least addresses Firefox and Chrome
Update 1
I've continued to work on Safari. Unfortunately Safari lacks the tooling of Chrome and Firefox to dig deep into media internals, so there's a lot of guesswork involved.
I had previously figured out that we were getting an error "The operation is not supported" when trying to call addSourceBuffer. So I created a one-off page to try and call just this method under different circumstances:
Maybe add a source buffer before play is called on the video
Maybe add a source buffer before the media source has been attached to a video element
Maybe add a source buffer with different codecs
etc
I found that the issue was the codec still, and that the error messaging about the "operation" not being permitted was slightly misleading. It was the parameters that were not permitted. Simply supplying "h264" worked for the MediaRecorder, but the SourceBuffer needed me to pass along the codec parameters.
One of the first things I tried was heading to the MDN sample page and copying the codecs they used there: 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"'. This gave the same "operation not permitted" error. Digging into the meaning of these codec parameters (like what the heck does 42E01E even mean?). While I wish I had a better answer, while Googling it I stumbled upon this StackOverflow post which mentioned using 'video/mp4; codecs="avc1.64000d,mp4a.40.2"' on Safari. I gave it a try and the console errors were gone!
Although the console errors are gone now, I'm still not seeing any video. So there is still work to do.
Update 2
Further investigation in the Debugger in Safari (placing multiple breakpoints and inspecting variables at each step of the process) found that handleDataAvailable was never being called in Safari. It looks like in Firefox and Chrome mediaRecorder.start(100) will properly follow the spec and call ondatavailable every 100 milliseconds, but Safari ignores the parameter and buffers everything into one massive Blob. Calling mediaRecorder.stop() manually caused ondataavailable to be called with everything that had been recorded up until that point
I tried using setInterval to call mediaRecorder.requestData() every 100 milliseconds, but requestData was not defined in Safari (much like how isTypeSupported was not defined). This put me in a bit of a bind.
Next I tried cleaning up the whole MediaRecorder object and creating a new one every 100 milliseconds, but this threw an error on the line await bufferedBlob.arrayBuffer(). I'm still investigating why that one failed
Update 3
One thing I recall about the MP4 format is that the "moov" atom is required in order to play back any content. This is why you can't download the middle half of an MP4 file and play it. You need to download the WHOLE file. So I wondered if the fact that I selected MP4 was the reason I was not getting regular updates.
I tried changing video/mp4 to a few different values and got varying results:
video/webm -- Operation is not supported
video/x-m4v -- Behaved like MP4, I only got data when .stop() was called
video/3gpp -- Behaved like MP4
video/flv -- Operation is not supported
video/mpeg -- Behaved like MP4
Everything behaving like MP4 led me to inspect the data that was actually being passed to handleDataAvailable. That's when I noticed this:
No matter what I selected for the video format, Safari was always giving me an MP4!
Suddenly I remembered why Safari was such a nightmare, and why I had mentally classified it as "damned-near impossible". In order to stitch together several MP4s would require a JavaScript transmuxer
That's when I remembered, that's exactly what I had done before. I worked with MediaRecorder and SourceBuffer just over a year ago to try and create a JavaScript RTMP player. Once the player was done, I wanted to add support for DVR (seeking back to parts of the video that had already been streamed), which I did by using MediaRecorder and keeping a ring buffer in memory of 1-second video blobs. On Safari I ran these video blobs through the transmuxer I had coded to convert them from MP4 to ISO-BMFF so I could concatenate them together.
I wish I could share the code with you, but it's all owned by my old employer - so at this point the solution has been lost to me. I know that someone went through the trouble of compiling FFMPEG to JavaScript using emscripten, so you may be able to take advantage of that.

Additionally, I suffered issues with MediaRecorder. At the time of Audio Record, Mime Types are different like
Mac chrome - Mime Type:audio/webm;codecs=opus
Mac Safari - Mime Type:audio/mp4
Windows/Android - Mime Type:audio/webm;codecs=opus
Iphone Chrome - Mime Type:audio/mp4
In PC, I was saving the file as M4a but Audio was not running in IOS. After some analysis and Testing. I decided to convert the file after Upload in Server and used ffmpeg and It worked like a charm.
<!-- https://mvnrepository.com/artifact/org.bytedeco/ffmpeg-platform -->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg-platform</artifactId>
<version>4.3.2-1.5.5</version>
</dependency>
/**
* Convert the file into MP4 using H264 Codac in order to make it work in IOS Mobile Device
* #param file
* #param outputFile
*/
private void convertToM4A(File file, File outputFile) {
try {
String ffmpeg = Loader.load(org.bytedeco.ffmpeg.ffmpeg.class);
ProcessBuilder pb = new ProcessBuilder(ffmpeg, "-i", file.getPath(), "-vcodec", "h264", outputFile.getPath());
pb.inheritIO().start().waitFor();
}catch (Exception e ){
e.printStackTrace();
}
}

Seeking in HTML5 video with Chrome

I'm having issues seeking in video's using Chrome.
For some reason, no matter what I do, video.seekable.end(0) is always 0.
When I call video.currentTime = 5, followed by console.log(video.currentTime), I see it is always 0, which seems to reset the video.
I've tried both MP4 and VP9-based webm formats, but both gave the same results.
What's more annoying is that Firefox runs everything perfectly. Is there something special about Chrome that I should know about?
Here's my code (which only works in Firefox):
<div class="myvideo">
<video width="500" height="300" id="video1" preload="auto">
<source src="data/video1.webm" type="video/webm"/>
Your browser does not support videos.
</video>
</div>
And here's the javascript:
var videoDiv = $(".myvideo").children().get(0)
videoDiv.load();
videoDiv.addEventListener("loadeddata", function(){
console.log("loaded");
console.log(videoDiv.seekable.end(0)); //Why is this always 0 in Chrome, but not Firefox?
videoDiv.currentTime = 5;
console.log(videoDiv.currentTime); //Why is this also always 0 in Chrome, but not Firefox?
});
Note that simply calling videoDiv.play() does actually correctly play the video in both browsers.
Also, after the movie file is fully loaded, the videoDiv.buffered.end(0) also gives correct values in both browsers.

It took me a while to figure it out...
The problem turned out to be server-side. I was using a version of Jetty to serve all my video-files. The simple configuration of Jetty did not support byte serving.
The difference between Firefox and Chrome is that Firefox will download the entire video file so that you can seek through it, even if the server does not support http code 206 (partial content). Chrome on the other hand refuses to download the entire file (unless it is really small, like around 2-3mb).
So to get the currentTime parameter of html5 video to be working in Chrome, you need a server that supports http code 206.
For anyone else having this problem, you can double check your server config with curl:
curl -H Range:bytes=16- -I http://localhost:8080/GOPR0001.mp4
This should return code 206. If it returns code 200, Chrome will not be able to seek the video, but Firefox will, due to a workaround in the browser.
And a final tip: You can use npm http-server to get a simple http-server for a local folder that supports partial content:
npm install http-server -g
And run it to serve a local folder:
http-server -p 8000

Work around if modifying server code is unfeasible. Make an API call for you video, then load the blob into URL.createObjectURL and feed that into the src attribute of your video html tag. This will load the entire file and then chrome will know the size of the file allowing seeking capabilities to work.
axios.get(`${url}`, {
responseType: "blob"
})
.then(function(response) {
setVideo(URL.createObjectURL(response.data));
})
.catch(function(error) {
// handle error
console.log(error);
});

If you wait for canplaythrough instead of loadeddata, it works.
See this codepen example.

You have 3 possibilities for the Video tag: MP4, OGG, WebM.
Not all formats work in all browsers.
Here, I'm thinking that WebM works in Firefox but not Chrome, so you should supply both alternative formats for MP4 and WebM files, by including a 2nd Source tag referring to the MP4 file.
E.g. src="data/video1.mp4" type="video/mp4"
The relevant version will be automatically selected by the browser.

I had a similar problem. I was listening for an the end event on a video and setting currentTime to the middle of the video to loop it continuously. It wasn't working in Safari or Chrome.
I think there may be a bug in Safari/Chrome where playhead position properties aren't available unless the media is currently playing.
My workaround was to start my loop just before the video end and not letting it actually end.
Try testing yours by starting playback first and then run your fuction to see if it works in Safari Chrome.

javascript: reading id3 tags from mp3 stream

I'm writing a Webradio Player for Firefox OS (mobilephone OS based on webstandards). Now I want to add a feature, that displays for example the actual Title like some radiostations are sending and VLC-Mediaplayer for example is able to display. All tested streams are using MP3. I'm playing the audio via the html-audio-tag. Until now i've tested https://github.com/aadsm/JavaScript-ID3-Reader and http://ericbidelman.tumblr.com/post/8343485440/reading-mp3-id3-tags-in-javascript. The JavaScript-ID3-Reader seams not to be able to handle streams. The other way doesn't writes an log via "console.log(title);". Does anybody know a way to add this feature?
Thanks

I've found out that it is impossible this way. But there is a much easier way:
var meta = document.getElementById("audio").mozGetMetadata();
The id audio refers to a audio tag. You can acces the title by meta.title. But there is a bug in firefox (firefox os, too) that causes that this isn't working with mp3-streams: https://bugzilla.mozilla.org/show_bug.cgi?id=908902

HTML5 audio starts from the wrong position in Firefox

I'm trying to play an mp3 file and I want to jump to specific location in the file. In Chrome 33 on Windows, the file jumps the correct position (as compared with VLC playing the mp3 locally) but in Firefox 28 on Windows it plays too far forward and in Internet Explorer 11 it plays too far behind.
It used to work correctly in Firefox 27 and earlier.
Is there a better way of doing this?
EDIT: The problem doesn't even require SoundManager2. You can replicate the same issue with just the <audio> tag in Firefox. These two lines are all the code you need to reproduce it:
<audio autoplay id="audio" src="http://ivdemo.chaseits.co.uk/enron/20050204-4026(7550490)a.mp3" controls preload></audio>
<button onclick="javascript:document.getElementById('audio').currentTime = 10;">Jump to 10 secs "...be with us in, er, 1 minute... ok" </button>
Try it here: http://jsfiddle.net/cpickard/29Gt3/
EDIT: Tried with Firefox Nightly, no improvement. I have reported it as bug 994561 in bugzilla. Still looking for a workaround for now.

The problem lies in the VBR encoding of the mp3.
Download that mp3 to disk and convert it to fixed bit rate, say with Audacity.
Run the example from disk:
<audio autoplay id="audio" src="./converted.mp3" controls preload></audio>
<button onclick="javascript:document.getElementById('audio').currentTime = 10;">
Jump to 10 secs "...be with us in, er, 1 minute... ok" </button>
and it works fine for me.
So my suggestion for workaround is is to upload an alternative fixed-bit mp3 file in place of the one you are using. Then it should work in the current FFx.

I work on SoundJS and while implementing audio sprites recently ran into similar issues. According to the spec, setting the position of html audio playhead can be inaccurate by up to 300ms. So that could explain some of the issues you are seeing.
Interestingly, your fiddle plays correctly for me in FF 28 on win 8.1 if I just let it play through from the start.
There are also some known issues with audio length accuracy that may also have an effect, which you can read about here.
If you want precision, I would definitely recommend using Web Audio where possible or a library like SoundJS.
Hope that helps.

I met the same issue, and I solved it by converting my MP3 file to the CBR(Constant Bit Rate) format. Then, it can solve the inconsistent issue between the currentTime and the real sound.
Choose the CBR format
Steps:
Download and install "Audacity" (it's a free for any platform)
Open your MP3 file
Click [File] -> [Export] -> [Options] -> [Constant] (See: Converting MP3 to Constant Bit Rate)
Audacity will ask you to provide the LAME MP3 encoder
(See: [download and install the LAME MP3 encoder])
There will be no inconsistent/asynchronous issue.
Also see:
HTML5 audio starts from the wrong position in Firefox
Inconsistent seeking in HTML5 Audio player
tsungjung411#gmail.com

I just tried your code with another audio url here, it seemed to work and i did not experience a delay of any sort in Firefox( v29) which i did previously.
<audio autoplay id="audio" src="http://mediaelementjs.com/media/AirReview-Landmarks-02-ChasingCorporate.mp3" controls preload></audio>
I guess to jump around an audio file, your server must be configured properly.
The client sends byte range requests to seek and play certain regions of a file, so the server must response adequately:
In order to support seeking and playing back regions of the media that
aren't yet downloaded, Gecko uses HTTP 1.1 byte-range requests to
retrieve the media from the seek target position. In addition, if you
don't serve X-Content-Duration headers, Gecko uses byte-range requests
to seek to the end of the media (assuming you serve the Content-Length
header) in order to determine the duration of the media.
Hope this helps..
You could also try looking into Web Audio API for sound-effect-like playback which gives you some guarantees about the playback delays.

After testing the fiddle it is noticable that there is some issue with FF , anywho , after searching sometime , the issue is due to "Performance lag" , but the good news is that someone has found a solution to that issue , you may want to read this :
http://lowlag.alienbill.com/
a single script will solve it all.