Develop Reference

WebAudioAPI decodeAudioData() giving null error on iOS 14 Safari

I have an mp3 audio stream player that works well in every desktop browser, using MediaSourceExtensions with a fallback to WebAudioAPI for those browsers that do not support MSE. iOS Safari is one such browser, and should theoretically support mp3 decoding via the Web Audio API without issues.
I've been struggling to get iOS Safari to properly play the mp3 audio chunks that are being returned from the stream. So far, it's the only browser that seems to have issues and I can't for the life of me figure out what's going on. Sadly, there isn't a whole lot of information on corner cases like this and the other questions here on StackOverflow haven't been any help.
Here's the relevant part of my js code where things are getting hung up. It's a callback function for an async fetch() process that's grabbing the mp3 data from the stream.
async function pushStream(value) {
// Web Audio streaming for browsers that don't support MSE
if (usingWebAudio) {
// convert the stream UInt8Array to an ArrayBuffer
var dataBuffer = value.stream.buffer;
// decode the raw mp3 chunks
audCtx.decodeAudioData(dataBuffer, function(newData) {
// add the decoded data to the buffer
console.log("pushing new audio data to buffer");
webAudioBuffer.push(newData);
// if we have audio in the buffer, play it
if (webAudioBuffer.length) {
scheduleWebAudio();
}
}, function(e) {
console.error(e);
});
What I'm seeing is the error callback being fired and printing null: null as its error message (very helpful). Every so often, I will see the console print pushing new audio data to buffer, but this seems to only happen about once every few minutes while the stream is playing. Almost all the stream data is erroring out during the decode and the lack of useful error messages is preventing me from figuring out why.
As far as I can tell, iOS safari should support mp3 streams without any issues. It also should support the decodeAudioData() function. Most of the other answers I was able to find related to trying to play audio before the user interacts with the screen. In my case, I start the audio using a play button on the page so I don't believe that's the problem either.
One final thing, I'm developing on Windows and using the remotedebug iOS adapter. This could possibly be the reason why I'm not getting useful debug messages, however all other debug and error prints seem to work fine so I don't believe that's the case.
Thanks in advance for any help!

Unfortunately there is a bug in Safari which causes it to reject the decodeAudioData() promise with null. From my experience this happens in cases where it should actually reject the promise with an EncodingError instead.
The bug can be reproduced by asking Safari do decode an image. https://github.com/chrisguttandin/standardized-audio-context/blob/9c705bd2e5d8a774b93b07c3b203c8f343737988/test/expectation/safari/any/offline-audio-context-constructor.js#L648-L663
In general decodeAudioData() can only handle full files. It isn't capable of decoding a file in chunks. The WebCodecs API is meant to solve that but I guess it won't be available on iOS anytime soon.
However there is one trick that works with MP3s because of their internal structure. MP3s are built out of chunks themselves and any number of those chunks form a technically valid MP3. That means you can pre-process your data by making sure that each of the buffers that you pass on to decodeAudioData() begins and ends exactly at those internal chunk boundaries. The phonograph library does for example follow that principle.

Is there a way to stop Web Audio API decodeAudioData method memory leak?

The Problem
When creating audio buffers using the Web Audio API, there are buffers created by the decodeAudioData method, which reside in memory and are apparently not accessible through JavaScript. They seem to hang around for the entire life of a browser tab, and never get garbage collected.
Possible Reason For the Problem
I know that these buffers are separated from the main thread and set on another thread for asynchronous decoding. I also know that the API spec says that decodeAudioData should not be allowed to decode the same input buffer twice, which I assume is why a copy of the decoded buffer and/or the encoded input buffer are kept around. However, on memory limited devices like Chromecast, this causes huge amounts of memory to accumulate and Chromecast crashes.
Reproducibility
In my example code, I fetch an mp3 using Ajax and then pass the arraybuffer into the decodeAudioData function. Normally within that function there is a onsuccess callback which can take the decoded AudioBuffer as a parameter. But here in my code, I don't even pass that in. Therefore I also don't do anything with the decoded buffer after decoding it. It is not referenced anywhere within my code. It is entirely left in the native code. However, every call to this function increases the memory allocation and it is never released. For example, in Firefox about:memory shows the audiobuffers there for the life of the Tab. Non-reference should be sufficient for the garbage collector to get rid of these buffers.
My main question then is, is there any reference to these decoded audio buffers, say within the audiocontext object, or somewhere else that I can try to remove them from memory? Or is there any other way that I can cause these stored and unreachable buffers to disappear?
My question differs from all the others currently on SO regarding decodeAudioData because I show that the memory leak happens even without the user storing any reference or even using the returned decoded audio buffer.
Code To Reproduce
function loadBuffer() {
// create an audio context
var context = new (window.AudioContext || window.webkitAudioContext)();
// fetch mp3 as an arraybuffer async
var url = "beep.mp3";
var request = new XMLHttpRequest();
request.open("GET", url, true);
request.responseType = "arraybuffer";
request.onload = function () {
context.decodeAudioData(
request.response,
function () {// not even passing buffer into this function as a parameter
console.log("just got tiny beep file and did nothing with it, and yet there are audio buffers in memory that never seem to be released or gc'd");
},
function (error) {
console.error('decodeAudioData error', error);
}
);
};
request.onerror = function () {
console.log('error loading mp3');
}
request.send();
}
To anticipate some possible responses.
I must use Web Audio API because I am playing four part harmony from four audio files on Chromecast and the html audio element does not support multiple simultaneous playback on Chromecast.
Probably any JS library you may reference [e.g. Howler.js, Tone.js, Amplitude.js etc.] is built upon the Web Audio API, and so they will all share this memory leak problem.
I know that the WAA is implementation dependent on a per browser basis. My primary concern at the moment is Chromecast, but the problem exists for every browser I've tried.
Therefore, I think it is a spec related issue where the spec requires the non-dupe encoding rule, and so implementers keep copies of the buffer around on a browser level thread so they can check them against new xhr inputs. If the spec writer's happen to read my question, is there not a way that the user can have the option for this behavior, and opt out of it if they wish in order to prevent the internal buffer storage on mobile and thin memory platforms?
I have not been able to find any reference to these buffers in any JS object.
I know that I can audio_context.close() and then hope for garbage collection of all the resources held by the audio_context, and then hope that I can reinstantiate the audio_context with a new one, but that has not empirically been timely enough for my application. Chromecast crashes before GC takes out the trash.

Pragmatic Workaround
I have found a method to solve the problem of the Web Audio API audiobuffers handing around indefinitely and crashing Chromecast and other mobile platforms. [[ I have not tested this on all browsers - your mileage may vary. ]]
LOADING STAGE
Load the document using Web Audio API inside an iFrame.
Load your audio buffers and do whatever you do to play them.
CLEARING STAGE
Call sourceNode.stop on all of the playing nodes you have reference to.
Call source.disconnect(); on all source nodes.
Call gainNode.disconnect(); on all gain nodes those source nodes are associated with (and whatever other kind of WAA nodes you might be using that have a disconnect method)
Set all referenced gainNodes and sourceNodes to null;
Null out any buffers you have referenced both decoded and your xhr fetched encoded audiobuffers;
KEY: Within the WAA page call audio_context.close(); then set audio_context=null; (this can be done from the parent of the iFrame using contentWindow).
Note: Some of these nulling steps may not be absolutely necessary, however this approach has worked for me.
RE-LOADING STAGE
Reload the iframe from the parent page. This will cause all of the audiobuffers to be garbage collected ON THE NEXT GC ROUND, including the ones in the hidden (non JS) areas of memory.
Your iframe will have to reinstantiate the web audio context and load its buffers and create nodes etc. just as you did when you first loaded it.
Notes: You must decide when you are going to use this clearing method (e.g. after so many buffers have been loaded and played). You can do it without an iframe, but you may have to reload the page once or twice to get garbage collection to fire. This is a pragmatic workaround for those who need to load lots of Web Audio API audio buffers on memory thin platforms like Chromecast or other mobile devices.
FROM PARENT
function hack_memory_management() {
var frame_player = document.getElementById("castFrame");
//sample is the object which holds an audio_context
frame_player.contentWindow.sample.clearBuffers();
setTimeout(function () {
frame_player.contentWindow.location.reload();
}, 1000);
}
INSIDE WAA IFRAME
CrossfadeSample.prototype.clearBuffers = function () {
console.log("CLEARING ALL BUFFERS -IT'S UP TO GC NOW'");
// I have four of each thing because I am doing four part harmony
// these are the decoded audiobuffers used to be passed to the source nodes
this.soprano = null;
this.alto = null;
this.tenor = null;
this.bass = null;
if (this.ctl1) {
//these are the control handles which hold a source node and gain node
var offName = 'stop';
this.ctl1.source[offName](0);
this.ctl2.source[offName](0);
this.ctl3.source[offName](0);
this.ctl4.source[offName](0);
// MAX GARGABE COLLECTION PARANOIA
//disconnect all source nodes
this.ctl1.source.disconnect();
this.ctl2.source.disconnect();
this.ctl3.source.disconnect();
this.ctl4.source.disconnect();
//disconnect all gain nodes
this.ctl1.gainNode.disconnect();
this.ctl2.gainNode.disconnect();
this.ctl3.gainNode.disconnect();
this.ctl4.gainNode.disconnect();
// null out all source and gain nodes
this.ctl1.source = null;
this.ctl2.source = null;
this.ctl3.source = null;
this.ctl4.source = null;
this.ctl1.gainNode = null;
this.ctl2.gainNode = null;
this.ctl3.gainNode = null;
this.ctl4.gainNode = null;
}
// null out the controls
this.ctl1 = null;
this.ctl2 = null;
this.ctl3 = null;
this.ctl4 = null;
// close the audio context
if (this.audio_context) {
this.audio_context.close();
}
// null the audio context
this.audio_context = null;
};
Update:
Sadly, even this does not reliably work and Chromecast can still crash given a few clear and loads of new mp3s. See "My present solution" elsewhere on this page.

Can you maybe use multiple audio-tags on Chromecast when you route each of them into the Web Audio graph (by using a MediaElementAudioSourceNode)?

My present solution
I could not find a final satisfactory solution for Chromecast using the Web Audio API and simultaneous playback of four mp3s - used for four part harmony. The 2nd Gen seems to simply not have enough resources to hold the audiobuffers and simultaneously decode four mp3 files using decodeAudioData without leaving too much garbage around and eventually crashing. I decided to go with surikov's webaudiofont which is built on top of the Web Audio API, and to use midi files. I never had a problem on desktop browsers or other devices with more resources, but I have to have it work on Chromecast. I have no problems at all now using webaudiofont.

I was facing the same problem. What eventually worked for me was to disconnected and delete all connected resources:
if (this.source) {
this.source.disconnect()
delete this.source
}
if (this.gain) {
this.gain.disconnect()
delete this.gain
}
await this.audioContext.close()
delete this.audioContext
delete this.audioBuffer
Just closing the audioContext is not enough. It seems that references will continue to exist preventing garbage collection.

A lot of answers I have seen seem to overcomplicate this. I have run into this same issue while rebuilding an audio system for an application I'm building, but then I realised it previously was not an issue, this was because everytime I played a new audio I closed the previous AudioContext and used the variable it was referenced in for a new AudioContext.
This means that the only two things one has to do to clear this overly memory usage is to use AudioContext.close(), and remove references to it, disconnecting nodes and such is not required.

Programmatically check and update only if an image has changed

I have an application which updates an image from time to time. The update interval is not predictable. The image itself is updated atomically on the web server via rename(). That is all this application does and there shall be no change on the Apache side such that the webserver can continue to only serve static files.
There is some AJAX script which displays the content and updates this image when it is changed. This is done using polling. The naive JavaScript version used a counter and updated pulled the image each second or so by adding a query timestamp. However 99% of the time this pulls the image unchanged.
The current not so naive version uses XMLHttpRequest aka. AJAX to check the If-Modified-Since-header, and if a change is detected the update is invoked.
The question now is, is there a better way to archive this effect? Perhaps look at the last paragraph of this text before you dive into this ;)
Here are the core code snippets of the current version. Please note that the code is edited for brevity, so var initialization left away and some lines removed which are not of interest.
First the usual, slightly extended AJAX binding:
// partly stolen at http://snippets.dzone.com/posts/show/2025
function $(e){if(typeof e=='string')e=document.getElementById(e);return e};
ajax={};
ajax.collect=function(a,f){var n=[];for(var i=0;i<a.length;i++){var v=f(a[i]);if(v!=null)n.push(v)}return n};
ajax.x=function(){try{return new XMLHttpRequest()}catch(e){try{return new ActiveXObject('Msxml2.XMLHTTP')}catch(e){return new ActiveXObject('Microsoft.XMLHTTP')}}};
ajax.send=function(u,f,m,a,h){var x=ajax.x();x.open(m,u,true);x.onreadystatechange=function(){if(x.readyState==4)f(x.responseText,x,x.status==0?200:x.status,x.getResponseHeader("Last-Modified"))};if(m=='POST')x.setRequestHeader('Content-type','application/x-www-form-urlencoded');if(h)h(x);x.send(a)};
ajax.get=function(url,func){ajax.send(url,func,'GET')};
ajax.update=function(u,f,lm){ajax.send(u,f,'GET',null,lm?function(x){x.setRequestHeader("If-Modified-Since",lm)}:lm)};
ajax.head=function(u,f,lm){ajax.send(u,f,'HEAD',null,lm?function(x){x.setRequestHeader("If-Modified-Since",lm)}:lm)};
The basic HTML part, it includes 2 images which are flipped after loading, and a third one (not referenced in the code snippets) to display archived versions etc., which prevents flipping the updates as well:
</head><body onload="init()">
<div id="shower"><img id="show0"/><img id="show1"/><img id="show2"/></div>
The initial part includes the timer. It is a bit more to it, to compensate for network delays on slow links, reduce the polling rate etc.:
function init()
{
window.setInterval(timer,500);
for (var a=2; --a>=0; )
{
var o=$("show"+a);
o.onload = loadi;
}
disp(0);
}
function disp(n)
{
shown=n;
window.setTimeout(disp2,10);
}
function disp2()
{
hide("show0");
hide("show1");
hide("show2");
show("show"+shown);
}
function hide(s)
{
$(s).style.display="none";
}
function show(s)
{
$(s).style.display="inline";
}
function timer(e)
{
if (waiti && !--waiti)
dorefresh();
nextrefresh();
}
function nextrefresh()
{
if (sleeps<0)
sleeps = sleeper;
if (!--sleeps)
pendi = true;
if (pendi && !waiti)
dorefresh();
}
From time to time dorefresh() is called to pull the HEAD, tracking If-Modified-Since:
function dorefresh()
{
waiti = 100; // allow 50s for this request to take
ajax.head("test.jpg",checkrefresh,lm);
}
function checkrefresh(e,x,s,l)
{
if(!l)
{
// not modified
lmc++;
waiti = 0;
}
else
{
lmc=0;
lm=l;
$("show"+loadn).src = "test.jpg?"+stamp();
waiti=100;
}
pendi=false;
sleeper++;
if (sleeper>maxsleep)
sleeper = maxsleep;
sleeps=0;
nextrefresh();
}
function stamp()
{
return new Date().getTime();
}
When the image is loaded it is flipped into view. shown usually is 0 or 1:
function loadi()
{
waiti=0;
$("show"+loadn).style.opacity=1;
if (shown<2)
disp(loadn);
loadn=1-loadn;
}
Please note that I only tested this code with Webkit based browsers yet.
Sorry, I cannot provide a working example, as my update source is non-public.
Also please excuse that the code is somewhat quick-n-dirty quality.
Strictly speaking HEAD alone is enough, we could look at the Last-Modified header of course.
But this recipe here also works for GET requests in a non-image situation.
AJAX GET in combination with images makes less sense, as this pulls the image as binary data.
I could convert that into some inline image, of course, but on bigger images (like mine) this will exceed the maximum URL length.
One thing which possibly can be done is using the browser cache.
That is pull the image using an ajax.update and then re-display the image from the cache.
However this depends on the cache strategy of a browser. On mobile devices the image might be too big to be cached, in that case it is transferred twice. This is wrong as usually mobile devices have slow and more important expensive data links.
We could use this method if the webserver would write a text file, like JSON or a JS snippet, which then is used to display the image.
However the nice thing about this code here is, that you do not need to provide additional information.
So no race conditions, no new weird states like in disk full cases, just works.
So one basic idea is to not alter the code on the webserver which generates the picture, just do it on the browser side.
This way all you need is a softlink from the web tree to the image and make sure, the image is atomically updated.
The downside of AJAX is the same origin policy, so AJAX can only check the HEAD of resources from the host which provided the running JavaScript code.
Greasemonkey or Scriptlets can circumvent that, but these cannot be deployed to a broad audience.
So foreign resources (images) sadly cannot be efficiently queried if they were updated or not.
At my side luclily both, the script and the image, originate from the same host.
Having said this all, here are the problems with this code:
The code above adds to the delay. First the HEAD is checked and if this shows that something has changed the update is done.
It would be nice to do both in one request, such that the update of the image does not require an additional roundtrip.
GET can archive that with If-Modified-Since, and it works on my side, however I found no way to properly display the result as an inlined image. It might work for some browsers, but not for all.
The code also is way too complex for my taste. You have to deal with possible network timeouts, not overwhelming limited bandwidth, trying to be friendly to the webserver, being compatible to as many browsers as possible, and so on.
Also I would like to get rid of the hack to use a query parameter just to pull an updated image, as this slowly fills/flushes the cache.
Perhaps there is an - unknown to me - way to "re-trigger" image refresh in the browser?
This way the browser could check with If-Modified-Since directly and update the image.
With JavaScript this could trigger a .load event then or similar.
At my side I even do not need that at all, all I want is to keep the image somewhat current.
I did not experiment with CANVAS yet. Perhaps somebody has an idea using that.
So my question just is, is there any better way (another algorithm) than shown above, except from improving code quality?

From what I understand, you have 2 sources of information on the server: the image itself and time of last update. Your solution polls on both channels and you want to push, right?
Start here: http://en.wikipedia.org/wiki/Comet_(programming), there should be a simple way to let the server update the client on a new image url. In case server and client support websockets it's a shortcut.
However, most simple solution assumes no image url change and runs
image.src = "";
image.src = url;
by using setInterval() and let the browser deal with cache and headers.

I have a JW Player implementation that won't work 'locally'

Users run my HTML files locally, straight from a CD.
I want to allow them to choose a bunch of videos and create a playlist on the fly.
This works very well if I run a web server but when I run the HTML itself it fails.
The player is created (using swfobject) and all my other code runs but playerReady never fires so I can never get the current play list to add to it.
Any ideas on how I can fix this or, more likely, work around it?

If the player is created, but you're not getting a playerReady, one of two things could be happening.
There's another playerReady on the page that's catching your playerReady. Make sure that there's just one playerReady on the page.
You haven't enabled JavaScript access for Flash. The code for that would look like this:
SWFObject:
var so = new SWFObject('player.swf','ply','470','320','9','#000000');
so.addParam('allowfullscreen','true');
so.addParam('allowscriptaccess','always');
so.addParam('wmode','opaque');
so.addVariable('file','video.flv');
so.write('mediaspace');
I should also note that there are some additional Flash security restrictions because you're accessing the player from disk. Namely, you can't access both a disk source and a network source (the Internet) simultaneously.
Best,
Zach
Developer, LongTail Video

HTML5 video currentTime and buffering

I have a buffering issue.
On my page (which I access remotely), I have a video that I position using the currentTime attribute.
However, it seems like, despite the fact that I do not autobuffer, the video is still buffered until this position.
Is that correct?
Is there a way I can seek to this position without reading the video until this point in time?

Assuming you are referencing the video asset by http, you probably need to enable pseudo-streaming on your webserver, otherwise the entire video will be transferred even though the browsers might be sending the &start= parameters on the request.
Take a look here: http://www.longtailvideo.com/support/jw-player/jw-player-for-flash-v4/12207/video-delivery-http-pseudo-streaming
and another comment here: http://www.longtailvideo.com/support/blog/11887/html5-video-not-quite-there-yet