Develop Reference

Tone.js audio filters not being heard

I'm trying to add filter effects to an audio stream I have playing on my website. I'm able to connect the Tone.js library to the audio stream but I'm not hearing any changes in the audio stream playing on the website. I'm not seeing any errors in the console and I've tried adjusting the filter from 50 to 5000 but nothing seems to have any effect on the audio. Do I need to set up the new Tone.Player() to actually hear the audio? If so, how do you go about setting up the Player if there is no src for the existing audio element.
$('#testBtn').click(async function () {
const audioElement = document.getElementById('theAudioStream');
const mediaElementSource = Tone.context.createMediaElementSource(audioElement);
const filter = new Tone.Filter(50, 'lowpass').toDestination();
Tone.connect(mediaElementSource, filter);
await Tone.start();
console.log("Started?");
});
The audio stream I'm trying to modify is set up from a JsSip call. The code to start the stream is as follows:
var audioStream = document.getElementById('theAudioStream')
//further down in code
currentSession.answer(options);
if (currentSession.connection) {
currentSession.connection.ontrack = function (e) {
audioStream.srcObject = e.streams[0];
audioStream.play();
}
}
I click the test button after the call has started so I know the audio stream is present before initializing the Tone.js Filters

Working solution:
Removing the audioStream.play() from where the JsSIP call is answered solves the issue.
I don't know the exact reason why this solves the issue (it might even be a workaround) but after much trial and error this way allows the audio to be available to ToneJS for effecting.
Any other solutions are welcome.

Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence

We are working on a Digital Audio Workstation kind of thing in the browser. We need to work with multiple audio files in a tab. We use new Audio(audioUrl) to be able to play the audio in our own audio mixer. It has been working for us up to now.
With the latest version of Chrome (92), we have the problem where the above code snippet causes the following error:
[Intervention] Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27
I cannot access the bug link provided, it says permission denied. And is there a suggested workaround to handle this?
UPDATE:
I moved away from using HTMLAudioElement to AudioBufferSourceNode. Seems like the only straightforward solution as Chrome team is discussing to limit them anyway. Note: We may need more than 1000 audio clips to be played back. This is in reference to the chromium discussion thread where they are going to increase the number of webmediaplayers to 1000 on the next release for August 5 2021.

Chrome 92 has introduced a limit on number of audio and video tags that can be allocated in a particular tab.
75 for desktop browsers and 40 for mobile browsers.
For now the only solution is to limit the number of audio and video tags created in the page. Try reusing the already allocated audio / video elements.
The number can only be increased by passing the following flag when starting up chrome, for example --max-web-media-player-count=5000
(Of course we cannot expect the end user to do this)
Related Source code here:
https://chromium-review.googlesource.com/c/chromium/src/+/2816118
Edit:
Before deallocating the audio/video elements setting the following seems to force clean up of the element.
mediaElement.remove();
mediaElement.srcObject = null;

const MaxWebMediaPlayerCount = 75;
class VideoProducer {
static #documentForVideo
static createVideo() {
if (!this.#documentForVideo || this.#documentForVideo.videoCount === MaxWebMediaPlayerCount) {
const iframeForVideo = document.body.appendChild(document.createElement('iframe'));
iframeForVideo.style.display = 'none';
iframeForVideo.contentDocument.videoCount = 0;
this.#documentForVideo = iframeForVideo.contentDocument;
}
this.#documentForVideo.videoCount++;
const video = this.#documentForVideo.createElement('video');
return video;
}
foo() {
const video = VideoProducer.createVideo();
// ...
}

Yeah me too it broke my game,
This is what I found as a workaround, hope this helps in the mean time:
function playSound( ) {
var jump_sound = new Audio("./jump.mp3");
jump_sound.play();
jump_sound.onended = function(){
this.currentSrc = null;
this.src = "";
this.srcObject = null;
this.remove();
};
}
Note: it still blocks if there's too many concurrent sound but with this code in place the blocking is temporary.

Chrome version 92.0.4515.131 seems to resolve the issue

Adding Audio with JavaScript: FireFox and Chrome show they're playing, but I'm not receiving any sound

My peer and I have a series of JavaScript objects which all have an audio property. The value of that property is a string which provides the relative path to where that audio file is stored on our server.
I wrote a JavaScript function which references those objects and plays their audio files when a new object loads. Everything seemed to be working fine during testing. He just added new audio files today and messaged me that our project wasn't playing audio in IE -- but it still worked in FF and Chrome.
When I checked the error, I noticed IE was getting an invalid state error from the playAudio JavaScript function that I wrote. I made the assumption the mp3 file didn't fully load, so IE was throwing the invalid state error because it couldn't execute the play() command on a file that wasn't available yet. So, tonight I added a media event listener to ensure the function didn't execute until the mp3 resource was available.
That fixed the IE issue, but now both Chrome and FireFox aren't playing sound. And by that I mean, they are receiving the mp3 resource, no errors are being thrown to the console, and they both show the audio icon on the browser's tab above the url bar as though media content were actually playing. It seems like they are playing, but I'm not getting sound through my actual speakers. I'm kind of at a loss as to why this is happening, and will have to look into it more tomorrow -- but I'm hoping someone here might know what could be the cause.
// The playAudio function pulls the audio file from the scenes object and only plays once.
// Afterward, the scenes object receives a new parameter, audio_played, which is set to true.
function playAudio(id) {
// check whether the variable currAudio has been defined.
// If it has, pause currAudio.
if (typeof currAudio !== 'undefined') {
currAudio.pause();
}
// check that the scenes object has an audio parameter
// and that the audio_played parameter is not true
if (scenes[id].audio && scenes[id].audio_played !== true) {
// create a new Audio object using the loaded scene's audio
// parameter and store it in the variable newAudio
var newAudio = new Audio(scenes[id].audio);
// wait for the audio file to load enough data
// to be played, then execute the nested scripts.
newAudio.addEventListener('canplay', function() {
// set newAudio to start at the beginning of the file.
newAudio.currentTime = 0;
// play the audio file
newAudio.play();
// add an audio_played parameter to the
// scenes object and set it's value to true.
scenes[id].audio_played = true;
// update the currAudio variable
var currAudio = newAudio;
}, false);
}
}

Get frame numbers in HTML5 Video

I am trying to capture each frame number of the video however it looks like there is no way of achieving it. So I started my own clock to match the frame numbers of the video but they never match and the difference keeps increasing as the video progress.
Please have a look at my bin. http://jsbin.com/dopuvo/4/edit
I have added the frame number to each frame of the video from Adobe After Effect so I have more accurate information of the difference. The Video is running at 29.97fps and the requestAnimationFrame is also set to increase the number at the same rate, however I am not sure where this difference is coming from.
Sometimes they match and sometimes they don't. I also tried doing it offline but I get the same results. Any help.

I found something on github for this. https://github.com/allensarkisyan/VideoFrame
I have implemented it in this fiddle: https://jsfiddle.net/k0y8tp2v/
var currentFrame = $('#currentFrame');
var video = VideoFrame({
id : 'video',
frameRate: 25,
callback : function(frame) {
currentFrame.html(frame);
}
});
$('#play-pause').click(function(){
if(video.video.paused){
video.video.play();
video.listen('frame');
$(this).html('Pause');
}else{
video.video.pause();
video.stopListen();
$(this).html('Play');
}
});
EDIT: updated fiddle to new video so it works again.
EDIT: As pointed out, the video is 25fps, so I updated it, and while I was there removed reliance on jQuery.
Non jQuery version:
https://jsfiddle.net/k0y8tp2v/1/
var currentFrame = document.getElementById('currentFrame');
var video = VideoFrame({
id : 'video',
frameRate: 25,
callback : function(frame) {
currentFrame.innerHTML = frame ;
}
});
document.getElementById('play-pause').addEventListener('click', function(e){
if(video.video.paused){
video.video.play();
video.listen('frame');
e.target.innerHTML = 'Pause';
}else{
video.video.pause();
video.stopListen();
e.target.innerHTML = 'Play';
}
});

The problem is that setTimeout is not really predictable, so you can't be sure that exactly one new frame has been displayed every time your function runs. You need to check the currentTime of the video every time you update your frame display and multiply that by the frame rate.
Here's a working example: http://jsbin.com/xarekice/1/edit It's off by one frame, but it looks like you may have two frames at the beginning marked "000000".
A few things about the video element that you may want to be aware of:
As you seem to have discovered, there's no reliable way to determine the frame rate, so you have to discover it elsewhere and hard-code it. There are some interesting things going on with video metrics, but they're non-standard, not widely supported and, in my experience, completely ineffective at determining the actual frame rate.
The currentTime is not always exactly representative of what's on the screen. Sometimes it's ahead a little bit, especially when seeking (which is why in my JSBin, I don't update the frame while you're seeking).
I believe currentTime updates on a separate thread from the actual video draw, so it kind of works like it's own clock that just keeps going. It's where the video wants to be, not necessarily where it is. So you can get close, but you need to round the results of the frame calculation, and once in a while, you may be off by one frame.

Starting in M83, Chromium has a requestVideoFrameCallback() API, which might solve your issue.
You can use the mediaTime to get a consistent timestamp, as outlined in this Github issue. From there, you could try something like this:
var frameCounter = (time, metadata) => {
let count = metadata.mediaTime * frameRate;
console.log("Got frame: " + Math.round(count));
// Capture code here.
video.requestVideoFrameCallback(frameCounter);
}
video.requestVideoFrameCallback(frameCounter)
This will only fire on new frames, but you may occasionally miss one (which you can detect from a discontinuity in the metadata.presentedFrames count). You might also be slightly late in capturing the frame (usually 16ms, or one call to window.requestAnimationFrame() later than when the video frame is available).
If you're interested in a high level overview of the API, here's a blogpost, or you can take a look at the API's offical github.

JS .play() on iPad plays wrong file...suggestions?

So, I am building a web app that has a div with text that changes on various user actions (it's stepping through an array of pieces of text). I'm trying to add audio to it, so I made another array with the sound files in the appropriate positions:
var phrases=['Please hand me the awl.','etc1','etc2','etc3'];
var phrasesAudio=['awl.mp3','etc1.mp3','etc2.mp3','etc3.mp3'];
And on each action completion, a 'counter' variable in incremented, and each array looks for the object at that counter
var audio = document.createElement("audio"),
canPlayMP3 = (typeof audio.canPlayType === "function" &&
audio.canPlayType("audio/mpeg") !== "");
function onAction(){
correct++;
document.getElementById('speech').innerHTML=phrases[correct];
if(canPlayMP3){
snd = new Audio(phrasesAudio[correct]);
}
else{
snd = new Audio(phrasesAudioOgg[correct]);
}
snd.play();
}
(the text replaces a div's HTML and I use .play() for the audio object)...usually works okay (and ALWAYS does in a 'real' browser), but on the iPad (the actual target device) after a few successful iterations, the TEXT continues to progress accurately, but the AUDIO will hiccup and repeat a sound file one or more times. I added some logging and looking there it reports that it's playing screw.mp3 (just an example, no particular file is more or less error prone), but in actuality it plays screwdriver.mp3 (the prior file, if there is an error, the audio always LAGS, never LEADS)...because of this I am thinking that the problem is with my use of .play()...so I tried setting snd=null; between each sound, but nothing changed...Any suggestions for how to proceed? This is my first use of audio elements so any advice is appreciated. Thanks.
edit: I've also tried setting the files with snd.src (based on https://wiki.mozilla.org/Audio_Data_API) and this caused no audio to play

for iPad you need to call snd.load() before snd.play() - otherwise you get "odd" behaviour...
see for some insight:
http://jnjnjn.com/187/playing-audio-on-the-ipad-with-html5-and-javascript/
Autoplay audio files on an iPad with HTML5
EDIT - as per comment from the OP:
Here https://developer.mozilla.org/En/Using_audio_and_video_in_Firefox you can find a tip on halting a currently playing piece of media with
var mediaElement = document.getElementById("myMediaElementID");
mediaElement.pause();
mediaElement.src = "";
and, following that with setting the correct src, load(), play() works great