I have an audioContext that gets its media from createMediaElementSource. I want to parse this audio on the go into AudioBuffers or something similar that I can send over to another client over websockets.
let audioElement = document.querySelector('video')
let audioContext = new window.AudioContext()
let source = audioContext.createMediaElementSource(audioElement)
source.connect(deliverToOtherClientOrSomething)
I tried making a AudioWorkletNode, but the problem with this approach is that it doesn't allow me to end the chain there, but forces me to forward the audio to some other AudioContext element, which is unwanted.
So, in the end this problem was solved by using an audio worklet node. When creating an AudioWorkletNode it is possible to pass options to it. One of the options is numberOfOutputs. By doing this my question is completely answered.
Mainfile
const sendProcessor = new AudioWorkletNode(audioContext, 'send-processor', {numberOfOutputs:0})
sendProcessor.port.onmessage = (event) => {
callback(event.data);
}
Processor file
process(inputs, outputs) {
this.port.postMessage(inputs[0][0]);
return true;
}
Related
I'm stuck with a problem in which whenever I pass the stream from createMediaStreamDestination to an audio element srcObject, no audio is being played. My implementation is based off of the response posted here Combine setSinkId with stereoPanner?
Initially, I have an audio element in which I isolate the sound so that it would only play from the left speaker
const audio = document.createElement('audio');
audio.src = audioUrl;
let audioContext = new AudioContext();
let source = audioContext.createMediaElementSource(audio);
let panner = audioContext.createStereoPanner();
let destination = audioContext.destination;
panner.pan.value = -1;
source.connect(panner).connect(destination);
The above plays sound fine when I add audio.play() but I want to be able to set specifically the speakers that the audio would play out of while keeping the panner changes. Since audioContext doesn't contain any possibility of setting the sinkId yet, I created a new audio element and mediastreamdestination and passed the mediaStream into the source object
const audio = document.createElement('audio');
audio.src = audioUrl;
let audioContext = new AudioContext();
let source = audioContext.createMediaElementSource(audio);
let panner = audioContext.createStereoPanner();
let destination = audioContext.createMediaStreamDestination();
panner.pan.value = -1;
source.connect(panner).connect(destination);
const outputAudio = new Audio();
outputAudio.srcObject = destination.stream;
outputAudio.setSinkId(audioSpeakerId);
outputAudio.play();
With the new code, however, when I start up my application, the outputAudio doesn't play any sound at all. Is there anything wrong with my code that is causing the outputAudio element not to play sound? I'm fairly new to web audio api and I tried implementing the code from the mentioned stackoverflow thread but it doesn't seem to be working for me. Any help would be appreciated!
In the description of your first code block you mention that you additionally also call audio.play() to start the audio. That's also necessary for the second code block to work. You need to start both audio elements.
Generally calling play() on an audio element and creating a new AudioContext should ideally happen in response to a user action to make sure the browser's autoplay policy doesn't block the audio.
If all goes well the state of your AudioContext should be "running".
I've worded my title, and tags in a way that should be searchable for both video and audio, as this question isn't specific to one. My specific case only concerns audio though, so my question body will be written specific to that.
First, the big picture:
I'm sending audio to multiple P2P clients who will connect and disconnect a random intervals. The audio I'm sending is a stream, but each client only needs the part of the stream from whence they connected. Here's how I solved that:
Every {timeout} (e.g. 1000ms), create a new audio blob
Blob will be a full audio file, with all metadata it needs to be playable
As soon as a blob is created, convert to array buffer (better browser support), and upload to client over WebRTC (or WebSockets if they don't support)
That works well. There is a delay, but if you keep the timeout low enough, it's fine.
Now, my question:
How can I play my "stream" without having any audible delay?
I say stream, but I didn't implement it using the Streams API, it is a queue of blobs, that gets updated every time the client gets new data.
I've tried a lot of different things like:
Creating a BufferSource, and merging two blobs (converted to audioBuffers) then playing that
Passing an actual stream from Stream API to clients instead of blobs
Playing blobs sequentially, relying on ended event
Loading next blob while current blob is playing
Each has problems, difficulties, or still results in an audible delay.
Here's my most recent attempt at this:
let firstTime = true;
const chunks = [];
Events.on('audio-received', ({ detail: audioChunk }) => {
chunks.push(audioChunk);
if (firstTime && chunks.length > 2) {
const currentAudio = document.createElement("audio");
currentAudio.controls = true;
currentAudio.preload = 'auto';
document.body.appendChild(currentAudio);
currentAudio.src = URL.createObjectURL(chunks.shift());
currentAudio.play();
const nextAudio = document.createElement("audio");
nextAudio.controls = true;
nextAudio.preload = 'auto';
document.body.appendChild(nextAudio);
nextAudio.src = URL.createObjectURL(chunks.shift());
let currentAudioStartTime, nextAudioStartTime;
currentAudio.addEventListener("ended", () => {
nextAudio.play()
nextAudioStartTime = new Date();
if (chunks.length) {
currentAudio.src = URL.createObjectURL(chunks.shift());
}
});
nextAudio.addEventListener("ended", () => {
currentAudio.play()
currentAudioStartTime = new Date();
console.log(currentAudioStartTime - nextAudioStartTime)
if (chunks.length) {
nextAudio.src = URL.createObjectURL(chunks.shift());
}
});
firstTime = false;
}
});
The audio-received event gets called every ~1000ms. This code works; it plays each "chunk" after the last one was played, but on Chrome, there is a ~300ms delay that's very audible. It plays the first chunk, then goes quiet, then plays the second, so on. On Firefox the delay is 50ms.
Can you help me?
I can try to create a reproducible example if that would help.
I want to make a recording where, I get multiple audio tracks from different mediaStream objects (some of them, remote). Use the getAudioTracks () method and add them to a mediaStream object using addTrack (). At the moment of passing this last object as a parameter for mediaRecorder I realize that it only records the audio track located in position [0]. That gives me to understand that mediaRecorder is capable of recording a track by type, is there any way to join these tracks into one to record them correctly using mediaRecorder? I would be grateful for any page that explains this if possible and if it exists
I was battling with this for a while and took me ages to realise that the MediaStream only ever recorded the first track I added. My solution was to get the Web Audio API involved. This example uses two UserMedia (e.g. a mic & Stereo Mix) and merges them. The UserMedia are identified by their deviceId as shown when you use await navigator.mediaDevices.enumerateDevices().
In summary:
Create an AudioContext()
Get your media using navigator.mediaDevices.getUserMedia()
Add these as a stream source to the AudioContext
Create an AudioContext stream destination object
Connect your sources to this single destination
And your new MediaRecorder() takes this destination as its MediaStream
Now you can record yourself singing along to your favourite song as it streams ;)
const audioContext = new AudioContext();
audioParams_01 = {
deviceId: "default",
}
audioParams_02 = {
deviceId: "7079081697e1bb3596fad96a1410ef3de71d8ccffa419f4a5f75534c73dd16b5",
}
mediaStream_01 = await navigator.mediaDevices.getUserMedia({ audio: audioParams_01 });
mediaStream_02 = await navigator.mediaDevices.getUserMedia({ audio: audioParams_02 });
audioIn_01 = audioContext.createMediaStreamSource(mediaStream_01);
audioIn_02 = audioContext.createMediaStreamSource(mediaStream_02);
dest = audioContext.createMediaStreamDestination();
audioIn_01.connect(dest);
audioIn_02.connect(dest);
const recorder = new MediaRecorder(dest.stream);
chunks = [];
recorder.onstart = async (event) => {
// your code here
}
recorder.ondataavailable = (event) => {
chunks.push(event.data);
}
recorder.onstop = async (event) => {
// your code here
}
Finish using a library built by muazKhan, which allows you to merge the streams and return them in one!
It's incredibly easy!
https://github.com/muaz-khan/MultiStreamsMixer
In the browser, I want to capture the stream of an audio tag that has an .mp3 as source, then send it live via WebRTC to the server. I don't want to hear it via the speakers.
Is it possible to call audioElement.play() without having speaker output?
new Audio() returns an HTMLAudioElement that connects to your browser's default audio output device. You can verify this in the dev console by running:
> new Audio().sinkId
<- ""
where the empty string output specifies the user agent default sinkId.
A flexible way of connecting the output of an HTMLAudioElement instance to non-default sink (for example, if you don't want to hear it through the speakers but just want to send it to another destination like a WebRTC peer connection), is to use the global AudioContext object to create a new MediaStreamAudioDestinationNode. Then you can grab the MediaElementAudioSourceNode from the Audio object holding your mp3 file via audioContext.createMediaElementSource(mp3Audio), and connect that to your new audio destination node. Then, when you run mp3Audio.play(), it will stream only to the destination node, and not the default (speaker) audio output.
Full example:
// Set up the audio node source and destination...
const mp3FilePath = 'testAudioSample.mp3'
const mp3Audio = new Audio(mp3FilePath)
const audioContext = new AudioContext()
const mp3AudioSource = audioContext.createMediaElementSource(mp3Audio)
const mp3AudioDestination = audioContext.createMediaStreamDestination()
mp3AudioSource.connect(mp3AudioDestination)
// Connect the destination track to whatever you want,
// e.g. another audio node, or an RTCPeerConnection.
const mp3AudioTrack = mp3AudioDestination.stream.getAudioTracks()[0]
const pc = new RTCPeerConnection()
pc.addTrack(track)
// Prepare the `Audio` instance playback however you'd like.
// For example, loop it:
mp3Audio.loop = true
// Start streaming the mp3 audio to the new destination sink!
await mp3Audio.play()
It seems that one can mute the audio element and still capture the stream:
audioElement.muted = true;
var stream = audioElement.captureStream();
What I'm trying to achieve is to make Chrome load a video file as data (via the Fetch API, XHR, whatever) and to play it using <video> while it's still being downloaded without issuing two separate requests for the same URL and without waiting until the file is completely downloaded.
It's easy to get a ReadableStream from the Fetch API (response.body), yet I can't find a way to feed it into the video element. I've figured out I need a blob URL for this, which can be created using a MediaSource object. However, the SourceBuffer#appendStream method, which sounds like just what is needed, isn't implemented in Chrome, so I can't connect the stream directly to the MediaSource object.
I can probably read the stream in chunks, create Uint8Arrays out of them, and use SourceBuffer#appendBuffer, but this means playback won't start immediately unless the chunk size is really small. Also it feels like manually doing something that all these APIs should be able to do out of the box. If there is no other solutions, and I go this way, what caveats should I expect?
Are there probably other ways to create a blob URL for a ReadableStream? Or is there a way to make fetch and <video> share a request? There are so many new APIs that I could easily miss something.
After hours of experimenting, found a half-working solution:
const video = document.getElementById('audio');
const mediaSource = new MediaSource();
video.src = window.URL.createObjectURL(mediaSource);
mediaSource.addEventListener('sourceopen', async () => {
const sourceBuffer = mediaSource.addSourceBuffer('audio/webm; codecs="opus"');
const response = await fetch(audioSRC);
const body = response.body
const reader = body.getReader()
let streamNotDone = true;
while (streamNotDone) {
const {value, done} = await reader.read();
if (done) {streamNotDone = false; break;}
await new Promise((resolve, reject) => {
sourceBuffer.appendBuffer(value)
sourceBuffer.onupdateend = (() => {
resolve(true);
})
})
}
});
It works with https://developer.mozilla.org/en-US/docs/Web/API/MediaSource
Also, I tested this only with webm/opus format but I believe it should work with other formats as well as long as you specify it.