In the browser, I want to capture the stream of an audio tag that has an .mp3 as source, then send it live via WebRTC to the server. I don't want to hear it via the speakers.
Is it possible to call audioElement.play() without having speaker output?
new Audio() returns an HTMLAudioElement that connects to your browser's default audio output device. You can verify this in the dev console by running:
> new Audio().sinkId
<- ""
where the empty string output specifies the user agent default sinkId.
A flexible way of connecting the output of an HTMLAudioElement instance to non-default sink (for example, if you don't want to hear it through the speakers but just want to send it to another destination like a WebRTC peer connection), is to use the global AudioContext object to create a new MediaStreamAudioDestinationNode. Then you can grab the MediaElementAudioSourceNode from the Audio object holding your mp3 file via audioContext.createMediaElementSource(mp3Audio), and connect that to your new audio destination node. Then, when you run mp3Audio.play(), it will stream only to the destination node, and not the default (speaker) audio output.
Full example:
// Set up the audio node source and destination...
const mp3FilePath = 'testAudioSample.mp3'
const mp3Audio = new Audio(mp3FilePath)
const audioContext = new AudioContext()
const mp3AudioSource = audioContext.createMediaElementSource(mp3Audio)
const mp3AudioDestination = audioContext.createMediaStreamDestination()
mp3AudioSource.connect(mp3AudioDestination)
// Connect the destination track to whatever you want,
// e.g. another audio node, or an RTCPeerConnection.
const mp3AudioTrack = mp3AudioDestination.stream.getAudioTracks()[0]
const pc = new RTCPeerConnection()
pc.addTrack(track)
// Prepare the `Audio` instance playback however you'd like.
// For example, loop it:
mp3Audio.loop = true
// Start streaming the mp3 audio to the new destination sink!
await mp3Audio.play()
It seems that one can mute the audio element and still capture the stream:
audioElement.muted = true;
var stream = audioElement.captureStream();
Related
I would like to modify the microphone input in real-time using an AudioWorklet, in a chrome extension.
For a simple example, I just want to transmit an oscillator wave, like in this plnkr, where one is transmitted to the output.
I first create the worklet:
await audioCtx.audioWorklet.addModule(WORKLET_PATH);
this.worklet = new AudioWorkletNode(audioCtx, 'oscillator');
this.worklet.connect(audioCtx.destination);
Then, I connect a microphone:
this.microphoneStream = await navigator.mediaDevices.getUserMedia({audio: true});
this.microphone = audioCtx.createMediaStreamSource(this.microphoneStream);
console.log('microphone', this.microphone.mediaStream.getTracks()[0].label);
this.microphone.connect(this.worklet as AudioWorkletNode);
And in the example plnkr, in processor.js, instead of modifying the output I modify the input, setting line 15 to be:
const output = inputs[0];
But unfortunately, that does not modify the microphone. If I just playback the microphone audio, I can't hear the oscillator
I want to mix different audio media streams in to one stream. I'm been doing this with Web Audio audiocontext and createMediaStreamSource.
But the final mixed audio is stuttering.
Have anyone an idea how to optimize this to avoid stuttering?
// init audio context
var audioContext = new AudioContext({ latencyHint: 0 });
var audioDestination = audioContext.createMediaStreamDestination();
// add audio streams
audioContext.createMediaStreamSource(audioStream1).connect(audioDestination);
audioContext.createMediaStreamSource(audioStream2).connect(audioDestination);
audioContext.createMediaStreamSource(audioStream3).connect(audioDestination);
audioContext.createMediaStreamSource(audioStream4).connect(audioDestination);
// get mixed audio stream tracks
var audioTrack = audioDestination.stream.getTracks()[0];
// get video track
var videoTrack = videoStream.getTracks()[0];
// combine video and audio tracks into single stream.
var finalStream = new MediaStream([videoTrack, audioTrack]);
// assign to video element
el_video.srcObject = finalStream;
You could try setting the latencyHint to 'playback' like this:
const audioContext = new AudioContext({ latencyHint: 'playback' });
This allows the browser to add a bit of latency to the audio graph which can help on underpowered devices. Setting the latencyHint to 0 on the other hand will tell the browser that it should do things as fast as possible which increases the risk of dropouts.
Having said that, the latencyHint is only a hint. The browser may very well ignore it. You can check what the browser is actually doing by inspecting the baseLatency property.
console.log(audioContext.baseLatency);
I have an audioContext that gets its media from createMediaElementSource. I want to parse this audio on the go into AudioBuffers or something similar that I can send over to another client over websockets.
let audioElement = document.querySelector('video')
let audioContext = new window.AudioContext()
let source = audioContext.createMediaElementSource(audioElement)
source.connect(deliverToOtherClientOrSomething)
I tried making a AudioWorkletNode, but the problem with this approach is that it doesn't allow me to end the chain there, but forces me to forward the audio to some other AudioContext element, which is unwanted.
So, in the end this problem was solved by using an audio worklet node. When creating an AudioWorkletNode it is possible to pass options to it. One of the options is numberOfOutputs. By doing this my question is completely answered.
Mainfile
const sendProcessor = new AudioWorkletNode(audioContext, 'send-processor', {numberOfOutputs:0})
sendProcessor.port.onmessage = (event) => {
callback(event.data);
}
Processor file
process(inputs, outputs) {
this.port.postMessage(inputs[0][0]);
return true;
}
Trying to follow the example here, which is basically a c&p of this
Think I got most of the parts down, except all the node.connect()'s
From what I understand, this sequence of code is needed to provide the audio analyzer with an audio stream:
var source = audioCtx.createMediaStreamSource(stream);
source.connect(analyser);
analyser.connect(audioCtx.destination);
I can't seem to make sense of it as it looks rather ouroboros-y to me.
And unfortunately, I can't seem to find any documentation on .connect() so quite lost and would appreciate any clarification!
Oh and I'm loading an .mp3 via pure javascript new Audio('db.mp3').play(); and am trying to use that as the source without creating an <audio> element.
Can a mediaStream object be created from this to feed into .createMediaStreamSource(stream)?
connect simply defines the output for the filters.
In this case, your source loads the stream into the buffer and writes to the input of the next filter which is defined by the connect function. This is repeated for your analyser filter.
Think of it as pipes.
here is a sample code snippet that I have written a few years back using web audio api.
this.scriptProcessor = this.audioContext.createScriptProcessor(this.scriptProcessorBufferSize,
this.scriptProcessorInputChannels,
this.scriptProcessorOutputChannels);
this.scriptProcessor.connect(this.audioContext.destination);
this.scriptProcessor.onaudioprocess = updateMediaControl.bind(this);
//Set up the Gain Node with a default value of 1(max volume).
this.gainNode = this.audioContext.createGain();
this.gainNode.connect(this.audioContext.destination);
this.gainNode.gain.value = 1;
sewi.AudioResourceViewer.prototype.playAudio = function(){
if(this.audioBuffer){
this.source = this.audioContext.createBufferSource();
this.source.buffer = this.audioBuffer;
this.source.connect(this.gainNode);
this.source.connect(this.scriptProcessor);
this.beginTime = Date.now();
this.source.start(0, this.offset);
this.isPlaying = true;
this.controls.update({playing: this.isPlaying});
updateGraphPlaybackPosition.call(this, this.offset);
}
};
So as you can see that my source is connected to a gainNode, which is connected to a scriptProcessor. When the audio starts playing, the data is passed from the source->gainNode->destination and source->scriptProcessor->destination. flowing through the "pipes" that connects them, which is defined by connect(). When the audio data pass through the gainNode, volume can be adjusted by changing the amplitude of the audio wave. After that it is passed to the script processor so that events can be attached and triggered while the audio is being processed.
In javascript, How can I connect an audio context to a video fetching its data from a blob (the video uses the MediaStream capabilities). No matter what I do the audio context returns an empty buffer. Is there any way to connect the two?
Probably, createMediaElementSource is not the right kind of processing node for this use-case.
Rather, you better off to use createMediaStreamSource node from WebAudio API in case you are trying to handle audio live stream, not fixed media source.
The createMediaStreamSource() method of the AudioContext Interface is used to create a new MediaStreamAudioSourceNode object, given a media stream (say, from a navigator.getUserMedia instance), the audio from which can then be played and manipulated.
The link has a more detailed example. However, the main difference for this MediaStreamAudioSourceNode is it can be created only using a MediaStream that you get from media-server or locally(through getUserMedia). In my experience, i couldn't find any way by using only the blob url from the <video> tag.
While this is an old question, I've searched for something similar and found a solution I want to share.
To connect the Blob, you may use a new Response instance. Here is an example for creating a wave form visualizer.
var audioContext = new (window.AudioContext || window.webkitAudioContext)();
var analyser = audioContext.createAnalyser();
var dataArray = new Uint8Array(analyser.frequencyBinCount);
var arrayBuffer = await new Response(yourBlob).arrayBuffer();
var audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
var source = audioContext.createBufferSource();
source.buffer = audioBuffer;
source.connect(analyser);
source.start(0);
Note: yourBlob needs to be a Blob instance.
You may find this fiddle usefull which records video and audio for 5 seconds, turns the recording into a Blob and than plays it back including audio wave visualization.