I would like to modify the microphone input in real-time using an AudioWorklet, in a chrome extension.
For a simple example, I just want to transmit an oscillator wave, like in this plnkr, where one is transmitted to the output.
I first create the worklet:
await audioCtx.audioWorklet.addModule(WORKLET_PATH);
this.worklet = new AudioWorkletNode(audioCtx, 'oscillator');
this.worklet.connect(audioCtx.destination);
Then, I connect a microphone:
this.microphoneStream = await navigator.mediaDevices.getUserMedia({audio: true});
this.microphone = audioCtx.createMediaStreamSource(this.microphoneStream);
console.log('microphone', this.microphone.mediaStream.getTracks()[0].label);
this.microphone.connect(this.worklet as AudioWorkletNode);
And in the example plnkr, in processor.js, instead of modifying the output I modify the input, setting line 15 to be:
const output = inputs[0];
But unfortunately, that does not modify the microphone. If I just playback the microphone audio, I can't hear the oscillator
Related
I want to mix different audio media streams in to one stream. I'm been doing this with Web Audio audiocontext and createMediaStreamSource.
But the final mixed audio is stuttering.
Have anyone an idea how to optimize this to avoid stuttering?
// init audio context
var audioContext = new AudioContext({ latencyHint: 0 });
var audioDestination = audioContext.createMediaStreamDestination();
// add audio streams
audioContext.createMediaStreamSource(audioStream1).connect(audioDestination);
audioContext.createMediaStreamSource(audioStream2).connect(audioDestination);
audioContext.createMediaStreamSource(audioStream3).connect(audioDestination);
audioContext.createMediaStreamSource(audioStream4).connect(audioDestination);
// get mixed audio stream tracks
var audioTrack = audioDestination.stream.getTracks()[0];
// get video track
var videoTrack = videoStream.getTracks()[0];
// combine video and audio tracks into single stream.
var finalStream = new MediaStream([videoTrack, audioTrack]);
// assign to video element
el_video.srcObject = finalStream;
You could try setting the latencyHint to 'playback' like this:
const audioContext = new AudioContext({ latencyHint: 'playback' });
This allows the browser to add a bit of latency to the audio graph which can help on underpowered devices. Setting the latencyHint to 0 on the other hand will tell the browser that it should do things as fast as possible which increases the risk of dropouts.
Having said that, the latencyHint is only a hint. The browser may very well ignore it. You can check what the browser is actually doing by inspecting the baseLatency property.
console.log(audioContext.baseLatency);
I'm building an application using PizzicatoJS + HowlerJS. Those libraries essentially allow me to play multiple audio files at the same time. Imagine a 4 audio tracks with each track containing an instrument like guitar, bass, drums, vocals, etc..
Everything plays fine when using PizzicatoJS's Group functionality or running a forEach loop on all my Howl sounds and firing .play(). However, I would like to download the final resulting sound I am hearing from my speakers. Any idea on how to approach that?
I looked into OfflineAudioContext, but I am unsure on how to use it to generate an audio file. It looks like it needs an Audio source like an <audio> tag. Is what I'm trying to do possible? Any help is appreciated.
I think the OfflineAudioContext can help with your use case.
Let's say you want to create a file with a length of 10 seconds. It should contain one sound playing from the start up to second 8. And there is also another sound which is supposed to start at second 5 and should last until the end. Both sounds are AudioBuffers (named soundBuffer and anotherSoundBuffer) already.
You could arrange and combine the sounds as follows.
const sampleRate = 44100;
const offlineAudioContext = new OfflineAudioContext({
length: sampleRate * 10,
sampleRate
});
const soundSourceNode = new AudioBufferSourceNode({
buffer: soundBuffer
});
soundSourceNode.start(0);
soundSourceNode.stop(8);
soundSourceNode.connect(offlineAudioContext.destination);
const anotherSoundSourceNode = new AudioBufferSourceNode({
buffer: anotherSoundBuffer
});
anotherSoundSourceNode.start(5);
anotherSoundSourceNode.stop(10);
anotherSoundSourceNode.connect(offlineAudioContext.destination);
offlineAudioContext
.startRendering()
.then((audioBuffer) => {
// save the resulting buffer as a file
});
Now you can use a library to turn the resulting AudioBuffer into an encoded audio file. One library which does that is for example audiobuffer-to-wav.
I am trying to stream Audio via Websocket.
I can get an AudioBuffer from the Microphone (or other Source) via Web-Audio-Api and stream the RAW-Audio-Buffer, but i think this would not be very efficient.
So i looked arround to encode the AudioBuffer somehow. - If the Opus-Codec would not be practicable,
i am open to alternatives and thankful for any hints in the right direction.
I have tried to use the MediaRecorder (from MediaStreamRecording-API) but it seems not possible to stream with that API, instead of plain recording.
Here is the Part how i get the RAW-AudioBuffer:
const handleSuccess = function(stream) {
const context = new AudioContext();
const source = context.createMediaStreamSource(stream);
const processor = context.createScriptProcessor(16384, 1, 1);
source.connect(processor);
processor.connect(context.destination);
processor.onaudioprocess = function(e) {
bufferLen = e.inputBuffer.length
const inputBuffer = new Float32Array(bufferLen);
e.inputBuffer.copyFromChannel(inputBuffer, 0);
let data_to_send = inputBuffer
//And send the Float32Array ...
}
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(handleSuccess);
So the Main Question is: How can i encode the AudioBuffer.
(and Decode it at the Receiver)
Is there an API or Library? Can i get the encoded Buffer from another API in the Browser?
The Web Audio API has a MediaStreamDestination node that will expose a .stream MediaStream that you can then pass through the WebRTC API.
But if you are only dealing with a microphone input, then pass directly that MediaStream to WebRTC, no need for the Web Audio step.
Ps: for the ones that only want to encode to opus, then MediaRecorder is currently the only native way. It will incur a delay, will generate a webm file, not only the raw data, and will process the data no faster than real-time.
Only other options now are to write your own encoders and run it in WabAssembly.
Hopefully in a near future, we'll have access to the WebCodecs API which should solve this use case among others.
In the browser, I want to capture the stream of an audio tag that has an .mp3 as source, then send it live via WebRTC to the server. I don't want to hear it via the speakers.
Is it possible to call audioElement.play() without having speaker output?
new Audio() returns an HTMLAudioElement that connects to your browser's default audio output device. You can verify this in the dev console by running:
> new Audio().sinkId
<- ""
where the empty string output specifies the user agent default sinkId.
A flexible way of connecting the output of an HTMLAudioElement instance to non-default sink (for example, if you don't want to hear it through the speakers but just want to send it to another destination like a WebRTC peer connection), is to use the global AudioContext object to create a new MediaStreamAudioDestinationNode. Then you can grab the MediaElementAudioSourceNode from the Audio object holding your mp3 file via audioContext.createMediaElementSource(mp3Audio), and connect that to your new audio destination node. Then, when you run mp3Audio.play(), it will stream only to the destination node, and not the default (speaker) audio output.
Full example:
// Set up the audio node source and destination...
const mp3FilePath = 'testAudioSample.mp3'
const mp3Audio = new Audio(mp3FilePath)
const audioContext = new AudioContext()
const mp3AudioSource = audioContext.createMediaElementSource(mp3Audio)
const mp3AudioDestination = audioContext.createMediaStreamDestination()
mp3AudioSource.connect(mp3AudioDestination)
// Connect the destination track to whatever you want,
// e.g. another audio node, or an RTCPeerConnection.
const mp3AudioTrack = mp3AudioDestination.stream.getAudioTracks()[0]
const pc = new RTCPeerConnection()
pc.addTrack(track)
// Prepare the `Audio` instance playback however you'd like.
// For example, loop it:
mp3Audio.loop = true
// Start streaming the mp3 audio to the new destination sink!
await mp3Audio.play()
It seems that one can mute the audio element and still capture the stream:
audioElement.muted = true;
var stream = audioElement.captureStream();
Trying to follow the example here, which is basically a c&p of this
Think I got most of the parts down, except all the node.connect()'s
From what I understand, this sequence of code is needed to provide the audio analyzer with an audio stream:
var source = audioCtx.createMediaStreamSource(stream);
source.connect(analyser);
analyser.connect(audioCtx.destination);
I can't seem to make sense of it as it looks rather ouroboros-y to me.
And unfortunately, I can't seem to find any documentation on .connect() so quite lost and would appreciate any clarification!
Oh and I'm loading an .mp3 via pure javascript new Audio('db.mp3').play(); and am trying to use that as the source without creating an <audio> element.
Can a mediaStream object be created from this to feed into .createMediaStreamSource(stream)?
connect simply defines the output for the filters.
In this case, your source loads the stream into the buffer and writes to the input of the next filter which is defined by the connect function. This is repeated for your analyser filter.
Think of it as pipes.
here is a sample code snippet that I have written a few years back using web audio api.
this.scriptProcessor = this.audioContext.createScriptProcessor(this.scriptProcessorBufferSize,
this.scriptProcessorInputChannels,
this.scriptProcessorOutputChannels);
this.scriptProcessor.connect(this.audioContext.destination);
this.scriptProcessor.onaudioprocess = updateMediaControl.bind(this);
//Set up the Gain Node with a default value of 1(max volume).
this.gainNode = this.audioContext.createGain();
this.gainNode.connect(this.audioContext.destination);
this.gainNode.gain.value = 1;
sewi.AudioResourceViewer.prototype.playAudio = function(){
if(this.audioBuffer){
this.source = this.audioContext.createBufferSource();
this.source.buffer = this.audioBuffer;
this.source.connect(this.gainNode);
this.source.connect(this.scriptProcessor);
this.beginTime = Date.now();
this.source.start(0, this.offset);
this.isPlaying = true;
this.controls.update({playing: this.isPlaying});
updateGraphPlaybackPosition.call(this, this.offset);
}
};
So as you can see that my source is connected to a gainNode, which is connected to a scriptProcessor. When the audio starts playing, the data is passed from the source->gainNode->destination and source->scriptProcessor->destination. flowing through the "pipes" that connects them, which is defined by connect(). When the audio data pass through the gainNode, volume can be adjusted by changing the amplitude of the audio wave. After that it is passed to the script processor so that events can be attached and triggered while the audio is being processed.