How usergetmedia to use the microphone in chrome and then stream to get raw audio? I need need to get the audio in linear 16.
Unfortunately, the MediaRecorder doesn't support raw PCM capture. (A sad oversight, in my opinion.) Therefore, you'll need to get the raw samples and buffer/save them yourself.
You can do this with the ScriptProcessorNode. Normally, this Node is used to modify the audio data programmatically, for custom effects and what not. But, there's no reason you can't just use it as a capture point. Untested, but try something like this code:
const captureNode = audioContext.createScriptProcessor(8192, 1, 1);
captureNode.addEventListener('audioprocess', (e) => {
const rawLeftChannelData = inputBuffer.getChannelData(0);
// rawLeftChannelData is now a typed array with floating point samples
});
(You can find a more complete example on MDN.)
Those floating point samples are centered on zero 0 and will ideally be bound to -1 and 1. When converting to an integer range, you'll want to clamp values to this range, clipping anything beyond it. (The values can sometimes exceed -1 and 1 in the event loud sounds are mixed together in-browser. In theory, the browser can also record float32 samples from an external sound device which may also exceed that range, but I don't know of any browser/platform that does this.)
When converting to integer, it matters if the values are signed or unsigned. If signed, for 16-bit, the range is -32768 to 32767. For unsigned, it's 0 to 65535. Figure out what format you want to use and scale the -1 to 1 values up to that range.
One final note on this conversion... endianness can matter. See also: https://stackoverflow.com/a/7870190/362536
The only two examples I've found that are clear and make sense are the following:
AWS Labs: https://github.com/awslabs/aws-lex-browser-audio-capture/blob/master/lib/worker.js
The AWS resource is very good. It shows you how to export your recorded audio to "WAV format encoded as PCM". Amazon Lex, which is a transcription service offered by AWS requires the audio to be PCM encoded and wrapped in a WAV container. You can merely adapt some of the code to make it work for you! AWS has some additional features such as "downsampling" which allows you to change the sample rate without affecting the recording.
RecordRTC: https://github.com/muaz-khan/RecordRTC/blob/master/simple-demos/raw-pcm.html
RecordRTC is a complete library. You can, once again, adapt their code or find the snippet of code that encodes the audio to raw PCM. You could also implement their library and use the code as-is. Using the "desiredSampleRate" option for audio config with this library negatively affects the recording.
They are both excellent resources and you'll definitely be able to solve your question.
You should look into MediaTrackConstraints.sampleSize property for the MediaDevices.getUserMedia() API. Using the sampleSize constraint, if your audio hardware permits you can set the sample size to 16 bits.
As far as the implementation goes, well that's what the links are and google are for...
here is some Web Audio API where it uses the microphone to capture and playback raw audio (turn down your volume before running this page) ... to see snippets of raw audio in PCM format view the browser console ... for kicks it also sends this PCM into a call to FFT to obtain the frequency domain as well as the time domain of the audio curve
<html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>capture microphone then show time & frequency domain output</title>
<script type="text/javascript">
var webaudio_tooling_obj = function () {
var audioContext = new AudioContext();
console.log("audio is starting up ...");
var BUFF_SIZE_RENDERER = 16384;
var SIZE_SHOW = 3; // number of array elements to show in console output
var audioInput = null,
microphone_stream = null,
gain_node = null,
script_processor_node = null,
script_processor_analysis_node = null,
analyser_node = null;
if (!navigator.getUserMedia)
navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia ||
navigator.mozGetUserMedia || navigator.msGetUserMedia;
if (navigator.getUserMedia){
navigator.getUserMedia({audio:true},
function(stream) {
start_microphone(stream);
},
function(e) {
alert('Error capturing audio.');
}
);
} else { alert('getUserMedia not supported in this browser.'); }
// ---
function show_some_data(given_typed_array, num_row_to_display, label) {
var size_buffer = given_typed_array.length;
var index = 0;
console.log("__________ " + label);
if (label === "time") {
for (; index < num_row_to_display && index < size_buffer; index += 1) {
var curr_value_time = (given_typed_array[index] / 128) - 1.0;
console.log(curr_value_time);
}
} else if (label === "frequency") {
for (; index < num_row_to_display && index < size_buffer; index += 1) {
console.log(given_typed_array[index]);
}
} else {
throw new Error("ERROR - must pass time or frequency");
}
}
function process_microphone_buffer(event) {
var i, N, inp, microphone_output_buffer;
// not needed for basic feature set
// microphone_output_buffer = event.inputBuffer.getChannelData(0); // just mono - 1 channel for now
}
function start_microphone(stream){
gain_node = audioContext.createGain();
gain_node.connect( audioContext.destination );
microphone_stream = audioContext.createMediaStreamSource(stream);
microphone_stream.connect(gain_node);
script_processor_node = audioContext.createScriptProcessor(BUFF_SIZE_RENDERER, 1, 1);
script_processor_node.onaudioprocess = process_microphone_buffer;
microphone_stream.connect(script_processor_node);
// --- enable volume control for output speakers
document.getElementById('volume').addEventListener('change', function() {
var curr_volume = this.value;
gain_node.gain.value = curr_volume;
console.log("curr_volume ", curr_volume);
});
// --- setup FFT
script_processor_analysis_node = audioContext.createScriptProcessor(2048, 1, 1);
script_processor_analysis_node.connect(gain_node);
analyser_node = audioContext.createAnalyser();
analyser_node.smoothingTimeConstant = 0;
analyser_node.fftSize = 2048;
microphone_stream.connect(analyser_node);
analyser_node.connect(script_processor_analysis_node);
var buffer_length = analyser_node.frequencyBinCount;
var array_freq_domain = new Uint8Array(buffer_length);
var array_time_domain = new Uint8Array(buffer_length);
console.log("buffer_length " + buffer_length);
script_processor_analysis_node.onaudioprocess = function() {
// get the average for the first channel
analyser_node.getByteFrequencyData(array_freq_domain);
analyser_node.getByteTimeDomainData(array_time_domain);
// draw the spectrogram
if (microphone_stream.playbackState == microphone_stream.PLAYING_STATE) {
show_some_data(array_freq_domain, SIZE_SHOW, "frequency");
show_some_data(array_time_domain, SIZE_SHOW, "time"); // store this to record to aggregate buffer/file
}
};
}
}(); // webaudio_tooling_obj = function()
</script>
</head>
<body>
<p>Volume</p>
<input id="volume" type="range" min="0" max="1" step="0.1" value="0.0"/>
<p> </p>
<button onclick="webaudio_tooling_obj()">start audio</button>
</body>
</html>
NOTICE - before running above in your browser first turn down your volume as the code both listens to your microphone and sends real time output to the speakers so naturally you will hear feedback --- as in Jimmy Hendrix feedback
Related
I am making an music app with js & jquery.
An user can add multiple audio files from local machine.
I want to apply an equalizer filter.
So I have used AudioContext. Something like this:
mediaElement = document.querySelector("#audioFiles_" + nameCounters);
sourceNode = audioContext.createMediaElementSource(mediaElement);
// Set filters
[60, 170, 350, 1000, 3500, 10000].forEach(function (freq, i) {
var eq = audioContext.createBiquadFilter();
eq.frequency.value = freq;
eq.type = "peaking";
eq.gain.value = 0;
filters.push(eq);
});
// Connect filters in serie
sourceNode.connect(filters[0]);
for (var i = 0; i < filters.length - 1; i++) {
filters[i].connect(filters[i + 1]);
}
// Master volume is a gain node
masterGain = audioContext.createGain();
masterGain.value = 1;
//connect the last filter to the speakers
filters[filters.length - 1].connect(masterGain);
// for stereo balancing, split the signal
stereoPanner = audioContext.createStereoPanner();
// connect master volume output to the panner
masterGain.connect(stereoPanner);
// Connect the stereo panner to analyser and analyser to destination
stereoPanner.connect(audioContext.destination);
This code runs for each file that is added.
Everything works just fine if there is only ONE audio file.
If I add multiple audio files, the filter gets applied multiple times on each subsequent file.
By the time the 10 file gets added, the audio is completely jarred.
link to demo:
https://mellowmonks.in/demos/mixtape/10/
I have checked out many example codes on the internet, however, they all cover working with a single audio file.
This is the code i have used as reference:
https://codepen.io/chi-hin-leung/pen/RwppEKG
I'll show you my code
audio_music = new Audio();
var track = audioContext.createMediaElementSource(audio_music);
//Import music files from other sources into base64 form.
audio_music.src = "data:audio/ogg;base64,"+ data.music;
var splitter = audioContext.createChannelSplitter(6);
var merger = audioContext.createChannelMerger(2);
track.connect(splitter);
//omitted in addition to 0 and 1 due to repetition of the similar content
gainNode0 = audioContext.createGain(); gainNode0.gain.setValueAtTime((musicvolume*0.1), audioContext.currentTime);
gainNode1 = audioContext.createGain(); gainNode1.gain.setValueAtTime((musicvolume*0.1), audioContext.currentTime);
splitter.connect(gainNode0, 0);
splitter.connect(gainNode1, 1);
var pitchshift0 = new Tone.PitchShift(pitch);
var pitchshift1 = new Tone.PitchShift(pitch);
Tone.connect(gainNode0, pitchshift0);
Tone.connect(gainNode1, pitchshift1);
Tone.connect(pitchshift0, merger, 0, 0);
Tone.connect(pitchshift1, merger, 0, 1);
Tone.connect(merger, audioContext.destination);
I am not familiar with the use of audioContext and tone.js, so I don't know if I understand correctly, but my intention is to separate input sources with six channels and process them in the order of gain adjustment, pitch shift, and marge, respectively.
This will do everything else, but you can't change the value of the pitch shift during playback.
I want a way to function similar to the setValueAtTime used in GainNode in pitch shift.
What should I do?
You can change the pitch by setting the pitch parameter:
pitchshift0.pitch = -12 // Semitone to shift the pitch to.
If you want to set this at a specific time during playback, you can use the Transport class to schedule this:
Tone.Transport.schedule(() => pitchshift0.pitch = -12, time /* The transport time you want to schedule it at */);
I get a WAV file from user upload (basically a file input) and have to do some manipulation with that:
Validate is it's a valid .wav file
If user uploaded a stereo file, extract a single channel (left)
Add w fade out at the end (50 last samples of the file)
My first thought was hey, there's an api for that (web audio), so I did something similar to:
const source = audioContext.createBufferSource();
const splitter = audioContext.createChannelSplitter(audioBuffer.numberOfChannels);
const gainNode = audioContext.createGain();
source.buffer = audioBuffer;
source.connect(splitter);
gainNode.gain.linearRampToValueAtTime(0, audioBuffer.duration);
splitter.connect(gainNode, 0);
Which in my thinking is taking the first channel out of the source and adding linear fade out (not really on last 50 samples, but that's not a point for now).
But...
How do I extract the output of that into a file? I know how to play manipulated sound frontend side, but am I able to turn it back into a file?
So at some point I assumed there's no way to do that, so I came up with a different solution, which uses low level file manipulation, that goes as follows:
const audioContext = new AudioContext();
// const arrayBuffer = await toArrayBuffer(file);
const audioBuffer = await decodeAudio(audioContext, arrayBuffer);
const channels = 1;
const duration = audioBuffer.duration;
const rate = audioBuffer.sampleRate;
const length = Math.ceil(duration * rate * channels * 2 + 44);
const buffer = new ArrayBuffer(length);
const view = new DataView(buffer);
let position = 0;
let offset = 0;
const setUint16 = (data) => {
view.setUint16(position, data, true);
position += 2;
};
const setUint32 = (data) => {
view.setUint32(position, data, true);
position += 4;
};
setUint32(0x46464952); // RIFF
setUint32(length - 8); // file length
setUint32(0x45564157); // WAV
setUint32(0x20746d66); // fmt
setUint32(16); // data size
setUint16(1); // PCM
setUint16(channels);
setUint32(rate);
setUint32(rate * 16 * channels);
setUint16(channels * 2);
setUint16(16);
setUint32(0x61746164); // "data"
setUint32(length - position - 4);
const leftChannel = audioBuffer.getChannelData(0);
let sample;
console.log('left', leftChannel);
console.log('length', length);
while (position < length) {
sample = leftChannel[offset];
sample = sample < 0 ? sample * 0x8000 : sample * 0x7FFF;
view.setInt16(position, sample, true);
position += 2;
offset++;
}
console.log('buffer', buffer);
const blob = new Blob([buffer], { type: file.type });
but seems it has a lot of flows, output is distorted / has different sample rate and so on...
My question then would be:
How do I extract a file out of a web audio api, if that's even possible? Cause that the best way to do that imho
If (1) is not possible, what am I doing wrong on the second approach?
WebAudio has no way of saving audio to a file. You can use MediaRecorder as one way, but I don't think that that's required to support WAV files. Or you can do it by hand as you show above. At a glance I don't see anything wrong with what you've done. Might be helpful to look at what Chrome does to save files in its test suite; it does basically what you do.
I'm trying to use a simple HTML range input to control the panning of my Web Audio API audio but I can only get 3 "positions" for my audio output:
-Center
-100% to the left
-100% to the right.
I would like to have something in between does positions, like 20% left and 80% right and so on...
The code that I'm using is:
//Creating the node
var pannerNode = context.createPanner();
//Getting the value from the HTML input and using it on the position X value
document.getElementById('panInput').addEventListener('change', function () {
pannerNode.setPosition(this.value, 0, 0);
});
And it refers to this input on my HTML file:
<input id="panInput" type="range" min="-1" max="1" step="0.001" value="0"/>
Does anyone knows what am I doing wrong?
You shouldn't need to use two panners - Panner is stereo. This old answer is a great one to this question:
How to create very basic left/right equal power panning with createPanner();
I've actually found simple left/right panning to be kind of difficult with the Web Audio API. It's really set up for surround / spatial stuff, and I honestly don't understand it very well.
The way that I usually do panning is like this:
var panLeft = context.createGain();
var panRight = context.createGain();
var merger = context.createMerger(2);
source.connect(panLeft);
source.connect(panRight);
panLeft.connect(merger, 0, 0);
panRight.connect(merger, 0, 1);
merger.connect(context.destination);
document.getElementById('panInput').addEventListener('change', function () {
var val = this.value;
panLeft.gain.value = ( val * -0.5 ) + 0.5;
panRight.gain.value = ( val * 0.5 ) + 0.5;
});
Basically, you send the signal to two gain nodes that you're going to use as your left and right channel. Then you take the value from your range element and use it to set the gain on each of the nodes.
This is sort of the lazy version though. In serious audio apps, there's usually a bit more math involved with the panning to make sure there aren't changes in overall level -- but hopefully this is enough to get you started.
I'm quite sure there is a better and easier way to do that but, for now, it definitely works for me.
If anyone else have a better/cleaner way of doing it, please share it here!
Thanks to Kevin Ennis for giving me this hint!
JavaScript File
//Create a splitter to "separete" the stereo audio data to two channels.
var splitter = context.createChannelSplitter(2);
//Connect your source to the splitter (usually, you will do it with the last audio node before context destination)
audioSource.connect(splitter);
//Create two gain nodes (one for each side of the stereo image)
var panLeft = context.createGain();
var panRight = context.createGain();
//Connect the splitter channels to the Gain Nodes that we've just created
splitter.connect(panRight,0);
splitter.connect(panLeft,1);
//Getting the input data from a "range" input from HTML (the code used on this range will be shown right on the end of this code)
var panPosition = document.getElementById("dispPanPositionLiveInput");
document.getElementById('panControl').addEventListener('change', function () {
var val = this.value;
panPosition.value = val;
panLeft.gain.value = ( val * -0.5 ) + 0.5;
panRight.gain.value = ( val * 0.5 ) + 0.5;
});
//Create a merger node, to get both signals back together
var merger = context.createChannelMerger(2);
//Connect both channels to the Merger
panLeft.connect(merger, 0, 0);
panRight.connect(merger, 0, 1);
//Connect the Merger Node to the final audio destination (your speakers)
merger.connect(context.destination);
HTML File
< input id="panControl" type="range" min="-1" max="1" step="0.001" value="0"/>
How can i modulate any of the AudioParams in Web Audio API, for example gain value of the GainNode using a Low Frequency Oscillator ?
https://coderwall.com/p/h1jnmg
var saw = context.createOscillator(),
sine = context.createOscillator(),
sineGain = context.createGainNode();
//set up our oscillator types
saw.type = saw.SAWTOOTH;
sine.type = sine.SINE;
//set the amplitude of the modulation
sineGain.gain.value = 10;
//connect the dots
sine.connect(sineGain);
sineGain.connect(saw.frequency);
You're not saving your actual nodes, just the value - so when you try to connect to oscillator.frequency, you're passing an integer value (400 - the frequency you saved in the node). Try http://jsfiddle.net/GCSEq/6/ - this stores the nodes, and properly routes to the AudioParam.
this.oscillator = context.createOscillator();
this.gain = context.createGainNode();
and
osctest2.play(osctest.oscillator.frequency , 1000);
(You were getting an error in the console.)