Develop Reference

Is there any way to track volume with the web audio api

I'd like to be able to track how loud the song gets in realtime, returning a number to represent the current volume of the song. The source is from a sample that plays when you click it. Ive seen tutorials on making a volume meter, but all I want is a number to measure this. here is my starting code.
const audioContext = new AudioContext();
const samplebutton = document.createElement('button')
samplebutton.innerText = 'sample'
samplebutton.addEventListener('click', async () => {
let volume = 0
const response = await fetch('testsong.wav')
const soundBuffer = await response.arrayBuffer()
const sampleBuffer = await audioContext.decodeAudioData(soundBuffer)
const sampleSource = audioContext.createBufferSource()
sampleSource.buffer = sampleBuffer
sampleSource.connect(audioContext.destination)
sampleSource.start()
function TrackVolume(){
}

You can calculate the volume by passing the audio through an AnalyserNode. With it you can get the amplitudes of every frequency at a specific moment in time and calculate the volume of the audio file.
const audioContext = new AudioContext();
const samplebutton = document.createElement('button')
samplebutton.innerText = 'sample'
samplebutton.addEventListener('click', async () => {
const response = await fetch('testsong.wav')
const soundBuffer = await response.arrayBuffer()
const sampleBuffer = await audioContext.decodeAudioData(soundBuffer)
const analyser = audioContext.createAnalyser();
const sampleSource = audioContext.createBufferSource()
sampleSource.buffer = sampleBuffer
const bufferLength = analyser.frequencyBinCount;
const dataArray = new Uint8Array(bufferLength);
sampleSource.connect(analyser);
analyser.connect(audioContext.destination)
sampleSource.start()
function caclculateVolume() {
analyser.getByteFrequencyData(dataArray)
let sum = 0;
for (const amplitude of dataArray) {
sum += amplitude * amplitude
}
const volume = Math.sqrt(sum / dataArray.length)
console.log(volume)
requestAnimationFrame(caclculateVolume)
}
caclculateVolume()
});

Best way to download depth data in javascript?

I am making a Three.js application that needs to download some depth data. The data consist of 512x256 depth entries, stored in a compressed binary format with the precision of two bytes each. The data must be readable from the CPU, so I cannot store the data in a texture. Floating point textures is not supported on many browsers anyway, such as Safari on iOS.
I have this working in Unity, but I am not sure how to go about downloading compressed depth like this using javascript / three.js. I am new to javascript, but seems it has limited support for binary data handling and compression.
I was thinking of switching to a textformat, but then memory footprint and download size is a concern. The user could potentially have to load hundreds of these depth buffers.
Is there a better way to download a readable depth buffer?

You can download a file as binary data with fetch and async/await
async function doIt() {
const response = await fetch('https://webglfundamentals.org/webgl/resources/eye-icon.png');
const arrayBuffer = await response.arrayBuffer();
// the data is now in arrayBuffer
}
doIt();
After that you can make TypedArray views to view the data.
async function doIt() {
const response = await fetch('https://webglfundamentals.org/webgl/resources/eye-icon.png');
const arrayBuffer = await response.arrayBuffer();
console.log('num bytes:', arrayBuffer.byteLength);
// arrayBuffer is now the binary data. To access it make one or more views
const bytes = new Uint8Array(arrayBuffer);
console.log('first 4 bytes:', bytes[0], bytes[1], bytes[2], bytes[3]);
const decoder = new TextDecoder();
console.log('bytes 1-3 as unicode:', decoder.decode(bytes.slice(1, 4)));
}
doIt();
As for a format for depth data that's really up to you. Assuming your format was just 16bit values representing ranges of depths from min to max
uint32 width
uint32 height
float min
float max
uint16 data[width * height]
Then after you've loaded the data you can use either muliplte array views.
const uint32s = new Uint32Array(arrayBuffer);
const floats = new Float32Array(arrayBuffer, 8); // skip first 8 bytes
const uint16s = new Uint16Array(arrayBuffer, 16); // skip first 16 bytes
const width = uint32s[0];
const height = uint32s[1];
const min = floats[0];
const max = floats[1];
const range = max - min;
const depthData = new Float32Array(width * height);
for (let i = 0; i < uint16s.length; ++i) {
depthData[i] = uint16s[i] / 0xFFFF * range + min;
}
If you care about endianness for some future world where there are any browsers running on big endian hardware, then you either write your own functions to read bytes and generate those values or you can use a DataView.
Assuming you know the data is in little endian format
const data = new DataView(arrayBuffer);
const width = data.getUint32(0, true);
const height = data.getUint32(4, true);
const min = data.getFloat32(8, true);
const max = data.getFloat32(12, true);
const range = max - min;
const depthData = new Float32Array(width * height);
for (let i = 0; i < uint16s.length; ++i) {
depthData[i] = data.getUint16(i * 2 + 16, true) / 0xFFFF * range + min;
}
Of you want more complex compression like a inflate/deflate file you'll need a library or to write your own.

Determine if there is a pause in speech using Web Audio API AudioContext

Trying to understand the Web Audio API better. We're using it to create an AudioContext and then sending audio to be transcribed. I want to be able to determine when there is a natural pause in speech or when the user stopped speaking.
Is there some data in onaudioprocess callback that can be accessed to determine pauses/breaks in speech?
let context = new AudioContext();
context.onstatechange = () => {};
this.setState({ context: context });
let source = context.createMediaStreamSource(stream);
let processor = context.createScriptProcessor(4096, 1, 1);
source.connect(processor);
processor.connect(context.destination);
processor.onaudioprocess = (event) => {
// Do some magic here
}
I tried a solution that is suggested on this post but did not achieve the results I need. Post: HTML Audio recording until silence?
When I parse for silence as the post suggests, I get the same result - either 0 or 128
let context = new AudioContext();
let source = context.createMediaStreamSource(stream);
let processor = context.createScriptProcessor(4096, 1, 1);
source.connect(processor);
processor.connect(context.destination);
/***
* Crete analyser
*
**/
let analyser = context.createAnalyser();
analyser.smoothingTimeConstant = 0;
analyser.fftSize = 2048;
let buffLength = analyser.frequencyBinCount;
let arrayFreqDomain = new Uint8Array(buffLength);
let arrayTimeDomain = new Uint8Array(buffLength);
processor.connect(analyser);
processor.onaudioprocess = (event) => {
/**
*
* Parse live real-time buffer looking for silence
*
**/
let f, t;
analyser.getByteFrequencyData(arrayFreqDomain);
analyser.getByteTimeDomainData(arrayTimeDomain);
for (var i = 0; i < buffLength; i++) {
arrayFreqDomain[i]; <---- gives 0 value always
arrayTimeDomain[i]; <---- gives 128 value always
}
}
Looking at the documentation for the getByteFrequencyData method I can see how it is supposed to be giving a different value (in the documentation example it will give a different barHeight), but it isn't working for me. https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode/getByteFrequencyData#Example

Reducing sample rate of a Web Audio spectrum analyzer using mic input

I'm using the Web Audio API to create a simple spectrum analyzer using the computer microphone as the input signal. The basic functionality of my current implementation works fine, using the default sampling rate (usually 48KHz, but could be 44.1KHz depending on the browser).
For some applications, I would like to use a lower sampling rate (~8KHz) for the FFT.
It looks like the Web Audio API is adding support to customize the sample rate, currently only available on FireFox (https://developer.mozilla.org/en-US/docs/Web/API/AudioContextOptions/sampleRate).
Adding sample rate to the context constructor:
// create AudioContext object named 'audioCtx'
var audioCtx = new (AudioContext || webkitAudioContext)({sampleRate: 8000,});
console.log(audioCtx.sampleRate)
The console outputs '8000' (in FireFox), so it appears to be working up to this point.
The microphone is turned on by the user using a pull-down menu. This is the function servicing that pull-down:
var microphone;
function getMicInputState()
{
let selectedValue = document.getElementById("micOffOn").value;
if (selectedValue === "on") {
navigator.mediaDevices.getUserMedia({audio: true})
.then(stream => {
microphone = audioCtx.createMediaStreamSource(stream);
microphone.connect(analyserNode);
})
.catch(err => { alert("Microphone is required."); });
} else {
microphone.disconnect();
}
}
In FireFox, using the pulldown to activate the microphone displays a popup requesting access to the microphone (as normally expected). After clicking to allow the microphone, the console displays:
"Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported".
The display of the spectrum analyzer remains blank.
Any ideas how to overcome this error? If we can get past this, any guidance on how to specify sampleRate when the user's soundcard sampling rate is unknown?

One approach to overcome this is passing audio packets captured from microphone to analyzer node via a script processor node that re-samples the audio packets passing through it.
Brief overview of script processor node
Every script processor node has an input buffer and an output buffer.
When audio enters the input buffer, the script processor node fires
onaudioprocess event.
Whatever is placed in the output buffer of script processor node becomes its output.
For detailed specs, refer : Script processor node
Here is the pseudo-code:
Create live media source, script processor node and analyzer node
Connect live media source to analyzer node via script processor
node
Whenever an audio packet enters the script processor
node, onaudioprocess event is fired
When onaudioprocess event is fired :
4.1) Extract audio data from input buffer
4.2) Re-sample audio data
4.3) Place re-sampled data in output buffer
The following code snippet implements the above pseudocode:
var microphone;
// *** 1) create a script processor node
var scriptProcessorNode = audioCtx.createScriptProcessor(4096, 1, 1);
function getMicInputState()
{
let selectedValue = document.getElementById("micOffOn").value;
if (selectedValue === "on") {
navigator.mediaDevices.getUserMedia({audio: true})
.then(stream => {
microphone = audioCtx.createMediaStreamSource(stream);
// *** 2) connect live media source to analyserNode via script processor node
microphone.connect(scriptProcessorNode);
scriptProcessorNode.connect(analyserNode);
})
.catch(err => { alert("Microphone is required."); });
} else {
microphone.disconnect();
}
}
// *** 3) Whenever an audio packet passes through script processor node, resample it
scriptProcessorNode.onaudioprocess = function(event){
var inputBuffer = event.inputBuffer;
var outputBuffer = event.outputBuffer;
for(var channel = 0; channel < outputBuffer.numberOfChannels; channel++){
var inputData = inputBuffer.getChannelData(channel);
var outputData = outputBuffer.getChannelData(channel);
// *** 3.1) Resample inputData
var fromSampleRate = audioCtx.sampleRate;
var toSampleRate = 8000;
var resampledAudio = downsample(inputData, fromSampleRate, toSampleRate);
// *** 3.2) make output equal to the resampled audio
for (var sample = 0; sample < outputData.length; sample++) {
outputData[sample] = resampledAudio[sample];
}
}
}
function downsample(buffer, fromSampleRate, toSampleRate) {
// buffer is a Float32Array
var sampleRateRatio = Math.round(fromSampleRate / toSampleRate);
var newLength = Math.round(buffer.length / sampleRateRatio);
var result = new Float32Array(newLength);
var offsetResult = 0;
var offsetBuffer = 0;
while (offsetResult < result.length) {
var nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio);
var accum = 0, count = 0;
for (var i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i++) {
accum += buffer[i];
count++;
}
result[offsetResult] = accum / count;
offsetResult++;
offsetBuffer = nextOffsetBuffer;
}
return result;
}
Update - 03 Nov, 2020
Script Processor Node is being deprecated and replaced with AudioWorklets.
The approach to changing the sample rate remains the same.

Downsampling from the constructor and connecting an AnalyserNode is now possible in Chrome and Safari.
So the following code, taken from the corresponding MDN documentation, would work:
const audioContext = new (window.AudioContext || window.webkitAudioContext)({
sampleRate: 8000
});
const mediaStream = await navigator.mediaDevices.getUserMedia({
audio: true,
video: false
});
const mediaStreamSource = audioContext.createMediaStreamSource(mediaStream);
const analyser = audioContext.createAnalyser();
analyser.fftSize = 256;
const bufferLength = analyser.frequencyBinCount;
const dataArray = new Uint8Array(bufferLength);
analyser.getByteFrequencyData(dataArray);
mediaStreamSource.connect(analyser);
const title = document.createElement("div");
title.innerText = `Sampling frequency 8kHz:`;
const wrapper = document.createElement("div");
const canvas = document.createElement("canvas");
wrapper.appendChild(canvas);
document.body.appendChild(title);
document.body.appendChild(wrapper);
const canvasCtx = canvas.getContext("2d");
function draw() {
requestAnimationFrame(draw);
analyser.getByteFrequencyData(dataArray);
canvasCtx.fillStyle = "rgb(0, 0, 0)";
canvasCtx.fillRect(0, 0, canvas.width, canvas.height);
var barWidth = canvas.width / bufferLength;
var barHeight = 0;
var x = 0;
for (var i = 0; i < bufferLength; i++) {
barHeight = dataArray[i] / 2;
canvasCtx.fillStyle = "rgb(" + (2 * barHeight + 100) + ",50,50)";
canvasCtx.fillRect(x, canvas.height - barHeight / 2, barWidth, barHeight);
x += barWidth + 1;
}
}
draw();
See here for a demo where both 48kHz and 8kHz sampled signal frequencies are displayed: https://codesandbox.io/s/vibrant-moser-cfex33

Swift 3 Getting audio stream

I am very new to Swift programming, so please pardon me.
What I am trying to do is Im trying to do a voice recording, and below is the method to get the stream from microphone of the iPhone. I am successful at storing the stream to a WAV file inside the device by using this method. I confirmed that the file is OK and is playing fine after the recording
osErr = ExtAudioFileWrite(destinationFile!, frameCount, inputDataList)
What i am unsuccessful in doing is capturing the audio stream, pass the stream to the WKWebview as base64 encoded data, converting the data into a Blob then pass the Blob as a source in HTML5 Audio tag using
URL.createObjectURL(new Blob( [blob], { type:{"audio/wav"} })
The Audio element just throws an Error and does not play the blob. I dont know if what I am getting at the sendAudioBuffer part of my code is the correct data that i should pass.
I am really very thankful for any bits of information. The original code where i got the method is from here
https://gist.github.com/hotpaw2/ba815fc23b5d642705f2b1dedfaf0107
func processMicrophoneBuffer( inputDataList: UnsafeMutablePointer<AudioBufferList>, frameCount : UInt32 ) {
let inputDataPtr = UnsafeMutableAudioBufferListPointer(inputDataList)
let mBuffers : AudioBuffer = inputDataPtr[0]
let count = Int(frameCount)
let bufferPointer = UnsafeMutableRawPointer(mBuffers.mData)
if let bptr = bufferPointer {
let dataArray = bptr.assumingMemoryBound(to: Float.self)
var sum : Float = 0.0
var j = self.circInIdx
let m = self.circBuffSize
for i in 0..<(count/2) {
let x = Float(dataArray[i+i ]) // copy left channel sample
let y = Float(dataArray[i+i+1]) // copy right channel sample
self.circBuffer[j ] = x
self.circBuffer[j + 1] = y
j += 2 ; if j >= m { j = 0 } // into circular buffer
sum += x * x + y * y
}
self.circInIdx = j // circular index will always be less than size
if sum > 0.0 && count > 0 {
let tmp = 5.0 * (logf(sum / Float(count)) + 20.0)
let r : Float = 0.2
audioLevel = r * tmp + (1.0 - r) * audioLevel
}
}
//This will write to a file
osErr = ExtAudioFileWrite(destinationFile!, frameCount, inputDataList)
//This will get the audio stream and send to WKWebview to be converted to Blob
sendAudioBuffer(buffer: Data(bytes: bufferPointer!, count: count))
}

... converting the
data into a Blob then pass the Blob as a source in HTML5 Audio tag
using
URL.createObjectURL(new Blob( [blob], { type:{"audio/wav"} })
You have wrong syntax in the row above. The correct one is:
URL.createObjectURL(new Blob([blob], {type:"audio/wav"}))
Also if your blob is really a base64 string, you can get rid of createObjectURL and set the src attribute of your <source> in HTML5 Audio tag as a data-uri e. g.:
document.querySelector('#my_audio source').src = 'data:audio/wav;base64,' + blob;
Hope it helps.