Playing audio from stream with HTML5 - javascript

I've got problem with playing audio from stream in web browser. I am streaming voice via BinaryJS as 2048-length Int16 arrays to client's browser and I am getting very noisy output. This is how I am receiving the stream:
var
client,
audioContext,
audioBuffer,
sourceNode,
sampleSize = 2048,
function main() {
try {
audioContext = new AudioContext();
} catch(e) {
alert('Web Audio API is not supported in this browser');
}
sourceNode = audioContext.createBufferSource();
sourceNode.connect(audioContext.destination);
connectClient();
playSound();
};
function playSound() {
sourceNode.start(0);
};
function onError(e) {
console.log('error' + e);
};
function connectClient() {
client = new BinaryClient('ws://localhost:3000');
client.on('open', function() {
console.log('stream opened');
});
client.on('stream', function(stream, meta){
stream.on('data', function(data){
var integers = new Int16Array(data);
var audioBuffer = audioContext.createBuffer(1, 2048, 4410);
audioBuffer.getChannelData(0).set(integers);
sourceNode.buffer = audioBuffer;
});
});
};
main();
What should I do to get clean audio?

I think your playback technique here is causing the noise – first of all, can you ensure that your stream event fires often enough to provide continuous data for Web Audio? If it doesn't, then the noise you hear is the pops and clicks after each bit of audio stops playing. Also, updating the data by swapping ArrayBuffers on sourceNode isn't a good idea.
I would suggest using a ScriptProcessorNode with your desired buffer size – this gives you the ability to insert raw data into the graph at audio rate, which should reduce the noise. Look at this demo to see how to set up the node, basically you just need to fill the outputBuffer with your stream data.
This will only really help if your streaming is faster than the audio rate!
Good luck,
jakub

Related

How to play media files sequentially without visible break?

I've worded my title, and tags in a way that should be searchable for both video and audio, as this question isn't specific to one. My specific case only concerns audio though, so my question body will be written specific to that.
First, the big picture:
I'm sending audio to multiple P2P clients who will connect and disconnect a random intervals. The audio I'm sending is a stream, but each client only needs the part of the stream from whence they connected. Here's how I solved that:
Every {timeout} (e.g. 1000ms), create a new audio blob
Blob will be a full audio file, with all metadata it needs to be playable
As soon as a blob is created, convert to array buffer (better browser support), and upload to client over WebRTC (or WebSockets if they don't support)
That works well. There is a delay, but if you keep the timeout low enough, it's fine.
Now, my question:
How can I play my "stream" without having any audible delay?
I say stream, but I didn't implement it using the Streams API, it is a queue of blobs, that gets updated every time the client gets new data.
I've tried a lot of different things like:
Creating a BufferSource, and merging two blobs (converted to audioBuffers) then playing that
Passing an actual stream from Stream API to clients instead of blobs
Playing blobs sequentially, relying on ended event
Loading next blob while current blob is playing
Each has problems, difficulties, or still results in an audible delay.
Here's my most recent attempt at this:
let firstTime = true;
const chunks = [];
Events.on('audio-received', ({ detail: audioChunk }) => {
chunks.push(audioChunk);
if (firstTime && chunks.length > 2) {
const currentAudio = document.createElement("audio");
currentAudio.controls = true;
currentAudio.preload = 'auto';
document.body.appendChild(currentAudio);
currentAudio.src = URL.createObjectURL(chunks.shift());
currentAudio.play();
const nextAudio = document.createElement("audio");
nextAudio.controls = true;
nextAudio.preload = 'auto';
document.body.appendChild(nextAudio);
nextAudio.src = URL.createObjectURL(chunks.shift());
let currentAudioStartTime, nextAudioStartTime;
currentAudio.addEventListener("ended", () => {
nextAudio.play()
nextAudioStartTime = new Date();
if (chunks.length) {
currentAudio.src = URL.createObjectURL(chunks.shift());
}
});
nextAudio.addEventListener("ended", () => {
currentAudio.play()
currentAudioStartTime = new Date();
console.log(currentAudioStartTime - nextAudioStartTime)
if (chunks.length) {
nextAudio.src = URL.createObjectURL(chunks.shift());
}
});
firstTime = false;
}
});
The audio-received event gets called every ~1000ms. This code works; it plays each "chunk" after the last one was played, but on Chrome, there is a ~300ms delay that's very audible. It plays the first chunk, then goes quiet, then plays the second, so on. On Firefox the delay is 50ms.
Can you help me?
I can try to create a reproducible example if that would help.

Steaming fragmented Webm over websocket to MediaSouce

I am trying to do the following:
On the server I encode h264 packets into Webm (MKV) container structure, so that each cluster gets a single frame packet.Only the first data chunk is different as it contains something called Initialization Segment.Here it is explained quite well.
Then I stream those clusters one by one in a binary stream via WebSocket to a broweser, which is Chrome.
It probably sounds weird that I use h264 codec and not VP8 or VP9, which are native codec for Webm Video Format. But it appears that html video tag has no problem to play this sort of video container. If I just write the whole stream to a file and pass it to video.src, it is played fine. But I want to stream it in real-time.That's why I am breaking the video into chunks and sending them over websocket.
On the client, I am using MediaSource API. I have little experience in Web technologies, but I found that's probably the only way to go in my case.
And it doesn't work.I am getting no errors, the streams runs ok, and the video object emits no warning or errors (checking via developer console).
The client side code looks like this:
<script>
$(document).ready(function () {
var sourceBuffer;
var player = document.getElementById("video1");
var mediaSource = new MediaSource();
player.src = URL.createObjectURL(mediaSource);
mediaSource.addEventListener('sourceopen', sourceOpen);
//array with incoming segments:
var mediaSegments = [];
var ws = new WebSocket("ws://localhost:8080/echo");
ws.binaryType = "arraybuffer";
player.addEventListener("error", function (err) {
$("#id1").append("video error "+ err.error + "\n");
}, false);
player.addEventListener("playing", function () {
$("#id1").append("playing\n");
}, false);
player.addEventListener("progress",onProgress);
ws.onopen = function () {
$("#id1").append("Socket opened\n");
};
function sourceOpen()
{
sourceBuffer = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.64001E"');
}
function onUpdateEnd()
{
if (!mediaSegments.length)
{
return;
}
sourceBuffer.appendBuffer(mediaSegments.shift());
}
var initSegment = true;
ws.onmessage = function (evt) {
if (evt.data instanceof ArrayBuffer) {
var buffer = evt.data;
//the first segment is always 'initSegment'
//it must be appended to the buffer first
if(initSegment == true)
{
sourceBuffer.appendBuffer(buffer);
sourceBuffer.addEventListener('updateend', onUpdateEnd);
initSegment = false;
}
else
{
mediaSegments.push(buffer);
}
}
};
});
I also tried different profile codes for MIME type,even though I know that my codec is "high profile.I tried the following profiles:
avc1.42E01E baseline
avc1.58A01E extended profile
avc1.4D401E main profile
avc1.64001E high profile
In some examples I found from 2-3 years ago, I have seen developers using type= "video/x-matroska", but probably alot changed since then,because now even video.src doesn't handle this sort of MIME.
Additionally, in order to make sure the chunks I am sending through the stream are not corrupted, I opened a local streaming session in VLC player and it played it progressively with no issues.
The only thing I suspect that the MediaCodec doesn't know how to handle this sort of hybrid container.And I wonder then why video object plays such a video ok.Am I missing something in my client side code? Or MediacCodec API indeed doesn't support this type of media?
PS: For those curious why I am using MKV container and not MPEG DASH, for example. The answer is - container simplicity, data writing speed and size. EBML structures are very compact and easy to write in real time.

rewriting Java code to JS - creating an audio from bytes?

I'm trying to rewrite some (very simple) android code I found written in Java into a static HTML5 app (I don't need a server to do anything, I'd like to keep it that way). I have extensive background in web development, but basic understanding of Java, and even less knowledge in Android development.
The only function of the app is to take some numbers and convert them into an audio chirp from bytes. I have absolutely no problem translating the mathematical logic into JS. Where I'm having trouble is when it gets to actually producing the sound. This is the relevant parts of the original code:
import android.media.AudioFormat;
import android.media.AudioManager;
import android.media.AudioTrack;
// later in the code:
AudioTrack track = new AudioTrack(AudioManager.STREAM_MUSIC, sampleRate, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT, minBufferSize, AudioTrack.MODE_STATIC);
// some math, and then:
track.write(sound, 0, sound.length); // sound is an array of bytes
How do I do this in JS? I can use a dataURI to produce the sound from the bytes, but does that allow me to control the other information here (i.e., sample rate, etc.)? In other words: What's the simplest, most accurate way to do this in JS?
update
I have been trying to replicate what I found in this answer. This is the relevant part of my code:
window.onload = init;
var context; // Audio context
var buf; // Audio buffer
function init() {
if (!window.AudioContext) {
if (!window.webkitAudioContext) {
alert("Your browser does not support any AudioContext and cannot play back this audio.");
return;
}
window.AudioContext = window.webkitAudioContext;
}
context = new AudioContext();
}
function playByteArray( bytes ) {
var buffer = new Uint8Array( bytes.length );
buffer.set( new Uint8Array(bytes), 0 );
context.decodeAudioData(buffer.buffer, play);
}
function play( audioBuffer ) {
var source = context.createBufferSource();
source.buffer = audioBuffer;
source.connect( context.destination );
source.start(0);
}
However, when I run this I get this error:
Uncaught (in promise) DOMException: Unable to decode audio data
Which I find quite extraordinary, as it's such a general error it manages to beautifully tell me exactly squat about what is wrong. Even more surprising, when I debugged this step by step, even though the chain of the errors starts (expectedly) with the line context.decodeAudioData(buffer.buffer, play); it actually runs into a few more lines within the jQuery file (3.2.1, uncompressed), going through lines 5208, 5195, 5191, 5219, 5223 and lastly 5015 before erroring out. I have no clue why jQuery has anything to do with it, and the error gives me no idea what to try. Any ideas?
If bytes is an ArrayBuffer it is not necessary to create a Uint8Array. You can pass ArrayBuffer bytes as parameter to AudioContext.decodeAudioData() which returns a Promise, chain .then() to .decodeAudioData(), call with play function as parameter.
At javascript at stacksnippets, <input type="file"> element is used to accept upload of audio file, FileReader.prototype.readAsArrayBuffer() creates ArrayBuffer from File object, which is passed to playByteArray.
window.onload = init;
var context; // Audio context
var buf; // Audio buffer
var reader = new FileReader(); // to create `ArrayBuffer` from `File`
function init() {
if (!window.AudioContext) {
if (!window.webkitAudioContext) {
alert("Your browser does not support any AudioContext and cannot play back this audio.");
return;
}
window.AudioContext = window.webkitAudioContext;
}
context = new AudioContext();
}
function handleFile(file) {
console.log(file);
reader.onload = function() {
console.log(reader.result instanceof ArrayBuffer);
playByteArray(reader.result); // pass `ArrayBuffer` to `playByteArray`
}
reader.readAsArrayBuffer(file);
};
function playByteArray(bytes) {
context.decodeAudioData(bytes)
.then(play)
.catch(function(err) {
console.error(err);
});
}
function play(audioBuffer) {
var source = context.createBufferSource();
source.buffer = audioBuffer;
source.connect(context.destination);
source.start(0);
}
<input type="file" accepts="audio/*" onchange="handleFile(this.files[0])" />
I solved it myself. I read more into the MDN docs explaining AudioBuffer and realized two important things:
I didn't need to decodeAudioData (since I'm creating the data myself, there's nothing to decode). I actually took that bit from the answer I was replicating and it retrospect, it was entirely needless.
Since I'm working with a 16 Bit PCM stereo, that meant I needed to use the Float32Array (2 Channels, each 16 Bit).
Granted, I still had a problem with some of my calculations that resulted in a distorted sound, but as far as producing the sound itself, I ended up doing this really simple solution:
function playBytes(bytes) {
var floats = new Float32Array(bytes.length);
bytes.forEach(function( sample, i ) {
floats[i] = sample / 32767;
});
var buffer = context.createBuffer(1, floats.length, 48000),
source = context.createBufferSource();
buffer.getChannelData(0).set(floats);
source.buffer = buffer;
source.connect(context.destination);
source.start(0);
}
I can probably optimize it a bit further - the 32767 part should happen before this, in the part where I'm calculating the data, for example. Also, I'm creating a Float32Array with two channels, then outputting one of them cause I really don't need both. I couldn't figure out if there's a way to create one channel mono file with Int16Array, or if that's even necessary\better.
Anyway, that's essentially it. It's really just the most basic solution, with some minimal understanding on my part of how to handle my data correctly. Hope this helps anyone out there.

Can I record the output of an <audio> without use of the microphone?

I have an <audio> element and I'm changing the speed, start/end bounds, and pitch. I want to see if it's possible to record the audio I hear in the browser. However I don't want to just record with the microphone because of the lower quality.
I could do the same effects server-side but I'd rather not since I'd be basically duplicating the same functionality with two different technologies.
In response to a flag vote since it's "unclear what I'm asking", I'll rephrase.
I have an <audio> element playing on the page. I have some javascript manipulating the play-rate, volume, etc. I then want my browser to record the audio as I hear it. This is not the microphone. I want to create a new audio file that is as close as possible to the one playing. If it's at 75%, then the new file will be at 75% volume.
In supporting browsers, you could use the MediaElement.captureStream() method along with the MediaRecorder API.
But note that these technologies are still in active development and that current implementations are still full of bugs.
E.g, for your case, current stable FF will stop the rendering of the original media audio if you change its volume while recording... I didn't had time to search for a bug report on it, but anyway, this is just one of the many bugs you'll find.
// here we will save all the chunks of our record
const chunks = [];
// wait for the original media is ready
audio.oncanplay = function() {
audio.volume = 0.5; // just for your example
// FF still does prefix this unstable method
var stream = audio.captureStream ? audio.captureStream() : audio.mozCaptureStream();
// create a MediaRecorder from our stream
var rec = new MediaRecorder(stream);
// every time we've got a bit of data, store it
rec.ondataavailable = e => chunks.push(e.data);
// once everything is done
rec.onstop = e => {
audio.pause();
// concatenate our chunks into one file
let final = new Blob(chunks);
let a = new Audio(URL.createObjectURL(final));
a.controls = true;
document.body.append(a);
};
rec.start();
// record for 6 seconds
setTimeout(() => rec.stop(), 6000);
// for demo, change volume at half-time
setTimeout(() => audio.volume = 1, 3000);
};
// FF will "taint" the stream, even if the media is served with correct CORS...
fetch("https://dl.dropboxusercontent.com/s/8c9m92u1euqnkaz/GershwinWhiteman-RhapsodyInBluePart1.mp3").then(resp => resp.blob()).then(b => audio.src = URL.createObjectURL(b));
<audio id="audio" autoplay controls></audio>
For older browsers, you could use the WebAudio API's createMediaElementSource method, to pass your audio element media through the API.
From there, you'd be able to extract raw PCM data to arrayBuffers and save it.
In following demo, I'll use recorder.js library which does greatly help for the extraction + save to wav process.
audio.oncanplay = function(){
var audioCtx = new AudioContext();
var source = audioCtx.createMediaElementSource(audio);
var gainNode = audioCtx.createGain();
gainNode.gain.value = 0.5;
source.connect(gainNode);
gainNode.connect(audioCtx.destination);
var rec = new Recorder(gainNode);
rec.record();
setTimeout(function(){
gainNode.gain.value = 1;
}, 3000);
setTimeout(function(){
rec.stop()
audio.pause();
rec.exportWAV(function(blob){
var a = new Audio(URL.createObjectURL(blob));
a.controls = true;
document.body.appendChild(a);
});
}, 6000);
};
<script src="https://rawgit.com/mattdiamond/Recorderjs/master/dist/recorder.js"></script>
<audio id="audio" crossOrigin="anonymous" controls src="https://dl.dropboxusercontent.com/s/8c9m92u1euqnkaz/GershwinWhiteman-RhapsodyInBluePart1.mp3" autoplay></audio>
As Kaiido mentions in his answer, captureStream() is one way of doing it. However, that is not fully supported in Chrome and Firefox yet. MediaRecorder does also not allow for track set changes during a recording, and a MediaStream coming from captureStream() might have those (depends on the application) - thus ending the recording prematurely.
If you need a supported way of recording only audio from a media element, you can use a MediaElementAudioSourceNode, pipe that to a MediaStreamAudioDestinationNode, and pipe the stream attribute of that to MediaRecorder.
Here's an example you can use on a page with an existing audio element:
const a = document.getElementsByTagName("audio")[0];
const ac = new AudioContext();
const source = ac.createMediaElementSource(a);
// The media element source stops audio playout of the audio element.
// Hook it up to speakers again.
source.connect(ac.destination);
// Hook up the audio element to a MediaStream.
const dest = ac.createMediaStreamDestination();
source.connect(dest);
// Record 10s of audio with MediaRecorder.
const recorder = new MediaRecorder(dest.stream);
recorder.start();
recorder.ondataavailable = ev => {
console.info("Finished recording. Got blob:", ev.data);
a.src = URL.createObjectURL(ev.data);
a.play();
};
setTimeout(() => recorder.stop(), 10 * 1000);
Note that neither approach works with cross-origin audio sources without a proper CORS setup, as both WebAudio and recordings would give the application the possibility to inspect audio data.

Web Audio API delay in playback when using socket.io to receive ArrayBuffer from server

Let me start with the fact that I am fairly new to the web-development game and have hardly used socket.io for over a week or two. I attempted playing audio from the ArrayBuffer received from socket.io corresponding to an MP3 file transfer using Web Audio API. The ArrayBuffer gets successfully decoded by WebAudio but the only issue I am having is that it takes about 10 seconds after receiving the initial chunk of the ArrayBuffer to start playing this song.
My understanding is that it waits for the entire file to get streamed and then starts the playback ?? Is there a better way of playing the track as soon as the first set of chunks arrive ?
This is how I am using it currently :
socket.on('browser:buffer', function(data) {
console.log(data.buffer);
source = audioContext.createBufferSource();
audioContext.decodeAudioData(data.buffer, function(decodedData) {
source.buffer = decodedData;
source.connect(audioContext.destination);
source.loop = true;
source.connect(audioContext.destination);
source.start(0);
});
}, function(error) {
console.error("decodeAudioData error", error);
});
Yes, you can you the <audio> and the createMediaElementSource(audio). This has benefits of self-dealing with the downloading :)
var audio = new Audio("shoot.mp3");
var context = new AudioContext();
var source = context.createMediaElementSource(audio);
audio.loop = true;
source.connect(context.destination);
audio.play();
https://jsfiddle.net/u8j4h4L4/1/
Other option is just the simplest if you actually don't need WebAudio:
var sound = new Audio("myaudio.mp3");
sound.play()

Categories

Resources