Develop Reference

Aligning audio for smooth playing with the web audio api

I am currently trying to figure how to play chunked audio with the web audio API, right off the bat everything does work.. however most transitions between chunks aren't as smooth as I want them to be, there's a very very brief moment of silence between most of them.
My current loading and playback code:
const response = await fetch(`${this.src}`)
const reader = response.body.getReader()
let timestamptowaituntil = 0
let tolog = []
let tolog2 = []
while (true) {
const { done, value } = await reader.read()
if (done) {
console.log(tolog)
console.log(tolog2)
console.log(this.ctx)
break
} else {
let audiodata = await this.ctx.decodeAudioData(value.buffer)
let source = this.ctx.createBufferSource()
source.buffer = audiodata
source.connect(this.ctx.destination)
source.start(timestamptowaituntil, 0, audiodata.duration)
timestamptowaituntil +=audiodata.duration
tolog.push(audiodata)
tolog2.push(source)
}
}
How could I go about eliminating these little moments of silence (or overlap)?
Edit: So far I've tried the following
Removing some milliseconds off the waiting time.
Removing the amount of time that is in the latency properties of the AudioContext.
Making a function to get the playback length of the UInt8Array form data using its bitrate (this indeed got me a slightly different result than the .duration property of an audioBuffer, but there still is tiny gaps)

After trying a ton of different approaches, I finally got a thought that solved the issue in the end.
My new idea was to simply play the first chunk when it arrives, and meanwhile collect as many chunks as possible, whenever a chunk is collected, its chained with the previous chunk to make one bigger chunk (this way also makes it works in firefox which requires the chunk to have a header for decoding). The playback of the first chunk is stopped 0.5-1 second before the .duration property claims it would end, this way any anomalies in detecting length are avoided. At that same time, the next chunk is played.
A few things I added to my code for this is the following:
A function to concat two chunks:
const concat = (arrayOne, arrayTwo) => {
let mergedArray = new Uint8Array(arrayOne.length + arrayTwo.length)
mergedArray.set([...arrayOne, ...arrayTwo])
return mergedArray
}
Extra offset when timing:
source.start(timestamptowaituntil, 0, audiodata.duration - .75)
timestamptowaituntil += (audiodata.duration - .75 + this.ctx.currentTime)
This along with some more minor edits has brought me to a solution that makes the chunk-swap impossible to hear (every now and then it is when the cpu is overloaded and the timing slowed).

Need Javascript help to avoid timeouts when loading audio files

I support several churches that don't have musicians, by providing a little website with a bunch of pure Javascript so they can select music for their services from a collection of about 1100 mp3 and m4a music files. Previously, they created playlists in iTunes or Media Player, but after a track completed, the player would immediately start the next track unless they quickly clicked 'Stop'. So my website allows them to select all their music ahead of time (up to 10 tracks), with a separate "Play" button for each. Hit "Play" and it plays that one track and stops. (Duh.)
I'm encountering delays in loading the files into my "audio" tags - and I need the file to load when they select it so I can display the track duration, which is frequently important to the selection of the music for the service. A delay doesn't occur very often, but often enough to be annoying. Also, the load will occasionally time out completely, even after several attempts. I've experimented played with various techniques, like using setTimeout with different values to allow several seconds before checking if it's loaded, or, loading 5 or 10 times with shorter timeout values until it's loaded. I created a test page that indicates that the timeouts vary greatly - from 2% to 5% of the time, to upwards of 25% occasionally (during tests of 1,000 to 10,000 random loads).
My first technique was relying on events (I tried both 'canplay' and 'canplaythrough' events with minimal difference):
const testAudio = document.getElementById('test-audio');
let timeStart = Date.now();
function loadMusic(p_file) {
testAudio.src = p_file;
testAudio.addEventListener('canplaythrough', musicLoaded);
timeStart = Date.now();
testAudio.load();
}
function musicLoaded() {
console.log('music loaded in ' + (Date.now()-timeStart) + 'ms');
testAudio.removeEventListener('canplaythrough', musicLoaded);
/* should I add/remove the listener each time I change the source file ? */
}
My second approach (from a post here: https://stackoverflow.com/questions/10235919/the-canplay-canplaythrough-events-for-an-html5-video-are-not-called-on-firefox) is to check the 'readyState' of the audio element after a specified timeout, rather than relying on an event. This question specifically addressed Firefox, so I should mention that in my tests Firefox has horrible load times for both the "events" and the "readyState" techniques. Chrome and Edge vary in the range of 2% to 6% load failure due to timeout and Firefox has 27% to 39% load timeouts.
let myTimeout = '';
function loadMusic(p_file) {
myTimeout = setTimeout(fileTimeout, 1000); /* I've tried various values here */
testAudio.src = p_file;
timeStart = Date.now();
testAudio.load();
}
function fileTimeout() {
if (testAudio.readyState > 3) {
console.log('music loaded in ' + (Date.now()-timeStart) + 'ms');
} else {
/* here, I've tried calling loadMusic again 5 to 10 times, which sometimes works */
/* or, just reporting that the load failed... */
console.log('music FAILED to load!');
}
}
I have a shared server hosting plan, and I suspect the delay might be due to traffic on my server. Unfortunately, my hosting service turns a deaf ear to anything that might be application or content related (not surprising). And this isn't worth upgrading to a dedicated server just to eliminate that variable. But I suspect that might be a major factor here.
I need a technique that will always work - even if it takes 30 seconds or more. As long as I can display an intermittent "Still loading..." type message I (and my users) would be satisfied. The "track X won't load" messages happen often enough to be annoying. Early on, I had a few files with bad characters in the file name that needed to be fixed before they would load. So the users think that problem persists. But I know I've fixed all them now.
Any and all suggestions are welcome - but I'd love to keep everything in plain Javascript.
Using an audio constructor:
function loadMusic(p_file) {
myTimeout = setTimeout(fileTimeout, 1000);
let audioConst = new Audio();
audioConst.src = p_file;
timeStart = Date.now();
}
function fileTimeout() {
if (audioConst.readyState > 3) {
console.log('music loaded in ' + (Date.now()-timeStart) + 'ms');
} else {
console.log('music FAILED to load!');
}
myTimeout = '';
}

Simple tensorflow.js model kills browser performance

I've made an app to run simple tfjs model on live camera in browser. The problem is that it kills the web page performance completely. The refresh rate is around 1-2fps and whole browser is laggy (other tabs, youtube movies, even system gui). What is strange, after the first model execution that is slow (model is compiled so it is understandable), model execution time is in range of 1.5-4ms (measured as in scrip below). So it should not be a problem for the browser.
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia({ video: true })
.then(function (stream) {
video!.srcObject = stream;
})
.catch(function (_) {
console.log("Something went wrong!");
});
}
let model = await tf.loadGraphModel('model/model.json')
let video = document.querySelector("#videoElement") as HTMLVideoElement;
async function do_inference()
{
let startTime = performance.now();
const image = tf.expandDims(tf.browser.fromPixels(video));
const image_fl32 = tf.cast(image, 'float32');
let final_pred = tf.tidy(() =>
{
return model.predict(image_fl32) as tf.Tensor4D;
});
image.dispose();
image_fl32.dispose();
final_pred.dispose();
let total = performance.now() - startTime;
console.log(total);
if (running)
{
requestAnimationFrame(do_inference);
}
}
Additional notes:
model is a very simple fully convolution model. Basically few layers of 3x3 and 5x5 convolutions. I also made a test with even smaller model (2 layers) and it has same problems.
the tfjs runs using webgl backend, changing it to wasm result in much greater execution time (around 150-200ms) but video is smooth. On production I would like to use even bigger model, so I cannot accept such execution time.
I've tested in on RTX 3060, 32 threads Ryzen, 128GB RAM. So PC should not be a problem
It seems that there are are no memory leaks on the tfjs side
the model is converted from tensorflow.keras.Model model using Model.save method and then using tensorflowjs_converter#3.9 with --output_format=tfjs_graph_model
Am I doing something wrong? Is there a way to make video smooth?

Create Seamless Loop of Audio - Web

I want to create a seamless loop of an audio file. But in all approaches I used so far, there was a noticeable gap between end & start.
This is what I tried so far:
First approach was to use the audio in the HTML and it loops but there is still a noticeable delay when going from the end of the track to the beginning.
<audio loop autoplay>
<source src="audio.mp3" type="audio/mpeg">
<audio>
Then I tried it from JavaScript with the same result:
let myAudio = new Audio(file);
myAudio.loop = true;
myAudio.play();
After that I tried this (according to this answer)
myAudio.addEventListener(
'timeupdate',
function() {
var buffer = .44;
if (this.currentTime > this.duration - buffer) {
this.currentTime = 0;
this.play();
}
},
false
);
I played around with the buffer but I only got it to reduce the gap but not leave it out entirely.
I turned to the library SeamlessLoop (GitHub) and got it to work to loop seamlessly in Chromium browsers (but not in the latest Safari. Didn't test in other browsers). Code I used for that:
let loop = new SeamlessLoop();
// My File is 58 Seconds long. Btw there aren't any gaps in the file.
loop.addUri(file, 58000, 'sound1');
loop.callback(soundsLoaded);
function soundsLoaded() {
let n = 1;
loop.start('sound' + n);
}
EDIT: I tried another approach: Looping it trough two different audio elements:
var current_player = "a";
var player_a = document.createElement("audio");
var player_b = document.createElement("audio");
player_a.src = "sounds/back_music.ogg";
player_b.src = player_a.src;
function loopIt(){
var player = null;
if(current_player == "a"){
player = player_b;
current_player = "b";
}
else{
player = player_a;
current_player = "a";
}
player.play();
/*
3104.897 is the length of the audio clip in milliseconds.
Received from player.duration.
This is a different file than the first one
*/
setTimeout(loopIt, 3104.897);
}
loopIt();
But as milliseconds in browsers are not consistent or granular enough this doesn't work too well but it does work much better than the normal "loop" property of the audio.
Can anyone guide me into the right direction to loop the audio seamlessly?

You can use the Web Audio API instead. There are a couple of caveats with this, but it will allow you to loop accurately down to the single sample level.
The caveats are that you have to load the entire file into memory. This may not be practical with large files. If the files are only a few seconds it should however not be any problem.
The second is that you have to write control buttons manually (if needed) as the API has a low-level approach. This means play, pause/stop, mute, volume etc. Scanning and possibly pausing can be a challenge of their own.
And lastly, not all browsers support Web Audio API - in this case you will have to fallback to the regular Audio API or even Flash, but if your target is modern browsers this should not be a major problem nowadays.
Example
This will load a 4 bar drum-loop and play without any gap when looped. The main steps are:
It loads the audio from a CORS enabled source (this is important, either use the same domain as your page or set up the external server to allow for cross-origin usage as Dropbox does for us in this example).
AudioContext then decodes the loaded file
The decoded file is used for the source node
The source node is connected to an output
Looping is enabled and the buffer is played from memory.
var actx = new (AudioContext || webkitAudioContext)(),
src = "https://dl.dropboxusercontent.com/s/fdcf2lwsa748qav/drum44.wav",
audioData, srcNode; // global so we can access them from handlers
// Load some audio (CORS need to be allowed or we won't be able to decode the data)
fetch(src, {mode: "cors"}).then(function(resp) {return resp.arrayBuffer()}).then(decode);
// Decode the audio file, then start the show
function decode(buffer) {
actx.decodeAudioData(buffer, playLoop);
}
// Sets up a new source node as needed as stopping will render current invalid
function playLoop(abuffer) {
if (!audioData) audioData = abuffer; // create a reference for control buttons
srcNode = actx.createBufferSource(); // create audio source
srcNode.buffer = abuffer; // use decoded buffer
srcNode.connect(actx.destination); // create output
srcNode.loop = true; // takes care of perfect looping
srcNode.start(); // play...
}
// Simple example control
document.querySelector("button").onclick = function() {
if (srcNode) {
srcNode.stop();
srcNode = null;
this.innerText = "Play";
} else {
playLoop(audioData);
this.innerText = "Stop";
}
};
<button>Stop</button>

There is a very simple solution for that, just use loopify it makes use of the html5 web audio api and works perfectly well with many formats, not only wav as the dev says.
<script src="loopify.js" type="text/javascript"></script>
<script>
loopify("yourfile.mp3|ogg|webm|flac",ready);
function ready(err,loop){
if (err) {
console.warn(err);
}
loop.play();
}
</script>
This will automatically play the file, if you want to have start and stop buttons for example take a look at his demo

NodeJS + Websockets + HTML5 video tag and 'streaming'

I'm trying to stream a large video file to the browser in a <video> tag using websockets.
The video plays fine, but it always waits until it's downloaded the entire video before playing, resulting in a large delay. Setting autoplay = true and preload="none" seems to have no effect on this. So I've looked into chunking the video out and then sending it to the browser as a blob URL. For the chunking I'm using Node-Chunking-Streams
My code so far:
var chunkingStreams = require('chunking-streams');
var SizeChunker = chunkingStreams.Chunker;
var input = fs.createReadStream('src-videos/redcliff450.webm'),
chunker = new SizeChunker({
chunkSize: 2000000
}),
output;
chunker.on('chunkStart', function(id, done) {
output = fs.createWriteStream('src-videos/output/' + id + '.webm');
done();
});
chunker.on('chunkEnd', function(id, done) {
output.end();
done();
});
chunker.on('data', function(chunk) {
output.write(chunk.data);
});
input.pipe(chunker);
//test out the video using just the first chunk
var smallChunk = fs.createReadStream('src-videos/output/0.webm');
client.send(smallChunk);
My plan is to make the chunks small enough to load quickly - say ~2MB - and then send the next one when the clients ready. My issue is though that the first chunk (0) only plays for 3 seconds or so, before skipping straight to the end and stopping. This happens in Chrome and FF.
Increasing the chunk size until it encompasses the whole video still only results in the first 3 seconds playing.
If I play the chunked video 0.webm directly from the HDD in VLC, it plays fine. If I download the stream from within the browser and play it in VLC, it only plays the first 3 seconds. This article describes what I'm looking to do, but over HTTP. Anyone have any pointers for websockets?

removing input.pipe(chunker); solved this. I'm not quite sure the reason for this though, so will investigate as to why.