I know that I can collect Audio Data of an currently played audio with getByteFrequenzyData() and I'll get back an Uint8Array.
Now I collect all data of one Audio File by pushing each animationFrame the currently data in an Array, do for example:
I have a audio file with duration of 20min.
Then I have after 20min all Audio data in one Array, which then looks kind a like this:
var data = [Uint8Array[1024], Uint8Array[1024], Uint8Array[1024], Uint8Array[1024], ... ];
Is there a faster way to get all these audio data, so I don't have to wait the full 20 minutes of the video, and get the audio data nearly instant?
It would be good to receive the audio information in fixed steps for, like 50ms or so!
Instead of using an AudioContext, use an OfflineAudioContext to process the data. This can run much faster than real time. To get consistent data at well defined times, you'll also need to use a browser that implements the recently added suspend and resume feature for offline contexts so that you can sample the data for getByteFrequencyData at well-defined time intervals.
Related
In our javascript app we are trying to extract a single frame from each video tag, which includes
'MediaStream's from each user (with audioTrack and videoTrack), which they received using navigator.mediaDevices.getUserMedia and then send said stream using peerjs API, with peerConnection.answer(stream).
Now I am trying to extract a single frame from MediaStream's videoTrack, which will then be sent to another server.
I have not dealt with mediaStreams in the past and will like to know any suggestions on how to implement it. will include in the next days entire code but will take some time to crop the relevant segments of the code. Thank you
I've been trying to figure out the best way to go about implementing an idea I've had for a while.
Currently, I have an icecast mp3 stream of a radio scanner, with "now playing" metadata that is updated in realtime depending on what channel the scanner has landed on. When using a dedicated media player such as VLC, the metadata is perfectly lined up with the received audio and it functions exactly as I want it to - essentially a remote radio scanner. I would like to implement something similar via a webpage, and on the surface this seems like a simple task.
If all I wanted to do was stream audio, using simple <audio> tags would suffice. However, HTML5 audio players have no concept of the embedded in-stream metadata that icecast encodes along with the mp3 audio data. While I could query the current "now playing" metadata from the icecast server status json, due to client & serverside buffering there could be upwards of 20 seconds of delay between audio and metadata when done in this fashion. When the scanner is changing its "now playing" metadata upwards of every second in some cases, this is completely unsuitable for my application.
There is a very interesting Node.JS solution that was developed with this exact goal in mind - realtime metadata in a radio scanner application: icecast-metadata-js. This shows that it is indeed possible to handle both audio and metadata from a single icecast stream. The live demo is particularly impressive: https://eshaz.github.io/icecast-metadata-js/
However, I'm looking for a solution that can run totally clientside without needing a Node.JS installation and it seems like that should be relatively trivial.
After searching most of the day today, it seems that there are several similar questions asked on this site and elsewhere, without any cohesive, well-laid out answers or recommendations. From what I've been able to gather so far, I believe my solution is to use a Javascript streaming function (such as fetch) to pull the raw mp3 & metadata from the icecast server, playing the audio via Web Audio API and handling the metadata blocks as they arrive. Something like the diagram below:
I'm wondering if anyone has any good reading and/or examples for playing mp3 streams via the Web Audio API. I'm still a relative novice at most things JS, but I get the basic idea of the API and how it handles audio data. What I'm struggling with is the proper way to implement a) the live processing of data from the mp3 stream, and b) detecting metadata chunks embedded in the stream and handling those accordingly.
Apologies if this is a long-winded question, but I wanted to give enough backstory to explain why I want to go about things the specific way I do.
Thanks in advance for the suggestions and help!
I'm glad you found my library icecast-metadata-js! This library can actually be used both client-side and in NodeJS. All of the source code for the live demo, which runs completely client side, is here in the repository: https://github.com/eshaz/icecast-metadata-js/tree/master/src/demo. The streams in the demo are unaltered and are just normal Icecast streams on the server side.
What you have in your diagram is essentially correct. ICY metadata is interlaced within the actual MP3 "stream" data. The metadata interval or frequency that ICY metadata updates happen can be configured in the Icecast server configuration XML. Also, it may depend on your how frequent / accurate your source is for sending metadata updates to Icecast. The software used in the police scanner on my demo page updates almost exactly in time with the audio.
Usually, the default metadata interval is 16,000 bytes meaning that for every 16,000 stream (mp3) bytes, a metadata update will sent from Icecast. The metadata update always contains a length byte. If the length byte is greater than 0, the length of the metadata update is the metadata length byte * 16.
ICY Metadata is a string of key='value' pairs delimited by a semicolon. Any unused length in the metadata update is null padded.
i.e. "StreamTitle='The Stream Title';StreamUrl='https://example.com';\0\0\0\0\0\0"
read [metadataInterval bytes] -> Stream data
read [1 byte] -> Metadata Length
if [Metadata Length > 0]
read [Metadata Length * 16 bytes] -> Metadata
byte length
response data
action
ICY Metadata Interval
stream data
send to your audio decoder
1
metadata length byte
use to determine length of metadata string (do not send to audio decoder)
Metadata Length * 16
metadata string
decode and update your "Now Playing" (do not send to audio decoder)
The initial GET request to your Icecast server will need to include the Icy-MetaData: 1 header, which tells Icecast to supply the interlaced metadata. The response header will contain the ICY metadata interval Icy-MetaInt, which should be captured (if possible) and used to determine the metadata interval.
In the demo, I'm using the client-side fetch API to make that GET request, and the response data is supplied into an instance of IcecastReadableStream which splits out the stream and metadata, and makes each available via callbacks. I'm using the Media Source API to play the stream data, and to get the timing data to properly synchronize the metadata updates.
This is the bare-minimum CORS configuration needed for reading ICY Metadata:
Access-Control-Allow-Origin: '*' // this can be scoped further down to your domain also
Access-Control-Allow-Methods: 'GET, OPTIONS'
Access-Control-Allow-Headers: 'Content-Type, Icy-Metadata'
icecast-metadata-js can detect the ICY metadata interval if needed, but it's better to allow clients to read it from the header with this additional CORS configuration:
Access-Control-Expose-Headers: 'Icy-MetaInt'
Also, I'm planning on releasing a new feature (after I finish with Ogg metadata) that encapsulates the fetch api logic so that all a user needs to do is supply an Icecast endpoint, and get audio / metadata back.
I am making an online game in node.js and trying to save a game replay on my game's server. I am using flatbuffers to serialize data for client-server communication and I thought it would be cool to save my game's state frame by frame in the file.
I created the following table in my .fbr file
table Entity {
id: ushort;
pos: Vec2;
}
table Frame {
entities: [Entity];
}
table Replay {
frames: [Frame];
}
Is there a way to write all game state frames to the file on the fly? I know that I could just buffer N frames and save them in separate replay files, but i feel that there should be a better way. I want my replay to be in a single file, otherwise it would be very inconvenient to use it afterwards.
The best way to do this is to make sure each individual FlatBuffer is a "size prefixed buffer" (it has a 32-bit size ahead of the actual buffer), which in JS you can create by calling this instead of the usual finish function: https://github.com/google/flatbuffers/blob/6da1cf79d90eb242e7da5318241d42279a3df3ba/js/flatbuffers.js#L715
Then you write these buffers one after the other into an open file. When reading, because the buffers start with a size, you can process them one by one. I don't see any helper functions for that in JS, so you may have to do this yourself: read 4 bytes to find out the size, then read size bytes, repeat.
Given an AudioNode, is there any way to directly get the audio data from it? The data could be an ArrayBuffer, an AudioBuffer, a TypedArray or something similar.
I don't want to use any kind of Media Stream stuff.
Connect a ScriptProcessorNode or AudioWorkletNode to the output of the node you're interested in. These will give you buffers of audio data. You'll have to figure out how and where to save all the data, but these nodes will give you the audio data. If you can, use AudioWorkletNode, which isn't subject to main thread loading issues.
i'm using the Javascript for a radio player, so, sometimes someone got a bad network and the connection drops out, i want to know if there's a function like buffering, for reload the stream if it crashes.
I hope that someone could answer, thanks.
Greetings,
Julia.
if your use case is a radio the data is streaming so you can't access past data, if the use case is just a media-player you can use the buffered property of the audio element to query which is the range of the media that has been buffered at the moment or the currentTime property to access the current execution time.
var myAudioElement = document.createElementbyTag('audio')
// let's suppose that the audio element has a radio streaming attached as source
myAudioElement.buffered // returns the last data that has been downloaded from the source
myAudioElement.currentTime // returns the time of the last data played
Maybe you should see something like the connection API to be aware of when your user go offline