I am developing audio chat. Sounds are recorded in a browser (Firefox 91.02, javascript) using MediaRecorder that produces blobs of audio type - I have used MediaRecorder.start(timeslice) method in order to get chunks (blobs) of the sound every milliseconds. The blobs will be sent to server, which retranslate them to another clients, which will listen in their browsers.
The test version records sound and play using Audio object it in the only browser (without sending to network). Each blob that I got every milliseconds I try to play independently.
The problem I have faced is: only the first blob was played correctly, other are not played (browser says: data format is not suitable). I guess this occurs because of I try to play sounds blobs independently.
The question is: is there a solution to record sound blobs which I can play independently - not joining them for player (except for stopping the mediarecorder and restarting periodically) ?
Related
I'm transferring a live audio stream between 2 Electron window processes using WebRTC. There are no ICE or STUN servers, or anything like that, the connection is established manually through Electron IPC communication (based on this code).
Note: from the technical point of view regarding the audio streams themselves, this is very similar (if not identical) to streaming between 2 browser tabs on the same domain, so this is primarily not a question regarding Electron itself, although Electron IPC would be obviously substituted with a browser equivalent.
The audio stream works, I can transmit audio from one window to another in real-time, as it is generated. That is, I can generate audio (Web Audio API) in window "A" and listen to it through an <audio> element in window "B", or do processing on it using a separate AudioContext in window "B" (although there is some latency).
However, the audio data is vastly altered during streaming: it became mono, its quality dropped, and there is significant latency. After fiddling around I've learned WebRTC does pretty much everything I don't need, including encoding the audio stream with an audio codec, encrypting the transfer, running echo cancellation, and so on.
I don't need these. I need to simply transfer raw audio data through local WebRTC without altering the audio in any way. It needs to be float32 accurate to the sample.
How can I do this with WebRTC?
Why use WebRTC then?
I need to do custom audio processing inside the Web Audio API.
The only way to do this is using a ScriptProcessorNode, which is unusuable in production code when there's essentially anything on the page, because it is broken by design (it processes audio on the UI thread, and causes audio glitching by even slight UI interactions).
So basically, because of this (and to the best of my knowledge), my only option is to transfer audio with WebRTC streams to another window process, perform ScriptProcessorNode processing there (nothing more is happening in that window, empty DOM, so the processing is always nice and smooth), then send the results back.
This works, but the audio is altered during streaming, which I want to avoid (see above).
Why not use AudioWorklet?
Because Electron is 5 versions behind Chrome unfortunately (version 59 at the moment), and simply does not ship AudioWorklet yet.
I am trying to setup live internet radio station using icecast server and want my stream to work in all modern browsers. My client only produces .ogg stream. All browsers doesn't play .ogg stream. For eg the .ogg stream I have setup works in chrome but doesn't work in IE. How should I make it run in all browsers?
Is there a way I can convert .ogg stream to .mp3 or any other format on the fly
Embed a audio player in the browser which can play .ogg stream.
Or Any other advice would be helpful.
Regards,
Hitesh Bhutani
You have several options:
Change encoding format from OGG to MP3 in your Virtual DJ software. Keep in mind that Firefox will not be able to play mp3 streams on some platforms using HTML5 audio tag due to licensing limitations.
Install some kind of transcoding software on your server (where you have Icecast installed and running), for example liquidosap (https://www.liquidsoap.info/). Liquidsoap can (among other things) take your stream as an input and transcode it to sereveral formats, for example - mp3, aac, ogg and then your Icecast server will have several mount points available, for example http://yourserver.com:8000/stream.mp3, http://yourserver.com:8000/stream.ogg, http://yourserver.com:8000/stream.aac and then you can create a small javascript that wil detect browser version and choose suitable stream.
Use HTML5 media player like jPlayer (http://jplayer.org/) or Soundmanager2 (http://www.schillmania.com/projects/soundmanager2/). These players can automatically detect browser version and select suitable stream type, also if they can't play the stream using HTML5 <audio> tag, they will fall back to internal Flash based player.
The most advanced way is to combine (2) and (3) methods, that will give you the most browser support.
Supported audio coding formats
I need to obtain frequency/pitch data from the microphone of an android device on the fly using JavaScript.
I have done this for desktop/laptop browsers with getUserMedia and Web Audio API, but these are not supported on the vast majority of Android devices.
I have tried using the cordova-plugin-media-capture however this opens an audio recorder which the user can then save or discard, and after saving you can use cordova-plugin-file to obtain the data as shown here: https://stackoverflow.com/a/32097634/5674976 but I need it not to open the audio recorder, and instead perhaps just a record button, and once it is recording to provide the audio data immediately (so that it can detect the frequency data in real-time).
I have seen recording functionality in place e.g. WhatsApp, Facebook Messenger etc. and so as a last resort - since I do not know Java - would it be possible to create a plugin using Java for Cordova?
edit: I have also looked at cordova-plugin-media https://github.com/apache/cordova-plugin-media which seems to provide amplitude data and current position data. I'm thinking I could figure out frequency by looking at the amplitude over time, or am I being naive?
I managed to record audio and also analyze the frequency without either getUserMedia or Web Audio API for Android.
Firstly I installed the cordova-plugin-audioinput plugin, which outputs a stream of audio samples (from the microphone), with custom configurations such as buffer size and sample rate. You can then use this data to detect specific frequencies.
I have an icecast setup running on a server. The clients that will be connecting to it are tags in web pages, either through HTML5 or Flash. I am currently using audio.js to achieve this (specifically, the flash fallback).
The problem is, the audio is being played concurrently but separately with a stream of images. (It's a 10-fps jpeg stream.) I need the audio to match up as much as possible with the images. Unfortunately, the audio is sometimes as much as 7 seconds delayed before it starts playing.
Some information:
The image stream cannot be delayed to match the audio. The audio must speed up to match the images.
The icecast server config has <burst-on-connect> set to 0 to minimize latency.
There is essentially no lag when playing via VLC (perhaps a few hundred ms, which is acceptable).
Put another way, when viewing the images and playing the audio via vlc, everything is sufficiently aligned. Unfortunately, using VLC is not an option in the endgame.
Since VLC has no lag, that tells me that the web browser (Chrome, firefox, IE) is buffering the audio before playing it.
The question: How do I prevent the web browser from buffering the audio? I want it to play immediately as soon as it has anything available. I'm currently using audio.js, but other similar technologies are acceptable.
Additional information: I've set audio.js to autoplay and preload=none.
Thanks for your help!
A buffer is always necessary. Networks are packet switched. Data comes in chunks, not continuously. In fact, there are many buffers:
Capture buffer (at the sound card)
Codec buffer (codecs work on a chunk of samples at a time)
Network buffer to server
Server-side buffer (typically very large, 10+ seconds)
Network buffer to client
Client buffer (typically 2-3 seconds)
Client codec buffer
Client sound device buffer
Each buffer adds latency, as you have noticed. The only buffer you really have control over is the server-side buffer, which is configured by the <burst-on-connect> setting. By setting the size of this buffer to a larger size, you can fill all downstream buffers very quickly, enabling an extremely fast start to playback. You have set this to zero, which means that the downstream buffers can only fill as fast as the data comes in from the encoder.
Client-side, you have absolutely no control over the buffering, and nor should you. Clients are free to implement a codec in whatever way they choose. Some codecs can begin streaming right away, and others can't. Some devices have to re-sample your audio to fit within their playback, and others don't.
What it sounds like you really want to do is synchronize a video stream and an audio stream. For that, you should be just streaming a video stream to begin with. Video is made to keep audio and video in sync. Icecast even supports streaming video in a few formats.
I am new in webRTC . As I know WebRTC is use for real time communication . In spec it seems that Stream can be created only by device outout( using GetUserMedia for microphone , camera or chrome tab capture api ) .But in my application i am getting real time Uint8DVideo( eg H264) data . Can i convert this uint8Data to MediaStream ?
I assume you don't use getUserMedia, but some arbitrary source.
Getting this video "buffer" to be displayed is tricky and not possible in every browser (only Chrome and soon Firefox). You don't need WebRTC to do that, but something called Media Source API AKA MSE (E for extensions).
The API is rather picky on it's accepted byte streams, and will not get any "video data". For H264, it will only except fragmented MP4. more info about that here.