Say i like to create my very own progressive streaming mechanicsm in Javascript because i'm finding the browser's built in streaming mechanism not fault-tollerant enough or i like to implement my own custom method over WebSocket. I would like to create a buffer which holds the already downloaded segments of a continous media file (say an arraybuffer or something like that). Is it possible to play this file even if it's not already downloaded from start-to-end?
My only idea was the Web Audio API which has a noteOn() function for preceisely timing the start of each segment. However i don't know how gapeless this would be. Also it introduces the problem that i have to know exactly where audio files can be cut safely on the server side so the next part can be decoded without any loss and gaps. E.g. mp3's bit reservoir stores audio data in neighbour audio frames even in CBR mode which makes things difficult.
What about creating a ScriptProcessorNode that feeds from your incoming buffers? The biggest issue is making sure that the segments are convertible to raw audio samples, but otherwise you could write a simple function in the onaudioprocess event handler that pulls in the next available buffer chunk and copies it into the node's output buffers. Since this would be a pull-on-demand mechanism, you wouldn't need to worry about timing segment playback.
Related
I'm developing an application that has business logic that heavily depends on sound precision, hence I'm using the Web Audio API. The app has the functionality to manipulate audio files in multiple ways (fade in and fade out, programmatically seek through audios, play one audio multiple times, etc).
I've tried to implement this functionality using MediaElementAudioSourceNode, but I had a hard time getting everything together. Using this type of node, I wasn't sure how to implement some of the features like scheduling start time in the AudioContext timeline.
In the first iteration, I've implemented a simple download and decode method where the audio files are first downloaded as a whole from the network, stored to IndexedDB for caching purposes, then decoded into AudioBuffer and played as AudioBufferSourceNode connected to AudioContext.
As we already know, the decoding function is quite slow for larger files. After running some tests, I've realized that decoding OPUS is slower than decoding MP3. Also, decoding MP3 is slower than decoding WAV.
For that reason, in the second implementation, I've used decode-audio-data-fast and MP3 files. The decoding process is now faster as they're decoded in chunks on multiple threads, but I have to download bigger files.
However, I'd like to avoid downloading and decoding if possible. If this is not possible, I'm wondering if I can progressively download files and feed the chunks to the new WebCodecs API, then route them to AudioContext to be played and manipulated. The files are hosted on an S3 bucket. In theory, as far as I know, this should be completely possible, however, due to a lack of examples and documentation, and probably my experience with this, I can't figure out how exactly to implement this.
I am trying to create a calling app that uses WebRTC and a feature I want to add is audio obfuscation.
I want the ability to change the audio pitch that I am sending either at the source audio or even at the receiving end.
I have tried various famous libraries like P5.js but still was unable to get the result.
Need some suggestions/sample code in which I can modulate real time audio or even simulate the feeling that I am modulating the real time pitch of the audio.
I would prefer javascript as it will be rendered on the client but I am even open to using some alternative like WebAssembly if that will help me do the job.
Thanks in advance!
I've been designing an html/js based media player whose goal is to give the user some simple playback and region movement functionality found in common DAW's. My ui is comprised of 5 individual tracks and playback controls. By using wavesurfer.js I've been able to successfully create any number of wavesurfer instances/regions inside these tracks, drag and position them anywhere I'd like, and play selected audio files.
In my experimentation with wavesurfer.js, I've found that each instance has its own playhead which indicates the current playback position of the selected audio file and allows the user to navigate playback within that instance.
My issue is that I would also like to have a single "master" playhead which is not contained within any particular instance and am unsure how to approach this.
While there are complex web DAW's out there, I haven't found a source that really helps me understand how to handle audio playback in the way I need..but I know it's been done before. I've read through the webaudio API and wavesurfer.js documentation and unless there's something I'm missing within wavesurfer's capabilities, I'm assuming I need to work with webaudio API to achieve this result - would I start by defining a new AudioContext object comprised of all of the present audio files as AudioNodes?
With multiple audio files/wavesurfer instances whose playback must be dependent on their user-determined positions within the context of the whole track workspace, how would I approach handling audio playback?
Thanks for any insight, I appreciate it.
If you work with for example 5 trackouts, you need to be aware that if you load them into audiobuffers (or audionodes like you call em) they will actually become pretty big concerning the RAM amount if they are longer. So perhaps you can reach your goal if you use five audio elements and set their state and position, though these audio elements can not that easily be accessed by filters and effects and such (Convolver etc.). It really depends on your setup and a prototype shouldn't be too hard to program, so if you are really passionate about it you can make both.
At my website 1ln.de you can load mp3 files (different) in the audio positions and RAM gets up to 30 % of 8 GB if you load 5 mp3 in memory. But RAM is much faster then streaming audio. Because the audio elements need to buffer their contents first and you can't handle and manipulate them as easy as audio buffers.
It really depends on the comlexity of your tool and what you want to achieve.
For professional usage and fast playback i would recommend you playing with audio buffers. It will take some time to decode the audio files first though. But you can handle when buffering is finished.
Hopefully I got your question right.
If you want to program a DAW feel free to say hi for some help.
In a Chrome extension, is it possible to create an waveform or spectrogram image (or canvas) element based on only a .wav file URL of an <audio> tag?
I assume that solving this will require multiple steps:
Load the sound file data.
Convert the data into a format that can be used for drawing, possibly an array of sample values, similar to the .dat format of soxformat.
Generate an image or draw on a canvas.
Please provide JavaScript code to turn this:
<audio src="http://goo.gl/hWyNYu" controls />
into this:
This needs to be done without having to play back the audio, as opposed to existing solutions and without using server-side solutions.
Rather than being a question about a specific step in the process, this question seeks a complete answer with complete code so anybody could test it first and understand it later.
Well, theoretically, you should be able to use those existing client-side solutions without playing back the audio, using an OfflineAudioContext. Unfortunately, those solutions both use ScriptProcessorNode, and from what I've heard, existing implementations are broken when using ScriptProcessorNode in an OfflineAudioContext, and not likely to be fixed. I suspect AnalyserNode may be broken in OfflineAudioContext too.
It would probably work to use an OfflineAudioContext to just "play back" the entire sound file, then draw your canvas based on the output buffer that is created.
Or you could use a regular AudioContext, but make sure the output isn't audible (say, by piping the sound through a gain node with gain of zero). This is really ugly, slow, and would interfere with any other Web Audio API usage on your page.
You could also try fetching the entire file just as a binary blob into an ArrayBuffer, and parse it yourself. The WAV file format is not all that complicated (or you might be able to find open source code to do this). If you wanted to handle compressed formats like MP3 this way, you would definitely not want to write the decoder from scratch.
EDIT:
I think the Web Audio-based solutions above are too complicated. You have to set up an AudioBuffer and decode the audio into it using decodeAudioData anyway. Once you've done that, there is no need to even create an AudioBufferSourceNode. You can just get the audio data directly from the AudioBuffer by calling getChannelData on it.
We are making an web based music editor and mixer based on the Web Audio api. Users can mix together multiple tracks, crop tracks, etc. The actual mixing together of the tracks just involves playing back all the sources at once.
We want to be able to add the option to save the mix and make it available for download to a user's computer. Is there some way to do this on the front end (like connecting all the sources to one destination/export node), or even the backend (we are using RoR)?
RecorderJS does exactly what you need, and it could not possibly be easier to use. Really, really great library.
https://github.com/mattdiamond/Recorderjs
P.S. Look into OfflineAudioContext and my answer to this question (Web audio API: scheduling sounds and exporting the mix) for info on doing a faster-than-realtime mixdown of your audio.
Users data looks to be on client side?
Basically when converting data with base64 into dataURI, datas are diplayed inline so they can be add 1 by 1 togheter into one single blob object, and be downloaded.
But this method is only good for smalls files, causing crash and freezing with most browser, this is only good for blob size less than 10mb after some personnal tests, this will be better soon for sure.
<audio controls><source src="data:audio/ogg;base64,BASE64.......BASE564......BASE64............."></audio>
or
<a href="data:audio/ogg;base64,BASE64...BASE64..BASE64....>Download</a>
Probably not your way, just an idea but your project is interesting ;)