I would like to see if it's possible to have direct access to Opus using getUserMedia or anything similar from the latest browsers.
I've been researching on it a lot but with no Good results.
I'm aware that either Opus or Speex are actually used in webkitSpeechRecognition API. I would like to do speech recognition but using my own server rather than Google's.
So there are a lot of suggestions about Emscripten but nobody did, so I ported the encoder opus-tools to JavaScript using Emscripten. Dependent on what one has in mind, there are now the following opportunities:
Encoding FLAC, WAVE, AIFF, RAW files || demo || Web Worker size: 1.3 MiB
Encoding raw stuff for immediately processing or sending without container || demo || Web Worker size: 0.6 MiB
Encoding to Ogg-Opus and WAV from getUserMedia stream
When using Mozilla Firefox, it's possible to use a MediaRecorder, which would also allow to convert arbitrary sound files into Opus format on supported platforms together with AudioContext.decodeAudioData()
We're using emscripten for encoding and decoding using gsm610 with getUserMedia, and it works incredibly well, even on mobile devices. These days javascript gives almost native performance, so emscripten is viable for compiling codecs. The only issue is potentially very large .js files, so you want to only compile the parts you are using.
Unfortunately, it isn't currently possible to access browser codecs directly from JavaScript for encoding. The only way to do it would be to utilize WebRTC and set up recording on the server. I've tried this by compiling libjingle with some other code out of Chromium to get it to run on a Node.js server... it's almost impossible.
The only thing you can do currently is send raw PCM data to your server. This takes up quite a bit of bandwidth, but you can minimize that by converting the float32 samples down to 16 bit (or 8 bit if your speech recognition can handle it).
Hopefully the media recorder API will show up soon so we can use browser codecs.
This is not a complete solution, #Brad's answer is actually the correct one at this time.
One way to do it is to compile Opus to Emscripten and hope that your PC can handle encoding using JavaScript. Another alternative is to use speex.js.
Related
i'm using wavesurfer.js for a tool i wrote. I used it to display a certain mp3 file. The problem i have is that if i load the mp3 file in both browsers the one in chrome gets chopped off in the beginning. I started debugging the problem and it seems that a call to audioContext.decodeAudioData from the AudioAPI results in the mp3 getting chopped off in chrome, the input in both cases is consistent (a 2781 bytearray goes in).
From firefox I get 121248 samples back and the layout looks good, from chrome I get 114330 and it's chopped off in the beginning.
I tested another file which is longer and it also gave me a difference of 6918 samples missing.
The same problem occurred under linux with FF where the samples returned are 124416. (gstreamer plugin)
(btw these are all comparable since the systems all use 48kHz output)
The plugin to decode mp3 under windows for firefox is the vlc plugin since ff itself is not capable to decode mp3 due to licensing issues.
the file is encoded as:
Audio file with ID3 version 2.4.0, contains: MPEG ADTS, layer III, v2.5, 24 kbps, 8 kHz, Monaural
originally it was a PCM 32kHz mono file
I can imagine that mp3 is not sample consistent due to different implementations of the decoders. (kinda answered my own question here)
What would be a consistent codec over multiple browsers that produces sampleconsistency? I know that Wav should produce comparable results, right?
How about ogg? can I assume consistency in the amount of samples since the codebase should be the same, or does these differences in samples stem from the fact that the audioAPI is built up differently in the different browsers (a bit counter intuitive from an API i would say)
I need to multiplex a couple streams between the browser and a Node.js server over a single Websocket connection. One stream is going to be used for sending binary data from the browser to the server, and the other is going to be used for a simple RPC.
I stumbled across BinaryJS which does exactly what I want. However, it has a specific problem with binary data and doesn't appear to be maintained regularly. Is there an alternative? My requirements:
Binary-compatible (no JSON serialization of binary data... that takes a ton of bandwidth)
Supports multiple, bidirectional streams
I actually don't care so much about browser support. My application relies on other modern APIs, so I'm only targeting current versions of Chrome and Firefox. Any ideas?
Brad I fixed the typed array issue with BinaryJS you were experiencing (in version 0.2.0). But you're right I haven't had much time to maintain it so you may run into other issues.
We are currently planning a website on which people can upload movies. When looking at YouTube you notice that some movies are uploaded twice or more times (by different users). To scale our application we're thinking about the following idea:
User selects movie file to be uploaded
A JavaScript will get the SHA256 hash from the file (it's more accurate then the MD5 hash) before it get's uploaded
The website will check if the hash already exists
If the hash doesn't exist, the file will be uploaded
If the hash does exist a message will be prompted or a reference to the already existing version on the server will be created. This without the video being uploaded.
Q: How do we analyze a file with JavaScript in order to get the SHA256 hash, and is SHA256 good enough or should we consider SHA512 (or another algorithm)?
Use the HTML5 File API to read the file: http://www.html5rocks.com/en/tutorials/file/dndfiles. Here is a JS code for calculating SHA-256: http://www.webtoolkit.info/javascript-sha256.html
I must add that I never tried this, but it seems to be possible. Alxandr is right, this would take very long for large videos, but you may try to use the WebWorker API in order not to freeze the browser: http://html5rocks.com/en/tutorials/workers/basics
Putting files aside for now, if the question is actually whether it's possible to get a SHA-256 hashes in JavaScript, the answer is yes. You can either reiplement this yourself (bad idea) or use a library like the Stanford JS Crypto library (good idea).
As far as the File API goes, it is implemented in the bleeding edge version of every major desktop browser, as well as the latest Android browser shipping. iOS is not supported as it doesn't really have a filesystem. Check out caniuse.com for the latest stats.
Simple answer, you can't. That is if you want to support all browsers at least. I think both Chrome and FireFox supports the reading of files on the client, and IE supports it with the help of ActiveX controls, but to get a solution that works in all browsers you have to use a plugin like Flash or Silverlight. Also, when doing file-uploads of video-magnitude (large+ files), I think going for flash or the likes from the start is a good idea anyhow, but that's just my opinion.
I have a camera feed coming into a linux machine using a V4l2 interface as the source for a gstreamer pipeline. I'm building an interface to control the camera, and I would like to do so in HTML/javascript, communicating to a local server. The problem is getting a feed from the gst pipeline into the browser. The options for doing so seem to be:
A loopback from gst to a v4l2 device, which is displayed using flash's webcam support
Outputting a MJPEG stream which is displayed in the browser
Outputting a RTSP stream which is displayed by flash
Writing a browser plugin
Overlaying a native X application over the browser
Has anyone had experience solving this problem before? The most important requirement is that the feed be as close to real time as possible. I would like to avoid flash if possible, though it may not be. Any help would be greatly appreciated.
You already thought about multiple solutions. You could also stream in ogg/vorbis/theora or vp8 to an icecast server, see the OLPC GStreamer wiki for examples.
Since you are looking for a python solution as well (according to your tags), have you considered using Flumotion? It's a streaming server written on top of GStreamer with Twisted, and you could integrate it with your own solution. It can stream over HTTP, so you don't need an icecast server.
Depending on the codecs, there are various tweaks to allow low-latency. Typically, with Flumotion, locally, you could get a few seconds latency, and that can be lowered I believe (x264enc can be tweaked to reach less than a second latency, iirc). Typically, you have to reduce the keyframe distance, and also limit the motion-vector estimation to a few nearby frames: that will probably reduce the quality and raise the bitrate though.
What browsers are you targeting? If you ignore Internet Explorer, you should be able to stream OGG/Theora video and/or WebM video direct to the browser using the tag. If you need to support IE as well though you're probably reduced to a flash applet. I just set up a web stream using Flumotion and the free version of Flowplayer http://flowplayer.org/ and it's working very well. Flowplayer has a lot of advanced functionality that I have barely even begun to explore.
I'm getting a pure binary MP3 stream from an ajax call. No headers, nothing. Just straight MP3 bits. (Actually is that really even a stream at all?)
I'd like to be able to play it in a web page (and, if possible, offer it for downloading).
Is this possible? And if so what's the best way to do it?
If it is not possible, what are some other ways to handle this?
You cant play music with pure javascript. you will need to get that stream and pass it to a flash player.
try JW flash player, though i am not sure if it can handle the type of stream you are talking about. you will have to do some research about what it can handle
as indeed mentioned, you'll need a mp3 playing plugin, flash being the most widely available.
The JMP3 jquery plugin makes that task easier for you. It does rely on a flash file for the sound processing.
If you're only targeting very modern browsers, you could make use of data URL:s, and just write an object element to your HTML, and link to that data URL. Note: I havn't tried this.
You might want to look at SoundManager 2: Javascript Sound for the Web
Its an open-source BSD licensed JavaScript script for dealing with sound.
It automatically hooks into HTML5 or Flash to produce the sound, depending on what is available.
I have done this using data uris and there is the browser compatibility issue, there is the problem of url length (basically > about 30,000 characters won't support IE) and there is also the problem of the browser taking forever to decode the base64 value (the buffering is also extermely slow).
With current web technology, I would say your best bet is to write the data to a temporary file on your server and then have something like Dewplayer load the temporary file and play it.