What I am doing is some kind of a video conference tool. I am doing some research on video processing recently and it seems straight forward by using the video element in combination with canvases. But, I am using WebRTC for video streaming to all connected clients, for which I need a MediaStream. So I am looking for a way to retrieve a MediaStream from the canvas element.
On my way, I found this project here called Whammy http://antimatter15.com/wp/2012/08/whammy-a-real-time-javascript-webm-encoder/ which creates a video file from a canvas, but as far as I understand, it is not made for live-streaming it.
One alternative approach would be to do the video processing on the remote client by sending him the stream and the information on how to process it. However this might work pretty well for few clients, but when it comes to multi-user conferences, I feel like it doesn't scale, since real-time video processing is still a calculation intense job. Every client would have to process all video streams from all connected clients.
For me it looks like a one-way-street. Getting video content into a canvas is pretty easy, the other way around is pretty hard. I thought, there might be a library for creating a MediaStream from a canvas element, but I found nothing. Any ideas on how to achieve that?
Best,
Felix
Since this question was posted, there has been not much, but a little progress on this front.
The MediaStream Recording standard allows for recording streams (such as WebRTC) into video file formats, using the MediaRecorder API.
It should work with recent Firefox as well as Chrome (video only) enabling the latter via the experimental flags ( chrome://flags -> Experimental Web Platform features).
Also see this resource for examples.
Related
I have been building a web radio player for live streams & static files (MP3 over http). I picked Howler.js as the back-end to normalize the quirks of HTML5 Audio (think: Autoplay, fading in/out, progress events).
Unfortunately, I ran into several bugs (e.g. incompatibility with iOS Control Center, event handling) and found some "creative" workarounds -- but, right now, I'm at the point where I have serious doubts that the cost to use Howler overweighs the benefits. Furthermore:
streaming MP3s with HTML5 Audio sometimes seems like an afterthought - the API does not really reliably tell you what's going on (buffer underruns, network errors). Or it glitches.
Howler seems more focused on the Web Audio API (think: games, sound fx) than on its HTML5 part, even if the first two demos on the home page ("Music Player", "Live Radio") suggest otherwise.
So now, before I ditch Howler and just go with the browser's native Audio API, and hopefully don't run into unforeseen problems: has someone been at this point and can advise?
Are there other libraries I have overlooked?
I realize this is a rather soft question, unfortunately -- any suggestions on where to ask or what other information to provide are greatly appreciated.
Howler is the most popular audio library on NPM with more than 12k weekly downloads. here is some other popular libraries.
ReactPlayer
wavesurfer.js
jPlayer
Sound.js
SoundManager 2
I'm a Javascript dilettante. I need to make a webpage for mobile viewing to deploy a dynamically created but ultimately linear audio piece. Essentially I would need to load a playlist, in which some tracks are fixed but others are randomly chosen from a larger pool; there also need to be timed pauses between some of the tracks. It would need only minimal controls, probably just play/pause.
I'm looking into Web Audio API and the basic HTML5 <audio> tag. My two main concerns for choosing between them are compatibility and simplicity of use.
On the compatibility point, I see that on the main page for the API itself it lists no support for Android, but on this more detailed rundown almost all browsers are listed as green. What's the best source to trust?
Assuming Web Audio API is viable for mobile deployment, do I need to use it? Would it make my life easier or is it just overpowered for my purposes? I see it has a handy onended event handler which I see myself using for queuing, and precise timing functions. It also seems to be more explicit about loading the files asynchronously with a callback function on success - I'd want to have a loading screen so that would be useful.
I'm a bit less clear on the capabilities of <audio>. I guess it must be able to do everything I want given HTML5 players have been built before Web Audio API came along - but is it more fiddly?
Web Audio works just fine on mobile.
Web Audio, in contrast to <audio>, breaks apart and give the developer precise control over the loading, decoding and playing of audio. If you need precise timing - like, beat-synching - of audio, you should probably use web audio. <audio> is pretty imprecise.
That said, a few caveats - as Web Audio by default uses in-memory buffers, it can use a lot more memory than <audio>, and it doesn't have native components to do streaming audio. The onended event is NOT the right way to do real chaining of audio, because it's a main-thread-Javascript callback (that is to say, any event handling like this might be delayed by other JS, garbage collection, etc. - and it might be off by 50 or 100 milliseconds). If you really care about timing, you have to plan ahead and use Web Audio scheduling. (This article I wrote describes this in more detail.)
I have a web program which makes use of Web Audio API. The issue here is that i want to make it compatible for IE. Is there any alternate for the Web Audio API, so that i can make the same code run on IE specifically?
What are your needs? If you need to do dynamic synthesis, audio routing, etc, you will only be able to achieve that with the Web Audio API, so your IE users are out of luck.
However, if all you need to do is play audio files, then I would recommend that you use howler.js. Howler has great compatibility across different browsers and operating systems, including various versions of IE.
Microsoft have been working on implementing the web audio API, it looks like a preview implementation is available. https://status.modern.ie/webaudioapi If you can wait until the next version of IE which I believe is expected to be released with Windows 10, which should be out this year IIRC, it may not be worth taking the time to implement an alternative.
That said, to answer your question, there's no way that I could find to generate audio with an oscillator in the web without the Web Audio API other than writing flash code, which has its own major disadvantages.
Minor note: you could possibly hack the html5 audio node to play back audio buffers that you generate in javascript code, but keeping everything in sync and preventing jumps in the waveform would be an awful task.
I understand that the best bet when streaming a video as a source in an HTML5 tag would be .mp4. But let's say that I have a source that only outputs fragmented to an ism/manifest.
Is there any way at all, whether through other libraries or messy hacks, that I can bring this video into something rendered as a tag onscreen? The closest I have found is Walkthrough: Building Your First HTML5 Smooth Streaming Player because it allows this to be done - but I neither have Windows 8, or want to have this running a server capable of .NET. I was hoping there was something, messy or not, that I could achieve this with entirely within javascript and executable locally without a deploy.
Thanks
Firstly, W3C does not provide a standard for adaptive bitrate streaming, yet. So for the time being most browsers only support simple progressive download playback.
Hence, there is no JS implementation of a Smooth Streaming player and Microsoft is not working on one, as far as I know.
The example you provide uses the "Microsoft Smooth Streaming Client SDK Beta 2 for Windows 8" which is a C++ library and is only available for Windows Store Apps development. It has nothing to do with browsers.
So, unfortunately this is not yet possible. Even more, I doubt that this will ever happen, because everybody is waiting for MPEG DASH to be finalized.
UPDATE.
Please, notice that you always can use Siverlight application for playing SmoothStreaming. The referenced HTML5 Player framework is capable of falling back to Silverlight.
no luck for Microsoft Smooth Streaming, but regarding MPEG-DASH which is similar (see http://blog.johndeutscher.com/2013/06/10/mpeg-dash-preview-from-windows-azure-media-services):
"Dash.js is permissively licensed (under the BSD license) and can therefore be studied and reused by anyone seeking to provide their own DASH-AVC/264 compliant player. The goal is to make it easier for third-parties to build adaptive streaming video players."
http://msopentech.com/blog/2013/06/20/ms-open-tech-contributes-to-open-source-adaptive-streaming-video-player/
also see:
https://dvcs.w3.org/hg/html-media/raw-file/tip/media-source/media-source.html
which is required for Dash.js
I have a camera feed coming into a linux machine using a V4l2 interface as the source for a gstreamer pipeline. I'm building an interface to control the camera, and I would like to do so in HTML/javascript, communicating to a local server. The problem is getting a feed from the gst pipeline into the browser. The options for doing so seem to be:
A loopback from gst to a v4l2 device, which is displayed using flash's webcam support
Outputting a MJPEG stream which is displayed in the browser
Outputting a RTSP stream which is displayed by flash
Writing a browser plugin
Overlaying a native X application over the browser
Has anyone had experience solving this problem before? The most important requirement is that the feed be as close to real time as possible. I would like to avoid flash if possible, though it may not be. Any help would be greatly appreciated.
You already thought about multiple solutions. You could also stream in ogg/vorbis/theora or vp8 to an icecast server, see the OLPC GStreamer wiki for examples.
Since you are looking for a python solution as well (according to your tags), have you considered using Flumotion? It's a streaming server written on top of GStreamer with Twisted, and you could integrate it with your own solution. It can stream over HTTP, so you don't need an icecast server.
Depending on the codecs, there are various tweaks to allow low-latency. Typically, with Flumotion, locally, you could get a few seconds latency, and that can be lowered I believe (x264enc can be tweaked to reach less than a second latency, iirc). Typically, you have to reduce the keyframe distance, and also limit the motion-vector estimation to a few nearby frames: that will probably reduce the quality and raise the bitrate though.
What browsers are you targeting? If you ignore Internet Explorer, you should be able to stream OGG/Theora video and/or WebM video direct to the browser using the tag. If you need to support IE as well though you're probably reduced to a flash applet. I just set up a web stream using Flumotion and the free version of Flowplayer http://flowplayer.org/ and it's working very well. Flowplayer has a lot of advanced functionality that I have barely even begun to explore.