Develop Reference

Audio manipulation using node.js

My team has been using the Web Audio API/Getusermedia in a product and we are going really well with our chrome and firefox users. But we still have a large base of users that we would love to reach, but due to technology barriers, we still can't (mostly, those are IE users), as their main browser does not support the technology, and they do not or can not change to a modern browser.
We are planning to get to those users, but we don't want to go to Flash, Flex, Silverlight or anything similar.
So, thinking about solutions, I thought that maybe I could pass by this difficulty if I moved the audio manipulation, from the browser to the server. NodeJS was the first answer when trying to figure out how to do it.
Would it be possible to be done using NodeJS? Are there any libraries available that would help us accomplish this? Are there any other technologies that would allow me to do this?
Thanks anyone that could help.

It could easily be done. Node is simply an IO engine designed for rapid response. If it needs to happen in real time then I imagine latency would be a usability-breaking issue due to networking restraints. If it doesn't, then I think it would be a great solution! :)
Either way here are a couple related resources
https://www.npmjs.org/package/webrtc.io <- latency optimization library intended for work with media streams
http://wac.ircam.fr/ an upcoming conference (Jan 2015) dedicated to the types of problems you are dealing with.
http://www.sitepoint.com/5-libraries-html5-audio-api/ A few web libraries for use with audio. #3 and #4 look like they are related to what you are trying to do

You can try using this (is in development):
Node Web Audio API
https://github.com/sebpiq/node-web-audio-api
Installation
npm install web-audio-api
Demo
node test/manual-testing/AudioContext-sound-output.js

Smooth Streaming .ism to an HTML5 video tag

I understand that the best bet when streaming a video as a source in an HTML5 tag would be .mp4. But let's say that I have a source that only outputs fragmented to an ism/manifest.
Is there any way at all, whether through other libraries or messy hacks, that I can bring this video into something rendered as a tag onscreen? The closest I have found is Walkthrough: Building Your First HTML5 Smooth Streaming Player because it allows this to be done - but I neither have Windows 8, or want to have this running a server capable of .NET. I was hoping there was something, messy or not, that I could achieve this with entirely within javascript and executable locally without a deploy.
Thanks

Firstly, W3C does not provide a standard for adaptive bitrate streaming, yet. So for the time being most browsers only support simple progressive download playback.
Hence, there is no JS implementation of a Smooth Streaming player and Microsoft is not working on one, as far as I know.
The example you provide uses the "Microsoft Smooth Streaming Client SDK Beta 2 for Windows 8" which is a C++ library and is only available for Windows Store Apps development. It has nothing to do with browsers.
So, unfortunately this is not yet possible. Even more, I doubt that this will ever happen, because everybody is waiting for MPEG DASH to be finalized.
UPDATE.
Please, notice that you always can use Siverlight application for playing SmoothStreaming. The referenced HTML5 Player framework is capable of falling back to Silverlight.

no luck for Microsoft Smooth Streaming, but regarding MPEG-DASH which is similar (see http://blog.johndeutscher.com/2013/06/10/mpeg-dash-preview-from-windows-azure-media-services):
"Dash.js is permissively licensed (under the BSD license) and can therefore be studied and reused by anyone seeking to provide their own DASH-AVC/264 compliant player. The goal is to make it easier for third-parties to build adaptive streaming video players."
http://msopentech.com/blog/2013/06/20/ms-open-tech-contributes-to-open-source-adaptive-streaming-video-player/
also see:
https://dvcs.w3.org/hg/html-media/raw-file/tip/media-source/media-source.html
which is required for Dash.js

Interfacing a midi keyboard or other real-time midi input with javascript?

I want to create a simple visualization tool that would allow to represent my playing a midi keyboard on the screen. I play a relatively novel instrument type, called the harmonic table:
http://en.wikipedia.org/wiki/Harmonic_table_note_layout
http://www.soundonsound.com/newspix/image/axis49.jpg
And want to build tools to ease their use and to teach others how to use them.
However, I can't find a good way to get get midi into javascript environment (or, for that matter, Flash, or Java without a large helping of jiggery-pokery slightly beyond my reach, and the use of code from what look to be rather stale and abandoned open source projects. Neither language I am too enthused to work in for this project in any case).
Is there a suitable library or application that I have missed, that will enable me to do this?

While searching around for another solution (Flash based, using the functions of the Red5 Open source flash server - really ugly, but I'm desperate at this point) I found someone who had done exactly what I needed using Java to interface with the hardware. They had started with a flash solution and recently ported to Javascript. Yay!
http://www.abumarkub.net/abublog/?p=505
Don't let the caveats about 'proof of concept' discourage you: the basic functionality appears solid, at least with everything I have been able to throw at it.
So now I'm on my way, and so is anyone else who want to build javascript based midi interfaces/synths/what have you.
I can manipulate real-time midi in javascript! This is much better than flying cars and jetboots.

I have made a NPAPI browser plugin that lets you communicate in Javascript with MIDI devices.
Currently it works on OSX in Chrome and Safari, and on Windows in Chrome.
I am working on support for other browsers (Firefox, Internet Explorer) and other operating systems (Linux, Android, iOs)
See: http://abumarkub.net/abublog/?p=754

EDIT:
I recently published this module https://github.com/hems/midi2funk it's a node.js module that listens to midi and broadcast it through socket.io so if you have the luxury of running a node.js service locally together with your client side you might get some fun out of it...
~~~~~
A few others handy links, i kinda ordered in what i think would be most important for you:
midibridge.js - A Javascript API for interacting with MIDI devices
midi.js sequencing in javascript
jasmid - A Javascript MIDI file reader and synthesiser
Second web midi api working draft published - 11/12/2012
Jazz Soft - MIDI IN / OUT PLUGIN FOR BROWSER
edit: just realised the thread is old, hopefully the links will help someone ( :

The Web MIDI API is now real in Google Chrome 43+. I even wrote a library to make it easier to work with it. If you are still interested and do not care that it currently only works in Chrome, check it out: https://github.com/cotejp/webmidi

Nowadays browsers supports MIDI listening. All you need is
navigator.requestMIDIAccess().then(requestMIDIAccessSuccess, requestMIDIAccessFailure);
and listen keys
function requestMIDIAccessSuccess(midi) {
var inputs = midi.inputs.values();
for (var input = inputs.next(); input && !input.done; input = inputs.next()) {
console.log('midi input', input);
input.value.onmidimessage = midiOnMIDImessage;
}
midi.onstatechange = midiOnStateChange;
}
See working example here

Most browsers don't allow access to any hardware except the keyboard and mouse - for obvious security reasons, so it's unlikely that you could access a midi device unless it's plugged in as one of those devices.
You could try finding a driver that would translate midi output to key presses, and then deal with those in the browser, but this would be a single-computer solution only.

I am really excited by the upcoming Web MIDI API. As far as I know, its only under discussion and hasn't made it into any browsers yet.
Combined with the Web Audio API which has started to be implemented in some browsers already, it will be possible to have complete software synthesis in the browser. Exciting times.

Since Web MIDI API is still a draft, there is no way of direct access to MIDI events in the browser.
A simple workaround could be to write a small server where you register MIDI events and communicate them to your javascript using a websocket. This could be done quite easily in Python.

Best option for capturing audio and video online

We are looking for various options which will help us to record audio and video through web on various platforms including iPhone and iPad? Recorded media will be saved on the server. Any suggestions would be helpful... We are looking for a cross browser approach.
Thanks and Regards

I hate to say, but the only way to do it on desktop in a cross-browser fashion would be Adobe Flash. On iOS you need to develop Apps for that.
HTML5 will provide Device API at some point of future to achieve your goal.

You have to use Flash. Flash can access your webcam and microphone.
Of course, Flash won't fly on iDevices.
There, you'll need to write a native app. :)

Because you need to gain hardware control, you'll most likely need a native application that can access the hardware drivers and API.
My guess is that Java may be able to do the job.
Here's another StackOverflow question that may have an embedded solution.
Nevermind: iFamily doesn't support Java either

Displaying a local gstreamer stream in a browser

I have a camera feed coming into a linux machine using a V4l2 interface as the source for a gstreamer pipeline. I'm building an interface to control the camera, and I would like to do so in HTML/javascript, communicating to a local server. The problem is getting a feed from the gst pipeline into the browser. The options for doing so seem to be:
A loopback from gst to a v4l2 device, which is displayed using flash's webcam support
Outputting a MJPEG stream which is displayed in the browser
Outputting a RTSP stream which is displayed by flash
Writing a browser plugin
Overlaying a native X application over the browser
Has anyone had experience solving this problem before? The most important requirement is that the feed be as close to real time as possible. I would like to avoid flash if possible, though it may not be. Any help would be greatly appreciated.

You already thought about multiple solutions. You could also stream in ogg/vorbis/theora or vp8 to an icecast server, see the OLPC GStreamer wiki for examples.
Since you are looking for a python solution as well (according to your tags), have you considered using Flumotion? It's a streaming server written on top of GStreamer with Twisted, and you could integrate it with your own solution. It can stream over HTTP, so you don't need an icecast server.
Depending on the codecs, there are various tweaks to allow low-latency. Typically, with Flumotion, locally, you could get a few seconds latency, and that can be lowered I believe (x264enc can be tweaked to reach less than a second latency, iirc). Typically, you have to reduce the keyframe distance, and also limit the motion-vector estimation to a few nearby frames: that will probably reduce the quality and raise the bitrate though.

What browsers are you targeting? If you ignore Internet Explorer, you should be able to stream OGG/Theora video and/or WebM video direct to the browser using the tag. If you need to support IE as well though you're probably reduced to a flash applet. I just set up a web stream using Flumotion and the free version of Flowplayer http://flowplayer.org/ and it's working very well. Flowplayer has a lot of advanced functionality that I have barely even begun to explore.