I was able to set up an Arduino to stream audio from a microphone into a linux server that's hosting an MQTT server. I then have a golang script that subscribes to the MQTT server, saves payload to disk as a binary file, and converts the binary file to a .WAV file with FFMPEG.
Is it possible to have a web browser use only client side code to subscribe to the same MQTT server, receive the audio payload from the Arduino, and stream the audio in near-real-time to the human listener's computer speakers? I see a Paho Javascript Client library that can help me connect to MQTT, but it seems to receive payloads as string, which isn't evident to me on how I'd stream audio content with. Hence, why I'm asking if it's even practical/feasible?
Or will I need to build another server-side script to stream MQTT data as audio data for a web client?
To ensure it works in all environments, ensure that you use MQTT over WebSocket to connect to the server.
Here is a discussion of this: Can a web browser use MQTT?
Look closer at the paho doc, there is a fiction to get the message payload as binary data using the message.payloadBytes field.
payloadBytes | ArrayBuffer | read only The payload as an ArrayBuffer
An example is described here:
https://www.hardill.me.uk/wordpress/2014/08/29/unpacking-binary-data-from-mqtt-in-javascript/
But basically you end up withan ArrayBuffer holding the binary data that you can then coonvert to a Typed Array and read back values at what ever offset you need.
Related
everyone .I need some help for some stream problems, here is my to do list.
I want stream my camera stream，microphone stream via broswer to a media server, and a python server
need to pull this stream to do some asr, tts things. After that ,It generate a reply stream and push back to media server ,and the broswer pull this stream. My plan is like this.
broswer stream to rtmp server use rtmp protocol
python server pull this stream use rtmp
python server push its reply to rtmp server
broswer pull it
but, my question is how can broswer stream rtmp to rtmp server? cause As far as I know I can just use webrtc to do this in broswer?
so the process maybe like this?
broswer stream to janus(or other webrtc server) use webrtc
python use webrtc to pull this stream
python server push its reply to janus
broswer pull this stream use webrtc
but, I'm not sure whether step 2 or step 3 can be done, cause I don't know how to use webrtc in a python environment and without a broswer and use my own stream(not a camera stream)
or can janus convert and push webrtc stream to rtmp server ?
Any helps will be appraciated, thanks.
I know that with mediasoup you can send the camera to the server with webrtc. Then you can use the server to retransmit it to your algos with rtp and lastly send the response back to the client using webrtc again.
I'm working on a project and I need to send an audio stream from a webpage (through javascript) to a server written in C++. Is this possible? How can I do this? I was thinking on use WebRTC and a WebRTC library for C++ but I don't really know hoy to achieve this.
In general I need some king of webserver in C++, that allows me to send/recieve audio stream and json and works with multiple web clients.
I have worked with Socket.io and once I coded a webserver in Java EE 7, with those I was able to send/recieve json from the webpage but I don't really know if I can send audio stream via websocket or json.
The question (or implementation in answer to the question) really consists of two parts, which are:
How to send audio stream from browser in Javascript
How to receive audio stream on server in C/C++
This is because sending data over the network only loosely couples the client and the server when they use the same protocol. You could write a server in C++, then write two different clients that communicate with it, one in Javascript, then also a desktop app written in Java.
Javascript on Client Side
For the client side, sending audio from the browser in Javascript should follow the normal libraries available for WebRTC; the WebRTC site has some useful information on this, including a video streaming example here (https://webrtc.github.io/samples/)
Some of the links which might be of interest on that page:
Audio-only getUserMedia() output to local audio element
Stream from a video element to a video element
There are some StackOverflow answers already about WebRTC and audio in javascript, here are a couple, these (and libraries) will be more plentiful than C++ questions on the topic:
Sending video and audio stream to server
Sending a MediaStream to host Server with WebRTC after it is captured by getUserMedia
For the C++ Server:
The WebRTC site has a link to the Native API for the libraries here (https://webrtc.org/native-code/native-apis/) and an excellent simple example of a peer connection WebRTC server using them is here (https://webrtc.googlesource.com/src/+/master/examples/peerconnection). It also has an implementation of a C++ client there, which may help in testing the server to get it working first, or see the general principles.
There are several solutions for saving a data blob to a file in a browser environment. I normally use FileSaver.js.
In this particular case, though, my data comes in from a websocket stream, and the resulting file might be so large (hundreds of MB) that concatenating it in memory really kills performance.
Is it at all possible to open a stream that data can be written to as it comes in?
Edit:
The websocket is an XMPP connection. The file transfer is negotiated via Stream Initiation (XEP-0095) using the SI File Transfer (XEP-0096) protocol; the data is then transferred via an In Band Bytestream (XEP-0047).
Basically, the web application gets the data in base64-encoded chunks of 4096 bytes, and I'd like to know if there is any way to let the user save the file before all the data is received.
The client sends a GET request to the server with the name of the audio file it wants. The server, which has access to the audio files, constructs a path to the mp3.
Right now I am calling result.sendFile('/audio/mysong.mp3') on the server, and getting the binary data on the client.
I haven't found a way to construct the mp3 object on the client to where I can just create a new Audio object and call .play(). Is there any way to do this?
Is there a better way to send the mp3 to the client?
Thanks
I connected an Arduino to my laptop using USB, and I can read the serial data using Processing.
Is there any way to get this data in real time into a local webbrowser? For example, a text field that shows the value from the serial port? It does not have to be connected to the internet.
The JavaScript version of Processing does not support the following code, which would have been the ideal solution.
The Processing code is:
myPort = new Serial(this, Serial.list()[0], 9600);
// read a byte from the serial port
int inByte = myPort.read();
// print it
println(inByte);
// now send this value somewhere...?
// ehm...
There is no way to directly access the local machine from the web browser. For security reasons browsers have very limited access to the machines resources.
To do this one option would be to write an extension for the browser of your choosing. Even though extensions also have a variety of limitations.
Option two would be to use local server to provide the functionality you need. Personally I recommend using node.js (it's light weight, fast and easy to implement). You can read/write serial data using https://github.com/rwaldron/johnny-five (as #kmas suggested) or https://github.com/voodootikigod/node-serialport and than you can use http://socket.io/ to create a simple service and connect to it though the browser. Socket.io uses WebSockets in modern browsers and works excepionally well for real-time connections.
I had a similar problem to solve. My Data acquisition system (DAQ) (like your arduino) relays data in HTTP, TCP, FTP, as well as serial. I had to capture it on the server and then send it to my webpage in real-time.
The hack I wrote uses nodejs at the server, and connects DAQ to the server using TCP sockets using the "net" module of nodejs and connects the server to the HTML page using socket.io.
The code and context can be found in "How to get sensor data over TCP/IP in nodejs?".
I use TCP as I wanted to transmit data over a long distance. You need to modify the socket protocol to serial.
For serial-to-TCP redirection, you may use bloom from sensorMonkey for Windows or their processing sketch for *nix/Mac OS.
if you want to send or receive serial data from Arduino to JavaScript in Processing code editor just go to Sketch -> Import Library -> Serial or just write import processing.serial.*;