Develop Reference

How to access input/output of worklet node from main file?

I have created a real time voice chat application for a game I am making. I got it to work completely fine using audiocontext.createScriptProcessor() method.
Here's the code, I left out parts that weren't relevant
//establish websocket connection
const audioData = []
//websocket connection.onMessage (data) =>
audioData.push(decodeBase64(data)) //push audio data coming from another player into array
//on get user media (stream) =>
const audioCtx = new AudioContext({latencyHint: "interactive", sampleRate: 22050,})
const inputNode = audioCtx.createMediaStreamSource(stream)
var processor = audioCtx.createScriptProcessor(2048, 1, 1);
var outputNode = audioCtx.destination
inputNode.connect(tunerNode)
processor.connect(outputNode)
processor.onaudioprocess = function (e) {
var input = e.inputBuffer.getChannelData(0);
webSocketSend(input) //send microphone input to other sockets via a function set up in a different file, all it does is base 64 encode then send.
//if there is data from the server, play it, else, play nothing
var output
if(audioData.length > 0){
output = audioData[0]
audioData.splice(0,1)
}else output = new Array(2048).fill(0)
};
the only issue is that the createScriptProccessor() method is deprecated. As recommended, I attempted to do this using Audio Worklet Nodes. However I quickly ran into a problem. I can't access the user's microphone input, or set the output from the main file where the WebSocket connection is.
Here is my code for main.js:
document.getElementById('btn').onclick = () => {createVoiceChatSession()}
//establish websocket connection
const audioData = []
//webSocket connection.onMessage (data) =>
audioData.push(data) //how do I get this data to the worklet Node???
var voiceChatContext
function createVoiceChatSession(){
voiceChatContext = new AudioContext()
navigator.mediaDevices.getUserMedia({audio: true}).then( async stream => {
await voiceChatContext.audioWorklet.addModule('module.js')
const microphone = voiceChatContext.createMediaStreamSource(stream)
const processor = new AudioWorkletNode(voiceChatContext, 'processor')
microphone.connect(processor).connect(voiceChatContext.destination)
}).catch(err => console.log(err))
}
Here is my code for module.js:
class processor extends AudioWorkletProcessor {
constructor() {
super()
}
//copies the input to the output
process(inputList, outputList) { // how do I get the input list data (the data from my microphone) to the main file so I can send it via websocket ???
for(var i = 0; i < inputList[0][0].length; i++){
outputList[0][0][i] = inputList[0][0][i]
outputList[0][1][i] = inputList[0][1][i]
}
return true;
}
}
registerProcessor("processor", processor);
So I can record and process the input, but I can't send input via WebSocket or pass in data that is coming from the server to the worklet node because I can't access the input list or output list from the main file where the WebSocket connection is. Does anyone know a way to work around this? Or is there a better solution that doesn't use audio worklet nodes?
Thank you to all who can help!

I figured it out, all I needed to do was use the port.onmessage method to exchange data between the worklet and the main file.
processor.port.onmessage = (e) => {//do something with e.data}

Getting a function to execute on a node server when data recieved

I'll try to keep this simple. I'm working on a paging app.
What I have: A terminal without internet, running a desktop app made from node.js using express,ejs, ect. I also have a physical server with internet access and a mssql database on this machine I have a node.js server that interfaces with the database, collects the data and send sends it over to terminal via websocket when the app is launched, the node.js app gets this data and is rendered to the web interface using ejs. This data is displayed in a form with a button that when clicked fires the app.post route. In the app.post that data is packaged in an array, and sent back to the server using a second websocket connection. I can then take that form data (a name, phone#, and a radio button value) and form a SMS message using and using twilio send a message to that number (with addition info)
Yes, I have two websocket connections one Server > client serves data to web app
and another client > server serves form data to the server.
What's working: I get data from the boh server/database > webserver/client and data is displayed correctly when I hit the button data is packaged and send back to the boh server.
On the boh server i have a function that get the data array and parses it out, and sends an SMS message using twilio
My issue: I have to restart the server app to get it to process the data and send the message, if I hit the page button, it does all the stuff in the background it should, packages the array and sends the info to the server. The server is waiting for the data to be sent and client has sent the data, however it will only send off the text message is a stop and restart the node.js server, if i do that, the server starts and runs through the initial process of getting the sql data, and sets it up to be called when the app launches, then continues on to read that data was sent from the client, received and it will parse the data from the SMS message through a function send the message, wait a few seconds and then grab the response and confirm delivery of message. I am quite sure I am missing something basic here, but I have different functions but nothing I do will seem to get it to fire when the data is received.
I'm new at node and not very advanced in js, but I understand on some level why its not firing as is right, script is running, data has not been sent, so it stops when it gets here, then i hit the button, and it does nothing, because the script is stopped, and it doesn't has know way of knowing that data was sent, but when i rerun the script, the data that was sent is still sitting there wait to be received, so it recognizes the open websocket, and the sent data and work appropriately, I feel like a am missing something on the server side that tell it to wait for the data send but have not been able to make it work
twilws.onopen = async() => {//when the page button is presses, it starts a websocket server on the client, if that makes sense
twilws.send('test')
if (twilws.readyState === WebSocket.OPEN) {
logger.info('Twilio Sockpuppet Connected')
} else {
logger.error('Twilio Sockpuppet Fail')
}
twilws.onmessage = async (e) => {//after i hit the page button the server is on and the array is sent, i am trying to get this to be waiting for data and when it arrives go, but it will only do that when i restart the script
try {
data = JSON.parse(e.data);//parse the data
logger.info('got the edata ' + e.data)
//main(data) this will fire but not when the button is pressed.
} catch (er) {
logger.error('socket parse error: ' + e.data);
}
}
}
twilws.onclose = () => {//close the connection, purges that data so that the websocket can be recreated and an array with new data sent.
logger.info('Web Socket Connection Closed');
twilws.close(1000, 'all done');
};
Here is the whole server
I believe i may be blocking something, sorry about the bad formatting i have been changing and trying different things for a couple weeks off and on now and have not had a chance to clean thing up.
const express = require("express");
const cors = require("cors");
const app = express();
const db = require("./app/models");
const twilioconfig = require('./configs/twilioConfig.js');
const path = require('path');
const WebSocket = require('ws');
const sql = require('mssql');
const WebSocketServer = require('websocket').server;
const WebSocketClient = require('websocket').client;
const WebSocketFrame = require('websocket').frame;
/*const WebSocketRouter = require('websocket').router;*/
const W3CWebSocket = require('websocket').w3cwebsocket;
const http = require('http');
const https = require('https');
const twilio = require('twilio');
const { Console } = require("console");
app.disable('view cache');
const pino = require('pino')
const SonicBoom = require('sonic-boom')
const logger = require('pino')()
const transport = pino.transport({
target: 'pino/file',
options: { destination: './logs/logs.txt', level: 'info', mkdir: true, append: true }
})
pino(transport)
/*const dbConfig2 = require("./app/config/db.config.js");*/
/*const config = require("./app/config/config.js");*/
const { client } = require("websocket");
const { err } = require("./node_modules/pino-std-serializers/index");
const { setInterval } = require("node:timers/promises");
app.use(pino)
//const webserver = app.listen(8080, function () {
// console.log('Node WEb Server is running..');
//});
var params = {
autoReconnect: false, //Enable/Disable reconnect when the server closes connection (boolean)
autoReconnectInterval: 1000, //Milliseconds to wait between reconnect attempts (number)
autoReconnectMaxRetries: 600, //Max number of reconnect attempts to allow (number)
requestTimeout: 30000, //Milliseconds to wait for a response before resending the request (number)
requestRetryInterval: 5000, //Milliseconds between request retry checks. This garbage collects the retry queue (number)
requestRetryQueueMaxLength: 10 //Max queue length of retry queue before old messages start getting dropped (number)
}
var wss = new WebSocket.Server({ port: 8081 })
sql.connect(config, function (err) {
if (err)
logger.error(err);
const sqlRequest = new sql.Request();
const sqlQuery = "SELECT TOP 5 guest_name,guest_phone_number,CONVERT(varchar,creation_time, 126) AS creation_time,CONVERT(varchar,last_modified_timestamp, 126) AS last_modified_timestamp,party_size from dbo.WaitList where status = '4' AND CAST(creation_time as date) = CAST( GETDATE() AS Date ) ORDER BY creation_time ASC";
logger.info("query passes preflight.....lets get data")
sqlRequest.query(sqlQuery, function (err, data) {
if (err) {
logger.error(err)
} else {
logger.info("We do preliminary query now.")
}
//console.table(data.recordset);
//logger.info('rows affected ' + data.rowsAffected);
//console.log(data.recordset[0]);
var array = [];
for (let i = 0; i < data.rowsAffected; i++) {
var a = data.recordset[i];
array.push(a);
}
wss.on('connection', ws => {
logger.info('Client connection established')
ws.on('message', function () {
sqlRequest.query(sqlQuery, function (err, data) {
if (err) {
logger.error(err)
}
//console.table(data.recordset);
//logger.info(data.rowsAffected);
//console.log(data.recordset[0]);
var array = [];
for (let i = 0; i < data.rowsAffected; i++) {
var a = data.recordset[i];
array.push(a);
}
})
wss.clients
.forEach(client => {
logger.info('sending data')
client.send(JSON.stringify(array))
})
})
})
})
});
const twilws = new W3CWebSocket('ws://172.16.0.101:8082', params);
twilws.onopen = () => {//when the page button is presses, it starts a websocket server on the client, if that makes sense
twilws.send('test')
if (twilws.readyState === WebSocket.OPEN) {
logger.info('Twilio Sockpuppet Connected')
} else {
logger.error('Twilio Sockpuppet Fail')
}
twilws.onmessage = async (e) => {//after i hit the page button the server is on and the array is sent, i am trying to get this to be waiting for data and when it arrives go, but it will only do that when i restart the script
try {
data = JSON.parse(e.data);//parse the data
logger.info('got the edata ' + e.data)
//main(data) this will fire but not when the button is pressed.
} catch (er) {
logger.error('socket parse error: ');
}
}
}
twilws.onclose = () => {//close the connection, purges that data so that the websocket can be recreated and an array with new data sent.
logger.info('Web Socket Connection Closed');
twilws.close(1000, 'all done');
};
//async function main(s) {
// /* these settings are loaded from configs/twilioConfig.js, go here to set the store account info and edit message body.*/
// const TWILIO_ACCOUNT_SID = twilioconfig.twilioOptions.TWILIO_ACCOUNT_SID;
// const TWILIO_AUTH_TOKEN = twilioconfig.twilioOptions.TWILIO_AUTH_TOKEN;
// const STORE_TWILIO_NUMBER = twilioconfig.twilioOptions.STORE_TWILIO_NUMBER;
// const TEXT_TWILIO_BODY = twilioconfig.twilioOptions.TEXT_TWILIO_BODY;
// var messarray = new Array([s])
// var num = JSON.stringify(messarray[0][0][0])
// var state = JSON.stringify(messarray[0][0][1])
// logger.info(num + ', ' + state)
// var readyin = state
// var custnum = num
// /*logger.info(TWILIO_ACCOUNT_SID + ' ' + TWILIO_AUTH_TOKEN)*/
// var customer = new twilio(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN);//these are our twilio account sid and token, set in twilioConfig, get from oneNote
// if (readyin == 0) {
// waitmsg = TEXT_TWILIO_BODY + "Your Table is Ready!";
// } else if (readyin == 15) {
// waitmsg = TEXT_TWILIO_BODY + "Your Table will be ready in about 15 minutes!";
// }
// logger.info(`Recieved Form data.....Twilio data read successfully....`);//log that we got the data from the gui form
// // send the message to the customer number through twilio, custnum from form, from is the store number, alson in twilioConfig
// customer.messages.create({
// to: custnum,
// from: STORE_TWILIO_NUMBER,
// body: waitmsg
// })
// .then(message => {
// var messageid = message.sid
// logger.info(messageid + ' Hey i found this, we might need it in a sec')
// const msid = JSON.stringify({ "MessageSid": messageid })
// const twilid = messageid
// setTimeout(getStatus, 5000);
// function getStatus() {
// customer.messages(messageid).fetch()
// .then(call => {
// const d = new Date(call.dateCreated).toLocaleString();
// const messageStatus = call.status
// var twil_response_array = [twilid, messageStatus, d]
// logger.info(twil_response_array)
// wss.on('connection', ws => {
// ws.on('message', function () {
// })
// wss.clients
// .forEach(client => {
// client.send(twil_response_array)
// })
// })
// })
// }
// //})
app.listen(8180, function () {
logger.info('Server is running..');
});
});

deinitialize audio recording started via getUserMedia

I'm trying to create audio stream from browser and send it to server.
Here is the code:
let recording = false;
let localStream = null;
const session = {
audio: true,
video: false
};
function start () {
recording = true;
navigator.webkitGetUserMedia(session, initializeRecorder, onError);
}
function stop () {
recording = false;
localStream.getAudioTracks()[0].stop();
}
function initializeRecorder (stream) {
localStream = stream;
const audioContext = window.AudioContext;
const context = new audioContext();
const audioInput = context.createMediaStreamSource(localStream);
const bufferSize = 2048;
// create a javascript node
const recorder = context.createScriptProcessor(bufferSize, 1, 1);
// specify the processing function
recorder.onaudioprocess = recorderProcess;
// connect stream to our recorder
audioInput.connect(recorder);
// connect our recorder to the previous destination
recorder.connect(context.destination);
}
function onError (e) {
console.log('error:', e);
}
function recorderProcess (e) {
if (!recording) return;
const left = e.inputBuffer.getChannelData(0);
// send left to server here (socket.io can do the job). We dont need stereo.
}
when function start is fired, the samples can be catched in recorderProcess
when function stop is fired, the mic icon in browser disappears, but...
unless I put if (!recording) return in the beginning of recorderProcess, it still process samples.
Unfortunately it's not a solution at all - the samples are still being received by recordingProcess and if I fire start functiono once more, it will get all samples from previous stream and from new one.
My question is:
How can I stop/start recording without such issue?
or if it's not best solution
How can I totally remove stream in stop function, to safely initialize it again anytime?

recorder.disconnect() should help.
You might want to consider the new MediaRecorder functionality in Chrome Canary shown at https://webrtc.github.io/samples/src/content/getusermedia/record/ (currently video-only I think) instead of the WebAudio API.

WebRTC Failed to Add a Third Peer: "Cannot set remote answer in state stable"

I am writing a multi-peer WebRTC video chat.
Two peers have no trouble connecting, no error or warning in console, and video works well, but I cannot add a third party to the chat successfully.
On the host (the first participant, Firefox), the error appear as "Cannot set remote answer in state stable" when trying to create an answer. At the second participant (Chrome), the error is "Failed to set remote answer sdp: Called in wrong state: STATE_INPROGRESS". At he third peer, the error is "the error is "Failed to set remote answer sdp: Called in wrong state: STATE_RECEIVEDINITIATE".
As it turn out, the first peer failed to have video with the third peer. Other two links appear fine.
Generally, my communication model is as below, self_id is a unique id per each peer in the session, and locate_peer_connection() will return the local peer_connection of the particular peer from which we receive message:
a new client send "peer_arrival" to the session using signalling server
all peers already in the session setlocaldescription, create offer and send to the new client
new client create answers to all other peers and setremotedescription
new client has video coming up
Signalling is done using WebSocket on a node.js server.
I have some of the core code below, some more note:
self_id is an unique id per client in a session
peer_connection stores peerConnection to other nodes, and peer_id store the respective user_id of these objects
local_stream is the local video stream from getUserMedia (already considered different browser)
Any insights in to the issue? Is there something wrong with my model?
// locate a peer connection according to its id
function locate_peer_connection(id) {
var index = peer_id.indexOf(id);
// not seen before
if (index == -1) {
add_peer_connection();
peer_id.push(id);
index = peer_id.length - 1;
}
return index;
}
// add a peer connection
function add_peer_connection() {
console.log('add peer connection');
// add another peer connection for use
peer_connection.push(new rtc_peer_connection({ "iceServers": [{ "url": "stun:"+stun_server }]}));
// generic handler that sends any ice candidate to the other peer
peer_connection[peer_connection.length - 1].onicecandidate = function (ice_event) {
if (ice_event.candidate) {
signaling_server.send(
JSON.stringify({
type: "new_ice_candidate",
candidate: ice_event.candidate,
id: self_id,
token:call_token
})
);
console.log('send new ice candidate, from ' + self_id);
}
};
// display remote video streams when they arrive using local <video> MediaElement
peer_connection[peer_connection.length - 1].onaddstream = function (event) {
video_src.push(event.stream); // store this src
video_src_id.push(peer_connection.length - 1);
if (video_src.length == 1) { // first peer
connect_stream_to_src(event.stream, document.getElementById("remote_video"));
// video rotating function
setInterval(function() {
// rorating video src
var video_now = video_rotate;
if (video_rotate == video_src.length - 1) {
video_rotate = 0;
} else {
video_rotate++;
}
var status = peer_connection[video_src_id[video_rotate]].iceConnectionState;
if (status == "disconnected" || status == "closed") { // connection lost, do not show video
console.log('connection ' + video_rotate + ' liveness check failed');
} else if (video_now != video_rotate) {
connect_stream_to_src(video_src[video_rotate], document.getElementById("remote_video"));
}
}, 8000);
// hide placeholder and show remote video
console.log('first remote video');
document.getElementById("loading_state").style.display = "none";
document.getElementById("open_call_state").style.display = "block";
}
console.log('remote video');
};
peer_connection[peer_connection.length - 1].addStream(local_stream);
}
// handle new peer
function new_peer(signal) {
// locate peer connection
var id = locate_peer_connection(signal.id);
console.log('new peer ' + id);
// create offer
peer_connection[id].createOffer(function(sdp) {
peer_connection[id].setLocalDescription(sdp,
function() { // call back
console.log('set local, send offer, connection '+ id);
signaling_server.send(
JSON.stringify({
token: call_token,
id: self_id,
type:"new_offer",
sdp: sdp
})
);
}, log_error);
}, log_error);
}
// handle offer
function new_offer_handler(signal) {
var id = locate_peer_connection(signal.id);
console.log('new offer ' + id);
// set remote description
peer_connection[id].setRemoteDescription(
new rtc_session_description(signal.sdp),
function() { // call back
peer_connection[id].createAnswer(function(sdp) {
peer_connection[id].setLocalDescription(sdp, function () {
console.log('set local, send answer, connection '+ id);
signaling_server.send(
JSON.stringify({
token: call_token,
id: self_id,
type:"new_answer",
sdp: sdp
})
);
},
log_error);
}, log_error);
}, log_error);
}
// handle answer
function new_answer_handler(signal) {
var id = locate_peer_connection(signal.id);
console.log('new answer ' + id);
peer_connection[id].setRemoteDescription(new rtc_session_description(signal.sdp),
function() {
console.log('receive offer answer, set remote, connection '+ id);
}
, log_error);
}
// handle ice candidate
function ice_candidate_handler(signal) {
var id = locate_peer_connection(signal.id);
console.log('get new_ice_candidate from ' + id);
if (typeof(RTCIceCandidate) != "undefined") {
peer_connection[id].addIceCandidate(
new RTCIceCandidate(signal.candidate)
);
} else { // firefox
peer_connection[id].addIceCandidate(
new mozRTCIceCandidate(signal.candidate)
);
}
}
function event_handler(event) {
var signal = JSON.parse(event.data);
if (signal.type === "peer_arrival") {
new_peer(signal);
} else if (signal.type === "new_ice_candidate") {
ice_candidate_handler(signal);
} else if (signal.type === "new_offer") { // get peer description offer
new_offer_handler(signal);
} else if (signal.type === "new_answer") { // get peer description answer
new_answer_handler(signal);
} else if (signal.type === "new_chat_message") { // chat message and file sharing info
add_chat_message(signal);
} else if (signal.type === "new_file_thumbnail_part") { // thumbnail
store_file_part(signal.name, "thumbnail", signal.id, signal.part, signal.length, signal.data);
if (file_store[signal.id].thumbnail.parts.length == signal.length) {
document.getElementById("file_list").innerHTML = get_file_div(signal.id, signal.name)+document.getElementById("file_list").innerHTML;
document.getElementById("file-img-"+signal.id).src = file_store[signal.id].thumbnail.parts.join("");
}
} else if (signal.type === "new_file_part") { // file
console.log('get new_file_part ' + signal.id);
store_file_part(signal.name, "file", signal.id, signal.part, signal.length, signal.data);
update_file_progress(signal.name, signal.id, file_store[signal.id].file.parts.length, signal.length);
}
}
// generic error handler
function log_error(error) {
console.log(error);
}

I might be wrong, but these two must be your major problems:
signaling_server.send(... I am not seeing any target here, so guessing that the server just broadcasts this message to everyone. When you are sending an sdp to already established peer connection, you are bound to get the error which you are getting now. My suggesting would be to add a target id in the message, either the server can forward it to that particular peer or, server can just broadcast, but event_handler of the peer can check if the target id of message is same as it's own id, if not, just ignore the message.
onicecandidate event, you are broadcasting the ICE candidates to all remote peers, again this is meant for single peer, another issue might be, addIceCandidate on PeerConnection before setting it's local and remote description would throw error, you need to some sort of mechanism to handle this( add ICE candidates only after setting the connections descriptions).
finally a suggestion. I am guessing peer_connection is an Array, if you change it Object, you can remove the redundancy of locate_peer_connection,
you can do something like.
if(peer_connection[signal.id]){
//do something...
}else{
peer_connection[signal.id] = new PeerConnection(...
}

i had the same problem when i was implementing one-to-many rtc broadcast, and what mido22 said is right. you might be sending/resetting existing established peer object with other incoming client. you have to create new RTCPeerConenction object for every new incoming client. the general work flow would be as follow.
peerconnection=[];//empty array
then you initialize your media devices with getUserMedia
and store media stream into global variable so that it can be added when creating offer.
once this is done you inform your singalling server with unique id
your signalling server may then broadcast this to all clients except from which it is received. each client will then check if that unique id does exist in there peerconnection array and like mido22 said you can do this as
if(peerconnection[signal.id])
{
//do something with existing peerconnections
}
else
{
peerconnection[signal.id]=new RTCPeerConnection({"stun_server_address"});
//register peerconnection[signal.id].onicecandidate callback
//register peerconnection[signal.id].onaddstream callback
//createoffer
//if local stream is ready then
peerconnection[signal.id].addStream(localStream);
//and rest of the stuff go as it is like in one-to-one call..
//setLocalDescriptor setRemoteDescriptor
}

Detecting that the peer's browser was closed in a webrtc videochat

I've been implementing a webrtc videochat.
Everything is working smoothly except for the case when the peer closes the browser.
I've been trying to handle this event by implementing an onended callback on the remote mediastream. Though, this callback does not seem to ever be called.
How can I detect that the peer's browser has been closed or that the connection was finished on the other side?

You can use the ICE connection status to determine this. If you disconnect one peer it takes some seconds (~5?) to recoginize it, but it works even without a signalling server.
(assuming you called your peer connection pc)
pc.oniceconnectionstatechange = function() {
if(pc.iceConnectionState == 'disconnected') {
console.log('Disconnected');
}
}

Use signaling gateway to send message to all connected peers that you're leaving; like this:
window.addEventListener('beforeunload', function () {
userLeft();
}, false);
window.addEventListener('keyup', function (e) {
if (e.keyCode == 116)
userLeft();
}, false);
function userLeft() {
signalingGateway.send({
userLeft: true,
whoLeft: 'user-id'
});
}
signalingGateway.on('message', function (signal) {
if (signal.userLeft && signal.whoLeft != 'current-user-id') {
var relevantPeer = listOfPeers[signal.whoLeft];
if (relevantPeer) {
relevantPeer.close();
relevantPeer = null;
}
var relevantLocalStreams = listOfLocalStreams[signal.whoLeft];
if (relevantLocalStreams.length) {
for (var i = 0; i < relevantLocalStreams.length; i++) {
if (relevantLocalStreams[i].stop) {
relevantLocalStreams[i].stop();
}
// it is suggested to stop media tracks instead!
}
}
}
});

I am using a solution like this:
const connection = new RTCPeerConnection(configuration);
connection.onconnectionstatechange = () => {
const connectionStatus = connection.connectionState;
if (["disconnected", "failed", "closed"].includes(connectionStatus)) {
console.log("disconnected");
}
};

I also had this issue. I've solved this by closing the connection at the desired moment(eg. on destructor).
A <- connected -> B
...
Close(A); // after this, B's ice status will go almost instantly to 'kDisconnected'.

Just Use
connection.onclose = function(event){
event.useid // gives the connection closed userid
}