Develop Reference

Getting a function to execute on a node server when data recieved

I'll try to keep this simple. I'm working on a paging app.
What I have: A terminal without internet, running a desktop app made from node.js using express,ejs, ect. I also have a physical server with internet access and a mssql database on this machine I have a node.js server that interfaces with the database, collects the data and send sends it over to terminal via websocket when the app is launched, the node.js app gets this data and is rendered to the web interface using ejs. This data is displayed in a form with a button that when clicked fires the app.post route. In the app.post that data is packaged in an array, and sent back to the server using a second websocket connection. I can then take that form data (a name, phone#, and a radio button value) and form a SMS message using and using twilio send a message to that number (with addition info)
Yes, I have two websocket connections one Server > client serves data to web app
and another client > server serves form data to the server.
What's working: I get data from the boh server/database > webserver/client and data is displayed correctly when I hit the button data is packaged and send back to the boh server.
On the boh server i have a function that get the data array and parses it out, and sends an SMS message using twilio
My issue: I have to restart the server app to get it to process the data and send the message, if I hit the page button, it does all the stuff in the background it should, packages the array and sends the info to the server. The server is waiting for the data to be sent and client has sent the data, however it will only send off the text message is a stop and restart the node.js server, if i do that, the server starts and runs through the initial process of getting the sql data, and sets it up to be called when the app launches, then continues on to read that data was sent from the client, received and it will parse the data from the SMS message through a function send the message, wait a few seconds and then grab the response and confirm delivery of message. I am quite sure I am missing something basic here, but I have different functions but nothing I do will seem to get it to fire when the data is received.
I'm new at node and not very advanced in js, but I understand on some level why its not firing as is right, script is running, data has not been sent, so it stops when it gets here, then i hit the button, and it does nothing, because the script is stopped, and it doesn't has know way of knowing that data was sent, but when i rerun the script, the data that was sent is still sitting there wait to be received, so it recognizes the open websocket, and the sent data and work appropriately, I feel like a am missing something on the server side that tell it to wait for the data send but have not been able to make it work
twilws.onopen = async() => {//when the page button is presses, it starts a websocket server on the client, if that makes sense
twilws.send('test')
if (twilws.readyState === WebSocket.OPEN) {
logger.info('Twilio Sockpuppet Connected')
} else {
logger.error('Twilio Sockpuppet Fail')
}
twilws.onmessage = async (e) => {//after i hit the page button the server is on and the array is sent, i am trying to get this to be waiting for data and when it arrives go, but it will only do that when i restart the script
try {
data = JSON.parse(e.data);//parse the data
logger.info('got the edata ' + e.data)
//main(data) this will fire but not when the button is pressed.
} catch (er) {
logger.error('socket parse error: ' + e.data);
}
}
}
twilws.onclose = () => {//close the connection, purges that data so that the websocket can be recreated and an array with new data sent.
logger.info('Web Socket Connection Closed');
twilws.close(1000, 'all done');
};
Here is the whole server
I believe i may be blocking something, sorry about the bad formatting i have been changing and trying different things for a couple weeks off and on now and have not had a chance to clean thing up.
const express = require("express");
const cors = require("cors");
const app = express();
const db = require("./app/models");
const twilioconfig = require('./configs/twilioConfig.js');
const path = require('path');
const WebSocket = require('ws');
const sql = require('mssql');
const WebSocketServer = require('websocket').server;
const WebSocketClient = require('websocket').client;
const WebSocketFrame = require('websocket').frame;
/*const WebSocketRouter = require('websocket').router;*/
const W3CWebSocket = require('websocket').w3cwebsocket;
const http = require('http');
const https = require('https');
const twilio = require('twilio');
const { Console } = require("console");
app.disable('view cache');
const pino = require('pino')
const SonicBoom = require('sonic-boom')
const logger = require('pino')()
const transport = pino.transport({
target: 'pino/file',
options: { destination: './logs/logs.txt', level: 'info', mkdir: true, append: true }
})
pino(transport)
/*const dbConfig2 = require("./app/config/db.config.js");*/
/*const config = require("./app/config/config.js");*/
const { client } = require("websocket");
const { err } = require("./node_modules/pino-std-serializers/index");
const { setInterval } = require("node:timers/promises");
app.use(pino)
//const webserver = app.listen(8080, function () {
// console.log('Node WEb Server is running..');
//});
var params = {
autoReconnect: false, //Enable/Disable reconnect when the server closes connection (boolean)
autoReconnectInterval: 1000, //Milliseconds to wait between reconnect attempts (number)
autoReconnectMaxRetries: 600, //Max number of reconnect attempts to allow (number)
requestTimeout: 30000, //Milliseconds to wait for a response before resending the request (number)
requestRetryInterval: 5000, //Milliseconds between request retry checks. This garbage collects the retry queue (number)
requestRetryQueueMaxLength: 10 //Max queue length of retry queue before old messages start getting dropped (number)
}
var wss = new WebSocket.Server({ port: 8081 })
sql.connect(config, function (err) {
if (err)
logger.error(err);
const sqlRequest = new sql.Request();
const sqlQuery = "SELECT TOP 5 guest_name,guest_phone_number,CONVERT(varchar,creation_time, 126) AS creation_time,CONVERT(varchar,last_modified_timestamp, 126) AS last_modified_timestamp,party_size from dbo.WaitList where status = '4' AND CAST(creation_time as date) = CAST( GETDATE() AS Date ) ORDER BY creation_time ASC";
logger.info("query passes preflight.....lets get data")
sqlRequest.query(sqlQuery, function (err, data) {
if (err) {
logger.error(err)
} else {
logger.info("We do preliminary query now.")
}
//console.table(data.recordset);
//logger.info('rows affected ' + data.rowsAffected);
//console.log(data.recordset[0]);
var array = [];
for (let i = 0; i < data.rowsAffected; i++) {
var a = data.recordset[i];
array.push(a);
}
wss.on('connection', ws => {
logger.info('Client connection established')
ws.on('message', function () {
sqlRequest.query(sqlQuery, function (err, data) {
if (err) {
logger.error(err)
}
//console.table(data.recordset);
//logger.info(data.rowsAffected);
//console.log(data.recordset[0]);
var array = [];
for (let i = 0; i < data.rowsAffected; i++) {
var a = data.recordset[i];
array.push(a);
}
})
wss.clients
.forEach(client => {
logger.info('sending data')
client.send(JSON.stringify(array))
})
})
})
})
});
const twilws = new W3CWebSocket('ws://172.16.0.101:8082', params);
twilws.onopen = () => {//when the page button is presses, it starts a websocket server on the client, if that makes sense
twilws.send('test')
if (twilws.readyState === WebSocket.OPEN) {
logger.info('Twilio Sockpuppet Connected')
} else {
logger.error('Twilio Sockpuppet Fail')
}
twilws.onmessage = async (e) => {//after i hit the page button the server is on and the array is sent, i am trying to get this to be waiting for data and when it arrives go, but it will only do that when i restart the script
try {
data = JSON.parse(e.data);//parse the data
logger.info('got the edata ' + e.data)
//main(data) this will fire but not when the button is pressed.
} catch (er) {
logger.error('socket parse error: ');
}
}
}
twilws.onclose = () => {//close the connection, purges that data so that the websocket can be recreated and an array with new data sent.
logger.info('Web Socket Connection Closed');
twilws.close(1000, 'all done');
};
//async function main(s) {
// /* these settings are loaded from configs/twilioConfig.js, go here to set the store account info and edit message body.*/
// const TWILIO_ACCOUNT_SID = twilioconfig.twilioOptions.TWILIO_ACCOUNT_SID;
// const TWILIO_AUTH_TOKEN = twilioconfig.twilioOptions.TWILIO_AUTH_TOKEN;
// const STORE_TWILIO_NUMBER = twilioconfig.twilioOptions.STORE_TWILIO_NUMBER;
// const TEXT_TWILIO_BODY = twilioconfig.twilioOptions.TEXT_TWILIO_BODY;
// var messarray = new Array([s])
// var num = JSON.stringify(messarray[0][0][0])
// var state = JSON.stringify(messarray[0][0][1])
// logger.info(num + ', ' + state)
// var readyin = state
// var custnum = num
// /*logger.info(TWILIO_ACCOUNT_SID + ' ' + TWILIO_AUTH_TOKEN)*/
// var customer = new twilio(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN);//these are our twilio account sid and token, set in twilioConfig, get from oneNote
// if (readyin == 0) {
// waitmsg = TEXT_TWILIO_BODY + "Your Table is Ready!";
// } else if (readyin == 15) {
// waitmsg = TEXT_TWILIO_BODY + "Your Table will be ready in about 15 minutes!";
// }
// logger.info(`Recieved Form data.....Twilio data read successfully....`);//log that we got the data from the gui form
// // send the message to the customer number through twilio, custnum from form, from is the store number, alson in twilioConfig
// customer.messages.create({
// to: custnum,
// from: STORE_TWILIO_NUMBER,
// body: waitmsg
// })
// .then(message => {
// var messageid = message.sid
// logger.info(messageid + ' Hey i found this, we might need it in a sec')
// const msid = JSON.stringify({ "MessageSid": messageid })
// const twilid = messageid
// setTimeout(getStatus, 5000);
// function getStatus() {
// customer.messages(messageid).fetch()
// .then(call => {
// const d = new Date(call.dateCreated).toLocaleString();
// const messageStatus = call.status
// var twil_response_array = [twilid, messageStatus, d]
// logger.info(twil_response_array)
// wss.on('connection', ws => {
// ws.on('message', function () {
// })
// wss.clients
// .forEach(client => {
// client.send(twil_response_array)
// })
// })
// })
// }
// //})
app.listen(8180, function () {
logger.info('Server is running..');
});
});

socket.io code organizing and timing ES6

i have an experimental app that is pretty simple, you click a display button and an image appears in real time to all clients, utilizing socket.io. now it does work for the most part. however i had an issue (my original question here: socket.io emit on connect) with an event not happening when the client connected, but i have resolved it with the help of this thread: socket.emit on sever side is ignored after connection?.
basically, my 'new-client-append event' retrieves data (html in the form of a string) so that when a new client connects, it shows the same data that current clients see (similar to connecting to a chat room and being able to see all chat history). i had to reorganize my code so my 'new-client-append' event would take place, and i can get it to work if i put data in manually. my new issue is now that i have had to reorganize my code, my 'new-client-append' event is dependent on a variable i set within the class, so it is no longer recognizable. i'm relatively new to JS, how can i get my variable to be recognized and why is this happening? i had tried moving the event in different places of my code with no luck. it is the this.mainContainer variable.
CLIENT
import $ from 'jquery';
import SaveInput from './SaveInput';
import io from 'socket.io-client';
// make connection
const socket = io.connect('localhost:3000');
**socket.on('new-client-append', (data) => {
console.log('NEW CLIENT ENTERED');
console.log('on new-client-clone ' + JSON.stringify(data));
this.mainContainer.append(data);
});**
socket.on('connect_error', function(){
console.log('fail');
});
class Display extends SaveInput {
constructor(){
this.mainContainer = $('.main-container');
this.pGrid = $('.pic-grid-container');
this.display = $('#btn-display');
this.buttons();
}
buttons (){
// click buttons
this.display.click(this.displayEls.bind(this));
//display images
displayEls() {
let img = 'https://secure.gravatar.com/avatar/22f38e0216f57af53a1776fb2a72c436?s=60&d=wavatar&r=g';
let $picContainer = $('<div class="picture-frame"></div>');
let $newImg = $('<img>');
// clone pic-grid-container
let htmlClone = this.pGrid.clone();
let stringClone = htmlClone.html();
// EMIT
//send image url
socket.emit('client-image', {
image: img
});
// send dom clone to server
socket.emit('new-client-append', {
clone: stringClone
});
// LISTEN
// append image in real time
socket.on('client-image', (data) => {
let foo = data.image.toString();
$newImg.attr('src', foo);
// console.log(data);
// console.log(foo);
$newImg.appendTo($picContainer);
this.pGrid.append($picContainer);
// console.log('html clone ' + JSON.stringify(htmlClone));
// console.log('string clone ' + stringClone);
});
}
export default Display;
SERVER
const express = require('express');
const socketIO = require('socket.io');
const http = require('http');
// app set up
const app = express();
const server = http.Server(app);
// const = new socket(server);
let port = process.env.PORT || 3000;
// static files
app.use(express.static('app'));
// socket setup & pass SERVER
const io = new socketIO(server);
let jqueryImage;
// on client connect
io.on('connection', (socket) => {
console.log('client has entered...');
socket.emit('new-client-append', jqueryImage);
// events
socket.on('client-image', function(data){
console.log('SERVER ' + data.image);
io.sockets.emit('client-image', data);
});
socket.on('new-client-append', function(data){
jqueryImage = data.clone;
console.log('jqueryImage ' + JSON.stringify(jqueryImage));
});
// errors
io.on('error', function (err) {
console.log(err);
});
io.on('connect_error', function(){
console.log('fail');
});
});
server.listen(port, () => {
console.log('server running....');
});
UPDATE
added suggested code, calling the maincontainer from the display object. but it is saying that it is not defined.
socket.on('new-client-append', (data) => {
console.log('NEW CLIENT ENTERED');
console.log('on new-client-clone ' + JSON.stringify(data));
**display.mainContainer.append(data);**
});
file where i create the objects
import SaveInput from './modules/SaveInput';
import Display from './modules/Display';
const saveInput = new SaveInput();
const display = new Display ();

Your socket.on function has
this.mainContainer.append(data);
This implies that you've attached a mainContainer property to the socket object. Because you've not done this, but rather declared it inside your Display (sub)class, it's attached to the Display object you created.. I can't seem to find it, but somewhere, wherever this script is required there is a
variable = new Display();
You must call this mainContainer object by it's name. Find that code that initiates the Display object and then use
theVarNameYouFound.mainContainer.append(thatThing);

Socket.io reconnection firing no matter what i do

Using socket.io 1.4.5.
It seems no matter what I do, I cannot prevent socket.io from firing the reconnection event or destroying the client socket when there are interruptions with the internet connection.
On the client side, I have:
reconnectionDelay: 99999999,
timeout: 99999999999,
reconnection: false,
And yet, if the internet disconnects, the socket will become undefined (after 20 seconds or so) and the reconnection event fires when the internet goes back on.
My ultimate goal is to use the same exact socket on the server and on the client (regardless of how long it's been since they've communicated) unless that socket is explicitly disconnected on the server. I cannot have the socket reconnecting at will, because I store data on the socket and use the socket.id extensively in my application. IF the socket and socket id were to suddenly change, the application breaks.

I think You've to pass some unique variable on handshake and keep data by that variable.
Regeneration of socket.id is normal behavior of socket.io.
In my practice I'm doing initial request to server to create slot variable and keep it slots collection in mongodb and after I'm creating connection by defining that slot variable as handshake.
or let's just simplify answer to Your question and use: https://www.npmjs.com/package/express-socket.io-session
also a little "hack" use namespaces logic: https://socket.io/docs/rooms-and-namespaces/#custom-namespaces
BONUS:
here is quick solution:
clientside:
function genUUID() {
var d = new Date().getTime();
var uuid = 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, function(c) {
var r = (d + Math.random()*16)%16 | 0;
d = Math.floor(d/16);
return (c=='x' ? r : (r&0x3|0x8)).toString(16);
});
return uuid;
};
var namespace = localStorage.getItem("namespace");
if(!namespace) {
namespace = genUUID();
localStorage.setItem("namespace", namespace);
}
var connect = function (ns) {
return io.connect(ns, {
query: 'ns='+ns // this is handshake variable `ns`
});
}
var socket = connect('/'+namespace);
backend:
const
url = require('url'),
sharedData = {};
io.sockets.on('connection', (socket) => {
const handshake = url.parse(socket.handshake.url, true);
const ns = handshake.query ? handshake.query.ns : 'default';
console.log('GOT CONNECTION TO NS: '+ns);
io.of(ns).on('connection', (socket) => {
if(!sharedData[ns]) sharedData[ns] = {};
// put socket code here
socket.on('some-event', (data) => {
sharedData[ns]['some-event'] = data.someData;
});
});
});

How to pass socket to cluster in node

I would like to get a multi-process node. Workers are listening clients connections. I need pass sockets to master process because master process emit message to clients. Workers also need socket to emit message to clients.
Socket is a circular object and can't pass to a master process.
My code:
const cluster = require('cluster');
const http = require('http');
var io = require('socket.io');
var users;
var clients = {};
if (cluster.isMaster) {
function messageHandler(msg) {
if (msg.usersarray) {
usersarray = msg.usersarray;
console.log(usersarray);
}else if(msg.socket){
clients[usersarray["manu"][0]] = msg.socket;
clients[usersarray["manu"][0]].emit("hola","hola");
}
}
// Start workers and listen for messages containing notifyRequest
const numCPUs = require('os').cpus().length;
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
Object.keys(cluster.workers).forEach((id) => {
cluster.workers[id].on('message', messageHandler);
});
}else {
// Create server & socket
var server = http.createServer(function(req, res){
// Send HTML headers and message
res.writeHead(404, {'Content-Type': 'text/html'});
res.end('<h1>Aw, snap! 404</h1>');
});
server.listen(3000);
io = io.listen(server);
// Add a connect listener
io.sockets.on('connection', function(socket) {
var hs = socket.handshake;
console.log("socket connected");
if(users == undefined){
users = {};
}
if(hs.query.usuario != undefined){
if(users[hs.query.usuario] == undefined){
users[hs.query.usuario] = new Array();
}
users[hs.query.usuario].push(socket.id); // connected user with its socket.id
clients[socket.id] = socket; // add the client data to the hash
process.send({ usersarray: users});
process.send({ socket: socket});
}
// Disconnect listener
socket.on('disconnect', function() {
console.log('Client disconnected.');
});
});
}
in line process.send({ socket: socket}); Node js get error "TypeError: Converting circular structure to JSON"
-I used some module to transform circular object but don't working.
-I tried to pass socket id and then in master process, created new socket with this id but I didn't know to use it.
There is any posibility to pass socket from worker to master process?
Node js version: v5.5.0

Hm, I don't think it is possible what you are trying to do. When you create a cluster it means that you create separate processes (master + workers) which can only talk over the pipe.
Talking over the pipe means they can only send strings to each other. process.send tries to serialize a Javascript object as JSON (--> making a string out of it) using JSON.stringify. JSON for example cannot have functions, circles, etc. I just checked the socket object, it is very complex and contains functions (such as socket.emit()), so you cannot just serialize it and send it over the pipe.
Maybe you can check this or this on how to use clustered WebSockets.
It doesn't seem very trivial.. Maybe you can just pass CPU intensive tasks to some worker processes (via cluster or just spawning them yourself), send the results back to the master and let him do all the communication with the client?

I understand your purpose of broadcasting to all the node worker processes in a cluster, although you can not send socket component as such but there is a workaround for the purpose to be served. I will try an explain with an example :
Step 1: When a client action requires a broadcast :
Child.js (Process that has been forked) :
socket.on("BROADCAST_TO_ALL_WORKERS", function (data)
{
process.send({cmd : 'BROADCAST_TO_ALL_WORKERS', message :data.message});
})
Step 2: On the cluster creation side
Server.js (Place where cluster forking happens):
if (cluster.isMaster) {
for (var i = 0; i < numCPUs; i++) {
var worker = cluster.fork();
worker.on('message', function (data) {
if (data.cmd === "BROADCAST_TO_ALL_WORKERS") {
console.log(server_debug_prefix() + "Server Broadcast To All, Message : " + data.message + " , Reload : " + data.reload + " Player Id : " + data.player_id);
Object.keys(cluster.workers).forEach(function(id) {
cluster.workers[id].send({cmd : "BROADCAST_TO_WORKER", message : data.message});
});
}
});
}
cluster.on('exit', function (worker, code, signal) {
var newWorker = cluster.fork();
newWorker.on('message', function (data) {
console.log(data);
if (data.cmd === "BROADCAST_TO_ALL_WORKERS") {
console.log(data.cmd,data);
Object.keys(cluster.workers).forEach(function(id) {
cluster.workers[id].send({cmd : "BROADCAST_TO_WORKER", message : data.message});
});
}
});
});
}
else {
//Node Js App Entry
require("./Child.js");
}
Step 3: To Broadcast in the child process
-> Put this before io.on("connection") in Child.js
process.on("message", function(data){
if(data.cmd === "BROADCAST_TO_WORKER"){
io.sockets.emit("SERVER_MESSAGE", { message: data.message, reload: data.reload, player_id : data.player_id });
}
});
I hope its clear. Please comment if its confusing ... I will try and make it clear.

really strange behaviour on node.js using connect-form, socket.io and express

the following code:
req.form.on('progress', function(bytesReceived, bytesExpected){
var percent = (bytesReceived / bytesExpected * 100) | 0;
// progressEvent.download(percent);
io.sockets.on('connection', function (socket) {
socket.emit('progress', { percent: percent});
client = socket;
});
});
written on an http post handler (express.js) sends socket messages to the client js, but it obviously creates a huge amount of listeners, in fact it warns me saying:
"node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit."
on the other hand this code:
io.sockets.on('connection', function (socket) {
progressEvent.on('progress', function(percentage) {
console.log(percentage);
socket.emit('progress', { percent: percentage});
});
});
Doesn't send any message back to the client, the ProgressEvent is:
var util = require('util'),
events = require('events');
function ProgressEvent() {
if(false === (this instanceof ProgressEvent)) {
return new ProgressEvent();
}
events.EventEmitter.call(this);
}
util.inherits(ProgressEvent, events.EventEmitter);
ProgressEvent.prototype.download = function(percentage) {
var self = this;
self.emit('progress', percentage);
}
exports.ProgressEvent = ProgressEvent;
I've been a good day on this strange problem I can't really see why socket.io doesn't send the socket message to the client.
the whole project is here: https://github.com/aterreno/superuploader
Thanks for your attention & help

You shouldn't listen to socket.io connections inside of the progress event. It looks like you're trying to get socket.io to connect when a user uploads a file, but that will not do that. Instead it will listen for a new connection each time the progress event fires on the upload, which I'm guessing is pretty often and it's why you're getting the warning about too many listeners.
What you want to do instead is on the client side, when you initialize an upload, tell the server through socket.io. Then the server links up that socket.io client with their upload through their session, http://www.danielbaulig.de/socket-ioexpress/
Something like this should do it
io.sockets.on('connection', function(socket) {
var session = socket.handshake.session;
socket.on('initUpload', function() {
session.socket = socket;
});
socket.on('disconnect', function() {
session.socket = null;
});
});
And then in your route
req.form.on('progress', function(bytesReceived, bytesExpected){
var percent = (bytesReceived / bytesExpected * 100) | 0;
if (req.session.socket) {
socket.emit('progress', percent);
}
});
This only works with one upload per session, but you get the idea.