Develop Reference

WebSocket needs browser refresh to update list

My project works as intended except that I have to refresh the browser every time my keyword list sends something to it to display. I assume it's my inexperience with Expressjs and not creating the route correctly within my websocket? Any help would be appreciated.
Browser
let socket = new WebSocket("ws://localhost:3000");
socket.addEventListener('open', function (event) {
console.log('Connected to WS server')
socket.send('Hello Server!');
});
socket.addEventListener('message', function (e) {
const keywordsList = JSON.parse(e.data);
console.log("Received: '" + e.data + "'");
document.getElementById("keywordsList").innerHTML = e.data;
});
socket.onclose = function(code, reason) {
console.log(code, reason, 'disconnected');
}
socket.onerror = error => {
console.error('failed to connect', error);
};
Server
const ws = require('ws');
const express = require('express');
const keywordsList = require('./app');
const app = express();
const port = 3000;
const wsServer = new ws.Server({ noServer: true });
wsServer.on('connection', function connection(socket) {
socket.send(JSON.stringify(keywordsList));
socket.on('message', message => console.log(message));
});
// `server` is a vanilla Node.js HTTP server, so use
// the same ws upgrade process described here:
// https://www.npmjs.com/package/ws#multiple-servers-sharing-a-single-https-server
const server = app.listen(3000);
server.on('upgrade', (request, socket, head) => {
wsServer.handleUpgrade(request, socket, head, socket => {
wsServer.emit('connection', socket, request);
});
});

In answer to "How to Send and/or Stream array data that is being continually updated to a client" as arrived at in comment.
A possible solution using WebSockets may be to
Create an interface on the server for array updates (if you haven't already) that isolates the array object from arbitrary outside modification and supports a callback when updates are made.
Determine the latency allowed for multiple updates to occur without being pushed. The latency should allow reasonable time for previous network traffic to complete without overloading bandwidth unnecessarily.
When an array update occurs, start a timer if not already running for the latency period .
On timer expiry JSON.stringify the array (to take a snapshot), clear the timer running status, and message the client with the JSON text.
A slightly more complicated method to avoid delaying all push operations would be to immediately push single updates unless they occur within a guard period after the most recent push operation. A timer could then push modifications made during the guard period at the end of the guard period.
Broadcasting
The WebSockets API does not directly support broadcasting the same data to multiple clients. Refer to Server Broadcast in ws documentation for an example of sending data to all connected clients using a forEach loop.
Client side listener
In the client-side message listener
document.getElementById("keywordsList").innerHTML = e.data;
would be better as
document.getElementById("keywordsList").textContent = keywordList;
to both present keywords after decoding from JSON and prevent them ever being treated as HTML.

So I finally figured out what I wanted to accomplish. It sounds straight forward after I learned enough and thought about how to structure the back end of my project.
If you have two websockets running and one needs information from the other, you cannot run them side by side. You need to have one encapsulate the other and then call the websocket INSIDE of the other websocket. This can easily cause problems down the road for other projects since now you have one websocket that won't fire until the other is run but for my project it makes perfect sense since it is locally run and needs all the parts working 100 percent in order to be effective. It took me a long time to understand how to structure the code as such.

Reconnect to Laravel Echo server after session disconnection

I am attempting to write an web application with a persistent echo connection to a laravel-echo-server instance, which needs to detect disconnections and attempt to reconnect gracefully. The scenario I am attempting to overcome now is a user's machine has gone to sleep / reawoke and their session key has been invalidated (echo server requires an active session in our app). Detecting this situation from an HTTP perspective is solved - I setup a regular keepAlive, and if that keepAlive detects a 400-level error, it reconnects and updates the session auth_token.
When my Laravel session dies, I cannot tell that has happened from an echo perspective. The best I've found is I can attach to the 'disconnect' event, but that only gets triggered if the server-side laravel-echo-server process dies, rather than the session is invalid:
this.echoConnection.connector.socket.on('connect', function() {
logger.log('info', `Echo server running`);
})
this.echoConnection.connector.socket.on('disconnect', function() {
logger.log('warn', `Echo server disconnected`);
});
On the laravel-echo-server side, I can tell that the connection is dead - it will show this error:
⚠ [7:03:30 PM] - 5TwHN2qUys5VEFP5AAAG could not be authenticated to private.1
I cannot figure out how to catch this failure event programmatically from the client. Is there a way to capture it? Again, I can tell the session is dead eventually because I poll the server regularly via a http keepAlive function, but I would definitely also like to tell directly from the echo connection if possible, as it polls at a much higher natural rate.
As a second (more important) question, if I detect that my session has died, what should I do to recycle the echo connection (after I have logged in again via HTTP and gotten a new auth_token)? Is there anything specific I should call / etc? I've had some success calling disconnect() then setting up the connection again from scratch, but I do see errors such as:
websocket.js:201 WebSocket is already in CLOSING or CLOSED state.
Here is my current (naive) reconnection code, which is my initial connection code with an attempt to disconnect first stapled onto it:
async attemptEchoReconnect() {
if (this.echoConnection !== null) {
this.echoConnection.disconnect();
this.echoConnection = null;
}
const thisConnectionParams = this.props.connections[this.connectionName];
const curThis = this;
this.echoConnection = new Echo({
broadcaster: 'socket.io',
host: thisConnectionParams.echoHost,
authEndpoint: 'api/broadcasting/auth',
auth: {
headers: {
Authorization: `Bearer ` + thisConnectionParams.authToken
}
}
});
this.echoConnection.connector.socket.on('connect', function() {
logger.log('info', `Echo server running`);
})
this.echoConnection.connector.socket.on('disconnect', function() {
logger.log('warn', `Echo server disconnected`);
});
this.echoConnection.join('everywhere')
.here(users => {
logger.log('info', `Rejoined presence channel`);
});
this.echoConnection.private(`private.${this.props.id}`)
.listen(...);
setTimeout(() => { this.keepAlive() }, 120 * 1000);
}
Any help would be so great - these APIs are not well documented to the end that I really want, and I am hoping I can get some stability with this connection rather than having to do something ugly like force restart.

For anyone who needs help with this problem, my above echo reconnection code seems to be pretty stable, along with a keepAlive function to determine the state of the HTTP connection. I am still a bit uncertain of the origin of the console errors I am seeing, but I suspect they have to do with connection loss during a sleep cycle, which is not something I am particularly worried about.
I'd still be interested in hearing other thoughts if anyone has any. I am somewhat inclined to believe long-term stability of an echo connection is possible, though it does appear you have to proactively monitor it with what tools you have available.

How to connect to a TCP server and pass a Javascript to it

I'm definitely a newbie with JS and node. I have telescope management software called SkyX Pro, and it has the ability to run a TCP Server on port 3040. I can connect to it using Netcat and hand it a Javascript starting with //* Javascript *// this works and allows me to startup cameras and other equipment and send commands for taking pictures etc. The issue is it needs to be run from a batch file which makes getting any information back to an HTML page tough (Like Camera, focuser and filter wheel status and temperatures).
The NC call looks like "NC localhost 3040 < Javascript-file.js
To get around the browser to local machine security issues I want to run this from node.js with maybe socket.io-client if possible, but I don't know the proper syntax for it.
I have seen plenty of client syntax sending hello's etc. but nothing send javascript and allowing for two-way connectivity that I can understand.
I have tried using:
var socket = io.connect('http://localhost');`enter code here`
socket.on('httpServer', function (data) {
console.log(data);
document.write(data + "\r\n");
socket.emit('tcp', "For TCP");
});
const net = require('net');
const client = new net.Socket();
client.connect({ port: 3040, host: process.argv[2] });
client.on('data', (data) => {
console.log(data.toString('utf-8'));
But I do not understand it well enough to troubleshoot why it's not working.
Any help would be wonderful, and please treat me like a baby that needs its step by step.
Cheer
Peter

Reading [1], We can assume socket-io isn't the perfect fit for you, because that Server you have sound like a typical tcp-socket server, not a socket.io server ( which requires special headers ) or a web-socket server.
So you only needs "net" library to do the job.
const net = require('net');
// module to send a message to TCP-socket server and wait for the response from socket-server
const sendAndReceive = async (client, message) => {
client.write(message);
let response = null
await ( new Promise( (resolve, reject) => {
client.on('data', function(data) {
response = data;
resolve()
});
}))
return response;
}
// send a single message to the socket-server and print the response
const sendJSCode = (message) => {
// create socket-client
const client = new net.Socket();
client.connect(3040, 'localhost', async function() {
console.log('Connected');
// send message and receive response
const response = await sendAndReceive(client, message)
// parse and print repsonse string
const stringifiedResponse = Buffer.from(response).toString()
console.log('from server: ', stringifiedResponse)
// clean up connection
client.destroy()
});
}
sendJSCode('var Out; \n Out="TheSky Build=" + Application.build \n\r')
This script will:
Initiate a socket client
on connection successfully, client sends a message
client receives back response from that message
client prints response to terminal
Note that TheSkyX has a limitation of 4096 bytes for each message[2], any more than that and we will need to chunk the message. So you may want to keep the js-code short and precise.
that snippet I gave is minimal, it doesn't handle errors from server. If you want, you can add client.on("error", .. ) to handle it.
Your point of connecting to the socket server directly from browser is very intriguing, unfortunately it is not allowed by modern browsers natively due to security concerns 3
[1] https://socket.io/docs/#What-Socket-IO-is-not:~:text=That%20is%20why%20a%20WebSocket%20client,to%20a%20plain%20WebSocket%20server%20either.
[2] https://www.bisque.com/wp-content/scripttheskyx/scriptOverSocket.html#MSearchField:~:text=set%20to%204096%20bytes

Why does Node.js HTTP server not respond to a request from Python?

I've got a working HTTP node.js server.
I then created a program on python that uses the socket module to connect to the above server
Please for the time being do not mind the try and except statements. The code's connectTO() function simply connects to a server like any other code, with the exception that it handles some errors. Then the program send the message "hello". Next in the while loop it repeatedly waits for an answer and when it receives one, it prints it.
When I connect to the Node.js http server from python, I do get the message:
"You have just succesfully connected to the node.js server"
Which if you look at my code means that the s.connect(()) command was successful. My problem is that when a request is send to the server, it's supposed to output a message back, but it doesn't.
I also tried sending a message to the server, in which case the server sends back the following message:
HTTP/1.1 400 Bad Request
So why is the server not responding to the requests? Why is it rejecting them?
Python Client:
from socket import AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR
import threading, socket, time, sys
s = socket.socket(AF_INET,SOCK_STREAM)
def connectTO(host,port):
connect = False
count = 0
totalCount = 0
while connect!= True:
try:
s.connect((host,port))
connect = True
print("You have just succesfully connected to the node.js server")
except OSError:
count += 1
totalCount += 1
if totalCount == 40 and count == 4:
print("Error: 404. Connection failed repeatedly")
sys.exit(0)
elif count == 4:
print("Connection failed, retrying...")
count = 0
else:
pass
connectTO("IP_OF_NODE.jS_SERVER_GOES_HERE",777)
message = "hello"
s.send(message.encode("utf-8"))
while True:
try:
data, addr = s.recvfrom(1024)
if data == "":
pass
else:
print(data.decode())
except ConnectionResetError:
print("it seems like we can't reach the server anymore..")
print("This could be due to a change in your internet connection or the server.")
s.close()
Node.js HTTP server:
function onRequest(req, res) {
var postData = "";
var pathname = url.parse(req.url).pathname;
//Inform console of event recievent
console.log("Request for "+pathanme+" received.");
//Set the encoding to equivelant one used in html
req.setEncoding("utf8");
//add a listener for whenever info comes in and output full result
req.addListener("data", function(postDataChunk) {
postData += postDataChunk;
console.log("Received POST data chunk: '"+postDataChunk+"'");
});
req.addListener("end", function() {
route(handle, pathname, res, frontPage, postData);
});
};
http.createServer(onRequest).listen(port,ip);
console.log("Server has started.");
Some of my Research
I should also note that after some research, it seems that an HTTP server accepts HTTP requests, but I don't understand most of what's on Wikipedia. Is this the reason why the server is not responding? And how do I fix that while still using the socket module.
Also there are a lot of similar questions on Stack Overflow, but none help me solve my problem. One of them describes my issue, and the only answer is about "handshakes". Google is also pointless here, but from what I understand it is simply a reaction between the server and the client which defines what the protocol will be. Could this be what I'm missing, and how do I implement it?
Some of these questions also use modules that I'm not ready to use yet like websocket. Either that or they describe a way in which the server connects to the client, which can be done by directly calling python code or connecting to it from Node.js express. I want the client to be the one connecting to an HTTP server, by the means of the socket module in python. For the sake of future visitors who are looking for something like this, here are some of these question:
How to connect node.js app with python script?
Python Client to nodeJS Server with Socket.IO
Python connecting to an HTTP server
A blog that also does something similar to what is described above: https://www.sohamkamani.com/blog/2015/08/21/python-nodejs-comm/
Here is an answer that doesn't actually seem that obvious, but also solves the issue with only the relevant code. People who don't yet no much about servers in general will have probably missed it:
how to use socket fetch webpage use python

You will need to construct an HTTP request.
Example: GET / HTTP/1.1\n\n
Try this:
from socket import AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR
import threading, socket, time, sys
s = socket.socket(AF_INET,SOCK_STREAM)
def connectTO(host,port):
connect = False
count = 0
totalCount = 0
while connect!= True:
try:
s.connect((host,port))
connect = True
print("You have just succesfully connected to the node.js server")
except OSError:
count += 1
totalCount += 1
if totalCount == 40 and count == 4:
print("Error: 404. Connection failed repeatedly")
sys.exit(0)
elif count == 4:
print("Connection failed, retrying...")
count = 0
else:
pass
connectTO("IP_OF_NODE.jS_SERVER_GOES_HERE",777)
message = "GET / HTTP/1.1\n\n"
s.send(message.encode("utf-8"))
while True:
try:
data, addr = s.recvfrom(1024)
if data == "":
pass
else:
print(data.decode())
except ConnectionResetError:
print("it seems like we can't reach the server anymore..")
print("This could be due to a change in your internet connection or the server.")
s.close()
Read this to learn more about HTTP.
Now, I would recommend using this python lib to do what you're trying to do. It makes things much easier. However, if you are 100% set on using raw sockets, then you should make the node server use raw sockets as well. (Assuming you will only be connecting via python). Here is an excellent tutorial

Is it possible to create a "fake" socket connection to a nodejs server that is secured through SSL?

I'm using socket.io-client to create a socket connection to my locally-running server. See my code below:
// Working example of connecting to a local server that is not SSL protected
var io = require('socket.io-client')
var socket = io.connect('http://localhost:3000', {reconnect: true});
socket.on('connect', function(){ console.log("inside 'connect'") } );
socket.on('connection', function(){ console.log("inside 'connection'") } );
socket.on('event', function(data){ console.log("inside 'event'") } );
socket.on('disconnect', function(){ console.log("inside 'disconnect'") } );
var payload = {email: 'fake#gmail.com', password: 'tester'};
var tokens = {browserId: 'b965e554-b4d2-5d53-fd69-b2ca5483537a'};
socket.emit("publish", {logic:"user", method:"signIn"}, payload, tokens, function(err, creds) {
console.log("inside the socket client emit callback. err: " + err);
console.log("creds: " + creds);
});
Now for my problem. As I stated in the comment at the top of that code, I can connect to my local nodejs server and get the response I expect when I turn off SSL encryption on my server. As soon as I turn SSL on, I stop getting any response at all from the code above. I don't see any message in my server logs or from the command line, where I'm running the code above with node.
My goal is to be able to run the code above, with SSL turned on in my server, and get the same response that I get when SSL is turned off. I've tried a bunch of variations on the code I included above, such as:
connecting to "https://localhost:3000"
connecting to "//localhost:3000"
connecting to "https://localhost:3443" (this is the port I have to connect to when I have the nodejs server running with SSL)
changing {reconnect:true} to {reconnect:true,secure:true}
I'm truly stumped, and I've been doing a bunch of research on the web and on my node server. It's my company's code and I didn't originally implement the SSL components, so I've spent a few hours looking at our code and trying to understand how adding SSL changes everything. I'm also a student and have about 2 years of experience behind me, so I'm good but I'm no expert. Have I said anything above that indicates if my task is impossible to achieve, or if maybe I have just overlooked something? Any leads on things to check out would be appreciated :)