My project works as intended except that I have to refresh the browser every time my keyword list sends something to it to display. I assume it's my inexperience with Expressjs and not creating the route correctly within my websocket? Any help would be appreciated.
Browser
let socket = new WebSocket("ws://localhost:3000");
socket.addEventListener('open', function (event) {
console.log('Connected to WS server')
socket.send('Hello Server!');
});
socket.addEventListener('message', function (e) {
const keywordsList = JSON.parse(e.data);
console.log("Received: '" + e.data + "'");
document.getElementById("keywordsList").innerHTML = e.data;
});
socket.onclose = function(code, reason) {
console.log(code, reason, 'disconnected');
}
socket.onerror = error => {
console.error('failed to connect', error);
};
Server
const ws = require('ws');
const express = require('express');
const keywordsList = require('./app');
const app = express();
const port = 3000;
const wsServer = new ws.Server({ noServer: true });
wsServer.on('connection', function connection(socket) {
socket.send(JSON.stringify(keywordsList));
socket.on('message', message => console.log(message));
});
// `server` is a vanilla Node.js HTTP server, so use
// the same ws upgrade process described here:
// https://www.npmjs.com/package/ws#multiple-servers-sharing-a-single-https-server
const server = app.listen(3000);
server.on('upgrade', (request, socket, head) => {
wsServer.handleUpgrade(request, socket, head, socket => {
wsServer.emit('connection', socket, request);
});
});
In answer to "How to Send and/or Stream array data that is being continually updated to a client" as arrived at in comment.
A possible solution using WebSockets may be to
Create an interface on the server for array updates (if you haven't already) that isolates the array object from arbitrary outside modification and supports a callback when updates are made.
Determine the latency allowed for multiple updates to occur without being pushed. The latency should allow reasonable time for previous network traffic to complete without overloading bandwidth unnecessarily.
When an array update occurs, start a timer if not already running for the latency period .
On timer expiry JSON.stringify the array (to take a snapshot), clear the timer running status, and message the client with the JSON text.
A slightly more complicated method to avoid delaying all push operations would be to immediately push single updates unless they occur within a guard period after the most recent push operation. A timer could then push modifications made during the guard period at the end of the guard period.
Broadcasting
The WebSockets API does not directly support broadcasting the same data to multiple clients. Refer to Server Broadcast in ws documentation for an example of sending data to all connected clients using a forEach loop.
Client side listener
In the client-side message listener
document.getElementById("keywordsList").innerHTML = e.data;
would be better as
document.getElementById("keywordsList").textContent = keywordList;
to both present keywords after decoding from JSON and prevent them ever being treated as HTML.
So I finally figured out what I wanted to accomplish. It sounds straight forward after I learned enough and thought about how to structure the back end of my project.
If you have two websockets running and one needs information from the other, you cannot run them side by side. You need to have one encapsulate the other and then call the websocket INSIDE of the other websocket. This can easily cause problems down the road for other projects since now you have one websocket that won't fire until the other is run but for my project it makes perfect sense since it is locally run and needs all the parts working 100 percent in order to be effective. It took me a long time to understand how to structure the code as such.
Related
I'm definitely a newbie with JS and node. I have telescope management software called SkyX Pro, and it has the ability to run a TCP Server on port 3040. I can connect to it using Netcat and hand it a Javascript starting with //* Javascript *// this works and allows me to startup cameras and other equipment and send commands for taking pictures etc. The issue is it needs to be run from a batch file which makes getting any information back to an HTML page tough (Like Camera, focuser and filter wheel status and temperatures).
The NC call looks like "NC localhost 3040 < Javascript-file.js
To get around the browser to local machine security issues I want to run this from node.js with maybe socket.io-client if possible, but I don't know the proper syntax for it.
I have seen plenty of client syntax sending hello's etc. but nothing send javascript and allowing for two-way connectivity that I can understand.
I have tried using:
var socket = io.connect('http://localhost');`enter code here`
socket.on('httpServer', function (data) {
console.log(data);
document.write(data + "\r\n");
socket.emit('tcp', "For TCP");
});
const net = require('net');
const client = new net.Socket();
client.connect({ port: 3040, host: process.argv[2] });
client.on('data', (data) => {
console.log(data.toString('utf-8'));
But I do not understand it well enough to troubleshoot why it's not working.
Any help would be wonderful, and please treat me like a baby that needs its step by step.
Cheer
Peter
Reading [1], We can assume socket-io isn't the perfect fit for you, because that Server you have sound like a typical tcp-socket server, not a socket.io server ( which requires special headers ) or a web-socket server.
So you only needs "net" library to do the job.
const net = require('net');
// module to send a message to TCP-socket server and wait for the response from socket-server
const sendAndReceive = async (client, message) => {
client.write(message);
let response = null
await ( new Promise( (resolve, reject) => {
client.on('data', function(data) {
response = data;
resolve()
});
}))
return response;
}
// send a single message to the socket-server and print the response
const sendJSCode = (message) => {
// create socket-client
const client = new net.Socket();
client.connect(3040, 'localhost', async function() {
console.log('Connected');
// send message and receive response
const response = await sendAndReceive(client, message)
// parse and print repsonse string
const stringifiedResponse = Buffer.from(response).toString()
console.log('from server: ', stringifiedResponse)
// clean up connection
client.destroy()
});
}
sendJSCode('var Out; \n Out="TheSky Build=" + Application.build \n\r')
This script will:
Initiate a socket client
on connection successfully, client sends a message
client receives back response from that message
client prints response to terminal
Note that TheSkyX has a limitation of 4096 bytes for each message[2], any more than that and we will need to chunk the message. So you may want to keep the js-code short and precise.
that snippet I gave is minimal, it doesn't handle errors from server. If you want, you can add client.on("error", .. ) to handle it.
Your point of connecting to the socket server directly from browser is very intriguing, unfortunately it is not allowed by modern browsers natively due to security concerns 3
[1] https://socket.io/docs/#What-Socket-IO-is-not:~:text=That%20is%20why%20a%20WebSocket%20client,to%20a%20plain%20WebSocket%20server%20either.
[2] https://www.bisque.com/wp-content/scripttheskyx/scriptOverSocket.html#MSearchField:~:text=set%20to%204096%20bytes
I implemented a simple chat for my website where users can talk to each other with ExpressJS and Socket.io. I added a simple protection from a ddos attack that can be caused by one person spamming the window like this:
if (RedisClient.get(user).lastMessageDate > currentTime - 1 second) {
return error("Only one message per second is allowed")
} else {
io.emit('message', ...)
RedisClient.set(user).lastMessageDate = new Date()
}
I am testing this with this code:
setInterval(function() {
$('input').val('message ' + Math.random());
$('form').submit();
}, 1);
It works correctly when Node server is always up.
However, things get extremely weird if I turn off the Node server, then run the code above, and start Node server again in a few seconds. Then suddenly, hundreds of messages are inserted into the window and the browser crashes. I assume it is because when Node server is down, socket.io is saving all the client emits, and once it detects Node server is online again, it pushes all of those messages at once asynchronously.
How can I protect against this? And what is exactly happening here?
edit: If I use Node in-memory instead of Redis, this doesn't happen. I am guessing cause servers gets flooded with READs and many READs happen before RedisClient.set(user).lastMessageDate = new Date() finishes. I guess what I need is atomic READ / SET? I am using this module: https://github.com/NodeRedis/node_redis for connecting to Redis from Node.
You are correct that this happens due to queueing up of messages on client and flooding on server.
When the server receives messages, it receives messages all at once, and all of these messages are not synchronous. So, each of the socket.on("message:... events are executed separately, i.e. one socket.on("message... is not related to another and executed separately.
Even if your Redis-Server has a latency of a few ms, these messages are all received at once and everything always goes to the else condition.
You have the following few options.
Use a rate limiter library like this library. This is easy to configure and has multiple configuration options.
If you want to do everything yourself, use a queue on server. This will take up memory on your server, but you'll achieve what you want. Instead of writing every message to server, it is put into a queue. A new queue is created for every new client and delete this queue when processing the last item in queue.
(update) Use multi + watch to create lock so that all other commands except the current one will fail.
the pseudo-code will be something like this.
let queue = {};
let queueHandler = user => {
while(queue.user.length > 0){
// your redis push logic here
}
delete queue.user
}
let pushToQueue = (messageObject) => {
let user = messageObject.user;
if(queue.messageObject.user){
queue.user = [messageObject];
} else {
queue.user.push(messageObject);
}
queueHandler(user);
}
socket.on("message", pushToQueue(message));
UPDATE
Redis supports locking with WATCH which is used with multi. Using this, you can lock a key, and any other commands that try to access that key in thet time fail.
from the redis client README
Using multi you can make sure your modifications run as a transaction,
but you can't be sure you got there first. What if another client
modified a key while you were working with it's data?
To solve this, Redis supports the WATCH command, which is meant to be
used with MULTI:
var redis = require("redis"),
client = redis.createClient({ ... });
client.watch("foo", function( err ){
if(err) throw err;
client.get("foo", function(err, result) {
if(err) throw err;
// Process result
// Heavy and time consuming operation here
client.multi()
.set("foo", "some heavy computation")
.exec(function(err, results) {
/**
* If err is null, it means Redis successfully attempted
* the operation.
*/
if(err) throw err;
/**
* If results === null, it means that a concurrent client
* changed the key while we were processing it and thus
* the execution of the MULTI command was not performed.
*
* NOTICE: Failing an execution of MULTI is not considered
* an error. So you will have err === null and results === null
*/
});
}); });
Perhaps you could extend your client-side code, to prevent data being sent if the socket is disconnected? That way, you prevent the library from queuing messages while the socket is disconnected (ie the server is offline).
This could be achieved by checking to see if socket.connected is true:
// Only allow data to be sent to server when socket is connected
function sendToServer(socket, message, data) {
if(socket.connected) {
socket.send(message, data)
}
}
More information on this can be found at the docs https://socket.io/docs/client-api/#socket-connected
This approach will prevent the built in queuing behaviour in all scenarios where a socket is disconnected, which may not be desirable, however if should protect against the problem you are noting in your question.
Update
Alternatively, you could use a custom middleware on the server to achieve throttling behaviour via socket.io's server API:
/*
Server side code
*/
io.on("connection", function (socket) {
// Add custom throttle middleware to the socket when connected
socket.use(function (packet, next) {
var currentTime = Date.now();
// If socket has previous timestamp, check that enough time has
// lapsed since last message processed
if(socket.lastMessageTimestamp) {
var deltaTime = currentTime - socket.lastMessageTimestamp;
// If not enough time has lapsed, throw an error back to the
// client
if (deltaTime < 1000) {
next(new Error("Only one message per second is allowed"))
return
}
}
// Update the timestamp on the socket, and allow this message to
// be processed
socket.lastMessageTimestamp = currentTime
next()
});
});
I need to implement long polling for a chat application. I've searched around, but I only find how to implement it in JavaScript using JQuery. How can I implement it using only native JavaScript and node.js? Can you guide me to some relevant articles or materials?
Q: How to do long polling in native Javascript in nodeJS?
A: I guess first of all you need to understand how the long polling model works. If you haven't had any clue then the RFC-6202 specification is a good starting point.
It is about the client sending a request to the server and waits until a response is returned.
From the specification we know that first the client will have to issue a http request which has an infinite or at least a high timeout value. Then the server, which is your nodeJs application is expected to stash all incoming requests into a data structure, basically a holding area. Your application will essentially hold on all the response object until an event gets triggered, then you reply to the responses appropriately.
Consider this Pseudo code:
const express = require('express');
const app = express();
const bodyParser = require('body-parser');
var requestCounter = 0;
var responses = {
/* Keyed by room Id =*/
"room_abc" : [ /* array of responses */]
};
app.get('/', function (req, res) {
requestCounter += 1;
var room = /* assuming request is for room_abc */ "room_abc";
// Stash the response and reply later when an event comes through
responses[room].push(res);
// Every 3rd request, assume there is an event for the chat room, room_abc.
// Reply to all of the response object for room abc.
if (requestCounter % 3 === 0) {
responses["room_abc"].forEach((res) => {
res.send("room member 123 says: hi there!");
res.end();
});
}
});
app.use(bodyParser.text({ type: 'text/*' }));
app.use(bodyParser.json());
app.listen(9999, function () {
console.log('Example app listening on port 9999!')
})
It is relatively time consuming to write a working example here but the code above is a good example of how you can implement long polling in NodeJS.
If you have postman installed or curl you can do HTTP calls to http://localhost:9999/ using method GET. You should noticed that on the first two calls you won't get a response and it is when you fired the 3rd one then you'll receive a response for all previous and current calls.
The idea here is you stash the request's response object first and when an event comes through, assuming on every 3rd HTTP call, you then loop through all of the responses and reply to them. For your chat application's case, the event that triggers a response would probably be when someone fires off a message to a chat room.
Why I can't close the server by requesting localhost:13777/close in browser (it continues to accept new requests), but it will gracefully close on timeout 15000? Node version is 0.10.18. I fell into this problem, trying to use code example from docs on exceptions handling by domains (it was giving me 'Not running' error every time I secondly tried to request error page) and finally came to this code.
var server
server = require("http").createServer(function(req,res){
if(req.url == "/close")
{
console.log("Closing server (no timeout)")
setTimeout(function(){
console.log("I'm the timeout")
}, 5000);
server.close(function(){
console.log("Server closed (no timeout)")
})
res.end('closed');
}
else
{
res.end('ok');
}
});
server.listen(13777,function(){console.log("Server listening")});
setTimeout(function(){
console.log("Closing server (timeout 15000)")
server.close(function(){console.log("Server closed (timeout 15000)")})
}, 15000);
The server is still waiting on requests from the client. The client is utilizing HTTP keep-alive.
I think you will find that while the existing client can make new requests (as the connection is already established), other clients won't be able to.
Nodejs doesn't implement a complex service layer on top of http.Server. By calling server.close() you are instructing the server to no longer accept any "new" connections. When a HTTP Connection:keep-alive is issued the server will keep the socket open until the client terminates or the timeout is reached. Additional clients will not be able to issue requests
The timeout can be changed using server.setTimeout() https://nodejs.org/api/http.html#http_server_settimeout_msecs_callback
Remember if a client has created a connection before the close event that connection can continually be used.
It seems that a lot of people do not like this current functionality but this issue has been open for quite a while:
https://github.com/nodejs/node/issues/2642
As the other answers point out, connections may persist indefinitely and the call to server.close() will not truly terminate the server if any such connections exist.
We can write a simple wrapper function which attaches a destroy method to a given server that terminates all connections, and closes the server (thereby ensuring that the server ends nearly immediately!)
Given code like this:
let server = http.createServer((req, res) => {
// ...
});
later(() => server.close()); // Fails to reliably close the server!
We can define destroyableServer and use the following:
let destroyableServer = server => {
// Track all connections so that we can end them if we want to destroy `server`
let sockets = new Set();
server.on('connection', socket => {
sockets.add(socket);
socket.once('close', () => sockets.delete(socket)); // Stop tracking closed sockets
});
server.destroy = () => {
for (let socket of sockets) socket.destroy();
sockets.clear();
return new Promise((rsv, rjc) => server.close(err => err ? rjc(err) : rsv()));
};
return server;
};
let server = destroyableServer(http.createServer((req, res) => {
// ...
}));
later(() => server.destroy()); // Reliably closes the server almost immediately!
Note the overhead of entering every unique socket object into a Set
I'm running Socket.io multi-threaded with the native cluster functionality provided by Node.js v0.6.0 and later (with RedisStore).
For every new change in state, the server iterates over each connection and sends a message if appropriate. Note: this isn't "broadcasting" to all connections, it's comparing server data with data the client sent on connection to decide whether to send the server data to that particular client. Consider this code sample:
io.sockets.clients().forEach(function (socket) {
socket.get('subscription', function (err, message) {
if(message.someProperty === someServerData) {
socket.emit('position', someServerData);
}
});
This worked fine when there was only one process, but now, the client receives a message for each Node process (ie. if there are 8 Node process running, all clients receive the messages 8 times).
I understand why the issue arises, but I'm not sure of a fix. How can I assign a 1-to-1 relation from one process to only on client. Perhaps something using NODE_WORKER_ID of Cluster?
This previous SO question seems somewhat related, although I'm not sure it's helpful.
This seems like a pretty common request. Surely, I must be missing something?
So if I get this straight you need to emit custom events from the server. You can do that by creating your own custom EventEmitter and triggering events on that emitter, for example:
var io = require('socket.io').listen(80);
events = require('events'),
customEventEmitter = new events.EventEmitter();
io.sockets.on('connection', function (socket) {
// here you handle what happens on the 'positionUpdate' event
// which will be triggered by the server later on
eventEmitter.on('positionUpdate', function (data) {
// here you have a function that checks if a condition between
// the socket connected and your data set as a param is met
if (condition(data,socket)) {
// send a message to each connected socket
// if the condition is met
socket.emit('the new position is...');
}
});
});
// sometime in the future the server will emit one or more positionUpdate events
customEventEmitter.emit('positionUpdate', data);
Another solution would be to have those users join the 'AWE150', so only they will receive updates for 'AWE150', like so:
var io = require('socket.io').listen(80);
io.sockets.on('connection', function (socket) {
if (client_is_interested_in_AWE) { socket.join('AWE150'); }
io.sockets.in('AWE150').emit('new position here');
});
Resources:
http://spiritconsulting.com.ar/fedex/2010/11/events-with-jquery-nodejs-and-socket-io/