I have a nodejs project that spawns multiple processes that communicate with socket io (the process both send data and receieve it).
Sometimes during feature development, other programmers might do mistakes that will cause my socket infrastructure code to send large messages that are over size X (for example, over 500MB).
I am looking for a way to catch such cases and log them, I also wan't that such specific request specifically will fail.
Right now the behavior is that the entire socket connections fails (not just the specific big messages) and I didn't find anyway to even catch the case in order to know that this is the cause.
Right now I limit the message size with the "maxHttpBufferSize" configuration:
Socket.IO doc about it here - "how many bytes or characters a message can be, before closing the session (to avoid DoS)."
But I can't seem to catch any error when this size exceeds and the entire connection fails which is very bad for me.
I did think about trying to add some checks around my sending data through socket code, to check myself if the message size is to big, but serializing the data to make this check will have a very heavy price on performance, and socket.io code should already be doing this before sending the data so I don't want to make this happen twice.
The best solution i'm hoping to find is a socket.io supported way to handle this, if there is a different way that is performace costy that can also be good.
Note: I am working with socket io 2.0.4.
If there is a fix only in higher version that's also acceptable.
Related
This is the case.
I'm reverse-engineering a chat. The client runs in a browser, in Javascript.
Nothing is documented, but by checking the HTTP requests/responses (using Burp Suite), I've been able to figure out everything.
For simplicity, let's say that "everything" means loading the last messages and also sending a message.
However, the chat client detects "events". E.g., when a new message arrives. And I'm pretty sure the program is not polling any endpoint for this.
So, there should be an existing active connection receiving these events.
I am sure it's not WebSockets. So I need to look elsewhere, and checking the Javascript code (which is fragmented, obfuscated and minified) seems to be a HUGE task without knowing what to look for.
I've used Wireshark / mitmproxy for clues but could not get anything more useful that what I get with Burp.
What's the cheapest route to find out where is this program connecting to? I'm not much versed in Javascript so I don't know what are the ways of connecting to a given host and receiving events.
Any hints?
Thank you very much!
As weird as it may sound, the events were hidden behind a default filter.
They come as text/event-stream, which is strangely recognized as CSS by Burp.
I have a chat application with huge chatrooms (15'000 user connected to one single room).
Only few have the right to write, so theoretically there should not be a huge load.
I have noticed that there are performance issue: when only one message is send, the server CPU load spikes to 30%-50% and the message gets delivered slowly (maybe 1 second later or worse if you write multiple messages)
I have analysed the performance with clinic-flame. I see that this code is the problem:
socket.to("room1").emit(/* ... */); which will trigger send() in engine.io and clearBuffer in the ws library.
Does someone know if I am doing something wrong and how to optimize the performance?
You can load balance sokets.io servers.
I haven't do this yet, but you have informations about doing this here: https://socket.io/docs/v4/using-multiple-nodes
If you use node.js as http server : https://socket.io/docs/v4/using-multiple-nodes#using-nodejs-cluster
Alternatively, you can consider using redis for that. There is a package that allow to distribute trafic on a cluster of process or server:
https://github.com/socketio/socket.io-redis-adapter
You do not have to manage load balance if you have up to 15k to 20k Users, follow the below instructions.
In socket code do not use await async.
In socket code do not use any DB query.
Use a socket to take data from someone and provide those data to others, if you want to store those data do it using APIs before or after you send it to the socket.
It's all about how you use it. queries and await all this will slow your response and load, there is always a way to do things differently.
The above 3 suggestions boost my socket performance, I hope it does the same for you.
I wrote chat application using NodeJs and Socket.io. It works fine but at the moment there is nothing stopping malicious user from flooding server with large number of messages. What would be the best way to avoid this kind of situation?
BTW I did quite a bit of research and currently it seems that the only option would be simply tracking frequency of messages that user sends to server and if its over certain threshold disconnect socket or simply ignore messages. Is there a better way?
I just started learning Node.js and as I was learning about the fs.watchFile() method, I was wondering if a chat website could be efficiently built with it (and fs.writeFile()), against for example Socket.IO which is stable, but I believe not 100% stable (several fallbacks, including flash).
Using fs.watchFile could perhaps also be used to keep histories of the chats quite simply (as JSON would be used on the spot).
The chat files could be formatted in JSON in such a way that only the last chatter's message is brought up to the DOM (or whatever to make it efficient to 'fetch' messages when the file gets updated).
I haven't tried it yet as I still need to learn more about Node, and even more to be able to compare it with Socket.IO, but what's your opinion about it? Could it be an efficient/stable way of doing chats?
fs.watchFile() can be used to watch changes to the file in the local filesystem (on the server). This will not solve your need to update all clients chat messages in their browsers. You'll still need web sockets, AJAX or Flash for that (or socket.io, which handles all of those).
What you could typically do in the client is to try to use Web Sockets. If browser does not support them, try to use XMLHttpRequest. If that fails, fallback to Flash. It's a lot of programming to do, and it has to be handled by node.js server as well. Socket.io does that for you.
Also, socket.io is pretty stable. Fallback to Flash is not due to it's instability but due to lack of browser support for better solutions (like Web Sockets).
Storing chat files in flatfile JSON is not a good idea, because if you are going to manipulating the files, you would have to parse and serialize entire JSON objects, which would become very slow as the size of the JSON object increased. The watch methods for the filesystem module also don't work on all operating systems.
You also can't compare Node.js to Socket.IO because they are entirely different things. Socket.IO is a Node module for realtime transport between the browser and the server. What you need is dependent on what you're doing. If you need chat history, then you should be using a database such as MongoDB or MySQL. Watching files for changes is not an efficient way and you should just send messages as they received.
In conclusion no, using fs.watchFile() and fs.writeFile() is a very bad idea, because race conditions would occur due to concurrent file writes, besides that fs.watchFile() uses polling to check if a file has changed. You should instead use Socket.IO and push messages to other clients / store them in a database as they are received.
You can use long pooling method using javascript setTimeout and setInterval
long pooling
basically long pooling working on Ajax reqest and server responce time.
server will respond after a certain time (like after 50 seconds ) if there is not notification or message else it will respond with data and from client side when client gets response client javascript makes another request for new update and wait till response this process is endless until server is running
I am working on a real time tracking application that uses Node.js and Socket.IO. In my tracking code that goes on a target site I have some code that grabs the user agent string of the browser and sends it back to the server. This USUALLY works fine but there are a few times where this data is set to undefined (this is where it happens).
For now, I just have a huge try/catch block on the server so it doesn't crash when running a method I've defined to detect what browser it is (it crashes when it tries to run the match() method). I'm assuming this is happening either from bots or from some other browser that has no user agent or has been tampered with. Am I wrong on that? Are there other reasons?
Does Socket.IO provide anything for browser detection? Either way I know I need to make the browser detection feature more robust but I'm just getting this project off the ground.
If there's no better way to do this, am I better off just checking to see if the data that was sent to the server is undefined and consider it as an "Other" browser?
See the difference in total connections and total browser numbers? At the moment, there's a difference of a little over 100. If this browser tracking issue wasn't happening the numbers should be exactly the same (because EVERY connection would have a browser, resolution, operating system, and a URL).
I'm not very familiar with socket.io, but it looks like you can get the request headers in socket.handshake.headers (Socket.io's Wiki – Authorizing). I don't know which browsers run JavaScript but don't have navigator.userAgent set, but maybe they'll send the User-Agent HTTP Header?
This actually had nothing to do with bots or tampered data like I thought it did. My problem was due to a rare race condition where the client would connect to the server but disconnect before sending the data to the server (the "beacon" event I have set up is where the client sends the data and the server receives it). So when the client went to disconnect, my server would look up the client but would return a undefined result because the data was never sent and stored in the first place. A rewarding experience but what a pain!