Develop Reference

Need some clarification on nodejs concepts

I am starting to learn more about how this "web world" works and that's why I am taking the free code camp course. I already took front-end development and I really enjoyed it. Now I am on the back end part.
The back end is much more foggy for me. There are many things that I don't get so I would hope that someone could help me out.
First of all I learned about the get method. so I did:
var http = require('http');
and then made a get request:
http.get(url, function callBack(response){
response.setEncoding("utf8");
response.on("data", function(data){
console.log(data);
});
});
Question 1)
So apparently this code "gets" a response from a certain URL. but What response? I didn't even ask for anything in particular.
Moving on...
The second exercise asks us to listen to a TCP connection and create a server and then write the date and time of that connection. So here's the answer:
var server = net.createServer(function listener (socket){
socket.end(date);
});
server.listen(port);
Question 2)
Okay so I created a TCP server with net.createServer() and when the connection was successful I outputted the date. But where? What did actually happen when I put date inside of socket.end()?
Last but not least...
in the last exercise I was told to create an HTTP server (what?) to server a text file for every time it receives requests, and here's what I did:
var server = http.createServer(function callback(request, response){
var read = fs.createReadStream(location);
read.pipe(response);
});
server.listen(port);
Question 3)
a) Why did I have to create an HTTP server instead of a regular TCP? what's the difference?
b)what does createReadStream do?
c) What does pipe() do?
If someone could help me, trying to make the explanation easier would help me a lot since I am, as you can see, pretty dumb on this subject.
Thank you a lot!

This is a little broad for Stackoverflow which favors focused questions that address specific problems. But I feel your pain, so…
Questions 1:
Http.get is roughly equivalent to requesting a webpage. The url in the function is the page you are requesting. The response will include several things like the HTTP response code, but also (most importantly) the content of the page, which is what you are probably after. On the backend this is normally used for hitting APIs that get data rather than actual web pages, but the transport mechanism is the same.
Question 2:
When you open a socket, you are waiting for someone else to request a connection. (The way you do when you use http.get(). When you output data you are sending them a response like the one you received in question 1.
Question 3:
HTTP is a higher level protocol than TCP. This basically means it is more specific and TCP is more general (pedants will take issue with that statement, but it's an easy way to understand it). HTTP defines the things like GET and POST that you use when you download a webpage. Lower down in the protocol stack HTTP uses TCP. You could just use TCP, but you would have to do a lot more work to interpret the requests that come in. The HTTP library does that work for you. Other protocols like FTP also use TCP, but they are different protocol than HTTP.

For this answer, you need to understand two things. An IP address is the numeric value of a website, it's the address to the server pointing to the site. A domain name is a conversion from IP to a NAMED system which allows humans an easier way to see the names of websites, so instead of typing numbers for websites, like 192.168.1.1, we can now just type names (www.hotdog.com). That's what your get request is doing, it's requesting the site.
socket.end is a method you're calling. socket.end "Half-closes the socket. i.e., it sends a FIN packet. It is possible the server will still send some data" from the nodejs.org docs, so basically it half closes your socket at the parameter you're sending in, which is todays current date.
HTTP is hyper text transfer protocol, TCP (transmissioncontrol protocol) is a link between two computers
3a HTTP is for browsers, so that's why you did it, for a web page you were hosting locally or something.
3b createreadstream() Returns a new ReadStream object. (See Readable Stream).
Be aware that, unlike the default value set for highWaterMark on a readable stream (16 kb), the stream returned by this method has a default value of 64 kb for the same parameter.
3c pipe:
The 'pipe' event is emitted when the stream.pipe() method is called on a readable stream, adding this writable to its set of destinations.

Understanding how to use Redis with Node.js and Server Sent Events

My Project is built with Nodejs as proxy server to communicate with an external API.
The API send product updates via Redis (pub/sub); The Proxy server handle the message and send it to the client via SSE (Server Sent Events).
It is the first time for me using Redis and SSE and looking online for tutorials seems to be easy to implement and I did it.
On the Client side I just created an EventSource and as soon as I receive an update I do something with it:
// Client Side
var source = new EventSource('/redis'); // /redis is path to proxy server
source.addEventListener('items', handleItemsCallback, false);
source.addEventListener('users', handleUsersCallback, false);
source.addEventListener('customers', handleCustomersCallback, false);
// Function sample...
function handleItemsCallback (msg) {
// Do something with msg...
}
In the Proxy server I created a controller with routing to /redis to handle Redis messages:
exports.redisUpdates = function (req, res) {
// Redis Authentication
var redisURL = url.parse(process.env.REDISCLOUD_URL);
var client = redis.createClient(redisURL.port, redisURL.hostname, {ignore_subscribe_messages: false});
client.auth(redisURL.auth.split(":")[1]);
// let request last as long as possible
req.socket.setTimeout(0);
// Subscribe to channels
client.subscribe('items', 'users', 'customers');
// Handle messages
client.on('message', function (channel, message) {
res.write('retry: 5000\n');
res.write('event: ' + channel + '\n');
res.write('data: ' + message + '\n\n');
res.flush(); // If I do not add this it doesn't push updates to the client (?)
});
//send headers for event-stream connection
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive'
});
res.write('\n');
};
Using it locally in a development environment it works fine but using it in Production generate several different errors, the App is hosted on Heroku and the Heroku Metrics show several H18, H12, H27 Errors;
Sometimes the /redis call return status 503;
What I wish to understand is if I'm using those services correctly, why all tutorials do not mention res.flush() and I discovered it by myself to let it work the first time...

In all fairness, this question is not really answerable for a few reasons. I don't know which tutorials you are talking about since you didn't reference any in the question. I cannot speak on behalf of those who wrote the unreferenced tutorials. They could just be wrong, or maybe the architecture of what you are trying to accomplish differs in some small way. I also don't know what framework or optional middleware you are using in your project.
Now, with all of that said there are a few things I can share that may help you out.
Most tutorials you find out there are probably not going to open a connection and read from the stream indefinitely. When the process ends, the http response is closed with .end() or something similar. Since an HTTP response is a write stream, it follows the same rules as any other stream. You can find a lot of good info about streams here:
https://github.com/substack/stream-handbook
Something important to understand is that a stream can have a buffer and most http frameworks enable compression which causes buffers to be used. The code sample in the next link is a good example of what a framework would do for you behind the scenes (a minimal implementation of course)
https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/#what-we-ve-got-so-far
Since you want the output to continue being updated, you either have to wait until the output buffer size is reached or you have to call .flush().
If you ARE using express, check out this next Stack Overflow post related to compression middleware. I believe you'll have to have it disabled for your /redis route.
Node Express Content-Length
I hope that helped a little. Like I said, its kind of hard to answer this question. ;)

Correct way to handle Websocket

I've a client to server Websocket connection which should be there for 40 seconds or so. Ideally it should be forever open.
The client continually sends data to server and vice-versa.
Right now I'm using this sequence:
var socket;
function senddata(data)
{
if (!socket)
{
socket = new WebSocket(url);
socket.onopen = function (evt) {
socket.send(data);
socket.onmessage = function (evt) {
var obj = JSON.parse(evt.data);
port.postMessage(obj);
}
socket.oneerror = function (evt) {
socket.close();
socket = null;
}
socket.onclose = function(evt){
socket = null;
}
}
}
else
{
socket.send(data);
}
}
Clearly as per current logic, in case of error, the current request data may not be sent at all.
To be frank it sometimes gives error that websocket is still in connecting state. This connection breaks often due to networking issues. In short it does not work perfectly well.
I've read a better design : How to wait for a WebSocket's readyState to change but does not cover all cases I need to handle.
Also I've Googled about this but could not get the correct procedure for this.
So what is the right way to send regular data through Websockets which handles well these issues like connection break etc?

An event you don't seem to cover is onclose. Which should work really well, since it's called whenever the connection terminates. This is more reliable than onerror, because not all connection disruptions result in an error.

I personally use Socket.IO, it enables real-time bidirectional event-based communication between client and server.
It is event driven. Events such as
on connection :: socket.on('conection',callback);
and
on disconnect :: socket.on('disconnect',callback);
are built in with socket.io so it can help you with your connection concerns. Pretty much very easy to use, check out their site if you are interested.

I use two-layer scheme on client: abstract-wrapper + websocket-client:
The responsibilities of the websocket-client are interacting with a server, recovering the connection and providing interfaces (event-emitter and some methods) to abstract-wrapper.
The abstract-wrapper is a high-level layer, which interacts with websocket-client, subscribes to its events and aggregating data, when the connection is temporary failed. The abstract-wrapper can provide to application layer any interface such as Promise, EventEmitter and so on.
On application layer, I just work with abstract-wrapper and don't worry about connection or data losing. Undoubtedly, it's a good idea to have here information about the status of connection and data sending confirmation, because it's useful.
If it is necessary, I can provide some code for example

This apparently is a server issue not a problem in the client.
I don't know how the server looks like here. But this was a huge problem for me in the past when I was working on a websocket based project. The connection would continuously break.
So I created a websocket server in java, and that resolved my problem.
websockets depend on lots of settings, like if you're using servlets then servlet container's settings matter, if you're using some php etc, apache and php settings matter, for example if you create a websocket server in php and php has default time-out of 30 seconds, it will break after 30 seconds. If keep-alive is not set, the connection wont stay alive etc.
What you can do as quick solution is
keep sending pings to a server after a certain amount of time (like 2 or 3 seconds, so that if a websocket is disconnected it is known to the client so it could invoke onclose or ondisconnect, I hope you know that there is no way to find if a connection is broken other than failing to send something.
check server's keep-alive header
If you have access to server, then it's timeouts etc.
I think that would help

Nodejs + Socket.IO - how to get total bytes transferred (read/write) after socket close

I have a simple socket.io client and server program, running on node.js. The client and server exchange messages between them for a few minutes, before disconnecting (like chat).
If there any function/method I can use to get the total bytes transferred (read/write), after the socket is closed?
At present I am adding up the message size for each each message sent and received by the client. But, as per my understanding, in socket.io depending on which protocol is used (websocket, xhr-polling, etc.), the size of the final payload being sent will differ due to the header/wrapper size. Hence, just adding message bytes won't give me an accurate measure of bytes transferred.
I can use monitoring tools like Wireshark to get this value, but I would prefer using a javascript utility to get this value. Search online, didn't give me any reasonable answer.
For pure websocket connections, I am being able to get this value using the functions: socket._socket.bytesRead and socket._socket.bytesWritten
Any help is appreciated!

As of socket v2.2.0 i managed to get byte data like this. Only problem these are specified when client closes browser window and reason parameter is transport error. If client uses socket.close() or .disconnect() or server uses .disconnect() then bytes are 0.
socket.on('disconnect', (reason) => {
let symbs = Object.getOwnPropertySymbols(socket.conn.transport.socket._socket);
let bytesRead = socket.conn.transport.socket._socket[symbs[3]];
let bytesWritten = socket.conn.transport.socket._socket[symbs[4]];
});

If you wanted such a feature that would work no matter what the underlying transport was below a socket.io connection, then this would have to be a fundamental feature of socket.io because only it knows the details of what it's doing with each transport and protocol.
But, socket.io does not have this feature built in for the various transports that it could use. I would conclude that if you're going to use the socket.io interface to abstract out the specific protocol and implementation on top of that protocol, then you give up the ability to know exactly how many bytes socket.io chooses to use in order to implement the connection on its chosen transport.
There are likely debug APIs (probably only available to browser extensions, not standard web pages) that can give you access to some of the page-wide info you see in the Chrome debugger so that might be an option to investigate. See the info for chrome.devtools.network if you want more info.

Private messaging through node.js

I'm making a multiplayer (2 player) browser game in JavaScript. Every move a player makes will be sent to a server and validated before being transmitted to the opponent. Since WebSockets isn't ready for prime time yet, I'm looking at long polling as a method of transmitting the data and node.js looks quite interesting! I've gone through some example code (chat examples, standard long polling examples and suchlike) but all the examples I've seen seem to broadcast everything to every client, something I'm hoping to avoid. For general server messages this is fine but I want two players to be able to square off in a lobby or so and go into "private messaging" mode.
So I'm wondering if there's a way to implement private messaging between two clients using nodejs as a validating bridge? Something like this:
ClientA->nodejs: REQUEST
nodejs: VALIDATE REQUEST
nodejs->ClientA: VALID
nodejs->ClientB: VALID REQUEST FROM ClientA

You need some way to keep track of which clients are in a lobby together. You can do this with a simple global array like so process.lobby[1] = Array(ClientASocket, ClientBSocket) or something similar (possibly with some additional data, like nicknames and such), where the ClientXSocket is the socket object of each client that connects.
Now you can hook the lobby id (1 in this case) onto each client's socket object. A sort of session variable (without the hassle of session ids) if you will.
// i just made a hashtable to put all the data in,
// so that we don't clutter up the socket object too much.
socket.sessionData['lobby'] = 1;
What this allows you to do also, is add an event hook in the socket object, so that when the client disconnects, the socket can remove itself from the lobby array immediately, and message the remaining clients that this client has disconnected.
// see link in paragraph above for removeByValue
socket.on('close', function(err) {
process.lobby[socket.sessionData['lobby']].removeByValue(socket);
// then notify lobby that this client has disconnected.
});
I've used socket in place of the net.Stream or request.connection or whatever the thing is.
Remember in HTTP if you don't have keep-alive connections, this will make the TCP connection close, and so of course make the client unable to remain within a lobby. If you're using a plain TCP connection without HTTP on top (say within a Flash application or WebSockets), then you should be able to keep it open without having to worry about keep-alive. There are other ways to solve this problem than what I've shown here, but I hope I got you started at least. The key is keeping a persistent object for each client.
Disclaimer: I'm not a Node.js expert (I haven't even gotten around to installing it yet) but I have been reading up on it and I'm very familiar with browser js, so I'm hoping this is helpful somehow.