Develop Reference

Express routes with Socket IO Node JS

I'm creating a microservice with NodeJS which will send & receive messages with socketIO. I'm following a boilerplate and setup the GET & POST routes with ExpressRouter
socket.on('sendmessage', (newMessage) => {
socket.in(userId).emit('newmessage', newMessage);
});
app.post('/sendMessage', expressAsyncHandler(async function (req, res) {
// saving the message to database and returning a response....
}))
I'm just confused about why do I need routes if I can handle all data saving, validation and authorization functionality too in socketIO code block? What is the best case for an application like that which ensures message delivery and no data loss?

Whether to use routes or not depends on the design and architecture of your application. If you need to implement additional functionality outside of sending and receiving messages, such as authentication or data validation, using routes with Express is a good option. If your application is solely focused on sending and receiving messages, then you can handle the functionality within the socket.on() event.
For ensuring message delivery and avoiding data loss, you can implement message acknowledgment or implement a message queue like RabbitMQ or Apache Kafka to buffer messages before they are processed by the server. This helps ensure that messages are not lost even if the server goes down or there is a connection issue.

How synchronise socketIO connection ID's on client and server?

I have a javascript GameClient that uses SocketIO to send messages to a nodeJs server. Multiple users can open the GameClient separately and send messages to the server.
GameClient
GameClient ---> NodeJS Server
GameClient
The server can send messages to specific clients using io.to(socketid).emit(). The code looks something like this:
CLIENT
this.socket = io({ timeout: 60000 })
this.socket.on('connect', () => Settings.getInstance().socketid = this.socket.id)
this.socket.on('reconnect', (attemptNumber:number) => console.log("reconnecting..."))
const json = JSON.Stringify({socketid:this.socket.id, name:"Old Billy Bob"})
this.socket.emit('user created', json)
SERVER (simplified for clarity, just keeping track of one user here)
user = {}
io.on('connection', (socket) => {
console.log('new connection')
socket.on('disconnect', () => {
console.log('user disconnected')
});
socket.on('user created', (json) => {
user = JSON.parse(json)
});
});
// demo code, send a message to our user
io.to(user.socketid).emit("message to one user")
PROBLEM
When the client browser tab becomes inactive for any reason at all, the client disconnects and reconnects and gets a new socket connection ID. This actually happens a lot in Chrome and Safari.
The server only knows the old connection id, so now it can't send direct messages any more. How do I keep the socket connection id synchronised on the client and server?
Since the server also gets a reconnected event, how does it know which user reconnected?

The answer to your question is quite simple: you need a way to identify who is who. And that is not socket.id because this only identifies sockets, not users, as you've already noticed.
So you need some authentication mechanism. Once a user authenticates you can reuse his true id (whether it is simply a name or an integer in a database is irrelevant). And then on the server side you keep a collection of pairs (true_id, socket_id). And whenever a message comes to that user, you broadcast it to all matched socket.io objects.
Edit: So here's the flow:
Client authenticates with the server, the server sends him his own true_id, which the client stores somewhere. The client may also store some session_id or maybe some other mechanism that will allow him fast reauthentication in case of disconnection (note: do not store credentials, its a security issue).
The server keeps track of (true_id, socket_id) pairs in the form of a double way, mutlivalue map (it's an implementation detail what kind of data structure should be used here, maybe two {} objects is enough). If a connection dies then (true_id, socket_id) entry is removed. Note that for a given true_id there still may be some other socket_id alive. And so it doesn't mean that the user disconnected. It only means that this particular channel is dead.
Users don't care about socket_id, they only care about true_id. What you emit is {target_id: true_id, ...} instead of {target_id: socket_id, ...} on the client side, when you want to send a direct message.
When the server receives such message with true_id inside, it retrieves all (true_id, socket_id) pairs and passes the message to all of these sockets (note: maybe you don't even need socket_id, you can simply store socket objects here). Although this is a business logic: do you allow multiple connections per user? I would. There are many edge cases here (like for example a client thinks that he disconnected, but the server thinks he is still connected, etc) and making this 100% correct is unfortunately impossible (due to the nature of networking). But with a bit of effort it is possible to make it work 99% of the time.
If a connection dies then it is your client's responsibility to automatically reconnect and reauthenticate. New socket_id for old true_id is generated on the server side.
Let me emphasize this again: clients don't care about socket_id at all. Because that doesn't identify them. This only identifies a channel. And only the server cares about this information.

Getting data from a API on the client side without giving the key

Currently I am making a page that display's data gathered from an API. Most of the data is updated on the server side every 4 hours but some of it is updated whenever a client requests the index route. As a result, there is a delay in the index file being sent since the data needs to be updated. I want to gather the updated data after the page has been requested and sent so there is no delay. My first idea was to make the request on the client side which will handle updating the display after the data is gathered but from my knowledge, I don't know how to do that without giving them the API key. Should I approach the problem this way or is there a better way to do it? I'm using Express for the back-end, Axios is used to make the get requests, and EJS is the template engine.
Here is the code:
// This is called before the data is send in a for loop
data.gameData[i].player_count = await SteamModule.liveGetCurrentPlayers(data.gameData[i].appid);
res.render('index', {data: data});
// This is the function that is called
liveGetCurrentPlayers: async (id) => {
const res = await axios.get(`${base}/ISteamUserStats/GetNumberOfCurrentPlayers/v1/?key=${key}&appid=${id}`, {timeout: 1000}).catch(err => {
Logger.error("Error getting updated user data per request");
return 'Error';
});
if(res.data) {
return res.data.response.player_count;
} else {
return 'Error';
}
}

Here's a bit of a drawing to explain what I've said in comments.....
(the code you showed should constantly update - without other info I can't help with whatever the other issue was, but this is the overall idea....)
Where:
Client requests data from you (your server)
Your server sends html and css to show a 'frame' of the page (no data, just something for them to see and feel like something is happening...)
Your server requests data from the API server (all the various "20-ish" things you said you wanted to serve....)
As the data is updated (or you may have it already), you send the data to the client, updating their 'frame' page with current data.

You can maintain your keys on the server side and add restriction that those API's can only be accessed by your client side URL. So you would access the API and it will maintain your session and handle the authorized KEY part as well.
Anything and everything on the client side is accessible if it running in your browser.
You can add security measures on the server but not on client side for protecting your key.

Running a node.js file from website

I have recently started using the Twilio platform to send SMS to my users. I am perfectly able to run the node.js file from the node terminal with the command:
node twilio.js
Now, my goal is to be able to send those SMS, but from my website. For instance, when the user provides his phone number and presses the "Send sms" button. How can I achieve this? I have been looking this up for a while and I came across Express platform, ajax post requests, http server, etc. But, I can't figure out how to use them. I currently make many ajax requests (POST and GET) on my site, but I'm not able to make a request to a node file.
Thanks in advance,
Here is the twilio.js file:
// Twilio Credentials
var accountSid = 'ACCOUNT SID';
var authToken = 'ACCOUNT TOKEN';
//require the Twilio module and create a REST client
var client = require('twilio')(accountSid, authToken);
client.messages.create({
to: 'TO',
from: 'FROM',
body: 'Message sent from Twilio!',
}, function (err, message) {
console.log(message.sid);
});

Being able to run any arbitrary script on your server from a webpage would be a huge security risk - don't do that. I'm not sure where you're hosting your site, or what technology stack you're running your site on, but since you mentioned Express and Node -- if you're using Express I'd recommend that you setup a route that handles an ajax request. When someone presses 'Send SMS' you send an ajax request to that route, and in the handler that gets invoked you place the Twilio logic.

Here is a very simple way to setup an Express request that calls you node module:
twilio.js:
// Twilio Credentials
var accountSid = 'ACCOUNT SID';
var authToken = 'ACCOUNT TOKEN';
//require the Twilio module and create a REST client
var client = require('twilio')(accountSid, authToken);
function sendSms(callback) {
client.messages.create({
to: 'TO',
from: 'FROM',
body: 'Message sent from Twilio!',
}, callback);
}
// Export this function as a node module so that you can require it elsewhere
module.exports = sendSms;
Here is a good start for Express.
server.js:
var express = require('express');
var app = express();
// Requiring that function that you exported
var twilio = require('/path/to/twilio.js');
// Creating a controller for the get request: localhost:8081/send/sms
app.get('/send/sms', function (req, res) {
twilio(function(err, message) {
if (err) res.send(err);
res.send('Message sent: ' + message);
});
});
// Creating an HTTP server that listens on port 8081 (localhost:8081)
var server = app.listen(8081, function () {
var host = server.address().address;
var port = server.address().port;
console.log("Example app listening at http://%s:%s", host, port);
});
Then you can run node server.js, go to your browser and go to the url: localhost:8081/send/sms and your message will be sent :)

I'd make it so the client sends a HTTP POST request to the server, and then the server will send the message on behalf of the client.
Easiest way is to use express. I'm a bit unsure of how you're serving your website from a Node.js app without using express. Do you have a custom solution or only a non-connected from end, or something like heroku or something? In any case, you can create a route that processes posts with the following:
app.post("send_twilio_message_route", function(req,res){
// this receives the post request -- process here
});
^ Note that doesn't actually create the express app. See my link below and they give examples of some of the nitty gritty and syntax.
So the above would be on the server, in your Node.js app. From the front-end client code that runs in the browser, you need to create a post. The easiest way and most likely way to do it is through $.post in Jquery. if you are using Angular there's a slightly different syntax but it's the same idea. You call post, point it to a url, and put in the body data.
Make the body of the post request contain data such the message, phone numbers,
authentication token maybe.
See this to be able to get the body from a post request and some more implementation details of how to set it up:
How to retrieve POST query parameters?
Depending on the nature of what you're doing you might consider having the sms processing stuff run separate from the actual web service. I would create the sms unique stuff as its own module and have a function retrieve the router so that you can mount is onto the app and move it about later. This might be overkill if you're doing something small, but I'm basically encouraging you to at the start put thought into isolating your services of your website, else you will create a mess. That being said, if it's just a small thing and just for you it might not matter. Depends on your needs.
Important: I highly encourage you to think about the malicious user aka me. If you don't add any authentication in the post body (or you could include it in the url but I wouldn't do that although it's equivalent), a malicious client could totally be a jerk and expend all of your twilio resources. So once you get it basic up in running, before deploying it to anything that people will see it, I recommend you add authentication and rate limiting.

Viewing Node.js code

I'm new to web development. I'm writing an application with Node.js and want to use MySQL. I'm not sure how web technology works, but would people visiting my site be able to see my database credentials? According to the MySQL documentation (https://github.com/felixge/node-mysql) this is how to connect to a MySQL database.
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'localhost',
user : 'me',
password : 'secret'
});
connection.connect();
connection.query('SELECT 1 + 1 AS solution', function(err, rows, fields) {
if (err) throw err;
console.log('The solution is: ', rows[0].solution);
});
connection.end();

No, they won't be able to see your code unless you run this on the client, which you shouldn't be able to or should, at all.
The example above runs on the backend and the query and data get displayed in the console, so it will only be accessible from the server and no clients connecting to it should be able see it, also you are not using anything to display a website ( based on the example above ) so you should not worry about it. If you want to, check out express or the http library for node.js.
After a while you should get the hang of it. Good luck!

People won't see your database credentials unless you publish the file in plain .js rather than executing it in node.js.
However, as an extra security measure, please prevent your database from connections outside the server, such as setting up a firewall to prevent port 3306 from outside connections, and use only 'localhost' users in your mysql database.