I need to understand the concept of handling websockets on multiple instances so that it could be shared across all the instances. For e.g I have three nodes running which is being connected by the load balancer. On receiving the data which needs to be emit on the specific socket. My initial idea was to create a hashmap or json objects to holds the websocket connections. But I realize that this couldn't be done in this way as it will be only specific to its particular instance. If I will receive the data on any of the instances then most of the data will not be going to emit on that websocket connection as it doesn't know on which instance the websocket is created. Is there a good way to handle websocket connections so that it could be shared on all instances. My ideas was to use redis or postgres sql database because they are shared among all the instances.
Also I have tried the postgres solution to store the websocket connection but when I save the connection it says
TypeError: Converting circular structure to JSON
Is there some good solution to handle websocket connections that could be shared among all instances. If database is a good solution how can I resolve
TypeError: Converting circular structure to JSON
I guess the thing you are looking for are Adapters, and it's being documented relatively well in the official socket.io documentation (/v4/adapter/).
When scaling to multiple Socket.IO servers, you will need to replace the default in-memory adapter by another implementation, so the events are properly routed to all clients.
You have several official adapters you can choose from :
Redis adapter
MongoDB adapter
Postgres adapter
Cluster adapter
Here is an example using the MongoDB Adapter :
npm install #socket.io/mongo-adapter mongodb
const { Server } = require("socket.io");
const { createAdapter } = require("#socket.io/mongo-adapter");
const { MongoClient } = require("mongodb");
// DATABASE CONNECTION
const DB = "mydb";
const COLLECTION = "socket.io-adapter-events";
const mongoClient = new MongoClient("mongodb://localhost:27017/?replicaSet=rs0", {
useUnifiedTopology: true,
});
const io = new Server();
const main = async () => {
await mongoClient.connect();
try {
await mongoClient.db(DB).createCollection(COLLECTION, {
capped: true,
size: 1e6
});
} catch (e) {
// collection already exists
}
const mongoCollection = mongoClient.db(DB).collection(COLLECTION);
// HERE COMES THE IMPORTANT PART :)
// CREATE MONGO DB ADAPTER AND USE IT
io.adapter(createAdapter(mongoCollection));
io.listen(3000);
}
main();
The adapter then will take care of the rest. Every packet that is sent to multiple clients (e.g. io.to("room1").emit() or socket.broadcast.emit()) will be
sent to all matching clients connected to the current server
inserted in a MongoDB capped collection, and received by the other Socket.IO servers of the cluster
Related
As i upgraded to version 4.x of socket.io on nodeJs i run into issues with certain things breaking.
Here is 2 issues i am facing one is to get a list of all rooms a socket is part of. When i call
socket.rooms after a client is connected i get a correct response that list all rooms in a set.
Based on what i read using io.sockets.sockets[536xdN11uf311xkFAAAD].rooms but that gets me the below error.
TypeError: Cannot read property 'rooms' of undefined
How can i get the rooms for a given socket in 4.x
Also how can i get the socket info for a given socket without calling io.sockets.sockets then doing a for each on it to find the socket i am looking for.
How can i get the rooms for a given socket in 4.x
socket.rooms is a Set object that lists the rooms a given socket is in. This is documented for socket.io v4.
If you have the socket.id instead of the socket object, you can do:
const sockets = await io.in(theSocketId).fetchSockets();
This will be an array of sockets, but if you passed in a socket id, there will be zero or one items in the array depending upon whether that socketid exists or not. This is in the doc here and is a fairly new way to do things.
So, putting these together:
async function getRooms(socketid) {
const sockets = await io.in(socketid).fetchSockets();
if (!sockets.length) {
throw new Error(`socketid ${socketid} is not found`);
}
return Array.from(sockets[0].rooms);
}
Note: you will have deal with promises when using this interface (likely because of the support for remote adapters via redis or other clustered adapters).
My main file app.js is connected to userDB
I want to add a second database postsDB
const mongoose = require("mongoose");
const app = express();
mongoose.set("useCreateIndex", true);
mongoose.set("useUnifiedTopology", true);
mongoose.connect("mongodb://localhost:27017/userDB", {useNewUrlParser: true});
If they are multiple databases in the same server, you can use useDb
const userDb = mongoose.connection.useDb('userDB');
const postsDb = mongoose.connection.useDb('postsDB');
Only do this if you have a good reason too though, the more standard approach would be to use multiple collections in the same database.
If the databases are in different servers, you'll need to use different connection pools entirely, and that is something mongoose does not support out-of-the-box. You can use the standard mongodb driver or split your app into different services, each with their own instance of mongoose, and own connection. This would be highly unusual within the same express application though.
So I figured the best way to learn is to try and fail over and over. I am building a webapp, at least trying to. I am curious how to go about using node to query my db. I am able to make a connection to the db with my single app.js file.
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'xxxxxxxx-us-east-1.rds.amazonaws.com',
port : '3306',
user : 'user',
password : 'password',
database : 'app'
});
connection.connect(function(err){
if(!err) {
console.log("Database is connected ... ");
} else {
console.log("Error connecting database ... ");
}
});
My problem, or lack of understand begins when I try to integrate this into my client-side js code. For instance say I wanted to trigger the db connection when a user uploads a photo.
var upload = s3.putObject({
Bucket: albumBucketName,
Key: photoKey,
Body: file,
ACL: "public-read",
});
var promise = upload.promise();
Can I include the app.js node file?
Sorry if this is a dumb question. I feel like I am missing some fundamental understanding of how to integrate the functionality of node with my current client side JS. Any help or further reading is appreciated--I am even curious about PHP solutions.
X
Server and client side code are separate. However you can create a Node module that harnesses the AWS and returns an appropriate response to the client after completed.
To do this, you need to create an endpoint that you post your data to from the client, then process with the same AWS modules only for Node. You also need to be able to access the connection instance from a different NodeJS module. This can be accomplished several ways. First, if the library that instantiates the connection tracks all of the connections, you should be able to require the library in a different module, then use the library's API to access one of the connections. Second, if you create only one instance of the connection and allow it to export, then you can import the module with that connection. Third, you can use something like a request/response pattern between the two modules, with the pattern instance declared globally.
I'm using node.js and mongoDB. Right now, for my test app, the connection to the db is in the main node file, but I guess this is a wrong practice.
What I want/need: a secure way (i.e. not storing password on files users can access) to connect to the db just when needed.
For example: I want several admin pages (users, groups, etc..). Each page should connect to the db, find some data, and display it. It also have a form for adding a document to the db and a delete option.
I thought maybe to create some kind of a connection function - send it what you want to do (add, update, find, delete), to where (collection name) and whatever it needs. But I can't just include this function, because then it'll reveal the password to the db. So what can I do?
Thanks!
I'm going to answer your question bit by bit.
Right now, for my test app, the connection to the db is in the main node file
This is fine, though you might want to put it in a separate file for easier reuse. NodeJS is a continuesly running process, so in theory you could serve all of your HTTP responses using the same connection to the database. In practice you'd want to create a connection pool, but the Mongodb driver for NodeJS already does this automatically.
Each page should connect to the db, find some data, and display it.
When you issue a query on the MongoDB driver, it will automatically use a connection from its internal connection pool, as long as you gave it the credentials when your application was starting up.
What I want/need: a secure way (i.e. not storing password on files users can access) to connect to the db just when needed.
I would advice to keep your application configuration (any variables that depend on the environment in which the app is running) in a separate file which you don't commit to your VCS. A module like node-config can help a great deal with that.
The code you will end up with, using node-config, is something like:
config/default.json:
{
"mongo": null
}
This is the default configuration file which you commit.
config/local.json:
{
"mongo": "mongo://user:pass#host:port/db"
}
The local.json should be ignored by your VCS. It contains secret sauce.
connection.js:
var config = require('config');
var MongoClient = require('mongodb').MongoClient;
var cache;
module.exports = function(callback){
if(cache){
return callback(cache);
}
MongoClient.connect(config.get('mongo'), function(err, db){
if(err){
console.error(err.stack);
process.exit(1);
}
cache = db;
callback(db);
});
}
An incomplete example of how you might handle reusing the database connection. Note how the configuration is gotten using config.get(*). An actual implementation should have more robust error handling and prevent multiple connections from being made. Using Promises would make all that a lot easier.
index.js:
var connect = require('./connection');
connect(function(db){
db.find({whatever: true})
});
Now you can just require your database file anywhere you want, and reuse the same database connection, which handles pooling for you and you don't have your passwords hard-coded anywhere.
I am just learning mongodb-native driver for nodejs.
I connect like this.
var mongo=require("mongodb")
var serv=mongo.Server("localhost", 27017)
var dbase=mongo.Db("MyDatabase", serv)
And that works. But if I try to create a new database connection using the same server I get an error.
var dbase2=mongo.Db("MyDatabase2", serv)
"Error: A Server or ReplSet instance cannot be shared across multiple Db instances"
But it works if a make a new server connection first.
var serv2=mongo.Server("localhost", 27017)
var dbase2=mongo.Db("MyDatabase2", serv2)
So my question is why there are 2 connection functions, one for Server and one for Db, when it seems like they must always be used together?
Why doesn't it go like this.
var dbase=mongo.Db("localhost", 27017, "MyDatabase")
I want to make my own function that does this, but I wonder if there is some other reason they are separate.
Thanks.
Here is a link to the solution on the mongo docs, for reference. (seems like the same solution the other poster mentioned)
http://mongodb.github.com/node-mongodb-native/markdown-docs/database.html#sharing-the-connections-over-multiple-dbs
The point of separating the connection to the mongo server, and then the DB is for cases like when you want to connect to a ReplSet server, or other custom params. This way, you have a separate process connecting to a mongodb server.
The database connection call is separate simply because of the case you have here: you dont simply want to connect to a mongo server and a single db, but multiple dbs. This separation of connecting to db and server allows this flexibility.
Another Solution: Use node-mongoskin
Mongoskin does what you want to... it allows connecting to server and db all in one command. Not a solution for mongo-native, but worth considering as an alternative library for your future projects.
var mongo = require('mongoskin');
var db = mongo.db('localhost:27017/testDB');
For what it's worth, you can do what you want to do by using Db#db(), which doesn't seem to appear in the official documentation but is listed in the source code of db.js as being a public API:
/**
* Create a new Db instance sharing the current socket connections.
*
* #param {String} dbName the name of the database we want to use.
* #return {Db} a db instance using the new database.
* #api public
*/
so you could do
var serv=mongo.Server("localhost", 27017);
var dbase=mongo.Db("MyDatabase", serv);
var dbase2=dbase.db("MyDatabase2");
Because these are two separate and distinct actions - you have to connect (or already have a connection) to DB server (computer) in order to query any of the databases on that particular server. You can create distinct database query connections for each of the databases that you will want to use, but at the same time you will be using the same connection to the server.
Most of the time you will NOT want to create a separate server connection for each of the databases (if there are many) because the server usually limits the number of connections.