Develop Reference

Serverless express close mongodb connexion

I am using serverless on aws with nodejs and mongodb atlas as database
At the moment I am using the trial version which allow maximum 500 connections.
Seems that my code is not disconnecting the database when process end
I am using express to manage it
First I had no connection close thinking that the connection will be closed automatically once the process end but no I had a lot of connections open.
Then I added a middleware to close my connections after the response has been sent, it was not working, I was thinking that serverless was stopping the process once the response was sent.
Not on each route I am closing mongo connection, for example
router.get('/website/:id/page', async (req, res, next) => {
try {
const pages = await pageDataProvider.findByWebsite(req.params.id);
await mongodbDataProvider.close();
res.json(pages);
} catch (error) {
next(error)
}
})
This is how I handle connections with mongo
const MongoClient = require('mongodb').MongoClient
const config = require('../config')
const MONGODB_URI = config.stage === 'test' ?
global.__MONGO_URI__ :
`mongodb+srv://${config.mongodb.username}:${config.mongodb.password}#${config.mongodb.host}/admin?retryWrites=true&w=majority`;
const client = new MongoClient(MONGODB_URI);
let cachedDb = null;
module.exports.connect = async () => {
if (cachedDb) return cachedDb;
await client.connect();
const dbName = config.stage === 'test' ? global.__MONGO_DB_NAME__ : config.stage;
const db = client.db(dbName)
cachedDb = db;
return db;
}
module.exports.close = async () => {
if (!cachedDb) return;
await client.close();
cachedDb = null;
}
I do not understand why I have so many connections open

Step 1
Isolate the call to the MongoClient.connect() function into its own module so that the connections can be reused across functions. Let's create a file mongo-client.js for that:
mongo-client.js:
const { MongoClient } = require('mongodb');
// Export a module-scoped MongoClient promise. By doing this in a separate
// module, the client can be shared across functions.
const client = new MongoClient(process.env.MONGODB_URI);
module.exports = client.connect();
Step 2
Import the new module and use it in function handlers to connect to database.
some-file.js:
const clientPromise = require('./mongodb-client');
// Handler
module.exports.handler = async function(event, context) {
// Get the MongoClient by calling await on the connection promise. Because
// this is a promise, it will only resolve once.
const client = await clientPromise;
// Use the connection to return the name of the connected database for example.
return client.db().databaseName;
}

I think its a programmatic error in your close method. Please have a closer look at
if (!cachedDb) return;
I think it should have been
if (cachedDb != null) return;
As stated in other response, I would strongly advice against closing the DB connections with each request. You should be looking for a pool mechanism, where a connection from the pool is handed to your application. The application can wait till it receives the connection
Closure of the DB connections should be handled at the time when the application is exiting (shutting/going down). This way application will at least try to close the connections gracefully.
Nonetheless, here is an adaptation your program
index.js
const express = require('express')
const app = express()
const port = 3000
const dbProvider = require('./dbProvider');
dbProvider.connect();
app.get('/testConnection',async (req, res, next) => {
console.log('Doing something for fetching the request & closing connection');
dbProvider.close();
console.log('After closing the connection');
})
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
})
dbProvider.js
let cachedDb = null;
let db = {};
module.exports.connect = async () => {
if (cachedDb) {
console.log('Returning Cachedb');
return cachedDb;
}
else{
console.log('Not a cachedDB');
}
db.setup = 1;
return db;
}
module.exports.close = async () => {
if (!cachedDb) {
console.log('Since its cached DB not closing the connection');
return;
}
db=null;
return;
}
And here is the console output:
-> node index.js
Not a cachedDB
Example app listening at http://localhost:3000
Doing something for fetching the request & closing connection
Since its cached DB not closing the connection
After closing the connection

According to this: https://docs.atlas.mongodb.com/best-practices-connecting-from-aws-lambda/
It's a good idea to add this line so you keep your connection pool between requests.
context.callbackWaitsForEmptyEventLoop = false;

IBM Watson WebSocket Connection failure. HTTP authentication failed; no valid credentials avaliable

I am working on a speech-to-text web app using the IBM Watson Speech to text API. The API is fetched on the click of a button. But whenever I click the button. I get the above-mentioned error. I Have stored my API key and URL in a .env file.
I tried a lot but keep on getting this error. Please Help me out as I am new to all this.
I got server.js from the Watson Github Repo
Server.js
'use strict';
/* eslint-env node, es6 */
const env = require('dotenv');
env.config();
const express = require('express');
const app = express();
const AuthorizationV1 = require('watson-developer-cloud/authorization/v1');
const SpeechToTextV1 = require('watson-developer-cloud/speech-to-text/v1');
const TextToSpeechV1 = require('watson-developer-cloud/text-to-speech/v1');
const vcapServices = require('vcap_services');
const cors = require('cors');
// allows environment properties to be set in a file named .env
// on bluemix, enable rate-limiting and force https
if (process.env.VCAP_SERVICES) {
// enable rate-limiting
const RateLimit = require('express-rate-limit');
app.enable('trust proxy'); // required to work properly behind Bluemix's reverse proxy
const limiter = new RateLimit({
windowMs: 15 * 60 * 1000, // 15 minutes
max: 100, // limit each IP to 100 requests per windowMs
delayMs: 0 // disable delaying - full speed until the max limit is reached
});
// apply to /api/*
app.use('/api/', limiter);
// force https - microphone access requires https in Chrome and possibly other browsers
// (*.mybluemix.net domains all have built-in https support)
const secure = require('express-secure-only');
app.use(secure());
}
app.use(express.static(__dirname + '/static'));
app.use(cors())
// token endpoints
// **Warning**: these endpoints should probably be guarded with additional authentication & authorization for production use
// speech to text token endpoint
var sttAuthService = new AuthorizationV1(
Object.assign(
{
iam_apikey: process.env.SPEECH_TO_TEXT_IAM_APIKEY, // if using an RC service
url: process.env.SPEECH_TO_TEXT_URL ? process.env.SPEECH_TO_TEXT_URL : SpeechToTextV1.URL
},
vcapServices.getCredentials('speech_to_text') // pulls credentials from environment in bluemix, otherwise returns {}
)
);
app.use('/api/speech-to-text/token', function(req, res) {
sttAuthService.getToken(function(err, token) {
if (err) {
console.log('Error retrieving token: ', err);
res.status(500).send('Error retrieving token');
return;
}
res.send(token);
});
});
const port = process.env.PORT || process.env.VCAP_APP_PORT || 3002;
app.listen(port, function() {
console.log('Example IBM Watson Speech JS SDK client app & token server live at http://localhost:%s/', port);
});
// Chrome requires https to access the user's microphone unless it's a localhost url so
// this sets up a basic server on port 3001 using an included self-signed certificate
// note: this is not suitable for production use
// however bluemix automatically adds https support at https://<myapp>.mybluemix.net
if (!process.env.VCAP_SERVICES) {
const fs = require('fs');
const https = require('https');
const HTTPS_PORT = 3001;
const options = {
key: fs.readFileSync(__dirname + '/keys/localhost.pem'),
cert: fs.readFileSync(__dirname + '/keys/localhost.cert')
};
https.createServer(options, app).listen(HTTPS_PORT, function() {
console.log('Secure server live at https://localhost:%s/', HTTPS_PORT);
});
}
App.js
import React, {Component} from 'react';
import 'tachyons';
//import WatsonSpeech from 'ibm-watson';
var recognizeMic = require('watson-speech/speech-to-text/recognize-microphone');
class App extends Component {
onListenClick = () => {
fetch('http://localhost:3002/api/speech-to-text/token')
.then(function(response) {
return response.text();
}).then(function (token) {
var stream = recognizeMic({
token: token, // use `access_token` as the parameter name if using an RC service
objectMode: true, // send objects instead of text
extractResults: true, // convert {results: [{alternatives:[...]}], result_index: 0} to {alternatives: [...], index: 0}
format: false // optional - performs basic formatting on the results such as capitals an periods
});
stream.on('data', function(data) {
console.log('error 1')
console.log(data);
});
stream.on('error', function(err) {
console.log('error 2')
console.log(err);
});
//document.querySelector('#stop').onclick = stream.stop.bind(stream);
}).catch(function(error) {
console.log('error 3')
console.log(error);
});
}
render() {
return(
<div>
<h2 className="tc"> Hello, and welcome to Watson Speech to text api</h2>
<button onClick={this.onListenClick}>Listen to Microphone</button>
</div>
);
}
}
export default App

Since the only code you show is fetching an authorisation token then I guess that that is what is throwing the authentication failure. I am not sure how old the code you are using is, but the mechanism you are using was used when the STT service credentials are userid / password. The mechanism became unreliable when IAM keys started to be used.
Your sample is still using watson-developer-cloud, but that has been superseded by ibm-watson. As migrating the code to ibm-watson will take a lot of rework, you can continue to use watson-developer-cloud.
If do you stick with watson-developer-cloud and you want to get hold of a token, with an IAM Key then use:
AuthIAMV1 = require('ibm-cloud-sdk-core/iam-token-manager/v1'),
...
tokenService = new AuthIAMV1.IamTokenManagerV1({iamApikey : apikey});
...
tokenService.getToken((err, res) => {
if (err) {
...
} else {
token = res;
...
}
});

ECONNRESET in Express.js (Node.js) with multiple requests

Given a standard Express.js setup
const express = require('express');
const app = express();
const router = express.Router();
router.get('/test/:id', (req, res) => {
return res.status(200).json({ hello: 'world' });
});
app.use('/api', router);
app.listen(3000, () => console.info('Up on port 3000));
I am making 1000 requests agains the endpoint, one after the other:
const fetch = require('node-fetch');
for (let i = 0; i < 1000; i++) {
let id = Math.floor(Math.random() * 12) + 1;
fetch(`http://localhost:3000/api/test/${id}`).then(res => res.json()).then(data => console.log(data)).catch(error => console.error(error));
}
I do see the data returned however, every now and then I see an ECONNRESET error. The amount of ECONNRESET error messages also vary: sometimes I get a few, sometimes a lot more. I do understand the message but I can't get my head around solving the issue behind it.
Here's a sample error:
{ FetchError: request to http://localhost:3000/api/test/8 failed, reason: connect ECONNRESET 127.0.0.1:3000
at ClientRequest.<anonymous> (node_modules/node-fetch/lib/index.js:1345:11)
at ClientRequest.emit (events.js:182:13)
at Socket.socketErrorListener (_http_client.js:399:9)
at Socket.emit (events.js:182:13)
at emitErrorNT (internal/streams/destroy.js:82:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
at process.internalTickCallback (internal/process/next_tick.js:72:19)
message:
'request to http://localhost:3000/api/departments/8 failed, reason: connect ECONNRESET 127.0.0.1:3000',
type: 'system',
errno: 'ECONNRESET',
code: 'ECONNRESET' }
Note that I have tried to make the request using axios, the built-in HTTP module all to avail. I'm sure the issue is with my Express app handling the request but not sure how to fix it exactly.
Update 1:
As per the suggestion in the comment, here's the async version:
async function f() {
const array = Array.from(Array(1000).keys());
for (const el of array) {
try {
let id = Math.floor(Math.random() * 12) + 1;
const result = await fetch(`http://localhost:3000/api/test/${id}`).then(res => res.json());
console.log(result);
return result;
} catch(e) {
console.log(e);
}
}
}
f();
Now I am receiving occasional ECONNREFUSED messages.
Update 2:
Based on Mazki516's answer here's the solution that works:
// previous require statements
const cluster = require('cluster');
const os = require('os');
if (cluster.isMaster) {
const cpuCount = os.cpus().length
for (let i = 0; i < cpuCount; i++) {
cluster.fork()
}
} else {
const app = express();
// rest of the route definitions
// also app.listen() etc...
}
cluster.on('exit', worker => {
console.log(`${worker.id} removed`);
cluster.fork();
});

One of the reasons you see this is because you make the calls in "parallel" .
You do start the calls one after the other , but the loops will end probably before the first results returned from the server.
The loop continues until the end , making the call stack filled with 1000 async requests to the server .
Your'e are hitting hardware/software limits and nothing is wrong with the code.
if you did want to build a server which can handle 1k (and much more) requests concurrently I would take a look into the "cluster" module of node .
Please notice that when doing network job between server , it's acceptable to use a concurrency limit . (for example: up to 4 requests concurrently)
but you can always scale your server beyond one machine and handle much more traffic .

Using node CRON job to call own request

I have the following snippet:
const express = require('express')
const app = express()
const cronJob = require('cron').CronJob
app.get('/test', (req, res) => {
// Do something here
}
new cronJob('* * * * * *', () => {
// Call localhost:3000/test here
}, null, true, 'Asia/Manila')
app.listen(3000, () => console.log('Successfully listened to app 3000'))
Usually on node, localhost:3000/test runs if this is called on the browser right? I wanted to make the CRON run this without typing it on the browser once the node app starts. If possible also regardless of the hostname, whether it's localhost or not, the CRON should make the request without being typed on the browser. Can this be done?

I read the comments above on the questions itself and decided to add my thoughts even thought it seems like you got a solution.
In my opinion it will be much more clean for you to call the "method" itself instead of hitting "http" for getting the response you need.
You have 2 options:
Hitting the "domain.com/test" endpoint with a request call.
Simply calling the same method the above url is doing.
In this way, you will "save" the overhead of need to "set-up" a new request to the express app with response and request headers. (example below)
let's say this is your code:
const handleTestData = () => return 'something';
app.get('/test', (req, res) => {
const result = handleTestData();
res.send(result);
}
new cronJob('* * * * * *', () => {
// instead of calling http and getting the response
// and adding overhead, just call the function
const result = handleTestData();
// do what you need with the result
}, null, true, 'Asia/Manila')

Emiting websocket message from routes

I'm trying to setup my server with websockets so that when I update something via my routes I can also emit a websocket message when something on that route is updated.
The idea is to save something to my Mongo db when someone hits the route /add-team-member for example then emit a message to everyone who is connected via websocket and is a part of whatever websocket room that corresponds with that team.
I've followed the documentation for socket.io to setup my app in the following way:
App.js
// there's a lot of code in here which sets what to use on my app but here's the important lines
const app = express();
const routes = require('./routes/index');
const sessionObj = {
secret: process.env.SECRET,
key: process.env.KEY,
resave: false,
saveUninitialized: false,
store: new MongoStore({ mongooseConnection: mongoose.connection }),
secret : 'test',
cookie:{_expires : Number(process.env.COOKIETIME)}, // time im ms
}
app.use(session(sessionObj));
app.use(passport.initialize());
app.use(passport.session());
module.exports = {app,sessionObj};
start.js
const mongoose = require('mongoose');
const passportSocketIo = require("passport.socketio");
const cookieParser = require('cookie-parser');
// import environmental variables from our variables.env file
require('dotenv').config({ path: 'variables.env' });
// Connect to our Database and handle an bad connections
mongoose.connect(process.env.DATABASE);
// import mongo db models
require('./models/user');
require('./models/team');
// Start our app!
const app = require('./app');
app.app.set('port', process.env.PORT || 7777);
const server = app.app.listen(app.app.get('port'), () => {
console.log(`Express running → PORT ${server.address().port}`);
});
const io = require('socket.io')(server);
io.set('authorization', passportSocketIo.authorize({
cookieParser: cookieParser,
key: app.sessionObj.key, // the name of the cookie where express/connect stores its session_id
secret: app.sessionObj.secret, // the session_secret to parse the cookie
store: app.sessionObj.store, // we NEED to use a sessionstore. no memorystore please
success: onAuthorizeSuccess, // *optional* callback on success - read more below
fail: onAuthorizeFail, // *optional* callback on fail/error - read more below
}));
function onAuthorizeSuccess(data, accept){}
function onAuthorizeFail(data, message, error, accept){}
io.on('connection', function(client) {
client.on('join', function(data) {
client.emit('messages',"server socket response!!");
});
client.on('getmessage', function(data) {
client.emit('messages',data);
});
});
My problem is that I have a lot of mongo DB save actions that are going on in my ./routes/index file and I would like to be able to emit message from my routes rather than from the end of start.js where socket.io is connected.
Is there any way that I could emit a websocket message from my ./routes/index file even though IO is setup further down the line in start.js?
for example something like this:
router.get('/add-team-member', (req, res) => {
// some io.emit action here
});
Maybe I need to move where i'm initializing the socket.io stuff but haven't been able to find any documentation on this or perhaps I can access socket.io from routes already somehow?
Thanks and appreciate the help, let me know if anything is unclear!

As mentioned above, io is in your global scope. If you do
router.get('/add-team-member', (req, res) => {
io.sockets.emit('AddTeamMember');
});
Then every client connected, if listening to that event AddTeamMember, will run it's associated .on function on their respective clients. This is probably the easiest solution, and unless you're expecting a huge wave of users without any plans of load balancing, this should be suitable for the time being.
Another alternative you can go:
socket.io lib has a rooms functionality where you can join and emit using the io object itself https://socket.io/docs/rooms-and-namespaces/ if you have a knack for this, it'd look something like this:
io.sockets.in('yourroom').broadcast('AddTeamMember');
This would essentially do the same thing as the top, only instead of broadcasting to every client, it'd only broadcast to those that are exclusive to that room. You'd have to basically figure out a way to get that users socket into the room //before// they made the get request, or in other words, make them exclusive. That way you can reduce the amount of load your server has to push out whenever that route request is made.
Lastly, if neither of the above options work for you, and you just absolutely have to send to that singular client when they initiate it, then it's going to get messy, because you have to have some sort of id to that person, and since you have no reference, you'd have to store all your sockets upon connection, and then make a comparison. I do not fully recommend something like this, because well, I haven't ever tested it, and don't know what type of repercussions could happen, but here is a jist of an idea I had:
app.set('trust proxy', true)
var SOCKETS = []
io.on('connection', function(client) {
SOCKETS.push(client);
client.on('join', function(data) {
client.emit('messages',"server socket response!!");
});
client.on('getmessage', function(data) {
client.emit('messages',data);
});
});
router.get('/add-team-member', (req, res) => {
for (let i=0; i< SOCKETS.length; i++){
if(SOCKETS[i].request.connection.remoteAddress == req.ip)
SOCKETS[i].emit('AddTeamMember');
}
});
Keep in mind, if you do go down this route, you're gonna need to maintain that array when users disconnect, and if you're doing session management, that's gonna get hairy really really quick.
Good luck, let us know your results.

Yes, it is possible, you just have to attach the instance of socket.io as long as you get a request on your server.
Looking to your file start.js you just have to replace your functions as:
// Start our app!
const app = require('./app');
app.app.set('port', process.env.PORT || 7777);
const io = require('socket.io')(app.app);
const server = app.app.listen(app.app.get('port'), () => {
server.on('request', function(request, response){
request.io = io;
}
console.log(`Express running → PORT ${server.address().port}`);
});
now when you receive an event that you want to emit some message to the clients you can use your io instance from the request object.
router.get('/add-team-member', (req, res) => {
req.io.sockets.emit('addteammember', {member: 6});
//as you are doing a broadcast you just need broadcast msg
....
res.status(200)
res.end()
});
Doing that i also were able to integrate with test framework like mocha, and test the events emited too...
I did some integrations like that, and in my experience the last thing to do was emit the msg to instances in the socket.
As a good practice the very begining of middleware functions i had were doing data validation, data sanitization and cleaning data.
Here is my working example:
var app = require('../app');
var server = require('http').Server(app);
var io = require('socket.io')(server);
io.on('connection', function(client) {
client.emit('connected');
client.on('disconnect', function() {
console.log('disconnected', client.id);
});
});
server.on('request', function(request, response) {
request.io = io;
});
pg.initialize(app.config.DATABASEURL, function(err){
if(err){
throw err;
}
app.set('port', process.env.PORT || 3000);
var server1 = server.listen(app.get('port'), function(){
var host = 'localhost';
var port = server1.address().port;
console.log('Example app listening at http://%s:%s', host, port);
});
});

Your io is actually the socket object, you can emit events from this object to any specific user by -
io.to(userSocketId).emit('eventName', data);
Or you can broadcast by -
io.emit('eventName', data);
Just create require socket.io before using it :)

You can use emiter-adapter to emit data to client in other process/server. It use redis DB as backend for emitting messages.

I did something similar in the past, using namespaces.
Let's say your client connect to your server using "Frontend" as the namespace.
My solution was to create the instance of socket.io as a class in a separate file:
websockets/index.js
const socket = require('socket.io');
class websockets {
constructor(server) {
this.io = socket(server);
this.frontend = new Frontend(this.io);
this.io.use((socket, next) => {
// put here the logic to authorize your users..
// even better in a separate file :-)
next();
});
}
}
class Frontend {
constructor(io) {
this.nsp = io.of('/Frontend');
[ ... ]
}
}
module.exports = websockets;
Then in App.js
const app = require('express')();
const server = require('http').createServer(app);
const websockets = require('./websockets/index');
const WS = new websockets(server);
app.use('/', (req, res, next) => {
req.websocket = WS;
next();
}, require('./routes/index'));
[ ... ]
Finally, your routes can do:
routes/index.js
router.get('/add-team-member', (req, res) => {
req.websocket.frontend.nsp.emit('whatever', { ... });
[ ... ]
});