Mongoose timeouts talking to AWS DocumentDB - javascript

We are trying to connect to AWS DocumentDB in a lambda which is built on Express in Serverless. To do this we're using mongoose and a connection function that looks something like
import mongoose from 'mongoose';
import logger from './utils/logger';
import fs from 'fs';
const READYSTATE_CONNECTED = 1;
const mongoDB = process.env.MONGODB_URI;
const certificateFilePath = __dirname + '/rds-combined-ca-bundle.pem';
logger.info(`Loading certificate file from ${certificateFilePath}`);
let ca = [fs.readFileSync(certificateFilePath)];
logger.info('Connection is ' + mongoose.connection.readyState);
if (mongoose.connection.readyState !== READYSTATE_CONNECTED) {
logger.info(`Connecting to mongo using env connection string ${mongoDB}`);
mongoose.connect(mongoDB, { useNewUrlParser: true, useUnifiedTopology: true, checkServerIdentity: false, ssl: true, sslCA: ca }).catch((err) => {
logger.error(`Unable to connect to mongoose due to ${err.reason}`);
console.error(err);
});
}
mongoose.Promise = global.Promise;
const db = mongoose.connection;
// eslint-disable-next-line no-console
db.on('error', console.error.bind(console, 'MongoDB connection error:'));
export default db;
The idea here being that we maintain a connection and reuse it to avoid the expense of creating new connections for each request that comes into the lambda. For the most part this works fine but every once in a while (perhaps 2x a day) we see problems connecting to the database. It seems to crash the lambda pretty hard and we have to trigger a change on the lambda to trick lambda into restarting our application after which all works fine again for another few hours. We run in 4 identical environments and it seems like the production environment is the only one which experiences this problem. Production is slightly busier than the other environments but really only by 50%.
The error looks like
2020-11-09T20:10:36.565Z d88c9b33-6b84-44cd-8c1d-297c6334aad5 ERROR MongooseServerSelectionError: connection timed out
at NativeConnection.Connection.openUri (/var/task/node_modules/mongoose/lib/connection.js:800:32)
at /var/task/node_modules/mongoose/lib/index.js:342:10
at /var/task/node_modules/mongoose/lib/helpers/promiseOrCallback.js:31:5
at new Promise (<anonymous>)
at promiseOrCallback (/var/task/node_modules/mongoose/lib/helpers/promiseOrCallback.js:30:10)
at Mongoose.connect (/var/task/node_modules/mongoose/lib/index.js:341:10)
at Object.<anonymous> (/var/task/src/mongoose.js:19:24)
at Module._compile (internal/modules/cjs/loader.js:1137:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
at Module.load (internal/modules/cjs/loader.js:985:32)
at Function.Module._load (internal/modules/cjs/loader.js:878:14)
at Module.require (internal/modules/cjs/loader.js:1025:19)
at require (internal/modules/cjs/helpers.js:72:18)
at Object.<anonymous> (/var/task/src/AppBuilder.js:17:1)
at Module._compile (internal/modules/cjs/loader.js:1137:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10) {
reason: TopologyDescription {
type: 'ReplicaSetNoPrimary',
setName: 'rs0',
maxSetVersion: null,
maxElectionId: null,
servers: Map {
'documentdbmasterinstance-xxxx.xxx.us-east-1.docdb.amazonaws.com:27017' => [ServerDescription],
'documentdbreplica1instance-xxxx.xxxx.us-east-1.docdb.amazonaws.com:27017' => [ServerDescription],
'documentdbreplica2instance-xxxx.xxxx.us-east-1.docdb.amazonaws.com:27017' => [ServerDescription]
},
stale: false,
compatible: true,
compatibilityError: null,
logicalSessionTimeoutMinutes: null,
heartbeatFrequencyMS: 10000,
localThresholdMS: 15,
commonWireVersion: 6
}
Thus far we've been unable to pinpoint any particular action which causes this. There does look to be a slight increase in connections to the database at the time but only to about 75 connections and we're running on r5.large which should allow 1700 connections so we're well off that limit.
I was unsure if the mention of ReplicaSetNoPrimary in the error log is a red herring but it doesn't seem to mentioned anywhere in similar issue reports. I am suspicious about if the connection is really timing out. None of the lambda invocations take more than 200ms.
I suppose the questions are:
Is there anything obvious in the connection code which would cause this?
Is there a better, more canonical way to establish and maintain connections in this express application turned lambda?
Is the ReplicaSetNoPrimary indicative that there is some issue with the documentdb electing a new primary or the primary being unreachable?
Any suggestions for more logging I could add to chase this down?
Edit:
Our connection strings look like
mongodb://redacted:redacted#prod-db.cluster-cvgzkbo26lzb.us-east-1.docdb.amazonaws.com:27017/database?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false

There could be multiple reasons for the connection timeout. The most common reason is to whitelist your IP address or enable public access so you can access the database.
The other reason could be the protocol you're using. For more information please share the format of the connection string so I can check and update my answer accordingly.

Related

Connecting Mongodb databse to node.js server

I have been trying to make a web app using node.js and express and decided to work with mongodb.
I am using mongodb node.js driver version: 4.3.1
I have tried all the possible ways to connect the node.js server with my mongodb atlas database.
My database also got connected using the following code in my db.js file:
const app = require('./app');
const MongoClient = require('mongodb').MongoClient
const Url = 'mongodb+srv://todoAppUser:<myOriginalPasswordHere>#cluster0.6lvjr.mongodb.net/myDatabase?retryWrites=true&w=majority';
MongoClient.connect(Url, function (err, client) {
if (err) throw err;
var db = client.db('myDatabase');
db.collection('products').findOne({}, function (findErr, result) {
if (findErr) throw findErr;
console.log(result.name);
client.close();
});
});
The above code works fine and gives the output as well.
But I want to use MVC (Model-view-Controller) framework for which I need to export the connection.
I made the following change in the above code:
MongoClient.connect(Url, function (err, client) {
if (err) throw err;
var db = client.db('myDatabase');
db.collection('products').findOne({}, function (findErr, result) {
if (findErr) throw findErr;
console.log(result.name);
module.exports = db
client.close();
});
});
After the change when I try to access my connection (const productCollection = require('./db').collection("product");) from any other file of mine, it gives me the following error:
const productCollection = require('./db').collection("product");
^
TypeError: require(...).collection is not a function
at Object.<anonymous> (D:\Kush- Complete Data\exp-projects\nodeApp\productController.js:1:43)
at Module._compile (internal/modules/cjs/loader.js:1072:14)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
at Module.load (internal/modules/cjs/loader.js:937:32)
at Function.Module._load (internal/modules/cjs/loader.js:778:12)
at Module.require (internal/modules/cjs/loader.js:961:19)
at require (internal/modules/cjs/helpers.js:92:18)
at Object.<anonymous> (D:\Kush- Complete Data\exp-projects\nodeApp\router.js:3:27)
at Module._compile (internal/modules/cjs/loader.js:1072:14)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
[nodemon] app crashed - waiting for file changes before starting...
May anyone please guide me and show a possible way out.
Thanks,
Kush
This can't work, because modules are evaluated synchronously. Your callback is called asynchronously but your module has been alread evaluated at this time. Means module.exports has no effect and require('./db').collection() is not defined. Please see Node.js Modules for details.
To solve your problem, handle the connection stored in a module internal variable and export a getter instead of the connection variable itself.
// db.js
let client;
let db;
function connectDB(url, dbName) {
client = new MongoClient(url);
return new Promise(function(resolve, reject) {
client.connect(function(err) {
if(err) return reject(err);
db = client.db(dbName);
return resolve(db);
});
});
}
function getCurrentDB() {
return db;
}
module.exports.connectDB = connectDB;
module.exports.getCurrentDB = getCurrentDB;
Then reuse your opened connection in other files like the following:
// other.js
const db = require("./db.js");
db.getCurrentDB().collection("product");
Of course, getCurrentDB() can only return a database connection if a connection has been established via connectDB() beforehand. So you have to wait for the resolution of the Promise.
[SOLVED]
I figured out, in newer versions of mongodb they have essentially changed the way of connecting node server to the database.
To establish a reusable connection (So that we can access the connected database from any other file), I created an async function in my db.js file where connection is established and then exported it. In the end of the file, I have called the function.
The code is as follows:
const {MongoClient} = require('mongodb')
const client = new MongoClient('mongodb+srv://todoAppUser:<password>#cluster0.6lvjr.mongodb.net/myDatabase?retryWrites=true&w=majority')
async function start(){
await client.connect()
console.log("Connected")
module.exports = client.db()
const app = require('./app')
app.listen(3000)
}
start()
and while calling it from another file:
const productCollection = require('./db').collection("product");
This code gives me no error and works perfectly fine.
With the help of the above code, one can use this conveniently while following the MVC (Model-View-Controller) framework.

Bot staying offline [duplicate]

This question already has answers here:
How do I fix CLIENT_MISSING_INTENTS error?
(5 answers)
Discord.js v12 code breaks when upgrading to v13
(1 answer)
Closed 1 year ago.
I've tried to create a bot on discord, and with multiple tutorials, I've followed all of the instructions, and it seems to work fine in the beginning. However, with all of the tutorials, I'm met with one problem. In my index.js file for my bot, when I open the VSCode terminal and type in the command node index.js (which is supposed to make my bot go online, I get the following error messages:
/Users/myname/Desktop/testbot/node_modules/discord.js/src/client/Client.js:544
throw new TypeError('CLIENT_MISSING_INTENTS');
^
TypeError [CLIENT_MISSING_INTENTS]: Valid intents must be provided for the Client.
at Client._validateOptions (/Users/myname/Desktop/testbot/node_modules/discord.js/src/client/Client.js:544:13)
at new Client (/Users/myname/Desktop/testbot/node_modules/discord.js/src/client/Client.js:73:10)
at Object.<anonymous> (/Users/myname/Desktop/testbot/index.js:2:13)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
at node:internal/main/run_main_module:17:47 {
[Symbol(code)]: 'CLIENT_MISSING_INTENTS'
}
I have node.js 16.13.0, and discord.js installed. My bot is a member of my server, but it's always offline. What can I do to fix it?
Here's the code for index.js:
const Discord = require('discord.js');
const bot = new Discord.Client();
const token = '[not putting my token online]';
const PREFIX = '';
bot.on('ready', () => {
console.log('Online!');
});
bot.login(token);
It seems like you don't have intents added with your bot which is needed to get it running according to the discord.js starter guide
// Require the necessary discord.js classes
const { Client, Intents } = require('discord.js');
const token = 'your token here';
// Create a new client instance
const client = new Client({ intents: [Intents.FLAGS.GUILDS] });
// When the client is ready, run this code (only once)
client.once('ready', () => {
console.log('Ready!');
});
// Login to Discord with your client's token
client.login(token);

Can't Connect to MongoDB using Mongoose

I'm following this video tutorial on the MERN stack and I'm unable to connect to MongoDB for some very strange reason. This issue has been frustrating me quite a bit since I'm probably just missing something very basic, so please forgive me if the answer is painfully obvious to you.
The video uses mlab for MongoDB, which is no longer available, so I'm instead using MongoDB Atlas. The key I'm supposed to use to connect my application to the database is this:
mongodb+srv://<username>:<password>#fs-shopping-list.6rzkd.mongodb.net/<dbname>?retryWrites=true&w=majority
My password doesn't contain any special characters, and my IP address is on the whitelist. As for dbname, I have one database named "data" with a collection called "items," so I'm using "data" for dbname.
The code in question that is causing my problem is in a file called server.js:
const db = require('./config/keys').mongoURI; // I keep my key in a separate file in the way shown in the video
// Connect to MongoDB
mongoose
.connect(db, { useNewUrlParser: true, useUnifiedTopology: true })
.then(() => console.log('MongoDB connected.'))
.catch(err => console.log(err));
I keep getting this error when I try to run the server (I edited out my name from some of the paths):
{ MongooseServerSelectionError: bad auth Authentication failed.
at NativeConnection.Connection.openUri (/fs_shopping_list/node_modules/mongoose/lib/connection.js:828:32)
at Mongoose.connect (/fs_shopping_list/node_modules/mongoose/lib/index.js:335:15)
at Object.<anonymous> (/fs_shopping_list/server.js:15:6)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)
message: 'bad auth Authentication failed.',
reason:
TopologyDescription {
type: 'ReplicaSetNoPrimary',
setName: null,
maxSetVersion: null,
maxElectionId: null,
servers:
Map {
'fs-shopping-list-shard-00-01.6rzkd.mongodb.net:27017' => [ServerDescription],
'fs-shopping-list-shard-00-02.6rzkd.mongodb.net:27017' => [ServerDescription],
'fs-shopping-list-shard-00-00.6rzkd.mongodb.net:27017' => [ServerDescription] },
stale: false,
compatible: true,
compatibilityError: null,
logicalSessionTimeoutMinutes: null,
heartbeatFrequencyMS: 10000,
localThresholdMS: 15,
commonWireVersion: null } }
Can someone please help me find out what I'm doing wrong so I can continue the tutorial? Thank you for your time.
EDIT: I don't think adding another account for database access will solve the problem, since admin accounts on MongoDB have the ability to read and write to their databases. The only thing I can think of that is possibly stopping me is maybe my Norton antivirus, although I'm not sure how to test this hypothesis.
Here an example of how I do it with mongoose:
const connectToMongo = async () => {
try {
await mongoose.connect(mongoUrl, { useNewUrlParser: true });
console.log('connected to MongoDB');
} catch(error) {
console.log('error connection to MongoDB:', error.message);
}
};
Here is an example of the mongoUrl: mongo+srv://username:password#cluster0-ipl5c.mongodb.net/collectionname?retryWrites=true
Please make sure that you create a user to read and write to the database that isn't the admin account. The URI string you get from the "Connect" button might use the admin account even though that's not the account you want to use in the URI string, so keep that in mind. If you did that and you're still unable to connect, please use this checklist:
check that correct IP is whitelisted
check that user has correct rights
check that you are using the right database name (collection)
check if you have special characters in your password (https://docs.atlas.mongodb.com/troubleshoot-connection/#special-pass-characters)
Try to add dbName in options:
await mongoose.connect('mongodb://root:example#mongo:27017', { dbName: "blog" });

Node Js mysql(and mysql2) ECONNRESET

i am currently trying to connect to a MySQL server on the internet using Node.Js with the mysql or the mysql2 NPM dependencies to use queries and other related stuff.
the code is simple...
//i import my dependency
const mysql = require('mysql2') //either 'mysql' or 'mysql2'
//i create my pool to create connections as needed
var conn = mysql.createPool({
host: 'some_database_i_have_access_to.mysql.uhserver.com',
user: 'valid_user',
password: 'valid_password',
database: 'some_existing_database'
})
//i try to connect (this is the part where it fails)
conn.getConnection((err,conn) => {
if (err) throw err //<- the error is thrown here
//i do query stuff
conn.query("SELECT * FROM atable",(err,res,firlds) => {
if(err) throw err
console.log(JSON.stringify(res))
})
//i close the connection
conn.end()
})
yet i always get an Error like this:
Error: read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
--------------------
at Protocol._enqueue (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\protocol\Protocol.js:144:48)
at Protocol.handshake (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\protocol\Protocol.js:51:23)
at Connection.connect (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\Connection.js:118:18)
at Object.<anonymous> (C:\Users\Aluno\Desktop\my-project\private\dtp-mysql.js:13:6)
at Module._compile (internal/modules/cjs/loader.js:707:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
at Module.load (internal/modules/cjs/loader.js:605:32)
at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
at Function.Module._load (internal/modules/cjs/loader.js:536:3)
at Module.require (internal/modules/cjs/loader.js:643:17)
all i know about the error is that the connection abruptly closes in one of the sides as stated in this question (Node js ECONNRESET), but nothing more, and creating singular connections does not solve this issue for me either.
any fixes to that?
you can also ref below url.
error while inserting LARGE volume data in mysql by using node.js (error code: 'ECONNRESET')
I have fixed this issue. It is caused by the default definition max_allowed_packet. Find max_allowed_packet in my.ini (C:\ProgramData\MySQL\MySQL Server 5.7). Update to 'max_allowed_packet=64M'. Restart mysql. Done.
Hi I know this was asked some time ago, but is it possibly because you're using:
conn.end()
Since you're using a Pooled connection, I think you can release the connection using
conn.release()
Or
conn.destroy()

OAuth2 with nodejs api and standard javascript app

I'm trying to create an sort of plugin that users can simply add to a website and it will make COR calls to my app and return JSON that will be handled by the client side javascript.
This is working how I want it to, but now I'm trying to make sure that the user logs into my app before being allowed to receive any JSON from my server side app.
From here on I'll refer to my Node.js API as Server and the straight JS plugin as Client
I found a npm plugin for node that handles OAuth2 on the Server, but I'm not sure I'm really understanding how to use it. Here's the link and I found this for taking care of it on the Client side.
Client -> App initializer:
define [
'oauth2'
], (oauth2) ->
App =
Models: {}
Collections: {}
Views: {}
initialize: () ->
$.get "/javascripts/mu-config.json", (config) =>
#api_url = config.api
#site = config.site
#credentials = config.credentials
#make_oauth_call()
make_oauth_call: ->
#xhr = new oauth2.OAuth2XMLHttpRequest
authorizeEndpoint: "#{this.api_url}/callback"
tokenEndpoint: "#{this.api_url}/oauth/access_token"
clientID: this.credentials.clientID
clientSecret: this.credentials.clientSecret
localStoragePrefix: "oauth2.#{this.site.name}"
requestAuthorization: (callback) ->
console.log 'what?'
console.log callback
#xhr.onreadystatechange = () ->
console.log "do something"
#xhr.open "GET", "#{this.api_url}/notes?site=1&user=1"
#xhr.setRequestHeader 'Content-type', 'application/x-www-form-urlencoded'
#xhr.send "site=1&user=1"
So what works here? Well the #xhr.open ... does in fact grab JSON from the Server, but that's about it. I'm not getting any errors from the Client, but the console.log 'what?' does not fire and I don't believe anything is getting authenticated.
Server -> oauth.coffee
token = null
credentials =
clientID: "sparkmasterflex"
clientSecret: "bob_the_builder"
site: 'http://marking_up.dev'
OAuth2 = require('simple-oauth2') credentials
authorization_uri = OAuth2.AuthCode.authorizeURL
redirect_uri: 'http://localhost:3000/callback'
scope: 'sites'
state: '55fce6241c8e6432e8dfee583141aa58'
res.redirect(authorization_uri)
OAuth2.AuthCode.getToken
code: "something here"
redirect_uri: "http://localhost:3000/callback"
, saveToken
saveToken = (error, result) ->
console.log('Access Token Error', error.message) if error
token = OAuth2.AccessToken.create(result)
module.exports = OAuth2
Server -> router
express = require("express")
db = require "../database"
oauth2 = require "../oauth"
router = express.Router()
# GET home page.
router.get "/", (req, res) ->
res.render 'index',
title: "Hello world"
# Initial page redirecting to Github
router.get '/auth', (req, res) ->
res.redirect authorization_uri
# Callback service parsing the authorization token and asking for the access token
# router.get '/callback', (req, res) ->
router.route('/callback')
.get (req, res) ->
code = req.query.code
console.log '/callback'
oauth2.AuthCode.getToken
code: code
redirect_uri: 'http://localhost:3000/callback'
, saveToken
saveToken = (error, result) ->
console.log('Access Token Error', error.message) if error
token = oauth2.AccessToken.create(result)
module.exports = router
Running the node server I get this error:
/Users/raymondke99/Sites/marking_up_api/oauth.js:19
res.redirect(authorization_uri);
^
ReferenceError: res is not defined
at Object.<anonymous> (/Users/raymondke99/Sites/marking_up_api/oauth.js:19:1)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/Users/raymondke99/Sites/marking_up_api/routes/index.js:7:10)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
I'm kinda at a loss here. The documentation for both of these seem pretty thorough but I still feel like I'm missing a huge chunk of information. Can anyone help and/or lead me to help?
Thank you
EDIT
I removed res.redirect() from oauth.coffee and I get the following error:
/Users/raymondke99/Sites/marking_up_api/node_modules/simple-oauth2/lib/core.js:16
throw new Error('Callback not provided on API call');
^
Error: Callback not provided on API call
at Object.api (/Users/raymondke99/Sites/marking_up_api/node_modules/simple-oauth2/lib/core.js:16:13)
at Object.getToken (/Users/raymondke99/Sites/marking_up_api/node_modules/simple-oauth2/lib/client/auth-code.js:34:8)
at Object.<anonymous> (/Users/raymondke99/Sites/marking_up_api/oauth.js:19:17)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/Users/raymondke99/Sites/marking_up_api/routes/index.js:7:10)
I have more than one router because I'm using expressjs and I'm not sure where I'm supposed to have the 'catch-all' redirect. Does it need to go into every router?
Why do you have "res.redirect(authorization_uri)" in the oath file? You seem to already have the GET endpoint in your router?

Categories

Resources