Develop Reference

Mongoose timeouts talking to AWS DocumentDB

We are trying to connect to AWS DocumentDB in a lambda which is built on Express in Serverless. To do this we're using mongoose and a connection function that looks something like
import mongoose from 'mongoose';
import logger from './utils/logger';
import fs from 'fs';
const READYSTATE_CONNECTED = 1;
const mongoDB = process.env.MONGODB_URI;
const certificateFilePath = __dirname + '/rds-combined-ca-bundle.pem';
logger.info(`Loading certificate file from ${certificateFilePath}`);
let ca = [fs.readFileSync(certificateFilePath)];
logger.info('Connection is ' + mongoose.connection.readyState);
if (mongoose.connection.readyState !== READYSTATE_CONNECTED) {
logger.info(`Connecting to mongo using env connection string ${mongoDB}`);
mongoose.connect(mongoDB, { useNewUrlParser: true, useUnifiedTopology: true, checkServerIdentity: false, ssl: true, sslCA: ca }).catch((err) => {
logger.error(`Unable to connect to mongoose due to ${err.reason}`);
console.error(err);
});
}
mongoose.Promise = global.Promise;
const db = mongoose.connection;
// eslint-disable-next-line no-console
db.on('error', console.error.bind(console, 'MongoDB connection error:'));
export default db;
The idea here being that we maintain a connection and reuse it to avoid the expense of creating new connections for each request that comes into the lambda. For the most part this works fine but every once in a while (perhaps 2x a day) we see problems connecting to the database. It seems to crash the lambda pretty hard and we have to trigger a change on the lambda to trick lambda into restarting our application after which all works fine again for another few hours. We run in 4 identical environments and it seems like the production environment is the only one which experiences this problem. Production is slightly busier than the other environments but really only by 50%.
The error looks like
2020-11-09T20:10:36.565Z d88c9b33-6b84-44cd-8c1d-297c6334aad5 ERROR MongooseServerSelectionError: connection timed out
at NativeConnection.Connection.openUri (/var/task/node_modules/mongoose/lib/connection.js:800:32)
at /var/task/node_modules/mongoose/lib/index.js:342:10
at /var/task/node_modules/mongoose/lib/helpers/promiseOrCallback.js:31:5
at new Promise (<anonymous>)
at promiseOrCallback (/var/task/node_modules/mongoose/lib/helpers/promiseOrCallback.js:30:10)
at Mongoose.connect (/var/task/node_modules/mongoose/lib/index.js:341:10)
at Object.<anonymous> (/var/task/src/mongoose.js:19:24)
at Module._compile (internal/modules/cjs/loader.js:1137:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
at Module.load (internal/modules/cjs/loader.js:985:32)
at Function.Module._load (internal/modules/cjs/loader.js:878:14)
at Module.require (internal/modules/cjs/loader.js:1025:19)
at require (internal/modules/cjs/helpers.js:72:18)
at Object.<anonymous> (/var/task/src/AppBuilder.js:17:1)
at Module._compile (internal/modules/cjs/loader.js:1137:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10) {
reason: TopologyDescription {
type: 'ReplicaSetNoPrimary',
setName: 'rs0',
maxSetVersion: null,
maxElectionId: null,
servers: Map {
'documentdbmasterinstance-xxxx.xxx.us-east-1.docdb.amazonaws.com:27017' => [ServerDescription],
'documentdbreplica1instance-xxxx.xxxx.us-east-1.docdb.amazonaws.com:27017' => [ServerDescription],
'documentdbreplica2instance-xxxx.xxxx.us-east-1.docdb.amazonaws.com:27017' => [ServerDescription]
},
stale: false,
compatible: true,
compatibilityError: null,
logicalSessionTimeoutMinutes: null,
heartbeatFrequencyMS: 10000,
localThresholdMS: 15,
commonWireVersion: 6
}
Thus far we've been unable to pinpoint any particular action which causes this. There does look to be a slight increase in connections to the database at the time but only to about 75 connections and we're running on r5.large which should allow 1700 connections so we're well off that limit.
I was unsure if the mention of ReplicaSetNoPrimary in the error log is a red herring but it doesn't seem to mentioned anywhere in similar issue reports. I am suspicious about if the connection is really timing out. None of the lambda invocations take more than 200ms.
I suppose the questions are:
Is there anything obvious in the connection code which would cause this?
Is there a better, more canonical way to establish and maintain connections in this express application turned lambda?
Is the ReplicaSetNoPrimary indicative that there is some issue with the documentdb electing a new primary or the primary being unreachable?
Any suggestions for more logging I could add to chase this down?
Edit:
Our connection strings look like
mongodb://redacted:redacted#prod-db.cluster-cvgzkbo26lzb.us-east-1.docdb.amazonaws.com:27017/database?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false

There could be multiple reasons for the connection timeout. The most common reason is to whitelist your IP address or enable public access so you can access the database.
The other reason could be the protocol you're using. For more information please share the format of the connection string so I can check and update my answer accordingly.

Node Js mysql(and mysql2) ECONNRESET

i am currently trying to connect to a MySQL server on the internet using Node.Js with the mysql or the mysql2 NPM dependencies to use queries and other related stuff.
the code is simple...
//i import my dependency
const mysql = require('mysql2') //either 'mysql' or 'mysql2'
//i create my pool to create connections as needed
var conn = mysql.createPool({
host: 'some_database_i_have_access_to.mysql.uhserver.com',
user: 'valid_user',
password: 'valid_password',
database: 'some_existing_database'
})
//i try to connect (this is the part where it fails)
conn.getConnection((err,conn) => {
if (err) throw err //<- the error is thrown here
//i do query stuff
conn.query("SELECT * FROM atable",(err,res,firlds) => {
if(err) throw err
console.log(JSON.stringify(res))
})
//i close the connection
conn.end()
})
yet i always get an Error like this:
Error: read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
--------------------
at Protocol._enqueue (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\protocol\Protocol.js:144:48)
at Protocol.handshake (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\protocol\Protocol.js:51:23)
at Connection.connect (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\Connection.js:118:18)
at Object.<anonymous> (C:\Users\Aluno\Desktop\my-project\private\dtp-mysql.js:13:6)
at Module._compile (internal/modules/cjs/loader.js:707:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
at Module.load (internal/modules/cjs/loader.js:605:32)
at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
at Function.Module._load (internal/modules/cjs/loader.js:536:3)
at Module.require (internal/modules/cjs/loader.js:643:17)
all i know about the error is that the connection abruptly closes in one of the sides as stated in this question (Node js ECONNRESET), but nothing more, and creating singular connections does not solve this issue for me either.
any fixes to that?

you can also ref below url.
error while inserting LARGE volume data in mysql by using node.js （error code: 'ECONNRESET'）
I have fixed this issue. It is caused by the default definition max_allowed_packet. Find max_allowed_packet in my.ini (C:\ProgramData\MySQL\MySQL Server 5.7). Update to 'max_allowed_packet=64M'. Restart mysql. Done.

Hi I know this was asked some time ago, but is it possibly because you're using:
conn.end()
Since you're using a Pooled connection, I think you can release the connection using
conn.release()
Or
conn.destroy()

Node.js steam trade bot

So I'm trying to code with node.js for the first time, and have very small prior coding knowledge. I'm trying to code a steam trading bot, but I'm already stumped at the basics of it. This is the code I have, titled "bot.js":
const SteamUser = require('steam-user');
const client = new SteamUser();
const config = require('./config.json');
const logOnOptions = {
accountName: config.username,
password: config.password
};
client.logOn(logOnOptions);
client.on('loggedOn', () => {
console.log('Succesfully logged on.');
});
In the config.json file I just have the username and password which are fine. My problem is that when typing in node bot.js after installing steam-user, I get this error:
module.js:471
throw err;
^
Error: Cannot find module 'C:\Users\Name\Desktop\bot\bot.js'
at Function.Module._resolveFilename (module.js:469:15)
at Function.Module._load (module.js:417:25)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:389:7)
at startup (bootstrap_node.js:149:9)
at bootstrap_node.js:504:3
Help?

This is not an error with your code, but an error node is throwing saying it can't find bot.js in your directory C:\Users\Name\Desktop\bot
Verify that that is the directory where bot.js lives and that the file name is indeed bot.js. Also make sure there are no hidden extensions, the file could possibly be bot.js.txt.

Running a js file with node

Currently, you just log on to the ssh with putty.
We're trying to run a js file which then basically creates an empty log file, currently we're just getting this error:
port : '1157', ^^^^ SyntaxError: Unexpected identifier at exports.runInThisContext (vm.js:73:16) at Module._compile (module.js:443:25) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:501:10) at startup (node.js:129:16) at node.js:814:3
Here's the code we think is the problem:
var mysqlInfo;
mysqlInfo = {
host : https//www.random.com, port : '2332',
user : 'test#localhost', password : '32323232',
database : 'test_installation',
charset : 'utf8_general_ci'
};
we're not sure how to format it
Another problem is we get multiple connecting issues, but I think this is the main problem.
details edited out for security, but same format as our actual file

There are few things which i noticed,
first install mysql node module
then require it in you js
the code which you wrote is for exporting module. and as per your question it seems you want to connect to mysql db.
please use the code given below and run it using node connect.js command it will work for you.
var mysql = require("mysql");
var con = mysql.createConnection({
host : https//www.random.com,
port : '2332',
user : 'test#localhost',
password : '32323232',
database : 'test_installation',
charset : 'utf8_general_ci'
});
con.connect(function(err){
if(err){
console.log('Error connecting to Db');
return;
}
console.log('Connection established');
});
Thanks
Amit

OAuth2 with nodejs api and standard javascript app

I'm trying to create an sort of plugin that users can simply add to a website and it will make COR calls to my app and return JSON that will be handled by the client side javascript.
This is working how I want it to, but now I'm trying to make sure that the user logs into my app before being allowed to receive any JSON from my server side app.
From here on I'll refer to my Node.js API as Server and the straight JS plugin as Client
I found a npm plugin for node that handles OAuth2 on the Server, but I'm not sure I'm really understanding how to use it. Here's the link and I found this for taking care of it on the Client side.
Client -> App initializer:
define [
'oauth2'
], (oauth2) ->
App =
Models: {}
Collections: {}
Views: {}
initialize: () ->
$.get "/javascripts/mu-config.json", (config) =>
#api_url = config.api
#site = config.site
#credentials = config.credentials
#make_oauth_call()
make_oauth_call: ->
#xhr = new oauth2.OAuth2XMLHttpRequest
authorizeEndpoint: "#{this.api_url}/callback"
tokenEndpoint: "#{this.api_url}/oauth/access_token"
clientID: this.credentials.clientID
clientSecret: this.credentials.clientSecret
localStoragePrefix: "oauth2.#{this.site.name}"
requestAuthorization: (callback) ->
console.log 'what?'
console.log callback
#xhr.onreadystatechange = () ->
console.log "do something"
#xhr.open "GET", "#{this.api_url}/notes?site=1&user=1"
#xhr.setRequestHeader 'Content-type', 'application/x-www-form-urlencoded'
#xhr.send "site=1&user=1"
So what works here? Well the #xhr.open ... does in fact grab JSON from the Server, but that's about it. I'm not getting any errors from the Client, but the console.log 'what?' does not fire and I don't believe anything is getting authenticated.
Server -> oauth.coffee
token = null
credentials =
clientID: "sparkmasterflex"
clientSecret: "bob_the_builder"
site: 'http://marking_up.dev'
OAuth2 = require('simple-oauth2') credentials
authorization_uri = OAuth2.AuthCode.authorizeURL
redirect_uri: 'http://localhost:3000/callback'
scope: 'sites'
state: '55fce6241c8e6432e8dfee583141aa58'
res.redirect(authorization_uri)
OAuth2.AuthCode.getToken
code: "something here"
redirect_uri: "http://localhost:3000/callback"
, saveToken
saveToken = (error, result) ->
console.log('Access Token Error', error.message) if error
token = OAuth2.AccessToken.create(result)
module.exports = OAuth2
Server -> router
express = require("express")
db = require "../database"
oauth2 = require "../oauth"
router = express.Router()
# GET home page.
router.get "/", (req, res) ->
res.render 'index',
title: "Hello world"
# Initial page redirecting to Github
router.get '/auth', (req, res) ->
res.redirect authorization_uri
# Callback service parsing the authorization token and asking for the access token
# router.get '/callback', (req, res) ->
router.route('/callback')
.get (req, res) ->
code = req.query.code
console.log '/callback'
oauth2.AuthCode.getToken
code: code
redirect_uri: 'http://localhost:3000/callback'
, saveToken
saveToken = (error, result) ->
console.log('Access Token Error', error.message) if error
token = oauth2.AccessToken.create(result)
module.exports = router
Running the node server I get this error:
/Users/raymondke99/Sites/marking_up_api/oauth.js:19
res.redirect(authorization_uri);
^
ReferenceError: res is not defined
at Object.<anonymous> (/Users/raymondke99/Sites/marking_up_api/oauth.js:19:1)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/Users/raymondke99/Sites/marking_up_api/routes/index.js:7:10)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
I'm kinda at a loss here. The documentation for both of these seem pretty thorough but I still feel like I'm missing a huge chunk of information. Can anyone help and/or lead me to help?
Thank you
EDIT
I removed res.redirect() from oauth.coffee and I get the following error:
/Users/raymondke99/Sites/marking_up_api/node_modules/simple-oauth2/lib/core.js:16
throw new Error('Callback not provided on API call');
^
Error: Callback not provided on API call
at Object.api (/Users/raymondke99/Sites/marking_up_api/node_modules/simple-oauth2/lib/core.js:16:13)
at Object.getToken (/Users/raymondke99/Sites/marking_up_api/node_modules/simple-oauth2/lib/client/auth-code.js:34:8)
at Object.<anonymous> (/Users/raymondke99/Sites/marking_up_api/oauth.js:19:17)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/Users/raymondke99/Sites/marking_up_api/routes/index.js:7:10)
I have more than one router because I'm using expressjs and I'm not sure where I'm supposed to have the 'catch-all' redirect. Does it need to go into every router?

Why do you have "res.redirect(authorization_uri)" in the oath file? You seem to already have the GET endpoint in your router?