I have just started learning about MongoDB and I am trying to host my node js application locally via MongoDB Server 6.0 (without using mongoose or atlas)
I copied the async javascript code given in the MongoDB docs. I made sure to run mongod before executing the below code
MongoDB server started
const { MongoClient } = require("mongodb");
// Connection URI
const uri =
"**mongodb://localhost:27017**";
// Create a new MongoClient
const client = new MongoClient(uri);
async function run() {
try {
// Connect the client to the server (optional starting in v4.7)
await client.connect();
// Establish and verify connection
await client.db("admin").command({ ping: 1 });
console.log("Connected successfully to server");
} finally {
// Ensures that the client will close when you finish/error
await client.close();
}
}
run().catch(console.dir);
It's throwing an error:
image of the error it's throwing
Problem is, the localhost alias resolves to IPv6 address ::1 instead of 127.0.0.1
However, net.ipv6 defaults to false.
The best option would be to start the MongoDB with this configuration:
net:
ipv6: true
bindIpAll: true
or
net:
ipv6: true
bindIp: localhost
Then all variants should work:
C:\>mongosh "mongodb://localhost:27017" --quiet --eval "db.getMongo()"
mongodb://localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.6.0
C:\>mongosh "mongodb://127.0.0.1:27017" --quiet --eval "db.getMongo()"
mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.6.0
C:\>mongosh "mongodb://[::1]:27017" --quiet --eval "db.getMongo()"
mongodb://[::1]:27017/?directConnection=true&appName=mongosh+1.6.0
If you don't run MongoDB as a service then it would be
mongod --bind_ip_all --ipv6 <other options>
NB, I don't like configuration
net:
bindIp: <ip_address>
in my opinion this makes only sense on a computer with multiple network interfaces. Use bindIp: localhost if you need to prevent any connections from remote computer (e.g. while maintenance or when used as backend database for a web-service), otherwise use bindIpAll: true
EDIT
I found the error. The mistake was very obvious: I did not include the
require("dotenv").config(); in the connection.js file. Without this, the database connection simply fails after a timeout because it does not have any connection details.
I found an update log from the Mariadb Node.js connector team stating they have a few errors where Mariadb does not provide sufficient error messages (it sometimes only offers a "timeout" without further information), so I changed what I was looking for, and found the mistake.
For anyone getting a similar error message, this can mean anything, so check all parts of your code!
Original Post
I am trying to get familiar with Nodejs and express, but ran into an issue that I can't seem to solve:
When creating a Mariadb database pool in a seperate file, and exporting the pool using module.exports, I am having trouble using the same pool in another file. I get a timeout error when trying to use the pool to query a database.
If I use the exact same code in the same file instead of two separate files, the query works perfectly, so I think there is something going wrong during module.exports = pool.
Am I missing something? Thanks in advance!
I have two files:
index.js:
// import express web framework
const express = require("express");
//create an express application
const app = express();
const pool = require('./database/connection')
const cors = require('cors');
//middleware
app.use(cors())
app.use(express.json())
getData = async () => {
data = await pool.query("call stored_procedure")
console.log (data)
}
getData()
app.listen(3001, () => {
console.log('Serving running on port 3001')
})
and connection.js:
//import mariadb library
const mariadb = require("mariadb")
//function that create mariadb connection pool for database
const createPool = () => {
try {
return (
mariadb.createPool({
connectionLimit: 10,
host: process.env.MARIADB_HOST,
user: process.env.MARIADB_USER,
password: process.env.MARIADB_PASSWORD,
database: process.env.MARIADB_DB,
port: 3306
})
)
}
catch (err) {
console.error('Failed to connect to database: ')
console.error(err)
}
}
const pool = createPool()
//export database connection pool
module.exports = pool
Running this app results in the following error (after some time):
path_to_dir/node_modules/mariadb/lib/misc/errors.js:57
return new SqlError(msg, sql, fatal, info, sqlState, errno, additionalStack, addHeader);
^
SqlError: (conn=-1, no: 45028, SQLState: HY000) retrieve connection from pool timeout after 10001ms
(pool connections: active=0 idle=0 limit=10)
at Object.module.exports.createError (path_to_dir/node_modules/mariadb/lib/misc/errors.js:57:10)
at Pool._requestTimeoutHandler (path_to_dir/node_modules/mariadb/lib/pool.js:345:26)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7) {
text: 'retrieve connection from pool timeout after 10001ms\n' +
' (pool connections: active=0 idle=0 limit=10)',
sql: null,
fatal: false,
errno: 45028,
sqlState: 'HY000',
code: 'ER_GET_CONNECTION_TIMEOUT'
}
I found the error. The mistake was very obvious: I did not include the require("dotenv").config(); in the connection.js file. Without this, the database connection simply fails after a timeout because it does not have any connection details. I found an update log from the Mariadb Node.js connector team stating they have a few errors where Mariadb does not provide sufficient error messages (it sometimes only offers a "timeout" without further information), so I changed what I was looking for, and found the mistake.
For anyone getting a similar error message, this can mean anything, so check all parts of your code!
I've been using PSQL for over 2 years now, this issue started occurring 3 Months ago.
The database would stop responding after a day of runtime until the affected Node.js process is restarted.
4 Days ago this issue got much worse, unless the host OS was restarted the database stops responding within minutes or less of process runtime.
This issue occurs in only one Node.js process, I have about 4 other Node.js processes running perfectly fine, so it's most likely an issue with my code.
Highest statistics for the affected process:
10 Sessions (constantly stays at that number)
90000 Transactions Per Second (Transactions)
140 Tuples in (Updates)
8000000 Tuples out (Returned)
180000 Block I/O (Hits)
I have tried:
Re-starting Postgres
Re-installing Postgres
using pg-pool (Runs into error: Connection timed out)
using pg-promise (I'm not sure how to apply this module without spamming tasks or connections)
No Errors are emitted, and the connection becomes increasingly slow over several minutes until the pgAdmin Dashboard basically flatlines and no further response is received.
Code:
Pool creation (initiated on startup):
const { Pool } = require('pg');
const auth = require('./auth.json');
const ch = require('./ClientHelper');
const pool = new Pool({
user: 'postgres',
host: 'localhost',
database: 'Ayako-v1.5',
password: auth.pSQLpw,
port: 5432,
});
pool.query('SELECT NOW() as now;', (err) => {
if (err) {
ch.logger("| Couldn't connect to DataBase", err.stack);
} else {
console.log('| Established Connection to DataBase');
}
});
pool.connect((err) => {
if (err) {
ch.logger('Error while logging into DataBase', err.stack);
}
});
pool.on('error', (err) => {
ch.logger('Unexpected error on idle pool client', err);
});
module.exports = pool;
Queries are executed via:
const query = async (query, arr, debug) => {
const pool = require('./DataBase');
if (debug === true) console.log(query, arr);
return pool.query(query, arr).catch((err) => {
console.log(query, arr);
module.exports.logger('Pool Query Error', err);
return null;
});
Queries arrive at the above query function but never receive a response.
File Links:
https://github.com/Larsundso/Ayako-v1.5/blob/main/Files/BaseClient/DataBase.js
https://github.com/Larsundso/Ayako-v1.5/blob/f2110f3cd73325b35a617fe58d19d8d9c46659d9/Files/BaseClient/ClientHelper.js#L215
Versions
PSQL - v14 |
Node.js - v17.8.0 |
Linux - Ubuntu 20.04.4 LTS
I appreciate everyone's help here, logging executed queries brought me on the right track.
The issue was the number of requests handled exceeded PostgreSQL's capabilities, making the queries stack up into timing out.
My solution to this is coupling redis with pSQL to circumvent unnecessarily accessing the Database.
It is my first time using postgres and I am trying to use it with node. I installed postgres in Ubuntu and created the user admin
sudo -i -u postgres
createuser --interactive
in my node code I tried to connect to postgres:
let { Client } = require('pg');
conn = new Client({host:'localhost', port:5432, database:'web-viewer', user: 'admin'});
conn.connect();
but I get this error
UnhandledPromiseRejectionWarning: error: password authentication failed for user "admin"
my pg_hba.conf was
local all postgres peer
then I changed to
local all postgres md5
to try to add a password to my admin user, but when I "createuser --interactive" again, I get this error
createuser: error: could not connect to database template1: FATAL: password authentication failed for user "postgres"
how can I prepare the postgreSQL to connect to node?
You didn't parse the password in your new Client definition:
in my node code I tried to connect to postgres:
let { Client } = require('pg');
conn = new Client({host:'localhost', port:5432, database:'web-viewer', user: 'admin'});
conn.connect();
change it like the following, using the information you saved during the user creation:
let { Client } = require('pg');
conn = new Client({
host:'localhost',
port:5432,
database:'web-viewer',
user: 'admin',
password: 'yourpassword'
});
conn.connect();
Follow the doc.
btw, to avoid using sensible information (your password), consider creating environment variables.
I know the post is 8 months old, but you never know.
How to solve model.find() function produces "buffering timed out after ... ms"? I'm using mongoose v 5.11.0, npm v6.14.8 and mongodb v
Here's the code.
var express = require('express');
var app = express();
var bodyParser = require('body-parser');
const assert = require('assert');
var mongoose = require('mongoose');
try {
var db = mongoose.connect('mongodb://localhost:27017', {useNewUrlParser: true, dbName: 'swag-shop' });
console.log('success connection');
}
catch (error) {
console.log('Error connection: ' + error);
}
var Product = require('./model/product');
var WishList = require('./model/wishlist');
//Allow all requests from all domains & localhost
app.all('/*', function(req, res, next) {
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "X-Requested-With, Content-Type, Accept");
res.header("Access-Control-Allow-Methods", "POST, GET");
next();
});
app.get('/product', function(request, response) {
Product.find({},function(err, products) {
if (err) {
response.status(500).send({error: "Could not fetch products. "+ err});
} else {
response.send(products);
}
});
});
app.listen(3004, function() {
console.log("Swag Shop API running on port 3004...");
});
The product model:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var product = new Schema({
title: String,
price: Number,
likes: {type: Number, default: 0}
});
module.exports = mongoose.model('Product', product);
Additionally, running the file also produces the following warnings:
D:\Test\swag-shop-api>nodemon server.js
[nodemon] 2.0.6
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node server.js`
success connection
Swag Shop API running on port 3004...
(node:28596) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_ARG_TYPE]: The "url" argument must be of type string. Received type function ([Function (anonymous)])
at validateString (internal/validators.js:122:11)
at Url.parse (url.js:159:3)
at Object.urlParse [as parse] (url.js:154:13)
at module.exports (D:\Test\swag-shop-api\node_modules\mongoose\node_modules\mongodb\lib\url_parser.js:15:23)
at connect (D:\Test\swag-shop-api\node_modules\mongoose\node_modules\mongodb\lib\mongo_client.js:403:16)
at D:\Test\swag-shop-api\node_modules\mongoose\node_modules\mongodb\lib\mongo_client.js:217:7
at new Promise (<anonymous>)
at MongoClient.connect (D:\Test\swag-shop-api\node_modules\mongoose\node_modules\mongodb\lib\mongo_client.js:213:12)
at D:\Test\swag-shop-api\node_modules\mongoose\lib\connection.js:820:12
at new Promise (<anonymous>)
at NativeConnection.Connection.openUri (D:\Test\swag-shop-api\node_modules\mongoose\lib\connection.js:817:19)
at D:\Test\swag-shop-api\node_modules\mongoose\lib\index.js:345:10
at D:\Test\swag-shop-api\node_modules\mongoose\lib\helpers\promiseOrCallback.js:31:5
at new Promise (<anonymous>)
at promiseOrCallback (D:\Test\swag-shop-api\node_modules\mongoose\lib\helpers\promiseOrCallback.js:30:10)
at Mongoose._promiseOrCallback (D:\Test\swag-shop-api\node_modules\mongoose\lib\index.js:1135:10)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:28596) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:28596) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
I tried increasing the bufferTimeoutMS or disabling the bufferCommands but still it won't work.
According to Documentation found in this link: https://mongoosejs.com/docs/connections.html#buffering
Mongoose lets you start using your models immediately, without waiting for mongoose to establish a connection to MongoDB.
That's because mongoose buffers model function calls internally. This
buffering is convenient, but also a common source of confusion.
Mongoose will not throw any errors by default if you use a model
without connecting.
TL;DR:
Your model is being called before the connection is established. You need to use async/await with connect() or createConnection(); or use .then(), as these functions return promises now from Mongoose 5.
The issue on model.find() error: Operation products.find() buffering timed out after 10000ms" was resolved by removing the node_module folder, *.json files and reinstalling the mongoose module.
The issue on the warnings was resolved by following this instructions https://mongoosejs.com/docs/deprecations.html
Well, I encountered the same problem and had very similar code. I got the same error when sending a get request while testing.
Eventually, I found the solution that my localhost DB wasn't running at that moment. Though it's a foolish error, but I had a hard time finding it.
This error poped becuase you are trying to access models before creating the connection with the database
Always link your mongodbconnection file (if you have created) in app.js by
var mongoose = require('./mongoconnection');
or just keep mongodb connection code in app.js
For me was 100% MongoDB Atlas issue.
I've created a cluster in Sao Paulo that for some reason wasn't working as expected. I've deleted it, create a new one in AWS / N. Virginia (us-east-1) and everything started working again.
i'm using this function to connect to the db and avoid some warnings
mongoose.connect(
url,
{ useNewUrlParser: true, useUnifiedTopology: true },
function (err, res) {
try {
console.log('Connected to Database');
} catch (err) {
throw err;
}
});
just use 127.0.0.1 instead of localhost
mongoose.connect('mongodb://127.0.0.1:27017/myapp');
Or use family:4 in mongoose.connect method like that
mongoose.connect('mongodb://localhost:27017/TESTdb', {
family:4
})
.then(() => {
console.log('FINE');
})
.catch(() => {
console.log("BAD");
})
I had the same problem.
After a long search I was able to find it.
I created a new user in MongoDB atlas settings. I changed the MongoDB connection value with the new user.
Changing DNS setting to 8.8.8.8 or changing mongodb connection settings to 2.2.12 did not work.
In my case my i forgot to import db.config file in server.js file
There has been a change in mongoose v5^ the spaghetti code has been refactored, It now returns a promise that resolves to the mongoose singleton. so you don't have to do this.
// You don't have todo this
mongoose.connect('mongodb://localhost:27017/test').connection.
on('error', handleErr).
model('Test', new Schema({ name: String }));
// You can now do this instead
mongoose.connect('mongodb://localhost:27017/test').catch(err);
Check here for references
What's new in Mongoose v5^
If this doesn't work for you, you can then change your connection URL > Select your driver and version to v2.2.12 or later
First you should check in which port mongodb currently running.
Use this command to check that port
sudo lsof -iTCP -sTCP:LISTEN | grep mongo
If there you find different port rather than 27017, you should change it
I was having this issue only on deployed lambda functions and everything worked fine on my local. The following worked for me.
Delete node_modules folder.
npm install
commit/push the new package-lock.json file
merge / run cicd pipeline / deploy.
For me, the issue was node version. I was getting the same error with nodejs version 17.
After trying all the suggestions on this thread, stumbled upon this open issue. Tried downgrading node, but that did not work, finally uninstalled node 17 completely and installed node 16 and the problem was solved!
You can check your node version on Mac using node --version
This means that, mongo connection has not been established like others have mentioned, go through your code and see if perhaps you forgot to create a mongoConnect() function to connect with your atlas URI
the best way is to put your initialization in a function, connect to db before starting the server. use a combination of async and a condition to check if environment variables are there(incase db url is in env) here is a sample code.
const start = async () => {
if (!process.env.DB_URI) {
throw new Error('auth DB_URI must be defined');
}
try {
await mongoose.connect(process.env.DB_URI!, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
});
console.log('Server connected to MongoDb!');
} catch (err) {
throw new DbConnectionError();
console.error(err);
}
const PORT = process.env.SERVER_PORT;
app.listen(PORT, () => {
console.log(`Server is listening on ${PORT}!!!!!!!!!`);
});
};
start();
You should check if string connection is correct, because in my case I forgot to include the .env file in my proyect. This file contains string connection for my server in digital ocean.
MONGO_URI="mongodb+srv://server:gfhyhfyh.mongo.ondigitalocean.com/db_customers"