Develop Reference

Why am I getting a timeout error when exporting a mariadb connection pool in Node.js?

EDIT
I found the error. The mistake was very obvious: I did not include the
require("dotenv").config(); in the connection.js file. Without this, the database connection simply fails after a timeout because it does not have any connection details.
I found an update log from the Mariadb Node.js connector team stating they have a few errors where Mariadb does not provide sufficient error messages (it sometimes only offers a "timeout" without further information), so I changed what I was looking for, and found the mistake.
For anyone getting a similar error message, this can mean anything, so check all parts of your code!
Original Post
I am trying to get familiar with Nodejs and express, but ran into an issue that I can't seem to solve:
When creating a Mariadb database pool in a seperate file, and exporting the pool using module.exports, I am having trouble using the same pool in another file. I get a timeout error when trying to use the pool to query a database.
If I use the exact same code in the same file instead of two separate files, the query works perfectly, so I think there is something going wrong during module.exports = pool.
Am I missing something? Thanks in advance!
I have two files:
index.js:
// import express web framework
const express = require("express");
//create an express application
const app = express();
const pool = require('./database/connection')
const cors = require('cors');
//middleware
app.use(cors())
app.use(express.json())
getData = async () => {
data = await pool.query("call stored_procedure")
console.log (data)
}
getData()
app.listen(3001, () => {
console.log('Serving running on port 3001')
})
and connection.js:
//import mariadb library
const mariadb = require("mariadb")
//function that create mariadb connection pool for database
const createPool = () => {
try {
return (
mariadb.createPool({
connectionLimit: 10,
host: process.env.MARIADB_HOST,
user: process.env.MARIADB_USER,
password: process.env.MARIADB_PASSWORD,
database: process.env.MARIADB_DB,
port: 3306
})
)
}
catch (err) {
console.error('Failed to connect to database: ')
console.error(err)
}
}
const pool = createPool()
//export database connection pool
module.exports = pool
Running this app results in the following error (after some time):
path_to_dir/node_modules/mariadb/lib/misc/errors.js:57
return new SqlError(msg, sql, fatal, info, sqlState, errno, additionalStack, addHeader);
^
SqlError: (conn=-1, no: 45028, SQLState: HY000) retrieve connection from pool timeout after 10001ms
(pool connections: active=0 idle=0 limit=10)
at Object.module.exports.createError (path_to_dir/node_modules/mariadb/lib/misc/errors.js:57:10)
at Pool._requestTimeoutHandler (path_to_dir/node_modules/mariadb/lib/pool.js:345:26)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7) {
text: 'retrieve connection from pool timeout after 10001ms\n' +
' (pool connections: active=0 idle=0 limit=10)',
sql: null,
fatal: false,
errno: 45028,
sqlState: 'HY000',
code: 'ER_GET_CONNECTION_TIMEOUT'
}

I found the error. The mistake was very obvious: I did not include the require("dotenv").config(); in the connection.js file. Without this, the database connection simply fails after a timeout because it does not have any connection details. I found an update log from the Mariadb Node.js connector team stating they have a few errors where Mariadb does not provide sufficient error messages (it sometimes only offers a "timeout" without further information), so I changed what I was looking for, and found the mistake.
For anyone getting a similar error message, this can mean anything, so check all parts of your code!

Unable to make request to my mariadb database using node server

I'm currently learning how to setup a node server and I'm making an API that performs some requests on my MariaDB database hosted on my VPS.
The problem is that when I make a POST request which makes a SQL request to the database, the connection times out and the server shuts down.
I have tried to add new users to MariaDB with all privileges, I tried use sequelize too.
But none of those solutions work, it still times out every time I make a query to my database.
I can connect to phpmyadmin and make some request on it, so I think that my database is running fine.
Here is my code:
router.post('/login', async function(req,res) {
let conn;
try {
// establish a connection to MariaDB
conn = await pool.getConnection();
// create a new query
var query = "select * from people";
// execute the query and set the result to a new variable
var rows = await conn.query(query);
// return the results
res.send(rows);
} catch (err) {
throw err;
} finally {
if (conn) return conn.release();
}
})
The way I connect to my database in my database.js file
const pool = mariadb.createPool({
host: process.env.DATABASE_HOST,
user: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
database: process.env.DATABSE_NAME,
});
// Connect and check for errors
module.exports={
getConnection: function(){
return new Promise(function(resolve,reject){
pool.getConnection().then(function(connection){
resolve(connection);
}).catch(function(error){
reject(error);
});
});
}
}
module.exports = pool;
And my error:
Node.js v17.0.1
[nodemon] app crashed - waiting for file changes before starting...
[nodemon] restarting due to changes...
[nodemon] starting `node server.js`
Server started
/Users/alexlbr/WebstormProjects/AlloEirb/server/node_modules/mariadb/lib/misc/errors.js:61
return new SqlError(msg, sql, fatal, info, sqlState, errno, additionalStack, addHeader);
^
SqlError: retrieve connection from pool timeout after 10001ms
at Object.module.exports.createError (/Users/alexlbr/WebstormProjects/AlloEirb/server/node_modules/mariadb/lib/misc/errors.js:61:10)
at timeoutTask (/Users/alexlbr/WebstormProjects/AlloEirb/server/node_modules/mariadb/lib/pool-base.js:319:16)
at Timeout.rejectAndResetTimeout [as _onTimeout] (/Users/alexlbr/WebstormProjects/AlloEirb/server/node_modules/mariadb/lib/pool-base.js:342:5)
at listOnTimeout (node:internal/timers:559:11)
at processTimers (node:internal/timers:500:7) {
text: 'retrieve connection from pool timeout after 10001ms',```

Three possibilities come to mind:
There is a typo in database name:
database: process.env.DATABSE_NAME
database: process.env.DATABASE_NAME
Your environment variables are not being properly set. Are you using dotenv to load these from an .env file?
https://www.npmjs.com/package/dotenv
If not, how are you setting the process.env values at runtime?
If the environment values are indeed set:
verify that these environment values are correct
verify which interface your MariaDB server is listening on:
It's possible the server is using a bind-address configuration and only listening on 127.0.0.1 (which is the default on Debian/Ubuntu)
You want to make sure the server is listening on: 0.0.0.0 (all interfaces, not only localhost)

How to solve Mongoose v5.11.0 model.find() error: Operation `products.find()` buffering timed out after 10000ms"

How to solve model.find() function produces "buffering timed out after ... ms"? I'm using mongoose v 5.11.0, npm v6.14.8 and mongodb v
Here's the code.
var express = require('express');
var app = express();
var bodyParser = require('body-parser');
const assert = require('assert');
var mongoose = require('mongoose');
try {
var db = mongoose.connect('mongodb://localhost:27017', {useNewUrlParser: true, dbName: 'swag-shop' });
console.log('success connection');
}
catch (error) {
console.log('Error connection: ' + error);
}
var Product = require('./model/product');
var WishList = require('./model/wishlist');
//Allow all requests from all domains & localhost
app.all('/*', function(req, res, next) {
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "X-Requested-With, Content-Type, Accept");
res.header("Access-Control-Allow-Methods", "POST, GET");
next();
});
app.get('/product', function(request, response) {
Product.find({},function(err, products) {
if (err) {
response.status(500).send({error: "Could not fetch products. "+ err});
} else {
response.send(products);
}
});
});
app.listen(3004, function() {
console.log("Swag Shop API running on port 3004...");
});
The product model:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var product = new Schema({
title: String,
price: Number,
likes: {type: Number, default: 0}
});
module.exports = mongoose.model('Product', product);
Additionally, running the file also produces the following warnings:
D:\Test\swag-shop-api>nodemon server.js
[nodemon] 2.0.6
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node server.js`
success connection
Swag Shop API running on port 3004...
(node:28596) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_ARG_TYPE]: The "url" argument must be of type string. Received type function ([Function (anonymous)])
at validateString (internal/validators.js:122:11)
at Url.parse (url.js:159:3)
at Object.urlParse [as parse] (url.js:154:13)
at module.exports (D:\Test\swag-shop-api\node_modules\mongoose\node_modules\mongodb\lib\url_parser.js:15:23)
at connect (D:\Test\swag-shop-api\node_modules\mongoose\node_modules\mongodb\lib\mongo_client.js:403:16)
at D:\Test\swag-shop-api\node_modules\mongoose\node_modules\mongodb\lib\mongo_client.js:217:7
at new Promise (<anonymous>)
at MongoClient.connect (D:\Test\swag-shop-api\node_modules\mongoose\node_modules\mongodb\lib\mongo_client.js:213:12)
at D:\Test\swag-shop-api\node_modules\mongoose\lib\connection.js:820:12
at new Promise (<anonymous>)
at NativeConnection.Connection.openUri (D:\Test\swag-shop-api\node_modules\mongoose\lib\connection.js:817:19)
at D:\Test\swag-shop-api\node_modules\mongoose\lib\index.js:345:10
at D:\Test\swag-shop-api\node_modules\mongoose\lib\helpers\promiseOrCallback.js:31:5
at new Promise (<anonymous>)
at promiseOrCallback (D:\Test\swag-shop-api\node_modules\mongoose\lib\helpers\promiseOrCallback.js:30:10)
at Mongoose._promiseOrCallback (D:\Test\swag-shop-api\node_modules\mongoose\lib\index.js:1135:10)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:28596) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:28596) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
I tried increasing the bufferTimeoutMS or disabling the bufferCommands but still it won't work.

According to Documentation found in this link: https://mongoosejs.com/docs/connections.html#buffering
Mongoose lets you start using your models immediately, without waiting for mongoose to establish a connection to MongoDB.
That's because mongoose buffers model function calls internally. This
buffering is convenient, but also a common source of confusion.
Mongoose will not throw any errors by default if you use a model
without connecting.
TL;DR:
Your model is being called before the connection is established. You need to use async/await with connect() or createConnection(); or use .then(), as these functions return promises now from Mongoose 5.

The issue on model.find() error: Operation products.find() buffering timed out after 10000ms" was resolved by removing the node_module folder, *.json files and reinstalling the mongoose module.
The issue on the warnings was resolved by following this instructions https://mongoosejs.com/docs/deprecations.html

Well, I encountered the same problem and had very similar code. I got the same error when sending a get request while testing.
Eventually, I found the solution that my localhost DB wasn't running at that moment. Though it's a foolish error, but I had a hard time finding it.

This error poped becuase you are trying to access models before creating the connection with the database
Always link your mongodbconnection file (if you have created) in app.js by
var mongoose = require('./mongoconnection');
or just keep mongodb connection code in app.js

For me was 100% MongoDB Atlas issue.
I've created a cluster in Sao Paulo that for some reason wasn't working as expected. I've deleted it, create a new one in AWS / N. Virginia (us-east-1) and everything started working again.
i'm using this function to connect to the db and avoid some warnings
mongoose.connect(
url,
{ useNewUrlParser: true, useUnifiedTopology: true },
function (err, res) {
try {
console.log('Connected to Database');
} catch (err) {
throw err;
}
});

just use 127.0.0.1 instead of localhost
mongoose.connect('mongodb://127.0.0.1:27017/myapp');
Or use family:4 in mongoose.connect method like that
mongoose.connect('mongodb://localhost:27017/TESTdb', {
family:4
})
.then(() => {
console.log('FINE');
})
.catch(() => {
console.log("BAD");
})

I had the same problem.
After a long search I was able to find it.
I created a new user in MongoDB atlas settings. I changed the MongoDB connection value with the new user.
Changing DNS setting to 8.8.8.8 or changing mongodb connection settings to 2.2.12 did not work.

In my case my i forgot to import db.config file in server.js file

There has been a change in mongoose v5^ the spaghetti code has been refactored, It now returns a promise that resolves to the mongoose singleton. so you don't have to do this.
// You don't have todo this
mongoose.connect('mongodb://localhost:27017/test').connection.
on('error', handleErr).
model('Test', new Schema({ name: String }));
// You can now do this instead
mongoose.connect('mongodb://localhost:27017/test').catch(err);
Check here for references
What's new in Mongoose v5^
If this doesn't work for you, you can then change your connection URL > Select your driver and version to v2.2.12 or later

First you should check in which port mongodb currently running.
Use this command to check that port
sudo lsof -iTCP -sTCP:LISTEN | grep mongo
If there you find different port rather than 27017, you should change it

I was having this issue only on deployed lambda functions and everything worked fine on my local. The following worked for me.
Delete node_modules folder.
npm install
commit/push the new package-lock.json file
merge / run cicd pipeline / deploy.

For me, the issue was node version. I was getting the same error with nodejs version 17.
After trying all the suggestions on this thread, stumbled upon this open issue. Tried downgrading node, but that did not work, finally uninstalled node 17 completely and installed node 16 and the problem was solved!
You can check your node version on Mac using node --version

This means that, mongo connection has not been established like others have mentioned, go through your code and see if perhaps you forgot to create a mongoConnect() function to connect with your atlas URI

the best way is to put your initialization in a function, connect to db before starting the server. use a combination of async and a condition to check if environment variables are there(incase db url is in env) here is a sample code.
const start = async () => {
if (!process.env.DB_URI) {
throw new Error('auth DB_URI must be defined');
}
try {
await mongoose.connect(process.env.DB_URI!, {
useNewUrlParser: true,
useUnifiedTopology: true,
useCreateIndex: true,
});
console.log('Server connected to MongoDb!');
} catch (err) {
throw new DbConnectionError();
console.error(err);
}
const PORT = process.env.SERVER_PORT;
app.listen(PORT, () => {
console.log(`Server is listening on ${PORT}!!!!!!!!!`);
});
};
start();

You should check if string connection is correct, because in my case I forgot to include the .env file in my proyect. This file contains string connection for my server in digital ocean.
MONGO_URI="mongodb+srv://server:gfhyhfyh.mongo.ondigitalocean.com/db_customers"

Node Js mysql(and mysql2) ECONNRESET

i am currently trying to connect to a MySQL server on the internet using Node.Js with the mysql or the mysql2 NPM dependencies to use queries and other related stuff.
the code is simple...
//i import my dependency
const mysql = require('mysql2') //either 'mysql' or 'mysql2'
//i create my pool to create connections as needed
var conn = mysql.createPool({
host: 'some_database_i_have_access_to.mysql.uhserver.com',
user: 'valid_user',
password: 'valid_password',
database: 'some_existing_database'
})
//i try to connect (this is the part where it fails)
conn.getConnection((err,conn) => {
if (err) throw err //<- the error is thrown here
//i do query stuff
conn.query("SELECT * FROM atable",(err,res,firlds) => {
if(err) throw err
console.log(JSON.stringify(res))
})
//i close the connection
conn.end()
})
yet i always get an Error like this:
Error: read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
--------------------
at Protocol._enqueue (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\protocol\Protocol.js:144:48)
at Protocol.handshake (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\protocol\Protocol.js:51:23)
at Connection.connect (C:\Users\Aluno\Desktop\my-project\node_modules\mysql\lib\Connection.js:118:18)
at Object.<anonymous> (C:\Users\Aluno\Desktop\my-project\private\dtp-mysql.js:13:6)
at Module._compile (internal/modules/cjs/loader.js:707:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:718:10)
at Module.load (internal/modules/cjs/loader.js:605:32)
at tryModuleLoad (internal/modules/cjs/loader.js:544:12)
at Function.Module._load (internal/modules/cjs/loader.js:536:3)
at Module.require (internal/modules/cjs/loader.js:643:17)
all i know about the error is that the connection abruptly closes in one of the sides as stated in this question (Node js ECONNRESET), but nothing more, and creating singular connections does not solve this issue for me either.
any fixes to that?

you can also ref below url.
error while inserting LARGE volume data in mysql by using node.js （error code: 'ECONNRESET'）
I have fixed this issue. It is caused by the default definition max_allowed_packet. Find max_allowed_packet in my.ini (C:\ProgramData\MySQL\MySQL Server 5.7). Update to 'max_allowed_packet=64M'. Restart mysql. Done.

Hi I know this was asked some time ago, but is it possibly because you're using:
conn.end()
Since you're using a Pooled connection, I think you can release the connection using
conn.release()
Or
conn.destroy()

Redis connection is lost after multiple calls to function

The program I am writing is a status display screen for alarms, each of which is represented by a channel.
When the server is started (run on a vagrant virtual machine), an Influx database is accessed, the data (comprising of 1574 'channels') is processed and put into a Redis database. This runs fine and the GUI is displayed with no issues when the webpage is refreshed, although it takes a long time to load (up to 20s), and nearly all of this time is spent in the method below.
However, after a few refreshes/moving around the site, it often crashes with the following error:
{ AbortError: Redis connection lost and command aborted. It might
have been processed.
at RedisClient.flush_and_error (/vagrant/node_modules/redis/index.js:362:23)
at RedisClient.connection_gone (/vagrant/node_modules/redis/index.js:664:14)
at RedisClient.on_error (/vagrant/node_modules/redis/index.js:410:10)
at Socket. (/vagrant/node_modules/redis/index.js:279:14)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at onwriteError (_stream_writable.js:417:12)
at onwrite (_stream_writable.js:439:5)
at _destroy (internal/streams/destroy.js:39:7)
at Socket._destroy (net.js:568:3) code: 'UNCERTAIN_STATE', command: 'HGETALL', args: [
'vista:hash:Result:44f59707-c873-11e8-93b9-7f551d0bdd1f' ], origin:
{ Error: Redis connection to 127.0.0.1:6379 failed - write EPIPE
at WriteWrap.afterWrite (net.js:868:14) errno: 'EPIPE', code: 'EPIPE', syscall: 'write' } }
This error is displayed 1574 times (once for each channel), and occurs when the program reaches this function:
Result.getFormattedResults = async function (cycle) {
const channels = await Channel.findAndLoad()
const formattedResults = await mapAsyncParallel(channels, async channel => {
const result = await this.findAndLoadByChannel(channel, cycle)
const formattedResult = await result.format(channel)
return formattedResult
})
return formattedResults
}
mapAsyncParallel() is as follows:
export const mapAsyncParallel = (arr, fn, thisArg) => {
return Promise.all(arr.map(fn, thisArg))
}
findAndLoadByChannel() finds the channel and loads it with this line:
const resultModel = await this.load(resultId)
And format() takes the model and outputs the data as in a JSON format
There are two 'fetch(...)' commands (which are needed and cannot be combined) in the front end, and the problem rarely occurs when I comment out one of them (either one). This is making me think it could be a max memory or max connections problem? (increasing maxmemory in the config file didn't help). Or a problem with using so many promises (a concept I am fairly new to).
This has only started to occur as I have added more functionality and I assume the function needs optimizing but I have taken over this project from someone else and am still quite new to node.js and redis.
Versions:
Vagrant: 2.0.1
Ubuntu: 16.04.5
Redis: 4.0.9
Node: 8.12.0
npm: 5.7.1

I've now moved all the 'getting' of the data (from redis) to the server side channels.controller file.
So, where before I would have:
renderPage: async (req, res) => {
res.render('page')
},
I now have a method like:
renderPage: async (req, res) => {
const data1 = getData1()
const data2 = getData2()
res.render('page', {data1, data2})
},
(Don't worry, these aren't my actual variable names)
Where the two 'data' variables were previously retrieved using the 'fetch' method.
I export the data once it's loaded into redis, and import it in the controller file, where I have the getters to combine it all into one return array.
The pages now take milliseconds to refresh and I haven't had any crashes