I have a server on sails nodejs and I am trying to connect my controllers with my MySQL db through a wrapper file that would create the connection pool. My purpose is that I use that pool everytime a function in any controller needs to interact with DB, and in such a way that connection is created at the time interaction starts and connection is closed at the time interaction is over. For this, I have created a wrapper file db.js
db.js
var mysql = require('mysql');
var connection = mysql.createConnection({
host:"localhost",
port: '3306',
user:"ye_old_username",
password:"ye_old_password",
database: "ye_old_schema"
});
module.exports = connection;
Now, I am creating a connection pool called ConnectionPool.js
ConnectionPool.js
var mysql = require('mysql'),
config = require("./db");
/*
* #sqlConnection
* Creates the connection, makes the query and close it to avoid concurrency conflicts.
*/
var sqlConnection = function sqlConnection(sql, values, next) {
// It means that the values hasnt been passed
if (arguments.length === 2) {
next = values;
values = null;
}
var connection = mysql.createConnection(config);
connection.connect(function(err) {
if (err !== null) {
console.log("[MYSQL] Error connecting to mysql:" + err+'\n');
}
});
connection.query(sql, values, function(err) {
connection.end();
if (err) {
throw err;
}
next.apply(this, arguments);
});
}
module.exports = sqlConnection;
I have followed the method answered on this question to create the connection pool: How to provide a mysql database connection in single file in nodejs
And finally, I am trying to run a function from a controller using the wrapper and the connection pool. The code inside the Controller is
var connPool = require('./ConnectionPool');
module.exports = {
testConn: function(req, res){
connPool('SELECT * from user where ?', {id: '1'}, function(err, rows) {
if(err){
sails.log.debug(err);
}else{
console.log(rows);
}
});
}
};
All the three files, the wrapper, the connection pool, and the controller are in the same Controllers folder.
Now, when I send a request to the URL through my client, that would invoke the testConn function inside the controller, I get the following response on server log:
[MYSQL] Error connecting to mysql:Error: ER_ACCESS_DENIED_ERROR: Access denied for user ''#'localhost' (using password: NO)
This error is coming from the line connection.connect(function(err) { in connection pool file.
When I try to log on my MySQL db through the same credentials on command line, I am through it. Therefore I believe that db.js file has some format related issue because of which a proper connection is not getting initiated. There can be other reason as well, but the reason I suspect seems to be very strong.
I need some guidance on solving this issue. Any help will be appreciated.
Related
Goal: Do a simple query to the database.
Expected results: "please print something!" and the results from the query are printed on the terminal.
Actual results: Nothing is printed on the terminal.
Errors: No error message.
Here is the db.js file:
var mysql = require('mysql');
var connection = mysql.createConnection({
host: 'localhost',
user: 'root',
password: 'root',
database: 'todoDB'
});
connection.connect();
connection.query('SELECT * FROM categories', function (err, res, fields) {
console.log("please print something!")
if (err) throw err;
console.log(res);
});
connection.end();
I execute this file using:
node db.js
On the mysql cli, I am able to do this query without any problem with the database name, credentials, and query given above.
I know that connection.connect() works since when I'm inputting the code below, the terminal prints "Database is connected!" I think the problem occurs at connection.query, but I am not sure why.
connection.connect(function(err){
if(!err){
console.log("Database is connected");
} else {
console.log("Error while connecting with database");
console.log(err);
}
});
I've looked through all the related questions on stackoverflow and tried them, but none of the solutions seems to resolve the problem that I have.
So it looks like mySQL V8 uses caching_sha2_password as the new authentication standard, which the nodeJS plugin does not support. Connect to your db and try creating a new user that uses the native password auth type.
CREATE USER 'foo'#'localhost' IDENTIFIED WITH mysql_native_password BY 'bar';
This question already has answers here:
where to destroy knex connection
(3 answers)
Closed 7 months ago.
I'm confused on where to use knex.destroy() in my Node API.
If I don't use knex.destroy() after I open the connection to make a call, the connection pool fills up over time, leading to error:
Unhandled rejection TimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
If I close the connection, which makes sense to me, when I'm done with it,
router.get('/users', function(req, res, next) {
var select = knex.select('*').from('users');
select.then((result) => {
res.send(result);
}).catch((error) => {
res.send(error);
}).finally(function() {
knex.destroy(); // close it when I'm done
});
});
The connection is closed for separate API calls:
Unhandled rejection Error: Unable to acquire a connection
at Client_PG.acquireConnection (/var/app/current/node_modules/knex/lib/client.js:331:40)
So where and when do I actually destroy the connection? Again, this Node application simply serves as an API. Each API call should open, then close, the connection, but knex doesn't seem to like this.
Router files that require knex: (I do this for each router file)
const knexService = require('../knexService');
const bookshelf = knexService.bookshelf;
const knex = knexService.knex;
let User = require('../models/User');
module.exports = function(app, router) {
router.get('/users', function(req, res, next) {
var select = knex.select('*').from('users');
select.then((result) => {
res.send(result);
}).catch((error) => {
res.send(error);
}).finally(function() {
knex.destroy(); // close it when I'm done
});
});
...
UserModel file
const knexService = require('../knexService');
const bookshelf = knexService.bookshelf;
var BaseModel = require('./BaseModel');
var addressModel = require('./Address').Address;
var User = BaseModel.extend({
tableName: 'users',
hasTimestamps: true,
addresses: function() {
return this.hasMany(addressModel);
}
});
KnexService.js
const knexfile = require('./knexfile');
const knex = require('knex')(knexfile.production);
const bookshelf = require('bookshelf')(knex);
module.exports.knex = knex;
module.exports.bookshelf = bookshelf;
KnexFile.js
module.exports = {
development: {
client: 'pg',
version: '7.2',
connection: {
...
knex.destroy() should be called when you want to knex to throw away all connections from pool and stop all timers etc. so that application can end gracefully.
So basically that should be called only on application exit unless you are doing some more complex stuff for example with connecting to multiple databases with multiple knex instances.
If you are running out of connections and pool fills up it means that you have problems in your code. Possible reasons could be:
making too many long lasting queries concurrently
creating transactions and never committing / rolling back so those connections are never returned to the pool
bug in knex / bookshelf / some middleware
You should try to pinpoint which parts of your app causes pool to fill up, remove all the extra code like bookshelf related stuff and find the minimal setup which you can use to replicate your problem (also remove all transactions to start with).
Are you really using postgresql 7.2 or are you connecting some custom postgresql compatible DB? That could cause some issues, but I don't think those would reveal themselves in that way, but by having broken connections to be left in pool.
Let's say I have a service app.js. Whenever a new client connects to the service , we will check if a mongodb connection is already establish for this client or not.
If it is not available then we will fetch the server ip, dbname ,collection name from a configuration file, connect to the db,and reply to the user.
Note: we can add a new client and corresponding info to Client Info at any time. (dynamically)
Client Info ClientId: ServerIp : Database Name :Collection Name I have tried to store mongo object in array so I can reuse them object based on database name from user's session data. But I keep running into circular json error. How do I store multi tenant database connections?
async.eachSeries(conf.clientDbs.clientsList, function(clientDetails,callback){
console.log(clientDetails);
mongodb.MongoClient.connect(conf.clientDbs.connection+clientDetails.dbName, function (err, database) {
if (err) {
console.log(err);
process.exit(1);
}
// Save database object from the callback for reuse.
var tempdbobj = {};
tempdbobj["obj"] = database
allDbs[clientDetails.team_id] = tempdbobj;
console.log("Database connection ready for "+clientDetails.team_id);
allDbs[clientDetails.team_id].obj.collection('collection_name').find({"ref_id":"111"}, function(dberr, testDoc){
if (dberr) {
console.log(dberr);
callback();
}
else {
console.log(testDoc);
callback();
}
});
});
});
I'm having trouble getting the pg package working on my local system. I've tried to run the following:
var pg = require('pg');
var con_string = "postgres://user:password#localhost:5432/documentation";
var client = new pg.Client();
client.connect(con_string, function(err, res) {
// stuff here
});
But I keep getting TypeError: callback is not a function.
Is there a setting that I need to change in order to connect to the db via a connection string? I have tried the username and password that I'm using in user:password above on a database on my local machine and I can connect just fine.
I've also tried in the node shell in the directory of the project where I installed pg and haven't had any luck.
Thanks
This the error that I get from running the answer below:
$ node pg_test.js
error fetching client from pool { [error: password authentication failed for user "jake"]
Directly from documentation at https://github.com/brianc/node-postgres:
var pg = require('pg');
var conString = "postgres://user:password#localhost:5432/documentation";
//this initializes a connection pool
//it will keep idle connections open for a (configurable) 30 seconds
//and set a limit of 10 (also configurable)
pg.connect(conString, function(err, client, done) {
if(err) {
return console.error('error fetching client from pool', err);
}
client.query('SELECT $1::int AS number', ['1'], function(err, result) {
//call `done()` to release the client back to the pool
done();
if(err) {
return console.error('error running query', err);
}
console.log(result.rows[0].number);
//output: 1
});
});
Background Information
I'm attempting my first node.js API/application. As a learning exercise, I'm trying to create some test cases initially delete all records in a table, insert 3 specific records, and then query for those 3 records.
Code
Here's the code I have cobbled together:
http://pastebin.com/duQQu3fm
Problem
As you can see from the code, I'm trying to put the database connection logic in a dbSession.js file and pass it around.
I am able to start up the http server by doing the following:
dev#devbox:~/nimble_node$ sudo nodejs src/backend/index.js
Server started and listening on port: 8080
Database connection successful
However, when I try to run my jasmine tests, it fails with the following error:
F
Failures:
1) The API should respond to a GET request at /api/widgets/
Message:
TypeError: Object #<MongoClient> has no method 'collection'
Stacktrace:
TypeError: Object #<MongoClient> has no method 'collection'
at resetDatabase (/home/dev/nimble_node/spec/resetDatabase.js:6:29)
at /home/dev/nimble_node/spec/e2e/apiSpec.js:23:25
at /home/dev/nimble_node/node_modules/async/lib/async.js:683:13
at iterate (/home/dev/nimble_node/node_modules/async/lib/async.js:260:13)
at async.forEachOfSeries.async.eachOfSeries (/home/dev/nimble_node/node_modules/async/lib/async.js:279:9)
at _parallel (/home/dev/nimble_node/node_modules/async/lib/async.js:682:9)
at Object.async.series (/home/dev/nimble_node/node_modules/async/lib/async.js:704:9)
at null.<anonymous> (/home/dev/nimble_node/spec/e2e/apiSpec.js:19:9)
at null.<anonymous> (/home/dev/nimble_node/node_modules/jasmine-node/lib/jasmine-node/async-callback.js:45:37)
Finished in 0.01 seconds
1 test, 1 assertion, 1 failure, 0 skipped
Database connection successful
Line 6 of resetDatabase is:
var collection = dbSession.collection('widgets');
Given that after the error appears, I get the "Database connection successful" message, I think what's happening is that when the tests request the dbSession library, the database hasn't finished running the code to connect. And therefore, I can't get the collection object.
I'm currently reading through the mongodb online manual to see if I can find some hints as to how to do something like this.
Any suggestions or pointers would be appreciated.
EDIT 1
To prove that there is a collection method on the MongoClient object, I changed the dbSession.js code to look like this:
'use strict';
var DBWrapper = require('mongodb').MongoClient;
var dbWrapper = new DBWrapper;
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
var collection = dbWrapper.collection('widgets');
console.log('just created a collection...');
}
});
module.exports = dbWrapper;
And now, when I start up the http server (index.js), notice the messages:
dev#devbox:~/nimble_node$ sudo nodejs src/backend/index.js
Server started and listening on port: 8080
Database connection successful
just created a collection...
It could be an async issue.
Your code in dbSessionjs
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
}
});
module.exports = dbWrapper;
Starts the connection at dbWrapper asynchronously, but exports dbWrapper right away, which is then imported in resetDatabase. Thus yes, the connect function may have not yet returned from the async function when you call it in resetDatabase (and is what the log suggests,as the error appears before the success log).
You could add a callback after dbWrapper.connect() returns, in order to actually only be able to use dbWrapper when the connection finished.
(With sqlite, this may not happen as it accesses the DB faster on the commandline).
This may not be your problem but looks like a candidate.
EDIT: Here's a possible example for a callback, but please take note it depends on what you need to do so there are a lot of different solutions. The key is to call a callback function when you are done initializing.
Another solution could be to simply wait, and/or poll (e.g. chcke a variable 'initialized').
'use strict';
var DBWrapper = require('mongodb').MongoClient;
var dbWrapper = new DBWrapper;
function doConnect(callback) {
console.log("Initializing DB connection...");
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
var collection = dbWrapper.collection('widgets');
console.log('just created a collection...');
console.log('calling callback...');
callback(dbWrapper);
} else {
console.log("Error connectingi: " + err);
}
});
};
doConnect(function(correctDbWrapper) {
//Now you can use the wrapper
console.log("Inside callback, now consuming the dbWrapper");
dbWrapper = correctDbWrapper;
var collection = dbWrapper.collection('widgets');
});
It's interesting though I never ran into this issue, although I have generally used similar code like yours. I guess because normally I have this DB initialization right at the top, and then have to do lots of initializations on the node app, which gives the app time enough to return from the connect call....