im not that experienced with node js but im developing something similar to how uber displays their cars in real time on a map.
So i have an sql database with a ton of cars and their gps location. The client sends their gps coordinates and a radius to the following function. some is in pseudo code for now.
var mysql = require('mysql');
var express = require('express');
var connection = mysql.createConnection({
host: "",
port: ,
user: "",
password: "",
database: ""
});
user.on('returnCars', function(gps, radius){
connection.query({
sql: "SELECT * FROM cars WHERE radius = ?",
values: [username] },
function(error, results, fields)
{
if(results)
{
user.emit('returnCars', results);
}
}
});
});
});
So as sql querys arnt instant, if there was 1000 people running this function at once it would surely clog up. All my research is telling me that this is the right way to do it so the only option would for it to be ran asnync right?
Also would it just be the returnCars function to run asynchronously? Im not sure if because the connection/ sql details variable isnt in a function or anything it would all try and read it at once so maybe it should go inside the function or something.
The code is far too fragmentary to really help you with, but in general:
If you're using Node's built-in HTTP serving or something layered on top of it like Express, your code for when a request is received is expected to run asynchronously and there's nothing special you need to do to make that happen.
If you're using any of the main npm modules for MySQL access, the functions you use to make queries will run asynchronously and there's nothing special you have to do to make that happen.
In your pseudocode example, you've shown a callback for your sqlQuery function stand-in. That's likely how you would use the MySQL access module you choose to use, either with direct callbacks like that or promises.
Related
I have a question regarding SQL connection pools. My team is using the knex.js library in one of our node applications to make database query's.
The application from time to time needs to switch databases. So my team created an initialization function that returns a knex object configured to the correct database. Then that object is used to do said query. To me this seems redundant and can cause bad performance, because we initiate a knex object every time need to do a query instead of reusing a single knex object. Which i could ignore if knex already does this when you which databases (and if anyone could shed light on this question as well that would be FANTASTIC !) . Moreover, (and this leads me to my question titled above) the connection pool properties are redefined. So does that mean we are creating new pools every time, or does the SQL ( SQL Sever in this case) reuse the connection pool you already defined ? The question might not be Knex specific, like if i used a library like knex for C#, and call that library a similar way, would SQL Server know not to make more connection pools?
Example code:
/** db.js
* #param {any} database
* #returns db: Knex
*/
module.exports = ( database ) => {
var knex = require('knex')({
client: 'mssql',
connection: {
database: database,
server: '127.0.0.1',
user: 'your_database_user',
password: 'your_database_password'
},
pool: {
min: 0,
max: 10,
idleTimeoutMillis: 5000,
softIdleTimeoutMillis: 2000,
evictionRunIntervalMillis: 500
}
});
return knex;
};
Index.js
var db = require('./db.js');
/**
* #returns users:Array
*/
const getUsers = async() => {
const users = await db('master')
.select()
.from('users_table')
.orderBy('user_id');
return users;
}
Short answer: The 'singleton' nature of the node require() statement prevents reinitialization of multiple occurrences of knex. So the initially created pool continues to be used for the duration of your process, not recreated, as long as you don't discard the db. variable reference.
More discussion...
... my team created an initialization function that returns a knex
object configured to the correct database. Then that object is used to
do said query. To me this seems redundant and can cause bad
performance, because we initiate a knex object every time need to do a
query instead of reusing a single knex object. Which i could ignore if
knex already does this when you switch databases...
var db = require('./db.js');
The node.js require statement creates a singleton object. (You probably already know) this means that the first time the module is called by your program using the require statement, the module and it's data will be initialized, but successive identical require calls will just reuse the same module reference and will not reinitialize the module.
... the connection pool properties are redefined. So does that mean
we are creating new pools every time, or does the SQL ( SQL Sever
in this case) reuse the connection pool you already defined ?
So since the require()-ed module is not reinitialized, then the originally created pool will not be re-created. Unless you discard the db variable reference (discussed more below).
The question might not be Knex specific, like if i used a library like
knex for C#, and call that library a similar way, would SQL Server
know not to make more connection pools?
Generally speaking, you need to build or acquire connection some code to properly manage a pool of connections throughout the life of your process. Knex and most other database wrappers do this for us. (Under the covers Knex uses this library before v0.18.3 and this one on/after.)
Properly initializing and then using the singly initialized pooling code throughout the life of your application process accomplishes this. Discarding the pool and recreating it within your process defeats the purpose of having pooling. Often pooling is setup as part of process initialization.
Also, this was probably just a misstatement within your question, but your Node.js module is making the connection pools, not the SQL Server.
... The application from time to time needs to switch databases. my
team created an initialization function that returns a knex object
configured to the correct database.
From that statement, I would expect to see code like the following:
var db = require('./db.js');
var dbOther = require('./dbOther.js');
... which each establishes a different database connection. If you are instead using:
var db = require('./db.js');
// ... do other stuff here in the same module ...
var db = require('./dbOther.js');
... then you are likely throwing away the original reference to your first database, and in that case, YES, you are discarding your DB connection and connection pool as you switch connections.
Or, you could do something like the following:
// initialize the 2 connection pools
const dbFirst = require('./db.js');
const dbOther = require('./dbOther.js');
// set the active connection
var db = dbFirst;
// change the active connection
db = dbOther;
Right now i am creating a very large application in Node JS. I am trying to make my code clean and short (Just like most of the developer). I've create my own js file to handle connection to mysql. Please see code below.
var mysql = require('mysql');
var config = {
'default' : {
connectionLimit : process.env.DB_CONN_LIMIT,
host : process.env.DB_HOST,
user : process.env.DB_USER,
password : process.env.DB_PASS,
database : process.env.DB_NAME,
debug : false,
socketPath : process.env.DB_SOCKET
}
};
function connectionFunc(query,parameters,callback,configName) {
configName = configName || "default";
callback = callback || null;
parameters = parameters;
if(typeof parameters == 'function'){
callback = parameters;
parameters = [];
}
//console.log("Server is starting to connect to "+configName+" configuration");
var dbConnection = mysql.createConnection(config[configName]);
dbConnection.connect();
dbConnection.query(query,parameters, function(err, rows, fields) {
//if (!err)
callback(err,rows,fields);
//else
//console.log('Error while performing Query.');
});
dbConnection.end();
}
module.exports.query = connectionFunc;
I am using the above file in my models, like below :
var database = require('../../config/database.js');
module.exports.getData = function(successCallBack){
database.query('SAMPLE QUERY GOES HERE', function(err, result){
if(err) {console.log(err)}
//My statements here
});
}
Using this coding style, everything works fine but when i am trying to create a function that will loop my model's method for some reason. Please see sample below :
for (i = 0; i < 10000; i++) {
myModel.getData(param, function(result){
return res.json({data : result });
});
}
It gives me an ER_CON_COUNT_ERROR : Too Many Conenction. The question is why i still get an error like these when my connection always been ended by this dbConnection.end();? I'm still not sure if i am missing something. I am still stuck on this.
My connection limit is 100 and i think adding more connection is a bad idea.
Because query data form the database is async.
In your loop the myModel.getData (or more precisely the underling query) will not halt/paus your code until the query is finished, but send the query to the database server and as soon as the database response the callback will be called.
The calling end on dbConnection will not close the connection immediately, it will just mark the connection to be closed as soon as all queries that where created with that connection are finished.
mysql: Terminating connections
Terminating a connection gracefully is done by calling the end() method. This will make sure all previously enqueued queries are still before sending a COM_QUIT packet to the MySQL server.
An alternative way to end the connection is to call the destroy() method. This will cause an immediate termination of the underlying socket. Additionally destroy() guarantees that no more events or callbacks will be triggered for the connection.
But with destroy the library will not wait for the result so the results are lost, destroy is rarely useful.
So with your given code you try to create 10000 connections at one time.
You should only use on connection by task, e.g. if a user requests data using the browser, then you should use one connection for this given request. The same is for timed task, if you have some task that is done in certain intervals.
Here an example code:
var database = require('./config/database.js');
function someTask( callback ) {
var conn = database.getConnection();
myModel.getData(conn, paramsA, dataReceivedA)
function dataReceivedA(err, data) {
myModel.getData(conn, paramsB, dataReceivedB)
}
function dataReceivedB(err, data) {
conn.end()
callback();
}
}
If you want to entirely hide your database connection in your model code. Then you would need to doe something like that:
var conn = myModel.connect();
conn.getData(params, function(err, data) {
conn.end();
})
How to actually solve this depends only many factors so it is only possible to give you hints here.
I'm writing a Meteor app that adds jobs to a queue that are to be processed by one or more external workers. I'm using vsivsi/meteor-job-collection (https://github.com/vsivsi/meteor-job-collection) and everything worked fine when I was (initially) processing jobs using method-based (i.e., non external) workers from within the Meteor app itself (which queues, creates and manages the jobs).
However, I want an external process, i.e., a Node.js program (not even a Meteor app) to listen for and subsequently process posted jobs instead and saw that this is indeed possible at https://github.com/vsivsi/meteor-job-collection#quick-example (third code block; "node.js Worker") - after all, this enables a more generic and Meteor-agnostic job-processing approach.
I created a separate Node.js program, successfully connected it to my running Meteor app using dpp, ddp-login and meteor-job and started the job listener in exactly the same fashion as I originally did within the Meteor app (when using method-based workers), but I get the following inexplicable error (only) when calling job.done() (other job methods seem to work flawlessly):
"Exception while invoking method 'jobs_jobDone' Error: Match error: Failed Match.OneOf or Match.Optional validation"
The code:
var DDP = require('ddp');
var DDPlogin = require('ddp-login');
var Job = require('meteor-job');
var ddp = new DDP({
host: "localhost",
port: 3000,
use_ejson: true
});
Job.setDDP(ddp);
ddp.connect(connectCB);
function connectCB(err) {
if (err)
throw err;
var options = { //Needed to be able to log in; METEOR_TOKEN is set as an environment variable and this works fine.
env: 'METEOR_TOKEN',
method: 'account',
account: null,
pass: null,
retry: 5,
plaintext: false
};
DDPlogin(ddp, options, ddpLoginCB);
}
function ddpLoginCB(err) {
if (err)
throw err;
Job.processJobs('jobs', 'dataGatherJob', {
concurrency: 8,
pollInterval: 250
}, processJobsCB);
}
function processJobsCB(job, cb) {
//This is where the actual work gets done; this callback works fine,
//is triggered when my Meteor app adds a new job, and the job object
//conforms to my expectations; just some example calls on/usages of the
//job object that work fine and are reactively updated and handled by my
//Meteor app:
//console.log(job); //Works fine.
//job.progress(50, 100); //Works fine.
//job.fail(); //Works fine.
job.done(); //Causes the Meteor-app (server) to generate an "Exception while invoking method 'jobs_jobDone' Error: Match error: Failed Match.OneOf or Match.Optional validation" error.
cb();
}
I do not understand why this is happening. The job id is identical to the one in the actual Mongo collection (the queue uses a Mongo collection to manage jobs) set by the server (I checked this) and besides that, other job operations (e.g., job.fail()) do work. I furthermore reset the "autopublish" and "insecure" packages for my Meteor app intermittently just to be sure that they are not causing the problem and this was not the case.
Any thoughts on this?
Many thanks in advance,
Danny.
Background Information
I'm attempting my first node.js API/application. As a learning exercise, I'm trying to create some test cases initially delete all records in a table, insert 3 specific records, and then query for those 3 records.
Code
Here's the code I have cobbled together:
http://pastebin.com/duQQu3fm
Problem
As you can see from the code, I'm trying to put the database connection logic in a dbSession.js file and pass it around.
I am able to start up the http server by doing the following:
dev#devbox:~/nimble_node$ sudo nodejs src/backend/index.js
Server started and listening on port: 8080
Database connection successful
However, when I try to run my jasmine tests, it fails with the following error:
F
Failures:
1) The API should respond to a GET request at /api/widgets/
Message:
TypeError: Object #<MongoClient> has no method 'collection'
Stacktrace:
TypeError: Object #<MongoClient> has no method 'collection'
at resetDatabase (/home/dev/nimble_node/spec/resetDatabase.js:6:29)
at /home/dev/nimble_node/spec/e2e/apiSpec.js:23:25
at /home/dev/nimble_node/node_modules/async/lib/async.js:683:13
at iterate (/home/dev/nimble_node/node_modules/async/lib/async.js:260:13)
at async.forEachOfSeries.async.eachOfSeries (/home/dev/nimble_node/node_modules/async/lib/async.js:279:9)
at _parallel (/home/dev/nimble_node/node_modules/async/lib/async.js:682:9)
at Object.async.series (/home/dev/nimble_node/node_modules/async/lib/async.js:704:9)
at null.<anonymous> (/home/dev/nimble_node/spec/e2e/apiSpec.js:19:9)
at null.<anonymous> (/home/dev/nimble_node/node_modules/jasmine-node/lib/jasmine-node/async-callback.js:45:37)
Finished in 0.01 seconds
1 test, 1 assertion, 1 failure, 0 skipped
Database connection successful
Line 6 of resetDatabase is:
var collection = dbSession.collection('widgets');
Given that after the error appears, I get the "Database connection successful" message, I think what's happening is that when the tests request the dbSession library, the database hasn't finished running the code to connect. And therefore, I can't get the collection object.
I'm currently reading through the mongodb online manual to see if I can find some hints as to how to do something like this.
Any suggestions or pointers would be appreciated.
EDIT 1
To prove that there is a collection method on the MongoClient object, I changed the dbSession.js code to look like this:
'use strict';
var DBWrapper = require('mongodb').MongoClient;
var dbWrapper = new DBWrapper;
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
var collection = dbWrapper.collection('widgets');
console.log('just created a collection...');
}
});
module.exports = dbWrapper;
And now, when I start up the http server (index.js), notice the messages:
dev#devbox:~/nimble_node$ sudo nodejs src/backend/index.js
Server started and listening on port: 8080
Database connection successful
just created a collection...
It could be an async issue.
Your code in dbSessionjs
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
}
});
module.exports = dbWrapper;
Starts the connection at dbWrapper asynchronously, but exports dbWrapper right away, which is then imported in resetDatabase. Thus yes, the connect function may have not yet returned from the async function when you call it in resetDatabase (and is what the log suggests,as the error appears before the success log).
You could add a callback after dbWrapper.connect() returns, in order to actually only be able to use dbWrapper when the connection finished.
(With sqlite, this may not happen as it accesses the DB faster on the commandline).
This may not be your problem but looks like a candidate.
EDIT: Here's a possible example for a callback, but please take note it depends on what you need to do so there are a lot of different solutions. The key is to call a callback function when you are done initializing.
Another solution could be to simply wait, and/or poll (e.g. chcke a variable 'initialized').
'use strict';
var DBWrapper = require('mongodb').MongoClient;
var dbWrapper = new DBWrapper;
function doConnect(callback) {
console.log("Initializing DB connection...");
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
var collection = dbWrapper.collection('widgets');
console.log('just created a collection...');
console.log('calling callback...');
callback(dbWrapper);
} else {
console.log("Error connectingi: " + err);
}
});
};
doConnect(function(correctDbWrapper) {
//Now you can use the wrapper
console.log("Inside callback, now consuming the dbWrapper");
dbWrapper = correctDbWrapper;
var collection = dbWrapper.collection('widgets');
});
It's interesting though I never ran into this issue, although I have generally used similar code like yours. I guess because normally I have this DB initialization right at the top, and then have to do lots of initializations on the node app, which gives the app time enough to return from the connect call....
Hello I noticed some strange behaviour in node.js with felix geisendörfers awesome mysql module.
I have the following structure in my express app.
app.js (main)
routesA.js
routesB.js
routesC.js
The routes use the Router function of express.
Every routing file (A-C) has to access the mysql server.
But I was to lazy to write a connection and the connection options in every routing script file.
So I made another file called DBServer.js It is as follows (inspired by some hints from someone here in stackoverflow):
var mysql = require('mysql');
exports.connect = function (){
var db_config = {
host: '127.0.0.1',
user: 'my username',
password: '********',
database: 'my database'
};
var connection;
function autoConnect() {
connection = mysql.createConnection(db_config);
connection.connect(function(err) {
if(err) {
console.log('DBServer Error: cannot connect to db. Reconnect attempt in 2 seconds...\nError: ', err);
setTimeout(autoConnect, 2000);
}
else{
console.log('DBServer connected successfully...');
}
});
connection.on('error', function(err) {
if(err.code === 'PROTOCOL_CONNECTION_LOST') {
console.log('DBServer Error: lost connection. Reconnect attempt in 2 seconds...\nError: ', err);
autoConnect();
}
else {
console.log('DBServer Error: minor error\nError: ', err);
}
});
}
autoConnect();
return connection
}
In every routing file I require this DBServer file via:
var db = require('./lib/DBServer').connect();
When I start my app, the console logs 3 times
DBServer connected successfully...
DBServer connected successfully...
DBServer connected successfully...
... as intended.
Everything works perfect. I run the App with forever and every time the script losses connection to the db-server (what happens from time to time) it reconnects again... as intended.
... BUT! Except of one script. Script routesA.js stops working when it comes to a mysql query. The script freezes but does not quit. I have to stop and restart it again.
There is no difference between the invocation of DBServer.js between routesA, routesB or routesC. And it works pretty good... But it seems that if routesA looses connection it does not reconnect again... routesB and routesC still works fine.
So I changed the way how script routesA.js connects to the database. I connect now in script routesA.js not via the DBServer.js and require but the manual way
var mysql = require('mysql');
var db = mysql.createConnection({
host: '127.0.0.1',
user: 'my username',
password: '******',
database: 'ma database'
});
db.connect();
Now it works... and runs for days without problems. But the reason why this works is, because now I don't have an error handling in script routesA.js.. so forever detects a script exit and restarts... and everything works again.
But I don't want that way. I want a proper error handling like in DBServer.js. As said this works for script B and C, but not script A...
I know that it is strange and difficult to say, what might be the problem. But maybe someone has had some similar problem.
Another question here is: How do you handle the database connections with multiple script files. I there a way to share one mysql connection for all script files you have in an app?
kind regards
martin
The reason probably has to do with the fact that you're returning the initial connection object on require(), but if you get disconnected, you reassign the connection variable which the external scripts do not have a reference to (they still only have a reference to the old/original connection object).
I should also note that if you're using the mysql2 module (compatible with mysql except much faster), there is a connection.ping() method that you can use to periodically ping the server to help keep the connection alive.