Develop Reference

Node js. Proper / Best Practice to create connection

Right now i am creating a very large application in Node JS. I am trying to make my code clean and short (Just like most of the developer). I've create my own js file to handle connection to mysql. Please see code below.
var mysql = require('mysql');
var config = {
'default' : {
connectionLimit : process.env.DB_CONN_LIMIT,
host : process.env.DB_HOST,
user : process.env.DB_USER,
password : process.env.DB_PASS,
database : process.env.DB_NAME,
debug : false,
socketPath : process.env.DB_SOCKET
}
};
function connectionFunc(query,parameters,callback,configName) {
configName = configName || "default";
callback = callback || null;
parameters = parameters;
if(typeof parameters == 'function'){
callback = parameters;
parameters = [];
}
//console.log("Server is starting to connect to "+configName+" configuration");
var dbConnection = mysql.createConnection(config[configName]);
dbConnection.connect();
dbConnection.query(query,parameters, function(err, rows, fields) {
//if (!err)
callback(err,rows,fields);
//else
//console.log('Error while performing Query.');
});
dbConnection.end();
}
module.exports.query = connectionFunc;
I am using the above file in my models, like below :
var database = require('../../config/database.js');
module.exports.getData = function(successCallBack){
database.query('SAMPLE QUERY GOES HERE', function(err, result){
if(err) {console.log(err)}
//My statements here
});
}
Using this coding style, everything works fine but when i am trying to create a function that will loop my model's method for some reason. Please see sample below :
for (i = 0; i < 10000; i++) {
myModel.getData(param, function(result){
return res.json({data : result });
});
}
It gives me an ER_CON_COUNT_ERROR : Too Many Conenction. The question is why i still get an error like these when my connection always been ended by this dbConnection.end();? I'm still not sure if i am missing something. I am still stuck on this.
My connection limit is 100 and i think adding more connection is a bad idea.

Because query data form the database is async.
In your loop the myModel.getData (or more precisely the underling query) will not halt/paus your code until the query is finished, but send the query to the database server and as soon as the database response the callback will be called.
The calling end on dbConnection will not close the connection immediately, it will just mark the connection to be closed as soon as all queries that where created with that connection are finished.
mysql: Terminating connections
Terminating a connection gracefully is done by calling the end() method. This will make sure all previously enqueued queries are still before sending a COM_QUIT packet to the MySQL server.
An alternative way to end the connection is to call the destroy() method. This will cause an immediate termination of the underlying socket. Additionally destroy() guarantees that no more events or callbacks will be triggered for the connection.
But with destroy the library will not wait for the result so the results are lost, destroy is rarely useful.
So with your given code you try to create 10000 connections at one time.
You should only use on connection by task, e.g. if a user requests data using the browser, then you should use one connection for this given request. The same is for timed task, if you have some task that is done in certain intervals.
Here an example code:
var database = require('./config/database.js');
function someTask( callback ) {
var conn = database.getConnection();
myModel.getData(conn, paramsA, dataReceivedA)
function dataReceivedA(err, data) {
myModel.getData(conn, paramsB, dataReceivedB)
}
function dataReceivedB(err, data) {
conn.end()
callback();
}
}
If you want to entirely hide your database connection in your model code. Then you would need to doe something like that:
var conn = myModel.connect();
conn.getData(params, function(err, data) {
conn.end();
})
How to actually solve this depends only many factors so it is only possible to give you hints here.

RxJS + Node.js Http Request [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Nodejs server side implementation: How to use https://www.npmjs.com/package/request with https://www.npmjs.com/package/rx to make GET request to https://www.reddit.com/r/javascript.json?
Goal: I attempting to accomplish constant streaming whenever there are data changes to whatever site api url I'm using.

Unfortunately you will be unable to receive updates when https://www.reddit.com/r/javascript.json is updated as there is no way for them to push this information to the client.
Services such as Github will let you register a webhook which will let them push data to an endpoint on your server. I am unsure if Reddit supports this.
As an alternate solution, and building on what AkkarinZA said in his answer, you could poll the json document using something similar to the following:
var fetchContent = function(url) {
return rx.Observable.create(function (observer) {
request(url, function (error, response, body) {
if (error) { observer.onError(); }
else { observer.onNext({response: response, body: body }); }
observer.onCompleted();
})
});
};
rx.Observable.interval(1000)
.map(function() { return 'https://www.reddit.com/r/javascript.json' })
.flatMap(fetchContent)
.map(/* do something */)
.subscribe();
Polling such as this isn't a good approach.

You want to create an observable with observers notified from the callback. Try something like:
rx.Observable.create(function (observer) {
request('https://www.reddit.com/r/javascript.json', function (error, response, body) {
if (error) { observer.onError(); }
else { observer.onNext({response: response, body: body }); }
observer.onCompleted();
})
})
.map(/* do something */)
.subscribe();

Firebase.createUser() - Create a return function [duplicate]

This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 7 years ago.
I would like to create a simple function in Javascript that returns me the result of calling Firebase.createUser() dependent on onComplete.
I've done like this:
function createAUser(email, password) {
if (email === "" || password === "") { return false; }
var
onComplete = function (error, userData) {
if (error) {
switch (error.code) {
case "EMAIL_TAKEN":
console.log("The new user account cannot be created because the email is already in use.");
break;
case "INVALID_EMAIL":
console.log("The specified email is not a valid email.");
break;
default:
console.log("Error creating user:", error);
}
return false;
} else {
console.log("Successfully created user account with uid:", userData.uid);
return true;
}
};
MyFirebaseRootRef.createUser({ email: email, password: password}, onComplete);
/* return something here, true/false based on onComplete */
}
Or.. are there any other way to get me what I want. What I'm essentially after is just to find a way to figure out not only through console.log() how the creating of a user went.
Sorry for typos/bad code, thanks for all responses!
Edit:
Thanks for all responses I've now looked into the callback & the asynchronous stuff (something like starting another thread, and then follow through with the function). I must give it some thought over data, like the stack data in the function must be release upon return, how can this data be followed through to the callback.
Anyhow sorry for duplicate, thanks again

You're dealing with asynchronous data.
The call to create the Firebase user goes over the network, which means we have to patiently wait for the user to come back. But, we don't want to block the only thread we have to do other operations. This means the call will go on the JavaScript event loop and when the network request completes (and the call stack is clear), we will finally get our data back.
To handle this elegantly in code you can use a Promise.
var user = { email: 'my#email.com', password: 'secret' };
var promise = new Promise() {
function(resolve, reject) {
var ref = new Firebase('<my-firebase-app>');
ref.createUser(user, function(error, authData) {
if (error) {
reject(error);
} else {
resolve(authData);
}
});
}
};
promise
.then(function(authData) {
})
.catch(function(error) {
});
Promises are native to modern browsers, but if you want to support older browsers you'll have to use a library like jQuery, Bluebird, or RSVP.

onComplete is a callback. This means that your code is sending a request and when it is completed, onComplete is being called. When you call createUser, the request is being sent and is not completed, so at the line after calling createUser you are not able to get the result of onComplete, because the request is not finished yet. As a result, you need to handle the complete event inside the function you associated to it.

Blocking Node in Script

I'm using Node.js to write system scripts that run on a server. Due to Node's asynchronous nature, my script is exiting before the database calls have a chance to complete and nothing is ever written to the database.
I'm using Mongoose as an ORM and talking to a MongoDB, if that makes any difference. Node.js offers SYNCHRONOUS method calls for this very reason, for example: https://nodejs.org/api/child_process.html
I guess my questions are:
1) Does mongoose offer a way to block so my scripting process can wait for the database call to return?
2) If not, is there another method I should consider other than something like:
(function wait () {
if (!SOME_EXIT_CONDITION) setTimeout(wait, 1000);
})();
3) Is node not the best tool for the job for writing scripts? I love node for web app development, and can write nested callbacks or work with promises all day long. But what about as a scripting language?
EDIT -----------------------------------------------
Below is an quick example of the script to provide more clarity of the situation:
#!/usr/bin/env node
# Please note the above that this is a bash script
var schema = mongoose.Schema({
// ... attributes ...
});
var model = new (mongoose.model('ModelObject'))();
model['attribute'] = 42;
console.log('This gets printed first');
model.save(function(err) {
console.log('Nothing in the callback gets printed because callback is never called');
if(err) { // Can't check for errors because this is never reached
console.log('This never gets printed to the screen');
console.log('And consequently nothing is ever saved to mongo');
} else {
console.log('This never gets printed either');
}
});
console.log('This gets printed second');

If your model does not get saved, there is a Mongo error. Following MongoDB conventions you have to check for errors:
model.save(function(error, savedItem) {
if(error) {
// nothing is saved
}
});
Otherwise, have you considered using Promises? It useful for chaining events and simpler error handling.
Promise = require('bluebird');
Promise.promisifyAll(mongoose.Query.base);
model.saveAsync().then(function(savedItem) {
// saved
})
.catch(function(error) {
// handle error
});

I think you are looking for this, check below if this help you.
var mongoose = require('mongoose'),
model1 = mongoose.model('model1'),
model2 = mongoose.model('model2');
model1.findOne({"type" : 'Active'}, function err(err, catConfig) {
if(!err.error){
//This will execute once above DB call is done!
model2.findOne(condition).remove(function(err, gAnalysis) {
//Lines of code that you want to execute after second DB call
});
}
});

I don't see you opening a connection to the database so presumably saving a model instance does nothing, not even call the callback with an error...
I've tested the below example:
test.js:
var mongoose = require('mongoose');
var kittySchema = mongoose.Schema({
name: String
});
var Kitten = mongoose.model('Kitten', kittySchema);
mongoose.connect('mongodb://localhost:27017/test', function (err) {
if (err) throw err;
var silence = new Kitten({ name: 'Silence' });
silence.save(function (err, saved) {
if (err) throw err;
console.log('Kitty Silence is saved!');
mongoose.disconnect(function (err) {
if (err) throw err;
console.log('done...');
});
});
});
Running node test.js prints this to the console:
Kitty Silence is saved!
done...
and examining my local test database shows that Silence is indeed saved.

NodeJS one-shot script using MongoDB connection pooling - how to terminate?

I'm aware of the best practice of MongoDB connection pooling in NodeJS of the singleton DB connection type like this
var db = null;
var connection = function getDBConnection(callback) {
if(db) { callback(null, db) } else { MongoClient.connect( .... ) }
}
module.exports = getDBConnection;
However, what I cannot get my head around at the moment is how to handle this in a one-shot script that, say, does some pre-initialization on the documents of a certain db collection:
getDBConnection(function (err, database) {
var collection = database.collection("objects");
var allObjectsArray = collection.find( /* ... */
).toArray(function (err, objects) {
if(err != null) console.log(err);
assert.equal(null, err);
_.each(objects, function (item) {
collection.update(
{ id: item.id},
{ $set: { /* ... */ }},
function (err, result) {
if(err != null) console.log(err);
assert.equal(null, err);
}
);
});
// database.close(); <-- this fails with "MongoError: Connection Closed By Application", thrown by the update callback
});
// database.close(); <-- this fails too, thrown by the toArray callback
});
If I call the script like that, it never terminates, due to the still open connection. If I close the connection at the bottom, it fails because of, well, a closed connection.
Considering that opening a new connection for every update is not really an option, what am I missing? Keeping the connection open may be fine for webapps, but for a one-shot script called from a shell script this really doesn't work out, does it?
Sorry if this question has arisen before, I've given it some research but have not quite been able to come up with a working answer for me...
Thanks!
Julian

As a "pooled connection" there is code running to keep the connection alive and establish more connections in the pool if required under the driver connection. So much like various "server code" methods, event loop handlers have been invoked and the process does not exit at the end of your code until these are de-registered.
Therefore your two choices to call after all your code has executed are either:
Call db.close() or in your code context specifically database.close() once all is done.
Call process.exit() which is a generic call in node.js applications which will shut the whole process down and therefore stop any other current event loop code. This actually gives you an option to throw an error on exit if you want your code to be "shell integrated" somewhere and look for the exit status.
Or call both. db.close() will allow execution to the next line of code and whatever you put there will also run.
But you have to wait until everything is called, so you can cannot use synchronous loops with asynchronous code in the middle:
async.each(objects,function(item,callback) {
collection.update(
{ "_id": item._id },
{
// updates
},
callback
);
},function(err) {
if (err) throw err;
database.close();
});