I am a bit confused why the Node.js script does not exit:
// myscript.js
var Model = bookshelf.Model.extend({
tableName: 'models'
});
var m = Model.forge({name: 'hello'});
m.save();
console.log("End");
But when running this with node myscript the script does not exit.
What is preventing the script from exiting?
How to cleanly exit after the model is saved?
Thanks!
What is preventing the script from exiting?
Knex holds a pool of connections to your database. Although your script's execution ends, behind the scenes Knex is still maintaining these, preventing the program from closing.
How to cleanly exit after the model is saved?
You can tear down all the database connections by calling knex.destroy(). You must do this after your queries are complete.
// myscript.js
var Model = bookshelf.Model.extend({
tableName: 'models'
});
Model.forge({name: 'hello'}).save().then(function(model) {
console.log('model has been saved', model);
}).catch(function(error) {
// it's bad practise to omit some sort of error reporting.
console.error('something went wrong!');
}).finally(function() {
// tear down connection.
return bookshelf.knex.destroy();
}).then(function () {
// This is the actual 'end' of the program.
console.log("End");
});
Related
Right now i am creating a very large application in Node JS. I am trying to make my code clean and short (Just like most of the developer). I've create my own js file to handle connection to mysql. Please see code below.
var mysql = require('mysql');
var config = {
'default' : {
connectionLimit : process.env.DB_CONN_LIMIT,
host : process.env.DB_HOST,
user : process.env.DB_USER,
password : process.env.DB_PASS,
database : process.env.DB_NAME,
debug : false,
socketPath : process.env.DB_SOCKET
}
};
function connectionFunc(query,parameters,callback,configName) {
configName = configName || "default";
callback = callback || null;
parameters = parameters;
if(typeof parameters == 'function'){
callback = parameters;
parameters = [];
}
//console.log("Server is starting to connect to "+configName+" configuration");
var dbConnection = mysql.createConnection(config[configName]);
dbConnection.connect();
dbConnection.query(query,parameters, function(err, rows, fields) {
//if (!err)
callback(err,rows,fields);
//else
//console.log('Error while performing Query.');
});
dbConnection.end();
}
module.exports.query = connectionFunc;
I am using the above file in my models, like below :
var database = require('../../config/database.js');
module.exports.getData = function(successCallBack){
database.query('SAMPLE QUERY GOES HERE', function(err, result){
if(err) {console.log(err)}
//My statements here
});
}
Using this coding style, everything works fine but when i am trying to create a function that will loop my model's method for some reason. Please see sample below :
for (i = 0; i < 10000; i++) {
myModel.getData(param, function(result){
return res.json({data : result });
});
}
It gives me an ER_CON_COUNT_ERROR : Too Many Conenction. The question is why i still get an error like these when my connection always been ended by this dbConnection.end();? I'm still not sure if i am missing something. I am still stuck on this.
My connection limit is 100 and i think adding more connection is a bad idea.
Because query data form the database is async.
In your loop the myModel.getData (or more precisely the underling query) will not halt/paus your code until the query is finished, but send the query to the database server and as soon as the database response the callback will be called.
The calling end on dbConnection will not close the connection immediately, it will just mark the connection to be closed as soon as all queries that where created with that connection are finished.
mysql: Terminating connections
Terminating a connection gracefully is done by calling the end() method. This will make sure all previously enqueued queries are still before sending a COM_QUIT packet to the MySQL server.
An alternative way to end the connection is to call the destroy() method. This will cause an immediate termination of the underlying socket. Additionally destroy() guarantees that no more events or callbacks will be triggered for the connection.
But with destroy the library will not wait for the result so the results are lost, destroy is rarely useful.
So with your given code you try to create 10000 connections at one time.
You should only use on connection by task, e.g. if a user requests data using the browser, then you should use one connection for this given request. The same is for timed task, if you have some task that is done in certain intervals.
Here an example code:
var database = require('./config/database.js');
function someTask( callback ) {
var conn = database.getConnection();
myModel.getData(conn, paramsA, dataReceivedA)
function dataReceivedA(err, data) {
myModel.getData(conn, paramsB, dataReceivedB)
}
function dataReceivedB(err, data) {
conn.end()
callback();
}
}
If you want to entirely hide your database connection in your model code. Then you would need to doe something like that:
var conn = myModel.connect();
conn.getData(params, function(err, data) {
conn.end();
})
How to actually solve this depends only many factors so it is only possible to give you hints here.
I'm using Node.js to write system scripts that run on a server. Due to Node's asynchronous nature, my script is exiting before the database calls have a chance to complete and nothing is ever written to the database.
I'm using Mongoose as an ORM and talking to a MongoDB, if that makes any difference. Node.js offers SYNCHRONOUS method calls for this very reason, for example: https://nodejs.org/api/child_process.html
I guess my questions are:
1) Does mongoose offer a way to block so my scripting process can wait for the database call to return?
2) If not, is there another method I should consider other than something like:
(function wait () {
if (!SOME_EXIT_CONDITION) setTimeout(wait, 1000);
})();
3) Is node not the best tool for the job for writing scripts? I love node for web app development, and can write nested callbacks or work with promises all day long. But what about as a scripting language?
EDIT -----------------------------------------------
Below is an quick example of the script to provide more clarity of the situation:
#!/usr/bin/env node
# Please note the above that this is a bash script
var schema = mongoose.Schema({
// ... attributes ...
});
var model = new (mongoose.model('ModelObject'))();
model['attribute'] = 42;
console.log('This gets printed first');
model.save(function(err) {
console.log('Nothing in the callback gets printed because callback is never called');
if(err) { // Can't check for errors because this is never reached
console.log('This never gets printed to the screen');
console.log('And consequently nothing is ever saved to mongo');
} else {
console.log('This never gets printed either');
}
});
console.log('This gets printed second');
If your model does not get saved, there is a Mongo error. Following MongoDB conventions you have to check for errors:
model.save(function(error, savedItem) {
if(error) {
// nothing is saved
}
});
Otherwise, have you considered using Promises? It useful for chaining events and simpler error handling.
Promise = require('bluebird');
Promise.promisifyAll(mongoose.Query.base);
model.saveAsync().then(function(savedItem) {
// saved
})
.catch(function(error) {
// handle error
});
I think you are looking for this, check below if this help you.
var mongoose = require('mongoose'),
model1 = mongoose.model('model1'),
model2 = mongoose.model('model2');
model1.findOne({"type" : 'Active'}, function err(err, catConfig) {
if(!err.error){
//This will execute once above DB call is done!
model2.findOne(condition).remove(function(err, gAnalysis) {
//Lines of code that you want to execute after second DB call
});
}
});
I don't see you opening a connection to the database so presumably saving a model instance does nothing, not even call the callback with an error...
I've tested the below example:
test.js:
var mongoose = require('mongoose');
var kittySchema = mongoose.Schema({
name: String
});
var Kitten = mongoose.model('Kitten', kittySchema);
mongoose.connect('mongodb://localhost:27017/test', function (err) {
if (err) throw err;
var silence = new Kitten({ name: 'Silence' });
silence.save(function (err, saved) {
if (err) throw err;
console.log('Kitty Silence is saved!');
mongoose.disconnect(function (err) {
if (err) throw err;
console.log('done...');
});
});
});
Running node test.js prints this to the console:
Kitty Silence is saved!
done...
and examining my local test database shows that Silence is indeed saved.
Background Information
I'm attempting my first node.js API/application. As a learning exercise, I'm trying to create some test cases initially delete all records in a table, insert 3 specific records, and then query for those 3 records.
Code
Here's the code I have cobbled together:
http://pastebin.com/duQQu3fm
Problem
As you can see from the code, I'm trying to put the database connection logic in a dbSession.js file and pass it around.
I am able to start up the http server by doing the following:
dev#devbox:~/nimble_node$ sudo nodejs src/backend/index.js
Server started and listening on port: 8080
Database connection successful
However, when I try to run my jasmine tests, it fails with the following error:
F
Failures:
1) The API should respond to a GET request at /api/widgets/
Message:
TypeError: Object #<MongoClient> has no method 'collection'
Stacktrace:
TypeError: Object #<MongoClient> has no method 'collection'
at resetDatabase (/home/dev/nimble_node/spec/resetDatabase.js:6:29)
at /home/dev/nimble_node/spec/e2e/apiSpec.js:23:25
at /home/dev/nimble_node/node_modules/async/lib/async.js:683:13
at iterate (/home/dev/nimble_node/node_modules/async/lib/async.js:260:13)
at async.forEachOfSeries.async.eachOfSeries (/home/dev/nimble_node/node_modules/async/lib/async.js:279:9)
at _parallel (/home/dev/nimble_node/node_modules/async/lib/async.js:682:9)
at Object.async.series (/home/dev/nimble_node/node_modules/async/lib/async.js:704:9)
at null.<anonymous> (/home/dev/nimble_node/spec/e2e/apiSpec.js:19:9)
at null.<anonymous> (/home/dev/nimble_node/node_modules/jasmine-node/lib/jasmine-node/async-callback.js:45:37)
Finished in 0.01 seconds
1 test, 1 assertion, 1 failure, 0 skipped
Database connection successful
Line 6 of resetDatabase is:
var collection = dbSession.collection('widgets');
Given that after the error appears, I get the "Database connection successful" message, I think what's happening is that when the tests request the dbSession library, the database hasn't finished running the code to connect. And therefore, I can't get the collection object.
I'm currently reading through the mongodb online manual to see if I can find some hints as to how to do something like this.
Any suggestions or pointers would be appreciated.
EDIT 1
To prove that there is a collection method on the MongoClient object, I changed the dbSession.js code to look like this:
'use strict';
var DBWrapper = require('mongodb').MongoClient;
var dbWrapper = new DBWrapper;
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
var collection = dbWrapper.collection('widgets');
console.log('just created a collection...');
}
});
module.exports = dbWrapper;
And now, when I start up the http server (index.js), notice the messages:
dev#devbox:~/nimble_node$ sudo nodejs src/backend/index.js
Server started and listening on port: 8080
Database connection successful
just created a collection...
It could be an async issue.
Your code in dbSessionjs
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
}
});
module.exports = dbWrapper;
Starts the connection at dbWrapper asynchronously, but exports dbWrapper right away, which is then imported in resetDatabase. Thus yes, the connect function may have not yet returned from the async function when you call it in resetDatabase (and is what the log suggests,as the error appears before the success log).
You could add a callback after dbWrapper.connect() returns, in order to actually only be able to use dbWrapper when the connection finished.
(With sqlite, this may not happen as it accesses the DB faster on the commandline).
This may not be your problem but looks like a candidate.
EDIT: Here's a possible example for a callback, but please take note it depends on what you need to do so there are a lot of different solutions. The key is to call a callback function when you are done initializing.
Another solution could be to simply wait, and/or poll (e.g. chcke a variable 'initialized').
'use strict';
var DBWrapper = require('mongodb').MongoClient;
var dbWrapper = new DBWrapper;
function doConnect(callback) {
console.log("Initializing DB connection...");
dbWrapper.connect("mongodb://localhost:27017/test", function(err, db) {
if (!err) {
console.log("Database connection successful");
dbWrapper = db;
var collection = dbWrapper.collection('widgets');
console.log('just created a collection...');
console.log('calling callback...');
callback(dbWrapper);
} else {
console.log("Error connectingi: " + err);
}
});
};
doConnect(function(correctDbWrapper) {
//Now you can use the wrapper
console.log("Inside callback, now consuming the dbWrapper");
dbWrapper = correctDbWrapper;
var collection = dbWrapper.collection('widgets');
});
It's interesting though I never ran into this issue, although I have generally used similar code like yours. I guess because normally I have this DB initialization right at the top, and then have to do lots of initializations on the node app, which gives the app time enough to return from the connect call....
I have a function which connect my local mongoDB database, but the connection suddenly doesn't work when I try to put it in a for-loop.
var connectMongo = require("./ConnectToMongoDB");
var insertDocument = require("./InsertDocument");
function spamMongoDBtest(){
process.nextTick(function(){
var max = 500;
for(var i = 0; i<max;i++){
setTimeout(function(){
connectMongo(insertDocument);
}, 50);
}
});
}
Why do I get this AssertionError:
AssertionError: null == { [MongoError: connect ECONNREFUSED] name:
'MongoError', message: 'connect ECONNREFUSED' }
connectMongo(insertDocument); DOES work in this function:
function spamMongoDB(){
process.nextTick(function(){
setInterval(function(){connectMongo(insertDocument); }, 100); });
}
Could you also share the connectMongo and insertDocument functions?
Perhaps you have too many open connections to Mongo. Try closing the connection to Mongo after each time you write the document.
You can use the async library to limit the concurrency of your test.
For example:
var concurrency = 25; //limit to 25 concurrent connections
async.eachLimit(new Array(500), concurrency, function(item, callback) {
connectMongo(function(){
insertDocument();
callback();
});
});
You are getting a connection error from MongoDb because you are trying to connect to database multiple times. You need to connect to database only once, and from that moment if the connection has been established the communication with database is opened.
You are opening the connection and once opened you can communicate with the database.
var mongoURI = "mongodb://localhost:3030/";
var MongoDB = mongoose.connect(mongoURI).connection;
MongoDB.on('error', function(err) { console.log(err.message); });
MongoDB.once('open', function() {
console.log("mongodb connection open");
insertDocument;
});
Inside the open function you can trigger the setInterval. Or if you want to stick with your code, each time you open a connection in the for loop you have to close it too. I don't know what connectMongo method contains, but after you invoke this method outside setTimeout method you have to close the connection. But this approach anyway will put on heavy duty the server, so my advice is to follow the method provided.
I'm aware of the best practice of MongoDB connection pooling in NodeJS of the singleton DB connection type like this
var db = null;
var connection = function getDBConnection(callback) {
if(db) { callback(null, db) } else { MongoClient.connect( .... ) }
}
module.exports = getDBConnection;
However, what I cannot get my head around at the moment is how to handle this in a one-shot script that, say, does some pre-initialization on the documents of a certain db collection:
getDBConnection(function (err, database) {
var collection = database.collection("objects");
var allObjectsArray = collection.find( /* ... */
).toArray(function (err, objects) {
if(err != null) console.log(err);
assert.equal(null, err);
_.each(objects, function (item) {
collection.update(
{ id: item.id},
{ $set: { /* ... */ }},
function (err, result) {
if(err != null) console.log(err);
assert.equal(null, err);
}
);
});
// database.close(); <-- this fails with "MongoError: Connection Closed By Application", thrown by the update callback
});
// database.close(); <-- this fails too, thrown by the toArray callback
});
If I call the script like that, it never terminates, due to the still open connection. If I close the connection at the bottom, it fails because of, well, a closed connection.
Considering that opening a new connection for every update is not really an option, what am I missing? Keeping the connection open may be fine for webapps, but for a one-shot script called from a shell script this really doesn't work out, does it?
Sorry if this question has arisen before, I've given it some research but have not quite been able to come up with a working answer for me...
Thanks!
Julian
As a "pooled connection" there is code running to keep the connection alive and establish more connections in the pool if required under the driver connection. So much like various "server code" methods, event loop handlers have been invoked and the process does not exit at the end of your code until these are de-registered.
Therefore your two choices to call after all your code has executed are either:
Call db.close() or in your code context specifically database.close() once all is done.
Call process.exit() which is a generic call in node.js applications which will shut the whole process down and therefore stop any other current event loop code. This actually gives you an option to throw an error on exit if you want your code to be "shell integrated" somewhere and look for the exit status.
Or call both. db.close() will allow execution to the next line of code and whatever you put there will also run.
But you have to wait until everything is called, so you can cannot use synchronous loops with asynchronous code in the middle:
async.each(objects,function(item,callback) {
collection.update(
{ "_id": item._id },
{
// updates
},
callback
);
},function(err) {
if (err) throw err;
database.close();
});