How can I change things around in my db connection call so that I can do db.collection():
// Create a Mongo connection
Job.prototype.getDb = function() {
if (!this.db)
this.db = Mongo.connectAsync(this.options.connection);
return this.db;
};
// I want to be able to do this
Job.prototype.test = function() {
return this.db.collection('abc').findAsync()...
};
// Instead of this
Job.prototype.test = function() {
return this.getDb().then(function(db) {
return db.collection('abc').findAsync()...
});
};
My code always calls getDb so the connection does get created so that is not an issue. ex:
this.getDb().then(test.bind(this));
But I actually string many of these calls around so looking for a cleaner approach.
This works - just wondering if there an overall better approach to dealing with this.
Job.prototype.getDb = function(id) {
var self = this;
return new P(function(resolve, reject) {
if (!self.db) {
return Mongo.connectAsync(self.options.connection)
.then(function(c) {
self.db = c;
debug('Got new connection');
resolve(c);
});
}
debug('Got existing connection');
resolve(self.db);
});
};
I suppose this is really just a mongo connection issue, perhaps not just promises. All the Mongo examples I see either just make all their calls inside of the connect callback or use some framework like Express and assign it on start.
I want to be able to do this
return this.db.collection('abc').findAsync()
No, that's impossible when you don't know whether the database is already connected or not. If you might need to connect at first, and that is asynchronous, then this.db must yield a promise, and you'll need to use then.
Notice that with Bluebird you can shorten that code a bit, and avoid that verbose .then() callback by using the .call() method:
Job.prototype.getDb = function() {
if (!this.db)
this.db = Mongo.connectAsync(this.options.connection);
return this.db;
};
Job.prototype.test = function() {
return this.getDb().call('collection', 'abc').call('findAsync');
};
Related
Beginning to feel really thick here. Read a lot and I believe I understand promises and async-await decently well. However, I seem to struggle to use the function elsewhere, such that I can obtain the result (e.g. i get pending in another js file with: let dbConnection = dbOperations.openDatabaseConnection();).
Could someone explain to me why do I keep getting pending from the below functions (same function written with promise and asyncawait)? I can console.log the dbConnection result as expected prior to my return within the function. Also, I am particularly keen to understand promises in this sense, as it seems that many npm packages seem to return promises (and with my experience at least the async-await does not sit well with that? -> using async does not wait for resolve in my experience).
// Establish database connection
function openDatabaseConnection() {
let dbConnection = {};
return mongodb.connect(dbUri).then(conn => {
dbConnection.connection = conn;
return dbConnection;
})
.then(() => {
dbConnection.session = dbConnection.connection.db(dbName);
//console.log(dbConnection);
return dbConnection;
})
.catch(err => {
throw err;
});
};
// Establish database connection
async function openDatabaseConnection() {
let dbConnection = {};
try {
dbConnection.connection = await mongodb.connect(dbUri);
dbConnection.session = await dbConnection.connection.db(dbName);
} finally {
//console.log(dbConnection);
return dbConnection;
};
};
Both functions return again a promise.
So in your statement let dbConnection = dbOperations.openDatabaseConnection();
you assign a promise.
Thus you need to do something like:
dbOperations.openDatabaseConnection().then((dbConn) => ..)
or
let dbConnection = await dbOperations.openDatabaseConnection();
(note this requires to be wrapped in an async function)
Async/await is just another way to work with Promises, just don't wait for something that isn't a Promise.
async function openDatabaseConnection() {
let dbConnection = {};
try {
dbConnection.connection = await mongodb.connect(dbUri);
// await here does not make sense, this function does not return a Promise
// dbConnection.session = await dbConnection.connection.db(dbName);
dbConnection.session = dbConnection.connection.db(dbName);
} finally {
//console.log(dbConnection);
// return will always execute, keep here only when it should
// return an empty object if the connection fails
return dbConnection;
};
};
More info on async/await
I've learned node.js and javascript lately. I loved node.js a lot, but I am working on a project coded in node.js, mongodb, cordova etc. I notice that I needed to use Promise Object in the code a lot.
I create a module in the project to query the db and bring results. In every exported function I need to declare a local function, then use promise, for example:
I have the following local functions in the Module:
var Initialize = function() {
return new Promise(function(resolve, reject) {
try {
MongoClient.connect("db_url_conn", function(err, database) {
if (err) return console.log(err)
db = database;
return resolve(db);
})
} catch (e) {
console.error(e);
}
});
};
then in every exported function in the module I needed to use:
mongoOperation.prototype.getLength = function() {
Initialize(function(db) {
return db;
}).then(function(db) {
getSize(db).then(function(length) {
console.log(length);
});
});
}
The Question is:
Is that normal according to the nature of node.js and JavaScript nature to use promise a lot?
Do I have any other choices to fulfill that?
Since MongoClient.connect() already returns a promise, you can simplify your code:
var Initialize = function() {
return MongoClient.connect("db_url_conn");
};
...
Initialize().then(function(db) { ... });
However, this will create a new client each time you call Initialize, where you should be reusing the client for better performance and to leverage the built-in connection pool:
var client = MongoClient.connect("db_url_conn");
var Initialize = function() { return client };
Lets say that we have two buttons, each on are calling the following method:
var NUMBER_OF_IMAGE_REQUEST_RETRIES = 3;
var IMAGE_REQUEST_TIMEOUT = 3000;
processImage: function(image_data) {
var main_response = $q.defer();
var hash = getImageHash(image_data);
var requestsCounter = -1;
requestImage = function() {
$http.post(apiUrl, {params: {data: hash},timeout: IMAGE_REQUEST_TIMEOUT})
.then(function(response) {
return main_response.resolve(response.data);
}, function(error) {
if (++requestsCounter < NUMBER_OF_IMAGE_REQUEST_RETRIES) {
requestLabelsImage();
} else {
return main_response.reject();
}
});
};
requestLabelsImage();
return main_response.promise;
}
The method passes an image related data to the server, the server process the data and then response. Every time a user press a different button different image_data is being send to the server.
The problem:
The user press button 1, the method is called with image_data_1, and then he/she immediately press button 2 and the method is called with image_data_2. The processImage function is called by another method, lets say doSomethingWithTheResponse which only cares about the latest user's action, but the image_data_2 is proceed faster by the servers, so the client gets image_data_2 before image_data_1, so the client believes that image_data_1 was related to the user's latest action, which is not the case. How can we ensure that the client is always getting the response that is related to the users latest action?
Note: The hash is different for the differente image_data requests.
I was thinking something like:
var oldhash = null;
processImage: function(image_data) {
var main_response = $q.defer();
var hash = getImageHash(image_data);
oldhash = hash;
var requestsCounter = -1;
requestImage = function(hash) {
if(hash === oldhash){
$http.post(apiUrl, {params: {data: hash},timeout: IMAGE_REQUEST_TIMEOUT})
.then(function(response) {
return main_response.resolve(response.data);
}, function(error) {
if (++requestsCounter < NUMBER_OF_IMAGE_REQUEST_RETRIES) {
requestLabelsImage(hash);
} else {
return main_response.reject();
}
});
}
else {
main_response.reject();
}
}
requestLabelsImage(hash);
return main_response.promise;
}
But I am not 100% sure that this is the right approach.
Simply disregard the previous requests.
You can create a repository of requests (array or dictionary implementation is okay). Call .abort() on the previous ones once another request is made -- when you add it in your storage.
If you want a dictionary, there is a good example here (tackles a different topic, though), but here is a modified snippet of his code which is related to your case:
var _pendingRequests = {};
function abortPendingRequests(key) {
if (_pendingRequests[key]) {
_pendingRequests[key].abort();
}
}
Where the key can be.. say... a category of your action. You can name constants for it, or it can be just the name of the button pressed. It can even be a URL of your request; completely up to you.
There is an excellent explanation of the whole concept here:
jquery abort() ajax request before sending another
https://stackoverflow.com/a/3313022/594992
If your UI allows for initiation multiple actions, while processing of those actions are mutually exclusive, then you should probably use promises, and track active promises.
button1.addEventListener("click", function(evt) {
startRunning( task1.start() );
});
button2.addEventListener("click", function(evt) {
startRunning( task2.start() );
});
With a task runner like:
function startRunning( promise ) {
while(runningTasks.length>0) {
cancel( runningTasks.unshift() );
});
runningTasks.push( promise );
}
Your cancel function can come from anything that can deal with promises, like Angular's service.cancelRequest, or you can write your own code that takes the promise and smartly breaks off its operation.
Of course, if you're not using promises, then you probably want to start doing so, but if you absolutely can't you can use a manager object like:
button1.addEventListener("click", function(evt) { task1(); });
button2.addEventListener("click", function(evt) { task2(); });
with
var manager = [];
function cancelAll() {
while(manager.lenght>0) {
var cancelfn = manager.unshift()
cancelfn();
}
return true;
}
function task1() {
var running = cancelAll();
manager.push(function() { running = false; });
asyncDo(something1, function(result) {
if(!running) return;
// do your real thing
});
}
function task1() {
var running = cancelAll();
manager.push(function() { running = false; });
asyncDo(something2, function(result) {
if(!running) return;
// do your real thing
});
}
And you can put cancels on as many aspects as you need. If you need to cancel running XHRs, you might be able to do so, if you have multiple steps in your result handling, cut off at each step start, etc.
This sounds like an ideal use-case for promises. Basically, whenever a new request is made, you want to cancel any existing promises. I am not versed in AngularJS, but the following ng-specific links might prove useful:
Angularjs how to cancel resource promise when switching routes
Canceling A Promise In AngularJS
I know this is wrong, but essentially I want to
connect to a db/orm via a promise
wait on that promise to fulfill and get the models (the return from the promise)
use the results for form a middleware generator function to place the models on the request
I suspect that this isn't the best approach, so essentially I have two questions:
Should I be rewriting my db/orm connect to a generator function (I have a feeling that is more inline with koa style)
Back to the original question (as I am sure I will not get a chance to rewrite all my business logic) - how do I wait on a promise to fulfill and to then return a generator function?
This is my poor attempt - which is not working, and honestly I didn't expect it to, but I wanted to start by writing code, to have something to work with to figure this out:
var connectImpl = function() {
var qPromise = q.nbind(object.method, object);
return qPromise ;
}
var domainMiddlewareImpl = function() {
let connectPromise = connectImpl()
return connectPromise.then(function(models){
return function *(next){
this.request.models = models ;
}
})
}
var app = koa()
app.use(domainMiddlewareImpl())
According to this, you can do the following:
var domainMiddlewareImpl = function() {
return function *(){
this.request.models = yield connectImpl();
};
};
A context sensitive answer based on the info provided by Hugo (thx):
var connectImpl = function() {
var qPromise = q.nbind(object.method, object);
return qPromise ;
}
var domainMiddlewareImpl = function () {
let models = null ;
return function *(next) {
if(models == null){
//////////////////////////////////////////////////////
// we only want one "connection", if that sets up a
// pool so be it
//////////////////////////////////////////////////////
models = yield connectImpl() ;
}
this.request.models = models.collections;
this.request.connections = models.connections;
yield next
};
};
My example the connectImpl is setting up domain models in an ORM (waterline for now), connecting to a database (pooled), and returning a promise for the ORM models and DB connections. I only want that to happen once, and then for every request through my Koa middleware, add the objects to the request.
i have a recursive query like this (note: this is just an example):
var user = function(data)
{
this.minions = [];
this.loadMinions = function()
{
_user = this;
database.query('select * from users where owner='+data.id,function(err,result,fields)
{
for(var m in result)
{
_user.minions[result[m].id] = new user(result[m]);
_user.minions[result[m].id].loadMinions();
}
}
console.log("loaded all minions");
}
}
currentUser = new user(ID);
for (var m in currentUser.minions)
{
console.log("minion found!");
}
this don't work because the timmings are all wrong, the code don't wait for the query.
i've tried to do this:
var MyQuery = function(QueryString){
var Data;
var Done = false;
database.query(QueryString, function(err, result, fields) {
Data = result;
Done = true;
});
while(Done != true){};
return Data;
}
var user = function(data)
{
this.minions = [];
this.loadMinions = function()
{
_user = this;
result= MyQuery('select * from users where owner='+data.id);
for(var m in result)
{
_user.minions[result[m].id] = new user(result[m]);
_user.minions[result[m].id].loadMinions();
}
console.log("loaded all minions");
}
}
currentUser = new user(ID);
for (var m in currentUser.minions)
{
console.log("minion found!");
}
but he just freezes on the while, am i missing something?
The first hurdle to solving your problem is understanding that I/O in Node.js is asynchronous. Once you know how this applies to your problem the recursive part will be much easier (especially if you use a flow control library like Async or Step).
Here is an example that does some of what you're trying to do (minus the recursion). Personally, I would avoid recursively loading a possibly unknown number/depth of records like that; Instead load them on demand, like in this example:
var User = function(data) {
this.data = data
this.minions;
};
User.prototype.getMinions = function(primaryCallback) {
var that = this; // scope handle
if(this.minions) { // bypass the db query if results cached
return primaryCallback(null, this.minions);
}
// Callback invoked by database.query when it has the records
var aCallback = function(error, results, fields) {
if(error) {
return primaryCallback(error);
}
// This is where you would put your recursive minion initialization
// The problem you are going to have is callback counting, using a library
// like async or step would make this party much much easier
that.minions = results; // bypass the db query after this
primaryCallback(null, results);
}
database.query('SELECT * FROM users WHERE owner = ' + data.id, aCallback);
};
var user = new User(someData);
user.getMinions(function(error, minions) {
if(error) {
throw error;
}
// Inside the function invoked by primaryCallback(...)
minions.forEach(function(minion) {
console.log('found this minion:', minion);
});
});
The biggest thing to note in this example are the callbacks. The database.query(...) is asynchronous and you don't want to tie up the event loop waiting for it to finish. This is solved by providing a callback, aCallback, to the query, which is executed when the results are ready. Once that callback fires and after you perform whatever processing you want to do on the records you can fire the primaryCallback with the final results.
Each Node.js process is single-threaded, so the line
while(Done != true){};
takes over the thread, and the callback that would have set Done to true never gets run because the thead is blocked on an infinite loop.
You need to refactor your program so that code that depends on the results of the query is included within the callback itself. For example, make MyQuery take a callback argument:
MyQuery = function(QueryString, callback){
Then call the callback at the end of your database.query callback -- or even supply it as the database.query callback.
The freezing is unfortunately correct behaviour, as Node is single-threaded.
You need a scheduler package to fix this. Personally, I have been using Fibers-promise for this kind of issue. You might want to look at this or another promise library or at async