Develop Reference

Function Return not Working

I am working in Node and trying to load the next sequence from my db. I am able to access the db, load and return the sequence within my function, but I am not able to access it outside of the function.
function getRunId() {
counters.findOne({_id: 'Run_ID'}, function(err, resp) {
if(err) {
console.log(err);
}
console.log('Seq: ' + resp.sequence); // Console Output = Seq: 1234
return resp.sequence;
});
};
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId); // Console Output = CRID: undefined
I've checked several pages worth of Stack Overflow issues relating to using callback's, async (node module), how to properly return values in the function, etc... but none of them get me closer to accessing currentRunId outside of the function.
Is this issue further complicated by the use of Mongo queries inside my function?

For anyone stumbling on this later, start by reading this answer.
I've dealt with this a few times so I understand the frustration. You are trying to mix sync and async code by doing this:
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId);
The trouble is that console.log('Run_ID: ' + currentRunId) is called immediately after you invoke getRunID() by assigning it to current RunID, and getRunID() resolves after console.log('Run_ID: ' + currentRunId), causing the currentRunId variable to be undefined.
But, you have some options to deal with this. Option one is to return a callback, and log the results of the callback instead. Option 2 is to use an ES6 promise. To use option 2, you need node version 7, and you need to use 'use strict' in your code.
Here are 3 examples built around a function stub that spoofs the results of findOne(). The getRunIdA() is your function, and getRunIdB, and getRunIdC are two example solutions to your current problem.
'use strict'
// A function stub which represents a simplified version of findOne.
// Accepts callback and returns a callback with the results of data
function findOne (callback) {
var data = {
sequence: 6
}
return callback(null, data)
}
// This is a simplified version of your function, which reproduces the undefined result
function getRunIdA () {
findOne(function (err, resp) {
if (err) {
console.log(err)
}
console.log('Seq: ' + resp.sequence)
return resp.sequence
})
}
// This is your function with a callback
function getRunIdB (callback) {
findOne(function (err, resp) {
if (err) {
console.log(err)
}
console.log('Seq: ' + resp.sequence)
return callback(resp.sequence)
})
}
// This is your function with a promise
var getRunIdC = new Promise(function (resolve, reject) {
resolve(findOne(function (err, resp) {
if (err) {
console.log(err)
}
return resp.sequence
}))
})
// Invoke your funciton; get undefined
var currentRunID = getRunIdA()
console.log('Run_ID: ' + currentRunID) // Run_ID: undefined
// Invoke getRunIdB using callback, get 6
getRunIdB(function (id) {
console.log('Run_ID: ' + id) // Run_ID: 6
})
// Invoke getRunIdC with a promise; get 6
getRunIdC.then(function (currentRunID) {
console.log('Run_ID: ' + currentRunID) // Run_ID: 6
})
/*
results for all 3:
Seq: 6
Run_ID: undefined
Seq: 6
Run_ID: 6
Run_ID: 6
*/
Give this a try by saving to your machine and running:
node test.js
Is this issue further complicated by the use of Mongo queries inside my function?
Nope, you just need to pass the results of your query to a promise or a callback so that you can work with the results somewhere else.
I hope this helps!
Edit: OP added the following code in a comment, which I will try to break down and address.
Unfortunately, using getRunIdB results in callback is not defined and using getRunIdC results in currentRunId is not defined
var currentRunID = '';
var getRunId = new Promise(function (resolve, reject) { resolve(counters.findOne({_id: 'Run_ID'}, function (err, resp) {
if (err) {
console.log(err)
}
return resp.sequence;
}))
});
getRunId.then(function (res) {
console.log('Run_ID: ' + res.sequence) // Run_ID: 1234
currentRunID = res.sequence;
})
console.log(currentRunID); // currentRunID is not defined
Check out an answer I gave to a similar question for more details on the JS concurrency model. Simply put, the getRunID() function is executing asynchronous code. What that means is that getRunID() doesn't get inserted into the message queue that determines what order javascript will execute until it's callbacks are completed. Thus, when you log currentRunID outside of the .then() function, the results is undefined because currentRunID is undefined.
I think that ultimately what OP is trying to do is to export the result of the function so that the something can be done with those results, this needs to be done within a callback like so:
getRunId.then(function (res) {
// Do stuff with the run ID here.
})

You are only returning on a callback function but not on the actual function.. Change your code to this:
function getRunId() {
var result = counters.findOne({_id: 'Run_ID'}, function(err, resp) {
if(err) {
console.log(err);
}
console.log('Seq: ' + resp.sequence); // Console Output = Seq: 1234
return resp.sequence;
});
return result; //<-- return result of your function is here
};
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId);

Async functions in meteor server

I'm using vpulim:node-soap to have a soap server running.
My meteor server startup contains this amongst various other code:
authRequestOperation: function(args,cb,headers,req) {
console.log(args);
var authResponceObject = {};
var futureAuthResponse = new Future();
Fiber(function(){
if(collectorUsers.findOne({username: args.username})){
console.log("Found User");
authResponceObject = {
username: args.username,
nonce: Random.id()
};
console.log("authResponceObject is: " + JSON.stringify(authResponceObject,null,4));
console.log("futureAuthResponse returning...");
futureAuthResponse.return(authResponceObject);
}
// console.log("futureAuthResponse waiting...");
// return futureAuthResponse.wait();
}).run();
console.log("authResponceObject after fiber is: " + JSON.stringify(authResponceObject,null,4));
return authResponceObject;
},
What I'm trying to do is:
I receive a user object from client.
I check if the user is present in the mongodb
If user is present, prepare response object
Respond to the client with the response object.
I have 1. working. However, the dute it being async call, the order of 2,3,4 is messed up.
Right now what's happening is:
receive client object
return response object (which is empty)
Check mongo
Prepare response object.
I'm not using Meteor.methods for the above.
How do I make this work in the right manner? I've tried juggling around wrapAsync and fiber/future but hitting dead ends.

I believe Meteor.bindEnvironment can solve your problem, try this code:
{
// ...
authRequestOperation: Meteor.bindEnvironment(function(args, cb, headers, req) {
console.log(args);
var authResponceObject = {};
if (collectorUsers.findOne({username: args.username})) {
console.log("Found User");
authResponceObject = {
username: args.username,
nonce: Random.id()
};
console.log("authResponceObject is: " + JSON.stringify(authResponceObject, null, 4));
}
return authResponceObject;
}),
// ...
}

JavaScript Callbacks and Splunk

I am in the process of using the Splunk Javascript API to gain access to some of its functionality, but I'm having trouble understanding JavaScript concepts behind callbacks.
An example from the docs:
var http = new splunkjs.ProxyHttp("/proxy");
var service = new splunkjs.Service(http, {
username: username,
password: password,
scheme: scheme,
host: host,
port: port,
version: version
});
Async.chain([
// First, we log in
function(done) {
service.login(done);
},
// Perform the search
function(success, done) {
if (!success) {
done("Error logging in");
}
service.search("search index=_internal | head 3", {}, done);
},
// Wait until the job is done
function(job, done) {
Async.whilst(
// Loop until it is done
function() { return !job.properties().isDone; },
// Refresh the job on every iteration, but sleep for 1 second
function(iterationDone) {
Async.sleep(1000, function() {
// Refresh the job and note how many events we've looked at so far
job.fetch(function(err) {
console.log("-- fetching, " + (job.properties().eventCount || 0) + " events so far");
iterationDone();
});
});
},
// When we're done, just pass the job forward
function(err) {
console.log("-- job done --");
done(err, job);
}
);
},
// Print out the statistics and get the results
function(job, done) {
// Print out the statics
console.log("Job Statistics: ");
console.log(" Event Count: " + job.properties().eventCount);
console.log(" Disk Usage: " + job.properties().diskUsage + " bytes");
console.log(" Priority: " + job.properties().priority);
// Ask the server for the results
job.results({}, done);
},
// Print the raw results out
function(results, job, done) {
// Find the index of the fields we want
var rawIndex = utils.indexOf(results.fields, "_raw");
var sourcetypeIndex = utils.indexOf(results.fields, "sourcetype");
var userIndex = utils.indexOf(results.fields, "user");
// Print out each result and the key-value pairs we want
console.log("Results: ");
for(var i = 0; i < results.rows.length; i++) {
console.log(" Result " + i + ": ");
console.log(" sourcetype: " + results.rows[i][sourcetypeIndex]);
console.log(" user: " + results.rows[i][userIndex]);
console.log(" _raw: " + results.rows[i][rawIndex]);
}
// Once we're done, cancel the job.
job.cancel(done);
}
],
function(err) {
callback(err);
}
);
Async.chain is defined here as being root.chain = function(tasks, callback). My understanding is that there are 5 functions in the tasks array which are executed one after the other, and pass the results from one to the other.
However I do not understand how and where "done", "success","job" and "results" are defined, or how it is that they are used as arguments within the body of their functions?
function(success, done) {
if (!success) {
done("Error logging in");
}
service.search("search index=_internal | head 3", {}, done);
}
here, how is it testing against success, and passing a string to done()?
and how does the two functions
function(job, done) {// Print out the statics ..}
&
function(results, job, done) { .. }
pass the results data from the first function to the second?
Apologies for the long question.

In Javascript, functions create new scope. That means it does not matter what the passed arguments were named before they were passed to the function.
var awesomeName = 'bob';
hi(awesomeName);
// name === undefined
function hi(name) {
// name === 'bob';
console.log('hi', name); // Outputs: 'hi bob' in console
}
// name === undefined
As you said, each task calls the next task as a callback. The last argument is always the next task function/callback. That means that Async.chain probably automagically adds the callbacks to the end of the arguments before calling each task function. done is just a conventional name to assign to the callback. Similarly, the other arguments are just descriptive names for the arguments passed by the previous task. In order to see why they are named that way, you should look at the function that is calling the callback.
For example:
service.login(done) probably has some kind of code in it that does something like this:
login: function(callback) {
var successful;
// Do Login logic here and assign true/false to successful
callback(successful);
}
The callback is the next task in the chain and has two arguments, success and done. success is just the first argument that login passed to it. Async.chain always passes another argument as the last argument: the next task function, which is just assigned the name done by convention. You could name it whatever you want within each function, as long as you refer to it as the same name within the function.
function cb(success, fuzzyCallback) {
if (!success) {
fuzzyCallback('Error!');
}
fuzzyCallback(null);
}

Using JS Promises to Execute Multiple Async Queries to Build an Object

After recently discovering JS promises, I have been studying them so that I might build a certain functionality that allows me to execute 4 async queries, use the result of each to build an object that I can finally send as a response to a request directed at my node app.
The final object is made up of 3 array properties containing the resulting rows of each query.
It seems that I've done something wrong handling the promises, though, because ultimately, game is not being built. It is sent as an empty object. Here's a JSFiddle.
What is my mistake?
Here's what I have so far:
function sendGame(req, res, sales, settings, categories) {
var game = new Object();
game.sales = sales;
game.settings = settings;
game.categories = categories;
JSONgame = JSON.stringify(game);
res.writeHead(200, {
'Access-Control-Allow-Origin': 'http://localhost',
'Content-Length': JSONgame.length,
'Content-Type': 'application/json'
});
res.write(JSONgame);
res.end();
console.log('Game: ' + JSON.stringify(game, null, 4));
console.log('--------------------------------------');
console.log('User ' + req.body.username + ' successfully retrieved game!');
}
function retrieveSales(req, connection, timeFrame) {
console.log('User ' + req.body.username + ' retrieving sales...');
connection.query('select * from sales_entries where date BETWEEN ? AND ?', timeFrame,
function (err, rows, fields) {
if (err) {
callback(new Error('Failed to connect'), null);
} else {
sales = [];
for (x = 0; x < rows.length; x++) {
sales.push(rows[x]);
}
//console.log('Sales: ' + JSON.stringify(sales, null, 4));
return sales;
}
});
}
retrieveCategories() and retrieveSettings() omitted for readability; they are the same as retrieveSales() mostly.
function gameSucceed(req, res) {
console.log('User ' + req.body.username + ' retrieving game...');
var timeFrame = [moment().days(0).format("YYYY-MM-DD HH:mm:ss"), moment().days(6).format("YYYY-MM-DD HH:mm:ss")];
var connection = createConnection();
connection.connect(function (err) {
if (err) return callback(new Error('Failed to connect'), null);
console.log('Connection with the Officeball MySQL database openned for game retrieval...');
var sales = retrieveSales(req, connection, timeFrame);
var settings = retrieveSettings(req, connection);
var categories = retrieveCategories(req, connection);
var all = q.all([sales, settings, categories]);
all.done(function () {
sendGame(req, res, sales, settings, categories);
});
});
}

Your problem is that you're not using promises. All your APIs use callbacks.
A promise is like a closed box:
A promise also has a method that opens the box, works on the value and returns another box on the value (also opening any additional boxes along the way). That method is .then:
In boxes, it does:
=>( . => ) =>
That is, it adds a handler that gets an open box and returns a box. Everything else just combines stuff. All .all does is wait for a list of promises to resolve, it is exactly like .then in the fact it waits for a result. Because promises are boxes, you can pass them around and return them which is very cool.
Generally:
Whenever you return from a promise handler (not a rejection), you are fullfilling it indicating normal flow continuation.
Whenever you throw at a promise handler, you are rejecting indication exceptional flow.
So basically in node speak:
Whenever you returned a null error and a response, you resolve the promise.
Whenever you returned an error and no response, you reject the promise.
So:
function myFunc(callback){
nodeBack(function(err,data){
if(err!== null){
callback(new Error(err),null);
}
callback(data+"some processing");
})
});
Becomes:
function myFunc(){
return nodeBack().then(function(data){ return data+"some processing"; });
}
Which I think is a lot clearer. Errors are propagated across the promise chain just like in synchronous code - it's very common to find synchronous analogs to promise code.
Q.all takes a list of promises and waits for them to complete, instead you want Q.nfcall to transform a callback based API to a promise one and then use Q.all on that.
That is:
var sales = Q.nfcall(retrieveSales,req, connection, timeFrame);
var settings = Q.nfcall(retrieveSettings,req, connection);
var categories = Q.nfcall(retrieveCategories, req, connection);
Q.nfcall takes a nodeback in the err,data convention and converts it to a promise API.
Also, when you do
return sales;
You are not really returning anything, since it returns synchronously. You need to use callback like in your error case or promisify it altogether. If you don't mind, I'll do it with Bluebird since it comes with much better facilities for dealing with these interop cases and does so much much faster, if you'd like you can switch promisifyAll for a bunch of Q.nfcall calls.
// somewhere, on top of file
connection = Promise.promisifyAll(connection);
// note I'm passing just the username - passing the request breaks separation of concerns.
var retrieveSales = Promise.method(username, connection, timeFrame) {
console.log('User ' + username + ' retrieving sales...');
var q = 'select * from sales_entries where date BETWEEN ? AND ?';
return connection.queryAsync(q, timeFrame).then(function(rows, fields){
return rows;
});
}
Note that suddenly you don't need a lot of boilerplate for making a query, you can use queryAsync directly instead if you'd like.
Now the code that wraps it becomes:
var gameSucceed = Promise.method(function gameSucceed(req, res) {
console.log('User ' + req.body.username + ' retrieving game...');
var timeFrame = [moment()....];
var connection = Promise.promisifyAll(createConnection());
return conn.connectAsync().then(function () {
console.log('Connection with the ...');
//sending req, but should really be what they use.
return Promise.all([retrieveSales(req,conn,timeFrame),
retrieveSettings(req,conn),
retrieveCategories(req,conn)]);
});
});
Now you can call sendGame(req, res, sales, settings, categories); outside of gameSucceed which doesn't hide what it does as much -
gameSucceed(req,res).spread(function(sales,settings,cats){
return sendGame(req,res,sales,settings,cats);
});

JavaScript nodejs mysql with queries in a loop

I feel a bit dumbstruck right now. I am fairly new to nodejs and javaScript and can't figure this one out. I guess it is because of the async nature of queries to mysql...
I made an example that shows my problem. I just want to cycle over a number of sql queries and do stuff with the results. for the sake of the example I just print out stuff. I know that I could use a single sql query like this "SELECT id, name FROM player WHERE id IN (1,2,3,4,5)" but this is not possible in the real application I am trying to write.
this is the relevant part of my nodejs app.js
var mysql = require("mysql");
var mysqlPool = mysql.createPool(conf.mysqlArbData);
for (var i = 0; i<5; i++){
mysqlPool.getConnection(function(err, connection) {
var detailSql = "SELECT id, name FROM player " +
"WHERE id = "+i;
if (err){
throw err;
}
connection.query(detailSql, function(err, detailRows, fields) {
connection.end();
console.log("detailSql="+detailSql);
if (err){
console.log("can't run query=" + detailSql +"\n Error="+err);
}
else{
console.log(detailRows[0].id + " " +detailRows[0].name);
}
});
});
};
Now the output:
web server listening on port 3000 in development mode
detailSql=SELECT id, name FROM player WHERE id = 5
5 Jyvaskyla
detailSql=SELECT id, name FROM player WHERE id = 5
5 Jyvaskyla
detailSql=SELECT id, name FROM player WHERE id = 5
5 Jyvaskyla
detailSql=SELECT id, name FROM player WHERE id = 5
5 Jyvaskyla
detailSql=SELECT id, name FROM player WHERE id = 5
5 Jyvaskyla
My question is, why do I get only the result for database entry with id=5? What needs to be changed in order to receive each individual result in the callback?

The problem is that getConnection is asynchronous and that Javascript doesn't have a block scope, which means that by the time the callback for getConnection is called, the i variable will point to the value of what it had last in the loop (which is 5).
You can use a trick to create a partial function (think of it as a function with the first argument already applied to it) for each turn of the loop, which will pass the current value of i as the first argument of the getConnection callback:
for (var i = 0; i<5; i++) {
mysqlPool.getConnection(function(i, err, connection) {
...
}.bind(mysqlPool, i));
};
FWIW, your code will open 5 connections (and perform 5 queries) to your database almost instantly (that's how asynchronous I/O works). That's probably not a big issue, but it's something worth realising if that 5 could get higher :)
Also, the for loop will generate [0, 1, 2, 3, 4], whereas in your example query, you write WHERE id IN (1, 2, 3, 4, 5).

For node, you can use let statement. It limit i scope in for loop.
for (let i = 0; i<5; i++) {
mysqlPool.getConnection(function(err, connection) {
console.log(i);
});
};

This is how I've been doing it. It's worked for UPDATE AND INSERT statements with 50,000+ items.
const mysql = mysql.createConnection({
host : 'HOST',
user : 'USER',
password : 'PASSWORD',
database : 'DATABASE_NAME',
port : PORT,
ssl : { ca : fs.readFileSync(process.cwd() + '/ssl/certs/ca-certificate.crt') },
});
async function processRequest(itemArray) {
//pre-processing of array (if needed...)
// initialize sql connection
mysql.connect();
const startProcess = await processItems(itemArray)
.catch(e => {
// end sql connection if there was an error running startProcess
mysql.end();
throw new Error(`Error # startProcess: ${e}`);
})
// end sql connection after startProcess is finished
mysql.end();
// optional return of startProcess. This occurs after processItems is done.
try {
if (startProcess.status == 'success') {
return {status: 'success', msg: 'function startProcess has compelted.';
}
}
catch (e) {
return {status: 'error', msg: 'function startProcess did not complete.', error: e;
}
function processItems(array){
return new Promise((resolve,reject) => {
for (let i in array) {
// perform DB operations inside loop
mysql.query(`UPDATE db SET ?`, array, (err, results, fields) => {
// exit if there's any sort of error
if (err) reject(err)
})
}
// resolve promise after loop completes
resolve({status: 'success'});
})
};
}
Here's the workflow:
mysql connection is opened (via mysql.connect())
processRequest is called (in my case, it's called from a Route)
the processItems function is called asynchronously
processItems returns a promise
processItems finishes where it either resolves the promise or rejects it
mysql connection is closed (via mysql.end())