I'm not very experienced with Nodejs and its async ways.
I'm trying to query different databases of redis.
I have a simple function to get a key from a redis database:
function get_key(client, key, db, callback) {
if (key) {
client.select(db, function(e, s) {
if (e) {
console.log('client.select err: ' + e);
} else if (s) {
client.get(key, function(e, s) {
callback(e, s);
return s;
});
}
});
}
return
}
And I'm using it to query multiple databases like this:
get_key(client, key1, 0, function(e, s) {
if (s) {
// do stuff with s;
} else {
debug('e: ' + e);
}
});
get_key(client, key2, 1, function(e, s) {
if (s) {
// do stuff with s;
} else {
debug('e: ' + e);
}
});
but it is not working working. But If I create different clients for each query, it works. I have queries in over a dozen databases, and wouldn't prefer to create a new client everytime I do a query.
Is there a better way to do this? the "node" way that I'm not familiar with? thank you.
Async functions take a callback function as a parameter. The function returns close to immedately, and the callback is called after the work is completed rather than blocking the caller. So your first call returns before its calls to redis are complete and the second call is being made immediately, which probably screws up the first call because they share the same client.
If you only have two calls to make, the simple answer is to put the second call within the callback of the first call. If you need to make a lot of async calls in series, look into using promises or some of the available libraries like async.
Related
I have an express.js app that needs to run a script on the server in order to derive some values using functions later. Here's the gist of it:
shell.exec(commandString);
readFolder();
renderPage();
Essentially, I need to run a script on the server, then run the second function, then run the third function. These need to happen subsequently, but it seems that javascript moves on ahead with the the second and third function no matter what I do. I've tried promises, async, callbacks. All of which I only partially understand and seem to get zero progress.
I will admit that I am a javascript novice. I am working on a project with others and this task fell to me. I doubt this is the best way to accomplish our ultimate goals, but I am left with little choice. please help.
I'll put the entire post here for reference:
//Run script when post is rec'd from root and send to results page
app.post("/", (req, res) => {
var commandString;
//take values and create complete command for Astrum script
commandString = 'bash /home/astrum/Main/Astrum.sh -s ' + req.body.speed + ' -h ' + req.body.host + ' -u ' + req.body.username + ' -p ' + req.body.password;
//execute command in shell
shell.exec(commandString);
readFolder();
renderPage();
//Iterate thru filenames to create arrays for links and link labels
function readFolder() {
fs.readdir('./reports/html/', (err, files) => {
//variable & method for links to html records pages
ipAddressesLink = files; //this is initialized earlier, globally
//variable and method to remove file extension for link labels in pug
ipAddresses = files.map(removeExtension); //this is initialized earlier, globally
});
}
//function to remove last five characters of each element
function removeExtension(value) {
return value.substring(0, value.length - 5);
};
//function to render the page
function renderPage() {
res.render("results", {ipAddressesLink, ipAddresses, title: 'Results'});
}
res.end();
});
You could write it this way:
shell.exec(commandString, (error, stdout, stderr) => {
// Calling the 1st function after shell command is executed
readFolder();
});
function readFolder() {
fs.readdir('./reports/html/', (err, files) => {
// Some stuff
...
// Calls the 2nd function after fs is done reading files
renderPage();
});
}
function renderPage() {
const options = { ... }; // IP addresses etc.
res.render(
"results",
options,
// Calls the final function after render is finished
sendResponse
);
}
function sendResponse(err, html) {
// Sends the response. It’s possible that res.send() is the better solution here
res.end();
}
It’s just the general structure of the callback chain, definitely not the cleanest one. If you want better code structure and readability try switching to async / await syntax.
Is shell here the child_process module? If it is then you can pass an optional callback argument and call your functions from there.
shell.exec(commandString, (error, stdout, stderr) => {
const files = readFolder();
renderPage(files);
});
function readFolder() {
...
return fs.readdirSync(files);
}
function renderPage(files) {
...
}
I understand the concept of callbacks in Javascript. For example this sort of code makes complete sense to me -
function processArray(arr, callback) {
var resultArr = new Array();
for (var i = arr.length-1; i >= 0; i--)
resultArr[i] = callback(arr[i]);
return resultArr;
}
var arr = [1, 2, 3, 4];
var arrReturned = processArray(arr, function(arg) {return arg * -1;});
// arrReturned would be [-1, -2, -3, -4]
I understand that callback is just a placeholder function which could be any function that we write later. However there are certain types of callbacks that I have no clue how to write. For example -
function processData (callback) {
fetchData(function (err, data) {
if (err) {
console.log("An error has occured. Abort everything!");
callback(err);
}
data += 1;
callback(data);
});
}
How is the callback to fetchData defined? How does the program know what is err and what is data?
Normally in node.js the structure of a callback is basically like this function(err, result)
This means that the first argument is the error and the second argument is the data. when you have no error you simply pass a null value. This is a standard already.
The second code you posted you should change it like follows:
function processData (callback) {
fetchData(function (err, data) {
if (err) {
console.log("An error has occured. Abort everything!");
callback(err);
}
data += 1;
callback(null, data);
});
}
In the above case, if you did not want to increase the databy 1 you could simply call fetchData(callback).
My two cents :
In JavaScript everything is object and so does function. So functions can be passed as argument to other functions. This enables you to use callback as arguement and then using those callbacks in your api implementation to execute user's task after accomplishing API's task. Here if you would try to re-read your code then you will realize your misconception. Other users have also quoted the same thing here.
EDIT
function fetchdata(callback){
var err = "This is Error message";
var data = {samplekey : "samplevalue"};
callback(err, data); // here in the implementation of fetchdata
// we are making those data and feeding to the callback function.
}
function callback(err, data){
// Implementation of call back
}
Here for making readable purpose I have put callback function separate. You can pass it simply as well.
I think you do not understand JavaScript completely.
I got this example from here
function doSomething(callback) {
// ...
// Call the callback
callback('stuff', 'goes', 'here');
}
function foo(a, b, c) {
// I'm the callback
alert(a + " " + b + " " + c);
}
doSomething(foo);
I'm new to js.
I am using express for node js, and mongoose as a mongo orm.
function direct_tags_search_in_db(tags){
var final_results = [];
for (var i=0; i<tags.length; ++i) {
var tag = tags[i];
Question.find({tags: tag}).exec(function(err, questions) {
final_results.push(questions);
if (i == tags.length -1 ){
return final_results;
}
});
}
};
I get empty results, because of the asynchronously of the find. But I don't know what the best approach for this.
Appriciate a little help, thanks.
You will often find that methods such as Question.find().exec that accept a function as an argument are async. It is especially common for methods that perform network requests or file system operations. These are most commonly referred to as a callback. That being the case, if you would like something to occur when the async task(s) complete, you need to also implement a callback.
Also, it is possible that your reference to tag is being changed in a way that is likely undesired. There are a number of solutions, here is a simple one.
function direct_tags_search_in_db(tags, callback){
var final_results = [];
// Array.forEach is able to retain the appropriate `tag` reference
tags.forEach(function(tag){
Question.find({tags: tag}).exec(function(err, questions) {
// We should be making sure to handle errors
if (err) {
// Return errors to the requester
callback(err);
} else {
final_results.push(questions);
if (i == tags.length -1 ){
// All done, return the results
callback(null, final_results);
}
}
});
});
};
You will notice that when we implement our own callback, that we follow the same common pattern as the callback for Question.find().exec(function(err, result){}); -- first argument a potential error, second argument the result. That is why when we return the results, we provide null as the first argument callback(null, final_results);
Quick example of calling this function:
direct_tags_search_in_db([1, 2, 3], function(err, results){
if (err) {
console.error('Error!');
console.error(err);
} else {
console.log('Final results');
console.log(results);
}
});
Another option for solving various async goals is the async module, promises, or otherwise.
Beginning with express and mongoose i often need to do some batch operations on collections.
However it usually involves callbacks which is a pain given how concurrency is coded in nodejs.
so basically
//given a collection C
var i = 0;
var doRecursive = function(i){
if(i<C.length){
C[i].callAsync(err,result){
i=+1;
return doRecursive(i);
}
}else{
return done();
}
}
doRecursive(i);
Now i dont remember what is the max stack before i get a stackover flow with node , but i guess with 10 000 elements , it wont do.
I wonder if there are other ways to handle this, if yes , what are they?
thanks
If the goal is to iterate an collection asynchronously, there are numerous control flow libraries available.
A good example is async and its reduce function:
async.reduce(C, 0, function (memo, item, callback) {
item.callAsync(function (err, result) {
if (err) {
callback(err);
} else {
callback(null, memo + result);
}
});
}, function (err, result) {
// ...
});
Note: It's not entirely clear what value you wanted to get from doRecursion, so this just uses addition for an example.
i think you can simply self-iterate instead of true recursion, since you're not drilling into a deep object:
function doRecursive (C, i){
i=i||0;
if(i<C.length){
C[i].callAsync(err, function(result){
doRecursive(C, ++i);
});
}else{
done();
}
};
doRecursive(C);
this does not create a tall stack if the code functions as labeled.
i localized C so that it executes faster and is potentially re-usable on other collections.
the pattern also makes it easy to defer it for long-running operations, just by changing
doRecursive(C, ++i);
to
setTimeout( doRecursive.bind(this, C, ++i), 50 );
I am having a synchronisation issue in Javascript. Code below. When I make the call to get mutual friends, although my function is still filling the array through the API callback, the printing of "Quite a bunch: 0" happens before the printing of the - console.log(friendID +" -> " + mutualfriends.data.length);
I know this must be a callback / asynch issue but I have no idea how to deal with it. I'm filling the array for a reason - need it to be filled for the next part.
code:
function getMutualFriends(friendID)
{
//console.log(response.data[i].id);
try{
FB.api('/me/mutualfriends/'+friendID, function(mutualfriends) {
//console.log(mutualfriends);
//console.log(mutualfriends.data.length);
console.log(friendID +" -> " + mutualfriends.data.length);
mutualFriendsList.push([friendID,mutualfriends.data.length]);
});
}
catch(err){
console.log('error caught: ' +err);
}
}
function getFriendsList()
{
FB.getLoginStatus(function(response){
FB.api('/me/friends', function(response) {
for(i=0; i<response.data.length;i++)
{
var friendID = response.data[i].id;
console.log(friendID);
friendsList.push(friendID);
}
console.log('Interesting, we gathered: '+friendsList.length+' friends,');
console.log('lets check mutual friends now');
for(j=0; j<friendsList.length;j++)
{
getMutualFriends(friendsList[j]);
}
console.log('Quite a bunch: ' + mutualFriendsList.length);
});//friends
});
}
You'll probably want to turn your "post-condition" code into a callback.
So where you now have:
for(j=0; j<friendsList.length;j++)
{
getMutualFriends(friendsList[j]);
}
console.log('Quite a bunch: ' + mutualFriendsList.length);
You'll want something like:
for(j=0; j<friendsList.length;j++)
{
getMutualFriends(friendsList[j], function(mutualFriendsList) {
console.log('Quite a bunch: ' + mutualFriendsList.length);
});
}
When you set it up like this, your getMutualFriends function can call the callback once it got the result:
function getMutualFriends(friendID, callback)
{
FB.api('/me/mutualfriends/'+friendID, function(mutualfriends) {
mutualFriendsList.push([friendID,mutualfriends.data.length]);
callback(mutualFriendsList);
});
}
This will do the callback once for every call to getMutualFriends. If you want the callback to only trigger once for all friends, you'll need to expand the concept a bit further.
Update:
You could combine the above "per friend" callback with #RGDev's condition to detect the last friend:
for(j=0; j<friendsList.length;j++)
{
getMutualFriends(friendsList[j], function(mutualFriendsList) {
if (mutualFriendsList.length == friendsList.length) {
console.log('Quite a bunch: ' + mutualFriendsList.length);
}
});
}
getMutualFriends is doing an async call on his own (by calling FB.api) so you cannot print the mutualFriendsList until it's done (actually, you are doing may calls to getMutualFiends so your mutualFriendsList won't be accurate until all of them finished their async processing).
Can't you rely on that collection to be async? I don't know what you are doing with that, but maybe (if you are drawing it in the screen by example) you can issue a redraw of that component every time the callback of FB.api('/me/mutualfriends/'+friendID, ...) is done.
I had a similar issue to you in the past, I solved it by using a backbone.js' collection instead of a simple javascript array. Then I bound a listener to the add event of the collection and print the qty of friends every time an object was added to it.
Good luck,
You could use a while loop to wait until the mutualFriendsList.length is equal to friendsList.length. which would mean that all of the ajax functions return functions had been fulfilled.
It would look something like:
while(mutualFriendsList.length != friendsList.length)
{
console.log("Still Waiting");
}
console.log('done')
// do your data processing
A little inelegant, but it should work