Running Functions Synchronously in NodeJS (MongoDB Operations/Async.js) - javascript

I'm trying to do something fairly simple it seems in NodeJS - I want to run functions, one at a time. All of these functions have callbacks. I have outlined my code below, as well as the function that they run for further reference.
My problem is that the first two are working absolutely fine - one at a time, but the third iteration simply ignores the first two functions and just goes anyway. This is causing a real problem, since my program works with putting objects into a database, and it's causing duplicate objects.
The overall goal is to simply have each function run one at a time. Is there anything I'm missing here? Thanks so much for your help!
Please note that in the functions below, I have simplified all parameters to "args" for easier reading.
Calling the functions:
addNewProject(args);
addNewProject(args);
addNewProject(args);
Inside the functions, I run this:
function addNewProject(args) {
var info = args;
queue.push(function (done) {
loopThroughDetails(info, projID, 0, function () {
console.log('complete');
done(null, true);
});
});
}
This calls loopThroughDetails(), which is an integration to work with async.series():
function loopThroughDetails(info, projID, i, callback) {
if (i < 500) {
getProjectDetails(projID + "-" + i, function (finished) {
if (JSON.stringify(finished) == "[]") {
info.ProjID = projID + "-" + i;
DB_COLLECTION_NAME.insert(info, function (err, result) {
assert.equal(err, null);
callback();
});
} else {
i++;
loopThroughDetails(info, projID, i, callback);
}
});
}
}
And after calling all this, I simply use async.series to accomplish the task:
async.series(queue, function () {
console.log('all done');
});
What am I doing wrong here? Thanks so much for any help you can give! :)

Firstly, there are many methods to achieve what you are looking for and most are subjective. I like to use an array.shift method when iterating synchronously when possible. The concept goes something like this.
// say you have an array of projects you need to add.
var arrayOfProjects = [{name: "project1"}, {name: "project2"}, {name: "project3"}];
// This takes the first project off of the array and assigns it to "next" leaving the remaining items on the array.
var nextProject = function (array) {
// if there are items left then do work. Otherwise done.
if (array.length > 0) {
// shift the item off of the array and onto "next"
var next = array.shift();
addNewProject(next);
}
}
var addNewProject = function (project) {
// Do stuff with the project
console.log("project name: ", project.name);
// When complete start over
nextProject(arrayOfProjects);
}
// Start the process
nextProject(arrayOfProjects);
Here is a working Example
if you inspect the page you will see the projects logged to the console in order.

Related

How to wait for callback while iterating an array

I am working with a transnational framework within Javascript. So I need to wait for the previous query to finish before I move on. For example...
// Explicit this won't work because the length is not static
var i = [1,2,3]
doSomething(i[0], function(){
doSomething(i[1], function(){
doSomething(i[2], function(){
commitTransaction()
}
})
})
From this example I can't figure out a way to do this dynamically. It feels like a queue/recursion problem but I can't seem to crack it.
Does anyone else have an idea? I can also wrap in promises so that is an option as well, although that seems less synchronous.
Use async.eachSeries. So your code would translate to:
var transaction = {...};
async.eachSeries([1, 2, 3], function(value, callback) {
doSomething(value, transaction, callback);
}, function(err) {
if(err) throw err; // if there is any error in doSomething
commitTransaction(transaction);
});
jsFiddle Demo
I would suggest making a queue to do this. It would take the array, the generic callback function and a final function to callback with. Basically, the best way to accomplish this is to allow your functions to expect to have values injected.
The core assumption is that it is understood the caller will allow their callback function to have the current value and next callback function injected. That basically means we will end up with a function I have named queueAll which looks like this
function queueAll(arr,cbIteration,final){
var queue = [function(){ cbIteration(arr[arr.length-1],final) }];
for(var i = arr.length-2; i > 0; i--){
(function(next,i){
queue.unshift(function(){ cbIteration(arr[i],next) });
})(queue[0],i)
}
cbIteration(arr[0],queue[0]);
}
It takes the final call, places it in the queue, and then iterates, placing subsequent callback functions in the queue with the current value closed over, as well as closing over the front of the queue which at that point is the next call back. It is fairly simple to use. Pass it an array, a callback which expects values to be injected, and a final function.
In your case it would look like
queueAll(i,function(item,next){
doSomething(item,next);
},function(){
commitTransaction();
});
Stack Snippet Demo
//## <helper queue>
function queueAll(arr,cbIteration,final){
var queue = [function(){ cbIteration(arr[arr.length-1],final) }];
for(var i = arr.length-2; i > 0; i--){
(function(next,i){
queue.unshift(function(){ cbIteration(arr[i],next) });
})(queue[0],i)
}
cbIteration(arr[0],queue[0]);
}
//## </helper queue>
//## <user defined functions>
function doSomething(val,callback){
setTimeout(function(){
console.log(val);
callback();
},val*10);
}
function commitTransaction(){
console.log("commit");
}
//## </user defined functions>
//## <actual use>
var arr = [10,20,30];
queueAll(arr,function(item,next){
doSomething(item,next);
},function(){
commitTransaction();
});
//## </actual use>
Actually, I think promises are exactly what you're looking for. But for a traditional callback approach, consider the following:
var state = false,
doSomething = function (value, callback) {
/* do stuff with value */
if (!state)
doSomething(newValue, callback);
else
callback();
};

Underscore _.each callback when finished?

Is there a callback for when underscore is finished it's _.each loop because if I console log immediately afterwards obviously the array I am populating with the each loop is not available. This is from a nested _.each loop.
_.each(data.recipe, function(recipeItem) {
var recipeMap = that.get('recipeMap');
recipeMap[recipeItem.id] = { id: recipeItem.id, quantity: recipeItem.quantity };
});
console.log(that.get('recipeMap')); //not ready yet.
The each function in UnderscoreJS is synchronous which wouldn't require a callback when it is finished. One it's done executing the commands immediately following the loop will execute.
If you are performing async operations in your loop, I would recommend using a library that supports async operations within the each function. One possibility is by using AsyncJS.
Here is your loop translated to AsyncJS:
async.each(data.recipe, function(recipeItem, callback) {
var recipeMap = that.get('recipeMap');
recipeMap[recipeItem.id] = { id: recipeItem.id, quantity: recipeItem.quantity };
callback(); // show that no errors happened
}, function(err) {
if(err) {
console.log("There was an error" + err);
} else {
console.log("Loop is done");
}
});
Another option is to build your callback function into the each loop on the last execution:
_.each(collection, function(model) {
if(model.collection.indexOf(model) + 1 == collection.length) {
// Callback goes here
}
});
Edit to add:
I don't know what your input/output data looks like but you might consider using _.map instead, if you're just transforming / rearranging the contents

Windows 8 Javascript asynchronous programming

I want to iterate through a list and rename a few files in Windows 8 / Javascript. Therefore, I wrote a function called "renameFile" and I call this function within a loop, like this:
list.forEach(function (value, index, array) {
var filename = index;
var newfilename = index+1;
renameFile(filename, newfilename);
});
function renameFile(filename, newfilename) {
Windows.Storage.ApplicationData.current.localFolder.getFileAsync(filename).then(function (sampleFile) {
sampleFile.renameAsync(newfilename).done(
function complete(result) {
},
function error(error) {
console.log("error" + error);
}
);
});
}
The problem is: The rename function is asynchronous and it seems that sometimes, the renaming works and sometimes not: In most cases, only the first element of the list is renamed. How can I tell the loop to wait, till the rename-process of the first item in the list is finished and then go an with the second, third, ... item?
Cheers
Well I did a fast lookup and you're out of luck. It seems like the StorageFolder class only has Asyn functions for what you're looking for. But it's not the end.
The easiest solution I have to do use a recursive function. And call the function with a different index.
function renameFile(filename, newfilename, list, index) {
Windows.Storage.ApplicationData.current.localFolder.getFileAsync(filename).then(function (sampleFile) {
sampleFile.renameAsync(newfilename).done(
function complete(result) {
renameList(list, index);
},
function error(error) {
console.log("error" + error);
}
);
});
function renameList(list, index) {
if(index >= list.length) return;
renameFile(index, index+1, list, index+1);
}
renameList(list, 0);
It's not very clean but it should work, this will force the code to be synchrone since you're calling from within a callback.

Functionscope and Callbacks in Node.js

so i have to calculate some share in a loop. In every iteration of that loop i have to get a variable called rent from an array. So i devided the calculate function from the database stuff.
var calculate = function() {
while(count < 100) {
var share = 50;
var shareArray = [];
for(var i = 0; i < 100; i++) {
var pension = share*2; // mathematical stuff
// Gets a rent from a database and returns it in a callback
getRent(modules, share, function(rent) {
share = rent*foo; // some fancy mathematical stuff going on here
// I need to get the share variable above out of its function scope
});
// I need the share variable right here
shareArray.push(share); // the value of share will be for i = 0: 50, i= 1: 50 ...
// This is not what i want, i need the share value from getRent()
}
count++;
}
}
Now as you may see i am presented with the following trouble. Because I'm working in node.js, the only way to get the rent variable from the modules array is through this callback function called getRent(). The thing is, i need the share value after this step but outside of getRent().
Is there any way i can do this?
This is the getRent() - Function:
var getRent = function(modules, share, callback) {
// Searching for a fitting rent in the modules array
// Just assume this is happening here
callback(rent);
};
So the question is: How can i "return" share:
getRent(modules, share, function(rent) {
share = rent*foo; // some fancy mathematical stuff going on here
// I need to get the share variable above out of its function scope
});
in any way?
If getRent is async there's no way to get the result synchronously. Fundamentally you don't know the value that getRent will end up supplying to its callback until it finally returns it. So it isn't a question of function scope, its a question of timing. You just have to wait for getRent to do its thing before you can get the value for rent. You need to refactor your code so that calculate is also async.
Something like:
// Refactor calculate to be async:
function calculate(cb) {
var data = [];
for ( var i=0; i<100; i++ ) {
getRent(function (rent) {
data.push(rent);
if ( data.length === 100 ) cb(data);
});
}
}
// And then use it async:
calculate(function (data) {
// data array arrives here with 100 elements
});
The above answer is perhaps similar to how you might achieve it with vanilla JS. Using the async library like miggs suggests is probably a good idea in the long run. But like I said, if you use vanilla JS, or the async lib, there's no getting away from the fact that you'll have to refactor towards asynchronicity both in this code and the code that calls it.
You want to use the whilst method of the async library (npm install async) to simplify this:
var count = 0;
var shareArray = [];
async.whilst(
function () {
return count < 100;
},
function (next) {
count++;
getRent(function(rent) {
// What does modules do anyway??
// Dont know where foo comes from...
shareArray.push(rent*foo); // some fancy mathematical stuff going on here
next();
});
},
function (err) {
console.log(shareArray);
// Do sth. with shareArray
}
);
If it was OK for you to request all 100 call in parallel, you could also use the parallel function.

Underscore js times series

I'm using UnderscoreJs with nodejs and have a need for the _.times() method. times() will invoke a function X number of times
This works as expected, however I need to iterate in a series, instead of in parallel which this appears to be doing.
Any idea if there's a way to use this in series w/ callback methods?
Given something like this:
function f() {
some_async_call({ callback: function(err, results) {...})
}
_(3).times(f);
Then the three f calls will happen in series but the some_async_call calls won't necessarily happen in series because they're asynchronous.
If you want to force your calls to run in series then you need to use the callback on the async call to launch the next one in the series:
function f(times, step) {
step = step || 0;
some_async_call({
callback: function(err, results) {
// Do something with `err` and `results`...
if(step < times)
f(times, step + 1);
}
});
}
f(3);
That approach will execute the three some_async_calls in series but, alas, the initial f(3) will return immediately. One solution to that problem is, of course, another callback:
function f(from_n, upto, and_finally) {
some_async_call({
callback: function(err, results) {
// Do something with `err` and `results`...
if(from_n < upto)
f(from_n + 1, upto, and_finally);
else
and_finally();
}
});
}
f(0, 3, function() { console.log('all done') });
Where does _.times in with all this? No where really. _.times is just a for loop:
_.times = function(n, iterator, context) {
for (var i = 0; i < n; i++) iterator.call(context, i);
};
_.times exists for completeness and to allow you to add for loop when using _.chain. You could probably shoe-horn it in if you really wanted to but you would be making a big ugly mess instead of simplifying your code.
You could use 250R's async idea but you'd have to build an array of three functions but _.range and _.map would be more appropriate for that than _.times:
// Untested off the top of my head code...
function f(callback) {
some_async_call({
callback: function(err, results) {
// Deal with `err` and `results`...
callback();
}
});
}
var three_fs = _(3).range().map(function() { return f });
async.series(three_fs);
But you still have to modify f to have a callback function and if you're always calling f three times then:
async.series([f, f, f]);
might be better than dynamically building the array with _.range and _.map.
The real lesson here is that once you get into asynchronous function calls, you end up implementing all your logic as callbacks calling callbacks calling callbacks, callbacks all the way down.
This async library might get you started
https://github.com/caolan/async#series
Or if you want to do it yourself, the idea is to do recursive calls after each function callback is called, here's the source code https://github.com/caolan/async/blob/master/lib/async.js#L101

Categories

Resources