I can't seem to find a general solution to this problem, even though I feel like I can't be the first to come across this. I also suspect it might be I'm taking the wrong approach in the first place here. Let me know.
I have an Expressjs App the interacts with various APIs (mostly OAuth2). Once a request comes in the App checks if it has an access token to fetch data from an API. In case the token is expired it will request a new one.
Now, when my App receives a second request in the meantime, requiring the exact same API, I want to avoid making a second call for an access token.
What I do is use a "Collector" where callbacks can be stored for a given key. The first request to store a callback for a key gets a collector callback to invoke once it has finished the task. All subsequent callbacks are enqueued and called later on with the collector callback.
This is the simple Collector class (CoffeeScript)
# Collect callbacks for tasks and execute all once the job is done
module.exports = class Collector
constructor: ()->
#tasks = {}
add: (key, callback)->
unless #tasks[key]
# Initiate callback list for the key with first callback
#tasks[key] = [callback]
return ()=>
# Call all collected callbacks for the key
(callback.apply {}, arguments for callback in #tasks[key])
# Reset callback list
#tasks[key] = null
else
# Add callback to existing list
#tasks[key].push callback
return false
I'm not sure if storing the callbacks inside this class is the right way, but to use a database (Redis) I would have to find a way to store callbacks…
Is there a better way to invoke multiple callbacks once a job is done?
Why don't you just aggregate your callbacks into an array which you cycle through, executing each contained function when your original call is complete?
It could be as simple as:
var dones = [];
dones.push(function (err, res) { console.log('hoo'); });
dones.push(function (err, res) { console.log('ray!'); });
function whenDone(err, res) {
_.each(dones, function (done) { done(err, res); } });
}
myWhateverFunction(whenDone);
You can wrap this into whatever data structure you want, if you want to make it prettier.
I don't have a specific answer to your problem, but you should check out the async module. I think it's a step in the right direction: https://github.com/caolan/async
Related
I have run into quite an annoying issue with my Javascript code due to its synchronous nature, in short the problem is I need to dump a load of data into an array.
This array is then written directly into a JSON file.
function main(callback) {
myarrary = []
//run some SQL query via stored procedure
for (result in query) {
myarray.push({
//my data is entered here
})
}
let data = JSON.stringify({myarray: somedata}, null, 4)
fs.writeFileSync('testingjson.json', data)
callback();
}
main(another_function)
My issue is that the data dump and callback are executed immediately, as such no data is transferred to the file and 'another_function` relies on data being there so that doesn't work either.
Therefore I need the dump to wait for the loop to finish appending myarray
I have also tried entering the dump and callback into my for loop however they execute on the first iteration.
Any help is appreciated!
Functionally your current code looks fine, except that you should not call the function as callback unless it is an asynchronous function.
Callbacks are usually associated with asynchronous API's. It is useful if you want main(...) to return immediately so that you can just continue with something else. Since that is the default behavior, the API is
fs.writeFile(filename, data[, options], callback)
So to convert your program to async, you need:
fs = require("fs");
function main(callback) {
// Oops. Why is query executed synchronously?
query = sqlSync() ;
fs.writeFile('delete.json', "Data", callback);
}
main(() => console.log("Done"))
You can also convert SQL calls to async if your API's support that (since it is an I/O operation).
Once you are comfortable with normal callbacks, you can try using promise or async-await as mentioned here.
if you have two methods who are returning callbacks you can use Promise.all([]). Here in Promise.all() you can write your database calls or other functions which are resolving/rejecting anything. From there you can get them in res object.
This is something that has been bothering me and I cant seem to find a straight answer.
Here is a form of Node function that I use a lot, it handles a web request and does a bit of IO:
function handleRequest(req, res) {
doSomeIo()
.then(function(result) {
res.send(result)
})
}
This function gets called and the res parameter is set to the current response object and it goes of into the land of IO, while its playing out there a second request comes through and sets the res to a new object? Now the first request comes back from IO, and uses the res object, is this now the first instance or the second i.e. does node essentially make a separate copy of everything each time the handleRequest function is called, with its own parameter values or is there only one instance of it and its parameters? Are the parameters in the the above function safe in an async environment or would it be better to do something like this:
function handleRequest(req, res) {
doSomeIo()
.then(function(res) {
return function(result) {
res.send(result)
}
}(res))
}
Or am I just completely ignorant of how Node and Java Script works, which seems pretty likely.
You don't have to worry at all about this case. The req object contains information about the HTTP request that the server received. Everything is sandboxed in per request basis. This will be helpful for you: What are "res" and "req" parameters in Express functions?
You can expect the current req and res object to remain the same among multiple events (I/O responses are essentially events) unless you do something to them, or you have other middleware that does something. There's no need to do anything like your second code snippet.
If you're new to JavaScript and/or the concept of closures, that's probably why you're uneasy with the syntax.
Each call to the function handleRequest() will not use the same variable values of previous calls to handleRequests(). An easy way to see this behavior would be to log the number of times the method was called, and the value of an incrementer in handleRequest().
Example:
In app.js (or whatever js file you initialize your server in), add the following:
var app = express(),
calls = 0; app.set('calls', calls);
Then in your handler add the following:
function handleRequest(req, res) {
doSomeIo()
.then(function(res) {
req.calls++;
return function(result) {
res.send(req.calls)
}
}(res))
}
You should notice that each call to the endpoint, no matter how quickly you make each call, count increases by 1 each time (get ready for race conditions).
I'm trying to sanitize inputs for an asynchronous function. Specifically, given a list of credentials for an API, I'm trying to filter out which ones are invalid by sending a test request to the API and examining the result for each one.
The problem I'm facing is this: I would like to collect the invalid keys into a single list. I would normally use the async library to execute the requests in sequence, using the series function. But, from the documentation:
If any functions in the series pass an error to its callback, no more functions are run, and callback is immediately called with the value of the error.
This isn't the desired behavior: I want to collect the errors in place of the responses (or both of them). Is this possible, using this library, without changing the way I'm interacting with the API?
The solution to this problem ended up being sort of hacky, but it works fine. I had a list of credentials and an async function apiCall, which looked like this:
var apiCall = function(arg, callback){
...
}
and the solution was to use mapSeries from async, but flip the callback arguments, like this:
async.mapSeries(credentials, function(creds, callback){
apiCall(creds, function(err, res){
callback(null, err);
});
},
function(nil, errors){
console.log(_.compact(errors));
});
where _.compact removes falsy elements from the array, getting rid of the nulls that non-error responses returned. This got me exactly what I was looking for.
I am quite new (just started this week) to Node.js and there is a fundamental piece that I am having trouble understanding. I have a helper function which makes a MySQL database call to get a bit of information. I then use a callback function to get that data back to the caller which works fine but when I want to use that data outside of that callback I run into trouble. Here is the code:
/** Helper Function **/
function getCompanyId(token, callback) {
var query = db.query('SELECT * FROM companies WHERE token = ?', token, function(err, result) {
var count = Object.keys(result).length;
if(count == 0) {
return;
} else {
callback(null, result[0].api_id);
}
});
}
/*** Function which uses the data from the helper function ***/
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
// this works
console.log(result);
});
// the problem is that I need result here so that I can use it else where in this function.
});
As you can see I have access to the return value from getCompanyId() so long as I stay within the scope of the callback but I need to use that value outside of the callback. I was able to get around this in another function by just sticking all the logic inside of that callback but that will not work in this case. Any insight on how to better structure this would be most appreciated. I am really enjoying Node.js thus far but obviously I have a lot of learning to do.
Short answer - you can't do that without violating the asynchronous nature of Node.js.
Think about the consequences of trying to access result outside of your callback - if you need to use that value, and the callback hasn't run yet, what will you do? You can't sleep and wait for the value to be set - that is incompatible with Node's single threaded, event-driven design. Your entire program would have to stop executing whilst waiting for the callback to run.
Any code that depends on result should be inside the getCompanyId callback:
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
//Any logic that depends on result has to be nested in here
});
});
One of the hardest parts about learning Node.js (and async programming is general) is learning to think asynchronously. It can be difficult at first but it is worth persisting. You can try to fight and code procedurally, but it will inevitably result in unmaintainable, convoluted code.
If you don't like the idea of multiple nested callbacks, you can look into promises, which let you chain methods together instead of nesting them. This article is a good introduction to Q, one implementation of promises.
If you are concerned about having everything crammed inside the callback function, you can always name the function, move it out, and then pass the function as the callback:
getCompanyId(token, doSomethingAfter); // Pass the function in
function doSomethingAfter(err, result) {
// Code here
}
My "aha" moment came when I began thinking of these as "fire and forget" methods. Don't look for return values coming back from the methods, because they don't come back. The calling code should move on, or just end. Yes, it feels weird.
As #joews says, you have to put everything depending on that value inside the callback(s).
This often requires you passing down an extra parameter(s). For example, if you are doing a typical HTTP request/response, plan on sending the response down every step along the callback chain. The final callback will (hopefully) set data in the response, or set an error code, and then send it back to the user.
If you want to avoid callback smells you need to use Node's Event Emitter Class like so:
at top of file require event module -
var emitter = require('events').EventEmitter();
then in your callback:
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
// this works
console.log(result);
emitter.emit('company:id:returned', result);
});
// the problem is that I need result here so that I can use it else where in this function.
});
then after your function you can use the on method anywhere like so:
getCompanyId(token, function(err, result) {
// this works
console.log(result);
emitter.emit('company:id:returned', result);
});
// the problem is that I need result here so that I can use it else where in this function.
emitter.on('company:id:returned', function(results) {
// do what you need with results
});
just be careful to set up good namespacing conventions for your events so you don't get a mess of on events and also you should watch the number of listeners you attach, here is a good link for reference:
http://www.sitepoint.com/nodejs-events-and-eventemitter/
Here is my problem i have one worker running that gathers up some data and then saves it to a schema. In those schemas i have post middleware functions that happen on a save. The problem is that my worker does not wait for the post middleware to finish working before it loops and saves again to the same schema. As my worker continues to run it slows down substantially as it queues up those functions over and over again before this get to finish. Those post middleware functions also save to other schemas which also execute there own post functions. SO as i you can tell that starts to really slow down as i am dealing with hundreds of thousands of objects being saved.
Example of what my code kind of looks likes:
Worker
tranFunc = (stuff, Object) ->
newObject = Object
newObject = new Object newObject
newObject.save (err) ->
The Save now jumps into the schema where some more stuff happens, but my worker loops before that one is done and keeps going
Schema
ObjectSchema.post("save", function() {
DOING STUFF
I have read that i could maybe use something like a next() as a callback to the worker but my attempts at using it were not successful.
I don't think I can answer this question in its current form but the basic idea is that
Async functions should use callbacks to communicate when they are done
Functions that call async code are be async themselves.
So in your case, it looks like the worker function is violating this principle, since its calling an async function without being async itself. You would fix this by adding a return callback as a parameter and calling said callbabk after everything is done (that is, inside the callback for the last async method you call)
//sort of like this:
tranFunc = (stuff, Object, doneCb) ->
newObject = Object
newObject = new Object newObject
newObject.save (err) ->
doneCb( /*put return values here as appropriate*/ )
Now whoever is calling tranFunc will be able to know when the inner save method has finished.