This is something that has been bothering me and I cant seem to find a straight answer.
Here is a form of Node function that I use a lot, it handles a web request and does a bit of IO:
function handleRequest(req, res) {
doSomeIo()
.then(function(result) {
res.send(result)
})
}
This function gets called and the res parameter is set to the current response object and it goes of into the land of IO, while its playing out there a second request comes through and sets the res to a new object? Now the first request comes back from IO, and uses the res object, is this now the first instance or the second i.e. does node essentially make a separate copy of everything each time the handleRequest function is called, with its own parameter values or is there only one instance of it and its parameters? Are the parameters in the the above function safe in an async environment or would it be better to do something like this:
function handleRequest(req, res) {
doSomeIo()
.then(function(res) {
return function(result) {
res.send(result)
}
}(res))
}
Or am I just completely ignorant of how Node and Java Script works, which seems pretty likely.
You don't have to worry at all about this case. The req object contains information about the HTTP request that the server received. Everything is sandboxed in per request basis. This will be helpful for you: What are "res" and "req" parameters in Express functions?
You can expect the current req and res object to remain the same among multiple events (I/O responses are essentially events) unless you do something to them, or you have other middleware that does something. There's no need to do anything like your second code snippet.
If you're new to JavaScript and/or the concept of closures, that's probably why you're uneasy with the syntax.
Each call to the function handleRequest() will not use the same variable values of previous calls to handleRequests(). An easy way to see this behavior would be to log the number of times the method was called, and the value of an incrementer in handleRequest().
Example:
In app.js (or whatever js file you initialize your server in), add the following:
var app = express(),
calls = 0; app.set('calls', calls);
Then in your handler add the following:
function handleRequest(req, res) {
doSomeIo()
.then(function(res) {
req.calls++;
return function(result) {
res.send(req.calls)
}
}(res))
}
You should notice that each call to the endpoint, no matter how quickly you make each call, count increases by 1 each time (get ready for race conditions).
Related
I am writing an express application using
NodeJS v8
express (latest version)
After looking at the onHeaders module and finding out how the module rewrites the HTTP head, I wanted to make use of that function of JavaScript.
I wanted to write a small session system using my SQL server. I am aware of the session module from express, but this module is not able to handle the specific tables and customization, I need.
For convenience reasons I wanted the session to be inserted into the request before the controllers and saved after all controllers finished. (e.g. the writeHead method has been called)
My code in the session looks like:
core = async function(req, res, next) {
res.writeHead = hijackHead(res.writeHead); // Hijack the writeHead method to apply our header at the last
}
//[...](Omitted code)
hijackHead = function(writeFunction) {
let fired = false;
return function hackedHead(statusCode) {
if ( fired ) {
return;
}
//[...](Omitted code)
debug("Session data has changed. Writing");
sessionManager.storeSessionData(session.identifier, session).then(() => { // Promise taking ~60ms to resolve
debug("Finished writing...");
callMe(); // Current idea of calling the writeHead of the original res
});
let that = this, // Arguments to apply to the original writeHead
args = arguments
function callMe() {
debug("Finished! Give control to http, statuscode: %i", statusCode);
writeFunction.apply(that, args); // Call the original writeHead from the response
debug("Sent!")
}
} // End of hackedHead
} // End of hijackHead
The core function is being passed to express as a middleware.
Additionally sessionManager.storeSessionData is a Promise storing data and fulfilling after that, taking ~60ms. The Promise has been testes and works perfectly.
When I now make a request using this Controller, the Node net Module returns the error:
TypeError: Invalid data, chunk must be a string or buffer, not object
at Socket.write (net.js:704:11)
at ServerResponse._flushOutput (_http_outgoing.js:842:18)
at ServerResponse._writeRaw (_http_outgoing.js:258:12)
at ServerResponse._send (_http_outgoing.js:237:15)
at write_ (_http_outgoing.js:667:15)
at ServerResponse.end (_http_outgoing.js:751:5)
at Array.write (/usr/lib/node_modules/express/node_modules/finalhandler/index.js:297:9)
at listener (/usr/lib/node_modules/express/node_modules/on-finished/index.js:169:15)
at onFinish (/usr/lib/node_modules/express/node_modules/on-finished/index.js:100:5)
at callback (/usr/lib/node_modules/express/node_modules/ee-first/index.js:55:10)
Since the new function needs about 30ms to react and return the Promise, the function finishes earlier causing Node to crash.
I already tried blocking the Node loop with a while, timeout or even a recursive function. Neither of them worked.
I tries to simplfy the code as much as possible and I hope that I didn't simplify it too much.
Now I am asking if anybody can help me, how to call the writeHead function properly after the Promise has resolved?
The issue with this is, that net.js directly responds to those methods when writeHead has finished. Even though the head has not been written, it tries to write the body.
Instead it is possible to catch the end()method which will await everything and then close the connection.
I understand the essence of callback functions in that the function is executed again after being passed as the parameter to another function. However, I'm confused as to where the variables inside the callback function come from as shown in the following node.js example:
router.get('/', function(req, res){
res.render('index', {});
});
How do the variables req and res get populated? An example explaining how I can just call res.render(...) without declaring res myself would be greatly appreciated.
They come from the same place they come from when a normal non callback function is invoked, at invocation time.
If you have this function,
function add (a, b) {
return a + b
}
You're fine with knowing that a and b come from when you invoke add,
add(1,2)
and it's the same principle with callbacks, don't let your brain get all twisted just because it's getting invoked later.
At some point the function you pass to router.get is going to be invoked, and when it does, it will receive req and res.
Let's pretend the definition for router.get looks like this
router.get = function(endpoint, cb){
//do something
var request = {}
var response = {}
cb(request, response) // invocation time
}
In the case of your example, it's just up to node to pass your function request and response whenever .get is invoked.
The whole point of the callback is that the invoked function calls it back.
In the case of router.get, it will insert the route (path, method, callback) in a lookup table; when a request comes in, Express will construct the response object, match the request's path and method against all the entries in the lookup table, take the callback from the matching entry and invoke callback(request, response) (passing the detected request and created response).
They get populated by whatever code is calling the callback. In your example, this is something inside the Express framework, though Express uses the node http library under the hood and adds additional functionality to the request and response objects provided by it.
But in code you write you can create a callback function signature that takes whatever params you want.
Someone please help me. I have been reading a lot of javascript documentation and fiddling with javascript.
I am able to use the functions but I don't quite get this essential syntactic jazz going on here
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(1337, 'myhost');
console.log('Server running at http://myhost:1337/');
I cant figure out why its okay to use req and res in the anonymous function above. It's like they live somewhere inside of http. They are not declared anywhere! They are made up variable names in an anonymous function that reference inner objects or something. It's crazy.
How does a callback function like this?
or like
stream.on('data', function(data){
// use data... i dont know where its coming from
// or how it got there, but use it
});
If you could post a small example that mimics this process and syntax or explain how these callback functions can work like this or how I can pass these undeclared variables into functions like this I would greatly appreciate it.
A similar example to the answer posted below.
var b = {
somefunction : function( data ){
var x = 100;
x = x+1; // 101
return data(x); // returns the callback function data w/ value x
}
};
b.somefunction(function(foo){
foo++; // 101 + 1
console.log(foo); // prints 102
});
The main issue is that Javascript is a functional language so you can pass functions as parameters to other functions. In other languages you may have experienced passing a pointer or handle to a function, for example. Consider a simple cases in which you have some math functions:
function add(a,b) {return (a+b)};
function subtract(a,b) {return (a-b)}:
Now I could create a new function:
function longWayAroundTheBarn(num1,num2,theFunc)
{
// invoke the passed function with the passed parameters
return theFunc(num1,num2);
}
And call it like this:
console.log(longWayAroundTheBarn(1,2,add));
> 3
Or even like this:
console.log(longWayAroundTheBarn(longWayAroundTheBarn(2,2,subtract),4,add);
> 4
Obviously this would be a silly use callbacks, but you can imagine generally that the ability to 'plug-in' a function this way can be pretty powerful.
Consider if you couldn't pass functions. This might be one way you would implement this:
function reallyLongWayAround(num1,num2,funcName)
{
if(funcName==='add')
return add(num1 ,num2);
else if (funcName === 'subtract')
return subtract(num1, num2);
}
You can imagine that besides being really tedious code to have to write and maintain, it's not nearly so powerful because the reallyLongWayAround function could only ever call code it knew about. By passing functions, my longWayAroundTheBarn doesn't care if I create new functions and pass it to it. Note that because of weak typing, it doesn't even need to care about the parameters it is passing. Maybe you want to implement something like
function businessDaysBetween(d1,d2)
{
// implementation left as a reader exercise
};
It would work just fine to call:
longWayAroundTheBarn(new Date(2014,1,15), new Date(2014,1,22),businessDaysBetween)
Returning to the specific case you raised, req and res are not 'local variables' as one answer indicates - they are called parameters or arguments. You aren't passing anything into them. They are being passed to you by the calling function. You could actually call them fred and barney if you wanted, although it would be a terrible idea. The main point is that they will be called populated with request and response objects.
You actually don't even have to have the parameters in your function signature, you could just have a callback as below, and make use of the second parameter passed to your function by reading the arguments array (Note, it's not actually an array but behaves similarly in many respects). This would be a terrible, terrible idea, but again, trying to illustrate the point.
var http = require('http');
http.createServer(function () {
arguments[1].writeHead(200, {'Content-Type': 'text/plain'});
arguments[1].end('Hello World\n');
}).listen(1337, 'myhost');
The req and res are actually local variables of the anonymous function.
I have an example below similar to the codes posted in your question.
// static class
var HelperClass = {
"doSomething" : function ( callback ) {
// do something here, for example ajax call to a server
var data = ajaxCallFromServer();
// return the data via callback
callback ( data );
};
};
// you codes here calling the HelperClass
// calling .doSomething method with an anonymous function callback
// that expects 1 parameter. This parameter is returned by the above
// code via callback ( data ). And in the anonymous function
// i called it as retData, you can call it whatever you like
HelperClass.doSomething( function ( retData ) {
// do soemthing with your retData here
});
You should use the documentation.
So for example for createServer, follow this page - http://nodejs.org/api/http.html#http_http_createserver_requestlistener
http.createServer([requestListener])# Returns a new web server object.
The requestListener is a function which is automatically added to the
'request' event.
Then, you check the 'request' event -
Event: 'request'# function (request, response) { }
Emitted each time there is a request. Note that there may be multiple
requests per connection (in the case of keep-alive connections).
request is an instance of http.IncomingMessage and response is an
instance of http.ServerResponse.
Regarding the stream, exactly same. The docs here - http://nodejs.org/api/stream.html#stream_writable_stream
Look for
Event: 'data'
The point is that http.createServer function takes an argument that is not a variable, but a function, if that makes sense. In javascript you can do that. And that function expects arguments that are specified in it's API. You can make it anonymous, like in your example, or declared like below:
function serverFunction(req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}
var http = require('http');
http.createServer(serverFunction).listen(1337, 'myhost');
but in the end it does not matter, it just behaves accordingly to what is specified in its the API.
I am quite new (just started this week) to Node.js and there is a fundamental piece that I am having trouble understanding. I have a helper function which makes a MySQL database call to get a bit of information. I then use a callback function to get that data back to the caller which works fine but when I want to use that data outside of that callback I run into trouble. Here is the code:
/** Helper Function **/
function getCompanyId(token, callback) {
var query = db.query('SELECT * FROM companies WHERE token = ?', token, function(err, result) {
var count = Object.keys(result).length;
if(count == 0) {
return;
} else {
callback(null, result[0].api_id);
}
});
}
/*** Function which uses the data from the helper function ***/
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
// this works
console.log(result);
});
// the problem is that I need result here so that I can use it else where in this function.
});
As you can see I have access to the return value from getCompanyId() so long as I stay within the scope of the callback but I need to use that value outside of the callback. I was able to get around this in another function by just sticking all the logic inside of that callback but that will not work in this case. Any insight on how to better structure this would be most appreciated. I am really enjoying Node.js thus far but obviously I have a lot of learning to do.
Short answer - you can't do that without violating the asynchronous nature of Node.js.
Think about the consequences of trying to access result outside of your callback - if you need to use that value, and the callback hasn't run yet, what will you do? You can't sleep and wait for the value to be set - that is incompatible with Node's single threaded, event-driven design. Your entire program would have to stop executing whilst waiting for the callback to run.
Any code that depends on result should be inside the getCompanyId callback:
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
//Any logic that depends on result has to be nested in here
});
});
One of the hardest parts about learning Node.js (and async programming is general) is learning to think asynchronously. It can be difficult at first but it is worth persisting. You can try to fight and code procedurally, but it will inevitably result in unmaintainable, convoluted code.
If you don't like the idea of multiple nested callbacks, you can look into promises, which let you chain methods together instead of nesting them. This article is a good introduction to Q, one implementation of promises.
If you are concerned about having everything crammed inside the callback function, you can always name the function, move it out, and then pass the function as the callback:
getCompanyId(token, doSomethingAfter); // Pass the function in
function doSomethingAfter(err, result) {
// Code here
}
My "aha" moment came when I began thinking of these as "fire and forget" methods. Don't look for return values coming back from the methods, because they don't come back. The calling code should move on, or just end. Yes, it feels weird.
As #joews says, you have to put everything depending on that value inside the callback(s).
This often requires you passing down an extra parameter(s). For example, if you are doing a typical HTTP request/response, plan on sending the response down every step along the callback chain. The final callback will (hopefully) set data in the response, or set an error code, and then send it back to the user.
If you want to avoid callback smells you need to use Node's Event Emitter Class like so:
at top of file require event module -
var emitter = require('events').EventEmitter();
then in your callback:
api.post('/alert', function(request, response) {
var data = JSON.parse(request.body.data);
var token = data.token;
getCompanyId(token, function(err, result) {
// this works
console.log(result);
emitter.emit('company:id:returned', result);
});
// the problem is that I need result here so that I can use it else where in this function.
});
then after your function you can use the on method anywhere like so:
getCompanyId(token, function(err, result) {
// this works
console.log(result);
emitter.emit('company:id:returned', result);
});
// the problem is that I need result here so that I can use it else where in this function.
emitter.on('company:id:returned', function(results) {
// do what you need with results
});
just be careful to set up good namespacing conventions for your events so you don't get a mess of on events and also you should watch the number of listeners you attach, here is a good link for reference:
http://www.sitepoint.com/nodejs-events-and-eventemitter/
I can't seem to find a general solution to this problem, even though I feel like I can't be the first to come across this. I also suspect it might be I'm taking the wrong approach in the first place here. Let me know.
I have an Expressjs App the interacts with various APIs (mostly OAuth2). Once a request comes in the App checks if it has an access token to fetch data from an API. In case the token is expired it will request a new one.
Now, when my App receives a second request in the meantime, requiring the exact same API, I want to avoid making a second call for an access token.
What I do is use a "Collector" where callbacks can be stored for a given key. The first request to store a callback for a key gets a collector callback to invoke once it has finished the task. All subsequent callbacks are enqueued and called later on with the collector callback.
This is the simple Collector class (CoffeeScript)
# Collect callbacks for tasks and execute all once the job is done
module.exports = class Collector
constructor: ()->
#tasks = {}
add: (key, callback)->
unless #tasks[key]
# Initiate callback list for the key with first callback
#tasks[key] = [callback]
return ()=>
# Call all collected callbacks for the key
(callback.apply {}, arguments for callback in #tasks[key])
# Reset callback list
#tasks[key] = null
else
# Add callback to existing list
#tasks[key].push callback
return false
I'm not sure if storing the callbacks inside this class is the right way, but to use a database (Redis) I would have to find a way to store callbacks…
Is there a better way to invoke multiple callbacks once a job is done?
Why don't you just aggregate your callbacks into an array which you cycle through, executing each contained function when your original call is complete?
It could be as simple as:
var dones = [];
dones.push(function (err, res) { console.log('hoo'); });
dones.push(function (err, res) { console.log('ray!'); });
function whenDone(err, res) {
_.each(dones, function (done) { done(err, res); } });
}
myWhateverFunction(whenDone);
You can wrap this into whatever data structure you want, if you want to make it prettier.
I don't have a specific answer to your problem, but you should check out the async module. I think it's a step in the right direction: https://github.com/caolan/async