Develop Reference

Expressjs middleware keeps variable changed

I am trying to do a simple thing which is obvious I believe in the code below:
module.exports = function(req, res, next) {
var dop = require('../../config/config').DefaultOptions;
console.log(require('../../config/config').DefaultOptions);
console.log(dop);
dop.firstPage = 'test.sjs';
next();
};
This is an Expressjs middle ware which is very simple but the interesting point is that next time I load a page both of the console.log results has been changed to 'firstPage: test.sjs'. It shouldn't act like this and it should only change the dop variable.
Anyone with the knowledge why this creepy thing is happening?
Thank you

The main issue is require() is cached, so require('../../config/config') returns reference to the same instance, and as a result changing in one place causes all other references and subsequent requires to get that modified instance.
The simplest solution would be have a function in config to return a config object, that way every time invoking the get config function you will get a new instance with essentially the same content. I.e.:
config.js:
module.exports = {
getDefaultOptions: function(){
return {foo: 'bar', ip: '1.1.1.1'}
}
};

Express - Passing mysql connection to scripts

I defined mysql connection with all parameters necessary to app.js, how can make visible to other scripts in routes/ by default, without requiring or redefining mysql parameters, just using client.query(..)?

A pattern I use is to set up my db object in a module once and export it: (let's call it utils/mySQL.js)
//I haven't used real mysql in node so excuse the pseudo-syntax:
var db = require('mysql-driver-thingy');
db.connect('localhost', 'sqlport', options...);
db.otherSetupFunctions();
console.log("Finished db setup. You should only see this message once! Cool.");
module.exports = db;
And then I can require the db object everywhere I need it. Since requires are cached, this does't actually call the setup methods multiple times.
In app.js:
var db = require('./utils/mySQL.js');
...
In models/user.js:
var db = require('../utils/mySQL.js');
...
A final option, which isn't recommended, is to pollute the global namespace. This seems to be the answer you're really after:
//set up your db
...
// and now make it available everywhere:
global.client = db.client
You can now magically use the client object in all your modules, without even requiring it.
There are many reasons globals are bad, though:
If your code and other code define globals, they could conflict and overwrite each other.
It's hard to find where you defined the db/client object, etc.

You can inject mysql connection into other scripts like this:
app.js
var mysqlConnection = new Conection(params);
require('controller/main.js)(mysqlConnection);
main.js
module.exports = function(mysqlConnection) {
// You can access your mysql connection here
};
UPDATE:
You can inject several variables same way. Also you still can export methods from module if you need this:
app.js
var mysqlConnection = new Conection(params);
var news = require('model/news.js)(app, mysqlConnection);
news.list(function(err, news) {
// Do something
});
news.js
module.exports = function(app, mysqlConnection) {
var methods = {};
// mysql connection and app available from here
methods.list = function(cb) {
mysqlConnection.list(function(err, data) {
cb(err, data);
});
};
return methods;
};

Node.js global variable property is purged

my problem is not about "memory leakage", but about "memory purge" of node.js (expressjs) app.
My app should maintain some objects in memory for the fast look-up's during the service. For the time being (one or two days) after starting the app, everthing seemed fine, until suddenly my web client failed to look-up the object bacause it has been purged (undefined). I suspect Javascript GC (garbage collection). However, as you can see in the psedu-code, I assigned the objects to the node.js "global" variable properties to prevent GC from purging them. Please give me some clue what caused this problem.
Thanks much in advance for your kind advices~
My node.js environments are node.js 0.6.12, expressjs 2.5.8, and VMWare cloudfoundry node hosting.
Here is my app.js pseudo-code :
var express = require("express");
var app = module.exports = express.createServer();
// myMethods holds a set of methods to be used for handling raw data.
var myMethods = require("myMethods");
// creates node.js global properties referencing objects to prevent GC from purging them
global.myMethods = myMethods();
global.myObjects = {};
// omited the express configurations
// creates objects (data1, data2) inside the global.myObjects for the user by id.
app.post("/createData/:id", function(req, res) {
// creates an empty object for the user.
var myObject = global.myObjects[req.prams.id] = {};
// gets json data.
var data1 = JSON.parse(req.body.data1);
var data2 = JSON.parse(req.body.data2);
// buildData1 & buildData2 functions transform data1 & data2 into the usable objects.
// these functions return the references to the transformed objects.
myObject.data1 = global.myMethods.buildData1(data1);
myObject.data2 = global.myMethods.buildData2(data2);
res.send("Created new data", 200);
res.redirect("/");
});
// returns the data1 of the user.
// Problem occurs here : myObject becomes "undefined" after one or two days running the service.
app.get("/getData1/:id", function(req, res) {
var myObject = global.myObjects[req.params.id];
if (myObject !== undefined) {
res.json(myObject.data1);
} else {
res.send(500);
}
});
// omited other service callback functions.
// VMWare cloudfoundry node.js hosting.
app.listen(process.env.VCAP_APP_PORT || 3000);

Any kind of cache system (whether is roll-your-own or a third party product) should account for this scenario. You should not rely on the data always being available on an in-memory cache. There are way too many things that can cause in-memory data to be gone (machine restart, process restart, et cetera.)
In your case, you might need to update your code to see if the data is in cache. If it is not in cache then fetch it from a persistent storage (a database, a file), cache it, and continue.

Exactly like Haesung I wanted to keep my program simple, without database. And like Haesung my first experience with Node.js (and express) was to observe this weird purging. Although I was confused, I really didn't accept that I needed a storage solution to manage a json file with a couple of hundred lines. The light bulb moment for me was when I read this
If you want to have a module execute code multiple times, then export a function, and call that function.
which is taken from http://nodejs.org/api/modules.html#modules_caching. So my code inside the required file changed from this
var foo = [{"some":"stuff"}];
export.foo;
to that
export.foo = function (bar) {
var foo = [{"some":"stuff"}];
return foo.bar;
}
And then it worked fine :-)

Then I suggest to use file system, I think 4KB overhead is not a big deal for your goals and hardware. If you familiar with front-end javascript, this could be helpful https://github.com/coolaj86/node-localStorage

URL routing in Node.js

Homework done:
The Node Beginner Book
How do I get started with Node.js [closed]
Structuring handlers in Node
Backstory: I wanted to try and write my own framework but I'm running into some troubles, most likely due to not understanding it fully.
What I want to achieve is a syntax that looks like this:
var app = require('./app'); //this part is understood and it works in my current code.
app.get('/someUrl', function(){ //do stuff here });
app.post('/someOtherUrl', function(){ //do stuff here });
I know of the Express-framework that has this same syntax but reading their source code still eludes me.
This might be a trivial task to achieve but I simply can't produce it, yet.
Trying to require('./app'); in a file deeper in the application produces a undefined object, so I'm guessing that a server is a singleton object.
So what have I tried?
My current code looks like this, and somehow I feel like this is the way to go, but I can't apparently do it like this.
I'm omitting all the require(); statements to keep it more readable.
server.js:
var app = module.exports = {
preProcess: function onRequest(request, response){
processor.preRequest(request); //this object adds methods on the request object
var path = urllib.parse(request.url).pathname;
router.route(urls, path, request, response);
},
createServer: function() {
console.log("Server start up done.");
return this.server = http.createServer(this.preProcess);
}
};
exports.app = app;
At the time of writing I'm experimenting with extending this object with a get() method.
index.js:
var app = require('./server');
app.createServer().listen('1337');
the router.route() bit basically sends the request onward into the application and inside the router.js-file I do some magic and route the request onward to a function that maps (so far) to the /urlThatWasRequested
This is the behavior I'd like to leave behind.
I know this might be a pretty tall order but all my code is easily discardable and I'm not afraid of rewriting the entire codebase as this is my own project.
I hope this is sufficient in explaining my question otherwise, please say what I should add to make this a bit more clear.
Thanks in advance!

I'm not exactly sure what your question is, but here's some thoughts:
1) You are creating a circular reference here:
var app = module.exports = {
// some other code
}
exports.app = app;
You add app as a property of app. There's no need for the last line.
2) You need to hold handlers in app. You can try something like this:
var app = module.exports = {
handlers : [],
route : function(url, fn) {
this.handlers.push({ url: url, fn: fn });
},
preProcess: function onRequest(request, response){
processor.preRequest(request);
var path = urllib.parse(request.url).pathname;
var l = this.handlers.length, handler;
for (var i = 0; i < l; i++) {
handler = this.handlers[i];
if (handler.url == path)
return handler.fn(request, response);
}
throw new Error('404 - route does not exist!');
},
// some other code
}
Note that you may alter this line: if (handler.url == path) in such a way that handler.url is a regular expression and you test path against it. Of course you may implement .get and .post variants, but from my experience it is easier to check whether a request is GET or POST inside the handler. Now you can use the code above like this:
app.route('/someUrl', function(req, res){ //do stuff here });
The other thing is that the code I've shown you only fires the first handler for a given URL it matches. You would probably want to make more complex routes involving many handlers (i.e. middlewares). The architecture would be a bit different in that case, for example:
preProcess: function onRequest(request, response){
var self = this;
processor.preRequest(request);
var path = urllib.parse(request.url).pathname;
var l = self.handlers.length,
index = 0,
handler;
var call_handler = function() {
var matched_handler;
while (index < l) {
handler = self.handlers[index];
if (handler.url == path) {
matched_handler = handler;
break;
}
index++;
}
if (matched_handler)
matched_handler.fn(request, response, function() {
// Use process.nextTick to make it a bit more scalable.
process.nextTick(call_handler);
});
else
throw new Error('404: no route matching URL!');
};
call_handler();
},
Now inside your route you can use
app.route('/someUrl', function(req, res, next){
//do stuff here
next(); // <--- this will take you to the next handler
});
which will take you to another handler (or throw exception if there are no more handlers).
3) About these words:
Trying to require('./app'); in a file deeper in the application produces a undefined
object, so I'm guessing that a server is a singleton object.
It isn't true. require always returns the reference to the module object. If you see undefined, then you've messed up something else (altered the module itself?).
Final note: I hope it helps a bit. Writing your own framework can be a difficult job, especially since there already are excelent frameworks like Express. Good luck though!
EDIT
Implementing .get and .set methods is actually not difficult. You just need to alter route function like this:
route : function(url, fn, type) {
this.handlers.push({ url: url, fn: fn, type: type });
},
get : function(url, fn) {
this.route(url, fn, 'GET');
},
post : function(url, fn) {
this.route(url, fn, 'POST');
},
and then in routing algorithm you check whether type property is defined. If it is not then use that route (undefined type means: always route). Otherwise additionally check if a request's method matches type. And you're done!

Down the rabbit hole - Where does "res" originate from in express and/or nodejs?

I'm trying to understand more of the express and nodejs internals. Looking in express' response.js file, it frequently assigns several methods to res, which seems to be a prototype.
Specifically, res is declared as res = http.ServerResponse.prototype.
Ok, so what is http? http is declared as http = require('http').
So looking in express' http.js file, we see exports = module.exports = HTTPServer;
and HTTPServer seems to be this method:
function HTTPServer(middleware){
connect.HTTPServer.call(this, []);
this.init(middleware);
};
And this is where I get stuck. According to my logic, it would seem that ServerResponse is being called on the HTTPServer method, which of course doesn't make sense. Therefore, I must be missing something.
UPDATE:
I just realized that express creates an instance of HTTPServer:
exports.createServer = function(options){
if ('object' == typeof options) {
return new HTTPSServer(options, Array.prototype.slice.call(arguments, 1));
} else {
return new HTTPServer(Array.prototype.slice.call(arguments));
}
};
So I'm guessing it's the case that ServerResponse is actually being called on that instance? But I still cannot locate ServerResponse...

I can't see any http.js file in express source files.
According to node.js documentation on http http = require('http') will load the http module, which has a ServerResponse object.
So express code enhances the ServerResponse object with additionnal methods.