We are using Koa to create a REST service.
One thing I would like to do is call an internal API from a middleware using a custom request object and parse the response before giving it to a real client.
If this is a bit hard to grasp, consider this pseudo code:
var myMiddleware = function(ctx, next) {
var fakeSubRequest = new Koa.Request('GET', '/some-internal-path');
var fakeSubResponse = yield koa.internalRequest(fakeSubRequest);
ctx.body = someTransformation(fakeSubResponse.body);
}
This entirely fictional. My point is that I want to 'simulate' a request in my application by creating a new request and receiving an new response, which I can then use to construct my actual response.
I don't want to call my routes and/or middlewares directly. I want it to go through the router because it must work server-wide.
Is it possible to simulate an internal request without doing a real HTTP request on localhost?
Few years later, I have an answer.
The best approach is to create a new Context object from scratch and try to 'Mock' Node.js IncomingMessage and ServerResponse objects.
This is not easy and a bit of a hack, but it's what Koa requires right now. I ended up starting a new Koa-inspired framework that does this better.
Related
I have a front end application, which I would like to return results with from an Express backend. Let's just call those results country and city for this reference.
I have done a bunch of searching, but I cannot find any solid resources on the relationship between the front end and middleware. Yes, I know what these things are, and the order in which they should flow, but the confusion sits with :
Do I need to connect my front end and middleware? How so?
If I am already connected to my backend from the front end, would I also have to connect to middleware?
How would I return the country and city from the middleware and/or express backend?
Any other info you think would be helpful for the greater dev community who is still learning would be beneficial.
While you could return data from a middleware, it's probably not what you are trying to do. A middleware is a piece of code that is executed between the time the request is receive by your backend, and the resource is fetch. In a middleware you could do things such as check if a user has access to a certain resource or authenticate a user by some sort of credential passed with the request.
Either way, the way you would, typically, do request from your front-end to your backend is via an XmlHttpRequest. Those request are usually Asynchronous, so they usage will not block the whole page while being executed. There are many ways you could create XmlHttpRequest. The native Javascript way is kinda ugly so I would suggest using the fetch api instead. You could also go with third party library if you need to do more complex stuff. I personnally like axios but this is up to you.
To give you a better understanding of what Express is doing, it's basically an infinite loop that waits for http request. You need to defined routes, that execute function that returns data.
Here is a basic example. Note that this script is executed via NodeJS :
// myserver.js
const express = require('express')
const app = express()
app.get('/cities', (req, res) => {
const cities = /** somehow get all the cities **/
res.json(cities);
})
/** the rest of the server... **/
/** For example, the route for Countries **/
In the previous example, we've built a basic server that listen to the url localhost:3000/cities and execute a function when this url is fetched. The said function will fetch all the cities and return them as JSON.
In your frontend, You would need to do a XmlHttpRequest that would call this url, to get the server to execute the function, which will return the data. Phew... I hope I did not lost you there.
A typical example would be a simple call using the fetch api.
Please note that this script is executed in the browser.
// myclient.js
async fetchAllCities() {
const cities = await fetch('http://localhost:3000/cities');
console.log(cities);
}
// just for fun, we add a click listener on a button and call the function defined above.
document.getElementById('myButton').addEventListener('click', async function() {
// we fetch the cities when we click on the button !
await fetchAllCities();
});
In the previous example, I am using the fetch function to call the url we declared in our Express server.
I'm also using Async / Await, which can be a little tricky, but it just mean Wait for the data to be there before going forward.
I highly suggest reading on the subject. Here are some references.
How do I return the response from an asynchronous call?
Understanding async/await on NodeJS.
Await from MDN
I hope this brief overview of XmlHttpRequest helped you get the base of how an API works.
Middleware is used to help the back-end do its job in processing incoming requests. It does not exist separate from the back-end. It's part of the back-end. For example, you might have middleware that checks to see if an incoming request is properly authorized/authenticated before the route can be handled by it's regular route handler.
Do I need to connect my front end and middleware? How so?
No. Your front-end sends requests to the back-end. The back-end may or may not use middleware to service the request. That's entirely up to the implementation in the back-end and what it needs to do for any given request.
If I am already connected to my backend from the front end, would I also have to connect to middleware?
No. You don't separately connect to middleware. You connect to your back-end and the back-end may or may not use middleware to do its job (something the front-end will have no knowledge of).
How would I return the country and city from the middleware and/or express backend?
You would have to show more details about what you're actually trying to return back from a request, but a common data format is JSON so you could construct a Javascript object with your desired response and then send it back to the client as the response from the incoming request using either res.json(someObj) or res.send(someObj) (both do the same thing if someObj is a Javascript object).
For example:
app.get("/getsomething", (req res) => {
// do some processing here to get cityResult and countryResult
// construct object to send back to client
const obj = { city: cityResult, country: countryResult};
// send this object as JSON back the the client as the response to this
// incoming request
res.json(obj);
});
I got an old website on a webserver. It is to big and not structured well enough but needs to be improved by e.g. Account management. As it is (in my opinion) at its end of lifetime, we do not want to put more effort in it but instead migrate to new technology. For that, we want to use node.js and AngularJS, because the whole project is more a webapp than it was at the beginning. As a migration concept, we want to include the old stuff via a kind of routing through the node.js server and replace it step by step. For that I looked into the "request" library without getting the right grip.
Goal is, to route some requests after authorization check to the old server, without leaving the new server (gate). For that I need to check and parse the gets and posts. Some other requests have to response by the node.js server itself.
As I think that I am not the only one with that approach I am asking for experience in that matter.
I had to do something similar, because we made a new API which was not compatible with the first version and some features were not implemented in the newer API so we had to do like a bridge. Authentication was happening in the first server, and then we were routing the request to the old API and then returning to the user.
The approach I took was using a module like request to make the call in the old server.
Assuming you are using expres for your new API, you can do something similar
var request = require('request');
app.get('/test', function(req, res) {
//authenticate stuff
var options = {
url: 'http://oldendpoint.com/test',
headers: {
//headers for authenticate in the old endpoint
}
};
request(options, function (error, response, body) {
if (!error && response.statusCode == 200) {
res.send(body); //this will send it back to your client
}
});
});
Basically you get a request to your new API (node.js app) in the /test endpoint and this, after auth or after whatever check, will forward the request to the old system, and then it will return some data which is forwarded to the client who made the request in the first place.
I want to design a live updating Node.js Express server, perhaps having a particular route, say /update, which loads a new configuration file. The only concern I have with this right now is that the server could potentially be in any state when the update happens. If I load a new configuration file while there is a JS message being processed for a user request, at the beginning of the user request there could be one configuration and before the request completes there could be a second configuration when the new config file is loaded. The only way I can think of to prevent this is to take down the server for at least one minute (keep the server live, but prevent any incoming requests altogether) and then update the server and put it back online, but then that's not really the best form of hot reloading or live updating is it?
How can I somehow trick the JS event loop so that the config file only gets loaded once all requests have completed and delay any new requests until after the config is loaded?
One algorithm would be:
set a flag "starting re-configuration"
above flag prevents any new requests from being processed (using Express middleware)
check that all current requests are completed (can't think of anything better than a polling loop here)
once above check is done, load new configuration
once configuration is loaded, switch the flag from (1)
Disclaimer: I have not tried this in production. In fact, I have not tried this at all. while I believe the idea is sane, there may be hidden pitfalls along the road which are not currently known to me.
There is one thing that many Node.js developers tend to forget/not fully realise:
There can always be only one JavaScript statement executed at a time.
It does not matter that you can do async I/O or execute a function later in time. No matter how hard you try, all the JS code that you write is executed in a single thread, no parallelism. Only the underlying implementation (which is completely out of our control) can do things in parallel.
This helps us, because as long as our update procedure is synchronous, no other JS code (i.e. client response) can be executed.
Configuration live-patching
The solution to prevent configuration change mid-request is rather simple:
Each request gets its own copy of the application's configuration.
If your application's configuration lives in a JavaScript object, you simply clone that object for each new request. This means that even if you suddenly change the configuration, it will only be applied to new incoming requests.
There are tons of modules for cloning (even deep cloning) objects, but since I believe mine is best I will use this opportunity for some small self-promotion - semantic-merge.
Code live-patching
This is a bit trickier, but should be generally possible with enough effort.
The trick here is to first remove/unregister current Express routes, clear Node's require cache, require the updated files again and re-register route handlers. Now Express will finish all pending requests using the old code (this is because Node cannot remove these old functions from memory as long as the code contains active object references - which it does - req and res) and use the newly required modules/routes for new incoming requests. Old code should get released from memory as soon as there are no more requests that started with the old code.
You must not use require anywhere during request processing, otherwise you risk the same problem as with changing configuration mid-request. You can of course use require in a module-level scope because that will be executed when the module itself is required, thus being synchronous.
Example:
// app/routes/users.js (or something)
// This is okay, because it is executed only once - when users.js
// itself is required
var path = require('path')
// This function is something you would put into app.use()
module.exports = function usersRoute (req, res, next) {
// Do not use require() here! It will be executed per-request!
}
I think that instead of looping a request to the server you can use a Websocket.
That way, when there's a change in the config file that you mentioned, the server can 'emit' a message to the users, so they refresh their data.
If you are using nodeJS and Express, this will help you:
Socket.io with NodeJS
The server will wait for the signal of some user or anybody and emit the signal to all the users, so they get the new data
Node.js:
var express = require('express');
var app = express();
var server = require('http').createServer(app);
var io = require('socket')(server);
var port = process.env.PORT || 3000;
server.listen(port, function () {
console.log('Server listening at port %d', port);
});
app.use(express.static("/PATH_TO_PROJECT"));
io.on('connection', function (socket) {
socket.on('someone update data', function (data) {
socket.to(socket.room).broadcast.emit('data updated', {params:"x"});
}
});
Meanwhile, the client will be listening if there's any change:
View.js:
var socket = io();
socket.on('new message', function (data) {
liveUpdate(data);
});
I hope I understood correctly what you asked
This is a good problem to solve.
A possible solution could be:
That you derive the controllers on each path from a parent controller. The parent controller can flag a property ON (a flag / file) when a request arrives and turn it OFF when the response is sent back.
Now subclass this parent controller for each express end-point facing the front end. If you make a request now to '/update', the update controller would know if the server is busy or not through the FLAG and send back a reply if the update was successful or not.
For update failures the front end could possibly post back to the '/update' end point with some back-off scheme.
This scheme might work for you ...
I'm new to JS and more specifically Node. Even after reading the api docs, I'm confused about what 'requestListener' is in the following method.
http.createServer([requestListener]);
Searching google revealed that 'requestListener' is a(n) (anonymous) function with the following signature:
function (request, response) { };
I suppose I'm reading the docs incorrectly, hopefully someone can point me in the right direction.
The docs say that the method call takes a function which will be called when a new request is received by your application. This function, as you correctly stated in your question, takes two arguments - a request and response objects.
You should inspect the contents of these objects to learn what information is available to you. Also, take a look at the API docs for request and response.
The function is optional; you could also attach the request handler in the following way:
var server = http.createServer()
server.on('request', function (req, res) {
// Process the request here
})
In practice, this function is called when someone opens up your website in their browser (i.e. issues a GET http request). The purpose of that function is to provide a HTTP response body back to the client, i.e. render a web page or perform any business logic as necessary.
To directly answer your question: it's a function that gets called when a request is received by the server and is given those two parameters.
At the very least you can experiment with doing a console.log(request, response) inside the function and see what is spit out in your terminal.
But that's only the beginning of the rabbit hole. You should read up on "callback functions", as they are integral to how Node (and quite a bit of client-side javascript) operates (asynchronously).
The http.createServer method creates a server object.
The server object has a listen method. If you invoke the listen method, for example:
createServer(requestListener).listen({ port:80 });
the server object will be listening on port 80, and when a Http request is received on that port, the server object will invoke the function requestListener, passing it two objects, of type request and response. So you can write your requestListener like, for example:
function requestListener(req, res) {
res.write("Hello world");
res.end();
}
and it will push out the string Hello world to every url that is hitting your this simple web server.
write is one of the many methods of the response object.
You can run the above few lines of code on your PC, and point your browser to http://localhost.
If you have other applications listening on port 80, then use a different port number in your listen method.
Using an anonymous function for requestListener is merely a different coding pattern, for better or for worse. My code above can be re-written as:
createServer((req, res) => {
res.write("Hello world");
res.end();
}).listen({ port:80 });
The above code is very rudimentary and will send the same response to every Http request that hits it. The code does not differentiate between localhost/page1 or localhost/page2, etc. So to do anything more, the requestListener have to be expanded significantly to parse different paths in the url and to decide on what content to send for each. But to do anything useful as a real web server without writing too much code, you will need a package. Express is an excellent one that drives many real life web servers.
How would I add route handlers to an http server that already exists and has been instantiated?
All the routers I've found (including express) seem to require that they be passed into the http.createServer() method.
For example with express:
var server = http.createServer(app);
My main criteria:
Add routes to an existing server the way something like sockjs does it.
Be agnostic to whatever router is already being used (if there is one)
Not rely on an existing router "app" object to add the routes (creating a new one using a routing library would be fine).
Example: passing server into SockJS
var http_server = http.createServer(); // agnostic
sockjs_server.installHandlers(http_server, options);
http_server.listen(...);
The way it's done in the sockjs source seems quite cryptic... but I think it involves traversing existing handlers and overwriting them with a custom router/handlers.
Thanks so much for any help!
Well, an http server is nothing but an EventEmitter. It has a request event which is the one that handles the requests coming from the clients.
So, one thing you can do is to make a wrapper function around the current handler function. For instance, let's suppose the existence of some express application:
var express = require('express');
var app = express();
app.get('/', function(req, res){
res.send('Hello World!');
});
var server = app.listen(8080);
So, now, you can simply go over the list of currently registered request listeners in the server, remove the old listener functions and wrap them in a new one that handles your request the way you want. For instance, you can now create your own router and determine through which pipeline to send a request depending on its contents (i.e. path, content type, accepted languages, etc).
server.listeners('request').forEach(function(listener){
server.removeListener('request', listener);
server.on('request', function(req, res){
console.log('Before');
listener(req,res);
console.log('After');
});
});
In the example above you can see that I run a couple of console.log statements around the actual execution of the listener function. In this case the listener function is actually the Express main request handler. The express handler represents a pipeline, and by creating this new wrapper function you just added a new pipe at the beginning of the pipeline.
This technique would allow you to handle the request first and decide whether you want to send your own response and terminate the request here, or send it through a different pipeline or send the request down the express pipeline (i.e. the old listener function available to your new request handler closure).