I got an old website on a webserver. It is to big and not structured well enough but needs to be improved by e.g. Account management. As it is (in my opinion) at its end of lifetime, we do not want to put more effort in it but instead migrate to new technology. For that, we want to use node.js and AngularJS, because the whole project is more a webapp than it was at the beginning. As a migration concept, we want to include the old stuff via a kind of routing through the node.js server and replace it step by step. For that I looked into the "request" library without getting the right grip.
Goal is, to route some requests after authorization check to the old server, without leaving the new server (gate). For that I need to check and parse the gets and posts. Some other requests have to response by the node.js server itself.
As I think that I am not the only one with that approach I am asking for experience in that matter.
I had to do something similar, because we made a new API which was not compatible with the first version and some features were not implemented in the newer API so we had to do like a bridge. Authentication was happening in the first server, and then we were routing the request to the old API and then returning to the user.
The approach I took was using a module like request to make the call in the old server.
Assuming you are using expres for your new API, you can do something similar
var request = require('request');
app.get('/test', function(req, res) {
//authenticate stuff
var options = {
url: 'http://oldendpoint.com/test',
headers: {
//headers for authenticate in the old endpoint
}
};
request(options, function (error, response, body) {
if (!error && response.statusCode == 200) {
res.send(body); //this will send it back to your client
}
});
});
Basically you get a request to your new API (node.js app) in the /test endpoint and this, after auth or after whatever check, will forward the request to the old system, and then it will return some data which is forwarded to the client who made the request in the first place.
Related
I'm using node-soap to make a SOAP call through an Express NodeJS application. I have very basic SOAP services wired up, most of which work perfectly.
There is one however that is returning a "resource not found" message (note this is a valid message from the service). Yet when I use the exact same header and body in Boomerang the resource is returning nicely. Is there a way of logging the full request that is being sent via node-soap itself (...rather than installing sniffer like fiddler)?
Sample of code I'm using:
soap.createClient(config.wsdl, (error, client) => {
client.addSoapHeader(config.soapHeader());
client[config.webMethodName](config.soapBody(policyNumber, agentNumber), (error, soapResponse) => {
//do stuff
});
});
This might help:
https://github.com/vpulim/node-soap#clientlastrequest---the-property-that-contains-last-full-soap-request-for-client-logging
Although I'm sure there is a normal request object buried in the node-soap implementation somewhere, the soapResponse object in your final callback might be of use for debugging as well.
I have an app that has a REST api. I want it so that the only requests that can be made to the REST api are ones originating from the app itself. How can I do that?
I am using a node.js+express server too.
EDIT: the app is fully a public web app.
Simply define the header in your request, what this does is, it allows requests only from a certain domain, and instantly rejects any other domain.
response.set('Access-Control-Allow-Origin', 'domain.tld');
EDIT: IF you're really keen against web scraping stuff, you could make a function to double check client's origin.
function checkOrigin (origin) {
if (origin === "your.domain.tld") {
return true;
} else {
return false;
}
}
/* Handling it in response */
if (checkOrigin(response.headers.origin)) {
// Let client get the thing from API
} else {
response.write("Send them error that they're not allowed to use the API");
response.end();
}
Above example should work for the default HTTP/HTTPS module, and should also work for Express, if I'm not mistaken.
EDIT 2: To back my claim up that it should also work for Express, I found this quotation at their documentation;
The req (request) and res (response) are the exact same objects that Node provides, so you can invoke req.pipe(), req.on('data', callback), and anything else you would do without Express involved.
I would recommend using an API key from the client. CORS filters are too easy to circumvent.
A simple approach for securing a How to implement a secure REST API with node.js
Overview from above post:
Because users can CREATE resources (aka POST/PUT actions) you need to secure your api. You can use oauth or you can build your own solution but keep in mind that all the solutions can be broken if the password it's really easy to discover. The basic idea is to authenticate users using the username, password and a token, aka the apitoken. This apitoken can be generated using node-uuid and the password can be hashed using pbkdf2
Then, you need to save the session somewhere. If you save it in memory in a plain object, if you kill the server and reboot it again the session will be destroyed. Also, this is not scalable. If you use haproxy to load balance between machines or if you simply use workers, this session state will be stored in a single process so if the same user is redirected to another process/machine it will need to authenticate again. Therefore you need to store the session in a common place. This is typically done using redis.
When the user is authenticated (username+password+apitoken) generate another token for the session, aka accesstoken. Again, with node-uuid. Send to the user the accesstoken and the userid. The userid (key) and the accesstoken (value) are stored in redis with and expire time, e.g. 1h.
Now, every time the user does any operation using the rest api it will need to send the userid and the accesstoken.
I have a JavaScript script to check if there is a new post in a certain subreddit. Reddit magicaly provides a JSON endpoint on every link. In this case I have the following endpoint:
https://www.reddit.com/r/webdev/new.json?limit=1
I then used request module for node.js to get the specefic data I need eg domain, selftext, author and domain. However this changes every single time a new post is published and I therefore use a setInterval function to check every few seconds if a new post is released (Its like some sort of polling). I am saving every posts unique id to mongo to prevent double posting since that would assume every post is new on every request.
So the structure is something like:
setInterval(function () {
request({
uri: redditEndpoint,
json: true
}, function (error, response, body) {}) });
// I then save the data sent to mongo as unique to prevent double posting
// I consume the data here
}, 1000);
I would like to move away from this method and move over to websockets. However I am not sure how to correctly implement a websocket on such an endpoint, preferably with socket.io
That websocket endpoint would need to be provided by the server - in this case from the reddit server. You can't create a websocket connection to a random server which does not offer websocket support.
The only thing you could do is build a proxy server, which polls the reddit server for new events (like what you are currently doing) and which then offers a websocket endpoint itself for other clients.
My Project is built with Nodejs as proxy server to communicate with an external API.
The API send product updates via Redis (pub/sub); The Proxy server handle the message and send it to the client via SSE (Server Sent Events).
It is the first time for me using Redis and SSE and looking online for tutorials seems to be easy to implement and I did it.
On the Client side I just created an EventSource and as soon as I receive an update I do something with it:
// Client Side
var source = new EventSource('/redis'); // /redis is path to proxy server
source.addEventListener('items', handleItemsCallback, false);
source.addEventListener('users', handleUsersCallback, false);
source.addEventListener('customers', handleCustomersCallback, false);
// Function sample...
function handleItemsCallback (msg) {
// Do something with msg...
}
In the Proxy server I created a controller with routing to /redis to handle Redis messages:
exports.redisUpdates = function (req, res) {
// Redis Authentication
var redisURL = url.parse(process.env.REDISCLOUD_URL);
var client = redis.createClient(redisURL.port, redisURL.hostname, {ignore_subscribe_messages: false});
client.auth(redisURL.auth.split(":")[1]);
// let request last as long as possible
req.socket.setTimeout(0);
// Subscribe to channels
client.subscribe('items', 'users', 'customers');
// Handle messages
client.on('message', function (channel, message) {
res.write('retry: 5000\n');
res.write('event: ' + channel + '\n');
res.write('data: ' + message + '\n\n');
res.flush(); // If I do not add this it doesn't push updates to the client (?)
});
//send headers for event-stream connection
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive'
});
res.write('\n');
};
Using it locally in a development environment it works fine but using it in Production generate several different errors, the App is hosted on Heroku and the Heroku Metrics show several H18, H12, H27 Errors;
Sometimes the /redis call return status 503;
What I wish to understand is if I'm using those services correctly, why all tutorials do not mention res.flush() and I discovered it by myself to let it work the first time...
In all fairness, this question is not really answerable for a few reasons. I don't know which tutorials you are talking about since you didn't reference any in the question. I cannot speak on behalf of those who wrote the unreferenced tutorials. They could just be wrong, or maybe the architecture of what you are trying to accomplish differs in some small way. I also don't know what framework or optional middleware you are using in your project.
Now, with all of that said there are a few things I can share that may help you out.
Most tutorials you find out there are probably not going to open a connection and read from the stream indefinitely. When the process ends, the http response is closed with .end() or something similar. Since an HTTP response is a write stream, it follows the same rules as any other stream. You can find a lot of good info about streams here:
https://github.com/substack/stream-handbook
Something important to understand is that a stream can have a buffer and most http frameworks enable compression which causes buffers to be used. The code sample in the next link is a good example of what a framework would do for you behind the scenes (a minimal implementation of course)
https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/#what-we-ve-got-so-far
Since you want the output to continue being updated, you either have to wait until the output buffer size is reached or you have to call .flush().
If you ARE using express, check out this next Stack Overflow post related to compression middleware. I believe you'll have to have it disabled for your /redis route.
Node Express Content-Length
I hope that helped a little. Like I said, its kind of hard to answer this question. ;)
I have been trying to develop a rather simple server in nodejs. Basically, what I am going for is a simple API that requires authentication (simple username/password style). What I do not need is any kind of frontend functionality (templating etc.). My problem is, I can't seem to get my head around the approach of express/node.
Specifically, my questions are:
How do I wire in the authentication? Do I pass several handlers into every route that requires authentication, or is there a more elegant way to do this?
How does the Express middleware (like app.use(express.bodyParser())) work? Do they alter contents of the request or response object? Specifically, if I use the body parser (internally formidable?), where do I access the request data this is supposed to parse?
When using authentication and I have, say, credentials stored in a database with more information about the individual client associated, at what point do I extract that information? I.e., when a user logs in, do I fetch the user record on login and pass it on, or do I fetch it in every handler that requires the information?
Ultimately, do you know of an open source application that I could take a look at? I'd like to see something that has simple authentication and maybe even utilizes formidable, since uploading a file is one of my requirements.
As I mentioned earlier, I believe my problem is ultimately a difficulty with the function-oriented approach in node (also, I have rather limited experience in webservice programming). If you know a resource where I could read up on how to approach architecting a nodejs app, please don't hesitate to point me to it.
How do I wire in the authentication? Do I pass several handlers into
every route that requires authentication, or is there a more elegant
way to do this?
You should use the session middleware. Here is some pseudo code:
var http = require('http');
var app = express();
var authorize = function(req, res, next) {
if(req.session && req.session.appname && req.session.appname === true) {
// redirect to login page
return;
}
next();
}
app.use(express.session());
app.all('/admin*', authorize, function(req, res, next) {
});
How does the Express middleware (like app.use(express.bodyParser()))
work? Do they alter contents of the request or response object?
Specifically, if I use the body parser (internally formidable?), where
do I access the request data this is supposed to parse?
Every middleware have an access to the request and response object. So, yes, it modifies it. Normally attach properties to it. This means that inside your handler (which is also a middleware) you may write:
if(req.body && req.body.formsubmitted && req.body.formsubmitted === 'yes') {
var data = {
title: req.body.title,
text: req.body.text,
type: req.body.type
}
// store the data
}
When using authentication and I have, say, credentials stored in a
database with more information about the individual client associated,
at what point do I extract that information? I.e., when a user logs
in, do I fetch the user record on login and pass it on, or do I fetch
it in every handler that requires the information?
I think that you should do the things the same way as in any other server side language. Keep the state of the user (logged/not-logged) inside a session. You may also keep the user's id and fetch the data for him whatever you need. It depends of your case, but you have the ability to cache information. Because node is not like PHP for example, I mean it's not dieing.
Ultimately, do you know of an open source application that I could
take a look at? I'd like to see something that has simple
authentication and maybe even utilizes formidable, since uploading a
file is one of my requirements.
Yep. I wrote an article about really simple MVC web site with admin panel. It is available here. And the code of it is here.
A simple way to implement authentication (if you don't want to use additional modules):
var checkAuth = function(req, res, next) {
if(!req.session.user)
{
// Redirect to login form
res.redirect("/login");
}
else
{
// Proceed to member's area
next();
}
};
app.get("/member/page", checkAuth, function(req, res) {
// render view, etc
});
bodyParser parses / converts the body of a POST request into an object, which helps with getting form submission values.
The route that handles your login form submission can access username / password like this:
var username = req.body.username;
var password = req.body.password;
At this point you'd query your database to select from users where the username and password matches (you'd want to use password encryption in a production environment).
If you get a record back in the query result, set it in the session. A simple way to do this is:
req.session.user = userRecord
(Adjust for your session middleware)
If you are looking for REST, I recommend using either Restify or booster
For authentication (distinct from authorization), use standard Basic, which can be handled by express.basicAuth() just to parse it and place it on the req object. Personally, I don't like basicAuth because it returns a 401 if there is no login, whereas the process of authenticating is different than determining if authentication is necessary.
For more advanced authentication, as well as session management, use cansecurity or passport. For authorization, you either can put individual middleware in each route, use cansecurity's middlewares, or use its declarative authorization.
Disclosure: I am the author of both booster and cansecurity.
If your goal is to build a RESTful API in Node.js, my best bet would be Restify, which uses a similar aproach of routes like Express, but eliminates all the high level stuff(templating, etc.) and ads backend functionalities(ie: body parser, ip blacklist, requests per hour).
For the authentication part, I would use another library perhaps, and wire it to a particular route. There are ORM's too that can solve your database needs(mongo and mysql are well supported, both for the "noSQL" fans and the classic db aproach ones).