I don't know where to start with this question!
Basically, I would like to build a control panel, web based, using node.js as the back end. This would run on my raspberry pi running Ubuntu Server.
The idea is that I can gain statistics (CPU, Temperature, Disk Space etc) and set up basic features like MongoDB database, hosting etc.
Now, this is obviously possible just like many web panels out there, however, is this possible with node.js.
I suppose the first question would be, can I start/stop services (reboot server, start MongoDB etc) via Node.Js, and secondly, can I get info from this to display in my web panel?
I tried Google but for the first time, I don't even know what question to ask :)
I found Node JS examples of running command line commands, however, when passing in simple commands, like "node -v" it crashed, so I am not sure this is the method used by other commercial server web panels.
Any advice would be great :)
Thanks
You should try this tutorial: https://stackabuse.com/executing-shell-commands-with-node-js/
From node doc for child_process:
https://nodejs.org/api/child_process.html
const { exec } = require('child_process');
exec('"/path/to/test file/test.sh" arg1 arg2');
// Double quotes are used so that the space in the path is not interpreted as
// a delimiter of multiple arguments.
exec('echo "The \\$HOME variable is $HOME"');
// The $HOME variable is escaped in the first instance, but not in the second.
I have a Express backend behind AWS Cloudfront. How properly set trust proxy for AWS Cloud Front:
app.set('trust proxy', function (ip) {
if ( ???????????? ) return true; // trusted IPs
else return false;
});
AWS Cloudfront use tons of ip address and is insecure validate all AWS ip address because anyone with an AWS EC2 instance have a valid IP.
The Problem
As you mentioned AWS CloudFront uses a long list of IP Address ranges. It's mentioned in their documenation. You can see them via this one liner (source, requires jq which you can get from brew in MacOs.):
curl 'https://ip-ranges.amazonaws.com/ip-ranges.json' | jq -r '.prefixes[] | select(.service=="CLOUDFRONT") | .ip_prefix'
(Update: or directly from http://d7uri8nf7uskq.cloudfront.net/tools/list-cloudfront-ips as mentioned in their doc.)
Right now, April 2021, it is giving me 122 ranges.
The Solution
You can make an AJAX call to this file in Node, parse the JSON file, get the list as an array of string (cloudFrontIps), and pass that to app.set('trust proxy', ['loopback', ...cloudFrontIps]).
Good news!
The good news is someone else has already done it! Check https://github.com/nhammond101/cloudfront-ip-ranges out.
Final Notes
It's obvious, but worth mentioning that getting this list in asynchronous! So, you might want to delay (e.g. await) your app start until this list is available. It's not a must though -- calling app.set after the HTTP server is up should work, thought for that short duration you will be recording CloudFront's IP.
You might want to call this file and get the new list periodically. The package is suggesting every 12 hours, using setTimeout.
My understanding is calling app.set on a running server will make the new list applicable on future calls immediately, without needing to restart. I am getting this impression by how X-Forward-For is examined on every request, and how app.set is calling compileTrust function on it's invocation. So, TL;DR: You shouldn't be needing to restart the server every 12 hours for this!
I look at express's code and it seems like app.set overrides (and not appends) the list every time you call it. So if you have some IPs of your own (e.g. your VPC's CIDR in AWS ELB), you have to manually add it to the list every time you call this app.set in your setTimeout.
I have a total of 7 machines in a test setup that run test scripts between any two combinations of those machines. These scripts require the IP address of the machines involved in the test run. I'd like to store each machine's IP address in a variable to not only make it easier to reference and call in our JavaScript test scripts, but also design the variables so that they'll be set by pinging the machine to get an up to date version of said address. I don't expect them to change, but you never know what might happen and this would allow to completely ignore them for the most part.
We're using Squish as our IDE for running JavaScript test scripts and Squish has an in house function OS.system that lets you run the command prompt.
It's probably obvious that I can't just use something like const pc1 = OS.system("ping pc-name") since ping gives a return far more than just the IP address.
I was wondering if there was a simpler way to pass a machine's ip address to a JavaScript variable.
I don't know which OS you are using but you can do something like that in Linux bash:
ping -c 1 google.de | grep PING | awk '{print $3}'
But I would recommend you to use a library for that.
You can take a look at this: (node.js)
https://nodejs.org/api/dns.html
Okay, I've spent over a week trying to figure this out to no avail, so if anyone has a clue, you are a hero. This isn't going to be an easy question to answer, unless I am being a dunce.
I am using node-http-proxy to proxy sticky sessions to 16 node.js workers running on different ports.
I use Socket.IO's Web Sockets to handle a bunch of different types of requests, and use traditional requests as well.
When I switched my server over to proxying via node-http-proxy, a new problem crept up in that sometimes, my Socket.IO session cannot establish a connection.
I literally can't stably reproduce it for the life of me, with the only way to turn it on being to throw a lot of traffic from multiple clients to the server.
If I reload the user's browser, it can then sometimes re-connect, and sometimes not.
Sticky Sessions
I have to proxy sticky sessions as my app authenticates on a per-worker basis, and so it routes a request based on its Connect.SID cookie (I am using connect/express).
Okay, some code
This is my proxy.js file that runs in node and routes to each of the workers:
var http = require('http');
var httpProxy = require('http-proxy');
// What ports the proxy is routing to.
var data = {
proxyPort: 8888,
currentPort: 8850,
portStart: 8850,
portEnd: 8865,
};
// Just gives the next port number.
nextPort = function() {
var next = data.currentPort++;
next = (next > data.portEnd) ? data.portStart : next;
data.currentPort = next;
return data.currentPort;
};
// A hash of Connect.SIDs for sticky sessions.
data.routes = {}
var svr = httpProxy.createServer(function (req, res, proxy) {
var port = false;
// parseCookies is just a little function
// that... parses cookies.
var cookies = parseCookies(req);
// If there is an SID passed from the browser.
if (cookies['connect.sid'] !== undefined) {
var ip = req.connection.remoteAddress;
if (data.routes[cookies['connect.sid']] !== undefined) {
// If there is already a route assigned to this SID,
// make that route's port the assigned port.
port = data.routes[cookies['connect.sid']].port;
} else {
// If there isn't a route for this SID,
// create the route object and log its
// assigned port.
port = data.currentPort;
data.routes[cookies['connect.sid']] = {
port: port,
}
nextPort();
}
} else {
// Otherwise assign a random port, it will/
// pick up a connect SID on the next go.
// This doesn't really happen.
port = nextPort();
}
// Now that we have the chosen port,
// proxy the request.
proxy.proxyRequest(req, res, {
host: '127.0.0.1',
port: port
});
}).listen(data.proxyPort);
// Now we handle WebSocket requests.
// Basically, I feed off of the above route
// logic and try to route my WebSocket to the
// same server regular requests are going to.
svr.on('upgrade', function (req, socket, head) {
var cookies = parseCookies(req);
var port = false;
// Make sure there is a Connect.SID,
if (cookies['connect.sid'] != undefined) {
// Make sure there is a route...
if (data.routes[cookies['connect.sid']] !== undefined) {
// Assign the appropriate port.
port = data.routes[cookies['connect.sid']].port;
} else {
// this has never, ever happened, i've been logging it.
}
} else {
// this has never, ever happened, i've been logging it.
};
if (port === false) {
// this has never happened...
};
// So now route the WebSocket to the same port
// as the regular requests are getting.
svr.proxy.proxyWebSocketRequest(req, socket, head, {
host: 'localhost',
port: port
});
});
Client Side / The Phenomena
Socket connects like so:
var socket = io.connect('http://whatever:8888');
After about 10 seconds on logging on, I get this error back on this listener, which doesn't help much.
socket.on('error', function (data) {
// this is what gets triggered. ->
// Firefox can't establish a connection to the server at ws://whatever:8888/socket.io/1/websocket/Nnx08nYaZkLY2N479KX0.
});
The Socket.IO GET request that the browser sends never comes back - it just hangs in pending, even after the error comes back, so it looks like a timeout error. The server never responds.
Server Side - A Worker
This is how a worker receives a socket request. Pretty simple. All workers have the same code, so you think one of them would get the request and acknowledge it...
app.sio.socketio.sockets.on('connection', function (socket) {
// works... some of the time! all of my workers run this
// exact same process.
});
Summary
That's a lot of data, and I doubt anyone is willing to confront it, but i'm totally stumped, don't know where to check next, log next, whatever, to solve it. I've tried everything I know to see what the problem is, to no avail.
UPDATE
Okay, I am fairly certain that the problem is in this statement on the node-http-proxy github homepage:
node-http-proxy is <= 0.8.x compatible, if you're looking for a >=
0.10 compatible version please check caronte
I am running Node.js v0.10.13, and the phenomena is exactly as some have commented in github issues on this subject: it just drops websocket connections randomly.
I've tried to implement caronte, the 'newer' fork, but it is not at all documented and I have tried my hardest to piece together their docs in a workable solution, but I can't get it forwarding websockets, my Socket.IO downgrades to polling.
Are there any other ideas on how to get this implemented and working? node-http-proxy has 8200 downloads yesterday! Sure someone is using a Node build from this year and proxying websockets....
What I am look for exactly
I want to accomplish a proxy server (preferrably Node) that proxies to multiple node.js workers, and which routes the requests via sticky sessions based on a browser cookie. This proxy would need to stably support traditional requests as well as web sockets.
Or...
I don't mind accomplishing the above via clustered node workers, if that works. My only real requirement is maintaining sticky sessions based on a cookie in the request header.
If there is a better way to accomplish the above than what I am trying, I am all for it.
In general I don't think node is not the most used option as a proxy server, I, for one use nginx as a frontend server for node and it's a really great combination. Here are some instructions to install and use the nginx sticky sessions module.
It's a lightweight frontend server with json like configuration, solid and very well tested.
nginx is also a lot faster if you want to serve static pages, css. It's ideal to configure your caching headers, redirect traffic to multiple servers depending on domain, sticky sessions, compress css and javascript, etc.
You could also consider a pure load balancing open source solution like HAProxy. In any case I don't believe node is the best tool for this, it's better to use it to implement your backend only and put something like nginx in front of it to handle the usual frontend server tasks.
I agree with hexacyanide. To me it would make the most sense to queue workers through a service like redis or some kind of Message Query system. Workers would be queued through Redis Pub/Sub functionality by web nodes(which are proxied). Workers would callback upon error, finish, or stream data in realtime with a 'data' event. Maybe check out the library kue. You could also roll your own similar library. RabbitMQ is another system for similar purpose.
I get using socket.io if you're already using that technology, but you need to use tools for their intended purpose. Redis or a MQ system would make the most sense, and pair great with websockets(socket.io) to create realtime, insightful applications.
Session Affinity(sticky sessions) is supported through Elastic LoadBalancer for aws, this supports webSockets. A PaaS provider(Modulus) does this exactly. Theres also satalite which provides sticky sessions for node-http-proxy, however I have no idea if it supports webSockets.
I've been looking into something very similar to this myself, with the intent of generating (and destroying) Node.js cluster nodes on the fly.
Disclaimer: I'd still not recommend doing this with Node; nginx is more stable for the sort of design architecture that you're looking for, or even more so, HAProxy (very mature, and easily supports sticky-session proxying). As #tsturzl indicates, there is satellite, but given the low volume of downloads, I'd tread carefully (at least in a production environment).
That said, since you appear to have everything already set up with Node, rebuilding and re-architecting may be more work than it's worth. Therefore, to install the caronte branch with NPM:
Remove your previous http-node-proxy Master installation with npm uninstall node-proxy and/or sudo npm -d uninstall node-proxy
Download the caronte branch .zip and extract it.
Run npm -g install /path/to/node-http-proxy-caronte
In my case, the install linkage was broken, so I had to run sudo npm link http-proxy
I've got it up and running using their basic proxy example -- whether or not this resolves your dropped sessions issue or not, only you will know.
I am new to web servers and node.js and I need some help.
I have no idea what to put in the .listen();
I think since I want it to connect to the internet the server needs to listen to port 80 but but I don't know what to put as the second value.
.listen(80, "What do I add here?");
Also i have a free domain name (www.example.co.cc) that is pointing to a dynamic dns (DnsExit) since I dynamic ip. I installed to program needed to update my ip address.
Is there anything I am missing?
Have you seen the example on the homepage of the Node.js project?
http://nodejs.org/
It clearly demonstrated .listen( 1337, "127.0.0.1" ); and then the next line reads Server running at http://127.0.0.1:1337/ - so the second argument is the IP you want to listen on. If you then take a look at the documentation you will see that this second argument is actually optional, if you omit it, Node.js will accept incoming connections directed at any IPv4 address.
http://nodejs.org/docs/v0.5.6/api/http.html#server.listen