Develop Reference

sending a websocket request through a proxy server in nodeJS [duplicate]

Generalizing that would be the question... how to make websockets to go through a proxy in node.js?
In my particular case I'm using pusher.com with the node.js client library they recommend. Looking inside the code I would like to know some hints on what I should change in order to make this library to work with a proxy... you can take a look in the code here
Maybe I should somehow replace or modified the websockets module that is being used by the library?
EDIT
Thanks for your answers/comments! A couple of things to take into consideration (excuse me if I'm wrong with some/all of them, just learning):
I don't want to create a proxy server. I just want to use an existent proxy server within my company in order to proxified my websockets requests (particularly pusher.com)
Just to let you know, if I use a proxifier like the one for windows Proxifier and I set up the rule to inspect for all connections to port 443 to go through the proxy server proxy-my.coporate.com:1080 (type SOCKS5) it works like a charm.
But I don't want to go this way. I want to programatically configuring this proxy server within my node js code (even if that involved to modified the pusher library I mentioned)
I know how to do this for HTTP using Request module (look for the section that mentions how to use a proxy).
I want a similarly thing for websockets.

From
https://www.npmjs.com/package/https-proxy-agent
var url = require('url');
var WebSocket = require('ws');
var HttpsProxyAgent = require('https-proxy-agent');
// HTTP/HTTPS proxy to connect to
var proxy = process.env.http_proxy || 'http://168.63.76.32:3128';
console.log('using proxy server %j', proxy);
// WebSocket endpoint for the proxy to connect to
var endpoint = process.argv[2] || 'ws://echo.websocket.org';
var parsed = url.parse(endpoint);
console.log('attempting to connect to WebSocket %j', endpoint);
// create an instance of the `HttpsProxyAgent` class with the proxy server information
var options = url.parse(proxy);
var agent = new HttpsProxyAgent(options);
// finally, initiate the WebSocket connection
var socket = new WebSocket(endpoint, { agent: agent });
socket.on('open', function () {
console.log('"open" event!');
socket.send('hello world');
});
socket.on('message', function (data, flags) {
console.log('"message" event! %j %j', data, flags);
socket.close();
});

Using a proxy for websockets should work roughly the same as for https connections; you should use the CONNECT method. At least that's what both the HTTP and HTML5 specs say. So if your proxy implements CONNECT, you're good to go.

Try node-http-proxy
It allows you to send http or websocket requests through a proxy.
var http = require('http'),
httpProxy = require('http-proxy');
//
// Create a basic proxy server in one line of code...
//
// This listens on port 8000 for incoming HTTP requests
// and proxies them to port 9000
httpProxy.createServer(9000, 'localhost').listen(8000);
//
// ...and a simple http server to show us our request back.
//
http.createServer(function (req, res) {
res.writeHead(200, { 'Content-Type': 'text/plain' });
res.write('request successfully proxied!' + '\n' + JSON.stringify(req.headers, true, 2));
res.end();
}).listen(9000);
Source: link

Most web proxies don't support websockets yet. The best workaround is to use encryption by specifying wss:// (websocket secure protocol):
wss://ws.pusherapp.com:[port]/app/[key]

When tunnelling a TLS connection, how to pass additional information?

I have created a bare-bones HTTP proxy that performs HTTP tunnelling using HTTP CONNECT method.
const http = require('http');
const https = require('https');
const pem = require('pem');
const net = require('net');
const util = require('util');
const createHttpsServer = (callback) => {
pem.createCertificate({
days: 365,
selfSigned: true
}, (error, {serviceKey, certificate, csr}) => {
const httpsOptions = {
ca: csr,
cert: certificate,
key: serviceKey
};
const server = https.createServer(httpsOptions, (req, res) => {
// How do I know I here whats the target server port?
res.writeHead(200);
res.end('OK');
});
server.listen((error) => {
if (error) {
console.error(error);
} else {
callback(null, server.address().port);
}
});
});
};
const createProxy = (httpsServerPort) => {
const proxy = http.createServer();
proxy.on('connect', (request, requestSocket, head) => {
// Here I know whats the target server PORT.
const targetServerPort = Number(request.url.split(':')[1]);
console.log('target server port', targetServerPort);
const serverSocket = net.connect(httpsServerPort, 'localhost', () => {
requestSocket.write(
'HTTP/1.1 200 Connection established\r\n\r\n'
);
serverSocket.write(head);
serverSocket.pipe(requestSocket);
requestSocket.pipe(serverSocket);
});
});
proxy.listen(9000);
};
const main = () => {
createHttpsServer((error, httpsServerPort) => {
if (error) {
console.error(error);
} else {
createProxy(httpsServerPort);
}
});
};
main();
The server accepts a HTTPS connection and responds with "OK" message without forwarding the request further.
As you can see in the code (see // Here I know whats the target server PORT.), I can obtain the target server's port within the HTTP CONNECT event handler. However, I am unable to figure out how to pass this information to the createHttpsServer HTTP server router (see // How do I know I here whats the target server port?).
When tunnelling a TLS connection, how to pass additional information?
The above code can be tested by running:
$ node proxy.js &
$ curl --proxy http://localhost:9000 https://localhost:59194/foo.html -k
The objective is to respond with "OK localhost:59194".

You can't add anything to a TLS stream (thankfully), short of tunneling it inside another protocol⁠—which is what the Connect method already does. But, since you have the HTTP proxy and the HTTPS server in the same codebase, you don't need to fling the TLS stream over the network another time. Instead, you want to parse the TLS stream, and then you can pass any variables to the code that handles it.
However, after parsing TLS you'll still have a raw HTTP stream, and you'll need an HTTP server to turn it into requests and to handle responses.
The quick and rather dirty way to go about it is to use Node's HTTPS server to both decode TLS and parse HTTP. But the server's API doesn't provide for dealing with sockets that are already connected, and server's code isn't cleanly separated from connection code. So you need to hijack the server's internal connection-handling logic—this is of course susceptible to breakage in case of future changes:
const http = require('http');
const https = require('https');
const pem = require('pem');
const createProxy = (httpsOptions) => {
const proxy = http.createServer();
proxy.on('connect', (request, requestSocket, head) => {
const server = https.createServer(httpsOptions, (req, res) => {
res.writeHead(200);
res.end('OK');
});
server.emit('connection', requestSocket);
requestSocket.write('HTTP/1.1 200 Connection established\r\n\r\n');
});
proxy.listen(9000);
};
const main = () => {
pem.createCertificate({
days: 365,
selfSigned: true
}, (error, {serviceKey, certificate, csr}) => {
createProxy({
ca: csr,
cert: certificate,
key: serviceKey
});
});
};
main();
To avoid creating an HTTPS server instance on every request, you can move the instance out and tack you data onto the socket object instead:
const server = https.createServer(httpsOptions, (req, res) => {
res.writeHead(200);
// here we reach to the net.Socket instance saved on the tls.TLSSocket object,
// for extra dirtiness
res.end('OK ' + req.socket._parent.marker + '\n');
});
proxy.on('connect', (request, requestSocket, head) => {
requestSocket.marker = Math.random();
server.emit('connection', requestSocket);
requestSocket.write('HTTP/1.1 200 Connection established\r\n\r\n');
});
With the above code, if you do several successive requests:
curl --proxy http://localhost:9000 https://localhost:59194/foo.html \
https://localhost:59194/foo.html https://localhost:59194/foo.html \
https://localhost:59194/foo.html https://localhost:59194/foo.html -k
then you'll also notice that they're processed on a single connection, which is nice:
OK 0.6113572936982015
OK 0.6113572936982015
OK 0.6113572936982015
OK 0.6113572936982015
OK 0.6113572936982015
I can't quite vouch that nothing will be broken by handing the socket to the HTTPS server while the proxy server already manages it. [The server has the presence of mind to not overwrite another instance on the socket object](https://github.com/nodejs/node/blob/v10.9.0/lib/_http_server.js#L331), but otherwise seems to be rather closely involved with the socket. You'll want to test it with longer-running connections.
As for the `head` argument, [which can indeed contain initial data](https://www.rfc-editor.org/rfc/rfc2817#section-5.2):
you might be able to put it back on the stream with requestSocket.unshift(head), but I'm not sure that it won't be immediately consumed by the proxy server.
Or, you might be able to chuck it over to the HTTPS server with requestSocket.emit('data', head) since the HTTP server seems to use the stream events, however TLS socket source calls read() for whatever reason, and that's mutually exclusive with the events, so I'm not sure how they even work with each other.
One solution would be to make your own wrapper for stream.Duplex that will forward all calls and events, except for read() in the case when this initial buffer exists—and then use this wrapper in place of requestSocket. But you'll then need to replicate the 'data' event also, in accordance with the logic of Node's readable streams.
Finally, you can try creating a new duplex stream, write head and pipe the socket to it, like you did initially, and use the stream in place of the socket for the HTTPS server—not sure that it will be compatible with HTTP server's rather overbearing management of the socket.
An cleaner approach is to decode the TLS stream and use a standalone parser for the resultant HTTP stream. Thankfully, Node has a tls module that is nicely isolated and turns TLS sockets into regular sockets:
proxy.on('connect', (request, requestSocket, head) => {
const httpSocket = new tls.TLSSocket(requestSocket, {
isServer: true,
// this var can be reused for all requests,
// as it's normally saved on an HTTPS server instance
secureContext: tls.createSecureContext(httpsOptions)
});
...
});
See caveats on tls.createSecureContext regarding replicating the behavior of the HTTPS server.
Alas, Node's HTTP parser isn't so usable: it's a C library, which necessitates quite a bit of legwork between the socket and the parser calls. And the API can (and does) change between versions, without warnings, with a larger surface for incompatibilities compared to the HTTP server internals used above.
There are NPM modules for parsing HTTP: e.g. one, two, but none seem too mature and maintained.
I also have doubts about the feasibility of a custom HTTP server because network sockets tend to require plenty of nurture over time due to edge cases, with hard-to-debug timeout issues and such—which should all be already accounted for in the Node's HTTP server.
P.S. One possible area of investigation is how the Cluster module handles connections: afaik the parent process in a cluster hands connection sockets over to the children, but it doesn't fork on every request—which suggests that the child processes somehow deal with connected sockets, in code that's outside of an HTTP server instance. However, since the Cluster module is now in the core, it may exploit non-public APIs.

Modifying MongoDB to allow cross origin requests

I am relatively new to MongoDB so I am still getting used to it.
Currently I am trying to post a json object to mongo from the client side using the code below in javascript.
var addUserButton = document.getElementById('add-user');
var userNameInput = document.getElementById('name');
addUserButton.onclick = function() {
var newUser = new Object();
newUser.name = userNameInput.value;
var newUserJson = { 'name': newUser.name};
$.post('127.0.0.1:27017/test', newUserJson);
};
When ever I run this code though I get error stating:
XMLHttpRequest cannot load %3127.0.0.1:27017/test. Cross origin
requests are only supported for protocol schemes... etc
I read up on this and was wondering if adding the following CORS handler to mongoDB would fix this. If it is correct, how would I go about adding this to MongoDB? I could not find documentation on how to add this CORS handler to mongo
function handleCors(req, res, callback) {
res.setHeader('Access-Control-Allow-Origin', '*')
res.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE,OPTIONS');
res.setHeader('Access-Control-Allow-Headers', 'Authorization');
// CORS OPTIONS request, simply return 200
if (req.method == 'OPTIONS') {
res.statusCode = 200;
res.end();
callback.onOptions();
return;
}
callback.onContinue();
};

Your work by concept is having issue.
Your post method is directly hitting to TCP Protocol here, which should not be.
You should call some http request based on some Rest API which is to be in Server.
At server routing, you should handle the CORS (http request) and then the db layer methods should get the data for Update/Select etc.
By the way, for Server Routing you can use high level node framework like expressjs.
If you want a complete example for this, you may go through want to look through this example and
the explanation can be found in at Single Page Application with Angular.js, Node.js and MongoDB - phloxblog.

Socket IO | How to get the client transport type on the serverside?

I need to know what transport method a client is using for some conditional statements on the nodeJS serverside.
Does anyone know how I can get that information? Is it held within the client object?

As of Socket.IO 1.0:
Client:
socket.on('connect', function() {
console.log(socket.io.engine.transport.name);
}
Server:
io.sockets.on('connection', function(socket) {
console.log(socket.conn.transport.name);
}

In socket.io 0.7.6
io.sockets.on('connection', function(client) {
console.log(io.transports[client.id].name);
});

April 2012, this works: socket.transport

I'm sure you can find it if you dig in the internals of a client object, although without knowing why you need this I have to recommend against this kind of check for 2 reasons:
Firstly, since it isn't in the API the developers have absolutely no responsibility to keep things backward compatible, so any given version might implement/store that information differently, which will only ripple into your own development and cause problems.
Secondly, and more importantly, I suggest you rethink your design, the communication with the server thru socket.io is built to be transparent to the method being used. There should be no difference on either side. That's the purpose of the library, designing an app that behaves otherwise is totally orthogonal to that idea.

for reference's sake and google stumbles:-
in case anyone is still using v0.9 (or possibly earlier)
you can access this info from client side like this:
var socket = io.connect();
console.log(socket.socket.transport.name); //log the name of the transport being used.
answer found on google groups https://groups.google.com/forum/#!topic/socket_io/yx_9wJiiAg0

I believe this will solve your problem. My trick here is to save the transport type on the HTTP Request object once the client connects. You can then pick it up in your callback later. First we tweak the Listener class:
var io = require('socket.io'),
io.Listener.prototype._onConnectionOld = io.Listener.prototype._onConnection;
io.Listener.prototype._onConnection = function(transport, req, res, up, head){
req.socketIOTransport = transport; // Take note of the transport type
this._onConnectionOld.call(this, transport, req, res, up, head);
};
And then below in the body of your app:
var socket = io.listen(app),
socket.on('connection', function(client){
console.log(client.request.socketIOTransport); // Lets check that transport
// ...
});
Hope this helps!

io.connect.managers['connect url/port'].engine.transport

Basic Ajax send/receive with node.js

So I'm trying to make a very basic node.js server that with take in a request for a string, randomly select one from an array and return the selected string. Unfortunately I'm running into a few problems.
Here's the front end:
function newGame()
{
guessCnt=0;
guess="";
server();
displayHash();
displayGuessStr();
displayGuessCnt();
}
function server()
{
xmlhttp = new XMLHttpRequest();
xmlhttp.open("GET","server.js", true);
xmlhttp.send();
string=xmlhttp.responseText;
}
This should send the request to server.js:
var http = require('http');
var choices=["hello world", "goodbye world"];
console.log("server initialized");
http.createServer(function(request, response)
{
console.log("request recieved");
var string = choices[Math.floor(Math.random()*choices.length)];
console.log("string '" + string + "' chosen");
response.on(string);
console.log("string sent");
}).listen(8001);
So clearly there are several things going wrong here:
I get the feeling the way I am "connecting" these two files isn't correct both in the xmlhttp.open method and in using response.on to send the string back to the front end.
I'm a little confused with how I call this page on localhost. The front end is named index.html and the sever posts to 8001. What address should I be go to on localhost in order to access the initial html page after I have initialized server.js? Should I change it to .listen(index.html) or something like that?
are there other obvious problems with how I am implementing this (using .responsetext etc.)
(sorry for the long multi-question post but the various tutorials and the node.js source all assume that the user already has an understanding of these things.)

Your request should be to the server, NOT the server.js file which instantiates it. So, the request should look something like this: xmlhttp.open("GET","http://localhost:8001/", true); Also, you are trying to serve the front-end (index.html) AND serve AJAX requests at the same URI. To accomplish this, you are going to have to introduce logic to your server.js that will differentiate between your AJAX requests and a normal http access request. To do this, you'll want to either introduce GET/POST data (i.e. call http://localhost:8001/?getstring=true) or use a different path for your AJAX requests (i.e. call http://localhost:8001/getstring). On the server end then, you'll need to examine the request object to determine what to write on the response. For the latter option, you need to use the 'url' module to parse the request.
You are correctly calling listen() but incorrectly writing the response. First of all, if you wish to serve index.html when navigating to http://localhost:8001/, you need to write the contents of the file to the response using response.write() or response.end(). First, you need to include fs=require('fs') to get access to the filesystem. Then, you need to actually serve the file.
XMLHttpRequest needs a callback function specified if you use it asynchronously (third parameter = true, as you have done) AND want to do something with the response. The way you have it now, string will be undefined (or perhaps null), because that line will execute before the AJAX request is complete (i.e. the responseText is still empty). If you use it synchronously (third parameter = false), you can write inline code as you have done. This is not recommended as it locks the browser during the request. Asynchronous operation is usually used with the onreadystatechange function, which can handle the response once it is complete. You need to learn the basics of XMLHttpRequest. Start here.
Here is a simple implementation that incorporates all of the above:
server.js:
var http = require('http'),
fs = require('fs'),
url = require('url'),
choices = ["hello world", "goodbye world"];
http.createServer(function(request, response){
var path = url.parse(request.url).pathname;
if(path=="/getstring"){
console.log("request recieved");
var string = choices[Math.floor(Math.random()*choices.length)];
console.log("string '" + string + "' chosen");
response.writeHead(200, {"Content-Type": "text/plain"});
response.end(string);
console.log("string sent");
}else{
fs.readFile('./index.html', function(err, file) {
if(err) {
// write an error response or nothing here
return;
}
response.writeHead(200, { 'Content-Type': 'text/html' });
response.end(file, "utf-8");
});
}
}).listen(8001);
console.log("server initialized");
frontend (part of index.html):
function newGame()
{
guessCnt=0;
guess="";
server();
displayHash();
displayGuessStr();
displayGuessCnt();
}
function server()
{
xmlhttp = new XMLHttpRequest();
xmlhttp.open("GET","http://localhost:8001/getstring", true);
xmlhttp.onreadystatechange=function(){
if (xmlhttp.readyState==4 && xmlhttp.status==200){
string=xmlhttp.responseText;
}
}
xmlhttp.send();
}
You will need to be comfortable with AJAX. Use the mozilla learning center to learn about XMLHttpRequest. After you can use the basic XHR object, you will most likely want to use a good AJAX library instead of manually writing cross-browser AJAX requests (for example, in IE you'll need to use an ActiveXObject instead of XHR). The AJAX in jQuery is excellent, but if you don't need everything else jQuery offers, find a good AJAX library here: http://microjs.com/. You will also need to get comfy with the node.js docs, found here. Search http://google.com for some good node.js server and static file server tutorials. http://nodetuts.com is a good place to start.
UPDATE: I have changed response.sendHeader() to the new response.writeHead() in the code above !!!

Express makes this kind of stuff really intuitive. The syntax looks like below :
var app = require('express').createServer();
app.get("/string", function(req, res) {
var strings = ["rad", "bla", "ska"]
var n = Math.floor(Math.random() * strings.length)
res.send(strings[n])
})
app.listen(8001)
https://expressjs.com
If you're using jQuery on the client side you can do something like this:
$.get("/string", function(string) {
alert(string)
})

I was facing following error with code (nodejs 0.10.13), provided by ampersand:
origin is not allowed by access-control-allow-origin
Issue was resolved changing
response.writeHead(200, {"Content-Type": "text/plain"});
to
response.writeHead(200, {
'Content-Type': 'text/html',
'Access-Control-Allow-Origin' : '*'});

Here is a fully functional example of what you are trying to accomplish. I created the example inside of hyperdev rather than jsFiddle so that you could see the server-side and client-side code.
View Code:
https://hyperdev.com/#!/project/destiny-authorization
View Working Application: https://destiny-authorization.hyperdev.space/
This code creates a handler for a get request that returns a random string:
app.get("/string", function(req, res) {
var strings = ["string1", "string2", "string3"]
var n = Math.floor(Math.random() * strings.length)
res.send(strings[n])
});
This jQuery code then makes the ajax request and receives the random string from the server.
$.get("/string", function(string) {
$('#txtString').val(string);
});
Note that this example is based on code from Jamund Ferguson's answer so if you find this useful be sure to upvote him as well. I just thought this example would help you to see how everything fits together.