Develop Reference

Use MQTTNet Server with MQTT.js client

I have started an MQTT server just like this example.
This code is hosted in an ASP.Net Core 2.0 application but I have tried console application with no luck.
I have also setup a client using the same demo as above and it connects perfectly. Also an Android client connects fine. But I have placed a MQTT.js client webpage but it will not connect with chrome showing net::ERR_CONNECTION_REFUSED error.
I believe that the problem is with server not supporting web sockets. Because if I start my client with WS type it will not connect.
var options = new MqttClientOptions
{
Server = "localhost",
//ConnectionType = MqttConnectionType.Tcp // Connects
ConnectionType = MqttConnectionType.Ws // Does not connect
};
Now MQTT.js supports TCP as far this link is telling. But I don't seem to be able to work it.
This is the code in my page javascript:
var client = mqtt.connect('tcp://localhost') //Also did mqtt://localhost
client.on('connect', function () {
client.subscribe('myTopic')
client.publish('myTopic', 'Hello mqtt')
})
client.on('message', function (topic, message) {
// message is Buffer
console.log(message.toString())
client.end()
})
I want to know how can I make javascript MQTT client to use TCP? (Any other js plugin maybe?) Or alternatively how can I enable WebSockets in MQTTNet.
Thanks for your help.

MQTT.js does support both native MQTT and MQTT over Websockets, but if you are embedding it in to a web app it can only use MQTT over Websockets because the browsers sandbox will not allow it to use arbitrary TCP connections.
As for enabling Websockets in the broker I can't see anything obvious in the code so you'll probably have to raise and issue against the github project to ask for details.

The next version of MQTTnet (2.5) will have support for WebSocket connections for the server via AspNetCore 2.0.

Javascript send string to python-script

I have an (html/js) application running on my localhost. I'd like to send some information (string) to a python script running in the background, which will just print the string, or save it as a .txt-file.
It seems websockets will do the job, but I cannot get around this (in my eyes a simple problem..?). All examples or libraries aims at specific usages or are depricated in the meanwhile. Also, maybe someone can point me to another principle like REST.?
I'm not really into web/ip/internetthings, but I need this to let the webpage initiate some python programs.
Any tips on how to achieve this?

I am doing something similar to have a web server (nodejs) control my raspberrypi (python).
I suggest you simply spawn your python script by your js server and make them communicate via stdin/stdout.
For example with nodejs:
var spawn = require('child_process').spawn;
var child = spawn(
'python3',
['./py/pi-ctrl.py']
);
var child_emit = function (message) {
child.stdin.write(message+"\n");
}
Then your js can just 'emit' anything to your python script, which listens to stdin:
while True:
line = input().split()

How to bring a gRPC defined API to the web browser

We want to build a Javascript/HTML gui for our gRPC-microservices. Since gRPC is not supported on the browser side, we thought of using web-sockets to connect to a node.js server, which calls the target service via grpc.
We struggle to find an elegant solution to do this. Especially, since we use gRPC streams to push events between our micro-services.
It seems that we need a second RPC system, just to communicate between the front end and the node.js server. This seems to be a lot of overhead and additional code that must be maintained.
Does anyone have experience doing something like this or has an idea how this could be solved?

Edit: Since Oct 23,2018 the gRPC-Web project is GA, which might be the most official/standardized way to solve your problem. (Even if it's already 2018 now... ;) )
From the GA-Blog: "gRPC-Web, just like gRPC, lets you define the service “contract” between client (web) and backend gRPC services using Protocol Buffers. The client can then be auto generated. [...]"
We recently built gRPC-Web (https://github.com/improbable-eng/grpc-web) - a browser client and server wrapper that follows the proposed gRPC-Web protocol. The example in that repo should provide a good starting point.
It requires either a standalone proxy or a wrapper for your gRPC server if you're using Golang. The proxy/wrapper modifies the response to package the trailers in the response body so that they can be read by the browser.
Disclosure: I'm a maintainer of the project.

Unfortunately, there isn't any good answer for you yet.
Supporting streaming RPCs from the browser fully requires HTTP2 trailers to be supported by the browsers, and at the time of the writing of this answer, they aren't.
See this issue for the discussion on the topic.
Otherwise, yes, you'd require a full translation system between WebSockets and gRPC. Maybe getting inspiration from grpc-gateway could be the start of such a project, but that's still a very long shot.

An official grpc-web (beta) implementation was released on 3/23/2018. You can find it at
https://github.com/grpc/grpc-web
The following instructions are taken from the README:
Define your gRPC service:
service EchoService {
rpc Echo(EchoRequest) returns (EchoResponse);
rpc ServerStreamingEcho(ServerStreamingEchoRequest)
returns (stream ServerStreamingEchoResponse);
}
Build the server in whatever language you want.
Create your JS client to make calls from the browser:
var echoService = new proto.grpc.gateway.testing.EchoServiceClient(
'http://localhost:8080');
Make a unary RPC call
var unaryRequest = new proto.grpc.gateway.testing.EchoRequest();
unaryRequest.setMessage(msg);
echoService.echo(unaryRequest, {},
function(err, response) {
console.log(response.getMessage());
});
Streams from the server to the browser are supported:
var stream = echoService.serverStreamingEcho(streamRequest, {});
stream.on('data', function(response) {
console.log(response.getMessage());
});
Bidirectional streams are NOT supported:
This is a work in progress and on the grpc-web roadmap. While there is an example protobuf showing bidi streaming, this comment make it clear that this example doesn't actually work yet.
Hopefully this will change soon. :)

https://github.com/tmc/grpc-websocket-proxy sounds like it may meet your needs. This translates json over web sockets to grpc (layer on top of grpc-gateway).

The grpc people at https://github.com/grpc/ are currently building a js implementation.
The repro is at https://github.com/grpc/grpc-web (gives 404 ->) which is currently (2016-12-20) in early access so you need to request access.

GRPC Bus WebSocket Proxy does exactly this by proxying all GRPC calls over a WebSocket connection to give you something that looks very similar to the Node GRPC API in the browser. Unlike GRPC-Gateway, it works with both streaming requests and streaming responses, as well as non-streaming calls.
There is both a server and client component.
The GRPC Bus WebSocket Proxy server can be run with Docker by doing docker run gabrielgrant/grpc-bus-websocket-proxy
On the browser side, you'll need to install the GRPC Bus WebSocket Proxy client with npm install grpc-bus-websocket-client
and then create a new GBC object with: new GBC(<grpc-bus-websocket-proxy address>, <protofile-url>, <service map>)
For example:
var GBC = require("grpc-bus-websocket-client");
new GBC("ws://localhost:8080/", 'helloworld.proto', {helloworld: {Greeter: 'localhost:50051'}})
.connect()
.then(function(gbc) {
gbc.services.helloworld.Greeter.sayHello({name: 'Gabriel'}, function(err, res){
console.log(res);
}); // --> Hello Gabriel
});
The client library expects to be able to download the .proto file with an AJAX request. The service-map provides the URLs of the different services defined in your proto file as seen by the proxy server.
For more details, see the GRPC Bus WebSocket Proxy client README

I see a lot of answers didn't point to a bidirectional solution over WebSocket, as the OP asked for browser support.
You may use JSON-RPC instead of gRPC, to get a bidirectional RPC over WebSocket, which supports a lot more, including WebRTC (browser to browser).
I guess it could be modified to support gRPC if you really need this type of serialization.
However, for browser tab to browser tab, request objects are not serializsed and are transfered natively, and the same with NodeJS cluster or thread workers, which offers a lot more performance.
Also, you can transfer "pointers" to SharedArrayBuffer, instead of serializing through the gRPC format.
JSON serialization and deserialization in V8 is also unbeatable.
https://github.com/bigstepinc/jsonrpc-bidirectional

Looking at the current solutions with gRPC over web, here is what's available out there at the time of writing this (and what I found):
gRPC-web: requires TypeScript for client
gRPC-web-proxy: requires Go
gRPC-gateway: requires .proto modification and decorations
gRPC-bus-websocket-proxy-server: as of writing this document it lacks tests and seems abandoned (edit: look at the comments by the original author!)
gRPC-dynamic-gateway: a bit of an overkill for simple gRPC services and authentication is awkward
gRPC-bus: requires something for the transport
I also want to shamelessly plug my own solution which I wrote for my company and it's being used in production to proxy requests to a gRPC service that only includes unary and server streaming calls:
gRPC-express
Every inch of the code is covered by tests. It's an Express middleware so it needs no additional modifications to your gRPC setup. You can also delegate HTTP authentication to Express (e.g with Passport).

Node.js http-proxy drops websocket requests

Okay, I've spent over a week trying to figure this out to no avail, so if anyone has a clue, you are a hero. This isn't going to be an easy question to answer, unless I am being a dunce.
I am using node-http-proxy to proxy sticky sessions to 16 node.js workers running on different ports.
I use Socket.IO's Web Sockets to handle a bunch of different types of requests, and use traditional requests as well.
When I switched my server over to proxying via node-http-proxy, a new problem crept up in that sometimes, my Socket.IO session cannot establish a connection.
I literally can't stably reproduce it for the life of me, with the only way to turn it on being to throw a lot of traffic from multiple clients to the server.
If I reload the user's browser, it can then sometimes re-connect, and sometimes not.
Sticky Sessions
I have to proxy sticky sessions as my app authenticates on a per-worker basis, and so it routes a request based on its Connect.SID cookie (I am using connect/express).
Okay, some code
This is my proxy.js file that runs in node and routes to each of the workers:
var http = require('http');
var httpProxy = require('http-proxy');
// What ports the proxy is routing to.
var data = {
proxyPort: 8888,
currentPort: 8850,
portStart: 8850,
portEnd: 8865,
};
// Just gives the next port number.
nextPort = function() {
var next = data.currentPort++;
next = (next > data.portEnd) ? data.portStart : next;
data.currentPort = next;
return data.currentPort;
};
// A hash of Connect.SIDs for sticky sessions.
data.routes = {}
var svr = httpProxy.createServer(function (req, res, proxy) {
var port = false;
// parseCookies is just a little function
// that... parses cookies.
var cookies = parseCookies(req);
// If there is an SID passed from the browser.
if (cookies['connect.sid'] !== undefined) {
var ip = req.connection.remoteAddress;
if (data.routes[cookies['connect.sid']] !== undefined) {
// If there is already a route assigned to this SID,
// make that route's port the assigned port.
port = data.routes[cookies['connect.sid']].port;
} else {
// If there isn't a route for this SID,
// create the route object and log its
// assigned port.
port = data.currentPort;
data.routes[cookies['connect.sid']] = {
port: port,
}
nextPort();
}
} else {
// Otherwise assign a random port, it will/
// pick up a connect SID on the next go.
// This doesn't really happen.
port = nextPort();
}
// Now that we have the chosen port,
// proxy the request.
proxy.proxyRequest(req, res, {
host: '127.0.0.1',
port: port
});
}).listen(data.proxyPort);
// Now we handle WebSocket requests.
// Basically, I feed off of the above route
// logic and try to route my WebSocket to the
// same server regular requests are going to.
svr.on('upgrade', function (req, socket, head) {
var cookies = parseCookies(req);
var port = false;
// Make sure there is a Connect.SID,
if (cookies['connect.sid'] != undefined) {
// Make sure there is a route...
if (data.routes[cookies['connect.sid']] !== undefined) {
// Assign the appropriate port.
port = data.routes[cookies['connect.sid']].port;
} else {
// this has never, ever happened, i've been logging it.
}
} else {
// this has never, ever happened, i've been logging it.
};
if (port === false) {
// this has never happened...
};
// So now route the WebSocket to the same port
// as the regular requests are getting.
svr.proxy.proxyWebSocketRequest(req, socket, head, {
host: 'localhost',
port: port
});
});
Client Side / The Phenomena
Socket connects like so:
var socket = io.connect('http://whatever:8888');
After about 10 seconds on logging on, I get this error back on this listener, which doesn't help much.
socket.on('error', function (data) {
// this is what gets triggered. ->
// Firefox can't establish a connection to the server at ws://whatever:8888/socket.io/1/websocket/Nnx08nYaZkLY2N479KX0.
});
The Socket.IO GET request that the browser sends never comes back - it just hangs in pending, even after the error comes back, so it looks like a timeout error. The server never responds.
Server Side - A Worker
This is how a worker receives a socket request. Pretty simple. All workers have the same code, so you think one of them would get the request and acknowledge it...
app.sio.socketio.sockets.on('connection', function (socket) {
// works... some of the time! all of my workers run this
// exact same process.
});
Summary
That's a lot of data, and I doubt anyone is willing to confront it, but i'm totally stumped, don't know where to check next, log next, whatever, to solve it. I've tried everything I know to see what the problem is, to no avail.
UPDATE
Okay, I am fairly certain that the problem is in this statement on the node-http-proxy github homepage:
node-http-proxy is <= 0.8.x compatible, if you're looking for a >=
0.10 compatible version please check caronte
I am running Node.js v0.10.13, and the phenomena is exactly as some have commented in github issues on this subject: it just drops websocket connections randomly.
I've tried to implement caronte, the 'newer' fork, but it is not at all documented and I have tried my hardest to piece together their docs in a workable solution, but I can't get it forwarding websockets, my Socket.IO downgrades to polling.
Are there any other ideas on how to get this implemented and working? node-http-proxy has 8200 downloads yesterday! Sure someone is using a Node build from this year and proxying websockets....
What I am look for exactly
I want to accomplish a proxy server (preferrably Node) that proxies to multiple node.js workers, and which routes the requests via sticky sessions based on a browser cookie. This proxy would need to stably support traditional requests as well as web sockets.
Or...
I don't mind accomplishing the above via clustered node workers, if that works. My only real requirement is maintaining sticky sessions based on a cookie in the request header.
If there is a better way to accomplish the above than what I am trying, I am all for it.

In general I don't think node is not the most used option as a proxy server, I, for one use nginx as a frontend server for node and it's a really great combination. Here are some instructions to install and use the nginx sticky sessions module.
It's a lightweight frontend server with json like configuration, solid and very well tested.
nginx is also a lot faster if you want to serve static pages, css. It's ideal to configure your caching headers, redirect traffic to multiple servers depending on domain, sticky sessions, compress css and javascript, etc.
You could also consider a pure load balancing open source solution like HAProxy. In any case I don't believe node is the best tool for this, it's better to use it to implement your backend only and put something like nginx in front of it to handle the usual frontend server tasks.

I agree with hexacyanide. To me it would make the most sense to queue workers through a service like redis or some kind of Message Query system. Workers would be queued through Redis Pub/Sub functionality by web nodes(which are proxied). Workers would callback upon error, finish, or stream data in realtime with a 'data' event. Maybe check out the library kue. You could also roll your own similar library. RabbitMQ is another system for similar purpose.
I get using socket.io if you're already using that technology, but you need to use tools for their intended purpose. Redis or a MQ system would make the most sense, and pair great with websockets(socket.io) to create realtime, insightful applications.
Session Affinity(sticky sessions) is supported through Elastic LoadBalancer for aws, this supports webSockets. A PaaS provider(Modulus) does this exactly. Theres also satalite which provides sticky sessions for node-http-proxy, however I have no idea if it supports webSockets.

I've been looking into something very similar to this myself, with the intent of generating (and destroying) Node.js cluster nodes on the fly.
Disclaimer: I'd still not recommend doing this with Node; nginx is more stable for the sort of design architecture that you're looking for, or even more so, HAProxy (very mature, and easily supports sticky-session proxying). As #tsturzl indicates, there is satellite, but given the low volume of downloads, I'd tread carefully (at least in a production environment).
That said, since you appear to have everything already set up with Node, rebuilding and re-architecting may be more work than it's worth. Therefore, to install the caronte branch with NPM:
Remove your previous http-node-proxy Master installation with npm uninstall node-proxy and/or sudo npm -d uninstall node-proxy
Download the caronte branch .zip and extract it.
Run npm -g install /path/to/node-http-proxy-caronte
In my case, the install linkage was broken, so I had to run sudo npm link http-proxy
I've got it up and running using their basic proxy example -- whether or not this resolves your dropped sessions issue or not, only you will know.

In Node.js, how do I make one server call a function on another server?

Let's say I have 2 web servers. Both of them just installed Node.js and is running a website (using Express). Pretty basic stuff.
How can Server-A tell Server-B to execute a function? (inside node.js)
Preferably...is there a npm module for this that makes it really easy for me?

How can Server-A tell Server-B to
execute a function?
You can use one of the RPC modules, for example dnode.

Check out Wildcard API, it's an RPC implementation for JavaScript.
It works between the browser and a Node.js server and also works between multiple Node.js processes:
// Node.js process 1
const express = require('express');
const wildcardMiddleware = require('#wildcard-api/server/express');
const {endpoints} = require('#wildcard-api/server');
endpoints.hello = async function() {
const msg = 'Hello from process 1';
return msg;
};
const app = express();
app.use(wildcardMiddleware());
app.listen(3000);
// Node.js process 2
const wildcard = require('#wildcard-api/client');
const {endpoints} = require('#wildcard-api/client');
wildcard.serverUrl = 'http://localhost:3000';
(async () => {
const msg = await endpoints.hello();
console.log(msg); // Prints "Hello from process 1"
})();
You can browse the code of the example here.

You most likely want something like a JSON-RPC module for Node. After some quick searching, here is a JSON-RPC middleware module for Connect that would be perfect to use with Express.
Also, this one looks promising too.

Update : The library I've made & linked below, isn't maintained currently. Please check out the other answers on this thread.
What you need is called RPC. It is possible to build your own, but depending on the features you need, it can be time consuming.
Given the amount of time I had to invest, I'd recommend finding a decent library that suits your purpose, instead of hand rolling. My usecase required additional complex features like selective RPC calls, for which I couldn't find anything lightweight enough, so had to roll my own.
Here it is https://github.com/DhavalW/octopus.