I want to have the following architecture:
A JSON REST API where real time statistic data is pushed to and stored in a Redis server.
A JSON REST API call where any number of clients (native or web) can receive this data after it has been stored - i.e. in real time.
The first client will just be a web app and I may build a native app later.
I'm wondering if my only option is for the clients to poll the REST API for changes? Ideally, I'd like the server to push updates as they arrive so I don't need to manage this polling.
Is my architecture suitable for what I want to achieve, or am I missing something?
A more efficient way than polling is to use websockets, such as Faye or Socket.IO. You can place an emit event under a data store event to immediately send data that's been stored.
With Socket.IO, you'd do that like this:
var io = require('socket.io').listen(80);
//note that you can listen on HTTP servers
//can also be used with Express applications, etc
//when data is stored, run this
io.sockets.emit('event', {
object: 'that is sent to client'
});
You could then use this to tell the client that there is new data, or you could directly send the newly stored data. Custom events can be defined, such as
io.sockets.emit('data_receive', function (data) {...});
and would be received client side like so:
var socket = io.connect('http://socket.location');
socket.on('data_recieve, function (data) {
//data is whatever sent from server
});
In Faye you'd do something like this:
var http = require('http');
var faye = require('faye');
var bayeux = new faye.NodeAdapter({
mount: '/faye',
timeout: 45
});
bayeux.listen(8000);
Then when data is stored, you'd run:
client.publish('/path', {
data: 'Hello world'
});
Any client that has created a client like so:
var client = new Faye.Client('http://socket:port/path');
client.subscribe('/path', function(data) {
alert('Received data: ' + data.text);
});
Will receive the data.
You have the option of Node.js and the websocket for push and pull in realtime.
To don't manage the queuing you still have the option of MQ.
Related
My project works as intended except that I have to refresh the browser every time my keyword list sends something to it to display. I assume it's my inexperience with Expressjs and not creating the route correctly within my websocket? Any help would be appreciated.
Browser
let socket = new WebSocket("ws://localhost:3000");
socket.addEventListener('open', function (event) {
console.log('Connected to WS server')
socket.send('Hello Server!');
});
socket.addEventListener('message', function (e) {
const keywordsList = JSON.parse(e.data);
console.log("Received: '" + e.data + "'");
document.getElementById("keywordsList").innerHTML = e.data;
});
socket.onclose = function(code, reason) {
console.log(code, reason, 'disconnected');
}
socket.onerror = error => {
console.error('failed to connect', error);
};
Server
const ws = require('ws');
const express = require('express');
const keywordsList = require('./app');
const app = express();
const port = 3000;
const wsServer = new ws.Server({ noServer: true });
wsServer.on('connection', function connection(socket) {
socket.send(JSON.stringify(keywordsList));
socket.on('message', message => console.log(message));
});
// `server` is a vanilla Node.js HTTP server, so use
// the same ws upgrade process described here:
// https://www.npmjs.com/package/ws#multiple-servers-sharing-a-single-https-server
const server = app.listen(3000);
server.on('upgrade', (request, socket, head) => {
wsServer.handleUpgrade(request, socket, head, socket => {
wsServer.emit('connection', socket, request);
});
});
In answer to "How to Send and/or Stream array data that is being continually updated to a client" as arrived at in comment.
A possible solution using WebSockets may be to
Create an interface on the server for array updates (if you haven't already) that isolates the array object from arbitrary outside modification and supports a callback when updates are made.
Determine the latency allowed for multiple updates to occur without being pushed. The latency should allow reasonable time for previous network traffic to complete without overloading bandwidth unnecessarily.
When an array update occurs, start a timer if not already running for the latency period .
On timer expiry JSON.stringify the array (to take a snapshot), clear the timer running status, and message the client with the JSON text.
A slightly more complicated method to avoid delaying all push operations would be to immediately push single updates unless they occur within a guard period after the most recent push operation. A timer could then push modifications made during the guard period at the end of the guard period.
Broadcasting
The WebSockets API does not directly support broadcasting the same data to multiple clients. Refer to Server Broadcast in ws documentation for an example of sending data to all connected clients using a forEach loop.
Client side listener
In the client-side message listener
document.getElementById("keywordsList").innerHTML = e.data;
would be better as
document.getElementById("keywordsList").textContent = keywordList;
to both present keywords after decoding from JSON and prevent them ever being treated as HTML.
So I finally figured out what I wanted to accomplish. It sounds straight forward after I learned enough and thought about how to structure the back end of my project.
If you have two websockets running and one needs information from the other, you cannot run them side by side. You need to have one encapsulate the other and then call the websocket INSIDE of the other websocket. This can easily cause problems down the road for other projects since now you have one websocket that won't fire until the other is run but for my project it makes perfect sense since it is locally run and needs all the parts working 100 percent in order to be effective. It took me a long time to understand how to structure the code as such.
I am building a simple IoT project in which a micro-controller publishes sensor data to a MQTT broker every second on a particular topic. I am using Node.js to build an app that uses the mqtt library to subscribe to this topic and update the sensor data to the Thingspeak channel using their API via a POST request (using the request library).
I have managed to build something that is almost working but I am facing a slight problem. I am not able to update all the sensor data since the POST request takes some time and during that time other sensor data is also published.
I am thinking of a solution in which I can queue the sensor data that needs to be updated via POST request.
Here is my current code:
var mqtt = require("mqtt")
var request = require("request")
var client = mqtt.connect('hostname')
client.on('connect', function () {
client.subscribe('topic')
})
var postReq = function(data) {
request.post('https://api.thingspeak.com/update', {form: {sensor_data: data}}, function(err,response,body){
console.log(body)
}}
}
client.on('message', function (topic, data) {
// message is Buffer
postReq(data)
})
Im trying to set up a node.js server to send messages to the client, which will then display the messages using a jquery notification library, I'm using this notifcation library if anyone's interested: http://needim.github.com/noty/
At the minute I have a postgres database set up with a table which has a a trigger on it to write to a listener.
The trigger is as follows:
CREATE OR REPLACE FUNCTION new_noti() RETURNS trigger AS $$
DECLARE
BEGIN
PERFORM pg_notify('watchers', TG_TABLE_NAME || ',msg,' || NEW.msg );
RETURN new;
END;
$$ LANGUAGE plpgsql;
Then I have a node.js server as follows:
var pg = require ('pg');
var pgConString = "pg://aydin:password#localhost/test"
var app = require('http').createServer(handler)
, io = require('socket.io').listen(app)
, url = require('url')
app.listen(8080);
function handler (request, respsonse) {
var client = new pg.Client(pgConString);
client.connect();
client.query('LISTEN "watchers"');
client.on('notification', function(msg) {
console.log(msg.payload);
sendMessage(msg.payload);
});
}
function sendMessage(message) {
io.sockets.emit('notification', {'message': message});
}
Then I have some client code as follows:
<script type="text/javascript">
var socket = io.connect('http://localhost:8080');
socket.on('notification', function (data) {
console.log(data.message);
newNoty(data);
});
function newNoty(data) {
noty({
"text":data.message,
buttons: [{
type: 'button green',
text: 'Go to'
}],
"theme":"noty_theme_twitter",
"layout":"bottomRight",
"type":"information",
"animateOpen":{
"height":"toggle"
},
"animateClose":{
"height":"toggle"
},
"speed":500,
"timeout":7500,
"closeButton":true,
"closeOnSelfClick":true,
"closeOnSelfOver":false,
"modal":false,
});
}
</script>
This doesn't work, it seems the node.js never receives the postgres notifications, I think this is because I am using the function handler and I'm not actually firing any requests to it from the client code. I'm not sure how to do this and whether it is the correct way?
Is there a function on which can fire on connections and not requests?
And am I even doing it the right way round? should there be a server on the client side which node.js sends messages to? How does it know when a client is available? Any help or pointers to tutorials would be much appreciated. Thankyou.
You're not actually setting up your database connection until the client sends an HTTP request. It looks like that may never happen due to same-origin issues (your client code appears to be coming from somewhere other than the server you've shown).
In any case, you probably want to set up the connection in response to a "connection" event from io.sockets (i.e. move the stuff that's currently in the HTTP request handler there). That's how it "knows when a client is available". Or maybe you should be doing it as part of initialization. Your client-side code seems OK, but it's out of context so it's hard to tell whether it really fits your needs.
I'm running Socket.io multi-threaded with the native cluster functionality provided by Node.js v0.6.0 and later (with RedisStore).
For every new change in state, the server iterates over each connection and sends a message if appropriate. Note: this isn't "broadcasting" to all connections, it's comparing server data with data the client sent on connection to decide whether to send the server data to that particular client. Consider this code sample:
io.sockets.clients().forEach(function (socket) {
socket.get('subscription', function (err, message) {
if(message.someProperty === someServerData) {
socket.emit('position', someServerData);
}
});
This worked fine when there was only one process, but now, the client receives a message for each Node process (ie. if there are 8 Node process running, all clients receive the messages 8 times).
I understand why the issue arises, but I'm not sure of a fix. How can I assign a 1-to-1 relation from one process to only on client. Perhaps something using NODE_WORKER_ID of Cluster?
This previous SO question seems somewhat related, although I'm not sure it's helpful.
This seems like a pretty common request. Surely, I must be missing something?
So if I get this straight you need to emit custom events from the server. You can do that by creating your own custom EventEmitter and triggering events on that emitter, for example:
var io = require('socket.io').listen(80);
events = require('events'),
customEventEmitter = new events.EventEmitter();
io.sockets.on('connection', function (socket) {
// here you handle what happens on the 'positionUpdate' event
// which will be triggered by the server later on
eventEmitter.on('positionUpdate', function (data) {
// here you have a function that checks if a condition between
// the socket connected and your data set as a param is met
if (condition(data,socket)) {
// send a message to each connected socket
// if the condition is met
socket.emit('the new position is...');
}
});
});
// sometime in the future the server will emit one or more positionUpdate events
customEventEmitter.emit('positionUpdate', data);
Another solution would be to have those users join the 'AWE150', so only they will receive updates for 'AWE150', like so:
var io = require('socket.io').listen(80);
io.sockets.on('connection', function (socket) {
if (client_is_interested_in_AWE) { socket.join('AWE150'); }
io.sockets.in('AWE150').emit('new position here');
});
Resources:
http://spiritconsulting.com.ar/fedex/2010/11/events-with-jquery-nodejs-and-socket-io/
I am using node.js building a TCP server, just like the example in the doc. The server establishes persistent connections and handle client requests. But I also need to send data to any specified connection, which means this action is not client driven. How to do that?
Your server could maintain a data structure of active connections by adding on the server "connection" event and removing on the stream "close" event. Then you can pick the desired connection from that data structure and write data to it whenever you want.
Here is a simple example of a time server that sends the current time to all connected clients every second:
var net = require('net')
, clients = {}; // Contains all active clients at any time.
net.createServer().on('connection', function(sock) {
clients[sock.fd] = sock; // Add the client, keyed by fd.
sock.on('close', function() {
delete clients[sock.fd]; // Remove the client.
});
}).listen(5555, 'localhost');
setInterval(function() { // Write the time to all clients every second.
var i, sock;
for (i in clients) {
sock = clients[i];
if (sock.writable) { // In case it closed while we are iterating.
sock.write(new Date().toString() + "\n");
}
}
}, 1000);