I have a file with the contents:
id,value,location
1234,pass,/temp/...
234,fail,/temp/r/...
2343,pass,/temp/status/...
The above file is update for about 1 hour continuously by some program. I need to send this file information to the browser and create a table and show all the data dynamically whenever user enters the link http://localhost:6666/getdata. How do I achieve this with:
1.cgi (python or perl)
or
2.nodejs
or
3.bottle framework.
as the backend.
There could be 10k entries after 1 hour.
Let us say the file was create at 12:00PM and a user requested http://localhost:6666/getdata at 12:10 PM. For next 50 minutes the data has to updated dynamically(continuously) which would feel like live data for the user.
To regularly send data from server to client, the usual design would be for the client to establish either a webSocket or socket.io connection to the server. That connection will then be long-lived and data can be sent either direction over the connection.
This allows the server to send data to the client whenever it wants to without waiting for the client to ask for data. The client then listens for that incoming data on the existing connection (with the appropriate event handlers) and processes the data when it arrives - doing whatever is appropriate for the data (like displaying it).
The socket.io library is a higher level abstraction built on top of webSocket and it offers a number of useful features beyond what webSocket offers (such as auto-reconnect, auto-detection of a dropped or non-function connection, a messaging layer, etc...) which are generally helpful (which is why that library is so popular for this use). There are socket.io libraries for both use in a browser and for many server platforms (including node.js).
Related
I need my website to get real time data from the server (it is for a project in html5, css3, javascript, php, mysql).
Initially I thought about websockets but maybe it is something "beast", since I don't need two-way communication, I just need to capture on the web (in real time without the user doing anything) the values of a mysql field and depending on the themselves do one thing or another in javascript.
My system could have about 1,000 users at a time.
What system do you recommend me? Would you know of any example?
So you basically have a website and you plan on receiving notifications from the server.That to me seems like a continous flow of data from the server to you the client at irregular intervals , and the reasonable way to do it (unless you plan on saving in an IP table your clients) is via websockets.This would be the push based approach.
Once you have established connection it remains open and you can get your data continously.
Another option like mentioned above would be to continously pull data from the server (query server for changes) and this could be done with HTTP.
So if you choose push based option you could you Websockets or you could look at Server Sent Events.
For pulling (request-response) you could use HTTP or something lighter like gRPC.
For more information check the options here
The question is not if the communication is one- or two-way but which communication partner initiates a speech act. If the real-time data of your application changes asynchronously and not in a certain rhythm then the server should send updates asynchronously to the webclient. And this is actually one of the standard usages of websockets which cannot be well implemented with HTTP request/response pairs (client would have to poll).
I need to refresh a part of my view without refreshing the whole page.
At my index.html page I have three panels, wich one shows the number of Tickets by it's status, I need to refresh this number every time a new ticket is created or updated. I used Java with Spring Boot and Thymelaf to build my application.
This is my view:
This is the way I'm doing it now:
model.addAttribute("resolvedTickets", atendimentoService.findAllTicketsByStatus(STATUS_RESOLVED).size());
I have tried to use web sockets but i can't figure out how to get this and refresh the panels.
In a standard web interaction, the client (i.e. your web browser) sends a request to your server. Your server receives the request, and sends back the information to show in your browser and then terminates the connection.
WebSockets are a way to create a persistent, two-way connection between the client and the server, but it requires cooperation from both. A lot of shared servers don't allow WebSockets, so you first have to make sure your server is capable of providing WebSockets. (I see from your screenshot that you're running on Heroku, which should have no problem running WebSockets.)
On the server side, you need to set up handling for incoming WebSocket requests. I don't know what language you've coded your server in, so I can't provide any guidance, but there are plenty of libraries that do the server-side part of WebSockets in most languages.
On the client side, you need to set up your WebSocket client. MDN has a great guide on WebSockets that explains what you'll need to do. Basically, all you'll have to do is listen for incoming messages and increment your counter.
var count = 0;
var exampleSocket = new WebSocket("ws://example.com/socket");
exampleSocket.onmessage = function(event) {
count++;
document.getElementById('myTicketCounter').innerHTML = count;
}
For some things, WebSockets are overkill. If you find that this is too much work for too little reward, you can also just set up an AJAX call to fire every few minutes that pings another page on your server and returns the number of tickets and updates accordingly. It won't be instantaneous, but if you don't need down-to-the-second resolution, it'll probably suffice. You can adjust the interval to be as long or as short as you want (to an extent; bombarding your server with constant requests will slow you down a bit).
I'm interested in creating what Disqus have done with their commenting system: http://highscalability.com/blog/2014/5/7/update-on-disqus-its-still-about-realtime-but-go-demolishes.html
The most impressive part of infrastructure is Nginx push stream module:
Still runs on 5 machines Nginx machines.
Uses NginxPushStream, which supprts EventSource, WebSocket, Long
Polling, and Forever Iframe.
All users are connected to these machines.
On a normal day each machine sees 3200 connections/s, 1 million
connections, 150K packets/s TX and 130K packets/s RX, 150 mbits/s TX
and 80 mbits/s RC, with <15ms delay end-to-end (which is faster than
Javascript can render a comment)
Had many issues with resource exhaustion at first. The configuration
for Nginx and the OS are given that help alleviate the problems,
tuning them to handle a scenario with many connections moving little
data.
Obviously this Nginx module doesn't support data storage. Only in-memory mechanism supported by push_stream_store_messages directive, but as author said:
The main target to stored messages is to deliver the message to a
subscriber that was offline when the message was published.
It is clear that Disqus don't publish messages to Nginx directly but rather though the Go backend that manages to store messages in Redis followed by publishing via internal POST to Nginx which keeps subscribers.
Does anyone have an experience with fetching messages history through Redis on a page load before even we use push stream module for newer messages? Do you have to push old messages history in one go or render as plain HTML followed by pubsub for new messages that appear after page is loaded?
The logic needs to be decoupled as much as possible. I don't intend to introduce blocking mechanism between user and Nginx for real time message communication. Would that be a good solution below?
Client pushes the message from the web page (via websockets)
Ajax request goes straight to push stream location and on Ajax complete callback it requests backend to store message in Redis (in the other direction it gets locked with backend)
Once user refreshes the page backend fetches Redis list and display the history
User can see the history and can post new messages
It requires only 2 backend requests to be developed: accept message and store in Redis, fetch the data and display on page load. Preferably light non-blocking backend required, like Lua module or even HTTP interface to Redis called Webdis.
I'd like to know smart people opinion on that mechanism from architecture point of view, no code example expected.
I have PHP on the server side, and HTML and javascript on the client side.
I am making an app where a stakeholder types a message that is broadcasted to multiple recievers of a group in real time.
I did some research on google and I understand I need to use WebSockets or Comet for real time push notifications. Is WebSocket or Comet mandatory for sending mass notifications to users?
Is my understanding correct? Any references to start with?
If the client is a browser, then the ONLY two ways a standard browser can connect to a server is via an Ajax (e.g. http) request or a webSocket connection. So, if you want a client to get notified of something from the outside world it has to use one of those two mechanisms.
HTTP requests are transitory. The client makes a request of a server, the server responds. HTTP requests are perfect for the client requesting information from the server. They are not very good at the server sending information to the client because normally the client is not connected. There are hacks and work-arounds where the client "polls" the server on some interval and maybe even the server uses longer running requests to try to simulate a "push" type system, but they are sub-optimal hacks at best.
webSockets are continuous connections. The client connects and the connection remains in place for as long as both sides want. This allows either side the ability to send a message to the other side whenever they want. That means the server can "push" data to the client whenever it wants. webSockets are efficient for push connections and are recommended (this is one of the main things they were designed for).
Comet is a library that was originally built for using HTTP to try to "hack" or "simulate" push before webSockets were invented and then before they were widely supported. I can think of no reason why one would want to use Comet instead of a webSocket unless you had such an old browser that webSocket was not supported.
So, if you are trying to do "realtime server push" to a browser, then you must have a continuously connected socket from the client which means webSocket (or something built on top of webSocket like socket.io).
For phone apps where you have access to the phone SDK, you can use the "push" system built into the OS to push some messages from server to client. This isn't quite the same as the two way webSocket channel, but since you asked about "push notifications", the OS push services available in both Android and IOS could also be an option for pushing notifications from server to client. Here's info on iOS notifications and Google Cloud Messaging
As of 2016, one can also use Server-sent events in all modern browsers except Microsoft browsers (not supported yet in Edge or IE) to push data from server to client. Here's a browser compatibility table. Server-sent events use a long lasting HTTP connection, a special MIME type and a supporting client in order to be able to send events from server to client at any time. Unlike webSockets, server-sent events are one way only (from server to client). A client would then use a traditional Ajax call in order to be able to send data to a server (whereas with a webSocket data can be sent either way over the same webSocket connection).
Here's a good description of how server-sent events work: How do server-sent events actually work?
Is your client application a SPA? (Single Page application)?
It's very important because if not, you have to consider that everytime a client change page, connection with websocket server will be lost.
In this case you have to manage a queue because if stakeholder send a multicast request when one client is disconnected, client won't receive nothing.
Polling won't solve this situation too and it's an orrible solution because mobile clients (for example) with typical internet plan, will consume megabytes for unuseful "ping" traffic.
A real example of polling is a child in a car asking his dad every minute if they are arrived to a destination!
So, Is there a solution without using spa?
Yes, using a "shared storage" between stakeholder and clients, and using websocket only for "wake up" online clients saying: Hey there is something new, go to check!
Everytime a client open a page it will receive from backend also not-read notifications, taken from the storage.
When a stakeholder want to notify something, it will just store the notification message in the shared storage and send a "pulse" to notification server.
Notification server will forward the "pulse" to online clients (just in case someone is stuck reading a page).
If a "pulse" is lost because a client is changing page there is no problem because the client will bring notifications from the storage.
Every page will contain this logic:
Retrive number or unread notifications (server side)
Connect to the notification server after 5 seconds (javascript side).
Hope it helps.
I would suggest that using webSockets is a more efficient way compared to other options, why is this? Well when a client receives a notification that there's a change in the server there is no need to create an AJAX call to the server to get that change, it can be sent to the client with the same webSocket connection more easily than AJAX. This means efficient code and a faster running App!
What would be best mechanism, for achieving ability, for users, that are logged in, receive messages, generated by server. As there is no way for a server, to send information to user, when it has new message to deliver, a user browser should poll with some specific interval, to receive in response new messages, additionally, there should be a way for server, to not send messages, that are already delivered to user. You could draw a connections with something like public chat mechanism, but the thing I need is message delay as close to realtime and ability to handle about 100 users simultaniously, making least traffic possible. Additional note: data is needed only when user is online, no need to store that data in server, for other users to read "history".
In my mind, there are one way of achieving this - global "message box" where server puts all messages, user browser is constantly polling the server, to check, if last received message ID is equal to last message ID in message box.
The question is, if this is right way to do that, or there are another ways for such tasks, as need for realtime data can be found everywhere: sensor data, multiplayer games, chat, stock market and more...
XEP-0124: Bidirectional-streams Over Synchronous HTTP (BOSH)
https://github.com/ssoper/jquery-bosh
Build a web-based notification tool with XMPP
Write real-time web applications with XMPP, PHP, and JavaScript
Hope this helps.
Isn't pushing a better strategy? Keep a tcp connection open between server and browser and stream changes to the browser when new information is available.
Take a look at html 5 websockets. (which does exactly this)
heres a demo
Have you looked at Comet?
Comet is a web application model in
which a long-held HTTP request allows
a web server to push data to a
browser, without the browser
explicitly requesting it.
If you search stackoverflow there is plenty of info about its use.