Develop Reference

Real time client information synchronization, best practices and advice

I am writing a PHP backend, JS/Jquery front end application that will allow users to "communicate" in near real time. That is the hope at least, my question is what is my best course of action? Am I best to use WebSockets to send data between the server and client or would use AJAX and some sort of timer (x amount of seconds) be better. My worry with the AJAX way is that it may be taxing on the server to have 10+ clients all asking for data every 15-30 seconds. I need it to be near real-time and so 5 minutes is not really realistic.
An example of what I am trying to do would be if I had 5 users all on a page and user 1 updates their status, I would want users 2,3,4, and 5 to see the update as fast as possible without having to refresh.
I am mixed on what I think is best and I don't want to start doing it one way and find out it is insecure or terrible after getting halfway done. What is my best route with an application like this?

Here's a list of popular possible solutions:
Short polling (what you're referring to in AJAX
Long Polling (AJAX too, but not too many requests)
Websockets
WebRTC
So, for short polling as you've said it consumes lots of resources so let's remove that from the list.
as for long polling, its idea is that a request is send to server and the server doesn't respond unless a new event has happened (keeps the request) but its rarely used in modern development. so If you're going to work with other developers its kind of bad decision.
for WebRTC, browser compatibility is not that great and still a draft in W3C.
Thus, you're left out with WebSockets, and yes they consume ram but not CPU. ram is much cheaper (and it doesn't consume that much too).
As for security they can be considered equal (except for WebRTC which is better because it is actually P2P Communication)
Side note: don't overthink it :)
Here's some resources that can help you:
https://webrtc.org/
https://github.com/walkor/phpsocket.io //Socket library for PHP similar to Socket.io
https://socket.io
What are the realtime communication protocols available for the web? List of the protocols
https://codeburst.io/polling-vs-sse-vs-websocket-how-to-choose-the-right-one-1859e4e13bd9 great article for polling, websockets & covers SSE too

there is one way to make the RTC, RealTime App, Just Use the Socket.io "WebSocket " for signaling and before that take a full view of these webPages:
https://bloggeek.me/
https://www.html5rocks.com/en/tutorials/webrtc
https://w3c.github.io/webrtc-pc/#rtcsignalingstate-enum
https://www.w3.org/TR/mediacapture-streams/#legacy-interface-extensions
and i start to development this technology with this book enter link description here it will open your view of RTC usage and All Details.

setInterval($.get or socket.io? [duplicate]

I am developing a web app and I was wondering which method should be suitable for my project.
Basically what I want to display to the users are some notifications which are taken from requests to other servers. My node.js app gets all the info and then it spread out to the users, saving a copy into my MongoDB.
The idea is quite simple but reading about methods I found these two techniques:
Ajax : Client side would be checking all time if there is new content on the server. This would be done by using a jquery ajax get to my server API (every 30/60 seconds).
Socket.io : The client connects once, and then a permanent TCP connection is maintained (more realtime).
Now I have explained the situation, I have the following questions :
Would I not have too many requests with ajax ? imagine I want a check every minute to the server, if we scale the app to 100 users, it will give me 100 queries per minute. Would it be "cheaper" in system resources to have a socket ?
Would the socket.io be a problem for mobile devices ? bandwith and performance. The response of the server is always info in JSON format.
I read that now.js could be used for this but it seems the project is no longer supported, so not sure if using it would be a good idea.
How is the caching on both methods ? I was considering to create a cache file for each user and this would be updated by the node.js in the server side. I guess this could work really well with ajax but what about socket.io ?
Is it true that socket.io is not compatible at all with many browsers ? My app would be more focused to mobile devices and I think this could make me think about choosing ajax instead.
Any alternative suggested ?
I hope this could clear my mind and others who are in the same situation :)
Thanks

Many of the generic tradeoffs between webSocket and Ajax are discussed here:
websocket vs rest API for real time data?
Some tradeoff issues for mobile devices are discussed here:
Cordova: Sockets, PushNotifications, or repeatedly polling server?
In a nutshell, if your data is primarily server-driven and then needs to be sent out to clients and you desire fairly good latency for when the clients see the new data, then that is the exact problem that webSockets are good for. webSockets work best in this situation because the client does not need to poll frequently, the server does not need to handle regular polling requests from lots of clients. Instead, each client just sets up the one persistent webSocket communication channel which the server can then send data down upon demand at any time.
Would I not have too many requests with ajax ? imagine I want a check
every minute to the server, if we scale the app to 100 users, it will
give me 100 queries per minute. Would it be "cheaper" in system
resources to have a socket ?
Sockets require very little resources when they are not active so yes, a peristent webSocket is more efficient than lots of clients polling endlessly. This is why webSockets were invented because they are better at solving this particular problem.
Would the socket.io be a problem for mobile devices ? bandwith and
performance. The response of the server is always info in JSON format.
socket.io is not a problem for bandwidth or performance. There are some mobile issues with trying to use webSockets in the background because mobile devices are also trying to do active power management, though a similar issue exists with client polling too.
How is the caching on both methods ? I was considering to create a
cache file for each user and this would be updated by the node.js in
the server side. I guess this could work really well with ajax but
what about socket.io ?
It is unclear what you are asking about caching? In a webSocket implementation, the server gets the data and then just sends it to each user. No server-side caching would generally be needed. In a client Ajax polling implementation, the server would have to store the data somewhere and "wait" for each client to then request the data. There is no "built-in" caching mechanism for either webSocket or Ajax.
Is it true that socket.io is not compatible at all with many browsers
? My app would be more focused to mobile devices and I think this
could make me think about choosing ajax instead.
socket.io is fully compatible with all browsers that have webSockets which is pretty much everything in use today except for IE9 and older. If you use the socket.io library, it will automatically fall back to long polling if webSockets don't exist. Your mobile issues are likely going to be similar whether you're doing regular polling or webSocket because the mobile device wants to power manage long running things, but you don't want to stop polling. I don't think this is a reason to avoid webSockets/socket.io. socket.io has some really nice auto-reconnect logic whenever it loses the connection which can be really useful.
In the mobile world, I think you're just going to find that you can't reliably do real-time notifications in the background without using some sort of native app component that can plug into the native "push" system on the device because that's the only system that is both battery efficient and fully compatible with power management. A web page is going to be power managed as soon as it is not the foreground task or when the device has been idle.

Advice on which technology to use for real time notifications

I have X amount of activity sensors connected to a server that inserts data to a database everytime a sensor is triggered. What I'm trying to do is create a web interface with a blue print of the facility (svg) and whenever a sensor is triggered, besides the db insert, I want it to show some sort of alert in my blue print. For that I need to keep an open connection to the server I think.
I was thinking of using web sockets, but it might be overkill since I only need to retrieve data from the server. But running an ajax call every second doesn't sound very efficient either. Are there any other alternatives?
Thank you

Some potential choices include:
WebSocket
Adobe® Flash® Socket
AJAX long polling
AJAX multipart streaming
Forever Iframe
JSONP Polling
Which actual transport you end up using will depend on the your requirements for browser support and what technology you are using on the server to handle these requests. The transport choice may also depend on your network topology - what types of load balancers you need to integrate with, proxies, etc.
There are many libraries available on both the client and server sides, many of which support more than one of these transports.
For example (not an exhaustive list):
socket.io for nodejs
WebSocket
Adobe® Flash® Socket
AJAX long polling
AJAX multipart streaming
Forever Iframe
JSONP Polling
SignalR for an asp/.net backend
WebSockets
Server-Sent Events
ForeverFrame
Long Polling
Atmosphere for a java backend
WebSockets
Server Side Events (SSE)
Long-Polling
Forever frame
JSONP
IMO - Websockets is NOT overkill for this type of problem and would lend itself nicely to this type of application.

Without specifically discussing frameworks or knowing what is running in the backend of your server(s), we have a few options to consider for the frontend:
Websockets
Websockets are designed for bidirectional communication, although it is kind of shocking how many users are surfing the web in a browser that doesn't support websockets. I always recommend a fallback for this, such as the other methods listed below.
SSE
SSE is an HTML5 spec and is still shaky at best. Try scrolling on a page while when an SSE event fires... It may be a little easier on the backend, put it sometimes hangs on the client side since it runs inside the same thread that the DOM is running in.
Long Polling
Keeps your connection open. It doesn't scale well with PHP, but performs swimmingly with Python+Twisted on the backend, or Node.Js
Good Old Ajax
Keep your requests small, and you still have a scalable solution. Yes, a full GET request is the most expensive, but is supported in just about every browser rolled out the past ten years. It is also worth noting that GET requests are easy to scale horizontally with more hardware.
In a perfect world:
You would break up your application into a few components, operating behind a reverse proxy such as Nginx. Then use Node.Js + Socket.IO handle the realtime aspects of your app.
Another option would be to use small Ajax requests, and offer websocket support for the browsers that support it. This is advice specifically for PHP in the backend.

WebSocket is certainly not overkill. On the contrary. With websockets, you have a bi-directional communication channel; this means, that the server can initiate communication whenever it seems fit (e.g. when sensor data changes).
In a previous project, I have used node.js together with socket.io, to monitor 50+ sensors. Data was updated in real-time in a browser. The data was visualized using smoothie.js.
Whenever a sensor value was updated, it was communicated to the browser. Some sensors only updated once a minute, others once a second, ...
Polling would have been overkill, because it would retrieve all data for all sensors, even from those that were not updated yet.

I had a similar problem and did a lot of research on this. As I understand it, there are three main options:
Short polling: Have an endpoint that your javascript client pings every second. This is the worst option, because the pings add latency up to one second to your communication, and depending on how you implement, the endpoint could query the database every second, adding unnecessary overhead.
Long polling: Have an endpoint that your javascript client pings that holds the connection until a) the event occurs or b) the connection times out. If the endpoint returns a response, the client gets the event information. If the endpoint does not return a response, no event has occurred, and the client sends a new request. This is a good option because the events can immediately trigger the response to the client, assuming you have an asynchronous interprocess communication layer (like 0MQ) to send the message without any sort of polling.
Websocket: Have your javascript client connect to a websocket server, which will send a message to your client immediately upon the event trigger.
I think a websocket is your best option, because it accommodates immediate communication of the event without all the request/response overhead. And most importantly, this is exactly what websockets are designed to do! As such, you will probably have to write the least amount of custom code with this solution.

There are two great commercial services that might work for you.
Firebase - a javascript hierarchical database and realtime
messaging/ synchronization platform, uses websockets and has other fallbacks
PubNub - a real time message passing and queue system, uses websockets

If I wanted to create an AJAX chat what communication technique should be used to maintain scalability?

I put together an AJAX chat a while back ASP.NET MVC and jQuery. The javascript would hit the server about every 7 seconds to check for new messages. Obviously this was horrible on performance as the chat grew and included more and more users. The site traffic grew exponentially with so many requests going on. A user could leave the computer on all day and not even be there and they would still be making hits every 7 seconds.
Is there a better way to do this? I have heard of something called "push" but I haven't really been able to wrap my head around it. I think I just need pointed in the right direction.
1.) What is the best way to develop an AJAX chat and have it be scalable?
2.) What is push and how would I just that with jQuery?

1.) What is the best way to develop an AJAX chat and have it be scalable?
I agree with #freakish about the complexity and potential lack of scaling of IIS.
However, there is a relatively new Microsoft option in the works called SignalR which could become a core part of ASP.NET. More details in this related SO Question:
AJAX Comet - Is there any solution Microsoft is working on or supports to allow it to be scalable?
2.) What is push and how would I just that with jQuery?
Partially answered elsewhere, but it's a long-held persistent connection between the server and the client which means the server can instantly 'push' data to the client when it has new data available.
jQuery does support making AJAX requests but the core library doesn't support expose ways of doing HTTP Long-Polling or HTTP Streaming. More information in this SO answer to 'Long Polling/HTTP Streaming General Questions'.

Server push is a technology that allows the server to push data back to the client without forcing client to make many requests (like every 7 seconds). It is not really a matter of javascript but rather good server scripting. The upcoming HTML5 will make it simple due to server-sent events and/or WebSockets. This will be a true TCP connection between different machines.
But if you intend to make a webpage compatible with older browsers, then the most common technique is the long polling. The client sends request to the server and the server does not respond to it until it has new data. If it does then the response is made and the client immediatly after receiving data calls the server with new request. In practice however this requires the server to be well-written (for example it has to maintain thousands of idle requests at the same time) and can become a rather big challenge for developers.
I hope this helps. :) Good luck!

The technique you should use is the real-time persistent long running connections over a web page using WebSockets. You can use this library.

Node.js is becoming quite popular to build something like this and supports socket connections so you could push the data out only when there is a new message. But that would be learning something completely new.
Another nice potential would be to use MVC's OutputCacheAttribute and use the SQL dependency option so your AJAX page could be cached and would only be a new request when a new chat message appears. Also you would want your controller to be an Asynchronous controller to help reduce the load on IIS.
Enjoy, optimization is always fun and very time consuming!

AJAX and Client-Server Architecture with JavaScript

I have to program websites, but I rather don't like the static HTML nature. I prefer more of a client-server architecture.
Now I've figured, that with XMLhttp, you can basically dynamically update your page and send/request for information/action to/from a server. So this would basically cover the client area.
But to complete a client-server architecture, it is necessary for the server to send/request information, too, without being queried.
Is there any way, for example for a chat server, to send back a received message to all clients (the clients use a web browser) without that the clients have to query in a fixed interval? I want to implement that one can see while you type something in.

There are several different ways to accomplish this. Some of them are already answered here, but I wanted to include a few more as well as my thoughts on them.
1. Polling
Frequent requests are made to the server to check for new info. This is the worst way to do this, but probably the easiest. If your site will have a low number of users, it might be worth doing it this way.
This can be accomplished by using setInterval(myFunction, n) in javascript to send XMLHttpRequests to the server every n milliseconds. Then, on the server, you respond to this with your new info, when you have it, or some message that implies no new info.
2. Long Polling
When the page is loaded, it makes a request to the server for new info. The server holds the connection open until there is something to send back. This method reduces the amount of network traffic used, but increases the resources used on the server. You can use this for a small number of users, but it doesn't scale very well.
The easiest way to do this is to have the page that handles the AJAX request simply wait for new information to be available, then respond. This can tie up a lot connections on your server. So, use with care.
3. COMET
COMET is basically long polling, but the server is setup properly for it. It knows that these connections aren't "real" and it uses less resources to handle them. This is a great solution for this problem, but it requires that the server is explicitly setup for this purpose. There are COMET servers and COMET addins for other popular servers, but it will require some setup and sometimes some money.
Implementing this on .NET isn't the easiest thing in the world. You can pay for solutions, try to find someone else's code that does something similar, or try to write it yourself. I've not found any decent free solutions for this. If someone else has, please comment.
4. RIA
Another solution would be to include Flash, Silverlight, or Java Applet on your page. People often do this by using a 1x1 object so that they can use Flash or Silverlight to talk to the server. If you don't mind adding the dependency, this is a decent solution. If you already know Silverlight or Flash, it could be relatively simple to implement.
You can find tutorials on the internet for each of these options.
5. Web Sockets
If you are on the cutting edge, you can look into Web Sockets. It's only available in the latest builds of modern browsers. It was part of HTML5, but it might be its own spec now. Regardless, it means that older browsers won't be able to handle it. But, if you don't mind limiting yourself to the latest of browsers, you can use this amazing feature.
I believe that Chromium is the only browser that currently supports it. However, there is work being done to implement this in Firefox and WebKit.
I'll spare you the controversy and simply say that this does exactly what you want it to. The Abstract of the spec says it all.
This specification defines an API that enables Web pages to use the Web Sockets protocol for two-way communication with a remote host.
Special Mention
If you are interested in the world of Node JS, you can't go wrong with Socket IO. It will implement the best of whichever technology is available to the browser.
Conclusion
The best option is Socket.IO on Node JS. However, for an ASP.Net solution, go for COMET or Web Sockets, if you can. Otherwise, using Flash/Silverlight isn't terrible. Finally, polling and long polling should be last resorts. You could always support one of these, then fall back to another if there isn't support for it in the client's browser.

Yes, you can use COMET.

The client has to tell the server when the client-user begins typing. You've got a couple options here.
Frequent requests from the server for the latest activity. This would be taking place for each user involved in the chat. The same request could be used to send user-specific activity to the server as well: "Jonathan is typing..."
Long-polling. This essentially requests information from the server, and the server keeps the connection opened until it has something to send back. So your requests are minimized, but your connections stay opened longer.
Depending on your traffic, type of data being transmitted, server-environment, and many other factors, one of these options may shine more than the other.

You can use Silverlight for push notifications. Look at PollingDuplexHttpBinding. Since you are using ASP.Net MVC, adding Silverlight will be easy.
Look at this page for more information.

Based upon the REST architecture the html system is based upon, the servers role is to simply act as a resource for the client to pull from. I am generalizing but there are tools to implement this type of action on the client side, rather than on the server side.
You are better off writing/using a library that can request updates from the server periodically. You can encapsulate these types of requests in a javascript object that can fire events. This way your client side script can act like it's getting notified from the server. Review some common stuff with COMET you can probably find some tools to help you client side code.
HTML 5 has some tentative attempts at this type of functionality, but if you want your app to work on older browsers, your better off using more stable methods, like AJAX requested updates.