Develop Reference

Inconsistent Date.now() values

I built an HTML5 multiplayer game that depends on having a reasonably accurate time sync between server and client. For the most part, the algorithm I use is very accurate -- all it does is estimate what the client-server time delta is, i.e. the difference between the current time on the server and the current time on client. For example, if the server time is exactly 5 seconds ahead of the client time, the time delta is 5000 ms.
The client and server (node.js) are both written in Javascript. The algorithm works as follows:
Record time on the client:
var clientTime = Date.now();
Ping the server. When the server receives the message, it immediately sends a response containing just one thing: the time on the server when the message was received.
var serverTime = Date.now();
// Send serverTime to the client
When the client receives the server response, immediately record the time:
var clientTime2 = Date.now();
Now, we know that when the server received the message, the client time must have been somewhere between clientTime and clientTime2.
If the server received the message when client time was clientTime (i.e. client->server request took 0ms somehow), then the time delta is
var delta1 = (serverTime - clientTime);
If the server received the message when client time was clientTime (i.e. server->client response took 0ms somehow), then the time delta is
var delta2 = (serverTime - clientTime2).
Thus we can safely say that the time delta is somewhere between delta1 and delta2. Now, repeat this process a bunch of times, each time narrowing the range based on whatever results you got, and you can get a pretty good estimate of the time delta.
I've tested this hundreds of times on 7 different browsers and multiple machines and have never had any issue with it. It's never been inconsistent.
The issue, though, is that my server logs show that, every now and then, a few people will get wildly inconsistent time sync results. Here is an actual example of one player's time sync:
The client went through 74 cycles of the above algorithm and successfully narrow the range of possible time deltas to: [-186460, -186431] without a single inconsistency. 29ms accuracy.
On the 75th cycle, possibly a few seconds after the 74th cycle, the client calculated the range of possible time deltas to be: [-601, -596]. 5ms accuracy, except for it's extremely inconsistent with the past 74 cycles: it's 3 minutes off!
I would blame this on crazy edge cases, except it happens almost 100 times a day... how could this happen? Is there any possible error when using Date.now()?

performance.now() instead of Date.now(), because performance.now() is monotonically increasing and not subject to clock drift. See the comments, thanks to everyone for their help!

Your difficulty is that you depend on estimating round-trip times to the server, over an Internet that has variance in round-trip times. Sometimes that variance will be unexpectedly substantial, as in cases where temporary congestion and large router buffers delay a single message far longer than normal. (Cf "bufferbloat".)
Your second difficulty is that you are using self-reported client times, and that means that if a client has a clock that's weird it will look to you like a round-trip estimation gone wrong. As another poster noted, internet time protocols will sometimes slew a clock rapidly to correct from local timekeeping anomalies.
What it sounds like is that you need some filtering in your code that takes into account previous results so that when you get an anomalous result you don't immediately accept it as true.

Delay an HTML5 notification?

I've been looking at the state of HTML notifications and service workers, and was wondering - is it possible to show a notification on a delay? Basically, I would like to be able to say "remind me in 30 minutes" (or whatever), then push a notification to the user 30 minutes later. That could be scheduled immediately, but I'm not seeing any functionality that allows it.
Am I missing something or is it impossible in the current state of (particularly) Chrome APIs?

This is possible but not straightforward with service workers, at least in their present form. It's not straightforward because a service worker can't keep itself awake for half an hour or wake itself up with a setTimeout or setInterval. The browser will just shut the worker down and will keep no record of any timeouts or intervals. You could wake it up with a message from an open tab, but you said that you don't want to have to have to keep an open tab, and if you assume an open tab then why even bother with the service worker anyway? As Jeff Posnick suggested in a comment, you could perhaps eventually use the Sync or PeriodicSync APIs, but, as he also points out, they aren't implemented yet and aren't really intended for this anyway.
You can accomplish what you want in current versions of Chrome using the Push API, but you'll have to include the server in the loop and set yourself up with a push notification service (i.e. GCM). Here's how it would work, roughly:
When you decide to delay a notification, let the server know about it
After a delay, the server sends out a push message for your user
The service worker is woken up in response to the push and creates a new notification
This last part will be a hassle, because currently you can't actually send any kind of payload with a push, so your service worker will need some way of figuring out what the notification is supposed to be. Maybe the server has a list of snoozed notifications and the service worker can get it from there, or maybe you saved them in IndexedDB.

Adapted from https://developer.cdn.mozilla.net/media/uploads/demos/e/l/elfoxero/c17223c414d8ddafb7808972b5617d9e/html5-notifications_1400214081_demo_package/:
<script>
var Notification = window.Notification || window.mozNotification || window.webkitNotification;
function show() {
window.setTimeout(function () {
var instance = new Notification("Hello World!");
}, 5000);
return false;
}
</script>
Notify me!

Ajax polling chat gets duplicates at message receiver front-end on speedy chat

I have developed a javascript chat (php on the backend) using:
1) long-polling to get new messages for the receiver
2) sessionStorage to store the counter of messages
3) setInterval to read new messages and if sessionStorageCounter < setIntervalCounter then the last message is shown to receiver.
4) javascript to create,update and write the chat dialogues
The module is working fine, but when users have a speedy chat the receiver' front end gets two or three same messages, (neither the counter fails, nor the query provides double inserts).
The code seems to be correct (that's why I don't provide the code), so the interval delay might be the reason (on reducing interval delay, nothing changes).
Do you think that the above schema is a bad practice and which schema do you think would eliminate the errors?

My approach, if solving it myself (as opposed to using an existing library that already handles this) would be:
Have the server assign a unique ID (GUID) to each message as it arrives.
On the clients, store the ID of the most recently received message.
When polling for new messages, do so with the ID of the last message successfully received. Server then responds by finding that message in its own queue and replaying all of the subsequent messages.
To guard against 'dropped' messages, each message can also carry the ID of the immediately-previous message (allowing the client to do consistency-checking)
If repolling does cause duplicates to be delivered from server to client, the presence of unique IDs on each message makes eliminating them trivial. Think of the server-side message queue as an event stream, with each client tracking their last-read position. The client makes no guesses about the appropriate order of messages, how many there are, etc - because its state consists entirely of 'what have I seen', there are few opportunities to get out of sync.

Since it's real time chat, the setInterval interval is probably small enough to ask the server for new messages two or three times simultaneously. Make sure that the server handler is synchronized and it is ignoring duplicated queries from the same user.

Proper way to monitor/control a server remotely over http in realtime

On my client (a phone with a browser) I want to see the stats of the server CPU,RAM & HDD and gather info from various logs.
I'm using ajax polling.
On the client every 5 sec (setInterval) I call a PHP file:
scan a folder containing N logs
read the last line of each log
convert that to JSON
Problems:
Open new connection every 5 sec.
Multiple AJAX calls.
Request headers (they are also data and so consume bandwidth)
Response headers (^)
Use PHP to read files every 5 sec. even if nothing changed.
The final JSON data is less than 5 KB, but I send it every 5 sec, and there are the headers and new connection every time, so basically every 5 sec., I have to send 5-10 KB to get 5 KB which are 10-20 KB.
Those are 60 sec / 5 sec = 12 new connections per minute and about 15 MB per hour of traffic if I leave the app open.
Lets say I have 100 users that I let monitor / control my server that would be around 1.5 GB outgoing traffic in one hour.
Not to mention that the PHP server is reading multiple files 100 times every 5 sec.
I need something that on the server reads the last lines of those logs every 5 sec and maybe writes them to a file, then I want to push this data to the client only if it's changed.
SSE (server sent events) with PHP
header('Content-Type: text/event-stream');
header('Cache-Control: no-cache');
while(true){
echo "id: ".time()."\ndata: ".ReadTheLogs()."\n\n";
ob_flush();
flush();
sleep(1);
}
In this case after the connection is established with the first user
the connection keeps open (PHP is not made for that) and so I save some space (request headers,response headers). This work on my server bu most server don't allow to keep the connection open for long time.
Also with multiple users I read the log multiple times.(slowing down my old server)
And I can't control the server ... I would need to use ajax to send a command...
I need WebSockets!!!
node.js and websockets
using node.js, from what i understand, i can do all this without consuming alot
of resources and bandwich. The connection keeps open so no unnecessary headers, i can recieve and send data.it handles multiple users very well.
And this is where i need your help.
the node.js server should in background update, and store the logs data every 5 sec if the files are modified.OR should that do the operating system with (iwatch,dnotify...)
the data should be pushed only if changed.
the reading of the logs should be happen only one time after 5 sec ... so not triggered by each user.
this is the first example i have found.....and modified..
var ws=require("nodejs-websocket");
var server=ws.createServer(function(conn){
var data=read(whereToStoreTheLogs);
conn.sendText(data)// send the logs data to the user
//on first connection.
setTimeout(checkLogs,5000);
/*
here i need to continuosly check if the logs are changed.
but if i use setInterval(checkLogs,5000) or setTimeout
every user invokes a new timer and so having lots of timers on the server
can i do that in background?
*/
conn.on("text",function(str){
doStuff(str); // various commands to control the server.
})
conn.on("close",function(code,reason){
console.log("Connection closed")
})
}).listen(8001);
var checkLogs=function(){
var data=read(whereToStoreTheLogs);
if(data!=oldData){
conn.sendText(data)
}
setTimeout(checkLogs,5000);
}
the above script would be the notification server, but i also need to find a solution to store somwhere the info of those multiple logs and do that everytime something is changed, in the background.
How would you do to keep the bandwich low but also the server resources.
How would you do?
EDIT
Btw. is there a way to stream this data simultaneosly to all the clinets?
EDIT
About the logs: i also want to be able to scale the time dilatation between updates... i mean if i read the logs of ffmpeg i ned the update every sec if possible... but when no conversion is active.. i need to get the basic machine info every 5min maybe ... and so on...
GOALS:
1. performant way to read & store somewhere the logs data (only if clinets connected...[mysql,file, it's possible to store this info inside the ram(with node.js??)]).
2. performant way to stream the data to the various clients (simultanously).
3. be able to send commands to the server.. (bidirectional)
4. using web languages (js,php...), lunix commands( something that is easy to implement on multiple machines).. free software if needed.
best approach would be:
read the logs, based on current activity, to the system memory and stream simultaneously and continuosly, with an already open connection, to the various clients with webSockets.
i'don't know anything that could be faster.
UPDATE
The node.js server is up and running, using the http://einaros.github.io/ws/ webSocketServer implementation, as it appears to be the fastest one.
I wrote with the help of #HeadCode the following code to handle properly the client situation & to keep the process as low as possible. checking various things inside the broadcast loop. Now the pushing & the client handling is at a good point.
var
wss=new (require('ws').Server)({port:8080}),
isBusy,
logs,
clients,
i,
checkLogs=function(){
if(wss.clients&&(clients=wss.clients.length)){
isBusy||(logs=readLogs()/*,isBusy=true*/);
if(logs){
i=0;
while(i<clients){
wss.clients[i++].send(logs)
}
}
}
};
setInterval(checkLogs,2000);
But atm i'm using a really bad way to parse the logs.. (nodejs->httpRequest->php).. lol. After some googling i found out that i totally could stream the output of linux software directly to the nodejs app ... i didn't checked... but maybe that would be the best way to do it. node.js also has a filesystem api where icould read the logs. linux has it's own filesystem api.
the readLogs()(can be async) function is still something i'm not happy with.
nodejs filesystem?
linuxSoftware->nodejs output implementation
linux filesystem api.
keep in mind that i need to scan various folders for logs and then parse somehow the outputted data, and this every 2 seconds.
ps.: i adde isBusy to the server variables in case the logReading sytem is async.
EDIT
Answer is not complete.
Missing:
A performant way to read,parse and store the logs somewhere (linux filesystem api, or nodejs api, so the i store directly into system memory)
An explaination if it's possible to stream data directly to multiple users .
apparently nodejs loops trough the clients and so (i think) sending multiple times the data.
btw is it possible/worth to close the node server if there are no clients and restart on new connections on the apache side. (ex: if i connect to the apache hosted html file a script launches the nodejs server again). doing so would further reduce the memory leaking???right?
EDIT
After some experimenting with websockets (some videos are in the comments) i learned some new stuff. Raspberry PI has the possibility to use some CPU DMA channels to to high frequency stuff like PWM... i need to somehow understand how that works.
When using sensors and stuff like that i should store everything inside the RAM, nodejs already does that?? (in a variable inside the script)
websocket remains the best choice as it's basically easely accessible from any device now, simply using a browser.

I haven't used nodejs-websocket, but it looks like it will accept an http connection and do the upgrade as well as creating the server. If all you care about receiving is text/json then I suppose that would be fine, but it seems to me you might want to serve a web page along with it.
Here is a way to use express and socket.io to achieve what you're asking about:
var express = require('express');
var app = express();
var http = require('http').Server(app);
var io = require('socket.io')(http);
app.use(express.static(__dirname + '/'));
app.get('/', function(req, res){
res.sendfile('index.html');
});
io.on('connection', function(socket){
// This is where we should emit the cached values of everything
// that has been collected so far so this user doesn't have to
// wait for a changed value on the monitored host to see
// what is going on.
// This code is based on something I wrote for myself so it's not
// going to do anything for you as is. You'll have to implement
// your own caching mechanism.
for (var stat in cache) {
if (cache.hasOwnProperty(stat)) {
socket.emit('data', JSON.stringify(cache[stat]));
}
}
});
http.listen(3000, function(){
console.log('listening on *:3000');
});
(function checkLogs(){
var data=read(whereToStoreTheLogs);
if(data!=oldData){
io.emit(data)
}
setTimeout(checkLogs,5000);
})();
Of course, the checkLogs function has to be fleshed out by you. I have only cut and pasted it in here for context. The call to the emit function of the io object will send the message out to all connected users but the checkLogs function will only fire once (and then keep calling itself), not every time someone connects.
In your index.html page you can have something like this. It should be included in the html page at the bottom, just before the closing body tag.
<script src="/path/to/socket.io.js"></script>
<script>
// Set up the websocket for receiving updates from the server
var socket = io();
socket.on('data', function(msg){
// Do something with your message here, such as using javascript
// to display it in an appropriate spot on the page.
document.getElementById("content").innerHTML = msg;
});
</script>
By the way, check out the Nodejs documentation for a variety of built-in methods for checking system resources (https://nodejs.org/api/os.html).
Here's also a solution more in keeping with what it appears you want. Use this for your html page:
<!DOCTYPE HTML>
<html>
<head>
<meta charset="utf-8">
<title>WS example</title>
</head>
<body>
<script>
var connection;
window.addEventListener("load", function () {
connection = new WebSocket("ws://"+window.location.hostname+":8001")
connection.onopen = function () {
console.log("Connection opened")
}
connection.onclose = function () {
console.log("Connection closed")
}
connection.onerror = function () {
console.error("Connection error")
}
connection.onmessage = function (event) {
var div = document.createElement("div")
div.textContent = event.data
document.body.appendChild(div)
}
});
</script>
</body>
</html>
And use this as your web socket server code, recently tweaked to use the 'tail' module (as found in this post: How to do `tail -f logfile.txt`-like processing in node.js?), which you will have to install using npm (Note: tail makes use of fs.watch, which is not guaranteed to work the same everywhere):
var ws = require("nodejs-websocket")
var os = require('os');
Tail = require('tail').Tail;
tail = new Tail('./testlog.txt');
var server = ws.createServer(function (conn) {
conn.on("text", function (str) {
console.log("Received " + str);
});
conn.on("close", function (code, reason) {
console.log("Connection closed");
});
}).listen(8001);
setInterval(function(){ checkLoad(); }, 5000);
function broadcast(mesg) {
server.connections.forEach(function (conn) {
conn.sendText(mesg)
})
}
var load = '';
function checkLoad(){
var new_load = os.loadavg().toString();
if (new_load === 'load'){
return;
}
load = new_load;
broadcast(load);
}
tail.on("line", function(data) {
broadcast(data);
});
Obviously this is very basic and you will have to change it for your needs.

I had made a similar implementation recently using Munin . Munin is a wonderful server monitoring tool, open source too which also provides a REST API. There several plugins available for your needs monitoring CPU, HDD and RAM usage of your server.

You need to build a push notification server. All clients who are listening, will then get a push notification when new data is updated. See this answer for more information: PHP - Push Notifications
As to how you would update the data, I'd suggest using OS-based tools to trigger a PHP script (command line) that will generate an "push" the json file out to any client currently listening. Any new client logging on to "listen" will get served the current json available, until it's updated.
This way you're not subject to 100 users using 100 connections and how much ever bandwidth to poll your server every 5 seconds, and only get updated when they need to know there's an update.

How about a service that reads all the log info (via IPMI, Nagios or whatever) and creates the output files on some schedule. Then anyone that wants to connect can just read this output rather than hammering the server logs. Essentially have one hit on the server logs then everyone else just reads a web page.
This could be implemented pretty easily.
BTW: Nagios has a v nice free edition

Answering just these bits of your question:
performant way to stream the data to the various clients (simultanously).
be able to send commands to the server.. (bidirectional)
using web languages (js,php...), lunix commands( something that is easy to implement on multiple machines).. free software if needed.
I'll recommend the Bayeux protocol as made simple by the CometD project. There are implementations in a variety of languages and it's really easy to use in its simplest form.
Meteor is broadly similar. It's an application development framework rather than a family of libraries, but it solves the same problems.

Some suggestions:
Munin for charts
NetSNMP (used by Munin, but you can also use Bash and Cron to build traps that send SMS texts on alerts)
Pingdom for remote alerts about how well the server is responding to ping and HTTP checks. It can SMS text you or call a phone, as well as have call escalation rules.

Flooding WebSocket

Am new to websocket and i implemented websocket on web application which the server-side is written in java and client-side is javascript. Server send notifications to client via websocket.
I wonder what would happened if client won't be fast enough to handle incoming messages as fast as server is sending them.
For example it is possible that server will be sending about 200 text messages per second, client is slow and is handling 100 messages per second.
I believe that browser queue incoming messages before it's processed but not sure. I what to also know how to check this buffer size and it's limit, and what would happen if buffer limit is reached.
Any idea on how i can simulate this kind of situation, i tried:
webSocket.onmessage = function (message) {
var bool = true;
var datenexexec = Date.now() + 1000;
while(bool) {
if(Date.now() > datenexexec){
bool = false;
}
}
}
but this causes the browser to only hang and later crash.
Thanks for help.

For sending data more rapidly than the client can read it, here's what will eventually happen.
The client receive buffer would fill up
TCP flow control will kick in and the server will be told to stop sending more packets on this socket.
The server will then buffer outgoing packets until flow control restrictions are removed
Eventually the server-side buffer limit will be hit and the underlying TCP would reject the socket write
This will return an error from the TCP send.
Depending upon what server-side library you are using for webSocket, you should get an error from a send operation at some point.
TCP is a reliable protocol so it will just buffer and transmit later until the buffer is full. It shouldn't lose packets by itself (unless the connection drops), but when buffers are full, it will give you an error that it can't send any more because the buffer is full.
As for the client-side code you tried, you can't busy/wait in Javascript for very long. That kills the event loop and eventually brings down the script engine.
The only way for you to simulate this is to try to actually send more packets than the client can process. You can code a "slow" client that takes maybe 250ms to process each packet in a short busy/wait loop and a "fast" server that sends a flood of packets and you should be able to simulate it.