Develop Reference

Implement Server Side Events with Grails

I want to implement Server side events with grails. What i want is that only when there is a change in the DataBase my DataTable should refresh. I want to user HTML5 Server Side events for this. My first question is that while using SSE i observed that the client keeps making a request to the server and if data is available it pulls it. Its similar to an Ajax call being sent every 3-4 seconds which can be changed but what i really want is that only and only when the Data in the DataBase changes should there be a refresh in the DataTable. Also i want to send JSON data to the client but am unable to send it in the right format. Below is my controller code.
def test(){
def action
something.each(){
action = "<a href=\"javascript:fetchDetails('"+it.id+"','comments'"+")\" class='btn btn-primary'><span class='glyphicon glyphicon-upload'></span></a>"
dataArr << [
it.Number,
it.product,
it.description,
OrgChart.findByid(it.Owner)?.displayName,
OrgChart.findByid(it.Coordinator)?.displayName,
startDate,
endDate,
it.status.status,
action
]
}
println dataArr
response.setContentType("text/event-stream;charset=UTF-8")
response << "data: ${dataArr}\n\n"
render "Hi"
}
}
Below is the gsp or the client side code
console.log("Starting eventSource");
var eventSource = new EventSource("/ops/test");
console.log("Started eventSource");
eventSource.onmessage = function(event) {
var data = JSON.stringify(event.data)
console.log("Message received: " + JSON.parse(data));
changeRequestsTable.clear().rows.add(JSON.parse(event.data)).draw()
};
eventSource.onopen = function(event) { console.log("Open " + event); };
eventSource.onerror = function(event) { console.log("Error " + event); };
console.log("eventState: " + eventSource.readyState);
// Stop trying after 10 seconds of errors
setTimeout(function() {eventSource.close();}, 10000);
I know i am a long way from implementing what i intend to but any help would be really appreciated

Going to answer this since it is getting to be a long conversation. As it stands the solution is too broad to give a proper answer since the angle as to how you do things could vary in such a dramatic range from using events that get triggered upon record save that then go off to either make client socket connections through application through to direct socket client connection at point of save that triggers something to be sent to clients. These methods are probably all more complex and more entangled and in short can be done all in a much easier way.
As the users go their interface get the users to make a ws connection to a backend listener. It can be the same location as in no room/separation (additional complexity needed).
As they join room declare a static concurrent map in the top of your websocket listener that collections each session and userId. I wouldn't do it the chat way since this is injected through a service that keeps it as a collection instead like this and changing RunnableFuture to be Websocket sessions like seen in the service link example.
Once you have this you can simply call a broadcast something like this that gets hold of your static concurrent map and for each session either broadcasts the entire new list in json and user processes html update with it or sends a trigger to say update page and they go off doing ajax call to update list.

See the following guide for a tutorial on how to implement Server Sent Events with Grails 3:
http://guides.grails.org/server-sent-events/guide/index.html

Real time message receiving from server (Database - MySQL, Frontend -Angularjs)

My code is working perfectly , But i need to refresh my page to get
new message . I want the real time message.
Angular controller's function -
$scope.getAllMessages = function() {
$scope.messages=[];
restservice.get('', "api/v1/message/getMessageList").then(function(response) {
if (response != null) {
for (var i = 0; i < response.length; i++) {
$scope.messages.push(response[i]);
}
}
});
}
My UI looks like -
<li ng-repeat="message in messages" class="{{(message.me == '1')? 'me' : 'other'}}">
<div>
{{message.userMessage}}
</div>
</li>
Please make it simple for me and give me a proper guideline/tutorial to do it. I'm bad at front end.

Another way is to use WebSockets.
Javascript has built-it WebSocket object wich you can use. And you also need to handle websockets on your server side. But for messaging websockets or long polling requests looks as more useful technologies, as for me.
WebSockets are very simple and if you want real-time, they could be useful. But you would have to implement websockets backend on your server. Here is a sample of WS usage from the JS side:
var exampleSocket = new WebSocket("ws://www.example.com/socketserver", "protocolOne");
exampleSocket.send("Here's some text that the server is urgently awaiting!");
exampleSocket.onmessage = function (event) {
console.log(event.data);
}
This is example from here: https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_client_applications

You need to call getAllMessages method on interval basis so that you get live data or real time data
Link here
https://docs.angularjs.org/api/ng/service/$interval
check with the above link for Example how to implement

To avoid getting ALL messages again i will considere do another .get maybe like this:
$interval(getLastMessagesFrom, 1000);
function getLastMessagesFrom () {
// Extract the last message_ID from $scope.messages, to do this maybe
// you will need to change your backend code to add the message ID in
// the message object. This deppends on your backend code & backend
// message response.
var last_message_id = $scope.messages[$scope.messages.length-1].id;
restservice.get('', "api/v1/message/getMessageList/" + last_message_id).then(function(response) {
if (response != null) {
// Now make sure that your server only returns messages with
// ID higher than last_message_id
for (var i = 0; i < response.length; i++) {
$scope.messages.push(response[i]);
}
}
});
}
Anyways this is still a bad practice if you have a lower time interval (1000 = 1 second) and a lot of concurrent users.
Considere use sockets to avoid repeated and unecessary calls.

use nodeJS's sockets
where you can achieve real-time experience.
e.g
once you inserted to database,you can fire JS command to emit msg realtime
io.sockets.connected[setsocketid].emit("clssifiedMessage", {msg:"Hi!!!how are yyou?" });

How do I publish a piece of data and then stop reacting to it?

I want to make a homepage where several pieces of data are published, but only when the user first visits the page : one would get the latest 10 articles published but that's it - it won't keep changing.
Is there a way to make the inbuilt pub/sub mechanism turn itself off after a set amount of time or number of records, or another mechanism?
Right now I'm using a very simple setup that doesn't "turn off":
latestNews = new Mongo.Collection('latestNews');
if (Meteor.isClient) {
Meteor.subscribe("latestNews");
}
if (Meteor.isServer) {
Meteor.publish('latestNews', function() {
return latestNews.find({}, {sort: { createdAt: -1 }, limit : 10});
});
}

The pub/sub pattern as it is implemented in Meteor is all about reactive data updates. In your case that would mean if the author or last update date of an article changes then users would see this change immediately reflected on their home page.
However you want to send data once and not update it ever again.
Meteor has a built-in functionality to handle this scenario : Methods. A method is a way for the client to tell the server to execute computations and/or send pure non-reactive data.
//Server code
var lastTenArticlesOptions = {
sort : {
createdAt : -1
},
limit : 10
}
Meteor.methods({
'retrieve last ten articles' : function() {
return latestNews.find({}, lastTenArticlesOptions).fetch()
}
})
Note that contrary to publications we do not send a Mongo.Cursor! Cursors are used in publications as a handy (aka magic) way to tell the server which data to send.
Here, we are sending the data the data directly by fetching the cursor to get an array of articles which will then be EJSON.stringifyied automatically and sent to the client.
If you need to send reactive data to the client and at a later point in time to stop pushing updates, then your best bet is relying on a pub/sub temporarily, and then to manually stop the publication (server-side) or the subscription (client-side) :
Meteor.publish('last ten articles', function() {
return latestNews.find({}, lastTenArticlesOptions)
})
var subscription = Meteor.subscribe('last ten articles')
//Later...
subscription.stop()
On the server-side you would store the publication handle (this) and then manipulate it.
Stopping a subscription or publication does not destroy the documents already sent (the user won't see the last ten articles suddenly disappear).

Speeding up an app that makes many Facebook API calls

I've got a simple app that fetches a user's complete feed from the Facebook API in order to tally the number of words he or she has written total on the site.
After he or she authenticates, the page makes a Graph call to /me/feed?limit100 and counts the number of responses and their dates. If there is a "next" cursor in the response, it then pings that next URL, which looks something like this:
https://graph.facebook.com/[UID]/feed?limit=100&until=1386553333
And so on, recursively, until we reach the time that the user joined Facebook. The function looks like this:
var words = 0;
var posts = function(callback, url) {
url = url || '/me/posts?limit=100';
FB.api(url, function(response) {
if (response.data) {
response.data.forEach(function(status) {
if (status.message) {
words += status.message.split(/ /g).length;
}
});
}
if (response.paging && response.paging.next) {
posts(callback, response.paging.next);
} else {
alert("You wrote " + words + " on Facebook!");
}
});
}
This works just fine for people who have posts a total of up to 4,000 statuses, but it really starts to crawl for power users with 10,000 lifetime updates or more. Each response from the API is only about 25Kb, but I cannot figure out what's straining the most.
After I've added the number of words in each status to my total word count, do I need to specifically destroy the response object so as not to overload memory?
Alternatively, is the recursion depth a problem? we're realistically talking about a total of 100 calls to the API for power users. I've experimented with upping the limit on each call to fetch larger chunks, but it doesn't seem to make a huge difference.
Thanks.

So, you're doing this with the JS SDK I guess, which mean this runs in the Browser... Did you try to run this in Chrome and then watch the network monitor to see about the response time etc.?
With 100 requests, this also means that the data object/JSON must be about the size of 2.5mb, which for some browsers/machines could be quite challenging I guess. Also, it must take quite a while to fetch the data from FB. What does the user see in the meantime?
Did you think of implementing this in the backend on the server side, and then just passing the results to the frontend?
For exmple use NodeJS together with SocketIO to do it on the server side and dynamically update the word count?

CouchDB, Node.js, Cradle - How to get data based on returned data

I am working on a messaging system using node.js + cradle and couchdb.
When a user pulls a list of their messages, I need to pull the online status of the user that sent them the message. The online status is stored in the user document for each registered user, and the message info is stored in a separate document.
Here is the only way I can manage to do what I need, but its hugely inefficient
privatemessages/all key = username of the message recipient
db.view('privatemessages/all', {"key":username}, function (err, res) {
res.forEach(function (rowA) {
db.view('users/all', {"key":rowA.username}, function (err, res) {
res.forEach(function (row) {
result.push({onlinestatus:row.onlinestatus, messagedata: rowA});
});
});
});
response.end(JSON.stringify(result));
});
Can someone tell me the correct way of doing this?
Thank you

Your code could return empty result because you are calling response at the time when user statuses may not yet be fetched from DB. Other problem is that if I received multiple messages from the same user, then call for his status may be duplicit. Below is a function which first fetch messages from DB avoiding duplicity of users and then get their statuses.
function getMessages(username, callback) {
// this would be "buffer" for senders of the messages
var users = {};
// variable for a number of total users I have - it would be used to determine
// the callback call because this function is doing async jobs
var usersCount = 0;
// helpers vars
var i = 0, user, item;
// get all the messages which recipient is "username"
db.view('privatemessages/all', {"key":username}, function (errA, resA) {
// for each of the message
resA.forEach(function (rowA) {
user = users[rowA.username];
// if user doesn't exists - add him to users list with current message
// else - add current message to existing user
if(!user) {
users[rowA.username] = {
// I guess this is the name of the sender
name: rowA.username,
// here will come his current status later
status: "",
// in this case I may only need content, so there is probably
// no need to insert whole message to array
messages: [rowA]
};
usersCount++;
} else {
user.messages.push(rowA);
}
});
// I should have all the senders with their messages
// and now I need to get their statuses
for(item in users) {
// assuming that user documents have keys based on their names
db.get(item, function(err, doc) {
i++;
// assign user status
users[item].status = doc.onlineStatus;
// when I finally fetched status of the last user, it's time to
// execute callback and rerutn my results
if(i === usersCount) {
callback(users);
}
});
}
});
}
...
getMessages(username, function(result) {
response.end(JSON.stringify(result));
});
Although CouchDB is a great document database you should be careful with frequent updates of existing documents because it creates entirely new document version after each update (this is because of it's MVCC model which is used to achieve high availability and data durability). Consequence of this behavior is higher disk space consumption (more data/updates, more disk space needed - example), so you should watch it and run database consumption accordingly.

I think your system could use an in memory hashmap like memcached. Each user status entry would expire after a time limit.
Mapping would be
[user -> lasttimeseen]
If the hashmap contains the user, then the user is online.
On some certain actions, refresh the lasttimeseen.
Then instead of pinging the whole world each time, just query the map itself and return the result.

I'm reminded of this presentation:
Databases Suck for Messaging
And its quote from Tim O'Reilly:
"On monday friendfeed polled flickr nearly 3 million times for 45000 users, only 6K of whom were logged in. Architectural mismatch."
As pointed out in the other answers, updates in CouchDB are expensive and should be avoided if possible, and there's probably no need for this data to be persistent. A cache or messaging system may solve your problem more elegantly and more efficiently.