I have built a web application using AngularJS (front-end) and PHP/MySQL (back-end).
I was wondering if there is a way to "watch" the MySQL database (without Node.js), so if one user adds some data to it, the changes are synced to other users too.
E.g. I know Firebase does that, but it's object oriented database and I am unable to do the advanced queries there like I do with SQL.
I was thinking to use $interval and $http and do ajax requests, so that way I could detect changes in the database. Well, that's possible, but it'll then do thousands of http requests to the server everyday and plus interpret php on each request.
I believe nothing is impossible, I just need an idea to do this, which I don't have, so that's why I am asking for a help here.
If you want a form of "real-time communication" you'll likely have to incorporate some form of long-polling from the client. Unless you use web sockets, but that's a big post about a bunch of different things. You're right to be concerned about bandwidth and demand on the DB though. So here's my suggestion:
If you don't have experience with web sockets then log your events in a separate table/view and use the pub/sub method to subscribe entities to an event, and broadcast that event to the table. Then long-poll against the watcher view to see when changes may have occurred. If one did occur then you query for the exact value.
Another option would be to use some query system with "deciders" that hold messages. Take a look at Amazon's SQS platform for a better explanation of how this could work. Basically you have a queue that holds messages and a decider chooses where to store the message using some hash or sorting method (to reduce run time). When the client requests an update, the decider finds any messages that would apply based on the hash/sort and returns them. Then you just have to decide how and when to destruct the messages.
The second option would require a lot more tinkering though, so it's really about your preference. I think what you'll find the difficulty to be is that most solutions have to deal with the fact that the message has to be delivered 1 or More times and you'll need to track when someone received the message and if it can now be deleted from the queue/event table or if you still need to wait. Otherwise you'll consume a lot of memory.
Related
I have struggled to find many resources on this online. I am developing an application that multiple users will be using at the same time. This means that one user may edit the database after another user has loaded the data from database. This means that this second user will not have an up to date view of the current state of the database. What is the best way to subscribe to database changes and deal with them. I am using a MEAN stack.
If you are trying to develop a real time system where changes are reflected instantly upon changes in database, you need to make use of web sockets. Since you are using Node.js as backend, see Socket.io
A good resource for implementation can be found here
However, if you plan on implementing web sockets, you will have to make significant changes to both your Node.js and Angular code.
Another method (which I would not recommend) is to make periodic api calls for those views which you want to reflect real time changes. You can make use of setInterval for this
Okay, let me start by saying that I know this is weird. I do.
But here goes:
Let's say I have an SQL database which stores my data. And let's say I don't have a choice in this, it has to be SQL. The application I'm building has somewhere in the region of 100,000 records in its database, and once every single record has been processed by the users of the application, they all go off and get sent to a different application entirely. So for a short period of time, this application will be in use, and then stops being used until the same time next year. While the application is in use, no external sources will be touching the database at all.
When the (Node) server starts up, it loads everything from the database, into an object literal on the server.
The client-side of this application, on a very basic level, makes requests (to an API on the server) for data, and sends updated versions of records back to the server once they've been processed.
So here's where it gets weird: Let's say I don't want to have the client-side application have to directly retrieve records from the database, nor do I want it to be able to write to them. So the data from the entire database already exists in memory on the server. There's a module on the server that can handle changing the representation of that data already (again, because the client application only interacts with APIs on the server, the database module exists to facilitate this).
Multiple users access the system at once, but due to the way the system works, it is not possible for two users to be sent the same record, so two users will never be sending an update back for the same record (records are processed individually, and sequentially).
So, let's say that I decided that, since I was already managing all of this data in memory on the server, I would just send an updated version of the current data, in its entirety, back to the database, every time it changed.
The question is, where does this rank on the crazy scale?
Performance, writing an entire database rather than single records, would obviously suffer. But, in a database that is only read from once (on start-up of the application), is that even a concern? If every operation other than "Write all the stuff when any of the stuff changes" happened in memory on the server, does it matter how long those updates actually take? If a new update to the database comes in whilst it's being updated, surely SQL will take care of this?
It feels like the correct way to do this of course, is to have each user directly getting their info from the database, and directly making updates to the database too (or at least interacting with API endpoints to make this happen), but, is just...not doing that, utter lunacy?
Like I said, I know it's weird, but other than the fact that "it feels kind of wrong", I'm not sure I'm convinced that it is in fact entirely wrong. So I figured that this place would have an opinion.
The way that I think it currently works is:
[SQL DB] is updated whenever a change happens on {in-memory DB}
{in-memory DB} is updated in various ways based on API calls to the server
makes requests for data, and sends updates to data, both of which are processed on the in-memory DB
Multiple requests can happen at the same time from the application, but mutliple users can not see the same record, because records are allocated to a given user before they're sent
Multiple updates can come from multiple users, each of which ultimately ends in the entire SQL database being saved to with the contents of the in-memory DB.
(Note: I'm not saying "is this the best way to do this". I'm just asking, is there a significant argument for caring about the performance of a database being written to, if it's not going to be read from again unless the server needs to be restarted)
What I think that I would do, in this situation, is to add an attribute to each cached record to indicate that the record is "dirty." In other words, that something has been done to it, by someone, since it was originally read from the database.
(You could also add an attribute that indicates that someone "has this particular record 'checked-out,'" so that you can be sure that two users are no updating the same record at the same time.)
At some convenient moment, you can then walk through the collection, posting the "dirty" records back to the database. Use an SQL Transaction, not only for efficiency but also to be sure that the final update to the database is atomic.
You will need to be very mindful of the possibility of race-conditions. One possible strategy is to use a Unix timestamp as a "dirty" indicator. A record is selected for posting to the database only if its "dirty-time" is greater-than or equal-to the timestamp when the commit-process was last run.
(And, P.S.: "no, I've seen even 'weirder' things than this, in all my crazy years in this crazy business ...)
I wrote a web page where there is a zone for user's comments.
Any authenticated users could post a comment.
As many users could post comments almost simultaneously, I want the comments list to be auto-refreshed.
Thus, I think about using WebSockets.
My thought are about a good/best practice for this use case:
Once a comment is posted, should WebSockets process read the current comments list on database and send a Json response containing all the new comments? This would allow the client to directly append the new comments on the DOM (JS).
Or should WebSocket just check the database (or queue if using a message queue (Redis, RabbitMQ etc..) for instance) and act just like: "Hey, I have new comments, click here if you want to see them !". This solution would only signal the presence of new comments, without bringing all those comments to the client. The workflow of retrieving the events would then involve by the client (by clicking on this sentence for instance) e.g using the traditional Ajax direction: client => server.
It is highly possible that a user posts a comment, then navigates to another page of the website. Therefore, a websocket response containing the whole new comments would be useless. A simple notification would then be possible, as most of known websites do for instance with the "+1" counter or more relevant to the "comments" scenario: "1 new comment available".
Which way should I choose?
I think to decide which data to push is mostly a matter of UI usability / user experience, as opposed to which technology is being used to interact with the server. We should avoid changing the UI with server pushed data in a way that would surprise the user in a negative way, for example having the comment feed constantly growing without any intervention from him.
But in the case of a realtime chart, it's probably better to push the data directly into the chart, that would be what the user expects.
In the case of the comment feed the reason why most sites go with the 'click to load' approach is because of user experience, so I think that is probably the best approach.
I use a combination of both....
In some pages the websocket communication contains the actual data--sort of like a stock ticker update.
And in other cases, the websocket communication just says -- all users viewing xyz data--refresh it. And then the browsers performs an ajax to obtain the new data and the grid is smartly refreshed in such a way that only the changed cells are modified on screen using innerHTML and the new rows are added and deleted rows are removed.
In cases like stackoverflow, it makes sense to show a message, "Got new stuff to show--want to see it?"
When I establish the websocket in the browser, I pass a page Id in the url and the cookies are passed too. So websocket server knows--the user cookie and the page which is being viewed.
Then in the database (or middle tier logic) communicates to the websocket server with messages such as: This message is for users viewing 'xyz' page: smartly refresh grid 'abc'. And the websocket server broadcasts the message.
Because the protocol allows you to pass anything you like, you have the ability to make it anyway you like.
My advise it to do what's best in each particular situation.
I have a CQRS application with eventual consistency between the event store and the read model. In it I have a list of items and under the list a "Create new" button. When a user successfully creates a new item he is directed back to the list but since the read model has not been updated yet (eventual consistency) the item is missing in the list.
I want to fake the entry in the list until the read model has been updated.
How do I best do that and how do I remove it when the new item is present in the actual list? I expect delays of about 60 seconds for the read model to catch up.
I do realize that there are simpler ways to achieve this behavior without CQRS but the rest of the application really benefits from CQRS.
If it matters the application is a c# mvc4 application. I've been thinking of solutions involving HTML5 Web Storage but want to know what the best practice is for solving this kind of problem.
In this situation, you can present the result in the UI with total confidence. There is no difference in presenting this information directly and reading it from the read model.
Your domain objects are up to date with the UI and that's what really matters here. Moreover, if you valid correctly your AR state in every operation and you keep track of the concurrency with the AR's version then you're safe and your model will be protected against invalid operations.
At the end, what are the probability of your UI going out of sync? This can happen if you there are many users modifying the information you're displaying at the same time. This can be avoided by creating task based UI and following the rule 'one command/operation in the AR per request'.
The read model can be unsynced until the denormalizers do their job.
In the other hand, if the command will generate a conversation (long running operation) between a saga and AR's then you cannot do this and must warn the user about it.
It doesn't matter that's a asp.net mvc app. The only solution I see, besides just telling the user to wait a bit, is to have another but this time synchronous event handler that generate the same model (of course the actual model generation should be encapsulated in a service) and sends it to a memory cache.
Being everything in memory makes it very fast and being synchronous means it's automatically executed before the request ends. I'm assuming the command is executed syncronously too.
Then in your query repository you also consider results from cache, removing it if that result is already returned by the db.
Personally, for things that I know I want to be available to the user and where the read model generation is trivial, I would use only synchronous event handlers. The user doesn't mind waiting a few seconds when submitting something and if updating a read model takes a few seconds, you know you have a backend problem.
I see that eventual consistency is applicable to application only if application environment has multiple front-end servers hosting application and all these servers has own copy of read model. All servers uses same copy of event store.
When something is changed to event store, read model that is used to read result to user must be updated in sync with event store. Rest of servers and read models managed by them can be updated using eventual consistency.
This way result to user (list of items) can be read from local read model copy because it is already updated in sync. No need for special complex fake updates/rollbacks.
Only case when user can see incomplete list is that user hits F5 to refresh list after update change and load balancing directs user request to front-end server which read model is not yet updated (60 second delay), but this can be avoided so that load balancing does not change users server in middle of session.
So, if application has only one front-end server, eventual consistency is not very usable or it does not give any benefits without some special fake updates/rollbacks with read model...
I am making a cooperation application like google docs, but I found this is different to do on the web. The problem is, when the user typing, another user should see the update at the same time. But what actually behind the screen? Is that when the user have an action, it sent a http request, and write into database. At the same time, another user get the action from database, and rendering the result that the user just type. If use this way to implement, the database need to keep read and write.....apart from this solution, how can I sync two people work on the fly? Thank you.
Check out something like this http://pusherapp.com/
Or http://www.tornadoweb.org/
Both are good at real time pushes without constant AJAX requests that will put a lot of strain on your server