I am building a web application which updates the HTML content of every client's web page depending on changes made by one client, for this purpose I am using Node.js's Socket.io so far.
I am using HTML5's contenteditable property to allow the client to manually edit the content of the div elements which I need to update for other clients as well.
The problem that I'm facing is that I don't understand what data to send over websockets to inform the server and, in turn, other clients of the changes made.
To send the whole innerHTML on adding and removing of every character means sending huge amount of data over websockets which results in bad performance and slow speed.
To send the appended data is not an option since we don't know at what position in the div element the data is appended or removed or edited.
Note that using the keyboard is not the only way how one client can change the HTML content of their copy of the web page. The required div element's data is changed depending on several activities by client using javascript.
Now I need to know how can I send the exact changed information over websockets on even the slightest change to get a realtime experience.
P.S. I intend not to use any existing modules like Share.js but suggestions are welcome.
This is not really a "question" rather a discussion and would be sort of "brainstorming" topic. As well it is pretty opinion based, as there is no specific one way of doing things talking so broad topic.
But I will take chance, and provide my pure IMHO the way I would approach this (sorry for being so personal):
Text Version Control - a way to keep track of version of text, something similar as git, but using character positions rather than lines. Each editable text should be version controlled, there should be one "master" that merges things in, and notifies clients of new versions. On client side, version changes (commits) should be applied the same way as server does merging. With respect of commits order.
Structure Updates - this is slightly different, as structure updates involve only HTML structure, and do not require versioning (there still is chance you would want versioning, or at least some history of actions, and there is racing conditions involved). Editing structure would notify server of such changes. It should make sure all clients are "on the same page" regarding structure.
Referencing - this can be hard bit. Referencing HTML elements can be pretty simple (selectors), but there is one complexity - when there is for example <ul> with many <li>, referencing each by index - fair enough, but what if you put new element in between? That will mess up references. So referencing should be unique, and fully depend on synchronised IDs that server would decide. You need to keep list of all references with pointer to html elements, so it is easy to access them directly.
Server <> Client politics. If you are talking fully about synchronised experience, then you have to follow authoritative politics of communication. Where only server really decides things, and clients render and only provide some input (requests). For example when client want to add element into HTML, he sends request and then server decides of such, and inserts it, once it is in, it will publish such event and then element will appear on clients and then they can edit content of this element. You can further enhance it by "going forward" without waiting for server but that should be done as extra layer, rather than part of main logic.
When one client is working on node (element/text), it should be the only who can work on it (to prevent conflicts). Commit will be considered as "in progress" and it should be single version commit. In beginning you can only update once user is "done" editing text or element. And later on add real-time visual capabilities of such process.
You have to research into amount of traffic this will inflict. With version controlling and event-based elements editing - it will be very efficient, but cost will be that server side should actually simulate whole DOM structure and do merging and other stuff.
Again this is primarily opinion based answer, and is not a small topic, it involves many fields. I think in terms "pure" and "good" approach to target, not the "fastest", there might be just much simpler solutions with some tradeoffs..
Related
I want to make it possible that multiple users can view the same website (a specific URL that all agree on) and all events of the users will be shared so that everyone has the same state. So that you can use a website with multiple people but the website thinks there is only one person, similar to when you use one computer with multiple people.
I have two ideas for how to do this:
The client-sided approach: everyone loads the same page with an iframe and then it detects all events of the users and sends these to each other so everyone has the same state.
Problems:
Each user might use a different browser and the website can be different for everyone and desynchronisation can also happen.
Emulating clicks might be difficult.
The server-sided approach: load the website only once on the server and then send all user events to the server and stream back the website's pixels back to the users.
Problems:
Streaming back the website's state (its look, the pixels) to all the users could be quite expensive, but maybe it could only update the pixels that actually changed, instead of all pixels of the website.
Because approach 1 doesn't seem very feasible, I would like to try approach 2, but I'm not sure where to start there. Do I make the server open the URL in its browser and let the system emulate clicks on the browser?
What is the best approach to solve this, are there more and better approaches?
ProseMirror is an open source web-based editor that was designed with collaborative editing in mind:
Live demo of collaborative editor
Source code for demo
I suggest using ProseMirror as a base, then modify it for your needs. ProseMirror defines a text-based document data structure. ProseMirror renders this data-structure as as a web-based editor, but you could render the data structure however you desire:
Render the document data structure as a web page.
Clicking the webpage alters the underlying document data structure.
ProseMirror takes care of synchronizing the document data structure on all the clients.
Other clients render the updated document.
If you are more interested in creating your own thing completely from scratch, I would study how ProseMirror did this:
A single authority (the server) maintains the "official" version of the document. Clients send requests to update the document, which the authority handles and broadcasts to all other clients.
Note peer-to-peer (called 'client-sided approach' in the question) can also have an authority, but there is added complexity to determine and maintain the authority. So server-client is simpler to implement
Minimize network bandwidth. Sending every pixel or click event is expensive. (Also clicks may result in different, diverging documents on different clients.) Ideally, transactions are sent, which are deltas describing how the document changed.
There was a similar question, where I went into more detail: Creating collaborative whiteboard drawing application
For the desynchronization problem you can create a website that'll look the same to all by using styles. Like fixed dimensions for all browsers and screen sizes. That'll help in rendering the same version and same dimension to all the users.
For click synchronization you can use node sockets. By using this you can manage a server-side directory for a group of users and share the same events using peer-to-peer sockets. This is similar to the group chat functionality. https://socket.io/ can help you in implementing this. Node Sockets are also used in client-side multi-user games for making the same connected experience to both the users.
This would be a client-server solution so that you can manage everything from the server and provide a good experience to your users.
Hope this helps. If you need additional information on this please drop a comment.
It looks like you need to create a reactive application. So, your web page
will serve any content to many users at the same time and they will be able to interact with this app almost in real time (milliseconds). All users will be able to see all interactions.
I have used the above scenario using Meteor, which is reactive by default. Of course you can use other frameworks too or try the difficult way to manipulate communication between clients by yourself, using some clever javascript.
I always find it difficult to decide if I should render my HTML server-side or build it client-side.
Let's say I want to render a dynamic HTML dropdown with following requirements:
Shows records from a database table at page load
Must stay up-to-date (users can add/remove records from the database table using the website)
I can't decide between
Option 1
Render template with empty dropdown server-side
Populate dropdown client-side using ajax requests (JSON)
Update dropdown client-side using ajax requests (JSON)
Concern: Rendering empty elements before populating feels ugly to me
Option 2
Render populated dropdown server-side
Update dropdown client-side using ajax requests (JSON)
Concern: Why would you render server-side if you are still updating it client-side?
What solution is more commonly used in web development? Feel free to share different approaches.
This is a little opinion bases. There are schools for the server-side, and other schools for the client-side. The main reason for the later is utilizing the client machine (which are free for you) to free the server resources (which you pay for). This also makes your app more scalable.
The example that you gave is not detailed enough, and it is only one aspect. I usually use these general rules:
If there are dynamic parts of your page (e.g. dropdown, grid, popup form, etc), I use Ajax to avoid reloading the entire page.
If the HTML is very simple and/or requires further processing on the client-side, then just send JSON data from the server and build the HTML on the client-side. For example, I usually do not send error message from the server. Instead, I only send status codes (e.g. Successful, AccessDenied, Error, etc...) and I inspect those codes on the client and display the associated message. This is specially useful when different messages are displayed in different colors and styles or contain other HTML tags like links.
If it is a complex popup form or grid of data, then I send the HTML from the server. It is usually much easier and more flexible to build complex HTML on the server. Also when there are sensitive information used to build the HTML, it is usually much easier to build it on the server, otherwise you'll have to send some flags from the server or, better, you need to split your HTML building process to sections depending on permissions.
As for that very specific example in your question, I would go with the first approach (all client-side), mainly for the DRY concept (Don't Repeat Yourself). You don't want to have two pieces of code doing the exact same thing, and have to remember to update both every time you need change something.
Note though, that you don't have to send empty drop-down if you don't like it. Instead of only updating options like your example is suggesting, you can actually rebuild the entire dropdown every time. I don't particularly like this approach, especially if there are event listeners and other references attached to the dropdown, but I just wanted to say other ways to do it. This method can be useful on some scenarios, especially if the dropdown is part of a bigger section of the page that this whole section requires updating (rebuilding) every time, in which case I usually opt for sending the HTML from the server.
Your concerns are valid, each case has its advantages and disadvantages as you mentioned.
I would, personally, go with the first approach (all client-side); mainly for code maintainability. With this approach, you only have to update the HTML client-side instead of both client-side and server-side.
However, there is an argument to be made for saving that one request by server-side rendering the values. Even though it might be insignificant, there is a small performance optimization.
That depends. are you worrying about SEO?
Are you using any kind of client-side framework? I will assume that you don't, since JavaScript frameworks have there own way to do this, if you want you can read more about angular/react/vuejs which is the hottest JavaScript frameworks those days that will solve this issue.
Client-side frameworks render HTML on client-side only, and use only Ajax API to receive data from the server.
But if you don't want to use any client-side framework and do it in the "classic" way, both ways are appreciated. I tend to like the first way, this is almost how client-side frameworks do it and makes the most sense, yes you render empty table but you only render the "container" of your data, if you're bothered by how it looks visually (Showing empty list before data is fetched) you can just show loading bar or hide the table till the data is fetched.
I've seen that Medium.com as well as Google Drive docs/sheets/etc. all use draft content saving so your editing doesn't get lost. Well Google's version is used for content versioning, but that's just an additional side effect as they're apparently not updating the same content but rather saving all of them getting content history records.
I'm interested in the way these two internally work.
If you observe Ajax calls being exchanged between the client and server one can see that both of them send some sort of partial content or user action saves. These kind of draft saving is very quick because even lengthy content can utilize tiny requests making draft saving a quick process. The problem is of course because these partial saves are now stateful, so they heavily rely on existing content state.
I'm also working on a web app that would benefit from such feature of tiny and quick saves, but I don't want to reinvent the wheel and jumping over all the hurdles. I'm not sure whether I should be
recording keystrokes and send them immediately which would then have to be replayed on the server to actually generate matching content state there.
should it work in any other way like recording keystrokes and sending them in predefined intervals?
not recording keystrokes at all but rather calculating content diffs in some way. Content diffs are done in CVSs like GitHub and I suppose this algorithm is very well defined by now and correctly shows removed as well as added content. If this process is not too complex for client-side processing I may go down this route and send small diffs
These are just three ways from the top of my head (although the first two are similar in nature) but there may be others that I haven't thought about.
I'm sure some of you may have done this before and have some experience as well as pros and cons of different techniques and if there's maybe an existing open source lib that can help simplify this without doing it all from scratch.
How should this be done?
I'm just looking for ideas/suggestions here; I'm not asking for a full on solution (although if you have one, I'd be happy to look at it)
I'm trying to find a way to only upload changes to text. It's most likely going to be used as a cloud-based application running on jQuery and HTML, with a PHP server running the back-end.
For example, if I have text like
asdfghjklasdfghjkl
And I change it to
asdfghjklXasdfghjkl
I don't want to have to upload the whole thing (the text can get pretty big)
For example, something like 8,X sent to the server could signify:
add an X to the 8th position
Or D8,3 could signify:
go to position 8 and delete the previous 3 terms
However, if a single request is corrupted en route to the server, the whole document could be corrupted since the positions would be changed. A simple hash could detect corruption, but then how would one go about recovering from the corruption? The client will have all of the data, but the data is possibly very large, and it is unlikely to be possible to upload.
So thanks for reading through this. Here is a short summary of what needs suggestions
Change/Modification Detection
Method to communicate the changes
Recovery from corruption
Anything else that needs improvement
There is already an accepted form for transmitting this kind of "differences" information. It's called Unified Diff.
The google-diff-match-patch provides implementations in Java, JavaScript, C++, C#, Lua and Python.
You should be able to just keep the "original text" and the "modified text" in variables on the client, then generate the diff in javascript (via diff-match-patch), send it to the server, along with a hash, and re-construct it (either using diff-match-patch or the unix "patch" program) on the server.
You might also want to consider including a "version" (or a modified date) when you send the original text to the client in the first place. Then include the same version (or date) in the "diff request" that the client sends up to the server. Verify the version on the server prior to applying the diff, so as to be sure that the server's copy of the text has not diverged from the client's copy while the modification was being made. (of course, in order for this to work, you'll need to update the version number on the server every time the master copy is updated).
You have a really interesting approach. But if the text files are really so large that it would need too much time to upload them every time, why do you have the send the whole thing to the client? Does the client really have to receive the whole 5mb text file? Wouldn't it be possible to send him only what he needs?
Anyway, to your question:
The first thing that comes to my mind when hearing "large text files" and modification detection is diff. For the algorithm, read here. This could be an approach to commit the changes, and it specifies a format for it. You'd just have to rebuild diff (or a part of it) in javascript. This will be not easy, but possible, as I guess. If the algorithm doesn't help you, possibly at least the definition of the diff file format does.
To the corruption issue: You don't have to fear that your date gets corrupted on the way, because the TCP protocol, on which HTTP is based, looks that everything arrives without being corrupted. What you should fear is the connection reset. Might be you can do something like a handshake? When the client sends an update to the server, the server applies the modifications and keeps one old version of the file. To ensure that the client has received the ratification from the server that the modification went fine (that's where the conneciton reset happens), the client sends back another ajax request to the server. If this one doesn't come to the server within sone definied time, the file gets reset on the server side.
Another thing: I don't know if javascript likes it to handle such gigantic files/data...
This sounds like a problem that versioning systems (CVS, SVN, Git, Bazaar) already solve very well.
They're all reasonably easy to set up on a server, and you can communicate with them through PHP.
After the setup, you'd get for free: versioning, log, rollback, handling of concurrent changes, proper diff syntax, tagging, branches...
You wouldn't get the 'send just the updates' functionality that you asked for. I'm not sure how important that is to you. Pure texts are really very cheap to send as far as bandwidth is concerned.
Personally, I would probably make a compromise similar to what Wikis do. Break down the whole text into smaller semantically coherent chunks (chapters, or even paragraphs), determine on the client side just which chunks have been edited (without going down to the character level), and send those.
The server could then answer with a diff, generated by your versioning system, which is something they do very efficiently. If you want to allow concurrent changes, you might run into cases where editors have to do manual merges, anyway.
Another general hint might be to look at what Google did with Wave. I have to remain general here, because I haven't really studied it in detail myself, but I seem to remember that there have been a few articles about how they've solved the real-time concurrent editing problem, which seems to be exactly what you'd like to do.
In summary, I believe the problem you're planning to tackle is far from trivial, there are tools that address many of the associated problems already, and I personally would compromise and reformulate the approach in favor of much less workload.
looking for some general advice and/or thoughts...
i'm creating what i think to be more of a web application then web page, because i intend it to be like a gmail app where you would leave the page open all day long while getting updates "pushed" to the page (for the interested i'm using the comet programming technique). i've never created a web page before that was so rich in ajax and javascript (i am now a huge fan of jquery). because of this, time and time again when i'm implementing a new feature that requires a dynamic change in the UI that the server needs to know about, i am faced with the same question:
1) should i do all the processing on the client in javascript and post back as little as possible via ajax
or
2) should i post a request to the server via ajax, have the server do all the processing and then send back the new html. then on the ajax response i do a simple assignment with the new HTML
i have been inclined to always follow #1. this web app i imagine may get pretty chatty with all the ajax requests. my thought is minimize as much as possible the size of the requests and responses, and rely on the continuously improving javascript engines to do as much of the processing and UI updates as possible. i've discovered with jquery i can do so much on the client side that i wouldn't have been able to do very easily before. my javascript code is actually much bigger and more complex than my serverside code. there are also simple calulcations i need to perform and i've pushed that on the client side, too.
i guess the main question i have is, should we ALWAYS strive for client side processing over server side processing whenever possible? i 've always felt the less the server has to handle the better for scalability/performance. let the power of the client's processor do all the hard work (if possible).
thoughts?
There are several considerations when deciding if new HTML fragments created by an ajax request should be constructed on the server or client side. Some things to consider:
Performance. The work your server has to do is what you should be concerned with. By doing more of the processing on the client side, you reduce the amount of work the server does, and speed things up. If the server can send a small bit of JSON instead of giant HTML fragment, for example, it'd be much more efficient to let the client do it. In situations where it's a small amount of data being sent either way, the difference is probably negligible.
Readability. The disadvantage to generating markup in your JavaScript is that it's much harder to read and maintain the code. Embedding HTML in quoted strings is nasty to look at in a text editor with syntax coloring set to JavaScript and makes for more difficult editing.
Separation of data, presentation, and behavior. Along the lines of readability, having HTML fragments in your JavaScript doesn't make much sense for code organization. HTML templates should handle the markup and JavaScript should be left alone to handle the behavior of your application. The contents of an HTML fragment being inserted into a page is not relevant to your JavaScript code, just the fact that it's being inserted, where, and when.
I tend to lean more toward returning HTML fragments from the server when dealing with ajax responses, for the readability and code organization reasons I mention above. Of course, it all depends on how your application works, how processing intensive the ajax responses are, and how much traffic the app is getting. If the server is having to do significant work in generating these responses and is causing a bottleneck, then it may be more important to push the work to the client and forego other considerations.
I'm currently working on a pretty computationally-heavy application right now and I'm rendering almost all of it on the client-side. I don't know exactly what your application is going to be doing (more details would be great), but I'd say your application could probably do the same. Just make sure all of your security- and database-related code lies on the server-side, because not doing so will open security holes in your application. Here are some general guidelines that I follow:
Don't ever rely on the user having a super-fast browser or computer. Some people are using Internet Explore 7 on old machines, and if it's too slow for them, you're going to lose a lot of potential customers. Test on as many different browsers and machines as possible.
Any time you have some code that could potentially slow down or freeze the browser momentarily, show a feedback mechanism (in most cases a simple "Loading" message will do) to tell the user that something is indeed going on, and the browser didn't just randomly freeze.
Try to load as much as you can during initialization and cache everything. In my application, I'm doing something similar to Gmail: show a loading bar, load up everything that the application will ever need, and then give the user a smooth experience from there on out. Yes, they're going to have to potentially wait a couple seconds for it to load, but after that there should be no problems.
Minimize DOM manipulation. Raw number-crunching JavaScript performance might be "fast enough", but access to the DOM is still slow. Avoid creating and destroying elements; instead simply hide them if you don't need them at the moment.
I recently ran into the same problem and decided to go with browser side processing, everything worked great in FF and IE8 and IE8 in 7 mode, but then... our client, using Internet Explorer 7 ran into problems, the application would freeze up and a script timeout box would appear, I had put too much work into the solution to throw it away so I ended up spending an hour or so optimizing the script and adding setTimeout wherever possible.
My suggestions?
If possible, keep non-critical calculations client side.
To keep data transfers low, use JSON and let the client side sort out the HTML.
Test your script using the lowest common denominator.
If needed use the profiling feature in FireBug. Corollary: use the uncompressed (development) version of jQuery.
I agree with you. Push as much as possible to users, but not too much. If your app slows or even worse crashes their browser you loose.
My advice is to actually test how you application acts when turned on for all day. Check that there are no memory leaks. Check that there isn't a ajax request created every half of second after working with application for a while (timers in JS can be a pain sometime).
Apart from that never perform user input validation with javascript. Always duplicate it on server.
Edit
Use jquery live binding. It will save you a lot of time when rebinding generated content and will make your architecture more clear. Sadly when I was developing with jQuery it wasn't available yet; we used other tools with same effect.
In past I also had a problem when one page part generation using ajax depends on other part generation. Generating first part first and second part second will make your page slower as expected. Plan this in front. Develop a pages so that they already have all content when opened.
Also (regarding simple pages too), keep number of referenced files on one server low. Join javascript and css libraries into one file on server side. Keep images on separate host, better separate hosts (creating just a third level domain will do too). Though this is worth it only on production; it will make development process more difficult.
Of course it depends on the data, but a majority of the time if you can push it client side, do. Make the client do more of the processing and use less bandwidth. (Again this depends on the data, you can get into cases that you have to send more data across to do it client side).
Some stuff like security checks should always be done on the server. If you have a computation that takes a lot of data and produces less data, also put it on the server.
Incidentally, did you know you could run Javascript on the server side, rendering templates and hitting databases? Check out the CommonJS ecosystem.
There could also be cross-browser support issues. If you're using a cross-browser, client-side library (eg JQuery) and it can handle all the processing you need then you can let the library take care of it. Generating cross-browser HTML server-side can be harder (tends to be more manual), depending on the complexity of the markup.
this is possible, but with the heavy intial page load && heavy use of caching. take gmail as an example
On initial page load, it downloads most of the js files it needed to run. And most of all cached.
dont over use of images and graphics.
Load all the data need to show in intial load and along with the subsequent predictable user data. in gmail & latest yahoo mail the inbox is not only populated with the single mail conversation body, It loads first few full email messages in advance at the time of pageload. secret of high resposiveness comes with the cost (gmail asks to load the light version if the bandwidth is low.i bet most of us have experienced ).
follow KISS principle. means keep ur desgin simple.
And never try to render the whole page using javascript in any case, you cannot predict all your endusers using the high config systems or high bandwidth systems.
Its smart to split the workload between your server and client.
If you think in the future you might want to create an API for your application (communicating with iPhone or android apps, letting other sites integrate with yours,) your would have to duplicate a bunch of code for all those devices if you go with a bare-bones server implementation of your application.