Develop Reference

Choosing DB model for an app similar to Notion, Block-based ("paragraphs") or document-based?

1. The problem
Lately, it seems that many note managers with "infinite" tree structure are choosing a block model (where each paragraph is an entry in the DB), instead of a document or file model.
Blocks
Documents
Notion Workflowy Remnote Dynalist Roam Research
Evernote Obsidian Bear app
If you find any errors in the table, please let me know.
We have been developing an app very similar to Notion for 8 months now, also using the block model, but we are considering making a radical change and switching to the document model. The structure of our blocks in MongoDB currently looks like this:
_id: "61fd3ede7f6d2cc7a53ca669"
children: Array
0: "61fd3ee87f6d2cc7a53ca66b"
1: "61fd3ef37f6d2cc7a53ca671"
2: "61fd3ef77f6d2cc7a53ca673"
backlinks: Array
type: "bullet"
parentPage: Array
_id: "61fd3ede7f6e2ccra53ca664"
userParent: "german-jablo"
permisionParent: "edit, comment, read"
parentParagraph: "61fd3ede7f6d2cc7a53ca668"
content: "<p>This is a paragraph</p>"
isCollapsed: false
createdAt: 2022-02-04T14:57:34.280+00:00
updatedAt: 2022-02-04T14:57:59.585+00:00
Many pages talk about the differences of both approaches (example) although in a very vague way, so we decided to open this thread to find a more scientific answer to the question.
Features of our app
app
Blocks that can be opened as documents
Blocks that can collapse or expand their children
Notion
Page type blocks
Toggle type blocks
Workflowy
All
All
Evernote
Documents
None
Our app
Page type blocks
All others
Our app has two types of "blocks". The page type (which, like in Notion, can be inserted into any note and generate a document "inside" the current document), and the rest of the blocks, which are equivalent to the "toggle" block type in Notion (i.e. they can be collapsed or their nested children can be expanded).
2. What we have tried
In trying to answer our question (which DB model would work best for our application), we've realized that the answer is probably "it depends". Perhaps both models have strengths or weaknesses in different types of operations or situations. That is why we formulated this comparison table describing how we believe the performance of both models would be for each of these operations.
Operation
Blocks
Documents
Apparent Winner
Fetch the contents of a page
Find all paragraphs in the DB.
Search the document in the DB.
Document
Render the content of a page**
Build the tree from the paragraphs recursively. You can omit the children of paragraphs whose isCollapsed property is true
Render the document
Document
Update the content of a paragraph in the DB
Only the modified paragraph is rewritten
The whole document is rewritten
Block
Alternatives for rendering very large documents *
Blocks can be fetched or rendered as you scroll (as Workflowy does), or as you expand child paragraphs that were collapsed.
I thought that Grifds could achieve similar behavior, breaking the document into smaller chunks and bringing them in piecemeal, but it doesn't support updating an individual chunk, or even the entire document. It could also corrupt an HTML by splitting it into binary format.
Block
Import or paste content
In addition to converting the clipboard to HTML and/or sanitizing it, you must set up paragraphs with tree structure recursively. Note: Roam Research e.g. supports importing in JSON format, but generally users do not handle this format beforehand.
Only convert the clipboard to HTML and/or sanitize the clipboard
Document
Copy content**
Clipboard must be sanitized and/or transformed
Correct by default**
Document
Real-Time Collaboration
At the document level, could use some tree-based (Json) library like Automerge, or combine with some CRDT library for paragraph level.
Could use tinymce solution.
Tie? Both seem to have their advantages and disadvantages.
*Render very large documents: Most users probably do not use notes larger than 250 kb (considering that multimedia files are referenced in a separate collection). Still, in the document model, the question arises: how can we load, render or edit large documents in manageable chunks? One idea we came up with is to split HTML documents that reach large dimensions into portions of a certain size in kb, instead of splitting them into paragraphs. (It would be like a kind of Gridfs that allows you to modify the file in parts.) Could this be a good idea?
**Should the DOM be nested? In order to be able to collapse or expand nested child paragraphs, note managers with a block model structure the DOM in a nested way (paragraphs are in divs, inside their parent divs, etc.). However, an alternative in the document model could be that when the user presses tab, only that block (HTML tag such as <p> or <li>) is assigned an attribute with a number less than or equal to 1, representing the nesting levels relative to the previous block. This way when you press tab to nest or shift-tab to un-nest, you only have to modify one attribute of an HTML element instead of many elements; and the DOM stays simple, without having nested blocks.
3. Our conclusions
We believe that for each of the rows in the comparison table, benchmarks could be done measuring the performance of both models.
Other people have done something similar here and here, comparing the performance of note managers using both models. The problem with those tests is that it is difficult to draw an accurate conclusion about the goodness of both models. Obsidian uses documents locally, so you don't have to sync notes. Roam Research is a very new and poorly optimized app. Standard Notes encrypts notes locally. In other words, it's not always apples to apples.
And even if tests could be done, we believe that the answer may even depend on how each user uses the application. Suppose user A usually organizes his notes in long documents using paragraph nesting (to collapse or expand them). On the other hand user B usually organizes his notes by creating new documents within documents. It is likely that a block model based manager would work better for user A while a document based one would work better for B.
So, we tried to push our doubt as far as we could, but we are still not sure of the answer Which of the two models do you think would offer better performance for our app and why?
4. Update
I just found some very interesting information. It seems that both TinyMCE and CKEditor (up to version 4), the view and the model converge with the HTML being based on content-editable. However, CKEditor 5 switched to MVC [source 1], [source 2].
I've done a short test pasting a large clipboard of a few MB in TinyMCE 5, CKEditor 4 and CKEditor 5, and the latter has been a bit slower. I hope soon to be able to do more tests with other things like dragging blocks or rendering large documents.
In a GitHub thread about CKEditor 5's performance when working with large documents, one of the contributors said "It works slower than the native content-editable element obviously, but the typing experience is pretty well".

Looks like you've done your homework. Database modeling is sometimes a bit of an art rather than a science. I think that with both models, you can achieve good performance if you optimize them well. So I would recommend that you go for the one that requires less work. Since you've been working on the block model for 8 months already, that's probably the best option for you.

Love how thorough you thought about the design of your application. I just want to add some suggestions you might look:
Obsidian is using a hybrid approach. It started document-based, but now supports block links and embeds while still being super-fast. From all programs tested in my benchmark you linked above, Obsidian was the fastest.
One of the most important functions of a notetaking tool is effectively searching the probably thousands of notes. I created an amazingly simple test (the "Spaghetti Parmesan"-test) where all block-based approaches currently fail. It is about searching for two ingredients (spaghetti and parmesan) in a recipe. When both ingredients are in different blocks, all common block-based applications ultimately fail to find the recipe. You can read more about this here. I also tried to start a discussion with some of the authors on Twitter but failed to get any serious results. If you want to continue the block-based approach, you might start designing a search algorithm that can handle search terms spread about blocks (if you haven't yet). I tried to outline an algorithm in the thread linked above, but not sure if it will really work with hundreds of thousands of blocks.

You seem to be making your choice based on what is easier to model or develop.
If you intend to compete in this increasingly crowded market, you need to consider what will give you an edge, and what will be sustainable in the long run.
Two of the main issues Evernote (longtime user and former employee) faced were:
a very badly designed back-end that was very expensive to scale and made sharing and collaboration a nightmare (engineering-wise)
a too loose document model that made it very hard to add new features to the editor (real time collaboration) and made collaboration very finicky even for simple features (checkboxes would often generate so many note conflicts as to be unusable)
On the other hand, block-based tools usually have really bad web clippers, because it's very hard to go from HTML to blocks.
Solve some of these hard problems!

How can I most easily identify bottlenecks in React render performance?

I'm having an issue with identifying bottlenecks in render performance while working on a JSON viewer. With few elements, it performs well, but at a certain point it becomes annoyingly slow.
Checking the profiler, it seems that elements are rendering fast enough, but I've noticed a few issues that I'm not sure how to pursue.
Overview
The app is a JSON viewer which allows you to expand / minimize all elements at once, as well as individual elements.
Performance is fine with few elements, but seems to decrease dramatically as the number of elements increases.
When profiling both my object filter method with performance.now() as well as checking the render time in React DevTools, the figures seem okay. I could be interpreting it wrong.
I've tried using React.memo() on stateless elements (particularly the key/value which is the most frequently rendered component), but it doesn't seem to improve the performance noticeably. Admittedly, I'm not sure if I understand the reasoning enough behind memoizing React components to implement this usefully.
Implementation
Currently, my app loads data into a parent which feeds into a component that loads the JSON tree using a recursive element.
Loading JSON feed from URL changes the state of the parent component, which is filtered using a helper method that uses values entered into an input field.
Issues
There are two functionalities which reproduce a slow response time with (not so big) JSON documents:
The expand all button
The first few keypresses on a filter query
With the current implementation, both filtering and expanding all triggers a display: none change on the child elements, and the behavior leads me to believe I'm doing something inefficiently to handle this use case.
Reproduction Steps
The code is available here: https://codesandbox.io/s/react-json-view-4z348
With a production build here (not performing any better): https://csb-4z348.vercel.app/
To reproduce the issue, play around with the Expand All function (plus sign next to filter input) and some filter inputs.
Then, try loading a JSON feed with more elements (you can test on my GitHub API feed) and try filtering/expanding all. Notice the major performance hit.
What I've noticed
When logging useEffect, minimizing seems to cause ~2x as many rerenders as expanding all.
As the filter input becomes more specific, the performance (logically) improves as less elements are being rendered.
Question
While I would appreciate a nudge in the right direction for this specific case, what I'm most curious about is how best to identify what is causing these performance issues.
I've looked into windowing the output, but it's not my first choice, and I'm pretty sure I'm doing something wrong, rather than the cause being too many elements rendered.
I appreciate your time, and thank you in advance for any tips you could provide!

It seems I've answered my own question. The problem was a reconciliation issue due to using UUID as a key prop in my child components, which caused them to re-render every time the minimize state changed. From the docs:
Keys should be stable, predictable, and unique. Unstable keys (like
those produced by Math.random()) will cause many component instances
and DOM nodes to be unnecessarily recreated, which can cause
performance degradation and lost state in child components.
I'll leave the steps here for anyone else who runs into this issue.
After (too long) digging around in performance profiler, I noticed that each time I minimized or expanded the elements, each child was being mounted again. After consulting Google with a more specific query, I found this blog post and realized that I was committing this flagrant performance error.
Once I found the source of the problem, I found many other references to it.
After fixing the key prop, interaction time got ~60% faster for minimize/expand all.
Finally, I memoized some other components related to the instant filter and finally it seems to be performing as well as I would like for the time being.
Thanks to anyone who took a look at this in the meantime, and I hope it's helpful for anyone who might come across this.

PhoneGap Memory Management

For few months now I have been developing an Android app using PhoneGap 2.8 and on the javascript side I have used Backbone and jQuery as my main frameworks. As my application has grown to a reasonable size, I have started to notice a considerable memory consumption. Having read different articles that explain why PhoneGap requires considerable amount of memory even to run, I still believe that I can do some optimization to how i use memory.
In BackBone we have a Router object that maps URI-s to specific functions, which render me something called a View object. Not only I implemented my router functions to create a view and render it, but I also store globally reference to currently being displayed view. So before a new view is created, I tell the old view to make some clean up (That is done recursively since views can contain more "sub" views). Under clean up I currently tell view to undelegate his events (I trust Backbone removes the event listeners). Not much more is done currently. Once new view is rendered, global variable will reference the new view. I trust that javascript GC will release the memory, used by the old view. Alas, I dont things this is happening- the more I navigate around my app, the more memory is being used up. I am aware that there is some memory leaking going on, but I can't figure out what is it, that takes memory. One thing I suspect is that old objects are not being garbage collected properly for some reason. I suspect that once I render new html (DOM) over some container, perhaps old DOM is causing memory leaks, perhaps some event handlers are being unnecessarily stored somewhere.
What I would like to know, if there is any tools or commands or tips on how can I debug/ trace/ measure where memory is being allocated. Is there a way to access all event listeners and measure them somehow (same for DOM). Any article to smart memory efficient techniques would also be appreciated. Currently only thing that I can thing of to do, is to start recursively deleting all attributes of the objects (in the end objects as well) I am willing to destroy.
Any suggestion is very welcome!
Thank you in advance.

I faced similar issues with my first phonegap app. Few techniques we managed to apply were
*old view - view getting navigated away
Unbind all events associated with old view
Remove all nodes attached to the view from dom, to make sure event are also removed
Remove old view object, model/collection, so that there are no instances remaining on the DOM
Moreover try to use prototyping as much as possible, as functions created via prototype occupy space in RAM only once. So if the view is created/initiated again, its associated/child functions aren't going to be pushed into RAM again
Most imp, make sure 'this' pointer isn't leaking anywhere between files. One of my workplace used to get stuck after 1.5 hrs of play and after a week debugging, we came to find out that there was a leakage of this pointed between 2 files/objects/views, which created a circular referencing and make the DOM to explode.
Also you can try to use Google Chrome's profiling tool
Few useful links
http://blog.socialcast.com/javascript-memory-management/
Backbone.js Memory Management, Rising DOM Node Count

jstree: incremental loading

The JStree plugin for jQuery allows you to load data to feed a navigable tree GUI element provided by the library.
For small trees, just load them all in memory, and you're done. But for large trees, that might not be a good solution. The all-in-memory approach doesn't scale.
Think 6000 nodes (or 60,000), most of which will never be relevant to the user viewing the page. Wouldn't it then be better to load only the first level of branches and incrementally load more branches following the user's clicks along what's already displayed? It certainly would.
You'd mark the tree where it has missing branches, then you'd load the missing branches on demand, remove the mark from the tree, graft the branch onto the tree, and proceed in that manner recursively if necessary.
How do you do imcremental loading? I found a question from 2009 pertaining to the same problem, but the API appears to have changed. Does anyone have a recipe of how to proceed for the current version of the library?
Note incremental loading is not the same as incremental rendering, which is another optimization already provided by the library.

Lumi, that is how the plugin works.
Have a look at the Demo page, about half way down, at the section "PHP & mySQL demo + event order". The example uses the JSON format for transmitting data, which is the de facto standard, but the plugin also supports other formats. When you expand a parent node, an AJAX request is made to load the next level of nodes.

Ajax heavy JS apps using excessive amounts of memory over time

I seem to have some pretty large memory leaks in an app that I am working on. The app itself is not very complex. Every 15 seconds, the page requests approx 40kb of JSON from the server, and draws a table on the page using it. It is cheaper to draw the table over because the data is usually always new. I am attaching a few events to the table, approx 5 per line, 30 lines in the table. I used jQuery's .html() method to put the new html into the container and overwrite the existing. I do this specifically so that jQuery's special cleanup functions go in and attempt to detach all events on the elements in the element that it is overwriting. I then also delete the large variables of html once they are sent to the DOM using delete my_var.
I have checked for circular references and attached events that are never cleared a few times, but never REALLY dug into it. I was wondering if someone could give me a few pointers on how to optimize a very heavy app like this. I just picked up "High Performance Javascript" by Nicholas Zakas, but didn't have much time to get into it yet.
To give an idea on how much memory this is using, after 4~ hours, it is using about 420,000k on chrome, and much more on Firefox or IE.
Thanks!

I'd suggest writing a test version of your script without events. DOM / JS circular references might be very hard to spot. By eliminating some variables from the equation, you might be able to narrow down your search a bit.

I have experienced the same thing. I had a piece of code that polled every 10 seconds and retrieved a count of the current user's errors (data entry/auditing), a simple integer. This was then used to replace the text inside a div (so the user knew when new errors were found in their work). If left overnight, the browser would end up using well over 1gb of memory!
The closest I came to solving the issue was reducing the polling to occur every 2 minutes instead and insist the user closed down at the end of the day. A better solution still would have been to use Ajax Push Engine to push data to the page ONLY when an error was created. This would have resulted in data being sent less frequently and thus less memory being used.

Are you saving anything to an abject / array? I've had this happen before with a chrome plugin where an array just kept getting larger and larger. This sounds like it might be your problem, especially considering you're fetching 40k.

A snippet would be great it seems as if you are creating new variables each time and the old ones aren't going out of scope therefore not being garbage collected.
Also try and encapsulate more of your JS using constructors and instances of objects ect. When the JS is just a list of functions and all the variables have a global scope rather than being properties of an instance your JS can take up a lot memory.

Why not draw the table and attach events only once and just replace the table data every 15 second?

1) Jquery Ajax wrapper, called recurrently, leads to memory leaks and the community is aware of that (although the issue on the ajax wrapper is not by far as ugly as your case).
2) When it comes to optimization, you've done the first step(using lightweight json calls) and delete method, but that the problem is in the "event attaching" area and html method.
What I mean, is that:
1) you are probably reattaching the listener after each html() call
2) you re-draw the hole table on each ajax call.
This indeed leads to memory leaks.
You have to:
1)draw the table (with the first time content) on the server side
2)$(document).ready you attach listeners to the table's cells
3)call json service with ajax, parse response
4)refill table with the parsed array data
Tell us what you've achived meanwhile :)

I was having a similar problem just this week. It turned out I had a circular reference in my database. I had an inventory item ABC123 flagged as being replaced by XYZ321 and I also had XYZ321 flagged as being replced by ABC123. Sometimes the circular reference is not in the PHP code.