So i'm seeking for a couple of questions to be answered. I am using a api which returns a list of products (15000+) how ever they use pagination so it only returns 20 per page.
I would like to be able to show all of this data on my shop so users can search through it etc... however, issue... it takes A LONG time to loop through it etc etc.
Is there a good method to do this? Shall I just loop through all the data and allow it to be added into an array once loaded? Is there something "special" we can do with the pagination?
I am new to this, and just seeking advice on the above.
Kind Regards,
Josh
There are a few thoughts that strike me straight away so let's cover those first:
From a pure UX perspective, it's very VERY unlikely that any user will ever need or click through 15k+ rows of whatever. So loading them all doesn't serve your user even if you could figure out how to do this in a efficient way.
Instead, look at what serves your users which likely in this case is some sort of filtering or search options. I would look into if your API has any support for things like categories (that should be a set smaller than 1 request maybe) which is much easier to display to get the user to cut down a lot of the data set. Then also look into if they offer some sort of query or search filter, maybe the names of whatever is being displayed. This further lets your users zoom down to a dataset that is manageable (roughly 100 items max). From there, 20 items per page is just 5 pages. Still though, you should only really load 1 page at a time and focus on better ways to offer SORTING, if you can find what the user needs on the first page, you don't need to load those 4 other pages. Hope that gives you some ideas of what to look for inside your API, or what to add if you can add it yourself.
If not, perhaps it would be worth considering loading the data into your own database and set up some background/nightly task that fetches any updates from the API and stores them. Then you build your own API around your own database that has functionality for filtering/searching.
A final option is indeed to simply ask for the first page, and then display that while you wait for the 2nd page to load. But this risks making an awful amount of wasted requests which wastes not only your users bandwidth but also puts pressure on the API for what is likely going to be wasted work. So there are a few other UX ideas around this as well, like infinite scrolling. Load the first 1 or 2 pages then stop, until the users scrolls past the first page and a half, then request page 3 etc. This way you only load pages as the user scrolls but it's a bit more fluid than pagination with numbering. Still, you'd likely want to offer some way to sort this set so that it becomes more likely that they'll find what they need in the first few "pages".
I have a use case where there are thousands of items on a list shown to the user. They are loaded a small batch at a time and I see the network traffic going in and out, I see data getting loaded. I see the DOM getting bigger, but the list itself in the UI stops updating (Chrome).
When I examine it, I see thousands of items in the code, when I select the items through console and make it count them, I see the proper number. But in the page itself, I don't see these items get displayed. The list uses drag-and-drop to put items from it into another list (and load additional data about them).
Not using jquery.datatables at the moment, but been meaning to migrate to them a long time ago. I can’t find a good example, though, everything I see uses pagination to split, but what if this is not an option?
How can I pinpoint what it is that is preventing the items from display? The number of entries will vary between 500 and 20 000.
Never mind. everything works as intended, duh. I was stupid and missed something very obvious: things had "display: none" for a very good reason about which I totally forgot (has to do with the core logic of the application). Next time hit me with a stick so I could remember to pay more attention.
Not sure what you meant by saying 'DOM getting bigger' but 'don't see items get displayed'.
Typically JS has a main thread which will handle functions/callbacks as well as view-refresh. So if you operation is blocking , the view will not be refreshing.
As for the pagination is not an option thing, you can consider using DOM-lazy-Loading mechanism where you only put what should be in the current viewport into the dom. As user scroll, you calculate the scroll height dynamically to add/remove items to/from the DOM. One thing to remember is you typically need to define a fixed height for your rows so that you could do the calculation. This lazy-loading way is a common way of solving this type of problem and is widely used by different frameworks like GXT, angular-gird..etc.
I am using ajax call to get json data.The data contains array of say 10000 size.
data.LocationQuality.forEach(function(datum){
}
I am using this loop to go through all the element and create tr with filling all the td's using datum.
Problem I am facing is that because of the large amount of data.The loop is taking lots of time and ultimately the browser shows not responding and I have to kill the page.
How to solve this problem?
There are a few things to be said here.
Firstly, as mentioned, pagination would definitely be the best solution for this. Keeping 10000 rows in memory is not a problem for browsers nowadays, but displaying 10000 rows in a table often does take ages. Only displaying 20 rows at a time for example, is much slower.
A solution like jqGrid does not require ajax (but does support it). You can load you data once and the grid will keep the data in memory and only displays a number of items at once, with navigational buttons to flip through pages.
You can get it here: http://www.trirand.com/blog/?page_id=6
Secondly, something should be said about the dom. I'm expecting that you are currently adding a row to the dom 10000 times. This is much much slower than building the table in memory and then adding the one table with 10000 rows in one step. It will still be slow, but not as slow as hitting the dom so many times.
So rather than
$.each(data, function (index, row) {
$('#yourTable').append('your row definition'); // hitting the dom for each row
}
do this instead
var table = $('<table></table>');
$.each(data, function (index, row) {
table.append('your row definition'); // adding it to the table in memory
}
$('#yourTable').replace(table); // hit the dom once
Without code this is a stab in the dark:
1: optimise your page creation loop. DOM operations are very slow - you might be able to improve things by doing them in a different way
2: download your data bit by bit. As the user scrolls your page make another AJAX call to get the next portion of the data.
3: download all the data, but then render only a page at a time.
You also might want to look at this: https://developer.mozilla.org/en-US/docs/Web/API/document.createDocumentFragment?redirectlocale=en-US&redirectslug=DOM%2Fdocument.createDocumentFragment
It allows you to create large parts of the DOM without adding them yet. That way you can build your table and append it to the DOM in one go, should be a lot faster then just appending rows all the time
You have problem displaying large number of data, and you can't get data in multiple queries?
So, get all data in one call, parse them and place in array, and do pagination with that "local" array.
Instead of requesting data from server to do pagination, just get data from "local" array.
I would like to display to user a table which contains ~500,000 rows.
The table should be calculated based on a file (i.e. the table is known at the beginning, and does not change).
I guess that building the HTML text of this table is not a good idea in terms of memory performance.
So how can I build such table ?
I would like user to be able to scroll the table using a vertical scroll bar. Is it possible to build on the fly only the visible part of the table ? I'm afraid to see delays because of this.
Is it a better idea to use server side programming rather than Javascript ?
I'm not restricted to any server side programming language, so any advise would be appreciated !
Send the first 250-ish rows to the user, as he scrolls down, perhaps past row 200, fetch the next 250 rows, append to the table and so on.
This is a (ui) design pattern known as "Infinite scroll".
Displaying 500,000 rows all at once to a user is a bad idea. Consider other options:
allow user to download file as CSV
show paginated (still not very useful)
provide filtering mechanisms (by date etc.) to allow the user to only see the data they need
If the users really needs to see all that data at once, then viewing it in the browser is one of the worst ways to do that - they should be using a tool made specifically for viewing data, like Tableau.
Classic example of where to use ajax.
Use javascript in combination with a server side script.
This really is a case for server-side pagination, I would say, maybe in combination with Ajax. A HTML table with 500.000 rows is going to crash a lot of browsers.
You're not specifying which server side technology you work with. If you update your question, I'm sure people will be able to give you some pointers.
Try this:
Clusterize.js
Tiny plugin to display large data sets easily
https://clusterize.js.org/
Is it possible to build on the fly only the visible part of the table ?
If you build a "fake" scroll (e.g. jquery slider) you can retrieve parts of the table instead of the whole.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I need to present a large number of rows of data (ie. millions of rows) to the user in a grid using JavaScript.
The user shouldn't see pages or view only finite amounts of data at a time.
Rather, it should appear that all of the data are available.
Instead of downloading the data all at once, small chunks are downloaded as the user comes to them (ie. by scrolling through the grid).
The rows will not be edited through this front end, so read-only grids are acceptable.
What data grids, written in JavaScript, exist for this kind of seamless paging?
(Disclaimer: I am the author of SlickGrid)
UPDATE
This has now been implemented in SlickGrid.
Please see http://github.com/mleibman/SlickGrid/issues#issue/22 for an ongoing discussion on making SlickGrid work with larger numbers of rows.
The problem is that SlickGrid does not virtualize the scrollbar itself - the scrollable area's height is set to the total height of all the rows. The rows are still being added and removed as the user is scrolling, but the scrolling itself is done by the browser. That allows it to be very fast yet smooth (onscroll events are notoriously slow). The caveat is that there are bugs/limits in the browsers' CSS engines that limit the potential height of an element. For IE, that happens to be 0x123456 or 1193046 pixels. For other browsers it is higher.
There is an experimental workaround in the "largenum-fix" branch that raises that limit significantly by populating the scrollable area with "pages" set to 1M pixels height and then using relative positioning within those pages. Since the height limit in the CSS engine seems to be different and significantly lower than in the actual layout engine, this gives us a much higher upper limit.
I am still looking for a way to get to unlimited number of rows without giving up the performance edge that SlickGrid currently holds over other implementations.
Rudiger, can you elaborate on how you solved this?
https://github.com/mleibman/SlickGrid/wiki
"SlickGrid utilizes virtual rendering to enable you to easily work with hundreds of thousands of items without any drop in performance. In fact, there is no difference in performance between working with a grid with 10 rows versus a 100’000 rows."
Some highlights:
Adaptive virtual scrolling (handle hundreds of thousands of rows)
Extremely fast rendering speed
Background post-rendering for richer cells
Configurable & customizable
Full keyboard navigation
Column resize/reorder/show/hide
Column autosizing & force-fit
Pluggable cell formatters & editors
Support for editing and creating new rows."
by mleibman
It's free (MIT license).
It uses jQuery.
The best Grids in my opinion are below:
Flexigrid: http://flexigrid.info/
jQuery Grid: http://www.trirand.com/blog/
jqGridView: http://plugins.jquery.com/project/jqGridView
jqxGrid: https://www.jqwidgets.com/
Ingrid: http://reconstrukt.com/ingrid/
SlickGrid http://github.com/mleibman/SlickGrid
DataTables http://www.datatables.net/index
ShieldUI http://demos.shieldui.com/web/grid-virtualization/performance-1mil-rows
Smart.Grid https://www.htmlelements.com/demos/grid/overview/
My best 3 options are jqGrid, jqxGrid and DataTables. They can work with thousands of rows and support virtualization.
I don't mean to start a flame war, but assuming your researchers are human, you don't know them as well as you think. Just because they have petabytes of data doesn't make them capable of viewing even millions of records in any meaningful way. They might say they want to see millions of records, but that's just silly. Have your smartest researchers do some basic math: Assume they spend 1 second viewing each record. At that rate, it will take 1000000 seconds, which works out to more than six weeks (of 40 hour work-weeks with no breaks for food or lavatory).
Do they (or you) seriously think one person (the one looking at the grid) can muster that kind of concentration? Are they really getting much done in that 1 second, or are they (more likely) filtering out the stuff the don't want? I suspect that after viewing a "reasonably-sized" subset, they could describe a filter to you that would automatically filter out those records.
As paxdiablo and Sleeper Smith and Lasse V Karlsen also implied, you (and they) have not thought through the requirements. On the up side, now that you've found SlickGrid, I'm sure the need for those filters became immediately obvious.
I can say with pretty good certainty that you seriously do not need to show millions of rows of data to the user.
There is no user in the world that will be able to comprehend or manage that data set so even if you technically manage to pull it off, you won't solve any known problem for that user.
Instead I would focus on why the user wants to see the data. The user does not want to see the data just to see the data, there is usually a question being asked. If you focus on answering those questions instead, then you would be much closer to something that solves an actual problem.
I recommend the Ext JS Grid with the Buffered View feature.
http://www.extjs.com/deploy/dev/examples/grid/buffer.html
(Disclaimer: I am the author of w2ui)
I have recently written an article on how to implement JavaScript grid with 1 million records (http://w2ui.com/web/blog/7/JavaScript-Grid-with-One-Million-Records). I discovered that ultimately there are 3 restrictions that prevent from taking it highter:
Height of the div has a limit (can be overcome by virtual scrolling)
Operations such as sort and search start being slow after 1 million records or so
RAM is limited because data is stored in JavaScript array
I have tested the grid with 1 million records (except IE) and it performs well. See article for demos and examples.
dojox.grid.DataGrid offers a JS abstraction for data so you can hook it up to various backends with provided dojo.data stores or write your own. You'll obviously need one that supports random access for this many records. DataGrid also provides full accessibility.
Edit so here's a link to Matthew Russell's article that should provide the example you need, viewing millions of records with dojox.grid. Note that it uses the old version of the grid, but the concepts are the same, there were just some incompatible API improvements.
Oh, and it's totally free open source.
I used jQuery Grid Plugin, it was nice.
Demos
Here are a couple of optimizations you can apply you speed up things. Just thinking out loud.
Since the number of rows can be in the millions, you will want a caching system just for the JSON data from the server. I can't imagine anybody wanting to download all X million items, but if they did, it would be a problem. This little test on Chrome for an array on 20M+ integers crashes on my machine constantly.
var data = [];
for(var i = 0; i < 20000000; i++) {
data.push(i);
}
console.log(data.length);
You could use LRU or some other caching algorithm and have an upper bound on how much data you're willing to cache.
For the table cells themselves, I think constructing/destroying DOM nodes can be expensive. Instead, you could just pre-define X number of cells, and whenever the user scrolls to a new position, inject the JSON data into these cells. The scrollbar would virtually have no direct relationship to how much space (height) is required to represent the entire dataset. You could arbitrarily set the table container's height, say 5000px, and map that to the total number of rows. For example, if the containers height is 5000px and there are a total of 10M rows, then the starting row ≈ (scroll.top/5000) * 10M where scroll.top represents the scroll distance from the top of the container. Small demo here.
To detect when to request more data, ideally an object should act as a mediator that listens to scroll events. This object keeps track of how fast the user is scrolling, and when it looks like the user is slowing down or has completely stopped, makes a data request for the corresponding rows. Retrieving data in this fashion means your data is going to be fragmented, so the cache should be designed with that in mind.
Also the browser limits on maximum outgoing connections can play an important part. A user may scroll to a certain position which will fire an AJAX request, but before that finishes the user can scroll to some other portion. If the server is not responsive enough the requests would get queued up and the application will look unresponsive. You could use a request manager through which all requests are routed, and it can cancel pending requests to make space.
I know it's an old question but still.. There is also dhtmlxGrid that can handle millions of rows. There is a demo with 50,000 rows but the number of rows that can be loaded/processed in grid is unlimited.
Disclaimer: I'm from DHTMLX team.
I suggest you read this
http://www.sitepen.com/blog/2008/11/21/effective-use-of-jsonreststore-referencing-lazy-loading-and-more/
Disclaimer: i heavily use YUI DataTable without no headache for a long time. It is powerful and stable. For your needs, you can use a ScrollingDataTable wich suports
x-scrolling
y-scrolling
xy-scrolling
A powerful Event mechanism
For what you need, i think you want is a tableScrollEvent. Its API says
Fired when a fixed scrolling DataTable has a scroll.
As each DataTable uses a DataSource, you can monitoring its data through tableScrollEvent along with render loop size in order to populate your ScrollingDataTable according to your needs.
Render loop size says
In cases where your DataTable needs to display the entirety of a very large set of data, the renderLoopSize config can help manage browser DOM rendering so that the UI thread does not get locked up on very large tables. Any value greater than 0 will cause the DOM rendering to be executed in setTimeout() chains that render the specified number of rows in each loop. The ideal value should be determined per implementation since there are no hard and fast rules, only general guidelines:
By default renderLoopSize is 0, so all rows are rendered in a single loop. A renderLoopSize > 0 adds overhead so use thoughtfully.
If your set of data is large enough (number of rows X number of Columns X formatting complexity) that users experience latency in the visual rendering and/or it causes the script to hang, consider setting a renderLoopSize.
A renderLoopSize under 50 probably isn't worth it. A renderLoopSize > 100 is probably better.
A data set is probably not considered large enough unless it has hundreds and hundreds of rows.
Having a renderLoopSize > 0 and < total rows does cause the table to be rendered in one loop (same as renderLoopSize = 0) but it also triggers functionality such as post-render row striping to be handled from a separate setTimeout thread.
For instance
// Render 100 rows per loop
var dt = new YAHOO.widget.DataTable(<WHICH_DIV_WILL_STORE_YOUR_DATATABLE>, <HOW YOUR_TABLE_IS STRUCTURED>, <WHERE_DOES_THE_DATA_COME_FROM>, {
renderLoopSize:100
});
<WHERE_DOES_THE_DATA_COME_FROM> is just a single DataSource. It can be a JSON, JSFunction, XML and even a single HTML element
Here you can see a Simple tutorial, provided by me. Be aware no other DATA_TABLE pluglin supports single and dual click at the same time. YUI DataTable allows you. And more, you can use it even with JQuery without no headache
Some examples, you can see
List item
Feel free to question about anything else you want about YUI DataTable.
regards,
I kind of fail to see the point, for jqGrid you can use the virtual scrolling functionality:
http://www.trirand.net/aspnetmvc/grid/performancevirtualscrolling
but then again, millions of rows with filtering can be done:
http://www.trirand.net/aspnetmvc/grid/performancelinq
I really fail to see the point of "as if there are no pages" though, I mean... there is no way to display 1,000,000 rows at once in the browser - this is 10MB of HTML raw, I kind of fail to see why users would not want to see the pages.
Anyway...
best approach i could think of is by loading the chunk of data in json format for every scroll or some limit before the scrolling ends. json can be easily converted to objects and hence table rows can be constructed easily unobtrusively
I would highly recommend Open rico.
It is difficult to implement in the the beginning, but once you grab it you will never look back.
I know this question is a few years old, but jqgrid now supports virtual scrolling:
http://www.trirand.com/blog/phpjqgrid/examples/paging/scrollbar/default.php
but with pagination disabled
I suggest sigma grid, sigma grid has embed paging features which could support millions of rows. And also, you may need a remote paging to do it.
see the demo
http://www.sigmawidgets.com/products/sigma_grid2/demos/example_master_details.html
Take a look at dGrid:
https://dgrid.io/
I agree that users will NEVER, EVER need to view millions of rows of data all at once, but dGrid can display them quickly (a screenful at a time).
Don't boil the ocean to make a cup of tea.