Practical limits on array size - javascript

Note: I'm really interested in "large" arrays sizes that work for other people. This is not the first time I've wondered about this and welcome feedback.
The scheme that I detail below requires development-effort towards that goal, so it's better to know going in whether I should eat the bandwidth and drop the idea, or whether this is a practical method for marginal gain on bandwidth. In other cases, for others who may be curious, or for future projects of mine, the gain may be larger.
Usually when I encounter a situation where a large javascript dataset is an option, there are usually other options and I tend to favor other options. I'm not actually sure what qualifies as large or too-large.
I'm wondering what safely works in production. I know that the theoretical limits but I can't find anything on the practical limits. As I detail below, my array is 14000 elemnts and 356k in size.
I realize, of course, that javascript is client side and depends on the specs of the client machine, my code, and to some extent, the version of the browser (in cases where related internal performance improved between versions).
Like any respectable modern website, it will have a mobile version as well and that's honestly where the memory concerns come in.
I don't actually think that this size is too large or near it for reasonable specs expectations of client machines, but I could be wrong and this is something I've long wondered about and haven't found any good information on. (Further, of course, the answer to this question would be quite different than it was N years ago).
Edit: I am aware of the 32-bit limitations on Array size, but I'm asking about what can work in healthy production and perform well.
Background:
I'm in the planning stages of a bible search engine.
I have a copy of the King James Version of the Bible filled with citations. Some of the citations are quite lengthy and as I'll be serving this over the net, I decided to rewrite the citations to an indexed array of the citations. That way I can rewrite the citations to shorter codes to shorter text and consume less bandwidth.
This leaves me with an array with 14,000 elements 356k in size, (216k of data) and the overall size of all the verses has shrunk by 20% (2mb, average of 65 bytes per verse). I would call this a gain.
A preview of the file looks like
['s=h07225','s=h0430','s=h0853','m=sm:th8804']
However, the potential downside is the memory consumed by using this array consistently with search results.

My recommendation is to use localStorage, this way you can AJAX the data in so it doesn't impact initial page load. Once cached on the client, you are free to use it as you please. Also, you might need to add a way to keep checking for your array. You can check to see if it exists, and AJAX it in again if it doesn't. Local storage can be removed by the browser at any point in time.
Another alternative is to put in a JavaScript web resource that gets cached in the browser. You use <script> tags to bring it in. This file will declare the global array in your application. There is an async attribute available to have it load asynchronously. The downside here is figuring out when your array becomes available.

Related

Should I pre-allocate an array or grow as I go

I have been trying to understand javascript better over the last few months. This has definitely made me want to get a Computer Science Degree. Anyways, I have sadly run into two related but conflicting points in relation to JavaScript.
According to this article, one should not prefill an array, but instead grow it when need be.
That sounded great and all, until I run into another article on wikipedia that stated that doing the above would be slow.
I am thinking of putting together some games from scratch, and being the code vegan that I am, plan on putting performance at the for front of my efforts. So is it better to have an array that grows or one that is pre-allocated? In HTML5 game development it is advised to use things such as object pools, which I tend to create using an array.
Rough guidelines:
One that is pre-allocated is better. If you grow one with for instance push or pop the JS engine needs to do a lot of additinal steps.
Even using an oversized array is way better than changing the size often. And you should operate on fixed size arrays whenever you can.
Here you can find more information regarding this.
Aray performance is Highly based on JS engine implementation:
Because javascript is a specification not an implementatation different browsers have different versions of the Javascript engine. These versions are updated regularly to enhance speed and take different approaches on how to optimize this.
When optimizing there are often tradeoffs between the speed of certain features including how arrays are manipulated/created. This means that the guidelines that I provided above might not be 100% accurate because one JS engine can have certain behaviour which the other JS engine lacks. This can cause differences in speed of manipulation/creation techniques on arrays.

Array sorting in Front-end or Back-end [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I am implementing a RESTful API that returns an array. I want to know if it will be more efficient to sort the array in descending order in my backend code or in the javascript?
Your API will be used by n clients. Performance-wise it would make sense to have each client do the sorting on their own instead of having the server do it for all the n clients. Simply, less CPU work for the server.
Furthermore, whether the result needs to be sorted or not depends on the nature of the application using the data. Let the application decide that. Some interfaces allow the user to decide what to sort by, thereby convenient to do it locally (without waiting for a background HTTP call).
However I would not overthink the performance part before you actually have a performance problem. It could also be that the data sorting is not really costly or the sorting has already been done depending on how information is kept internally (in DBMS-s, for example).
Edit
With up to 20 rows without sorting, it really makes no important difference - make the API implementing developers' life easier and do the small sorting on the frontend side.
I would actually try to hand it off to the database. Write the query where the array returned from the DB is already sorted. The most computing power and most efficient sorting is probably on the DB.
This goes to sorting, filtering in general IMO.
Depends on from where data come, how much of it do you have, and what goals you want to achieve.
Frontend based solution is cost less server CPU but could become terrible user experience. Imagine an array of 100000 features sorted in IE installed on old PC? It could hang the browser.
So if you have hot much data to process or CPU economy on server is important for you use frontend, otherwise backend.
A RESTful API is built for the purpose of a developer using it. The values it returns, the errors it presents, even the headers are all part of the "user experience" a developer has when using your API.
Now you need to weigh that against performance concerns. Is it potentially lots of data? Can you limit that data through paging, etc.? Doing a long sort operation on your server can, in unchecked situations, provide a severely degraded experience for not only that user but also others.
That said, a server typically has a lot more power for sorting and even (using the right language/algorithm) could provide a multithreaded approach to the problem. So if you're not worried about performance impacts server side, in almost all cases I would do it on the server.
Think about it this way .. Do you want the server to do the work, or the individual's browser ... So efficiency comes down to THEIR system hardware and software .. Anytime you can not "pawn off" the load to the browser, it's going to be slightly more overhead on the server .. However, there are less questions to be had -- The short answer is nobody knows, since you can't know all aspects of your clients system.
With that said, it's just array parsing. With larger (by large, I mean HUGE) datasets ... What you're asking might be slightly noticeable between server-side and client side processing of the array. But if it's not "HUGE" -- It is most likely to be six one way half dozen the other from a performance stand point.
Best practice, in my opinion, is to keep the code where you can control the overhead. Server-side.
That depends on your needs. If you have small amount of data, you can do all in FE, so user will have not to load the same array for different sortings.
With a lot of data and pagination it's better to do it on back-end, of course.

Why write modular javascript? [closed]

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
The benefits of well-factored and modular code, in my understanding are re-usability and organization. Code written in a big chunk all in one file is difficult to read, and re-using small portions of the code requires careful copy-pasting, rather than include statements.
In particular, with regards to Javascript, I came across an example recently that got me thinking about this. A comment was made on SO to the effect that, if you are not including your javascripts conditionally on a page-by-page basis, this "represents a failure to modularize JS code properly". However, from a code re-use and organization point of view, there is no reason to consider what happens at page load time. The code will be just as readable if it is written in a bunch of separate files and then mashed together and minified before being served. The rails asset pipeline, for example, does just this.
When I first encountered the asset pipeline, my mind reeled and I started wondering "how do I make javascripts load only when needed?" I read a few SO questions and an article on the matter, and began to think that maybe I shouldn't worry about what happens to my code after it "compiles".
Is the purpose of writing modular code purely a human-level activity, should we stop worrying about modularity after the code starts running? In the case of Javascript, should we be concerned that our scripts are being mashed together before being included?
I think the one thing that you are not really talking about in this with regards to performance is actual HTML browser download behavior. I believe you have to walk a fine line between only displaying the javascript needed on a page by page basis and leveraging browser caching and download behavior.
For example, say you have 20 different javascript snippets that are going to be used on every page. In this case it is a no-brainer to compile/minify them into a single file, as the fewer files your browser needs to download, the better. This single file would also be able to be cached, that is assuming it is a static file or appearing to be static (via headers sent) if it is dynamically compiled.
Now say of those 20 snippets, 15 are used on every page and the others are used intermittently. Of course you put all 15 of the always used snippets into a single file. But what about the others? In my opinion you need to consider the size and frequency of use of the files. If they are small and used relatively frequently, I might consider putting them into the main file, with the thought that the extra size in the main file is outweighed by the need to have additional request to download the content later. If the code is large, I would tend to only use it where necessary. Of course once it is used, it should remain in cache.
This approach might best be suited for a web application where users are expect to typically have multiple page loads per session. Of course if you are designing an advertising landing pages or seomthing where the user only may see that single page, you might lean on keeping the initial javasciprt download as small as possible and only loading new javascript in as necessary based on user interaction.
Every aspect of this question boils down to "it depends".
Are you writing an enterprise-level application, which results in 80,000 lines of code, when you stuff it all together?
If so, then yes, compilation time is going to be huge, and if you stuff that in the <head> of your document, people are going to feel the wait time.
Even if it's already cached, compile time alone will be palpable.
Are you writing dozens of widgets which might never be seen by an end-user?
Especially on mobile?
If so, then you might want to save them the download/compile time, and instead load your core functionality, and then load extra functionality on-demand, as more studies are showing that the non-technical end-user expects their mobile-internet experience to be similar to their desktop experience, not only in terms of content, but in general wait-times.
Fewer and fewer people are willing to accept 5s-8s for a mobile experience (or a desktop experience on mobile) to get to the point of interactivity, just based on the "well, it's mobile, so it'll take longer" train of thought.
So again, if you've got an 80,000 line application, or a 300kB JS file, or are doing a whole lot of XML parsing, et cetera, prior to load, without offering a separate mobile experience, your stats on mobile are bound to hurt -- especially if you're a media site or a commercial/retail site.
So the correct answer to your question is to say that there is no correct answer to your question, excepting that there are good ideas and bad ideas, based on the target-devices, the intent of the site/application, the demographic, the code-base, the anticipation that users will frequent the site (and thus benefit from cached assets), the frequency of updates to the codebase (having one updated module, with 20 cached modules, versus a fully-invalid 21-module chunk, due to one updated line, with a client-base of 250,000 customers, is a consideration for several reasons)...
...and more...
Figure out what you're doing.
Figure out what you need to do to make that happen.
Figure out how to do it, while providing your customers a good experience.
Know how to combine files.
Know how to load on demand.
Know how to build a light bootstrap, which can intelligently load modules (and/or learn require/AMD).
Use these as tools to offer your users the best experience possible, given what you're trying to accomplish.

Javascript Coding - Good Practices/Standards [closed]

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I've been working on a project (Javascript with some AJAX, JQuery, etc) and my project has gotten rather large; I'm finding myself using more and more clientside Javascript arrays, and am now considering a 2D Javascript array as well.
I know from incremental testing that my current implementation is very much manageable in terms of browser resources and performance, but was wondering if there was a general consensus as to how "heavy" a website could be in terms of Javascript memory usage and processing before it could be declared "bloated."
Also, if I'm already treading into dangerous territory by using this much storage and processing (I feel like I'm coding a rather substantial Java/C app), what is the best way to lighten the load?
Thanks!
One man's "bloated" is another man's "robust". In other words, there is no "right" answer to this question, as it depends heavily on your audience and your site. For some sites, every millisecond of load time matters, while for others a 20 second load time is perfectly acceptable. It really just depends.
The best advice that I can suggest is to use one of the many site performance analysis tools (eg. YSlow) to get a sense of just how slow your app actually is. These tools can also give you a better idea of what is making your site slow; for instance, you might think it's the amount of JS code you have, but really the number of JS files (every HTTP request has a cost) might be the bigger factor.
Once you have some objective, metricable sense of how slow your site is, you can then take the time to consider your audience and determine how slow is "too slow" for them. And once you've done that, you can then consider all of the different suggestions that YSlow (or whatever tool you choose) offers, and pick which ones (if any) make the most sense for your site.
you have a big problem that the internet connection speed is varied and that peoples hardware is varied
OVERVIEW
with my company we run on a 3 second system, if they page is not working in 3 seconds of it starting to download we have a problem but this could be server connection speed issue, images are to big Javascript is quite quick most of the time but if you worried about memory and speed with javascript trying to keep your script tidy helps,
TIDY
By tidy i mean if you have one giant js file will all your functions in or every thing in one $(document).ready() when its only used on 1 or 2 of your pages try splitting it up a little, but don't split it up too much as connecting to a server to download more files is a lot longer than that of running it i have script about half a MB that run fine.
TESTING
A good way to test that i use is to have a Windows install in Virtual Box and reduce the memory to the common amount users would have 256MB (really old), 512(old), 1024(a lot of users), 2GB+(high end users) and remember your target if your building a JS heavy site do you want to support IE6 if not then your main target is windows vista or newer and as such a minimum of 512MB of Ram is required just remember to test in the different browser FF(3.x.x, 4/5), IE(7/8/9) and chrome because different browser use different amounts of memory
Main Causes of Bounce Rate,
Page takes to long to load... Images are the worst cause of this
Page Crashes Browser... Javascript is looping when a certain thing happens people wont come back if there browser crashes
There will be people that always complain about JavaScript usage cater for the masses not the unique
UPDATE
Another good way to keep script small is to use this, also what google use for there min Script, it will drop file size (so download time) and will lower memory a little as variables name will be using less space in memory
http://code.google.com/closure/compiler/
One thing you can do if you've got a lot of data is to take advantage of localstorage in browsers that support it.
Other than that I can't think of a clear definition of bloat.. Does your website load quickly on modest connections? How does it do on slower browsers? If it does well in those scenarios then you're good to go.

How much external data is too much? (XML or JSON)

I have written pure JavaScript front ends before and started noticing performance decrease when working with large stores of data. I have tried using xml and json, but in both cases, it was a lot for the browser to handle.
That poses my question, which is how much is too much?
You can't know, not exactly and not always. You can make a good guess.
It depends on the browser, OS, RAM, CPU, what else is running at that moment, how fast their connection is, what else they're transferring, etc.
Figure out several situations you expect for your average user, and test those. Add for various best, worst, and interesting (e.g. mobile, tablet) cases.
You can, of course, apply experience and extrapolate from your specific cases, and the answer will change for the future.
But don't fall into the trap of "it works for me!"
I commonly see this with screen resolutions: as those have increased, it's much more popular to have multiple windows visible at the same time. In 1995 it was rare for me to not have something maximized; now fifteen years later, it's exactly the opposite.
Yet sometimes people will design some software or a website, use lower contrast[1], maximize it, and connect to a server on localhost—and that's the only evaluation they do.
[1] Because they know what the text says and don't need to read it themselves, so lower contrast looks aesthetically better.
In my opinion, if you need to stop and think about this issue, then the data is too much. In general you should design your applications so that users with a low-end netbooks and/or slow internet connections are still able to run them. Also keep in my mind that more often than not your application isn't the only page your users are visiting at the same time.
My recommendation is to use Firefox with Firebug to do some measurements. See how long a request takes to complete in a modest configuration. If it takes noticeable time for the browser to render data, then you'd better off doing a redesign.
A good guiding principle should be that instead of worrying about whether the browser can handle the volume of data you're sending it, worry about whether your user can handle it. It all depends on the presentation of course (i.e., a lot of data bound for a visualization tool that'll render a complex graph in a canvas is different than a lot of raw numbers bound for a gigantic table), but in my experience a user's brain reaches data overload before the browser/network/client computer.
It really depends on the form that your external data is going to take in your Javascript. If you want to load all your data at once and keep it in memory as a large object with lots of properties (associative array), then you will find that most current desktops can only handle about 100k entries (with small key-value pairs) before performance really degrades.
If it is possible, you should see if there are ways to only load the data that is needed by the user for a given request / interaction. You can use AJAX to request needed data and prefetch data that you think the user may need.

Categories

Resources