Develop Reference

Most efficient way to populate/distribute indexedDB [datastore] with 350000 records

So, I have a main indexedDB objectstore with around 30.000 records on which I have to run full text search queries. Doing this with the ydn fts plugin this generates a second objectstore with around 300.000 records. Now, as generating this 'index' datastore takes quite long I figured it had be faster to distribute the content of the index as well. This in turn generated a zip file of around 7MB which after decompressing on the client side gives more than 40MB of data. Currently I loop over this data inserting it one by one (async, so callback time is used to parse next lines) which takes around 20 minutes. As I am doing this in the background of my application through a webworker this is not entirely unacceptable, but still feels hugely inefficient. Once it has been populated the database is actually fast enough to be even used on mid to high end mobile devices, however the population time of 20 minutes to one hour on mobile devices is just crazy. Any suggestions how this could be improved? Or is the only option minimizing the number of records? (which would mean writing my own full text search... not something I would look forward to)

Your data size is quite large for mobile browser. Unless user constantly using your app, it is not worth sending all data to client. You should use server side for full text search, while catching opportunistically as illustrated by this example app. In this way, user don't have to wait for full text search indexing.
Full-text search require to index all tokens (words) except some stemming words. Stemming is activated only when lang is set to en. You should profile your app which parts is taking time. I guess browser is taking most of the time, in that case, you cannot do much optimization other than parallelization. Sending indexed data (as you described above) will not help much (but please confirm by comparing). Also Web worker will not help. I assume your app have no problem with slow respond due to indexing.
Do you have other complaint other than slow indexing time?

Website Performance Testing: How best to approximate computer performance?

I have some browser-intensive CSS and animation in my webpage and I'd like to determine if the user has a fast PC or not so i can scale things accordingly to provide the best experience.
I am using http://detectmobilebrowser.com's script to detect all mobile devices, and I am going to include the clause /android|ipad|ipod|playbook|silk/i.test(a) to include all tablet devices as well.
However this doesn't and cannot really address the actual hardware. It doesn't go very far at all to paint a picture of what I'm looking for.
An iPhone 4S, for example, will be quite a lot more capable than many of the devices matched by the mobile user agent detector, and this provides no way for it to set itself apart.
Somebody might run Google Chrome on a Pentium II machine (somehow) and want to view my page. (This person probably does not have an iPhone 4S)
Obviously to actually get an idea for this I'll have to do some actual performance testing, and as with performance testing with any kind of application, it makes sense to only test the performance of the type of tasks that the application actually performs.
Even with this in mind I feel like it would be difficult to obtain any reasonably accurate numbers before the performance testing routine will have taken too long and the user will have became impatient. So this probably means go ahead with it unless I want the first initial impression to be perfect. Well, this actually happens to be the case. So I can't get away with measuring performance "after the first run" and adjusting the parameters later.
So what I've got left is to basically try to perform a similar task on initial page load, in a way that is dependent on browser rendering and processing speed, while not presenting anything to the user (so that to the user they still think the page is loading), and then preferably within a second or two obtain accurate enough numbers to set parameters for the actual page to animate and present in a pleasing manner that doesn't resemble a slideshow.
Maybe I could place a full-page white <div> over my test case so that I can prevent the user from seeing what's going on and hope that the browser will not be smart by avoiding doing all the work.
Has anybody ever done this?
I know people are going to say, "you probably don't need to do this", or "there's gotta be a better way" or "reduce the amount of effects".
The reason for doing any of the things I'm doing on the page are so that it looks good. That's the entire point of it. If I didn't care about that as much this question wouldn't exist. The goal is to give the javascript the ability to determine enough parameters to provide an awesome experience on a powerful computer, and also a passable experience on a less capable computer. When more power is available, it should be harnessed. So hopefully that can explain why such suggestions are not valid answers to the question.

I think this is a great question because it puts the user's experience first and foremost.
Several ideas come to mind:
Microsoft has published many tests demonstrating the performance of IE 9 and 10. Many of these tests focus on graphic performance, such as this one, which appears to use this JavaScript file to measure performance. There may be some code/concepts you can use.
A media-intensive page probably takes a few seconds to load anyway, so you have a little breathing room if you begin your tests while the rest of the content loads. For example, initiate AJAX/image requests, run your tests, and then handle the responses.
To test graphic performance, what about using a loading graphic as the performance test? I'm not usually a fan of "loading" screens, but if the site may take a few seconds to load, and the end result is better UX, then it isn't a bad idea.
The white screen idea may work if you draw a bunch of white shapes on it (not sure if any engines are smart enough to optimize this away because it is the same color).
Ultimately, I would err on the side of better performance and lower fidelity, and a less accurate (but fast) test versus making the user wait for too long.

Rather than measuring the user's CPU performance once and determining how many fancy visual effects to use from that, I would measure the amount of time taken by the CPU-intensive bits every time they execute (using new Date()), compare that to expected minimum and maximum values (which you will have to determine), and dynamically adjust the "effect level" up and down as appropriate.
Say if the user starts up a program in the background which eats a lot of CPU time. If you use this idea, your page will automatically tone down the visual effects to save CPU cycles. When the background program finishes, the fancy effects will come back. I don't know if your users will like this effect (but I am sure they will like the fact that their browser stays responsive when the CPU is overloaded).

This is a poor solution but it worked at the time: I used to generate two random matrices of about 100x100, multiply them (the school boy way) 100 times and time it. It took less than 1 second on regular machines and a bit more than 2 seconds in the slowest machine I could find (EeePC 1000H). After that I could say "well, this CPU can do X floating point operations per second", which is very inaccurate and probably wrong but the results were very stable and with very low standard deviations, so I guess you can call it a poor measure of javascript mathematical performance, which can tell you something about the CPU of that computer.
You can also check if it's WebGL enabled, and it will leave out all windows operating systems older than Vista. Since those don't have the hardware to run Vista or greater, those are slower PCs. You can check it with this:
function hasWebGL () {
if (typeof window.WebGLRenderingContext !== 'undefined') {
var canvas = document.createElement('canvas');
var gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl') || canvas.getContext('webkit-3d') || canvas.getContext('moz-webgl');
if(gl) {
return(true);
} else {
return(false);
}
}
}

Do spaces/comments slow Javascript down?

I was wondering, do whitespaces and comments slow down JavaScript? I'm doing a brute force attack which takes some time (30 seconds). Removing whitespaces does not show a significant growth in speed, but I think the browser just does have to parse more.
So, is it of any use to remove unnecessary whitespaces and comments to speed the whole up?

People usually use minimizers to reduce the SIZE of the script, to improve download speed, rather than to make any difference in speed of parsing the script.
Whitespace and comments will have little effect in how long it takes a browser to execute, as the parser needs to check if it is whitespace, or a comment, but in reality this will be so minute with current computing power, it would be impossible to notice any impact.
SIZE however is still important even with the large bandwidth available in our broadband world.

Whitespaces and comments increase the size of the JavaScript file, which slows down the actual downloading of the file from the server - minification is the process of stripping unnecessary characters from a JavaScript file to make it smaller and easier to download.
However, since you mention a brute force attack, the bottleneck is probably not the download. Try using a profiler to find what slows you down.

There is always a point in minifying, combining and gzipping your assets, to ease server load.
Minifying is the act you refer to, of stripping away unnecessary whitespace and comments, to make the download speed smaller.
Combining will most likely show an even greater increase in page rendering speed; it is the act of merging all your javascript files into one, and all your css files into one (it can also be done for most images, but that taks requires some more work). This is done to reduce the amount of requests the browser has to make towards your server, to be able to display the page.
GZipping is the act of further compressing the data, in a zipped format, to the browsers that indicate that they'll accept such data. This further reduces size, but adds some extra work load at both ends. You're likely to see a net gain from it.
Depending on what environment you're working in, there are different components that'll help you with this, that usually covers all of the above in one go.
The time your code takes to download from the server has a direct effect on how long the page takes to render. JavaScript is blocking, meaning that a JS block will prevent any furhter rendering, until the block has executed entirely. As such, where you put your javascript files (i.e. in which point in the rendering process they'll be requested), how many requests it takes for it to be completely downloaded, and how much data there is to download, will have an impact on your page load, as it appears to the user.
Once the browser has parsed your code, be it javascript, css or html, it'll have created internal representations of the part it needs to keep remembering, and the actual formatting will no longer affect it.

I don't think whitespace in js-code slows down the execution of it. As far as I understand a javascript interpreter strips all comments and redundant whitespace before processing. It can influence download time en thus loading time of a web page however.
Take a look here for a bit of extra information.

It has little to no impact on actual processing speed, however...
Smaller size => less bandwith => less costs => ??? => profit!

How can I estimate browser's Javascript capabilities?

I serve a web page which makes the client do quite a lot of Javascript work as soon as it hits. The amount of work is proportional to the amount of content, which varies a lot.
In cases where there is a huge amount of content, the work can take so long that clients will issue their users with one of those "unresponsive script - do you want to cancel it?" messages. In cases with hardly any content, the work is over in the blink of an eye.
I have included a feature where, in cases where the content is larger than some value X, I include a "this may take a while" message to the user which is displayed before the hard work starts.
The trouble is choosing a good value for X since, for this particular page, Chrome is so very much faster than Firefox which is faster than IE. I'd like to warn all users when appropriate, but avoid putting the message up when it's only going to be there for 100ms since this is distracting. In other words, I'd like the value for X to also depend on the browser's Javascript capabilities.
So does anyone have a good way of figuring out a browser's capabilities? I'm currently considering just explicitly going off what the browser is, but that seems hacky, and there are other factors involved I guess.

If the data is relatively homogeneous, one method might be to have a helper function that checks how long a particular subset of the data has taken to go through, and make a conservative estimate of how long the entire set will take.
From there, decide whether to display the message or not.

This may not be where you want to go, but do you have a good idea why the javascript can take so long? Is it downloading a bunch of content over the wire or is the actual formatting/churning on the browser the slow part?
You might even be able to do something incrementally so that while the whole shebang takes a long time but users see content 'build' and thus don't have to be warned.

Why not just let the user decide what X is? (e.g. like those "display 10 | 20 | 50 | 100" per page choosers) Then you don't have to do any measurement/guesswork at all; you can let them make the optimal latency / information content tradeoff.

This is somewhat misleading; usually when one discusses a browser's JS capabilities, it's referring to the actual abilities of the browser, such as does it support native XMLHTTP? Does it support ActiveX? etc.
Regardless, there is no way to reliably deduce the processing power or speed of a browser. One might think that you could run some simple stress-tests, compute the result and compare to a list of past performances to see where the current user's browser ranks, and possibly use this information to arrive at an estimated time. The problem here, is that these calculations can not only be influenced by activities in the browser (or merely on the OS); for instance, you run your profiling script, and the user's AV scanner starts up because its 5pm; what normally might take 2s, takes 20s.
On thing to ask yourself, is: Does this processing have to take place right NOW? As n8wrl and Beska alluded to, you might need to code your own method whereby you break-up the work to be done into chunks and then you operate on them one at a time and using something like setTimeout(). This will give the engine time to 'breathe' -- and thus hopefully avoid the 'unresponsive script' warnings. Each of these chunks could also be used to update a progress bar (or similar) that gives the user some indication that work is being done.
Or you could take the approach like GMail - they flash a very small, red "Loading..." text area in the corner of the window. Sometimes its there for a few seconds, sometimes it's not there long enough to read it. Other times it blinks on-and-off several times. But you know when its doing something.
Lastly, also on the point of incrementally 'building' the page, you could inspect the source of Chrome's new tab page. Note: you can't view this using "view source"; instead, choose the "javascript console" option (while on the new tab page) and then look at the HTML source there. There should be a comment that explains their general strategy, like such:
<!-- This page is optimized for perceived performance. Our enemies are the time
taken for the backend to generate our data, and the time taken to parse
and render the starting HTML/CSS content of the page. This page is
designed to let Chrome do both of those things in parallel.
1. Defines temporary content callback functions
2. Fires off requests for content (these can come back 20-150ms later)
3. Defines basic functions (handlers)
4. Renders a fast-parse hard-coded version of itself (this can take 20-50ms)
5. Defines the full content-rendering functions
If the requests for content come back before the content-rendering functions
are defined, the data is held until those functions are defined. -->
Not sure if that helps, but I think it does give insight into how some of the big players handle challenges such as this.

What is considered a fast or slow load w/ respect to a web page

I just built a web page that is employing several different javascript elements. I am just curious as to what is considered a fast vs. a slow load time. Mine is coming out to be about 490ms w/ four different javascript pieces. Is that good, bad or average? Wondering if I need to optimize my js elements or not.

There's a paper from 1991 called “The information visualizer, an information workspace”, by Card et all (Xerox Palo Alto Research Center) that says:
0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.
1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data.
10 seconds is about the limit for keeping the user's attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.
A detailed overview can be found here.
Bottom line: if we want a fluid user experience, our apps need to operate at 10+ 'fps' :)

Just run your site through this tool and you'll know (almost) all there is to know about loading times. Or use Google's Speed Tracer if you use Chrome.

Try to think from the users point of view:
Instead of measuring the response time of xyz XHR request, consider the function the user wants to perform - "post a comment on this website"? Then measure the total time to achieve that goal. If the users work, as a whole, is too much or takes too long then they will go elsewhere.
eg. I can have 10 XHR functions that return in 15ms each it might seem blazing fast. But if the user has to click 10 different places to post a comment, they're going to get sick of my interface pretty quick too.
(extreme example)

This is going to vary from system to system. But if it's taking less than a few seconds to load everything (completely - dom ready), that's pretty good if you ask me. I recall keeping my entire project size down to about 50kb, back before we had all of this fancy ajax :) Cherish your broadbands and asynchronous calls!

If I remember well a podcast about StackOverflow (I can't find which one it is anymore, on DNR or on Hanselminute), above 1 sec the user starts to have the time to be distracted by something else and loose focus. At 10 seconds, I would probably already have closed your page!
of course it depends on what kind of action it is. but if it's a repetitive task, 1 second is the maximum I would say. That's what I aim to in general. It feels instant when you go under that limit.

The general rule of thumb is that the maximum attention span of a web surfer is 10 seconds*. Anything longer and they'll use another site.
*note: I read that somewhere but can't remember where.