Cloning knockout observables with lodash.clonedeep - javascript

I needed some ideas regarding my current issue...
I'm currently improving a small but common part of a big software. This means I may not replace functionality without creating extreme testing overhead or even impacting functionality.
I'm trying to clone a data structure that besides normal data contains lots of knockout observables. Doing this I see serious performance differences between Chrome and IE.
It could be possible that there are cyclic references, it could also be possible, that knockout observables somehow do not like being cloned. Or perhaps the Internet
Explorer reached some kind of data size limit that can be extended. Whatever the reason is, Chrome somehow does better.
The problem gets intensified because of these reasons:
I need this huge data structure of several tens of thousands "beautified JSON"(!!) lines to simulate
real life and to create this big problem. This amount of data may not be diminished to a debugable extent because the smaller the data is, the less the timing
difference to find and finally it vanishes. To publish the data structure is also not possible for me because I'm not able to make such an amount of data anonymous. And with this
size of data it is not really possible to find data problems while debugging.
I do not think that Google is that superior to Microsoft, so I expect the problem to be data dependent or to be able to be eliminated by some browser settings.
Overall in the internet I can read, that lodash.clonedeep() would be slow or that the Internet Explorer would be slow. If so, this may not be my problem. In my opinion this does not explain the huge differences on Chrome an the Internet Explorer.
Such an answer is too simple for me and I need some improvement because most of our customers use the Internet Explorer and this is impossible to change.
I did run the following lines in Chrome and Internet Explorer.
console.time('jquery.extend()');
dataCopy = $.extend(true, {}, viewModel.api.getData());
console.timeEnd('jquery.extend()');
console.time('lodash.clonedeep()');
dataCopy = _.cloneDeep(viewModel.api.getData());
console.timeEnd('lodash.clonedeep()');
console.time('JSON.parse(JSON.stringify())');
dataCopy = JSON.parse(JSON.stringify(viewModel.api.getData()));
console.timeEnd('JSON.parse(JSON.stringify())');
The output is as follows
Chrome Version 62.0.3202.94:
jquery.extend(): 1159.470947265625ms
lodash.clonedeep(): 2783.241943359375ms
JSON.parse(JSON.stringify()): 1139.403076171875ms
Internet Explorer Version 11.0.9600.18837:
jquery.extend(): 10.802,27ms
lodash.clonedeep(): 28.302,65ms
JSON.parse(JSON.stringify()): 10.479,681ms
Does anybody have an idea regarding
why this huge difference exists?
a way to accelerate IE (almost) to the speed of Chrome in this context?
a hint what to look at? / to find out what the problem is?
a way to find out what is going on in lodash.clonedeep() with such huge data? (Something more efficient than breakpoints.)
possible debug output messages from anywhere that could help?
is there a known impact of knockout observables to lodash.clonedeep() on IE?
whether knockout observables somehow do not like to be cloned?
whether cyclic structures are a problem for cloning at all? (If so, why is Chrome that faster?)
I'm searching for resolutions as well as for best practices.
Thanks to all of you!

Related

Why is getting from Map slower than getting from object?

I'm considering migrating my state management layer to using Map versus using a standard object.
From what I've read, Map is effectively a hash table whereas Objects use hidden classes under the hood. Generally it's advised, where the properties are likely to be dynamically added or removed it's more efficient to use Map.
I set up a little test and to my surprise accessing values in the Object version was faster.
https://jsfiddle.net/mfbx9da4/rk4hocwa/20/
The article also mentions fast and slow properties. Perhaps the reason my code sample in test1 is so fast is because it is using fast properties? This seems unlikely as the object has 100,000 keys. How can I tell if the object is using fast properties or dictionary lookup? Why would the Map version be slower?
And yes, in practice, looks like a premature optimization, root of all evil ... etc etc. However, I'm interested in the internals and curious to know of best practices of choosing Map over Object.
(V8 developer here.)
Beware of microbenchmarks, they are often misleading.
V8's object system is implemented the way it is because in many cases it turns out to be very fast -- as you can see here.
The primary reason why we recommend using Map for map-like use cases is because of non-local performance effects that the object system can exhibit when certain parts of the machinery get "overloaded". In a small test like the one you have created, you won't see this effect, because nothing else is going on. In a large app (using many objects with many properties in many different usage patterns), it's still not guaranteed (because it depends on what the rest of the app is doing) but there's a good chance that using Maps where appropriate will improve overall performance -- if the overall system previously happened to run into one of the unfortunate situations.
Another reason is that Maps handle deletion of entries much better than Objects do, because that's a use case their implementation explicitly anticipates as common.
That said, as you already noted, worrying about such details in the abstract is a case of premature optimization. If you have a performance problem, then profile your app to figure out where most time is being spent, and then focus on improving those areas. If you do end up suspecting that the use of objects-as-maps is causing issues, then I recommend to change the implementation in the app itself, and measure (with the real app!) whether it makes a difference.
(See here for a related, similarly misleading microbenchmark, where even the microbenchmark itself started producing opposite results after minor modifications: Why "Map" manipulation is much slower than "Object" in JavaScript (v8) for integer keys?. That's why we recommend benchmarking with real apps, not with simplistic miniature scenarios.)

Javascript Pouch DB doesn't load on Safari & Firefox

We have built a small script and a database, based on PouchDB in order to display all the products of one of our clients in a so called "product tree".
You can find the product tree here: http://www.bodyrevitaliser.nl/nl/service/product-tree/
As you can see the tree is loading properly only in Chrome. If you check the console in safari and Firefox the DB seems to be loaded as well but something seems to be blocking the tree itself to be loaded.
What are you thoughts? Any ideas what might be causing this and solutions.
The problem with your code is that your usage of promises is not correct. I strongly recommend you read this blog post: We have a problem with promises. I know it's long, but it's worthwhile to read the whole thing.
In particular, read the section called "WTF, how do I use forEach() with promises?", because that is exactly the mistake that you're making. You are doing a bunch of insertions inside of a $.each, and then you are immeditely doing an allDocs() inside the same function. So you have zero guarantees that any documents have actually been inserted into PouchDB by the time you try to read from PouchDB. Perhaps it will, perhaps it won't, but it all depends on subtle timing differences between different browsers, so you can't count on it.

Is there a way to control Chrome GC?

I am working with quite large volume of data.
Mechanism:
JavaScript is reading WebSQL database, then assembles data into Object that has tree structure.
Then applies to tree object knockout.js (makes elements observable) then data-binds
and then applies Jquery Mobile UI at the end.
Whole process takes unacceptable amount of time.
I have already optimized algorithm that makes tree object out of data,
also optimised conversion to observables mechanism by pushing items directly into ko.observable arrays and calling hasMutated only once.
I am applying knockout.js IF bindings to not process invisible tree nodes in UI until parent is opened.
Performance here is key.
After inspecting page load in timeline in Chrome developer tools I have noticed that Garbage Collector is doing cleans on every concurrent call when I am building tree object.
Question: Is there a way to temporarily disable Chrome GC and then enable it again after I am done with page processing?
P.S I know I could add reference to part that gets collected, basically introduce object that dominates and prevents GC collection, but this would require substantial changes through the code, and I am not sure I could keep it long enough, and it is likely to introduce memory leak. Surely there must be better way
No, there is no way to disable the garbage collector. There cannot be, because what is Chrome supposed to do when more memory is requested but none is available?
(Also, the garbage collector is very fine-grained and complicated; your screenshot is a bit too small to be readable, but in all likelihood what you're seeing are small steps of incremental work to keep up with allocations, and/or "minor GC" cycles that only operate on the relatively small area of the heap where new allocations happen.)
If you want to reduce time spent in GC, then the primary way how to achieve that is to allocate fewer and/or smaller objects. Yes, that can mean changing your application's design so that objects are reused instead of being short-lived, or similar changes in strategy.
If you allocate a lot, you will see a lot of GC activity, there is just no way around that. This is true even in languages/runtimes that are not considered "garbage collected", e.g. in C/C++ using new/delete a lot also has a performance cost.

How to "follow" the references / memory profile of specific Javascript values, or how to tie references in memory profiler to source code

I'd like to know if it's possible (with any browser / dev tools) to pick a specific value or closure variable while debugging and "follow" or "watch" it somehow into future execution points on the page. Basically, a memory profiler attached to just a single value, which would show during debugging or snapshots whether that value is still being retained either directly or indirectly. Alternatively, I'd like to know if it's possible to look at references in the memory profiler/snapshot view in, say, Chrome, and tie those references to actual points in the source code.
My problem is that I am debugging a memory leak caused by rebuilding a DOM tree for a portion of a fairly complex page. Even taking a very controlled memory snapshot that just looks at a single redraw (removing the old DOM tree and adding a new one, where I know that I'm unintentionally retaining a reference to a small part of the old one), there are still hundreds of objects to look through, and to be quite honest I find the memory profiler in Chrome to be very confusing to navigate through. And even when I find references that might be of interest, I'm at a loss as to how to tie them to points in the code - it's great to know that I'm retaining an HTMLDivElement somewhere but that could be almost any of the files...
So basically, I'm unsure how to proceed, and the two solutions I'm asking about are the only things I can think of, if there is any way to do them. Sorry that this is such a vague question, I am open to other ways of tackling this as well.

But why's the browser DOM still so slow after 10 years of effort?

The web browser DOM has been around since the late '90s, but it remains one of the largest constraints in performance/speed.
We have some of the world's most brilliant minds from Google, Mozilla, Microsoft, Opera, W3C, and various other organizations working on web technologies for all of us, so obviously this isn't a simple "Oh, we didn't optimize it" issue.
My question is if i were to work on the the part of a web browser that deals specifically with this, why would I have such a hard time making it run faster?
My question is not asking what makes it slow, it's asking why hasn't it become faster?
This seems to be against the grain of what's going on elsewhere, such as JS engines with performance near that of C++ code.
Example of quick script:
for (var i=0;i<=10000;i++){
someString = "foo";
}
Example of slow because of DOM:
for (var i=0;i<=10000;i++){
element.innerHTML = "foo";
}
Some details as per request:
After bench marking, it looks like it's not an unsolvable slow issue, but often the wrong tool is used, and the tool used depends on what you're doing cross-browser.
It looks like the DOM efficiency varies greatly between browsers, but my original presumption that the dom is slow and unsolvable seems to be wrong.
I ran tests against Chrome, FF4, and IE 5-9, you can see the operations per second in this chart:
Chrome is lightning fast when you use the DOM API, but vastly slower using the .innerHTML operator (by a magnitude 1000-fold slower), however, FF is worse than Chrome in some areas (for instance, the append test is much slower than Chrome), but the InnerHTML test runs much faster than chrome.
IE seems to actually be getting worse at using DOM append and better at innerHTML as you progress through versions since 5.5 (ie, 73ops/sec in IE8 now at 51 ops/sec in IE9).
I have the test page over here:
http://jsperf.com/browser-dom-speed-tests2
What's interesting is that it seems different browsers seem to all be having different challenges when generating the DOM. Why is there such disparity here?
When you change something in the DOM it can have myriad side-effects to do with recalculating layouts, style sheets etc.
This isn't the only reason: when you set element.innerHTML=x you are no longer dealing with ordinary "store a value here" variables, but with special objects which update a load of internal state in the browser when you set them.
The full implications of element.innerHTML=x are enormous. Rough overview:
parse x as HTML
ask browser extensions for permission
destroy existing child nodes of element
create child nodes
recompute styles which are defined in terms of parent-child relationships
recompute physical dimensions of page elements
notify browser extensions of the change
update Javascript variables which are handles to real DOM nodes
All these updates have to go through an API which bridges Javascript and the HTML engine. One reason that Javascript is so fast these days is that we compile it to some faster language or even machine code, masses of optimisations happen because the behaviour of the values is well-defined. When working through the DOM API, none of this is possible. Speedups elsewhere have left the DOM behind.
Firstly, anything you do to the DOM could be a user visible change. If you change the DOM, the browser has to lay everything out again. It could be faster, if the browser caches the changes, then only lays out every X ms (assuming it doesn't do this already), but perhaps there's not a huge demand for this kind of feature.
Second, innerHTML isn't a simple operation. It's a dirty hack that MS pushed, and other browsers adopted because it's so useful; but it's not part of the standard (IIRC). Using innerHTML, the browser has to parse the string, and convert it to a DOM. Parsing is hard.
Original test author is Hixie (http://nontroppo.org/timer/Hixie_DOM.html).
This issue has been discussed on StackOverflow here and Connect (bug-tracker) as well. With IE10, the issue is resolved. By resolved, I mean they have partially moved on to another way of updating DOM.
IE team seems to handle the DOM update similar to Excel-macros team at Microsoft, where it's considered a poor practice to update the live-cells on the sheet. You, the developer, is supposed to take the heavy lifting task offline and then update the live team in batch. In IE you are supposed to do that using document-fragment (as opposed to document). With new emerging ECMA and W3C standards, document-frags are depreciated. So IE team has done some pretty work to contain the issue.
It took them few weeks to strip it down from ~42,000 ms in IE10-ConsumerPreview to ~600 ms IE10-RTM. But it took lots of leg pulling to convince them that this IS an issue. Their claim was that there is no real-world example which has 10,000 updates per element. Since the scope and nature of rich-internet-applications (RIAs) can't be predicted, its vital to have performance close to the other browsers of the league. Here is another take on DOM by OP on MS Connect (in comments):
When I browse to http://nontroppo.org/timer/Hixie_DOM.html, it takes
~680ms and if I save the page and run it locally, it takes ~350ms!
Same thing happens if I use button-onclick event to run the script
(instead of body-onload). Compare these two versions:
jsfiddle.net/uAySs/ <-- body onload
vs.
jsfiddle.net/8Kagz/ <-- button onclick
Almost 2x difference..
Apparently, the underlying behavior of onload and onclick varies as well. It may get even better in future updates.
Actually, innerHTML is less slow than createElement.
In an effort to optimize I found js can parse enormous json effortlessly. Json parsers can have a huge number of nested function calls without issues. One can toggle between display:none and display:block thousands of elements without issues.
But if you try create a few thousand elements (or even if you simply clone them) performance is terrible. You don't even have to add them to the document!
Then, when they are created, insert and remove from the page works supper fast again.
It looks to me like the slowness has little to do with their relation to other elements of the page.

Categories

Resources