I'm making a rather extensive game using Javascript. It is a kind of online game maker that allows players to upload media files and use them to create worlds. Unfortunately, it is rather prone to crashing the browser tab at unpredictable moments. So far, I have found no pattern to this - sometimes it happens within a few minutes, other times it can run for hours without a problem.
I have tried enabling logging in Chrome, but the crashes don't seem to generate an error report in the chrome_debug file.
I thought it might be that the program was using too much memory (given the game's open-ended nature, some worlds can involve downloading rather large data files - though this seems unrelated to when the crash actually happens - while large worlds do seem to be more crash-prone, they do not always crash when the world's data is loaded).
I tried using Electron to turn it into an executable app, but the app still crashes. That shouldn't happen if it's a memory issue, right?
Is there any way of finding out what is causing the code to crash?
Most unpredictable crashes in Javascript are caused by memory leaks - objects that are still stored in memory and not being picked up by the garbage collector. Every object in Javascript is stored in a variable somewhere within the global scope, or is associated with another object that is itself connected to the global scope. When a "branch" of the "tree" is removed and can no longer be accessed by the global scope, the garbage collector destroys it.
However, if an object is not being removed from the global scope when it should be, it remains in memory. This usually happens when objects are added to an array but are not removed from that array when they are no longer in use. Over time, these objects build up until the process crashes due to memory overload.
To find memory leaks in Chrome, press F12 and open the Performance tab. By recording the page over time, you can view the amount of memory being used. The green line (nodes) is the most important here - it refers to the number of objects in memory. If nodes are constantly increasing over time (there will always be increases and decreases, but if the overall level is constantly rising) this generally means there's a memory leak.
To find which specific objects are causing the problem, open the Memory tab to take snapshots or timeline profiles of the memory heap. This gives you a count of the specific objects that are in memory at any given time. If there are more of some kind of object than there should be, that's where the leak is.
I want to limit the memory or the number of entries in the JavaScript / Web Inspector console.
Is it already limited somehow (except by available and accessible memory)?
If not, is it possible to just clear out the oldest entries in the console log while retaining newer ones, i.e. something like console.clear(10000)?
Is there anything like a limit or a selective clear() in any JavaScript engine?
The issue is that I want to log debug information in a single-page app in an embedded web view but do not want to constantly leak memory this way.
While some JavaScript developer tools might implement an internal limit, WebKit's Web Inspector apparently does not, at least it does not expose any documented interface to it.
So basically there are two feasible solutions (besides patching WebKit or using a tool that supports a limit), as Cerbrus and Jan Dvorak have also mentioned.
Create a proxy to the various console methods to buffer any log output and dump on request only, and limit the internal buffer.
A simple ring buffer is not very fast, so an alternative would be to use two buffers where the older one is discarded and overwritten when the current becomes full.
The drawback of using a buffer is that you cannot immediately expand objects but only on first access, which may lose vital debug data if they get overwritten (Qt 5's console expands simple objects and arrays during logging).
Periodically clear the console to limit memory usage.
However, the console may be cleared at the wrong time.
I am working with quite large volume of data.
Mechanism:
JavaScript is reading WebSQL database, then assembles data into Object that has tree structure.
Then applies to tree object knockout.js (makes elements observable) then data-binds
and then applies Jquery Mobile UI at the end.
Whole process takes unacceptable amount of time.
I have already optimized algorithm that makes tree object out of data,
also optimised conversion to observables mechanism by pushing items directly into ko.observable arrays and calling hasMutated only once.
I am applying knockout.js IF bindings to not process invisible tree nodes in UI until parent is opened.
Performance here is key.
After inspecting page load in timeline in Chrome developer tools I have noticed that Garbage Collector is doing cleans on every concurrent call when I am building tree object.
Question: Is there a way to temporarily disable Chrome GC and then enable it again after I am done with page processing?
P.S I know I could add reference to part that gets collected, basically introduce object that dominates and prevents GC collection, but this would require substantial changes through the code, and I am not sure I could keep it long enough, and it is likely to introduce memory leak. Surely there must be better way
No, there is no way to disable the garbage collector. There cannot be, because what is Chrome supposed to do when more memory is requested but none is available?
(Also, the garbage collector is very fine-grained and complicated; your screenshot is a bit too small to be readable, but in all likelihood what you're seeing are small steps of incremental work to keep up with allocations, and/or "minor GC" cycles that only operate on the relatively small area of the heap where new allocations happen.)
If you want to reduce time spent in GC, then the primary way how to achieve that is to allocate fewer and/or smaller objects. Yes, that can mean changing your application's design so that objects are reused instead of being short-lived, or similar changes in strategy.
If you allocate a lot, you will see a lot of GC activity, there is just no way around that. This is true even in languages/runtimes that are not considered "garbage collected", e.g. in C/C++ using new/delete a lot also has a performance cost.
I've created a very simple test case that creates a Backbone view, attaches a handler to an event, and instantiates a user-defined class. I believe that by clicking the "Remove" button in this sample, everything will be cleaned up and there should be no memory leaks.
A jsfiddle for the code is here: http://jsfiddle.net/4QhR2/
// scope everything to a function
function main() {
function MyWrapper() {
this.element = null;
}
MyWrapper.prototype.set = function(elem) {
this.element = elem;
}
MyWrapper.prototype.get = function() {
return this.element;
}
var MyView = Backbone.View.extend({
tagName : "div",
id : "view",
events : {
"click #button" : "onButton",
},
initialize : function(options) {
// done for demo purposes only, should be using templates
this.html_text = "<input type='text' id='textbox' /><button id='button'>Remove</button>";
this.listenTo(this,"all",function(){console.log("Event: "+arguments[0]);});
},
render : function() {
this.$el.html(this.html_text);
this.wrapper = new MyWrapper();
this.wrapper.set(this.$("#textbox"));
this.wrapper.get().val("placeholder");
return this;
},
onButton : function() {
// assume this gets .remove() called on subviews (if they existed)
this.trigger("cleanup");
this.remove();
}
});
var view = new MyView();
$("#content").append(view.render().el);
}
main();
However, I am unclear how to use Google Chrome's profiler to verify that this is, in fact, the case. There are a gazillion things that show up on the heap profiler snapshot, and I have no idea how to decode what's good/bad. The tutorials I've seen on it so far either just tell me to "use the snapshot profiler" or give me a hugely detailed manifesto on how the entire profiler works. Is it possible to just use the profiler as a tool, or do I really have to understand how the whole thing was engineered?
EDIT: Tutorials like these:
Gmail memory leak fixing
Using DevTools
Are representative of some of the stronger material out there, from what I've seen. However, beyond introducing the concept of the 3 Snapshot Technique, I find they offer very little in terms of practical knowledge (for a beginner like me). The 'Using DevTools' tutorial doesn't work through a real example, so its vague and general conceptual description of things aren't overly helpful. As for the 'Gmail' example:
So you found a leak. Now what?
Examine the retaining path of leaked objects in the lower half of the Profiles panel
If the allocation site cannot be easily inferred (i.e. event listeners):
Instrument the constructor of the retaining object via the JS console to save the stack trace for allocations
Using Closure? Enable the appropriate existing flag (i.e. goog.events.Listener.ENABLE_MONITORING) to set the creationStack property during construction
I find myself more confused after reading that, not less. And, again, it's just telling me to do things, not how to do them. From my perspective, all of the information out there is either too vague or would only make sense to someone who already understood the process.
Some of these more specific issues have been raised in #Jonathan Naguin's answer below.
A good workflow to find memory leaks is the three snapshot technique, first used by Loreena Lee and the Gmail team to solve some of their memory problems. The steps are, in general:
Take a heap snapshot.
Do stuff.
Take another heap snapshot.
Repeat the same stuff.
Take another heap snapshot.
Filter objects allocated between Snapshots 1 and 2 in Snapshot 3's "Summary" view.
For your example, I have adapted the code to show this process (you can find it here) delaying the creation of the Backbone View until the click event of the Start button. Now:
Run the HTML (saved locally of using this address) and take a snapshot.
Click Start to create the view.
Take another snapshot.
Click remove.
Take another snapshot.
Filter objects allocated between Snapshots 1 and 2 in Snapshot 3's "Summary" view.
Now you are ready to find memory leaks!
You will notice nodes of a few different colors. Red nodes do not have direct references from Javascript to them, but are alive because they are part of a detached DOM tree. There may be a node in the tree referenced from Javascript (maybe as a closure or variable) but is coincidentally preventing the entire DOM tree from being garbage collected.
Yellow nodes however do have direct references from Javascript. Look for yellow nodes in the same detached DOM tree to locate references from your Javascript. There should be a chain of properties leading from the DOM window to the element.
In your particular you can see a HTML Div element marked as red. If you expand the element you will see that is referenced by a "cache" function.
Select the row and in your console type $0, you will see the actual function and location:
>$0
function cache( key, value ) {
// Use (key + " ") to avoid collision with native prototype properties (see Issue #157)
if ( keys.push( key += " " ) > Expr.cacheLength ) {
// Only keep the most recent entries
delete cache[ keys.shift() ];
}
return (cache[ key ] = value);
} jquery-2.0.2.js:1166
This is where your element is being referenced. Unfortunally there is not much you can do, it is a internal mechanism from jQuery. But, just for testing purpose, go the function and change the method to:
function cache( key, value ) {
return value;
}
Now if you repeat the process you will not see any red node :)
Documentation:
Eliminating memory leaks in Gmail.
Easing JavaScript Memory Profiling In Chrome DevTools.
Here's a tip on memory profiling of a jsfiddle: Use the following URL to isolate your jsfiddle result, it removes all of the jsfiddle framework and loads only your result.
http://jsfiddle.net/4QhR2/show/
I was never able to figure out how to use the Timeline and Profiler to track down memory leaks, until I read the following documentation. After reading the section entitled 'Object allocation tracker' I was able to use the 'Record Heap Allocations' tool, and track some some Detached DOM nodes.
I fixed the problem by switching from jQuery event binding, to using Backbone event delegation. It's my understanding that newer versions of Backbone will automatically unbind the events for you if you call View.remove(). Execute some of the demos yourself, they are set up with memory leaks for you to identify. Feel free to ask questions here if you still don't get it after studying this documentation.
https://developers.google.com/chrome-developer-tools/docs/javascript-memory-profiling
Basically you need to look at the number of objects inside your heap snapshot. If the number of objects increases between two snapshots and you've disposed of objects then you have a memory leak. My advice is to look for event handlers in your code which do not get detached.
There is an introduction video from Google, which will be very helpful to find JavaScript memory leaks.
https://www.youtube.com/watch?v=L3ugr9BJqIs
You also might want to read :
http://addyosmani.com/blog/taming-the-unicorn-easing-javascript-memory-profiling-in-devtools/
It explains the use of the chrome developer tools and gives some step-by-step advices on how to confirm and locate a memory leak using heap snapshot comparison and the different hep snapshot views available.
You could also look at the Timeline tab in developer tools. Record the usage of your app and keep an eye on the DOM Node and Event listener count.
If the memory graph would indeed indicate a memory leak, then you can use the profiler to figure out what is leaking.
A couple of important notes in regards to identifying memory leaks using Chrome Developer tools:
1) Chrome itself has memory leaks for certain elements such as password and number fields. https://bugs.chromium.org/p/chromium/issues/detail?id=967438. Avoid using those while debugging as they polute your heap snapshot when searching for detached elements.
2) Avoid logging anything to the browser console. Chrome will not garbage collect objects written to the console, hence affecting your result. You can suppress output by placing the following code in the beginning of you script/page:
console.log = function() {};
console.warn = console.log;
console.error = console.log;
3) Use heap snapshots and search for "detach" to identify detached DOM elements. By hovering objects, you get access to all the properties including id and outerHTML which may help identify each element.
If the detached elements are still too generic to recognize, assign them unique IDs using the browser console prior to running your test, e.g.:
var divs = document.querySelectorAll("div");
for (var i = 0 ; i < divs.length ; i++)
{
divs[i].id = divs[i].id || "AutoId_" + i;
}
divs = null; // Free memory
Now, when you identify a detached element with, lets say id="AutoId_49", reload your page, execute the snippet above again, and find the element with id="AutoId_49" using the DOM inspector or document.querySelector(..). Naturally this only works if your page content is predictable.
How I run my tests to identify memory leaks
1) Load page (with console output suppressed!)
2) Do stuff on page that could result in memory leaks
3) Use Developer Tools to take a heap snapshot and search for "detach"
4) Hover elements to identify them from their id or outerHTML properties
I second the advice to take a heap snapshot, they're excellent for detecting memory leaks, chrome does an excellent job of snapshotting.
In my research project for my degree I was building an interactive web application that had to generate a lot of data built up in 'layers', many of these layers would be 'deleted' in the UI but for some reason the memory wasn't being deallocated, using the snapshot tool I was able to determine that JQuery had been keeping a reference on the object (the source was when I was trying to trigger a .load() event which kept the reference despite going out of scope). Having this information at hand single-handedly saved my project, it's a highly useful tool when you're using other people's libraries and you have this issue of lingering references stopping the GC from doing its job.
EDIT:
It's also useful to plan ahead what actions you're going to perform to minimize time spent snapshotting, hypothesize what could be causing the problem and test each scenario out, making snapshots before and after.
Adding my 2 cents here with the tools available in 2021: https://yonatankra.com/how-to-profile-javascript-performance-in-the-browser/
There's a short video version here: https://yonatankra.com/detect-memory-leak-with-chrome-dev-tools
I'm suspecting that if i set div.innerHTML="" instead of using while(div.firstChild)div.removeChild(div.firstChild) the memory will be hogged until the page refreshes or the browser closes.
My question is how do we actully go about testing if my hypothesis is true?
Firstly, whether memory is returned to the system or not depends on the Javascript garbage collector (gc) and therefore results will vary from browser to browser.
It's difficult to measure memory usage by looking at the process as there are several layers of memory management in place. To see how this can have an impact, consider that a huge javascript object might have been erased forever, but that memory might still not have yet been released back to the operating system because the web browser might keep hold of it in case you need to create more big objects. Another example is that most gc routines only run periodically, so it's possible that object is still in memory but will be reclaimed later.
However, it's pretty easy to determine if a particular operation is leaking memory as all you have to do is repeat it in an endless loop. e.g.
remove references to existing html elements
construct new html elements
add them to the page
Try this code:
var div = document.getElementById("test")
while(true) {
// remove operation, change me
while(div.firstChild) {
div.removeChild(div.firstChild);
}
// create some new content
for(var i=0; i<1000; i++) {
var p = document.createElement('p');
p.appendChild(document.createTextNode('text'));
div.appendChild(p);
}
}
My results in Chrome, using the Task Manager (shift+esc):
Leaving the loop running endlessly without deleting anything eventually results in the "Aw! Snap" screen, which indicates memory has been exhausted
Using the removeChild technique leads to memory usage stabilizing at around 750MB
Using the innerHTML technique leads to memory usage stabilizing at around 750MB
If memory isn't leaking you'll notice a pattern similar to this: Increases to 300, drops to 150, increases to 400, drops to 250, eventually stabilising. This is the memory management system running out of memory, triggering the gc to reclaim memory that has been deallocated and increasing the available memory footprint each time until reaching a soft limit that has been set to avoid the process impacting others. This is a typical memory management scheme and more can be found by reading this wikipedia article: http://en.wikipedia.org/wiki/Garbage_collection_(computer_science)
Since both results stabilize, I'd conclude that (for Chrome at least) both techniques work the same though you might get different results from other browsers.
As there doesn't seem to be a difference, removeChild should be preferred as innerHTML is not included in the W3C standard and is frowned upon by some developers. See more here: Alternative for innerHTML?
The reason innerHTML and removeChild have no differences is because underneath, they both have to de-reference elements to stop them from being visible on the screen. Memory leaks are most likely to occur when you have circular references (A points to B, B points to A, nobody else points to either) but this is only a problem in older browsers. This link has some good guidelines about how to avoid js memory leaks: Javascript memory management pitfalls?