Phantomjs dies by high memory consumption - javascript

We are using phantomjs to run our qunit tests page on our TFS build server. Our version of test runner is built from below example
https://github.com/ariya/phantomjs/blob/master/examples/run-qunit.js
Over a period of time number of tests increased from hundreds to couple of thousands and on a fine day phantomjs started crashing. It literally dies saying upload the dump and when you see the dump it 0kb !!
When we took a closer look at it on process explorer we found that memory consumption by phantomjs keeps going up as phantomjs is running tests and eventually crashes somewhere 833MB.
Yes the same amount of memory was being utilized by chrome and IE ! And Yes-Yes our tests were leaking memory :(. We did fixed it, memory utilization is lowered by 50% on chrome and IE and we expected phantomjs will handle it now. But no, phantomjs still kept crashing, process explorer shows same memory consumption.
http://phantomjs.org/api/webpage/method/close.html
According to above documentation phantomjs releases heap allocation just on close ? Could that be the reason why our fixed test consumed less memory on chrome but not phantomjs ? And last how to fix this ? How to make phantomjs keep garbage collecting javascript objects to reduce heap allocation ?
Update 1 - 07/28
We took a work around. I did modified my script to execute my tests module by module. In loop after executing all tests for a module I call page.close so it releases the memory for each module and never keeps building the dead heap of objects. Not closing this question since since its a workaround and not a solution. Hope creators will fix this sometime.

There is a static method, QWebPageSettings::clearMemoryCache, that invokes WebKit's garbage collection. However, it clears all QWebPage memory cache for every instantiated QWebPage object and is therefore, currently, unsuitable for including as an option in PhantomJS.
The Github pull request is available here:
https://github.com/ariya/phantomjs/pull/11511
Here's the Google Groups discussion:
https://groups.google.com/forum/#!msg/phantomjs/wIDp9J7B-bE/v5U31_mTbswJ
Until a workaround is available, you might break up your unit tests into blocks on separate pages. It will take a change to QtWebkit's implementation and how memory/cache is handled across QWebPage objects.
Update September 2014:
https://github.com/ariya/phantomjs/commit/5768b705a0
It looks like support for clearing memory cache was added, but there is a note about my original comment in the commit.

I managed to work around it by setting the /LARGEADDRESSAWARE flag
If you have visual studio installed, run from a visual studio command prompt
editbin /LARGEADDRESSAWARE <pathto>/PhantomJS.exe

Related

Is there any memory limit for Google Chrome browser?

What is the default memory limit for a single tab in chrome ??
It's hard to answer correctly and the answer could be figured out only in benchmarks for particular task with particular build.
There are several thing that could allow you to use from 1 to 2 GB of memory:
Chrome version: because different V8 engines have different memory limits.
Chrome platform: 32-bit or 64-bit.
Operating System itself: chrome could be built with different flags for different platforms.
Again, Chrome version: there are some discussion on the internet about recompiling chrome with some compilation flags that could allow to use more memory, then chrome developers decide to exclude such flags because of some reasons. Who know — maybe they will include them again or will increase default heap size.
Video memory: in some cases page's content may require some video memory and, if your video adapter has no sufficient amount, Chrome will take it from general memory, which will increase memory usage for the page.
Right now, with 64bit Chrome version 47, on Windows 8.1 I can take up to 1.8GB with one tab, then it crashes.
Update:
As I can see some magic has happened and limit has changed.
For Chrome version 63, on x64 Windows 10 OS, I could allocate up to 3.5 GB memory, parsing a huge JSON string and then displaying it on the page.
The number is taken from Chrome's Task manager and from Process Explorer's Private Bytes metric.
There was some talking about a very similar topic. Here
You should try the following:
Right click on the Chrome icon and go to properties. Chrome should be here:
"C:\Documents and Settings\%USER%\Local Settings\Application Data\Google\Chrome\Application\chrome.exe"
Where %USER% is your username on your PC, obviously ;)
At the end of the line add --purge-memory-button
It should look like this:
"C:\Documents and Settings\%USER%\Local Settings\Application Data\Google\Chrome\Application\chrome.exe" --purge-memory-button
Now, when Chrome works, press shift+Esc, and now you have a new option, "Purge Memory" which frees up memory. The tabs that do not need attention at that particular time will be purged from your RAM.
You can also add one of these lines:
Never voluntarily relinquish memory
--memory-model=high
Voluntarily reduce working set when switching tabs
--memory-model=medium
Voluntarily reduce working set when switching tabs and also when the
--memory-model=low
Browser is not actively being used
You can have several lines after the target place "C:\Documents and Settings\%USER%\Local Settings\Application Data\Google\Chrome\Application\chrome.exe"
Let's say we wanna use the "Purge memory" line and the "Low memory model" line. It would look like this:
"C:\Documents and Settings\%USER%\Local Settings\Application Data\Google\Chrome\Application\chrome.exe" --purge-memory-button --memory-model=low
You can have a perfect answer once you benchmark on a specific platform with a specific version. The outcome will only be applicable to that platform and chrome version you experimented because there is so many factors that may have a different effect in results including,
The Chrome Version may have different v8 engines and not all v8 engines have the same memory limits. Also, the 32 bit and 64-bit architecture come into picture when we benchmark the chrome. The default heap size may varies from v8 version to version.
The system configuration on which you run the chrome. chrome requires the graphics memory/video memory and when you don't have that, It will use internal memory resulting in more memory usage.
It's also related to process itself, every thread requires some memory for bookkeeping, notably its stack. 32-bit processes can only address 4 GB of memory. 64-bit processes don't have problems with address space, but rather with the actual allocations. However, programs should not approach any such limit or need to worry about it and the same applies to chrome as well.
I tried once to export and print the database from PHPMyAdmin from chrome. It had 1.22GB of size in DB and chrome crashed. Same thing I performed to file export and magically it worked fine.
About the v8 engine, it has 512MB memory limit on 32bit system and 1434MB(1.4 GB) on 64bit system. The chrome has v8 it's heart, this is a major factor affecting the memory limit.
I was able to increase the memory limit for my nw.js (chrome) based destkop app by passing following command line options:
Node.js: node --max_old_space_size=12288 script.js
Google Chrome: chrome.exe --js-flags="--max_old_space_size=12288"
NW.js: add "js-flags": "--max_old_space_size=12288" to the package.json.
Adjust 12288 (12GB) to suit your needs
if you're running 32-bit Chrome, you can't allocate more than 2 GB, because that's the process limit for a 32-bit Windows process. for 64bit chrome there is 4gb limit.
Chrome doesn't allow allocations over 4Gb

JavaScript heap memory is constant, but the browser process private bytes are growing. Where does the memory difference come from?

I am troubleshooting what appears to be a memory leak in our configuration page. The page is used to change the configuration of our service and also displays health diagnostics. This means that we are querying the service periodically for configuration and instrumentation information (typically we use a query interval of 30sec, but to troubleshoot I am querying at 100ms intervals). We rely on knockoutjs, datajs, jquery and spinjs.
I've found that if I leave the page open overnight at the 100ms query interval that the private bytes for the chrome browser tab grows from about 50MB to 335MB. I have four pages with the issue, but am focused on one during my troubleshooting effort. Using chrome://memory-redirect/ I can see the page (process id 26148) memory.
However, the JavaScript heap memory appears to be flat over the same period at 3.6MB. Using the heap profiling tools in Chrome it shows that all of my object allocations are garbage collected.
In the above picture the gray allocations indicate that the objects have been cleaned up by the GC.
The memory timeline also is constant.
I also forced two GCs and confirmed that the number of documents, nodes and listeners was constant between the two GCs.
My questions are:
Where is the process memory being used that is not part of the JavaScript heap?
Given our JavaScript heap memory is flat, could that extra memory be a memory leak caused by our JavaScript code?
Thanks for all the help!
You are comparing apples and oranges - Garbage collected memory in a sub-heap and whole application memory.
You've used Chrome to inspect the JavaScript heap and found evidence that indicates the JavaScript part of your application is running OK.
You're also using a tool to monitor the global memory usage of Chrome itself. That is all the memory that Chrome is using for any task, including tasks not directly related to your application, but to the functioning of the browser itself.
Perhaps you've found a use case that triggers a memory leak in the Chrome internals?
Or perhaps it isn't a memory leak, but memory fragmentation in the non-garbage collected internal heaps used by Chrome?
According to this web page Chrome is written in a mixture of C, C++, Java, JavaScript and Python. This means we have deterministic memory allocators for C and C++ and three different types of garbage collected heaps for Java, JavaScript and Python. Bad news: Python's handling of integers isn't so kind on memory use when it comes to garbage collection (last time I checked, which was a few years ago, maybe they've improved it).
But I've had Chrome sessions run for weeks without issue. So I do wonder what is happening.
You don't say which OS you are using but if you are using Microsoft Windows then you could use C++ Memory Validator to inspect where each allocation was made (full callstack, how many bytes, etc) while Chrome is running (Launch Chrome from C++ Memory Validator, load your applicaton, let it do it's thing then go to the Memory tab and click Refresh - it will display all the live allocations that can be tracked - any statically linked heaps won't be trackable as you won't have the symbols to allow them to be hooked). OK, you don't have symbols to make teh callstacks readable but you can still identify allocations happening at the same place. That may give you a clue as to the cause of the leak/fragmentation so that can report this to the Chrome devs for a closer look.
Do you get the same behaviour in Firefox? If you then you could do what I suggest with C++ Memory Validator but do it on a build of Firefox that you've built yourself - you'll have symbols and source and know exactly where the problem is.
Disclaimer. I am the designer of C++ Memory Validator.

Why do I fail to debug a nodejs app in Intellij IDEA 11?

I have a single process node.js application, which I wish to debug with Intellij IDEA 11 32 bits (node.js is 32 bits too).
So, I place an initial breakpoint and run. The debugger stops at the breakpoint, but then it refuses to do any of the following:
step into
go to another breakpoint
pause execution
When I step into, it seems just to run, without stepping through the code. Once running, it ignores any subsequent breakpoints and does not break when I press the pause button.
This issue drives me crazy.
Any ideas on how should I troubleshoot it?
EDIT
More info. After IDEA breaks on the first breakpoint (the only successful time) I try to inspect the variables and am unable to see any. IDEA is stuck on "Collecting data..." The watch window does not work too.
EDIT2
Justed posted an issue to their bug tracking system - http://youtrack.jetbrains.com/issue/IDEA-112925
I've been noticing that IntelliJ's node.js debugger kinda sucks. It's death by 1000 cuts. I love IntelliJ to death, its such a nice IDE. But for node, the debugger has a million different scenarios where breakpoints don't work properly, and another million where it doesn't properly give you access to the in-scope variable values, and another million where it doesn't step properly...
I'm gonna hafta try looking for another tool..
UPDATE 2014-01-13: I've been using IntelliJ's debugger for a while now (having found no other good tool). It seems some of the problems with it are problems with node or v8 itself. Most of the problems I was having have either actually been solved by newer versions of IntelliJ, or by using this workaround:
1. create a file called proxyDebug.js
2. put the following content in it:
require('path/to/the/script/you/want/to/debug.js')
3. point your debugger to that file
Apparently the node.js debugging hooks go buggy at the entrypoint script, so by creating this proxy entrypoint that we don't care at all about, there won't be any weird bugs caused by that in the scripts you do care about. The bugs this workaround fixed were missed breakpoints, predefined variables (exports, module, process, etc) being inaccessible by the debugger, and one or two other things I can't remember.
Last WebStorm version I tried (7.0.3) actually takes ages in collecting data but eventually works. If it seems stuck to you, try to leave it for a minute

Unexplained growth in chrome private memory

I am in the process of profiling a javascript library I wrote, looking for memory leaks. The library provides an API and service to a back-end. It does not do any html or dom manipulation. It does not load any resources (images, etc). The only thing it does is make xhr requests (using jquery), including one long poll, and it passes and receives data to and from the UI via events (using a Backbone event bus).
I have tested this library running it overnight for 16 hours. The page that loads it does nothing but load the library and sends a login request to start the service. There were no html, css, or other dom changes over the course of the test.
All that happened over the course of the test was the library sent a heartbeat (xhr request) to the server every 15 seconds, and received a heartbeat via the long poll every 30 seconds.
I ran the test with the chrome task manager open, and with the chrome debugger open, in order to force GCs from the timeline.
At the start of the tests, after I forced an initial GC - these were the stats from the chrome task manager:
Memory - 11.7mb
Javascript memory - 6.9 mb / 2.6mb live
Shared memory - 21.4 mb
Private memory 19mb
After 16hrs I forced a GC - these were the new stats:
Memory - 53.8mb
Javascript memory - 6.9 mb / 2.8mb live
Shared memory - 21.7 mb
Private memory 60.9mb
As you can see the JS heap grew by only 200k.
Private memory grew by 42mb!
Can anyone provide possible explanations of what caused the private memory growth? Does having the chrome debugger open cause or affect the memory growth?
One other thought I had is that forcing the GC from the timeline debugger only reclaimed memory from the JS heap - so other memory was not reclaimed. Therefore this may not be a 'leak' per se, as it may eventually be collected - though I'm not sure how to confirm this. Especially without knowing what this memory represents.
Lastly, I did read that xhr results are also stored in private memory. Does anyone know if this is true? If so, the app did perform approx 5700 xhr requests over this timeframe. If most of the 42mb was due to this, that would mean approx 7k was allocated - which seems high, considering the payloads were very small. If this is the case; when would this memory be released, is there anything I can do to cause it to be released, and would having the chrome debugger open impact this?
I was unable to find information on precisely when and what goes to private memory versus javascript memory. However I can answer this: "would having the chrome debugger open impact this" ... YES.
Having the developer tools opened causes the browser to gather/retain/display a lot more information about each and every XHR that is being made. This data is not gathered/retained when the developer tools are closed (as most know since it is sometimes bloody annoying when you open the dev tools to late and it missed that one request you cared about).
You can open the dev tools, trigger the GC, then close the dev tools, opening them only again to trigger the GC when you want to take the metric. Also you can use this sucker chrome://memory-redirect/ to keep track of the growth without opening the dev tools.

Chrome freezes on my backbone page: how to debug?

The project I'm working on involves a "planning" screen, which is entirely made with backbone.js (the other pages of the app are not).
My issue is that from times to times, chrome freezes and the web view stop responding to any interaction. Sometimes, I can manage to quit chrome itself, but usually the controls does not answer either.
I'm pretty convinced this is related to the js code. It seems to me that when a script takes too much time, or loops indefinitely, Chrome can detect this and interrupt the script. However, since this is not the case, I'm thinking that too many js objects stay in memory.
Whatever the cause is, I would like to know which chrome dev tools can help me here. While I'm not a beginner in js, asides setting breakpoints and calling console.log, I have no idea how to debug JS apps. I'm not opposed to use another browser if the dev tools are more suited.
Thanks a lot for your time !
FTR : This is a rails 3.2.8 app, using mongodb, & Backbone.js 0.9.2. The js code is written in coffeescript. This issue happened on my macbook air 2012 running mountain lion, as well as on the client machine which runs on windows 7. The issue appeared at least on chrome 22 & 23.
Using the Javascript CPU profiler, I was able to find the group of functions that seems to be responsible for the freezing.
I'm still open to any advice/ressource on debugging javascript code.
Make a console.log in the loop and check if its the same point freezing on all chrome versions. There is a limit see Browser Javascript Stack size limit.
Maybe add some code? Because there could be some memory leaks especially with event handlers!
What I would do is the long and weary approach to getting your hands dirty with console.log. --- in this case.
In your case, and THX for the hint by the way (finding the offender with the CPU profiler, I'll try that next time round), I guess the offenders might be some functions "callbacking" themselves .. probably via some sort of event-handling/bubbling/callback-combination.
What happens then usually is that it just doesn't recognize the endless-loop, because the callback-stack is kinda "broken". Therefor it will never throw a browser-error.
If you have any luck, it doesn't kill the browser quick enough to kill the console. I did have that at times, the console killed (remaining silent), the coffeescript files not even loaded in the debugger (I use a JIT-coffee-to-js-translator as in JS-MVC) ... the page frozen or not doing anything ...
So, if indeed you are lucky and the debugger spits out your console.logs you can thereby guess where your unwanted loop is hiding. Just by looking at the repeated order of the output statements.
of course you know you can set breakpoints right? even conditional breakpoints
I'm currently struggling a bit untangling a similar sounding loop - i guess in case of emergency some alert() calls in your code would at least slow the loop down
or maybe some wait calls (bad idea)
my loop is running so fast I loose my console log!

Categories

Resources