Why node.js process taking more memory than allocated - javascript

Can someone please explain why my Node.js process is taking more than allocated memory?
I assigned 4G of memory to Nodejs process (maximum supported on 64bit machine, as per Nodejs doc), but I have seen process touching 5.6g of RSS memory (way higher than 4g that I assigned)
This is how I am running the process
node -max-old-space-size=4096 processName.js
This is what my TOP command shows (RSS #4.6g)

max-old-space-size controls one aspect of node.js memory usage within the interpreter as used for the storage of Javascript objects (sometimes referred to as the V8 heap), not the entire memory usage of the whole process. For example, max-old-space-size has nothing to do with how much memory the native code portions of node.js use at all.
So, total memory usage can always be more than max-old-space-size.

Related

Should I worry about high RSS in node if the heaptotal / heapused is low?

So I'm a starting programmer and I honestly can't find any information about what node's rss means. Everything just says its the total amount of memory allocated to the process. great! so is that a problem?
I'm writing a small discord bot in node and I noticed my rss going all over the place.
45.5 MB used for shard 1 (10.1 MB HeapUsed, 16.9 MB HeapTotal)
37.2 MB used for shard 1 (7.1 MB HeapUsed, 9.6 MB HeapTotal)
75.3 MB used for shard 1 (7.2 MB HeapUsed, 9.6 MB HeapTotal)
These are measurements about 5 seconds apart after starting up the process. It stays at that ~75MB mark.
I'm wondering if I should actually worry about this memory usage or if it's totally fine. Say I run this program on a host with only 2GB of ram. Would having an RSS of 1800MB be bad or would it just cap itself there and only improve the heapused/heaptotal?
Also is there any way to check what the process is assigning memory to in node?
My question really just is, Should I worry about the RSS or should I just ignore it and only look at the heap?
RSS is the total portion of the memory allocated for the running node process at hand system wise.
This means that even though heap portions of the memory are low, total rss could grow arbitrarily high unless the system where the node process resides decides to free it -possibly because it's running out of memory for other processes-
Most of the time, this is a "normal" behaviour of node, although some discussions have marked this as "a possible hidden memory leak" within node itself.
When the time comes, system should signal the process to free the unused memory that has being "reserved" for node.
Other times though, rss can grow bigger due to binary dependencies on your project. Let's say, those having memory leaks within them, system wouldn't be able to free that memory when the system requires so.
TL;DR
Unless you're having an specific problem regarding the rss memory space allocation -in which case, you'll definitely know-, don't panic about this behaviour, as most of the time the underlying system should take care of it.
Pay attention to your binary dependencies though. Those will not show on the heap size reported by V8's engine, and in case of a memory leak within them, that portion of memory will no be able to be freed given the time has come.
source

nodejs memory leak : process usage increase more than heap

I got some memory leak issue on my nodejs app executed by openshift docker soulution.
when I try to monitor memory usage using process rss, I found process memory increased over time.
I'm trying to catch memory usage in process heap but memwatch, heapdump module can't show anything.
npm module showed heap size, and diff that size is under 50mb.
but process memory is still increasing and it uses over 150mb.
I thought It caused application leak issue so I try --expose-gc and called global.gc() but never helped.
how can I see where the process use memory or
does nodejs use OS memory more than that's max heap size?
(I showed memory usage increased over 4GB)
I want to fix it or want to see how nodejs use that memory.
thanks for read & answer :)

Chrome keeps source code of functions around

I have some JavaScript Functions with source code coming from larger JavaScript-files. They are created this way:
const function = Function("foo", "...<large JS source code>...");
When looking at the memory snapshots in Chrome, the source code of these functions is retained and create a large memory overhead. Is it possible to "release"/"drop" source code of functions in JavaScript?
EDIT
I am really keeping the function itself around and wonder whether it’s possible to tell the function to drop the source code from memory.
Internally Chrome uses the V8 JavaScript engine. This engine is a just-in-time JavaScript engine which means that it takes your JavaScript and it compiles it to machine code when it needs it. This is expensive; so V8 will cache the result of the compilation and (in most cases) will re-use the previously compiled code if you call the function again.
I believe function memory reclamation is handled by the V8 garbage collector. When a variable (or function) falls out of scope (there aren't any references left to it anywhere, including closures) then the garbage collector is free to reclaim that memory including the source code and cached machine code. The garbage collector runs periodically and will clean up any memory from anything still in scope. Generally speaking you shouldn't try to force garbage collection on your own, it should happen automatically, but with Chrome there is a way to force garbage collection using the developer tools.
If you remove any references to your function (remember this includes closures) and force garbage collection you should see the memory reclaimed. Use the Chrome Developer memory tools to see if your function has been reclaimed or not (look at "heap snapshots").
There's one other caveat to this: even if the memory is reclaimed it won't necessarily be released back to the operating system or even cleared. Many applications handling large amounts of small memory allocations will try to improve performance by trying to re-use previously allocated memory before asking the operating system for more. So if you're using a low-level memory inspector you may still see your code hanging around in memory even if it's been garbage collected and there aren't any useful references to it. Without diving pretty deep into the V8 internals it probably isn't possible to determine from a memory dump if your code is still in memory because of a memory leak or because Chrome allocated the memory and simply hasn't released it back to the operating system after internally cleaning up references to that memory.

JavaScript heap memory is constant, but the browser process private bytes are growing. Where does the memory difference come from?

I am troubleshooting what appears to be a memory leak in our configuration page. The page is used to change the configuration of our service and also displays health diagnostics. This means that we are querying the service periodically for configuration and instrumentation information (typically we use a query interval of 30sec, but to troubleshoot I am querying at 100ms intervals). We rely on knockoutjs, datajs, jquery and spinjs.
I've found that if I leave the page open overnight at the 100ms query interval that the private bytes for the chrome browser tab grows from about 50MB to 335MB. I have four pages with the issue, but am focused on one during my troubleshooting effort. Using chrome://memory-redirect/ I can see the page (process id 26148) memory.
However, the JavaScript heap memory appears to be flat over the same period at 3.6MB. Using the heap profiling tools in Chrome it shows that all of my object allocations are garbage collected.
In the above picture the gray allocations indicate that the objects have been cleaned up by the GC.
The memory timeline also is constant.
I also forced two GCs and confirmed that the number of documents, nodes and listeners was constant between the two GCs.
My questions are:
Where is the process memory being used that is not part of the JavaScript heap?
Given our JavaScript heap memory is flat, could that extra memory be a memory leak caused by our JavaScript code?
Thanks for all the help!
You are comparing apples and oranges - Garbage collected memory in a sub-heap and whole application memory.
You've used Chrome to inspect the JavaScript heap and found evidence that indicates the JavaScript part of your application is running OK.
You're also using a tool to monitor the global memory usage of Chrome itself. That is all the memory that Chrome is using for any task, including tasks not directly related to your application, but to the functioning of the browser itself.
Perhaps you've found a use case that triggers a memory leak in the Chrome internals?
Or perhaps it isn't a memory leak, but memory fragmentation in the non-garbage collected internal heaps used by Chrome?
According to this web page Chrome is written in a mixture of C, C++, Java, JavaScript and Python. This means we have deterministic memory allocators for C and C++ and three different types of garbage collected heaps for Java, JavaScript and Python. Bad news: Python's handling of integers isn't so kind on memory use when it comes to garbage collection (last time I checked, which was a few years ago, maybe they've improved it).
But I've had Chrome sessions run for weeks without issue. So I do wonder what is happening.
You don't say which OS you are using but if you are using Microsoft Windows then you could use C++ Memory Validator to inspect where each allocation was made (full callstack, how many bytes, etc) while Chrome is running (Launch Chrome from C++ Memory Validator, load your applicaton, let it do it's thing then go to the Memory tab and click Refresh - it will display all the live allocations that can be tracked - any statically linked heaps won't be trackable as you won't have the symbols to allow them to be hooked). OK, you don't have symbols to make teh callstacks readable but you can still identify allocations happening at the same place. That may give you a clue as to the cause of the leak/fragmentation so that can report this to the Chrome devs for a closer look.
Do you get the same behaviour in Firefox? If you then you could do what I suggest with C++ Memory Validator but do it on a build of Firefox that you've built yourself - you'll have symbols and source and know exactly where the problem is.
Disclaimer. I am the designer of C++ Memory Validator.

JavaScript Memory: Heap, Native, WebGL/GPU

Is it possible to see all sources of memory for JavaScript in Chrome? As far as I know the three above are what would be available.
The Heap is your basic GC-able JS objects. "Native Memory" is not part of the Heap .. like DOM, TypedArrays, 2D context ImageData and so on. WebGL too is a source of memory.
I'd like to know how much my code is using. Chrome recently dumped their Native profiler. Heap profiling is simply not sufficient for large memory web-apps.
Is there a way to get useful information on what percent of these memory sources my code is using?
I estimate the native memory by using the chrome task manager (more tools | Task Manager). It shows the private memory, GPU memory and javascript memory. private memory - javascript memory would be an approximation of native memory. But it can't indicate how much memory is allocated by different kinds of resources.
It also shows the GPU memory if the page has a canvas.

Categories

Resources