NodeJS: Reached heap limit Allocation failed - JavaScript heap out of memory - javascript

I have a VPS (Debian) with 2 GB RAM where I run pm2 in order to execute a node.js server application. I have a cache and every night, at 3 am I want to dump it into a file as a backup.
Every day at 3, I got the following error:
<--- Last few GCs --->
[717119:0x63345b0] 66921944 ms: Mark-sweep 956.0 (1043.7) -> 954.9
(1043.7) MB, 737.3 / 0.0 ms (average mu = 1.000, current mu = 0.349)
allocation failure; scavenge might not succeed [717119:0x63345b0]
66922975 ms: Mark-sweep 962.8 (1043.7) -> 961.9 (1059.7) MB, 978.0 /
0.0 ms (average mu = 0.999, current mu = 0.052) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap
out of memory 1: 0xb6e500 node::Abort() [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
2: 0xa7e632 [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
3: 0xd47f20 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char
const*, bool) [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
4: 0xd482c7
v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char
const*, bool) [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
5: 0xf25685 [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
6: 0xf37b6d
v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,
v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node
/home/userName/web/appName.com/public_html/appNameServer/stoc> 7:
0xf1226e
v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int,
v8::internal::AllocationType, v8::internal::AllocationOrigin,
v8::internal::AllocationAlignment) [node /home/userName/web/app> 8:
0xf13637
v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int,
v8::internal::AllocationType, v8::internal::AllocationOrigin,
v8::internal::AllocationAlignment) [node /home/userName/web/stoc> 9:
0xef3b80 v8::internal::Factory::AllocateRaw(int,
v8::internal::AllocationType, v8::internal::AllocationAlignment) [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
10: 0xeeb5f4
v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int,
v8::internal::AllocationType, v8::internal::Map,
v8::internal::AllocationAlignment) [node /home/userName/web/stoc>11:
0xeed988
v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int,
v8::internal::AllocationType) [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
12: 0x131ddec v8::internal::IncrementalStringBuilder::Extend() [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
13: 0x10349d8
v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>)
[node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
14: 0x1035c52 v8::internal::JsonStringifier::Result
v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>,
bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>15:
0x10378af v8::internal::JsonStringifier::Result
v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>,
bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>16:
0x10378af v8::internal::JsonStringifier::Result
v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>,
bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>17:
0x10378af v8::internal::JsonStringifier::Result
v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>,
bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>18:
0x10378af v8::internal::JsonStringifier::Result
v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>,
bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>19:
0x10378af v8::internal::JsonStringifier::Result
v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>,
bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>20:
0x10378af v8::internal::JsonStringifier::Result
v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>,
bool, v8::internal::Handle<v8::internal::Object>) [node /home/di>21:
0x1037f13
v8::internal::JsonStringifier::SerializeJSReceiverSlow(v8::internal::Handle<v8::internal::JSReceiver>)
[node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
22: 0x1039897 v8::internal::JsonStringifier::Result
v8::internal::JsonStringifier::Serialize_(v8::internal::Handle<v8::internal::Object>,
bool, v8::internal::Handle<v8::internal::Object>) [node /home/d>23:
0x103ae5f v8::internal::JsonStringify(v8::internal::Isolate*,
v8::internal::Handle<v8::internal::Object>,
v8::internal::Handle<v8::internal::Object>,
v8::internal::Handle<v8::internal::Object>) [node /hom>24: 0xdcdbf7
v8::internal::Builtin_JsonStringify(int, unsigned long*,
v8::internal::Isolate*) [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
25: 0x16e9af9 [node
/home/userName/web/appName.com/public_html/appNameServer/appNameServer.js]
I know the error is straight forward and the trivial answer would be to add more RAM, but I still want to share some more data:
Before the crash, the app consumed 640mb (gross), meaning the cache itself consumes less than that.
When I run this node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))' the result is 1662.
When I run the caching dump locally, it takes around 3 seconds to export the items but viewing pm2 logs on the server, it takes 14 seconds between trying to export the cache items until the server is back up again, as if the export takes much longer or maybe the crash/restart takes that much, not sure.
2022-12-27T03:00:00.016Z: exporting to json 2|appName|
2022-12-27T03:00:14.747Z: server is running on https at port...
Based on the numbers above, 640mb (gross) and the fact that there's at least one more 1gb left for node, I don't understand why there's a heap out of memory in the first place.
Please advise, thanks

NODE_OPTIONS (--max_old_space_size)
In my case
$ node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
4144
but
$ export NODE_OPTIONS=--max_old_space_size=8192 && node -e 'console.log(v8.getHeapStatistics().heap_size_limit/(1024*1024))'
8240
so maybe you can modify amount of available memory using this variable.
Writing file using buffer
Probably you are writing file using
fs.writeFileSync('/tmp/test.txt', content);
but maybe you can switch to
fs.createWriteStream(path, [options])
then you will not have to store all content of file in memory in the same time.
Profiling using IDE
I solved similar cases using tools like this:
https://www.jetbrains.com/help/webstorm/v8-cpu-and-memory-profiling.html#ws_node_cpu_profiling
in console you can simply start your script with --porf flag. It will created huge, hard to read, but very detailed logs telling about memory allocation.

Related

case:27(ESC) doesn't work with microsoft edge

As the title say's, the other cases work in google chrome, firefox, iexplore and opera. Does someone know why this is and what the correct code for edge is so that the case would work and go to another page (that's what #messagebtn does).
Further i want to thank everyone for their time.
document.addEventListener("keyup",function(e){
var key = e.which||e.keyCode;
switch(key){
//enter
case 13:
document.getElementById("messagebtn").click();
break;
//space
case 32:
document.getElementById("messagebtn").click();
break;
//escape
case 27:
document.getElementById("messagebtn").click();
break;
}
});

How to differentiate key event from numpad and barcode reader?

I'm working on a software that is expected to use a barcode reader. Everything worked fine till a customer started using a new barcode reader and some bindings we got in the numpad keyboard started triggering which did not happened before (we actually tried various barcode readers).
I'm pretty sure it can be fixed from the barcode reader configuration but most of our customers are small shops with no knowledge about stuff like that. So it would be really important for us to fix it from the code.
Here's a mixup code between what I got and what I would like to acomplish, obviouslyI'm ignoring parts of the original code to make it easy readable.
$('body').keydown(function(key){
var keyCode = key.keyCode;
switch(keyCode){
case 96:
case 97:
case 98:
case 99:
case 100:
case 101:
case 102:
case 103:
case 104:
case 105:
case 110:
if(not_actually_from_keyboard_but_barcode){
//step out
}
else{
doNumPadStuff(keyCode);
}
break;
});
Would appreciate any kind of solution or work around to the problem.
Maybe the below helps
The numeric keypad may produce different keycodes depending on the NumLock state.
$(document).keyup(function(e) {
/* OPTIONAL: Only if you want other elements to ignore event */
e.preventDefault();
e.stopPropagation();
var keyCode = (e.keyCode ? e.keyCode : e.which);
debuger //check what keyCode is saying. to get this press f12 to get browser debugger.
switch(keyCode){
case 96: //numpad 0
case 97: //numpad 1
case 98: //numpad 2
case 99: //numpad 3
case 100: //numpad 4
case 101: //numpad 5
case 102: //numpad 6
case 103: //numpad 7
case 104: //numpad 8
case 105: //numpad 9
case 110: //numpad .
if(not_actually_from_keyboard_but_barcode){
//step out
}
else{
doNumPadStuff(keyCode);
}
break;
});

My ANTLR4 based C++ parser dies on out of memory error when I try to parse expanded header

Hi ANTLR experts out there, I need your help!
I have been using ANTLR for quite a while and wrote several parsers using it ANTLR4. The version I use now is 4.5.3.
Ok I am writing a c++ parser with Javascript target. Until this time things have been quite smooth. Then I tried to parse iostream.h (actually with a small main and used Xcode llvm to preprocess). It is more than 30000 lines of code of 1.2MB in size. There are errors but those can be fixed. The problem is I get this out of memory error.
375416 ms: Mark-sweep 1349.2 (1422.3) -> 1349.0 (1432.3) MB, 719.3 / 0 ms [allocation failure] [GC in old space requested].
376141 ms: Mark-sweep 1349.0 (1432.3) -> 1349.0 (1433.3) MB, 725.6 / 0 ms [allocation failure] [GC in old space requested].
376900 ms: Mark-sweep 1349.0 (1433.3) -> 1349.0 (1433.3) MB, 758.7 / 0 ms [last resort gc].
377632 ms: Mark-sweep 1349.0 (1433.3) -> 1349.0 (1433.3) MB, 731.9 / 0 ms [last resort gc].
==== JS stack trace =========================================
Security context: 0x973d6c9e59
1: Join(aka Join) [native array.js:179] [pc=0x109304f3a6b5] (this=0x973d604189 ,w=0x109e7ca91511 ,x=108,N=0x1a20734ed919 ,M=0x973d6b4a11 )
2: InnerArrayJoin(aka InnerArrayJoin) [native array.js:~343] [pc=0x109304f1afac] (this=0x973d604189 ,N=0x1a20734ed919 ,w=0...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Abort trap: 6
My main now is like this
try {
text = fs.readFileSync(filepath, 'utf8');
var chars = new antlr4.InputStream(text);
var lexer = new CPP14Lexer.CPP14Lexer(chars);
var tokens = new antlr4.CommonTokenStream(lexer);
var parser = new CPP14Parser.CPP14Parser(tokens);
var printer;
try {
// SLL mode
parser.buildParseTrees = true;
parser.setTrace(this.trace);
var cache = new PredictionContextCache();
var sim = new ParserATNSimulator(parser, parser._interp.atn, parser._interp.decisionToDFA, cache);
sim.parser._interp.predictionMode = PredictionMode.SLL;
var tree = sim.parser.translationunit();
printer = new listener.CPPXXPrinter(tokens, lexer, parser, this.debug);
antlr4.tree.ParseTreeWalker.DEFAULT.walk(printer, tree);
}
catch ...
It tries to build a syntax tree for the entire input. My question is, is there any ways to do things in memory saving mode, for example not creating the tree at all and just parse the input?
Thanks
-Yoshi

coffee script switch without break

Is it possible to use switch in coffeescript without break?
switch code switch (code) {
when 37 then case 37: break;
when 38 then -> case 38: break;
when 39 then case 39: break;
when 40 case 40:
... ...
I thought this will work but failed:
switch code
when 37 then continue
when 38 then continue -> not valid
when 39 then continue
when 40
...
Not really. From the docs:
Switch statements in JavaScript are a bit awkward. You need to remember to break at the end of every case statement to avoid accidentally falling through to the default case. CoffeeScript prevents accidental fall-through, and can convert the switch into a returnable, assignable expression. The format is: switch condition, when clauses, else the default case.
What you can do, though, is specify several values in a case, if they are to be treated equally:
switch day
when "Mon" then go work
when "Tue" then go relax
when "Thu" then go iceFishing
when "Fri", "Sat"
if day is bingoDay
go bingo
go dancing
when "Sun" then go church
else go work
You can use line continuation to help with this. For example:
name = 'Jill'
switch name
when 'Jill', \
'Joan', \
'Jess', \
'Jean'
$('#display').text 'Hi!'
else
$('#display').text 'Bye!'
Check it out in action here.
It's totally possible, just use a classic javascript and pass it through with backtics
`
switch (code) {
case 37:
case 38:
case 39:
case 40:
// do the work of all four
default:
//default
}
`
Old question already, but if you place the commas on the next line, it works as expected, without the backslash line continuation showed by #Ron Martinez
switch code
when 37
, 38
, 39
, 40
console.log "Some Number"
else
console.log "Default"
Which will compile to:
switch (code) {
case 37:
case 38:
case 39:
case 40:
return console.log("Some Number");
default:
return console.log("Default");
}

non-ascii char as arguments

printargv.js:
console.log(Buffer.byteLength(process.argv[2]));
In cmd.exe (with chcp=65001,font='Lucida Console'), I ran:
node printargv.js Ā
(Note: unicode code point of Ā is U+0100.) The script outputted:
1
I expected the script to print a number greater than 1 but it doesn't. Does anyone know why?
edit:
i think that node 'parses' initial arguments incorrectly for cmd.exe after i tried the below code:
var i = require('readline').createInterface(process.stdin,process.stdout);
i.question('char: ', function(c){
console.log( Buffer.byteLength(c) );
i.close();
process.stdin.destroy();
});
the output is 2
Your program is not receiving the Ā, it's receiving an A instead. I used this program to test:
var n;
for (n = 0; n < process.argv.length; ++n) {
console.log(n + ": '" + process.argv[n] + "'");
}
console.log("length: " + process.argv[2].length);
console.log("code: " + process.argv[2].charCodeAt(0));
console.log("size: " + Buffer.byteLength(process.argv[2]));
On Ubuntu using UTF-8 in the console, I got:
$ node test.js Ā
0: 'node'
1: '/home/tjc/temp/test.js'
2: 'Ā'
length: 1
code: 256
size: 2
...which is correct.
On Windows 7 using chcp 65001 and Lucida Console, I got:
C:\tmp>node temp.js Ā
0: 'node'
1: 'C:\tmp\temp.js'
2: 'A'
length: 1
code: 65
size: 1
Note that the Ā became an A at some point along the way.
As I said in my comment on the question, I can only assume there's some issue with Lucida Console, or cmd.exe's handling of UTF-8, or perhaps node.exe's handling of Unicode from the console on Windows (I used the pre-built 0.5.7 version).
Update: This might be something to take up with the NodeJS folks, since Windows appears to get it right on its own. If I put this code in a test.vbs file:
WScript.Echo WScript.Arguments(0)
WScript.Echo AscW(WScript.Arguments(0))
I get a correct result:
C:\tmp>cscript /nologo test.vbs Ā
Ā
256
...suggesting that the terminal is passing the argument correctly to the program. So it could be an issue with the Windows node.exe build.

Categories

Resources