Node.js not closing files created by fs.createReadStream() - javascript

On my server, every time a user uses our service we have to grab a JSON file for them from the server. I do this by using fs.createReadStream() inside of my own function.
function getJSONFromServer(filepath, callback){
var data = fs.createReadStream(filepath);
data.on('error', function (error) {
console.log("Caught", error);
callback(undefined, error);
});
var jsonFile = "";
data.on('data', function(chunk) {
jsonFile += chunk;
});
data.on('end', function() {
var jsonData = JSON.parse(jsonFile);
callback(jsonData);
data.destroy();
data.close();
});
}
This does the job, but it does not close the connection to the file. So after reading 1024 files (the limit on my server), Node.js will then produce the error EMFILE, too many open files. Then I have to kill our Node.js server, open it again and that will clear the "open files".
I check the amount of files open by lsof -i -n -P | grep nodejs. It displays something like this:
nodejs 13707 node 10u IPv4 1163695 0t0 TCP 127.0.0.1:55643->127.0.0.1:27017 (ESTABLISHED)
nodejs 13707 node 11u IPv4 1163697 0t0 TCP 127.0.0.1:55644->127.0.0.1:27017 (ESTABLISHED)
for as many files that are open.
I've tried using graceful-fs. I've tried calling stream.destroy() and stream.close(), but I still get the same issue. My server is essentially a ticking time bomb because we get a heavy, steady flow of users and after so many users have connected it will just stop working.
Also, ulimit -n [open file amount] does not work, and even if it did, this is not a long term solution because I'd like my file connections to close and not sit open for no reason.
I'm using Node.js version v0.10.25, Ubuntu 15.04 (GNU/Linux 3.19.0-42-generic x86_64) and the latest version of graceful-fs if that helps!
Thanks for any help you can provide.

This has got to be the stupidest mistake I've ever made. Regardless, here's the answer. I hope I can save someone from dealing with this error and almost ripping their hair out.
I was running my app with nodejs and not node. Turns out, if you do nodejs --version, it will likely return a version that is very old, which was v0.10.25 for me. node --version however was v5.6.0. Obviously this massive jump in versions would fix some stuff, so I ran the app with node app.js instead of nodejs app.js and I haven't had the issue at all since. There are now only 6 open files, whereas before we had over 1000 with time.
Damn it feels good to have this off my chest.

Related

Node JS ctrl + C doesn't stop server (after starting server with "npm start")

When I start my server with node app.js in the command line (using Git Bash), I can stop it using ctrl + C.
In my package.json file i got this start-script that allows me to use the command npm start to start the server:
"scripts": {
"start": "node app"
},
When I do this, the server starts as normal:
$ npm start
> nodekb#1.0.0 start C:\Projects\nodekb
> node app.js
Server started on port 3000...
But when i ctrl + C now, the server does not get stopped (the node process still remains in task manager). This means that I get an error when I try to do npm start again, because port 3000 is still being used.
I'm following a tutorial on youtube (video with timestamp), and when this guy ctrl + C and then runs npm start again, it works as normal.
Any ideas why my server process is not stopped when I use ctrl + C?
My app.js file if needed:
var express = require("express");
var path = require("path");
//Init app
var app = express();
//Load View Engine
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "pug");
//Home Route
app.get("/", function(req, res) {
res.render("index", {
title: "Hello"
});
});
//Add route
app.get("/articles/add", function (req, res) {
res.render("add_article", {
title: "Add Article"
});
});
//Start server
app.listen(3000, function() {
console.log("Server started on port 3000...");
});
Thanks!
Ctrl + C does not kill the server. The resolution to the issue was using following code snippet in server.js:
process.on('SIGINT', function() {
console.log( "\nGracefully shutting down from SIGINT (Ctrl-C)" );
// some other closing procedures go here
process.exit(0);
});
This worked for me.
You can also check for other solutions mentioned at Graceful shutdown in NodeJS
I tried it on normal windows cmd, and it worked as it should there. Looks like it's a problem with git bash.
I encountered this problem in MSYS2 proper, even in latest build (x64 2018-05-31).
Luckily, Git for Windows maintain a customized MSYS2 runtime. They have patches that have not been sent upstream, including a patch that fixes emulation of SIGINT, SIGTERM and SIGKILL.
Discussion: https://github.com/nodejs/node/issues/16103
I was able to make my "MSYS2 proper" platform use Git for Windows' MSYS2 runtime, by following these instructions.
Repeated here for posterity:
Install inside MSYS2 proper
This guide assumes that you want the 64-bit version of Git for Windows.
Git for Windows being based on MSYS2, it's possible to install the git package into an existing MSYS2 installation. That means that if you are already using MSYS2 on your computer, you can use Git for Windows without running the full installer or using the portable version.
Note however that there are some caveats for going this way. Git for Windows created some patches for msys2-runtime that have not been sent upstream. (This had been planned, but it was determined in issue #284 that it would probably not be happening.) This means that you have to install Git for Windows customized msys2-runtime to have a fully working git inside MSYS2.
Here the steps to take:
Open an MSYS2 terminal.
Edit /etc/pacman.conf and just before [mingw32] (line #71 on my machine), add the git-for-windows packages repository:
[git-for-windows]
Server = https://wingit.blob.core.windows.net/x86-64
and optionally also the MINGW-only repository for the opposite architecture (i.e. MINGW32 for 64-bit SDK):
[git-for-windows-mingw32]
Server = https://wingit.blob.core.windows.net/i686
Authorize signing key (this step may have to be repeated occasionally until https://github.com/msys2/msys2/issues/62 is fixed)
curl -L https://raw.githubusercontent.com/git-for-windows/build-extra/master/git-for-windows-keyring/git-for-windows.gpg |
pacman-key --add - &&
pacman-key --lsign-key 1A9F3986
Then synchronize new repository
pacboy update
This updates msys2-runtime and therefore will ask you to close the window (not just exit the pacman process). Don't panic, simply close all currently open MSYS2 shells and MSYS2 programs. Once all are closed, start a new terminal again.
Then synchronize again (updating the non-core part of the packages):
pacboy update
And finally install the Git/cURL packages:
pacboy sync git:x git-doc-html:x git-doc-man:x git-extra: curl:x
Finally, check that everything went well by doing git --version in a MINGW64 shell and it should output something like git version 2.14.1.windows.1 (or newer).
Note: I found that the git-extra package installed by step 7 was quite intrusive (it adds a message "Welcome to the Git for Windows SDK!" to every terminal you open), so I removed it with pacman -R git-extra.
Note 2: I also found that Git for Windows' MSYS2 runtime opens in a different home directory than did MSYS2 proper's. This also means it reads in the wrong bash profile. I fixed this by adding an environment variable to Windows in the Control Panel: HOME=/C/msys64/home/myusername
I use git bash on my Windows machine and have run into this issue in the last month or so.
I still do not know what's causing it but I've found another way to stop it.
Open Task Manager
Go into the Processes tab
Look for node.exe and then press End Process
This has allowed me to stop the server quickly.
I had the same problem working with npm. But finally, I knew it was a problem with git itself.
There was a comment by dscho on GitHub 15 days ago. He said that they're working to fix this problem in the next release. He also shared the exact msys-2.0.dll file that can fix the problem for the people who can't wait.
Personally, I couldn't wait :p. So, I gave it a try, downloaded the file, and throw it in the git folder as he said. And the problem gone! It was awesome!
But please be sure to take a backup before you replace the file.
I also tried to kill it after running express as I used to; using taskkill /im node.exe on the cmd but there was no process to be found.
Check out this issue on GitHub,and search for the name of the file msys-2.0.dll to get to the comment faster.
Sometimes the node process hangs.
Check for the process ID using ps You may want to grep for node and then kill the process using kill -9 [PID]
Use Ctrl+\ to send the SIGQUIT signal. It will close the server.
Reference - https://en.wikipedia.org/wiki/Signal_(IPC)
I was able to fix this by switching to nodemon to run the server.
npm install --save-dev nodemon
package.json:
"scripts": {
"start": "nodemon app"
},
I was trying to get json-server to quit a custom server script, but it always left a child process running on Windows. It seems to be a specific problem running express via npm on Windows. If you run the server directly via the c:>node server.js then it seems to quit correctly.
I was able to debug this issue by checking the ports using TCP View, and realizing that my Node server was running even though I had pressed ctrl-C to stop it. I suggest killing the terminal you are running node from entirely.
Use Ctrl + C, then input: >pm2 stop all
This will stop all server or when you get stack with nodejs.
Inside package.json under scripts I had this line react-scripts start&. Notice it ends with an & which would send the process to the background and ctrl+c will not work. Somehow trying to bring this to the foreground with fg also did not work. Solved the problem by removing the &.
if you use Node.js Exec Extention to run your project from f8,
you can use also f9 to cancel running..
This is more than likely just a problem with your console not accurately sending the command to the process. This is pretty common, especially when using third party consoles like cmdr / conemu.
The solution?
Just hit ctrl+c several times until it closes :P

MongoDB Cloud9 Connection

So, I am wondering if there is a way to connect to the mongoDB I have setup in my Cloud9 from an html. I mean, I have already connected to the db from the terminal and everything is working like a charm but I need to do some stuff inside my script in an html document and when I try calling the function which contains this code it does nothing
var MongoClient = require('mongodb').MongoClient
, format = require('util').format;
MongoClient.connect('mongodb://127.0.0.1:27017/ingesoft', function (err, db) {
if (err) {
throw err;
} else {
console.log("successfully connected to the database");
}
db.close();
});
I have saved the same code into a "file.js" and ran it from console using node file.js and it outputs into the console log "successfully connected to the database", plus the terminal which is running mongo's connection shows me one more connection to the db. The thing is, when I try to run that code inside my script it doesn't work. Sorry for my ignorance I am new to mongo.
Any help would be much appreciated
To simplify your question, here's what's going on:
node file.js containing the code in your question is working
pasting the same code to your html file is not
So, getting to the bottom of the issue, let's ask first: what's the difference between running node file.js and putting the code in html?
The difference is that node ... is running on your Cloud9 workspace (let's call it the server machine).
Your MongoDB server is also running on that server machine
The mongodb npm package you installed is also present on the server machine
The url: mongodb://127.0.0.1:27017/ingesoft references 127.0.0.1 which is the localhost for your server
whereas with the code on your browser:
The code is being run on your customer's machine
That machine doesn't have your Mongodb server
Browser's usually don't support require
You can do requires if you bundle code and use something like webpack or browserify. Did you perhaps do that?
If you did indeed package everything, was the mongodb package that you're requiring packaged?
Can the mongodb package be run from the client side?
The url: mongodb://127.0.0.1:27017/ingesoft references 127.0.0.1 which is the localhost for your customer's machine
Basically, as you can see from the above, the two are very different.
If you want to talk to your db, a lot of people go the following route:
Make a server application that implements some form of REST API
That REST API talks to your DB
Your client code knows how to talk to the REST API and get the required data
That way, you only talk to your MongoDB using your server, and the client can talk to your server via the internet.
This is, of course, an oversimplification, but I hope this resolves your confusion.

Is ZeroMQ for Node.js compatible with Electron?

I am have a huge headache from trying to get the ZMQ Node bindings working with Electron, especially on Windows. I am working on Windows 7 and Ubuntu 16.04 and both of them have two separate issues.
On Windows, I get an error when I try to do require('zmq')
C:\vueelectron\app\node_modules\bindings\bindings.js:91 Uncaught Error: Could not locate the bindings file. Tried:
→ C:\vueelectron\app\node_modules\zmq\build\zmq.node
→ C:\vueelectron\app\node_modules\zmq\build\Debug\zmq.node
→ C:\vueelectron\app\node_modules\zmq\build\Release\zmq.node
→ C:\vueelectron\app\node_modules\zmq\out\Debug\zmq.node
→ C:\vueelectron\app\node_modules\zmq\Debug\zmq.node
→ C:\vueelectron\app\node_modules\zmq\out\Release\zmq.node
→ C:\vueelectron\app\node_modules\zmq\Release\zmq.node
→ C:\vueelectron\app\node_modules\zmq\build\default\zmq.node
→ C:\vueelectron\app\node_modules\zmq\compiled\6.1.0\win32\x64\zmq.node
I've tried compiling with VS 2013 and 2015, rebuilt multiple times, used electron-rebuild nothing seems to be working.
On Linux it loads up fine but the problem is that when I send a message, it seems to get stuck in a loop somewhere and it keeps sending sending hundreds of messages and goes on doing that indefinitely. This was resolved by upgrading from the version of ZMQ in the Ubuntu repositories to the latest one downloaded from the ZeroMQ website.
This is the code I used in my index.html file of my Electron app.
const electron = require('electron')
const zmq = require('zmq')
const socket = zmq.socket('req')
socket.connect('tcp://10.10.0.51:3111')
socket.on('message', function (data) {
console.log(socket.identity + ': answer data ' + data)
})
socket.send('test')
Has anyone else been able to get Electron + ZMQ working? If so, what is your development enviroment like? Thanks.
The problem is the unmatched node.js binary that is delivered by Electron and your version of node. The long answer is that you need to compile Electron and ZeroMQ with the same Node.js headers. Here is the response from Electron community http://github.com/electron/electron/issues/6805. There's a short answer now though!
Use zeromq in place of zmq (same API). zeromq provides prebuilt binaries for electron and node.js for OS X, Windows, and macOS/OS X. After installing zeromq, rebuild for the version of electron you're using:
npm rebuild zeromq --runtime=electron --target=1.4.5
Thanks to the zeromq.js team and have fun with ZeroMQ!
It might be safer to put access to your queue behind an api layer. You might have better success with stability too, native modules in electron can be very tricky.
And but that I mean have a REST server which your electron application communicates with. It would send a message to that api, which then queues the message for your application. Restrict access to the queue at the network level to only the api server.

Node.js - Server crashes when started on startup, doesn't crash when manually started

I am running Node.js and Socket.io for online chat.
I have created a file in:
/etc/init/example.conf
it has two lines:
start on startup
exec forever start /var/www/example.com/html/server.js //location of server file.
Whenever I start file upload in chat application, it crashes but instantly restarts.
Whenever I kill node process though, and start it manually - it works fine.
I also can't get any logs or anything from terminal as when it's auto started - it doesn't print me anything to terminal.
I am still new to Node.js and Linux in general.
Node.js is running on Express + Jade.
How do I determine specific cause?
I managed to solve my problem, after a bit of searching around I found out about tail command.
My issue was a bit harder to trace because Node.js was a process started by autostart so when I launched terminal and connected to server, process was just running in background basically and I wouldn't get any output (including exception messages).
Anyway, solution that worked for me was:
I typed
ps aux | grep node //to find PID of node process
then I went to following directory
cd /proc/[pid of running node service]/fd
In fd directory there are few objects you can get output from but if you want to attach and listen to servers output including uncaught exceptions, you need 1.
So:
tail -f 1
that way I was able to cause website to crash and see the output.

How to write first application in Node.js in Windows Environment?

I have downloaded Node.js from their site in my windows xp os. I have installed it through the microsoft installer. I didn't know how to write my first application, where to save them and how to execute them in Windows. I have got a node.js command prompt but I can't use it.
I have searched a lot but there is only instruction for linux and mac. I didn't find any suitable tutorial or example that I can start a node application from scratch.
If anybody can put some documentation or steps or any such tutorial where I can get help of this, it will be great for me.
As this blog entry by Eric, it is ridiculously easy to get a node.js server setup and responding to requests on 127.0.0.1:#port#.
Really easy. I just did it all in... Longer than write this text.
Download an OS-appropriate node.js1: http://nodejs.org/
Now, create a .txt (plaintext) file in the same folder as your node.exe and rename that to server.js.
Put this Javascript in that file:
var http = require('http');
http.createServer(function (request, response) {
console.log('request starting...');
response.writeHead(200, { 'Content-Type': 'text/html' });
var html = '<p>Hello World!</p>';
response.end(html, 'utf-8');
}).listen(8125);
console.log('Server running at http://127.0.0.1:8125/');
Open a cmd.exe and cd c:\path\to\the\node\executable, where node.exe resides.
From there, run:
node server.js
You should see it say:
Server running at http://127.0.0.1:8125
Open a browser and put http://127.0.0.1:8125/ in the address bar.
You should see it say:
Hello World!
That's it. That's pretty easy. The proverbial Hello World! in 15 seconds or less. Now how to get it to parse a Javascript file and return that output instead of simple HTML...
1. Note, I got the executable, node.exe, made a folder somewhere, put the executable in that folder. No installer. Works like a charm.
Here is the new way develop node.js app with Windows Environments and Visual Studio >= 2012
https://nodejstools.codeplex.com/

Categories

Resources