Develop Reference

How to solve nodejs uncaughtException: Connection already released error and MaxListenersExceededWarning?

I am building an express server to receive request (a dict with 10 items) from my react front end and then save the data to database. Below is my code. I found that my code is work and the query does save the record back to Db. But in each for loop, this error is returned in server. What cause this error and the MaxListenersExceededWarning?
The request data:
{{.....}, {.....}, {.....}, {.....}, {.....}} #10 item
Code:
connection.js:
const p = mysql.createPool({
"connectionLimit" : 100,
"host": "example.org",
"user": "test",
"password": "test",
"database": "test",
"multipleStatements": true
});
const getConnection = function(callback) {
p.getConnection(function(err, connection) {
callback(err, connection)
})
};
module.exports = getConnection
routers.js
router.post('/test', (req, res) => {
getConnection(function(err, conn){
if (err) {
return res.json({ success: false, error: err })
} else {
const dict = req.body;
Object.keys(dict).forEach(function(r){
#putting dict's value to query
query = "UPDATE ......;"
conn.query(query, function (err, result, fields) {
conn.release()
console.log(query)
if (err) {
console.log("err")
return res.json({ success: false, error: err });
}
});
});
}
});
return res.json({ success: true });
});
Error:
error: uncaughtException: Connection already released
Error: Connection already released
at Pool.releaseConnection (/home/node_modules/mysql/lib/Pool.js:138:13)
at PoolConnection.release (/home/node_modules/mysql/lib/PoolConnection.js:35:15)
at Query.<anonymous> (/home/routes/test.js:276:22)
at Query.<anonymous> (/home/node_modules/mysql/lib/Connection.js:526:10)
at Query._callback (/home/node_modules/mysql/lib/Connection.js:488:16)
at Query.Sequence.end (/home/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
at Query._handleFinalResultPacket (/home//node_modules/mysql/lib/protocol/sequences/Query.js:149:8)
at Query.OkPacket (/home//node_modules/mysql/lib/protocol/sequences/Query.js:74:10)
at Protocol._parsePacket (/home//node_modules/mysql/lib/protocol/Protocol.js:291:23)
at Parser._parsePacket (/home//node_modules/mysql/lib/protocol/Parser.js:433:10)
(node:15881) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 finish listeners added. Use emitter.setMaxListeners() to increase limit

One connection (conn) is being retrieved from the pool, and is used to launch 10 queries in the forEach loop.
When the first query finishes to run, the first step of its callback is: conn.release(). The connection is released.
When the second query finishes to run, its callback also tries to release the connection,causing the error.
This problem might be solved in multiple ways:
Solve using a counter
In the callback of the database query, before calling call.release, check the number of queries already processed, and only close the connection when the last product is being processed.
const dict = req.body;
// initialize counter
let itemCount = 0
, errors = []
Object.keys(dict).forEach(function(r){
#putting dict's value to query
query = "UPDATE ......;"
conn.query(query, function (err, result, fields) {
// check whether this is the last callback
if (itemCount === dict.length-1) {
conn.release()
let result = errors.length ? { success: false, error: errors } : { success: true }
res.json(result)
}
// increment counter
itemCount++
console.log(query)
if (err) {
console.log("err")
errors.push(err)
}
});
});
Edit: There is also an issue with the res.json calls: inside the code in the question, res.json({ success: true }) is always executed, without waiting for the queries' execution results. The modified code sample above calls res.json only once after the execution of all queries, this is the only place where res.json should be called. This implies modifying the client-side code so that it can handle an array of errors, rather than only one error.
Solve by using a recursive function instead of for loop.
It is not a good practice to use for loops for the execution of asynchronous code. You might run into Maximum call stack size exceeded errors whenever the data volume gets too large.
Instead, create a recursive function (e.g. updateDictItem) to process one update query at a time. Read more about the asynchronous patterns in node.js in this article.
Other possible enhancements
Rather than firing ten database queries, it is worth considering grouping all the updates in one MERGE update statement, otherwise doing all the updates in a TRANSACTION.

How to get the exit code from a file ran using javascript child_process.execFile

Here is my python code:
#!/bin/python
import sys
sys.exit(4)
Here is my javascript
var exec = require('child_process').execFile
exec('./script.py', (err, data) => { if(err) { console.log(err.status) } })
But it doesn't work because there is not such thing as err.status
what I want to have in my console logs is '4'.

There isn't err.status, but there is err.code.
This should work:
var exec = require('child_process').execFile
exec('./script.py', (err, data) => { if(err) { console.log(err.code) } })
There is also err.signal and err.killed
From the nodejs docs:
On success, error will be null. On error, error will be an instance of
Error. The error.code property will be the exit code of the child
process while error.signal will be set to the signal that terminated
the process. Any exit code other than 0 is considered to be an error.
(https://nodejs.org/api/child_process.html#child_process_child_process_execfile_file_args_options_callback)

execFile's callback get 3 arguments:
error, if there was.
stdout, the stdout of process.
stderr, the stderr of the process.
So, by checking the stderr, you should be able to achieve the expected result:
var exec = require('child_process').execFile
exec('./script.py',(err,stdout,stderr)=>{console.log(stderr)})
See this in the Node.js docs

Trouble with callbacks, error catching and MongoDB

I've been working on an application which allows me to add companies to a database. Originally my code was pure spaghetti, so I wanted to modularize it properly. For this purpose, I added routes, a controller and a dao.
This is how my code looks right now
Routes
app.post('/loadcompanies', (req, res)=> {
companiesController.loadcompany(req.body, (results)=>{
console.log(results);
res.send(200, "working!");
})
})
Controller
module.exports.loadCompany = (body, callback)=>{
companiesDao.loadCompany(body, callback);
}
Dao
module.exports.loadCompany = (company, callback)=>{
MongoClient.connect(conexionString, (err, database) => {
if (err) console.log(err);
db = database;
console.log(company);
db.collection('companies').insert(company, (err, result)=>{
callback({message:"Succesfully loaded company", company:result});
});
})
}
My current concern is that working with errors when modularizing like this is confusing. I tried adding a try-catch method around the db insert and throwing and error if there is one, but that doesn't seem to work. Other things I've tried is returning the error in the callback, like this:
if (err) callback (err, null);
but I end up getting a "Can't set headers after they are sent." error.
How would you handle errors in this situation? For example, in the case that someone tries to add a duplicate entry in an unique element.

You should be able to simply do the error checking inside the callback for the insert function:
db.collection('companies').insert(company, (err, result)=>{
if (err) {
callback(err, null);
return;
}
callback(null, {message:"Succesfully loaded company", company:result});
});
If you get an error like you say, that's probably because the database is actually returning an error. You could also make your errors more specific, like:
module.exports.loadCompany = (company, callback)=>{
MongoClient.connect(conexionString, (err, database) => {
if (err) {
callback(new Error('Connection error: ' + err.Error());
return;
}
db = database;
console.log(company);
db.collection('companies').insert(company, (err, result)=>{
if (err) {
callback(new Error('Insertion error: ' + err.Error());
return;
}
callback(null, {message:"Succesfully loaded company", company:result});
});
})

Here is your loadCompany done in async / await format.
Notise there is no need for error checking, errors will propagate as expected up the promise chain.
Note I've also changed loadCompany to be an async function too, so to call it you can simply do var ret = await loadCompany(conpanyInfo)
module.exports.loadCompany = async (company)=>{
let db = await MongoClient.connect(conexionString);
console.log(company);
let result = await db.collection('companies').insert(company);
return {message:"Succesfully loaded company", company:result};
}

MongoDB cursor.each throws error

I am trying to execute the following nodejs/mongoDB code:
var tickers = [];
MongoClient.connect(mongoUrl, function(err, db) {
assert.equal(null, err);
console.log("Connected correctly to server."); //ok
var cursor = db.collection('STI').find();
cursor.each(function(err, doc){
assert.equal(err, null);
console.log(doc.symbol); //executes fine
tickers.push(doc.symbol);
})
});
console.log(tickers);
The symbols are logging out fine to the console but after that the code throws an error 'TypeError: Cannot read property 'symbol' of null'.
The code seems to forget the doc.symbol by the time it gets to executing the 'tickers.push' part. How do I fix this?
Update:
I tried shifting the console.log(tickers) into the cursor.each callback and it prints out the array which each iteration, so the symbol pushing is happening. however i still get the same error
Update: full error message
/Users/kevin/Projects/yahooscrape/node_modules/mongodb/lib/utils.js:98
process.nextTick(function() { throw err; });
^
TypeError: Cannot read property 'symbol' of null
at /Users/kevin/Projects/yahooscrape/index.js:21:19
at handleCallback (/Users/kevin/Projects/yahooscrape/node_modules/mongodb/lib/utils.js:96:12)
at /Users/kevin/Projects/yahooscrape/node_modules/mongodb/lib/cursor.js:736:16
at handleCallback (/Users/kevin/Projects/yahooscrape/node_modules/mongodb/lib/utils.js:96:12)
at /Users/kevin/Projects/yahooscrape/node_modules/mongodb/lib/cursor.js:670:5
at handleCallback (/Users/kevin/Projects/yahooscrape/node_modules/mongodb-core/lib/cursor.js:154:5)
at setCursorDeadAndNotified (/Users/kevin/Projects/yahooscrape/node_modules/mongodb-core/lib/cursor.js:463:3)
at nextFunction (/Users/kevin/Projects/yahooscrape/node_modules/mongodb-core/lib/cursor.js:644:7)
at Cursor.next [as _next] (/Users/kevin/Projects/yahooscrape/node_modules/mongodb-core/lib/cursor.js:685:3)
at nextObject (/Users/kevin/Projects/yahooscrape/node_modules/mongodb/lib/cursor.js:655:8)

Use cursor.forEach() instead of cursor.each(). Cursor.forEach() is an officially implemented mongodb method that does not throw the errors shown in the question.

POST response error after collection.insert with NodeJS Mongo module

I've got this error when trying to POST
> process.nextTick(function() { throw err; });
> ^
>
> TypeError: first argument must be a string or Buffer
> at ServerResponse.OutgoingMessage.end (_http_outgoing.js:524:11)
Errors shows that something's wrong with utils and cursor both from mongodb module, but what are they?
Everything works nice on GET but brakes on POST (postman and passing as text {"name":"Computer","price":2500}) - i cannot trace which module or instance is braking the code.
This is my conn with db:
// Our primary interface for the MongoDB instance
var MongoClient = require('mongodb').MongoClient;
// Used in order verify correct return values
var assert = require('assert');
var connect = function (databaseName, callBack) {
var url = 'mongodb://localhost:27017/' + databaseName;
MongoClient.connect(url,
function (error, database) {
assert.equal(null, error);
console.log("Succesfully connected to MongoDB instance!");
callBack(database);
});
};
exports.find = function (databaseName, collectionName, query, callback) {
connect(databaseName, function (database) {
var collection = database.collection(collectionName);
collection.find(query).toArray(
// Callback method
function (err, documents) {
// Make sure nothing went wrong
assert.equal(err, null);
// Print all the documents which we found, if any
console.log("MongoDB returned the following documents:");
console.dir(documents)
callback(err, documents);
// Close the database connection to free resources
database.close();
})
})
};
exports.insert = function (databaseName, collectionName, object, callback) {
connect(databaseName, function (database) {
var collection = database.collection(collectionName);
collection.insert(document, {w: 1}, function (err, documents) {
console.log("Added a new document");
console.log(documents[0]);
callback(err, documents[0]);
});
})
};
exports.remove = function (databaseName, collectionName, object, callback) {
connect(databaseName, function (database) {
var collection = database.collection(collectionName);
collection.remove(object, function (err, result) {
callback(err, result);
database.close();
});
})
};

The issue is actually pretty straightforward, so I'm surprised that you're not getting a better error message.
In your code:
collection.insert(document, {w: 1}, function (err, documents) {
console.log("Added a new document");
console.log(documents[0]); // I expect this to log undefined
callback(err, documents[0]);
});
The second argument passed into the collection.insert callback is actually a results object, not the documents that were inserted. So, documents[0] ends up being undefined because it's not an array of documents. Thus, when you trying to send undefined as a response, it's failing.
If you intention is to pass the newly created documents, you're going to have to use the result object to get the _id and attach it to the document you inserted.
As a side note, I would consider keeping a connection open to your database rather than creating a new connection every time you want to talk with Mongo.