Develop Reference

Error "Given transaction number * does not match" in mongodb and nodejs

I want to modify two schema while adding data. For that I used ACID transaction of mongodb with nodejs as follow. But, when I run program it displays the error like
(node:171072) UnhandledPromiseRejectionWarning: MongoError: Given transaction number 3 does not match any in-progress transactions. The active transaction number is 2
at MessageStream.messageHandler (/home/user/Projects/project/node_modules/mongodb/lib/cmap/connection.js:272:20)
at MessageStream.emit (events.js:375:28)
at MessageStream.emit (domain.js:470:12)
addData = async(request: Request, response: Response) => {
const session = await stockSchema.startSession()
try {
const userData = request.body
let data = {}
const transaction = await session.withTransaction(async() => {
try {
userData.products.map(async(item: any) => {
await inventorySchema.findOneAndUpdate({ _id: item.materialID }, { $inc: {qty: -item.qty }}, { session });
});
data = new stockSchema(userData);
await data.save({ session });
} catch (error) {
await session.abortTransaction()
throw new Error("Could not create data. Try again.");
}
});
if (transaction) {
session.endSession()
return returnData(response, data, 'Data created successfully.');
} else {
throw new Error("Could not create data. Try again.");
}
} catch (error: any) {
session.endSession();
return Error(response, error.message, {});
}
}

So you might have figured out the answer to this already, but anyway, after months of frustration, and no clear answer on the internet, i finally figured it out.
The problem with your code above is that you are passing session into a database operation (the .findOneAndUpdate function above) that is running within .map . Meaning, your 'transaction session' is being used concurrently, which is what is causing the error. read this: https://www.mongodb.com/community/forums/t/concurrency-in-mongodb-transactions/14945 (it explains why concurrency with transactions creates a bit of a mess.)
Anyway, instead of .map, use a recursive function that fires each DB operation one after another rather than concurrently, and all your problems will be solved.
You could use a function something like this:
const executeInQueue = async ({
dataAry, //the array that you .map through
callback, //the function that you fire inside .map
idx = 0,
results = [],
}) => {
if (idx === dataAry.length) return results;
//else if idx !== dataAry.length
let d = dataAry[idx];
try {
let result = await callback(d, idx);
results.push(result);
return executeInQueue({
dataAry,
callback,
log,
idx: idx + 1,
results,
});
} catch (err) {
console.log({ err });
return results.push("error");
}
};

Nodejs, close mongo db connection via callback

I have the problem with callbacks, async thinking etc.
Execution program:
Connect to mongoDb.
Create url - https://example.com + add part from locArray.
Send get request (for each).
Save data to mongo db.
Close connection.
Problem:
If the connection was closed on last line in jsonDataFromApi - "server instance pool was destroyed" before all data from each request was saved to db
So callback(db) was sent to another place - closeMongoDb
but error was appeared
"Cannot read property 'close' of undefined".
I think, the problem is with async, send callbacks etc.
const MongoClient = require('mongodb').MongoClient;
const Array = require('node-array');
const request = require('request');
var locationArray = [
'location1',
'location2',
'location3',
'location4'
];
var dataFromLocApi = (loc, callback) => {
request({
url: `https://example.com/${loc}`,
json: true
}, (error, response, body) => {
if (error){
callback('Error connection to url.');
} else{
callback(undefined, body.result);
}
});
};
var jsonDataFromApi = (urldb, callback) => {
MongoClient.connect(urldb, (err, db) => {
if (err) {
console.log('MongoDb connection error.');
}
console.log('MongoDb - connected.');
locationArray.forEachAsync(function(loc, index, arr) {
dataFromLocApi(loc, (errorMessage, results) => {
if (errorMessage) {
console.log(errorMessage);
} else {
console.log(JSON.stringify(results, undefined, 2));
db.collection('testCollection').insert(results, function(error, record) {
if (error)
throw error;
console.log("data saved");
});
}
});
}, function() {
console.log('complete');
});
callback(db);
});
}
var closeMongoDb = (urldb, callback) => {
jsonDataFromApi(urldb, (error, db) => {
if (error){
callback('Close connection - failure');
} else{
db.close();
console.log('MongoDb connections was closed.');
}
});
}
closeMongoDb('mongodb://127.0.0.1:27017/testDb', (err, db) => {
console.log('DONE');
} );

There is definitely a problem with asynchrony there.
You're not waiting for the items to be processed before calling the db.close().
Also, the functions that you have defined have the unclear semantics. For example, the function closeMongoDb should basically close the DB and that's it. But here does the other job: fetches the data and closes the DB afterwards.
Also, I'd probably use the async module instead of node-array as the last one seems to solve other problem.
I've refactored the code. Please read my comments. I tried to make it as clear as possible.
const MongoClient = require("mongodb").MongoClient;
const request = require("request");
// We are going to use the async module
// This is a classical module to handle async behavior.
const async = require("async");
// As you can see this function accepts a callback
// If there is an error connecting to the DB
// it passes it up to the caller via callback(err)
// This is a general pattern
const connectToDb = function(urldb, callback) {
MongoClient.connect(urldb, (err, db) => {
if (err) {
console.log("MongoDb connection error.");
callback(err);
return;
}
// If everything is OK, pass the db as a data to the caller.
callback(undefined, db);
});
};
// This method fetches the data for a single location.
// The logic with errors/data is absolutely the same.
const getData = (loc, callback) => {
request(
{
url: `https://example.com/${loc}`,
json: true
},
(error, response, body) => {
if (error) {
callback("Error connection to url.");
return;
}
callback(undefined, body.result);
}
);
};
// This function goes over each location, pulls the data and saves it to the DB
// Last parameter is a callback, I called it allDataFetchedCb to make it clear
// that we are calling it after ALL the locations have been processed
// And everything is saved to the DB.
const saveDataFromLocations = function(locations, db, allDataFetchedCb) {
// First param here is an array of items
// The second one is an async function that we want to execute for each item
// When a single item is processed we call the callback. I named it 'locProcessedCB'
// So it's clear what happens.
// The third parameter is a callback that is going to be called when ALL the items
// have been processed.
async.each(
locations,
function(loc, locProcessedCb) {
getData(loc, (apiErr, results) => {
if (apiErr) {
console.log(apiErr);
// Well, we couldn't process the item, pass the error up.
locProcessedCb(apiErr);
return;
}
console.log(
`Obtained the data from the api: ${JSON.stringify(
results,
undefined,
2
)}`
);
db.collection("testCollection").insert(results, function(dbError) {
if (dbError) {
// Also an error, we couldn't process the item.
locProcessedCb(dbError);
return;
}
// Ok the item is processed without errors, after calling this
// So we tell the async.each function: ok, good, go on and process the next one.
locProcessedCb();
});
});
},
function(err) {
// We gonna get here after all the items have been processed or any error happened.
if (err) {
allDataFetchedCb(err);
return;
}
console.log("All the locations have been processed.");
// All good, passing the db object up.
allDataFetchedCb(undefined, db);
}
);
};
// This function is an entry point.
// It calls all the above functions one by one.
const getDataAndCloseDb = function(urldb, locations, callback) {
//Well, let's connect.
connectToDb(urldb, (err, db) => {
if (err) {
callback(err);
return;
}
// Now let's get everything.
saveDataFromLocations(locations, db, (err, db) => {
if (err) {
callback(err);
return;
}
// If somehow there is no db object, or no close method we wanna know about it.
if (!db || !db.close) {
callback(new Error("Unable to close the DB Connection."));
}
// Closing the DB.
db.close(err => {
// If there's no error err === undefined or null
// So this call is equal to callback(undefined);
callback(err);
});
});
});
};
const locationArray = ["location1", "location2", "location3", "location4"];
// Finally calling the function, passing all needed data inside.
getDataAndCloseDb("mongodb://127.0.0.1:27017/testDb", locationArray, err => {
if (err) {
console.error(
`Unable to fetch the data due to the following reason: ${err}`
);
return;
}
console.log("Done successfully.");
});
I didn't run this code as I don't have the URL etc. So please try it yourself and debug if needed.

Break promise map if throw error inside the loop

I'm using Hapi Js and Sequelize to do something about my API and in this case I need to check everything first before go the next step.
This is my code
return promise.map(array, function (values) {
models.Goods.find({
where: {
id: values.id
}
}).then(function (res) {
if (!res || res.length === 0) {
throw new Error("Item not found");
}
});
}).then(function (res) {
//do something after check
//next step
}).catch(function(error) {
console.log(error.message);
});
I need to check if the id is in my database or not before go the next step but in this code if there is any error the throw new Error("Item not found"); never go to the catch function so I try to do something to get the error function. I changed the code inside the promise.map, I put catch function in models.Goods and console.log the error, the error shows up but the promise.map still running and go to the //next step section and not stop.
Please help me how to break the promise.map if there is an error in models.Goods
Thank you

I think you only have forgotten to return the models, this
return promise.map(array, function (values) {
models.Goods.find({
where: {
should be:
return promise.map(array, function (values) {
return models.Goods.find({
where: {
you could omit the return key word if using arrow functions.
Here is an example, I also put in some object destructuring.
return promise.map(array, ({id}) =>
models.Goods.find({
where: {id}
}).then(res => {
if (!res || res.length === 0) {
throw new Error("Item not found");
}
}) // can't have ; here now
).then(res => {
// do something after check
// next step
}).catch(error => {
console.log(error.message);
});

When the user is not found the query itself was successful, so the success callback is triggered. But since nothing matched your query, null is returned. Which is why its not triggering an error in the first place. As for the second part.
You cannot catch an error thrown in an asynchronous callback function using promises, since its context will be lost.
Using promises, the correct solution will be to reject the wrapping promise.
Promise.reject(new Error('fail')).then(function(error) {
// not called
}, function(error) {
console.log(error); // Stacktrace
});

Break out of Bluebird promise chain in Mongoose

I've studied several related questions & answers and still can't find the solution for what I'm trying to do. I'm using Mongoose with Bluebird for promises.
My promise chain involves 3 parts:
Get user 1 by username
If user 1 was found, get user 2 by username
If both user 1 and user 2 were found, store a new record
If either step 1 or step 2 fail to return a user, I don't want to do step 3. Failing to return a user, however, does not cause a database error, so I need to check for a valid user manually.
I can use Promise.reject() in step 1 and it will skip step 2, but will still execute step 3. Other answers suggest using cancel(), but I can't seem to make that work either.
My code is below. (My function User.findByName() returns a promise.)
var fromU,toU;
User.findByName('robfake').then((doc)=>{
if (doc){
fromU = doc;
return User.findByName('bobbyfake');
} else {
console.log('user1');
return Promise.reject('user1 not found');
}
},(err)=>{
console.log(err);
}).then((doc)=>{
if (doc){
toU = doc;
var record = new LedgerRecord({
transactionDate: Date.now(),
fromUser: fromU,
toUser: toU,
});
return record.save()
} else {
console.log('user2');
return Promise.reject('user2 not found');
}
},(err)=>{
console.log(err);
}).then((doc)=>{
if (doc){
console.log('saved');
} else {
console.log('new record not saved')
}
},(err)=>{
console.log(err);
});

Example
All you need to do is something like this:
let findUserOrFail = name =>
User.findByName(name).then(v => v || Promise.reject('not found'));
Promise.all(['robfake', 'bobbyfake'].map(findUserOrFail)).then(users => {
var record = new LedgerRecord({
transactionDate: Date.now(),
fromUser: users[0],
toUser: users[1],
});
return record.save();
}).then(result => {
// result of successful save
}).catch(err => {
// handle errors - both for users and for save
});
More info
You can create a function:
let findUserOrFail = name =>
User.findByName(name).then(v => v || Promise.reject('not found'));
and then you can use it like you want.
E.g. you can do:
Promise.all([user1, user1].map(findUserOrFail)).then(users => {
// you have both users
}).catch(err => {
// you don't have both users
});
That way will be faster because you don't have to wait for the first user to get the second one - both can be queried in parallel - and you can scale it to more users in the future:
let array = ['array', 'with', '20', 'users'];
Promise.all(array.map(findUserOrFail)).then(users => {
// you have all users
}).catch(err => {
// you don't have all users
});
No need to complicate it more than that.

move your error handling out of the inner chain to the place you want to actual catch/handle it. As i don't have mongo installed, here is some pseudocode that should do the trick:
function findUser1(){
return Promise.resolve({
user: 1
});
}
function findUser2(){
return Promise.resolve({
user: 2
});
}
function createRecord(user1, user2){
return Promise.resolve({
fromUser: user1,
toUser: user2,
});
}
findUser1()
.then(user1 => findUser2()
.then(user2 => createRecord(user1, user2))) // better nest your promises as having variables in your outside scope
.then(record => console.log('record created'))
.catch(err => console.log(err)); // error is passed to here, every then chain until here gets ignored
Try it by changing findUser1 to
return Promise.reject('not found 1');

First, I would recommend using throw x; instead of return Promise.reject(x);, simply for readibility reasons. Second, your error logging functions catch all the errors, that's why your promise chain is continuing. Try rethrowing the errors:
console.log(err);
throw err;

Don't put error logging everywhere without actually handling the error - if you pass an error handler callback you'll get back a promise that will fulfill with undefined, which is not what you can need. Just use
User.findByName('robfake').then(fromUser => {
if (fromUser) {
return User.findByName('bobbyfake').then(toUser => {
if (toUser) {
var record = new LedgerRecord({
transactionDate: Date.now(),
fromUser,
toUser
});
return record.save()
} else {
console.log('user2 not found');
}
});
} else {
console.log('user1 not found');
}
}).then(doc => {
if (doc) {
console.log('saved', doc);
} else {
console.log('saved nothing')
}
}, err => {
console.error("something really bad happened somewhere in the chain", err);
});
This will always log one of the "saved" or "something bad" messages, and possibly one of the "not found" messages before.
You can also use exceptions to achieve this, but it doesn't really get simpler:
var user1 = User.findByName('robfake').then(fromUser => {
if (fromUser)
return fromUser;
else
throw new Error('user1 not found');
});
var user2 = user1.then(() => // omit this if you want them to be searched in parallel
User.findByName('bobbyfake').then(toUser => {
if (toUser)
return toUser;
else
throw new Error('user2 not found');
})
);
Promise.all([user1, user2]).then([fromUser, toUser]) =>
var record = new LedgerRecord({
transactionDate: Date.now(),
fromUser,
toUser
});
return record.save();
}).then(doc => {
if (doc) {
console.log('saved', doc);
} else {
console.log('saved nothing')
}
}, err => {
console.error(err.message);
});

Access variable outside mongoose find method - node js

In my node js program, I look into my mongoose database, and find and return the values in that collection - there is only one value.
var myValueX;
myCollection.find(function(err, post) {
if (err) {
console.log('Error ' + err)
} else {
myValueX = post[0].valuex;
}
});
console.log('Have access here' + myValueX);
Now, I want to be able to use myValueX outside this find method. How can I do this?
When I try the console.log above, I get undefined back - is this possible to achieve

To access myValueX after it had been assigned in find's callback, you have two options, the first (naturally) is inside the callback itself, or inside a function called by the callback to which you send myValueX as an argument.
The better solution in my opinion is to use promises.
A simple promise-using solution is as follows:
function findPromise(collection) {
return new Promise((resovle, reject) => {
collection.find((err, post) => {
if (err)
return reject(err)
// if all you want from post is valuex
// otherwise you can send the whole post
resolve(post[0].valuex)
})
})
}
findPromise(collection)
.then((valueX) => {
// you can access valuex here
// and manipulate it anyway you want
// after it's been sent from the `findPromise`
console.log("valueX: ", valueX)
})
.catch((err) => {
console.log("An error occurred. Error: ", err)
})