Express js deadlock in promise inside for loop - javascript

I have an application with node.js and expressjs and now I have a problem when I tried to insert several records into the database.
I have an array with 4 records and the applications insert 3 and the last one returns me a deadlock error from SQL.
I think that problem could be because of the promise inside the for a loop.
Next you have a resume of the code:
processOrder: (req, res) => {
//Collecting data for variables
//Promise to wait for data
Promise.all([data]).then(function(result)
{
//iterate an array
for (var i = 0; i < result[0].length; i++)
{
var new_record = request.query("INSERT INTO table_test (name, local, date) VALUES ('"+result[0][i].name+"', '"+result[0][i].local+"', '"+result[0][i].date+"')");
//Promise to wait for insert
Promise.all([ new_record]).then(function(result)
{
console.log("Record number "+i+" inserted");
}).catch(function(err) {
console.log(err);
console.log("Erro collecting data");
});
}
console.log("END");
res.send('true');
}).catch(function(err) {
console.log(err);
console.log("Erro collecting data");
});
}
The output is:
END
Record number 1 inserted
Record number 2 inserted
Record number 3 inserted
deadlock error
The "END" console.log should be the last log and not the first.
How can I solve this situation? Please explain to me what I'm doing wrong.
Thank you

I rewrote your code below.
You can try.
import { request } from "http";
processOrder: (req, res) => {
//Collecting data for variables
//Promise to wait for data
Promise.all([data])
.then(function(result){
const requests = result[0].map(record => {
// assume request.query returns promise
return request.query("INSERT INTO table_test (name, local, date) VALUES ('"+record.name+"', '"+record.local+"', '"+record.date+"')");
})
Promise.all(requests)
.then(function(result){
console.log("END");
res.send('true');
})
.catch(function(err) {
console.log(err);
console.log("Erro collecting data");
});
})
.catch(function(err) {
console.log(err);
console.log("Erro collecting data");
});
}
If you want to log "Record Number X inserted", then
make promised function which execute request.query and console.log.
function requestPromise(queryStmt, index){
return new Promise(resolve, reject){
request.query(queryStmt)
.then(result => {
console.log(`Record Number ${index} inserted`);
resolve(result);
})
.catch(err){
reject(err);
})
}
}
and then call above function like this
Promise.all([data])
.then(function(result){
const requests = result[0].map((record, index) => {
return requestPromise("INSERT INTO table_test (name, local, date) VALUES ('"+record.name+"', '"+record.local+"', '"+record.date+"')", index);
})

You are doing asynchronus stuff inside your for-loop (your query). The output of "END" is outside of the loop. You are starting 4 Promises, but you are nowhere waiting for them to resolve. So the end is printed first because the for-loop is finished as soon as the Promises are created (not really finished, but javascript contnious with the next command).
You could await the Promise inside your for loop to resolve this.
Instead of
Promise.all([ new_record]).then(function(result)
you can use
const result = await new_record;
Therefore you have to mark the whole function as async

Related

Error "Given transaction number * does not match" in mongodb and nodejs

I want to modify two schema while adding data. For that I used ACID transaction of mongodb with nodejs as follow. But, when I run program it displays the error like
(node:171072) UnhandledPromiseRejectionWarning: MongoError: Given transaction number 3 does not match any in-progress transactions. The active transaction number is 2
at MessageStream.messageHandler (/home/user/Projects/project/node_modules/mongodb/lib/cmap/connection.js:272:20)
at MessageStream.emit (events.js:375:28)
at MessageStream.emit (domain.js:470:12)
addData = async(request: Request, response: Response) => {
const session = await stockSchema.startSession()
try {
const userData = request.body
let data = {}
const transaction = await session.withTransaction(async() => {
try {
userData.products.map(async(item: any) => {
await inventorySchema.findOneAndUpdate({ _id: item.materialID }, { $inc: {qty: -item.qty }}, { session });
});
data = new stockSchema(userData);
await data.save({ session });
} catch (error) {
await session.abortTransaction()
throw new Error("Could not create data. Try again.");
}
});
if (transaction) {
session.endSession()
return returnData(response, data, 'Data created successfully.');
} else {
throw new Error("Could not create data. Try again.");
}
} catch (error: any) {
session.endSession();
return Error(response, error.message, {});
}
}
So you might have figured out the answer to this already, but anyway, after months of frustration, and no clear answer on the internet, i finally figured it out.
The problem with your code above is that you are passing session into a database operation (the .findOneAndUpdate function above) that is running within .map . Meaning, your 'transaction session' is being used concurrently, which is what is causing the error. read this: https://www.mongodb.com/community/forums/t/concurrency-in-mongodb-transactions/14945 (it explains why concurrency with transactions creates a bit of a mess.)
Anyway, instead of .map, use a recursive function that fires each DB operation one after another rather than concurrently, and all your problems will be solved.
You could use a function something like this:
const executeInQueue = async ({
dataAry, //the array that you .map through
callback, //the function that you fire inside .map
idx = 0,
results = [],
}) => {
if (idx === dataAry.length) return results;
//else if idx !== dataAry.length
let d = dataAry[idx];
try {
let result = await callback(d, idx);
results.push(result);
return executeInQueue({
dataAry,
callback,
log,
idx: idx + 1,
results,
});
} catch (err) {
console.log({ err });
return results.push("error");
}
};

Using async/await with for loop and mongodb

I want to send an order-confirmation email(with nodemailer) to customers when they finished the order.
In this confirmation email I want to insert some data from the buyed products.
I've used many async/await combinations but none of them succeeded.
The code I got for now:
async function getProductData(products) {
let productsArray = products;
try {
let productsHtml = "";
for await (const product of productsArray) { // loop through the products
db.collection("collectionname").findOne(query, (err, data) => { // get data for every product
if (err) console.log(err)
else {
let productHtml = `<p>productname: ${data.productname}</p>
<img src="${data.imageSrc}">`;
productsHtml += productHtml; // add the product html string to productsHtml
}
})
}
} catch (error) {
console.log(error);
}
return productsHtml; //return the productsHtml string. But it returns "", because it doesn't await the mongodb function
}
async function sendConfirmationMail() {
let productsHtml = await getProductData(products); //"products" is an array
//nodemailer code
let transporter = nodemailer.createTransport({
// configuration
});
let email = await transporter.sendMail({
//email information
html: `<p>Thank you for your order!</p>
${productsHtml}`;, // insert the html string for all products into the mail
});
}
The problem is that the return statement in getProductData() fires before the for loop finishes. And then returns undefined.
Does anyone have tips on how I can implement async/await the correct way here?
You are not awaiting your database call. You are passing it a callback, so it does not return a Promise :
.findOne(query, (err, data) => {
This has a callback. You are passing it a function as second argument, that takes (err,data) as its own arguments. You need to remove this callback and use the async/await syntax with try/catch. Something like
const data = await db.collection("collectionname").findOne(query);
On the other hand you don't need to for await (const product of productsArray). You are iterating an array of static values, not an array of Promises. Remove this await.

How can I push data to an array inside an async request to the database?

I am trying to make a request to the database (mongoDB) and save its return in a list of objects but the list is not getting filled. Here is the code
router.get('/all/:studentEmail', auth, async (req, res) => {
try {
const student = await Student.findOne({ email: req.params.studentEmail });
if (!student) {
return res.status(404).json({ msg: 'Student not found' });
}
var list = [];
student.exercises.map(async (exercise) => {
list.unshift(await Exercise.findById(exercise));
});
res.json(list);
} catch (err) {
console.error(err.message);
res.status(500).send('Server error');
}
});
The database query await Exercise.findById(exercise) returns correctly the object, but res.json(list); returns empty. Do anyone know how to solve it?
The base issue is that res.json() executes way before student.exercises.map(async (exercise) => { completes. Putting await into map doesn't wait for each and every item in the async loop to process. Either use something like Promise.all() or use a for loop (other strategies can be used also). Decide which to use based on whether you can process in parallel or need to process in series. Try the following using Promise.all to execute async requests parallel using then on each Promise to execute an operation against list:
router.get("/all/:studentEmail", auth, async (req, res) => {
try {
const student = await Student.findOne({ email: req.params.studentEmail });
if (!student) {
return res.status(404).json({ msg: "Student not found" });
}
var list = [];
await Promise.all(
student.exercises.map((exercise) =>
Exercise.findById(exercise).then((result) => list.unshift(result))
)
);
res.json(list);
} catch (err) {
console.error(err.message);
res.status(500).send("Server error");
}
});
Also, an alternative to unshift and just return the results if they are not nested, if they are nested you can consider flat():
const list = await Promise.all(
student.exercises.map((exercise) => Exercise.findById(exercise))
);
return res.json(list);
Hopefully that helps!

Wait promises inside loop in nodejs

I know this question may already be asked. But I didn't understand how things are worked.that is why I am creating the new thread.
con.query(sql,[req.params.quizId],(err,rows,fields)=>{
//rows contains questions
if(err) throw err;
else{
let object={};
rows.forEach((item,index)=>{
object=item;
//here iam passing question id to get choices a async function
getChoices(item.id)
.then(data=>{
object.choices=data;
//save the question array
response.push(object);
//res.send(response);
});
})
res.send(response) //return empty array
}
});
function getChoices(questionId) {
let sql='SELECT id,text FROM `question_choices` where question_id=?';
return new Promise((resolve, reject) => {
con.query(sql,[questionId],(err,rows,fields)=>{
if(err) throw err;
else {
resolve(rows);
}
})
})
}
I tried several things but none is worked. I think for loop didn't wait for the promise to complete and it sends the response directly. Some async problems are happening there.
I can able to get all questions from database and for each question I need to get corresponding choices that I want.
something like this
[{id:'xx', text:'yy',choices:[{id:'c',text:'kk']},etc]
forEach runs synchronously. You're looking for Promise.all, which accepts an array of Promises, and resolves to an array of the resolved values once all of the Promises resolve. To transform your rows array to an array of Promises, use .map.
Also, when there's an error, you should call reject so that you can handle errors in the consumer of the Promise (the con.query callback), otherwise, when there's an error, it'll hang forever without you knowing about it:
con.query(sql,[req.params.quizId],(err,rows,fields)=>{
if(err) throw err;
Promise.all(rows.map((item) => (
getChoices(item.id)
.then((choices) => ({ ...item, choices }))
)))
.then((response) => {
res.send(response);
})
.catch((err) => {
// handle errors
})
});
function getChoices(questionId) {
const sql='SELECT id,text FROM `question_choices` where question_id=?';
return new Promise((resolve, reject) => {
con.query(sql,[questionId],(err,rows,fields)=>{
if(err) reject(err);
else resolve(rows);
});
});
}

From where should I call module.exports to get a not null value?

Where should I call module.export, I assume, it's supposed to be a callback function.
But I'm confused as to where am I supposed to call the callback function.
I'm still confused with the solution, too complicated for me.
sql.connect(config, function(err) {
if (err)
console.log(err);
// create Request object
var request = new sql.Request();
// query to the database and get the records
request.query('select part_num,qty from CRM.CRM.Fishbowl_Inventory where not location = \'Shipping\'',
function(err, recordset) {
if (err)
console.log(err)
// send records as a response
var details = recordset;
});
});
module.exports = details;
Confusion:
Extremely sorry to bother you guys but I want to be sure that I'm doing no harm to our database by involving any database request through Javascript.
I'm testing directly with our production database, hence cautious
So as Max provided in his answer the following code
const connectToSql = require('./connectToSql');
connectToSql()
.then(details => {
console.log(details);
//Here I can do as much logic as I want
//And it won't affect my database or call multiple requests on my DB
})
.catch(err => {
console.log(err);
});
I can understand I'm asking super silly questions, very sorry about that.
You can't export the result of your function. You want to export a function that will return your value. Like this:
function connectToSql(config) {
return new Promise((resolve, reject) => {
sql.connect(config, function (err) {
if (err) {
console.log(err);
reject(err);
}
// create Request object
var request = new sql.Request();
// query to the database and get the records
request.query('select part_num,qty from CRM.CRM.Fishbowl_Inventory where not location = \'Shipping\'',
function (requestErr, recordset) {
if (err) {
console.log(requestErr);
reject(requestErr);
}
resolve(recordset);
});
});
});
}
module.exports = connectToSql;
Because your function is async, I returned a promise that will return your result. Also, your second error from your query is named the same as your first error from the connection. That would cause problems.
Example of how to use this:
const connectToSql = require('./connectToSql');
connectToSql()
.then(details => {
console.log(details);
})
.catch(err => {
console.log(err);
});

Categories

Resources