I build a scraper where the scraped data gets compared with already existing data to avoid duplicates, create new entries and update old entries. I'm doing this with a for loop, which loops over a findOne function where are two awaits in. The problem is that my for loop is ignoring (because it's sync?) my awaits and goes over to a part, where it is important that all of these awaits are done.
async function comparedata(length) {
console.log("Starting comparing entries in data");
for (let x = 0; x < length; x++) {
const model = new dataModel({
link: dataLinks[x],
name: dataGames[x].replace('Download', ' '),
logo: dataLogos[x],
provider: 'data',
});
model.collection.findOne({ "link": dataLinks[x] }, async function (err, found) {
if (err) throw err;
if (found == null) {
await model.save().then((result) => {
console.log(result) // Is not happening because the for loop goes through to the next function and closes the server
}).catch((err) => { console.log(err) });
}
else if (found != null) {
if (dataGames[x] != found.name) {
await model.collection.findOneAndUpdate({ link: dataLinks[x] }, { $set: { name: dataGames[x] } });
}
}
})
}
closeServer()//Closes the server is happening before new entries or updates are made.
}
My idea was to work with promises, but even if I tried to, it was just getting resolved too fast and closes the server again.
You should be able to simplify your logic like this:
async function comparedata(length) {
console.log('Starting comparing entries in data');
try {
for (let x = 0; x < length; x++) {
let found = await dataModel.findOne({ link: dataLinks[x] });
if (!found) {
found = await dataModel.create({
link: dataLinks[x],
name: dataGames[x].replace('Download', ' '),
logo: dataLogos[x],
provider: 'data',
});
} else if (found.name !== dataGames[x]) {
found.name = dataGames[x];
await found.save();
}
console.log(found);
}
} catch (e) {
console.log(e);
}
closeServer();
}
For the first iteration of for loop, the findOne's callback is put in the callback queue by the event loop and it proceeds with the next iteration not waiting for the awaits, this goes till the last iteration and after the last iteration it immediately calls the closeServer(), after this closeServer() call the tasks put into the callback queue (i.e the findOne's callbacks) are considered and executed by the event loop. Inorder to get an understanding of this, you have to learn about the event loop and how the event loop executes the javascript code. Please check about the event loop working mechanism youtube video here
You can use the promise style execution of findOne and overcome this issue.
And according to me, using await & then() functionalities on the same statement is not a good practice.
My suggestion,
async function comparedata(length) {
console.log("Starting comparing entries in data");
for (let x = 0; x < length; x++) {
const model = new dataModel({
link: dataLinks[x],
name: dataGames[x].replace('Download', ' '),
logo: dataLogos[x],
provider: 'data',
});
// using .exec() at the end allows us to go with the promise way of dealing things rather than callbacks
// assuming that 'dataModel' is the Schema, so I directly called findOne on it
const found = await dataModel.findOne({ "link": dataLinks[x] }).exec();
if (found == null) {
// wrap the await in try...catch for catching errors while saving
try{
await model.save();
console.log("Document Saved Successfully !");
}catch(err) {
console.log(`ERROR while saving the document. DETAILS: ${model} & ERROR: ${err}`)
}
} else if (found != null) {
if (dataGames[x] != found.name) {
await model.collection.findOneAndUpdate({ link: dataLinks[x] }, { $set: { name: dataGames[x] } });
}
}
if (x > length)
sendErrorMail();
}
closeServer()//Closes the server is happening before new entries or updates are made.
}
NOTE: Please refer the Mongoose official documentation for updates.
I'm not super familiar with mongoose api, but if you are unable to use a promisified version then you can use the Promise constructor to "jump" out of a callback:
const found = await new Promise((resolve, reject) => {
model.collection.findOne({ "link": dataLinks[x] }, function (err, found) {
if (err) {
reject(err);
return;
};
resolve(found);
}
});
This might help you work things out.
Related
I want to modify two schema while adding data. For that I used ACID transaction of mongodb with nodejs as follow. But, when I run program it displays the error like
(node:171072) UnhandledPromiseRejectionWarning: MongoError: Given transaction number 3 does not match any in-progress transactions. The active transaction number is 2
at MessageStream.messageHandler (/home/user/Projects/project/node_modules/mongodb/lib/cmap/connection.js:272:20)
at MessageStream.emit (events.js:375:28)
at MessageStream.emit (domain.js:470:12)
addData = async(request: Request, response: Response) => {
const session = await stockSchema.startSession()
try {
const userData = request.body
let data = {}
const transaction = await session.withTransaction(async() => {
try {
userData.products.map(async(item: any) => {
await inventorySchema.findOneAndUpdate({ _id: item.materialID }, { $inc: {qty: -item.qty }}, { session });
});
data = new stockSchema(userData);
await data.save({ session });
} catch (error) {
await session.abortTransaction()
throw new Error("Could not create data. Try again.");
}
});
if (transaction) {
session.endSession()
return returnData(response, data, 'Data created successfully.');
} else {
throw new Error("Could not create data. Try again.");
}
} catch (error: any) {
session.endSession();
return Error(response, error.message, {});
}
}
So you might have figured out the answer to this already, but anyway, after months of frustration, and no clear answer on the internet, i finally figured it out.
The problem with your code above is that you are passing session into a database operation (the .findOneAndUpdate function above) that is running within .map . Meaning, your 'transaction session' is being used concurrently, which is what is causing the error. read this: https://www.mongodb.com/community/forums/t/concurrency-in-mongodb-transactions/14945 (it explains why concurrency with transactions creates a bit of a mess.)
Anyway, instead of .map, use a recursive function that fires each DB operation one after another rather than concurrently, and all your problems will be solved.
You could use a function something like this:
const executeInQueue = async ({
dataAry, //the array that you .map through
callback, //the function that you fire inside .map
idx = 0,
results = [],
}) => {
if (idx === dataAry.length) return results;
//else if idx !== dataAry.length
let d = dataAry[idx];
try {
let result = await callback(d, idx);
results.push(result);
return executeInQueue({
dataAry,
callback,
log,
idx: idx + 1,
results,
});
} catch (err) {
console.log({ err });
return results.push("error");
}
};
I got two cases which I tried to understand what is happen but it still not clear enough form me ,so I read that if I want to run async call in for loop or map I need to use Promise.all
but let me share what happen with me
First I used map to update many records in my database ,It updated some and not all the data
AllAuthToken.map(async singleToken =>{
let deviceIds = uuidv4();
let newDeviceArray = {
[deviceIds]: singleToken.deviceToken,
};
await AuthToken.updateOne(
{ _id: singleToken._id },
{
$set: {
tokensDeviceArray: [newDeviceArray],
deviceId: [deviceIds],
},
},
{ $new: true }
);
}
Then after this happened I used for loop and it updated all the data
for (let i = 0; i < AllAuthToken.length; i++) {
let deviceIds = uuidv4();
let newDeviceArray = {
[deviceIds]: AllAuthToken[i].deviceToken,
};
await AuthToken.updateOne(
{ _id: AllAuthToken[i]._id },
{
$set: {
tokensDeviceArray: [newDeviceArray],
deviceId: [deviceIds],
},
},
{ $new: true }
);
}
So what happen so that the first case failed and the second passed
The reason is that .map executes its callback one after the other without waiting. The await does its thing, but that only delays the code that follows after that await. But it does not prevent the .map iteration to continue with the next call of the callback, as it is not part of the async function in which the await occurs.
The for loop on the other hand, is part of the same async function as where the await occurs, and so the iteration only continues when the awaited promise resolves.
If you prefer to have the database requests executed in "parallel", i.e. where you launch the next request without waiting for the previous one to resolve, then use Promise.all, like this:
let promises = AllAuthToken.map(singleToken => {
let deviceIds = uuidv4();
let newDeviceArray = {
[deviceIds]: singleToken.deviceToken,
};
return AuthToken.updateOne(
{ _id: singleToken._id },
{
$set: {
tokensDeviceArray: [newDeviceArray],
deviceId: [deviceIds],
},
},
{ $new: true }
);
};
await Promise.all(promises);
/* ... now all is done ... */
So this has been troubling me for a while, I have an array of objects that I want to insert into my SQLite DB. Each of the objects have 5 parameters and I have the SQL Query in place to run it. I was using a loop to iterate through the array and populate each of the objects via db transactions to SQLite. However, the db tasks are asynchronous which leads to the loop being completed before the task is run and incorrect data being populated into the db. The while loop in the code below doesn't work and I have tried the same thing with a for loop to no avail.
var i=0;
while(i<rawData.length){
console.log(rawData[i],i)
db.transaction(function (tx) {
console.log(rawData,i," YAY")
tx.executeSql(
'Update all_passwords SET title=?,userID=?,password=?,notes=?,category=? WHERE ID =? ',
[rawData[i].title,rawData[i].userID,rawData[i].password,rawData[i].notes,rawData[i].category,rawData[i].id],
(tx, results) => {
console.log("saved all data")
tx.executeSql(
"SELECT * FROM all_passwords ORDER BY id desc",
[],
function (tx, res) {
i++
console.log("Print Out Correct Data")
for(var i=0;i<res.rows.length;i++){
console.log(res.rows.item(i), i )
}
});
}
);
console.log("EXIT")
}
,
(error) => {
console.log(error);
}
);
}
I'm not familiar using async tasks with hooks but I believe that might be a potential solution. My intention is to populate the rawaData array of objects into the SQLDb in one go while I use a state to maintain the loading screen.
I did refer the below sources but wasn't able to come up with anything concrete.
react native insertion of array values using react-native-sqlite-storage
https://medium.com/javascript-in-plain-english/how-to-use-async-function-in-react-hook-useeffect-typescript-js-6204a788a435
Thanks in advance!
I made a little write up for you on how I would solve it. Read the comments in the code. If anything is unclear feel free to ask!
const rawData = [
{ title: "title", userID: "userID", password: "password", notes: "notes", category: "category", id: "id" },
{ title: "title_1", userID: "userID_1", password: "password_1", notes: "notes_1", category: "category_1", id: "id_1" },
{ title: "title_2", userID: "userID_2", password: "password_2", notes: "notes_2", category: "category_2", id: "id_2" }
];
// You can mostly ignore this. It's just a mock for the db
const db = {
tx: {
// AFAIK if there is a transaction it's possible to execute multiple statements
executeSql: function(sql, params, success, error) {
// just for simulating an error
if (params.title === "title_2") {
error(new Error("Some sql error"));
} else {
console.log(sql, params.title);
success();
}
}
},
transaction: function(tx, error) {
// simulating async
setTimeout(() => {
return tx(this.tx);
}, parseInt(Math.random() * 1000));
}
}
// Lets make a class which handles our dataccess in an async way
class DataAccess {
// as transaction has callback functions it's wrapped in a promise
// on success the transaction is resolved
// if there is an error it will be thrown
transaction = () => {
return new Promise(resolve => {
db.transaction(tx => resolve(tx), error => {
throw error;
});
});
}
// the actual executeSql function which "hides" all the transaction stuff
// awaits a transaction and executes the sql on it
// if the execution was successfull resolve
// if not throw the error
executeSql = async(sql, params) => {
const tx = await this.transaction();
tx.executeSql(sql, params, () => Promise.resolve(), error => {
throw error;
});
}
}
const dal = new DataAccess();
// all sql execute tha was possible
async function insert_with_execute() {
// promise all does not guarantee execution order
// but it is a possibility to await an array of promises (async functions)
await Promise.all(rawData.map(async rd => {
try {
await dal.executeSql("sql_execute", rd);
} catch (error) {
console.log(error.message);
}
}));
}
// no sql executed cause of error and all in the same transaction
async function insert_with_transaction() {
const tx = await dal.transaction();
for (let i = 0; i < rawData.length; i++) {
tx.executeSql("sql_transaction", rawData[i], () => console.log("success"), error => console.log(error.message));
}
}
async function test() {
await insert_with_execute();
console.log("---------------------------------")
await insert_with_transaction();
}
test();
Apparently the best approach to take is using anonymous functions that create a separate instance of execution for each value of i. This is a good example of how to do it....
Javascript SQL Insert Loop
I know this is a common problem for people, and I've looked up some guides and looked at the docs, but I'm not understanding what is happening, and would love some help.
Here is the controller function I am working on:
exports.appDetail = function (req, res) {
appRepo.findWithId(req.params.id, (err, myApp) => {
if (err) {
console.log(err)
}
getDeviceData(myApp.devices, (err, myDeviceData) => {
if (err) console.log(err)
console.log(JSON.stringify(myDeviceData) + ' || myDeviceData')
// construct object to be returned
let appsDataObject = {
name: myApp.name,
user_id: myApp.user_id,
devices: myDeviceData,
permissions: myApp.permissions
}
return res.status(200).send(appsDataObject)
})
})
}
// write async function here
const getDeviceData = function (devices, callback) {
let devicesDataArray = []
async.each(devices, function (device, cb) {
deviceRepo.findById(new ObjectID(device), (err, myDevice) => {
if (err) {
cb(err)
}
// get device data, push to devices array
let deviceObj = {
name: myDevice.name,
version: myDevice.version
}
devicesDataArray.push(deviceObj)
console.log(JSON.stringify(devicesDataArray) + ' || devicesDataAray after obj push')
})
cb(null, devicesDataArray)
}, function (err) {
// if any of the file processing produced an error, err would equal that error
if (err) console.log(err)
})
callback(null, devicesDataArray)
}
I originally wrote this with a for loop and a callback, but I think it was impossible to do that way (I'm not sure about that though). If there is a better way to make an asynchronous loop, please let me know.
On ~line 8 there is a log statement myDeviceData. This should be returning the data I want through a callback, but this log statement always comes back empty. And since the other log statements show that the data is being formatted correctly, the problem must be with returning the data I need through the callback of getDeviceData(). Presumably, the callback(null, devicesDataArray) should do this.
I'm not understanding how these async functions are supposed to work, clearly. Can someone please help me understand how I should get values from these async.each functions? Thank you.
EDIT:
I refactored the code to try and make it clearer and approach the problem better, and I have pinpointed where the problem is. At the beginning the this function I define devicesDataArray as an empty array, and at the end I return the array. Everything inside happens as it should, but I don't know how to tell the return to wait until the array isn't empty, if that makes sense, Here is the new code:
let getData = async function (devices) {
let devicesDataArray = []
for (let i = 0; i < devices.length; i++) {
deviceRepo.findById(new ObjectID(devices[i]), async (err, myDevice) => {
if (err) {
console.log(err)
}
console.log(JSON.stringify(myDevice) + ' || myDevice')
let deviceObj = await {
name: myDevice.name,
version: myDevice.version
}
console.log(JSON.stringify(deviceObj) + ' || deviceObj')
await devicesDataArray.push(deviceObj)
console.log(JSON.stringify(devicesDataArray) + ' || devicesDataArray after push')
})
}
console.log(JSON.stringify(devicesDataArray) + ' || devicesDataArray before return')
return Promise.all(devicesDataArray) // problem is here.
}
Any help towards understanding this is appreciated.
Here we are using a Promise
The Promise object is returned immediately and we perform our async code inside there.
const someFuncThatTakesTime = () => {
// Here is our ASYNC
return new Promise((resolve, reject) => {
// In here we are synchronous again.
const myArray = [];
for(x = 0; x < 10; x++) {
myArray.push(x);
}
// Here is the callback
setTimeout(() => resolve(myArray), 3000);
});
};
someFuncThatTakesTime().then(array => console.log(array));
first, I would prefer to avoid using callback because it will make your code unreadable,
also you can handle async using javascript methods without any needs to use other modules.
the reason for your problem is your async.each function executed without waiting what it has inside
and you have here deviceRepo.findById which is async function and need to wait for it
I refactored your code using async await and I wish this will simplify the problem
exports.appDetail = function async(req, res) {
try {
const myApp = await appRepo.findWithId(req.params.id)
const myDeviceData = await getDeviceData(myApp.device)
console.log(JSON.stringify(myDeviceData) + ' || myDeviceData')
// construct object to be returned
let appsDataObject = {
name: myApp.name,
user_id: myApp.user_id,
devices: myDeviceData,
permissions: myApp.permissions
}
return res.status(200).send(appsDataObject)
} catch (err) {
console.log(err)
}
}
// write async function here
const getDeviceData = function async(devices) {
let devicesDataArrayPromises = devices.map(async(device)=>{
const myDevice= await deviceRepo.findById(new ObjectID(device))
let deviceObj = {
name: myDevice.name,
version: myDevice.version
}
return deviceObj
})
const devicesDataArray = await Promise.all(devicesDataArrayPromises)
console.log(JSON.stringify(devicesDataArray) + ' || devicesDataAray after obj push')
return devicesDataArray
}
I am currently building a Nodejs, Express, Sequelize (w. PostgreSQL) app, and have run into a few problems with using promises together with transactions and loops.
I am trying to figure out how to use a for loops in a transaction. I am trying to loop through a list of members and create a new user in the database for each of them.
I know the following code is wrong but it shows what I am trying to do.
Can anyone point me in the right direction?
var members = req.body.members;
models.sequelize.transaction(function (t) {
for (var i = 0; i < members.length; i++) {
return models.User.create({'firstname':members[i], 'email':members[i], 'pending':true}, {transaction: t}).then(function(user) {
return user.addInvitations([group], {transaction: t}).then(function(){}).catch(function(err){return next(err);});
})
};
}).then(function (result) {
console.log("YAY");
}).catch(function (err) {
console.log("NO!!!");
return next(err);
});
You should use a Promise.all
var members = req.body.members;
models.sequelize.transaction(function (t) {
var promises = []
for (var i = 0; i < members.length; i++) {
var newPromise = models.User.create({'firstname':members[i], 'email':members[i], 'pending':true}, {transaction: t});
promises.push(newPromise);
};
return Promise.all(promises).then(function(users) {
var userPromises = [];
for (var i = 0; i < users.length; i++) {
userPromises.push(users[i].addInvitations([group], {transaction: t});
}
return Promise.all(userPromises);
});
}).then(function (result) {
console.log("YAY");
}).catch(function (err) {
console.log("NO!!!");
return next(err);
});
I don't believe you need to catch within sequelize transactions as I think it jumps out to the catch on the transaction
Sorry for formatting. On mobile.
Promise.all will wait for all promises to return (or fail) before running the .then, and the .then callback will be all the promise data from each array
You'll need to use the built in looping constructs of bluebird which ships with sequelize:
var members = req.body.members;
models.sequelize.transaction(t =>
Promise.map(members, m => // create all users
models.User.create({firstname: m, email: m, 'pending':true}, {transaction: t})
).map(user => // then for each user add the invitation
user.addInvitations([group], {transaction: t}) // add invitations
)).nodeify(next); // convert to node err-back syntax for express
Depending on your implementation of Node.js this may help. I have the same setup using express, POSTGRES and sequelize.
Personally I'd prefer the async/await (ES6) implementation over then/catch as it is easier to read. Also creating a function that can be called externally improves re-usability.
async function createMemeber(req) {
let members = req.body.members;
for (var i = 0; i < members.length; i++) {
// Must be defined inside loop but outside the try to reset for each new member;
let transaction = models.sequelize.transaction();
try {
// Start transaction block.
let user = await models.User.create({'firstname':members[i], 'email':members[i], 'pending':true}, {transaction});
await user.addInvitations([group], {transaction}));
// if successful commit the record. Else in the catch block rollback the record.
transaction.commit();
// End transaction block.
return user;
} catch (error) {
console.log("An unexpected error occurred creating user record: ", error);
transaction.rollback();
// Throw the error back to the caller and handle it there. i.e. the called express route.
throw error;
}
}
}
if someone is looking for a solution with typescript v4.0.5 using async and await here is what worked out for me. Maybe you can use it on your javascript aplication too but it will depend on the version of it.
const array = ['one','two','three'];
const createdTransaction = sequelize.transaction();
const promises = array.map(async item => {
await model.create({
name: item,
},
{ transaction: createdTransaction },
);
});
Promise.all(promises).then(async values => {
await createdTransaction.commit();
});
First: https://caolan.github.io/async/docs.html
So, easily:
// requiring...
const async = require('async');
// exports...
createAllAsync: (array, transaction) => {
return new Promise((resolve, reject) => {
var results = [];
async.forEachOf(array, (elem, index, callback) => {
results.push(models.Model.create(elem, {transaction}));
callback();
}, err => {
if (err) {
reject(err);
}
else {
resolve(results);
}
});
});
}