I have a large csv and I need to read t and insert to mongodb
Csv contains user name, category name and policy name.
Need to insert Users into User collection with category id and policy id. Csv provides only the category name and policy name. So I need to fetch category id from collection using its name.
If category name not exist, create a new one and returns its id. Same case for policy.
So I tried
fs.createReadStream('./data_sheet.csv')
.pipe(csv())
.on('data', async (row) => {
// console.log(row)
let res = await Category.findOneOrCreate({ name: row.cat.trim() });
console.log(res)
})
.on('end', () => {
console.log('CSV file successfully processed');
});
categorySchema.statics.findOneOrCreate = async function findOneOrCreate(condition) {
try {
const self = this
let agent = await self.findOne(condition)
console.log("condition")
console.log(condition)
console.log("agent")
console.log(agent)
if (agent) return agent._id
else {
agent = await self.create(condition)
return agent._id
}
} catch (e) {
console.log(e)
}
}
This is not working in proper manner. What is the proper way to do this?
If by not working you mean, the data is not coming up for category then make sure you follow the right async approach or else provide more info.
There are several things to keep in mind,
You might need to create a cronjob for recursive process
Import the csv file into an array of Object
Loop over the object to match with category Id
If not then create a category Id
Return to the main function
update the document [here you can either update all or update one at a time using the step 2 looping we did]
Related
I am building a chatbot on Dialogflow and using Firestore as my database. I was originally using a custom auto ID generator, but I am now trying to use Firestores in built one, and getting errors with my current code.
function AddWaterConsumption (agent) {
// Get parameter from Dialogflow with the string to add to the database
var databaseEntry = {
"PatientID": agent.parameters.Patient_ID,
"DailyConsumption": agent.parameters.Water_consumption_user,
"DailyWaterPlan": agent.parameters.Daily_water_plan,
};
let Patient_ID = agent.parameters.Patient_ID;
console.log(`Okay, we're trying to write to database: ${databaseEntry}`);
// here is where changes are needed
const dialogflowAgentRef = db.collection("WaterConsumption").doc(genRand(8)); // need to change to add()
console.log(`Found agent ref: ${dialogflowAgentRef}`);
return db.runTransaction(t => {
t.set(dialogflowAgentRef, databaseEntry); // this will also need to be changed
return Promise.resolve('Write complete');
}).then(doc => {
agent.add(`I have updated your daily water consumption`);
agent.add(`Is anything else I can do for you?.`);
}).catch(err => {
console.log(`Error writing to Firestore: ${err}`);
agent.add(`Failed to write "${dialogflowAgentRef}" to the Firestore database.`);
});
}
I have changed the important lines to:
console.log(`Okay, we're trying to write to database: ${databaseEntry}`);
const dialogflowAgentRef = db.collection("WaterConsumption"); // changed here
console.log(`Found agent ref: ${dialogflowAgentRef}`);
return db.runTransaction(t => {
t.set(db.collection("WaterConsumption").add(databaseEntry), databaseEntry); // and here
Which does successfully write to the db and auto generates an ID. Although an error is being caught:
Argument "documentRef" is not a valid DocumentReference. Input is not a plain JavaScript object
This makes no sense:
t.set(db.collection("WaterConsumption").add(databaseEntry), databaseEntry);
In this line the db.collection("WaterConsumption").add(databaseEntry) is a call to Firestore to create a document outside of the transaction). What you want instead is to just create a new DocumentReference with a unique ID, which (as shown in the 3rd snippet in the documentation on adding documents) can be done with doc().
So:
const newDocRef = db.collection("WaterConsumption").doc();
t.set(newDocRef, databaseEntry);
I’m new to mongo DB. What will be the schema or schemas what is user inputted, how we can recognize those in our code, or rather what will be the query (using javascript ).
Question:
Suppose we have a book schema and saved some books in the database, now we have to make a post api /getParticularBooks
take any input and use it as a condition to fetch books that satisfy that condition
e.g
if body had { name: “hi”} then you would fetch the books with this name
if body had { year: 2020} then you would fetch the books in this year
Or maybe the both at same time
hence the condition will differ based on what you input in the request body
You can start by creating a function which verify if the user request body doesn't have any input property which isn't define in your schema.
I would do it like that
const verifyProperty = (user_request_body) => {
const properties = [list of your schema property];
for(let i in properties){
const string = properties[i]
if(!(string in user_request_body))
return false;
}
return true;
}
And now, based on the return value of the verifyProperty function, you can send a query or return an error message.
if(verifyProperty(req.body) == false){
return res.json({message: "Invalid properties"})
}
const data = await Books.find(req.body);
return res.json({data})
In order to use await, your function must be async
I am trying to create an add to cart button which fetches the data from product database using the id of specific product which I selected. I am trying to push the object found using the same Id into a normal javascript array and then to display it using ejs methods. While I was tring I found I am unable to push the data in object form.
Summary:
On 7th line I have declared an array and in that array I want to store some objects which I have fetched frome a db model.
On 15th line I am trying to push the object form into my array so that I could iterate through the objects to display them on my page using ejs. But I am unable to do that.
screenshots:
Here's the final result I'm getting even after trying to push objects in array:
empty array logged
Here are the objects I'm trying to push:
Objects
Code:
app.get("/cart", (req, res) => {
if (req.isAuthenticated()) {
const findcartdata = req.user.username;
userData.findOne({email: findcartdata}, (err, BookId) => {
// console.log(BookId.cartItemId);
const idArray = BookId.cartItemId;
var bookArray = [];
idArray.forEach((data) => {
productData.findOne({_id: data}, (err, foundBookData) =>{
// console.log(foundBookData);
if(err){
console.log(err);
}
else{
bookArray.push(foundBookData);
}
})
});
console.log(bookArray);
// res.render("cart", {
// cartBookArray: BookId.cartItemId
// })
});
} else {
res.redirect("/login");
}
})
In above code i found the user's email using passport authentication user method and using that email I wanted to add the products in a different javascript array (which I am goint to pass to my ejs file of cart and then iterate it on list) using those array of Id which I got from another model called userData. The problem is I am able to find userData of each Id but unable to store them as an array of objects.
Looks like a timing issue, your code completes before the database downloads the objects and pushes them to your array.
This should fix your issue:
// ...
const idArray = BookId.cartItemId;
var bookArray = [];
for (const data of idArray) {
const foundBookData = await productData.findOne({_id: data}).catch(console.error);
if (!foundBookData) continue;
bookArray.push(foundBookData);
}
console.log(bookArray);
// ...
By the way, make sure to make the whole function asynchronous as well, which would be done by changing this line:
userData.findOne({email: findcartdata}, async (err, BookId) => { // ...
I am stuck in what I thought was a very simple use case: I have a list of client ids in an array. All I want to do is fetch all those clients and "watch" them (using the .onSnapshot).
To fetch the client objects, it is nice and simple, I simply go through the array and get each client by their id. The code looks something like this:
const accessibleClients = ['client1', 'client2', 'client3']
const clients = await Promise.all(
accessibleClients.map(async clientId => {
return db
.collection('clients')
.doc(clientId)
.get()
})
)
If I just needed the list of clients, it would be fine, but I need to perform the .onSnapshot on it to see changes of the clients I am displaying. Is this possible to do? How can I get around this issue?
I am working with AngularFire so it is a bit different. But i also had the problem that i need to listen to unrelated documents which can not be queried.
I solved this with an object which contains all the snapshot listeners. This allows you to unsubscribe from individual client snapshots or from all snapshot if you do not need it anymore.
const accessibleClients = ['client1', 'client2', 'client3'];
const clientSnapshotObject = {};
const clientDataArray = [];
accessibleClients.forEach(clientId => {
clientSnapshotArray[clientId] = {
db.collection('clients').doc(clientId).onSnapshot(doc => {
const client = clientDataArray.find(client => doc.id === client.clientId);
if (client) {
const index = clientDataArray.findIndex(client => doc.id === client.clientId);
clientDataArray.splice(index, 1 , doc.data())
} else {
clientDataArray.push(doc.data());
}
})
};
})
With the clientIds of the accessibleClients array, i create an object of DocumentSnapshots with the clientId as property key.
The snapshot callback function pushes the specific client data into the clientDataArray. If a snapshot changes the callback function replaces the old data with the new data.
I do not know your exact data model but i hope this code helps with your problem.
I am trying to get data from another database before reading data in the table. However, I can't seem to find a way to access it properly.
The best I've got so far is based on some other examples both on Microsoft's documentation and on StackOverflow but they all seem to fail.
table.read(function (context) {
var results = context.tables("table2").read();
var text = results[0].column;
context.query.where({ columnName: text });
return context.execute();
});
I get an error when doing this saying that column doesn't exist.
As per your description, if I do not misunderstand, you want to query table2 in table1 operations in EasyTables scripts.
we can leverage "use()" to custom middleware to specify middleware to be executed for every request against the table as the description on the document of azure-mobile-apps sdk at
E.G.
var queries = require('azure-mobile-apps/src/query');
var insertMiddleware = function(req,res,next){
var table = req.azureMobile.tables('table2'),
query = queries.create('table2')
.where({ TestProperty : req.body.testproperty });
table.read(query).then(function(results) {
if(results){
req.someStoreData = somehander(results); //some hander operations here to get what you want to store and will use in next step
next();
}else{
res.send("no data");
}
});
};
table.insert.use(insertMiddleware, table.operation);
table.insert(function (context) {
console.log(context.req.someStoreData);
return context.execute();
});
More example:
async function filterByAllowedDomain(context) {
var domains = await context.tables('domains')
.where({ allowed: true })
.read();
var categories = await context.tables('categories')
.where(function (ids) {
return this.domainId in ids;
}, domains.map(d => d.id))
.read();
context.query.where(function (ids) {
return this.categoryId in ids;
}, categories.map(c => c.id));
return context.execute(); }
The tables module in azure-mobile-apps-node sdk contains functionality for adding tables to an Azure Mobile App. It returns a router that can be attached to an express app with some additional functions for registering tables. Which actually leverage Azure SQL (SQL Server database service on Azure).
Hope it helps.