I am trying to insert an array of JSON into Postgres. Everything appears to run successfully (the table gets created), but there is no data inserted, and no error given. I am using the async/await method for transactions, but I can't understand what is failing.
(async() => {
const client = await pool.connect()
let val= [
{"table_pk":1,"NAME":"my Great Name","ROLE":"name1"},
{"table_pk":2,"NAME":"new Name","ROLE":"name1"},
{"table_pk":3,"NAME":"someone's funny name","ROLE":"name1"}
]
try {
await client.query('BEGIN');
await client.query('DROP TABLE IF EXISTS myTable')
await client.query(`CREATE TABLE myTable(table_pk INTEGER PRIMARY KEY, name TEXT, role TEXT)`)
await client.query(`INSERT INTO myTable SELECT * FROM jsonb_populate_recordset(NULL::myTable, $1::jsonb)`,[JSON.stringify(val)]);
await client.query('COMMIT');
}
catch (e) {
await client.query('ROLLBACK');
throw e
}
finally {
client.release();
}
})().catch(e => console.error(e.stack))
EDIT
Updated the code to account for missing INSERT, now rows will insert, but not all the data.
The INSERT failed because JSONB is case sensitive when it comes to column names. I had Upper Case column names in the val data and they should have been lower. Thanks to #vitaly-t for pointing out the missing INSERT in my original post.
Related
I am building a chatbot on Dialogflow and using Firestore as my database. I was originally using a custom auto ID generator, but I am now trying to use Firestores in built one, and getting errors with my current code.
function AddWaterConsumption (agent) {
// Get parameter from Dialogflow with the string to add to the database
var databaseEntry = {
"PatientID": agent.parameters.Patient_ID,
"DailyConsumption": agent.parameters.Water_consumption_user,
"DailyWaterPlan": agent.parameters.Daily_water_plan,
};
let Patient_ID = agent.parameters.Patient_ID;
console.log(`Okay, we're trying to write to database: ${databaseEntry}`);
// here is where changes are needed
const dialogflowAgentRef = db.collection("WaterConsumption").doc(genRand(8)); // need to change to add()
console.log(`Found agent ref: ${dialogflowAgentRef}`);
return db.runTransaction(t => {
t.set(dialogflowAgentRef, databaseEntry); // this will also need to be changed
return Promise.resolve('Write complete');
}).then(doc => {
agent.add(`I have updated your daily water consumption`);
agent.add(`Is anything else I can do for you?.`);
}).catch(err => {
console.log(`Error writing to Firestore: ${err}`);
agent.add(`Failed to write "${dialogflowAgentRef}" to the Firestore database.`);
});
}
I have changed the important lines to:
console.log(`Okay, we're trying to write to database: ${databaseEntry}`);
const dialogflowAgentRef = db.collection("WaterConsumption"); // changed here
console.log(`Found agent ref: ${dialogflowAgentRef}`);
return db.runTransaction(t => {
t.set(db.collection("WaterConsumption").add(databaseEntry), databaseEntry); // and here
Which does successfully write to the db and auto generates an ID. Although an error is being caught:
Argument "documentRef" is not a valid DocumentReference. Input is not a plain JavaScript object
This makes no sense:
t.set(db.collection("WaterConsumption").add(databaseEntry), databaseEntry);
In this line the db.collection("WaterConsumption").add(databaseEntry) is a call to Firestore to create a document outside of the transaction). What you want instead is to just create a new DocumentReference with a unique ID, which (as shown in the 3rd snippet in the documentation on adding documents) can be done with doc().
So:
const newDocRef = db.collection("WaterConsumption").doc();
t.set(newDocRef, databaseEntry);
I’m new to mongo DB. What will be the schema or schemas what is user inputted, how we can recognize those in our code, or rather what will be the query (using javascript ).
Question:
Suppose we have a book schema and saved some books in the database, now we have to make a post api /getParticularBooks
take any input and use it as a condition to fetch books that satisfy that condition
e.g
if body had { name: “hi”} then you would fetch the books with this name
if body had { year: 2020} then you would fetch the books in this year
Or maybe the both at same time
hence the condition will differ based on what you input in the request body
You can start by creating a function which verify if the user request body doesn't have any input property which isn't define in your schema.
I would do it like that
const verifyProperty = (user_request_body) => {
const properties = [list of your schema property];
for(let i in properties){
const string = properties[i]
if(!(string in user_request_body))
return false;
}
return true;
}
And now, based on the return value of the verifyProperty function, you can send a query or return an error message.
if(verifyProperty(req.body) == false){
return res.json({message: "Invalid properties"})
}
const data = await Books.find(req.body);
return res.json({data})
In order to use await, your function must be async
I have a large csv and I need to read t and insert to mongodb
Csv contains user name, category name and policy name.
Need to insert Users into User collection with category id and policy id. Csv provides only the category name and policy name. So I need to fetch category id from collection using its name.
If category name not exist, create a new one and returns its id. Same case for policy.
So I tried
fs.createReadStream('./data_sheet.csv')
.pipe(csv())
.on('data', async (row) => {
// console.log(row)
let res = await Category.findOneOrCreate({ name: row.cat.trim() });
console.log(res)
})
.on('end', () => {
console.log('CSV file successfully processed');
});
categorySchema.statics.findOneOrCreate = async function findOneOrCreate(condition) {
try {
const self = this
let agent = await self.findOne(condition)
console.log("condition")
console.log(condition)
console.log("agent")
console.log(agent)
if (agent) return agent._id
else {
agent = await self.create(condition)
return agent._id
}
} catch (e) {
console.log(e)
}
}
This is not working in proper manner. What is the proper way to do this?
If by not working you mean, the data is not coming up for category then make sure you follow the right async approach or else provide more info.
There are several things to keep in mind,
You might need to create a cronjob for recursive process
Import the csv file into an array of Object
Loop over the object to match with category Id
If not then create a category Id
Return to the main function
update the document [here you can either update all or update one at a time using the step 2 looping we did]
I'm trying to get the Discord bot to create a database that is basically the user map (a row for each user and columns for ID, nick name, avatar URL, etc) when it receives a !getdata command.
I've gotten to the point where the database successfully takes data, in this case the username and user ID, but it displays all the unique values in two columns as long comma separated values (i.e. the username column displays 'user1,user2,user3').
I'm sure this is by design, but I'm really struggling with restructuring. I'd like to either have it take all the data from an object map (client.users or message.guild.members) but I cannot figure it out.
The other option, which is what I'm trying now, is to create a row for each user and then fill in the values that I want to store, but I'm getting nowhere fast.
I'm very new with SQLite (and node/DiscordJS/JS for that matter), so any advice is greatly appreciated.
Index.js
const Discord = require('discord.js');
const client = new Discord.Client();
const sql = require('sqlite3');
let db = new sql.Database("users.sqlite", (err) => {
if (err) {
console.log('Error connecting to the database', err)
} else {
console.log('Database connected.')
}
})
let token = process.env.CLIENT_TOKEN;
let prefix = process.env.PREFIX ;
client.on('ready', () => {
console.log(`Logged in as ${client.user.tag}!`);
db.run(`CREATE TABLE IF NOT EXISTS users(username TEXT, id TEXT)`);
})
client.on('message', function(message) {
if (!message.content.startsWith(prefix));
const args = message.content.slice(prefix.length).split(/ +/);
const command = args.shift().toLowerCase();
if (command === 'getdata') {
let username = message.guild.members.map(m=>m.user.username);
let userid = message.guild.members.map(m=>m.user.id);
db.run(`INSERT OR REPLACE INTO users(username, id) VALUES(?,?)`, [`${username}`,`${userid}`]);
return message.channel.send(`User database updated.`);
}
});
client.login(token);
If you're curious as to the formatting or way things are written, the answer is two fold:
I'm pretty new at this
This was the only way I could get the values in the database to return something other than null
Thanks in advance,
First off, welcome to the site.
I hope that I can shine some light here without diving into refactoring your code or making you change anything major.
One thing sticks out to me as to why you are storing an array instead of a single value.
let username = message.guild.members.map(m=>m.user.username);
let userid = message.guild.members.map(m=>m.user.id);
The .map call returns an array, not a single value.
Each user that issues a command is part of the message object. If I remember correctly, you would want this to be something like...
(simplified version)
const { username, id } = message.member.user;
db.run(`INSERT OR REPLACE INTO users(username, id) VALUES(?,?)`, [username, id]);
// ...
User documentation can be found here
Edit:
If you wanted to build the database for all users in that one command you could do something like the following with a bulk insert... (quick and dirty)
db.serialize(() => {
db.run('BEGIN TRANSACTION;');
// execute inserts in transaction
for (const m of message.guild.members) {
db.run('INSERT OR REPLACE INTO users(username, id) VALUES(?,?);', [m.user.username, m.user.id]);
}
// commit all inserts :)
db.run('COMMIT;')
});
message.channel.send('User database updated.');
Control flow documenation
Hopefully this points you in the right direction :)
I am attempting to perform an update to a MongoDB document (using mongoose) by first using .findById to get the document, then updating the fields in that document with new values. I am still a bit new to this so I used a tutorial to figure out how to get it working, then I have been updating my code for my needs. Here is the tutorial: MEAN App Tutorial with Angular 4. The original code had a schema defined, but my requirement is for a generic MongoDB interface that will simply take whatever payload is sent to it and send it along to MongoDB. The original tutorial had something like this:
exports.updateTodo = async function(todo){
var id = todo.id
try{
//Find the old Todo Object by the Id
var oldTodo = await ToDo.findById(id);
}catch(e){
throw Error("Error occured while Finding the Todo")
}
// If no old Todo Object exists return false
if(!oldTodo){
return false;
}
console.log(oldTodo)
//Edit the Todo Object
oldTodo.title = todo.title
oldTodo.description = todo.description
oldTodo.status = todo.status
console.log(oldTodo)
try{
var savedTodo = await oldTodo.save()
return savedTodo;
}catch(e){
throw Error("And Error occured while updating the Todo");
}
}
However, since I don't want a schema and want to allow anything through, I don't want to assign static values to specific field names like, title, description, status, etc. So, I came up with this:
exports.updateData = async function(update){
var id = update.id
// Check the existence of the query parameters, If they don't exist then assign a default value
var dbName = update.dbName ? update.dbName : 'test'
var collection = update.collection ? update.collection : 'testing';
const Test = mongoose.model(dbName, TestSchema, collection);
try{
//Find the existing Test object by the Id
var existingData = await Test.findById(id);
}catch(e){
throw Error("Error occurred while finding the Test document - " + e)
}
// If no existing Test object exists return false
if(!existingData){
return false;
}
console.log("Existing document is " + existingData)
//Edit the Test object
existingData = JSON.parse(JSON.stringify(update))
//This was another way to overwrite existing field values, but
//performs a "shallow copy" so it's not desireable
//existingData = Object.assign({}, existingData, update)
//existingData.title = update.title
//existingData.description = update.description
//existingData.status = update.status
console.log("New data is " + existingData)
try{
var savedOutput = await existingData.save()
return savedOutput;
}catch(e){
throw Error("An error occurred while updating the Test document - " + e);
}
}
My original problem with this was that I had a lot of issues getting the new values to overwrite the old ones. Now that that's been solved, I am getting the error of "TypeError: existingData.save is not a function". I am thinking the data type changed or something, and now it is not being accepted. When I uncomment the static values that were in the old tutorial code, it works. This is further supported by my console logging before and after I join the objects, because the first one prints the actual data and the second one prints [object Object]. However, I can't seem to figure out what it's expecting. Any help would be greatly appreciated.
EDIT: I figured it out. Apparently Mongoose has its own data type of "Model" which gets changed if you do anything crazy to the underlying data by using things like JSON.stringify. I used Object.prototype.constructor to figure out the actual object type like so:
console.log("THIS IS BEFORE: " + existingData.constructor);
existingData = JSON.parse(JSON.stringify(update));
console.log("THIS IS AFTER: " + existingData.constructor);
And I got this:
THIS IS BEFORE: function model(doc, fields, skipId) {
model.hooks.execPreSync('createModel', doc);
if (!(this instanceof model)) {
return new model(doc, fields, skipId);
}
Model.call(this, doc, fields, skipId);
}
THIS IS AFTER: function Object() { [native code] }
Which showed me what was actually going on. I added this to fix it:
existingData = new Test(JSON.parse(JSON.stringify(update)));
On a related note, I should probably just use the native MongoDB driver at this point, but it's working, so I'll just put it on my to do list for now.
You've now found a solution but I would suggest using the MongoDB driver which would make your code look something along the lines of this and would make the origional issue disappear:
// MongoDB Settings
const MongoClient = require(`mongodb`).MongoClient;
const mongodb_uri = `mongodb+srv://${REPLACE_mongodb_username}:${REPLACE_mongodb_password}#url-here.gcp.mongodb.net/test`;
const db_name = `test`;
let db; // allows us to reuse the database connection once it is opened
// Open MongoDB Connection
const open_database_connection = async () => {
try {
client = await MongoClient.connect(mongodb_uri);
} catch (err) { throw new Error(err); }
db = client.db(db_name);
};
exports.updateData = async update => {
// open database connection if it isn't already open
try {
if (!db) await open_database_connection();
} catch (err) { throw new Error(err); }
// update document
let savedOutput;
try {
savedOutput = await db.collection(`testing`).updateOne( // .save() is being depreciated
{ // filter
_id: update.id // the '_id' might need to be 'id' depending on how you have set your collection up, usually it is '_id'
},
$set: { // I've assumed that you are overwriting the fields you are updating hence the '$set' operator
update // update here - this is assuming that the update object only contains fields that should be updated
}
// If you want to add a new document if the id isn't found add the below line
// ,{ upsert: true }
);
} catch (err) { throw new Error(`An error occurred while updating the Test document - ${err}`); }
if (savedOutput.matchedCount !== 1) return false; // if you add in '{ upsert: true }' above, then remove this line as it will create a new document
return savedOutput;
}
The collection testing would need to be created before this code but this is only a one-time thing and is very easy - if you are using MongoDB Atlas then you can use MongoDB Compass / go in your online admin to create the collection without a single line of code...
As far as I can see you should need to duplicate the update object. The above reduces the database calls from 2 to one and allows you to reuse the database connection, potentially anywhere else in the application which would help to speed things up. Also don't store your MongoDB credentials directly in the code.