Develop Reference

Executing mysql queries sequentially nodejs

I am pretty new to nodejs and am using it to construct an api with a mysql database and have run into an issue where I am unable to execute mysql queries sequentially.
The database structure is that there are three tables. Table a in a one to many relation with table b, which is in a one to many relation with table c.
I need a GET endpoint where aid is given and it returns a result with an array of items of b with the array of items of c nested inside it.
Sample Result:
{
logcode: "",
logmessage: "",
bitems: [{
bitemid: 1
citems: [{
citemid: 1
} {
citemid: 2
}]
}{
bitemid: 2
citems: [{
citemid: 3
}]
}]
}
Currently I am attempting to do this by first executing a query where I retrieve all the values of type b that correspond to the received key of entity a and then running a foreach loop over the results and extracting the bitem ids and then running another query within the foreach loop to get all items of table c with that specific foreign key.
async function (req, res) {
let functionname = 'getAllItems'
const conLocalPool = db.conLocalPool.promise();
var data = {}
const atoken = req.get('token');
var str = ``;
str = `call ${functionname}('${atoken}')`;
console.log(str);
try {
const [rows, fields] = await conLocalPool.query(str)
data = rows[0][0]
if (data.logcode === 20100) {
data.bitems = rows[1];
data.bitems.forEach(async (bitem) => {
var stmt = `Select * from \`table-c\` where bitemid=${bitem.id}`
try {
const [citemrows, citemfields] = await conLocalPool.query(stmt);
console.log(citemrows[1])
bitem.citems = citemrows[1];
return true;
}
catch (err) {
console.log(err);
return false;
}
})
}
res.status(200).send(data);
return true;
}
catch (err) {
res.status(500).send({
error: err.message
});
return false;
}
}
Using this function, I am able to get a response which contains all the bitems related to the aitemtoken but without the citems related to each individual bitem.
I would like some help on how to execute the first query and then the later queries based on the response that it retrieves.

According to your last comment, I'm sure that you can achieve it by doing JOIN. There are a few ways to do it but consider this example below:
SELECT * FROM owner o
JOIN dogs d ON o.id=d.owner_id
JOIN dog_toys t ON d.id=t.dog_id
WHERE o.id=1;
Assuming that you have 3 tables (owner, dogs & dog_toys) and each of the table have a relation of owner.id=dogs.owner_id and dogs.id=dog_toys.dog_id, you can simply JOIN all three of them in one query and modify the returned results in SELECT.
I've created a sample fiddle here : https://www.db-fiddle.com/f/ukBgL4M3NzkyTDC6nMGkJn/1

Snowflake Javascript Procedure Truncate Table not effective

I spent most of this week writing out some Javascript stored procedure in Snowflake to handle a few things. But I can't for the life of me figure out why Truncate Table doesn't work.
try {
var get_tables = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'SCHEMA' AND TABLE_CATALOG = 'DATABASE' ORDER BY TABLE_NAME DESC;"
var tableStmt = snowflake.createStatement( {sqlText: get_tables} );
var rs = tableStmt.execute();
while (rs.next()) {
var table = rs.getColumnValue(1);
var truncateTable = "TRUNCATE TABLE DATABASE.SCHEMA." + table
var truncateStmt = snowflake.createStatment( {sqlText: truncateTable} );
var truncateEx = truncateStmt.execute();
}
}
This pattern of creating queries is in their documentation and seems to work for every other query I've put together. But it fails for this one and I don't get a syntax error or anything. I've left out the catch and stuff because this is the only relevant part.
I have additionally tried
var truncateTable = "TRUNCATE TABLE DATABASE.SCHEMA." + table + ";"
I have tested this specific query against tables in a normal SQL query in Snowflake and confirmed that it works.
Am I missing something here?

this will work:
create or replace procedure util_db.public.truncate_table("table_name" varchar,"truncate" boolean)
returns string
language javascript
strict
execute as owner
as
$$
var sql_command = "Truncate table " + table_name ;
try {
if (truncate){
snowflake.execute (
{sqlText: sql_command}
);
return "Succeeded."; // Return a success/error indicator.
}
}
catch (err) {
throw(err);
// return "Failed: " + err; // Return a success/error indicator.
}
$$
;

Since you the SQL executes inside a try block, it won't display errors. Maybe you're handling that in a catch, but it's not shown.
If you comment out the try block, it should display the error when run. Maybe it's a permissions issue. Stored procedures can run with caller rights or owner rights. If the caller does not have the privilege to read from INFORMATION_SCHEMA or truncate the tables in the list, it will error out. You can call using owner's rights if that makes sense in the situation.
What happens when you run it like this?
//try {
var get_tables = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'SCHEMA' AND TABLE_CATALOG = 'DATABASE' ORDER BY TABLE_NAME DESC;"
var tableStmt = snowflake.createStatement( {sqlText: get_tables} );
var rs = tableStmt.execute();
while (rs.next()) {
var table = rs.getColumnValue(1);
var truncateTable = "TRUNCATE TABLE DATABASE.SCHEMA." + table
var truncateStmt = snowflake.createStatment( {sqlText: truncateTable} );
var truncateEx = truncateStmt.execute();
}
//}

MongoDB - Mongoose - TypeError: save is not a function

I am attempting to perform an update to a MongoDB document (using mongoose) by first using .findById to get the document, then updating the fields in that document with new values. I am still a bit new to this so I used a tutorial to figure out how to get it working, then I have been updating my code for my needs. Here is the tutorial: MEAN App Tutorial with Angular 4. The original code had a schema defined, but my requirement is for a generic MongoDB interface that will simply take whatever payload is sent to it and send it along to MongoDB. The original tutorial had something like this:
exports.updateTodo = async function(todo){
var id = todo.id
try{
//Find the old Todo Object by the Id
var oldTodo = await ToDo.findById(id);
}catch(e){
throw Error("Error occured while Finding the Todo")
}
// If no old Todo Object exists return false
if(!oldTodo){
return false;
}
console.log(oldTodo)
//Edit the Todo Object
oldTodo.title = todo.title
oldTodo.description = todo.description
oldTodo.status = todo.status
console.log(oldTodo)
try{
var savedTodo = await oldTodo.save()
return savedTodo;
}catch(e){
throw Error("And Error occured while updating the Todo");
}
}
However, since I don't want a schema and want to allow anything through, I don't want to assign static values to specific field names like, title, description, status, etc. So, I came up with this:
exports.updateData = async function(update){
var id = update.id
// Check the existence of the query parameters, If they don't exist then assign a default value
var dbName = update.dbName ? update.dbName : 'test'
var collection = update.collection ? update.collection : 'testing';
const Test = mongoose.model(dbName, TestSchema, collection);
try{
//Find the existing Test object by the Id
var existingData = await Test.findById(id);
}catch(e){
throw Error("Error occurred while finding the Test document - " + e)
}
// If no existing Test object exists return false
if(!existingData){
return false;
}
console.log("Existing document is " + existingData)
//Edit the Test object
existingData = JSON.parse(JSON.stringify(update))
//This was another way to overwrite existing field values, but
//performs a "shallow copy" so it's not desireable
//existingData = Object.assign({}, existingData, update)
//existingData.title = update.title
//existingData.description = update.description
//existingData.status = update.status
console.log("New data is " + existingData)
try{
var savedOutput = await existingData.save()
return savedOutput;
}catch(e){
throw Error("An error occurred while updating the Test document - " + e);
}
}
My original problem with this was that I had a lot of issues getting the new values to overwrite the old ones. Now that that's been solved, I am getting the error of "TypeError: existingData.save is not a function". I am thinking the data type changed or something, and now it is not being accepted. When I uncomment the static values that were in the old tutorial code, it works. This is further supported by my console logging before and after I join the objects, because the first one prints the actual data and the second one prints [object Object]. However, I can't seem to figure out what it's expecting. Any help would be greatly appreciated.
EDIT: I figured it out. Apparently Mongoose has its own data type of "Model" which gets changed if you do anything crazy to the underlying data by using things like JSON.stringify. I used Object.prototype.constructor to figure out the actual object type like so:
console.log("THIS IS BEFORE: " + existingData.constructor);
existingData = JSON.parse(JSON.stringify(update));
console.log("THIS IS AFTER: " + existingData.constructor);
And I got this:
THIS IS BEFORE: function model(doc, fields, skipId) {
model.hooks.execPreSync('createModel', doc);
if (!(this instanceof model)) {
return new model(doc, fields, skipId);
}
Model.call(this, doc, fields, skipId);
}
THIS IS AFTER: function Object() { [native code] }
Which showed me what was actually going on. I added this to fix it:
existingData = new Test(JSON.parse(JSON.stringify(update)));
On a related note, I should probably just use the native MongoDB driver at this point, but it's working, so I'll just put it on my to do list for now.

You've now found a solution but I would suggest using the MongoDB driver which would make your code look something along the lines of this and would make the origional issue disappear:
// MongoDB Settings
const MongoClient = require(`mongodb`).MongoClient;
const mongodb_uri = `mongodb+srv://${REPLACE_mongodb_username}:${REPLACE_mongodb_password}#url-here.gcp.mongodb.net/test`;
const db_name = `test`;
let db; // allows us to reuse the database connection once it is opened
// Open MongoDB Connection
const open_database_connection = async () => {
try {
client = await MongoClient.connect(mongodb_uri);
} catch (err) { throw new Error(err); }
db = client.db(db_name);
};
exports.updateData = async update => {
// open database connection if it isn't already open
try {
if (!db) await open_database_connection();
} catch (err) { throw new Error(err); }
// update document
let savedOutput;
try {
savedOutput = await db.collection(`testing`).updateOne( // .save() is being depreciated
{ // filter
_id: update.id // the '_id' might need to be 'id' depending on how you have set your collection up, usually it is '_id'
},
$set: { // I've assumed that you are overwriting the fields you are updating hence the '$set' operator
update // update here - this is assuming that the update object only contains fields that should be updated
}
// If you want to add a new document if the id isn't found add the below line
// ,{ upsert: true }
);
} catch (err) { throw new Error(`An error occurred while updating the Test document - ${err}`); }
if (savedOutput.matchedCount !== 1) return false; // if you add in '{ upsert: true }' above, then remove this line as it will create a new document
return savedOutput;
}
The collection testing would need to be created before this code but this is only a one-time thing and is very easy - if you are using MongoDB Atlas then you can use MongoDB Compass / go in your online admin to create the collection without a single line of code...
As far as I can see you should need to duplicate the update object. The above reduces the database calls from 2 to one and allows you to reuse the database connection, potentially anywhere else in the application which would help to speed things up. Also don't store your MongoDB credentials directly in the code.

How to use multiple callbacks when looping data

I'm trying to get HTML form data, loop it through, change it a bit and insert it to database. I have tried like below app.js.
How can I make callbacks so that formdata what I have modified is available for .create function?
I have searched from everywhere and I always end up in dead end and undefined variable somehow.
app.js:
//Find the day where to save
Day.findById(req.params.id, function(err, day) {
if (err) {
console.log(err);
res.redirect("/diary");
} else {
// Search function to find data with _id
function ingredientIdQuery(reqBodyId) {
var ingQuery = Ingredient.find({_id:reqBodyId});
return dbQuery;
}
// This loops through HTML formdata and formats it for mongoose model
for (var i = 0; i < req.body.amount.length; i++) {
if (req.body.amount[i] !== "") {
var amount = Number(req.body.amount[i]);
var singleMealTempObj = {};
singleMealTempObj.amount = amount;
var _id = req.body.id[i];
var query = ingredientIdQuery(_id);
// Executing the query for the data I need with id
query.exec(function(err, ingr){
if(err) {
return console.log(err);
} else {
singleMealTempObj.ingredient = ingr[0];
singleMealTempArr.push(singleMealTempObj);
}
});
}
}
}
// This inserts data into day
Meal.create(singleMealTempArr, function(err, singleMealObject) {
if (err) {
console.log(err);
} else {
day.meals.push(singleMealObject);
day.save();
res.redirect("/day/" + day._id + "/dayshow");
}
});
});
});
Edit:
Thanks for reply and notices! While I was trying to do everything to get this work I missed those few things like declaring variables. Sorry for that. I threw the towel in to the cage at this point.
flow goes like this:
User sends HTML form data to app.js which is inside object of two arrays (id[] and amount[]). Amount array needs to be looped through if it has value other than 0. Same index id array value is used to fetch data from database. This data what is found from database with id from id[] is used with same index amount[] and it should be saved to mongo.
I can get the values from HTML form ok. but I have tried to make a search in Mongo in a for loop (query.exec in the code) I get the data ok. When I log the data outside the database query, variable is undefined.
I hope this clarifys a bit what I'm trying to achieve.
I'll continue this later... :)

I guess issue originates because of this function.
function ingredientIdQuery(reqBodyId) {
var ingQuery = Ingredient.find({_id:reqBodyId});
return dbQuery;
}
Is find function asynchronous or synchronous?
Also you are returning dbQuery but dbQuery does not seem to be changed inside the function.

Couple I noticed that may fix this:
You never define singleMealTempArr, so when you try to push data to it, you are gonna run into problems.
Your ingredientIdQuery function returns dbquery - which also isn't defined. You actually call it ingQuery. Even so...are you positive that this will return the data that you want?
// lets loop through all the form fields in req.body.amount
for (var i = 0; i < req.body.amount.length; i++) {
// keep going unless the form field is empty
if (req.body.amount[i] !== "") {
// assign all the form fields to the following vars
var amount = Number(req.body.amount[i]);
var singleMealTempObj = {};
singleMealTempObj.amount = amount;
var _id = req.body.id[i];
var query = ingredientIdQuery(_id);
// we are executing the ingredientIdQuery(_id), better
// double-check that this query returns the result we are
// looking for!
query.exec(function(err, ingr){
if(err) {
return console.log(err);
} else {
singleMealTempObj.ingredient = ingr[0];
// now that we've gone through and mapped all the form
// data we can assign it to the singleMealTempArr
// WOOPS! Looks like we forgot to assign it!
singleMealTempArr.push(singleMealTempObj);
}
});
}
}
}

How to design a chained Model.find() function?

I'm trying to write a ORM in Node.js. I want to declare a class named Model which will be used to declare a data object, like:
Users = new Model(someModelRules);
newUser = new Users(userInfomation);
the data model User have a function named find(). Now, I want to make find() chained, like:
Users.find(" name = 'John' ")
.orderedBy("age").desc()
.limit(0,10)
or maybe just a simply find:
Users.find(" name = 'John' ")
to code this find function, I believe I must build the SQL first,and do the SQL query at the end of this find chain.
I don't know how to do this, all I can think of is to add a function like: doQuery(), so I will know it's time to do the SQL query when the doQuery() function was called, like:
Users.find(" name = 'John' ")
.orderedBy("age").desc()
.limit(0,10)
.doQuery();
I know this is a simply solution, but I don't want the extra doQuery() function. :(
So, how should I design this? It would so nice of you if you can show me some example code with comments.
Thx! (sorry for my poor English)
ps. I know the ORM2 has a find function I just want, but I wanna know how to code it and I can barely understand the code in ORM2 as there are no comments. (I'm not gonna use orm2.)
================================= SOLUTION ==============================
Inspired by #bfavaretto :
function User() {
this.find = function(id, condition) {
return new findChain(id, condition);
}
}
function findChain(id, condition) {
this._id = id
this._condition = condition
this.queryTimerSet = false;
this.scheduleQuery = function () {
var self = this;
if(!self.queryTimerSet) {
console.log('[TEST CASE: ' + self._id + '] Insert query into eventLoop');
setTimeout(function(){
console.log('[TEST CASE: ' + self._id + '] Start query: '+self._condition);
}, 0);
self.queryTimerSet = true;
} else {
console.log('[TEST CASE: ' + self._id + '] No need to insert another query');
}
}
this.orderedBy = function(column) {
console.log('[TEST CASE: ' + this._id + '] orderedBy was called');
this._condition = this._condition + ' ORDER BY ' + column
this.scheduleQuery();
return this;
}
this.desc = function() {
// simply add DESC to the end of sql
this._condition = this._condition + ' DESC'
}
this.scheduleQuery();
}
var user = new User();
user.find(1,'SELECT * FROM test').orderedBy('NAME1').desc();
user.find(2,'SELECT * FROM test').orderedBy('NAME2');
user.find(3,'SELECT * FROM test');
runnning this code, you will get the result:
[TEST CASE: 1] Insert query into eventLoop
[TEST CASE: 1] orderedBy was called
[TEST CASE: 1] No need to insert another query
[TEST CASE: 2] Insert query into eventLoop
[TEST CASE: 2] orderedBy was called
[TEST CASE: 2] No need to insert another query
[TEST CASE: 3] Insert query into eventLoop
[TEST CASE: 1] Start query: SELECT * FROM test ORDER BY NAME1 DESC
[TEST CASE: 2] Start query: SELECT * FROM test ORDER BY NAME2
[TEST CASE: 3] Start query: SELECT * FROM test
I believe there must be a better way to achieve this, but this is the best I can get for now.
Any comments?

It is possible to achieve that if you schedule the doQuery logic to run asynchronously (but as soon as possible). I am thinking on something like this:
function User() {
// Flag to control whether a timer was already setup
var queryTimerSet = false;
// This will schedule the query execution to the next tick of the
// event loop, if it wasn't already scheduled.
// This function is available to your model methods via closure.
function scheduleQuery() {
if(!queryTimerSet) {
setTimeout(function(){
// execute sql
// from the query callback, set queryTimerSet back to false
}, 0);
queryTimerSet = true;
}
}
this.find = function() {
// ... logic that builds the sql
scheduleQuery();
return this;
}
this.orderedBy = function() {
// ... logic that appends to the sql
scheduleQuery();
return this;
}
// etc.
}
One totally different approach is to have a single method for building the SQL, and passing the ORDER BY and LIMIT parameters in an options object. Then your call would look like this:
user.find({
what : " name = 'John' ",
orderedBy : "age DESC",
limit : "0,10"
});
This is more suited for SQL queries than what you're trying to do. What you have looks like noSQL stuff like MongoDB, where fetching the records and sorting are separate operations (I think).

You will always have to have a execute/doQuery function at the end of the chain.
This is because all the other functions before the doQuery help build the query that needs to be executed at the end.