Develop Reference

AWS S3 / Javascript callback issue

So, I'm having a problem with JavaScript asynchronous execution when making an API call to AWS S3.
I have a sequence of nested callbacks that are working fine up until a specific S3 call that my code is not waiting for. Here's my code:
getThumbUrls(contentIndex, function(data) {
console.log('Returning from getThumbUrls');
// let's just display thumbUrls[0] for now...
console.log('The thumbUrls are ' + data[0]);
});
getThumbUrls() looks like this:
function getThumbUrls(contentIndex, callback) {
console.log('Entering getThumbUrls');
var thumbUrls = [];
JSON.parse(contentIndex).forEach(videoKey => {
// get the thumbnail: bucket-name/thumbnails/<first-key>
console.log('videoKey = ' + videoKey);
getThumbFileName(videoKey, function(thumbFileName) {
console.log('Returning from getThumbFileName');
console.log('Returned thumb filename is ' + thumbFileName);
thumbUrls.push(CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName);
});
});
callback(thumbUrls);
}
And getThumbFileName() looks like this:
function getThumbFileName(videoKey, callback) {
console.log('Entering getThumbFileName...');
const s3 = new AWS.S3({
apiVersion: '2006-03-01',
params: {
Bucket: 'my-bucket-name'
}
});
// Get the name of the file.
params = {
Bucket: 'my-bucket-name',
Delimiter: '/',
Prefix: videoKey + '/' + THUMBS_FOLDER,
MaxKeys: 1
};
var urlKey;
//console.log('listObjects params = ' + JSON.stringify(params, null, 4));
s3.listObjectsV2(params, (err, data) => {
if (err) {
console.log(err, err.stack);
callback(err);
return;
}
var thumbsKey = data.Contents;
// MaxKeys was 1 bc first thumbnail key is good enough for now. Therefore, only one iteration.
thumbsKey.forEach(function (keys) {
console.log('thumbKey = ' + keys.Key);
urlKey = keys.Key;
});
});
callback(urlKey);
//callback('20161111-TheWind.jpg');
}
Obviously, what's happening is that execution doesn't wait for the s3.listObjectsV2 call to finish. I've verified that the entire flow works properly when all getThumbFileName() does is callback with the filename.
Would someone kindly show me how to force execution to wait for s3.listObjectsV2 to complete before calling back with undefined?

As discussed, you should avoid callbacks approach when dealing with asynchronous operations over iterations, due their difficulty.
(You can skip this section if you don't want to know motivation behind promises approach).
Just to mention, in a callback approach, you must have to wait for all callbacks to complete in your getThumbUrls(), using a if which will check if all callbacks has been called, then just call callback(thumbUrls); with all responses pushed into your thumbUrls array:
function getThumbUrls(contentIndex, callback) {
const thumbUrls = [];
// counter which will increment by one for every callback
let counter = 0;
JSON.parse(contentIndex).forEach(videoKey => {
getThumbFileName(videoKey, function (thumbFileName) {
thumbUrls.push(CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName);
// for each callback response you must add 1 to a counter and then
counter++;
// check if all callbacks already has been called
if (counter === JSON.parse(contentIndex).length) {
// right here, thumbsUrls are filled with all responses
callback(thumbUrls);
}
});
});
}
So, you can make use of Promises, and a Promise.all will be enough for you to handle all responses from api. You can study over internet and check your code below, which is using a promise approach. I've added some comments to help you understanding what is happening.
// when using promises, no callbacks is needed
getThumbUrls(contentIndex)
.then(function (data) {
console.log('Returning from getThumbUrls');
// let's just display thumbUrls[0] for now...
console.log('The thumbUrls are ' + data[0]);
})
// when using promises, no callbacks is needed
function getThumbUrls(contentIndex) {
console.log('Entering getThumbUrls');
// not needed anymore, Promise.all will return all values
// var thumbUrls = [];
// Promise.all receives an array of promises and returns to next .then() all results
// changing forEach to map to return promises to my Promise.all
return Promise.all(JSON.parse(contentIndex).map(videoKey => {
console.log('videoKey = ' + videoKey);
// returning a promise
return getThumbFileName(videoKey)
.then(function (thumbFileName) {
console.log('Returning from getThumbFileName');
console.log('Returned thumb filename is ' + thumbFileName);
return CLOUDFRONT_URL + videoKey + '/thumbnails/' + thumbFileName;
});
}))
}
// when using promises, no callbacks is needed
function getThumbFileName(videoKey) {
console.log('Entering getThumbFileName...');
const s3 = new AWS.S3({
apiVersion: '2006-03-01',
params: {
Bucket: 'my-bucket-name'
}
});
// Get the name of the file.
params = {
Bucket: 'my-bucket-name',
Delimiter: '/',
Prefix: videoKey + '/' + THUMBS_FOLDER,
MaxKeys: 1
};
// urlKey not need anymore
// var urlKey;
// most of AWS functions has a .promise() method which returns a promise instead calling callback funcions
return s3.listObjectsV2(params).promise()
.then(function (data) {
var thumbsKey = data.Contents;
//if you want to return only first one thumbsKey:
return thumbsKey[0];
})
.catch(function (err) {
console.log(err, err.stack);
callback(err);
return;
})
}
Hope this helps you out in your study.

Would someone kindly show me how to force execution to wait
That's the wrong question. You are not trying to get execution to "wait," or, at least, you shouldn't be. You just need to call the callback in the right place -- inside the callback from s3.listObjectsV2(), not outside.
function getThumbFileName(videoKey, callback) {
...
s3.listObjectsV2(params, (err, data) => {
if (err) {
...
}
var thumbsKey = data.Contents;
// MaxKeys was 1 bc first thumbnail key is good enough for now. Therefore, only one iteration.
thumbsKey.forEach(function (keys) {
console.log('thumbKey = ' + keys.Key);
urlKey = keys.Key;
});
callback(urlKey); // right
});
// wrong // callback(urlKey);
}
The way you wrote it, the callback fires after s3.getObjectsV2() begins to run -- not after it finishes (calls its own callback).

Watson Conversation - Oracle DB Integration

Good morning/afternoon all, I am trying to get Watson to return a response manually set from our Oracle Databases.
I am using async to sequentially access the database and return the response because the first problem I had was the DB query would not happen until after Watson had already returned a response. Async.waterfall fixes this.
My current problem: In the console I see everything logged correctly. The Database is queried, response.output.text is set correctly and then returned to Watson but no response ever appears in my chat.. If I set the response.output.text = "asdfa" before the async, asdfa is returned as expected.
I am at a loss trying to figure this out so I appreciate any and all help. Please let me know if any more information is required.
// Send the input to the conversation service
conversation.message(payload, function (err, data) {
if (err) {
return res.status(err.code || 500).json(err);
}
return res.json(updateMessage(payload, data));
});
});
function updateMessage(input, response) {
var responseText = null;
if (!response.output) {
response.output = {};
} else {
// checkNames check
if (response.output.nodes_visited[0] === 'slot_11_1519333387192' && response.entities[0].entity === 'confirm') {
/* This code actually returns asdfa as a response from Watson.. */
// response.output.text = "asdfa";
// return response;
async.waterfall([
// this function queries the database
// TODO: module out the Oracle connection parts once POC is completed
function queryDB(callback) {
console.log('Starting queryDB');
var query = "SELECT column_name FROM table#prod WHERE column_id = '" + response.context.VSUID + "'";
oracledb.getConnection('hr',
function (err, connection) {
var conn = oracleGetConnection(err, connection);
conn.execute(query, {}, {
outFormat: oracledb.OBJECT
},
function (err, result) {
console.log('result from Oracle: ', result);
// pass a null error and the result of the query
callback(null, result.rows[0]);
});
});
},
// this function formats the result of the query
// TODO: this should not be it's own function. This can happen at the same time the db gets the row
function formatName (arg1, callback) {
console.log('this should happen after query..');
console.log('arg1: ', arg1);
var r = JSON.stringify(arg1);
r = r.substring((r.indexOf(':') + 1) + 1, r.length - 2);
console.log('Name is: ', r);
// pass a null error and the formatted name
callback(null, r);
}
],
// Final function to be ran after the two above have completed
function finalFunction (err, result) {
if (err) {
console.log('uh oh async err: ', err);
} else {
console.log('This is final Function');
// set output text
response.output.text = 'Is your name ' + result + '?';
// response.context.dbResponse = 'Is your name ' + result + '?';
// response.output.text = "asdfasdfasd";
// console.log('This is the value of response\n\n', response);
// var resp = returnResponse(input, response);
response.context.dbResponse = response.output.text[0];
return returnResponse(input, response);
// return response;
}
});
// response.output.text = "asdfa";
console.log('This is response.output.text ', response.output.text);
return response;
} else {
//If no special if case to query the db just run Watson Conversation stock
return returnResponse(input, response);
}
}
}
Here is a sample console log.
This logs the Input from the user:
name 111111111
This logs the Response from Watson:
Is 111111111correct?
This logs the intent recognized, if any:
nameCheck
This logs the entity recognized, if any:
VSUID
This logs the text that is being returned to the user: [ 'Is 111111111correct?'
]
Starting queryDB
Connected to database
result from Oracle: { outBinds: undefined,
rowsAffected: undefined,
metaData: [ { name: 'TABLE_FIRST_NAME' } ],
rows: [ [ 'Tyler' ], [ 'Tyler' ] ],
resultSet: undefined }
this should happen after query..
arg1: [ 'Tyler' ]
Name is: "Tyler
This is final Function
This logs the Input from the user:
yes
This logs the Response from Watson:
Is your name "Tyler?
This logs the entity recognized, if any:
confirm
This logs the text that is being returned to the user: Is your name "Tyler?

Writing code like the following will open you up to SQL injection vulnerabilities (and likely performance problems too):
var query = "SELECT column_name FROM table#prod WHERE column_id = '" + response.context.VSUID + "'";
Please read the section of the documentation on bind variables.
On to your question...
You are treating updateMessage as though it is a synchronous function, but it is performing asynchronous work. A function that is performing asynchronous work will need an asynchronous API, such as Node.js style callbacks, Promises, or AsyncFunctions (async/await).
If you see line 73 of the code you supplied, you're "returning" the response object, but that is outside of the async.waterfall call. Even the return on line 67 will not work because of the asynchronous nature of Node.js.
Here's my latest attempt at describing how all of this works:
https://www.youtube.com/watch?v=iAdeljxq_hs
You can access the slides and sample code here:
https://www.dropbox.com/s/quu7oxiug0gh6ua/Understanding%20Async%20Processing%20and%20Patterns%20in%20Node.js.zip?dl=0
In the sample code's code > header-detail directory, you'll see 5 different files that start with header-detail-with- followed but the name of a different API choice you could make. You will have to make a similar choice with your updateMessage API.
To run a test, use the ddl.sql file to create the target tables, then edit db-config.js as needed for your environment, and finally run node test.js 1 from a terminal in that directory. You can change the number at the end to run a different test file.

Function Return not Working

I am working in Node and trying to load the next sequence from my db. I am able to access the db, load and return the sequence within my function, but I am not able to access it outside of the function.
function getRunId() {
counters.findOne({_id: 'Run_ID'}, function(err, resp) {
if(err) {
console.log(err);
}
console.log('Seq: ' + resp.sequence); // Console Output = Seq: 1234
return resp.sequence;
});
};
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId); // Console Output = CRID: undefined
I've checked several pages worth of Stack Overflow issues relating to using callback's, async (node module), how to properly return values in the function, etc... but none of them get me closer to accessing currentRunId outside of the function.
Is this issue further complicated by the use of Mongo queries inside my function?

For anyone stumbling on this later, start by reading this answer.
I've dealt with this a few times so I understand the frustration. You are trying to mix sync and async code by doing this:
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId);
The trouble is that console.log('Run_ID: ' + currentRunId) is called immediately after you invoke getRunID() by assigning it to current RunID, and getRunID() resolves after console.log('Run_ID: ' + currentRunId), causing the currentRunId variable to be undefined.
But, you have some options to deal with this. Option one is to return a callback, and log the results of the callback instead. Option 2 is to use an ES6 promise. To use option 2, you need node version 7, and you need to use 'use strict' in your code.
Here are 3 examples built around a function stub that spoofs the results of findOne(). The getRunIdA() is your function, and getRunIdB, and getRunIdC are two example solutions to your current problem.
'use strict'
// A function stub which represents a simplified version of findOne.
// Accepts callback and returns a callback with the results of data
function findOne (callback) {
var data = {
sequence: 6
}
return callback(null, data)
}
// This is a simplified version of your function, which reproduces the undefined result
function getRunIdA () {
findOne(function (err, resp) {
if (err) {
console.log(err)
}
console.log('Seq: ' + resp.sequence)
return resp.sequence
})
}
// This is your function with a callback
function getRunIdB (callback) {
findOne(function (err, resp) {
if (err) {
console.log(err)
}
console.log('Seq: ' + resp.sequence)
return callback(resp.sequence)
})
}
// This is your function with a promise
var getRunIdC = new Promise(function (resolve, reject) {
resolve(findOne(function (err, resp) {
if (err) {
console.log(err)
}
return resp.sequence
}))
})
// Invoke your funciton; get undefined
var currentRunID = getRunIdA()
console.log('Run_ID: ' + currentRunID) // Run_ID: undefined
// Invoke getRunIdB using callback, get 6
getRunIdB(function (id) {
console.log('Run_ID: ' + id) // Run_ID: 6
})
// Invoke getRunIdC with a promise; get 6
getRunIdC.then(function (currentRunID) {
console.log('Run_ID: ' + currentRunID) // Run_ID: 6
})
/*
results for all 3:
Seq: 6
Run_ID: undefined
Seq: 6
Run_ID: 6
Run_ID: 6
*/
Give this a try by saving to your machine and running:
node test.js
Is this issue further complicated by the use of Mongo queries inside my function?
Nope, you just need to pass the results of your query to a promise or a callback so that you can work with the results somewhere else.
I hope this helps!
Edit: OP added the following code in a comment, which I will try to break down and address.
Unfortunately, using getRunIdB results in callback is not defined and using getRunIdC results in currentRunId is not defined
var currentRunID = '';
var getRunId = new Promise(function (resolve, reject) { resolve(counters.findOne({_id: 'Run_ID'}, function (err, resp) {
if (err) {
console.log(err)
}
return resp.sequence;
}))
});
getRunId.then(function (res) {
console.log('Run_ID: ' + res.sequence) // Run_ID: 1234
currentRunID = res.sequence;
})
console.log(currentRunID); // currentRunID is not defined
Check out an answer I gave to a similar question for more details on the JS concurrency model. Simply put, the getRunID() function is executing asynchronous code. What that means is that getRunID() doesn't get inserted into the message queue that determines what order javascript will execute until it's callbacks are completed. Thus, when you log currentRunID outside of the .then() function, the results is undefined because currentRunID is undefined.
I think that ultimately what OP is trying to do is to export the result of the function so that the something can be done with those results, this needs to be done within a callback like so:
getRunId.then(function (res) {
// Do stuff with the run ID here.
})

You are only returning on a callback function but not on the actual function.. Change your code to this:
function getRunId() {
var result = counters.findOne({_id: 'Run_ID'}, function(err, resp) {
if(err) {
console.log(err);
}
console.log('Seq: ' + resp.sequence); // Console Output = Seq: 1234
return resp.sequence;
});
return result; //<-- return result of your function is here
};
var currentRunId = getRunId();
console.log('Run_ID: ' + currentRunId);

Why does Node.js appear to pass arguments differently in these two uses?

Check out this super-simple node.js program:
var g = { a : 1, b : 2 }
function callBack (key, value) {
console.log("Callback called with key: " + key + "\nAnd value: " + value) ;
}
function doNothing (key, value, cb) {
true ;
console.log(key + ": doing nothing") ;
cb() ;
}
function doLoop () {
for (k in g) {
f = function () {
callBack(k, g[k]) ;
}
doNothing(k, g[k], f) ;
}
}
doLoop() ;
When run, it produces this output:
a: doing nothing
Callback called with key: a
And value: 1
b: doing nothing
Callback called with key: b
And value: 2
OK. That makes sense - each time the callback is called, it has the correct arguments.
Now look at this program:
var mysql = require('mysql') ;
var dbClient = undefined ;
var db_uri = "mysql://xxx:xxx#127.0.0.1/xxx" ;
var schema = {
redirects : "(id int AUTO_INCREMENT, key VARCHAR(50), url VARCHAR(2048))",
clicks : "(ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP, IP VARBINARY(16))"
} ;
function createOnEmpty(err, results, fields, tableName, create_def) {
console.log("createOnEmpty called on " + tableName) ;
if (err) {
console.error(err) ;
process.exit(1) ;
} else {
if (0 == results.length) {
dbClient.query(["create table ", tableName, create_def].join(" "),
function (err, results, fields) {} ) ;
} else {
console.log(tableName + " table already exists.") ;
}
}
console.log("\n\n") ;
}
function setupSchema() {
for (table in schema) {
console.log("Checking for table: " + table) ;
// FIXME: Why does this always seem to pass clicks as tablename?!
dbClient.query(
"show tables LIKE '" + table + "'",
function (err, results, fields) {
createOnEmpty(err, results, fields, table, schema[table])
}
);
}
}
function handleDBConnect(err) {
if (err) {
console.error("ERROR: problem connecting to DB: " + err.code) ;
process.exit(1) ;
} else {
console.log("Connected to database.") ;
// Automatically set up the schema, if the tables don't exist
setupSchema() ;
}
}
function MySQLConnect() {
dbClient = mysql.createConnection(db_uri) ;
dbClient.connect(handleDBConnect) ;
}
MySQLConnect() ;
It outputs:
Connected to database.
Checking for table: redirects
Checking for table: clicks
createOnEmpty called on clicks
createOnEmpty called on clicks
The loop seems to be giving the argument 'clicks' as the argument 'table' both times, even though the variable has clearly been switched to 'redirects'.
I figure I must have some fundamental misunderstanding of how JavaScript/Node works here.

To understand this behaviour you need to understand this 2 core js concepts:
Closures
Eventloop
Let say we have global variable a, log function which outputs a to console, and main function which calls log twice, first time with timeout (asynchronously), second time just simple function call
var a = 42;
function log() {
console.log(`a is ${a}`);
}
function main() {
setTimeout(log, 100);
a = 13;
log();
}
main();
This code produces the following output:
a is 13
a is 13
Why the hell the first time a is 13?
When you call setTimeout, it doesn't block main js thread for 100ms, it just adds log function to callback queue. The next line is a = 13. As a wasn't declared inside function body with var keyword, 13 assigned to a which was declared on the first line of code. Then we have the first line of output as a result of the last line of main function. Now we have empty callstack, nothing else happening in our code, but we still have log function in callback queue. After 100ms passed, if and only if callstack is empty (which is our case), log function could be called second time. It logs 'a is 13' once again, as a-s value was already reassigned.
This is a short explanation of how async callbacks work in javascript, and this is the reason why createOnEmpty called on clicks twice. dbClient.query is asynchronous and by the time it was called first time, your for loop finished its execution and table value is clicks.
Quick and dirty solution of your problem will be
for (table in schema) {
console.log("Checking for table: " + table) ;
(function (table) {
dbClient.query("show tables LIKE '" + table + "'",
function (err, results, fields) {
createOnEmpty(err, results, fields, table,
schema[table]) } );
)(table);
}
This immediately called function memorizes table value on each loop iteration in scope
Check out this excellent talk about eventloop and how async stuff works in js
How closures work

By the time the callback of dbClient.query is called the loop has already finished, leaving your (implicitly global) table variable at the last key in schema.
You need to scope it using an anonymous function(see the various answers there) or use a callback based iteration like so:
function setupSchema() {
Object.keys(schema).forEach(function (table) {
console.log("Checking for table: " + table) ;
// FIXME: Why does this always seem to pass clicks as tablename?!
dbClient.query(
"show tables LIKE '" + table + "'",
function (err, results, fields) {
createOnEmpty(err, results, fields, table, schema[table])
}
);
});
}

Node & MySQL - connection.query inside connection.query - Object property not accessible

I got a pretty awkward problem.
I create a pool, connect to the database, create a connection and query, get the results, do a bunch of stuff, then I have to create another connection and query, but actually it has to be dynamically so I loop over my Array teacherHours containing the data.
Then more Code is happening, and I have to create an extra loop, because certain elements of my teacherHours Array have to try multiple times to get the correct response from the upcoming query.
So another loop follows, which is supposed to loop as long as availableHours > 0. Now, here is where it all goes left.
A buch of code happens inside the second loop, I prepare my second query, call connection.query() and inside of the callback function I prepare my third query (after doing some other stuff) and this is actually where Node kicks me out.
It gives me TypeError: Cannot read property 'tid' of undefined. tid needs to be accessed for my third query, so I try to access it just like I did before but Node doesn't allow it.
I know that the queries return useful data (rows) so it can't be a problem of querying but receiving no data. Actually I console.log("the rowRIDS"+rowRIDS); the result of the second query and I see that it returns 2 rows and just after that it gives me the error.
What is also strange to me, all the console.logs inside my my two loops are being logged, and my console.log of the second query (containing the 2 rows) are being logged after the loops ran through, since the queries are nested shouldn't the returned 2 rows and the error appear within the first iteration of the loop, since the code should access the second query at that point.
BTW, I've tried to set a hardcoded number instead of the tid just to get the next property datum to be an error. I kind of got a feeling as if the variable teacherHours is out of scope, but it is supposed to be a global variable.
To get a better feeling of what I'm talking about I copied the code and uncommented all the javascript code, where I populate and calculate stuff. Any help would be really great, its been almost 7 hours of try & error without any luck. Thank You!
pool.getConnection(function(err, connection){
if (err) throw err;
connection.query('SELECT * FROM teachers_teaching_tbl WHERE fwdid = 1 ', function(err, rows, fields) {
if (err) {
console.error('error querying: ' + err.stack);
return;
}
rowArray=rows;
console.log(rowArray);
//
// HERE HAPPENS
// A LOOOOT OF STUFF
//
// AND teacherHours IS BEING POPULATED
//
// THEN COMES A FOR LOOP
for(var i=0; i<teacherHours.length;i++){
//
// MORE STUFF
//
//AND ANOTHER LOOP
while(availableHours>0){//do{ ORIGINALLY I TRIED WITH A DO WHILE LOOP
//
// AGAIN A BUNCH OF STUFF
//
// NOW I'M PREPARING MY NEXT QUERY
//
var myQueryGetFreeRoom=" SELECT rms.rid FROM rooms_tbl as rms WHERE NOT EXISTS ( ";
myQueryGetFreeRoom+=" SELECT NULL FROM classes_tbl as cls ";
myQueryGetFreeRoom+=" WHERE ( (cls.bis > '"+bisMinus1+"' AND cls.bis <= '"+realBis+"' ) OR ( cls.von > '"+bisMinus1+"' AND cls.von < '"+realBis+"' ) ) AND (cls.datum = '"+teacherHours[i].datum.getFullYear()+"-"+(teacherHours[i].datum.getMonth()+1)+"-"+teacherHours[i].datum.getDate()+"') AND (cls.rid=rms.rid) ) ";
//
//
connection.query(myQueryGetFreeRoom, function(err, rowRIDS, fields) {
if (err) {
console.error('error querying: ' + err.stack);
return;
}
roomIDs=rowRIDS;
console.log("the rowRIDS"+rowRIDS);
//
// MORE STUFF
// HAPPENING
//
if(roomIDs.length>0){
//
// PREPARING QUERY NO.3 - WHICH IS WHERE MY ERROR POINTS - TO THE USE OF tid PROPERTY
//
var myQueryBookClass = " INSERT INTO classes_tbl ( rid , tid , belegtAnz, datum, von , bis ) ";
myQueryBookClass+=" VALUES ( "+Math.floor(Math.random() * roomIDs.length)+", "+teacherHours[i].tid+" , 0, '"+teacherHours[i].datum.getFullYear()+"-"+(teacherHours[i].datum.getMonth()+1)+"-"+teacherHours[i].datum.getDate()+"' , '"+bisMinus1+"' , '"+realBis+"' ) ";
console.log("myQueryBookClass: "+myQueryBookClass);
availableHours = 0;
//
// HERE WAS SUPPOSED TO FOLLOW QUERY 3 - myQueryBookClass
//
// BUT SINCE I DONT EVEN GET INSIDE HERE IT IS IN COMMENTS
//
/*connection.query(myQueryBookClass, function(err, insertRows, fields){
if(err){
console.error('error querying: '+err.stack);
return;
}
console.log("Inserted Rows: "+ insertRows);
}); */
} else {
availableHours= availableHours - 1;
//
// STUFF HAPPENING
//
}
});
availableHours= availableHours - 1;
}//while(availableHours>0);
//
}
connection.release(function(err){
if (err){
console.error('error disconnecting: ' + err.stack);
return;
}
});
});
});

I think you are coming from a non-async language like Python, Java, etc. which is why Node, i.e. JavaScript, seems to screw things up for you, but actually it isn't.
The problem you have in your code is that you execute async functions like query synchronously all at the same time in the same while loop. You need to use a module like async which helps to run and collect results asynchronously.
Here is the updated code.
var async = require('async'),
connection;
async.waterfall([
function (cb) {
pool.getConnection(cb);
},
function (conn, cb) {
connection = conn;
connection.query('SELECT * FROM teachers_teaching_tbl WHERE fwdid = 1', cb);
},
function (rows, fields, cb) {
rowArray = rows;
console.log(rowArray);
// HERE HAPPENS
// A LOOOOT OF STUFF
//
// AND teacherHours IS BEING POPULATED
//
// THEN COMES A FOR LOOP
async.eachSeries(teacherHours, function (teacherHour, done) {
// MORE STUFF
//
//AND ANOTHER LOOP
async.whilst(function () {
return availableHours > 0;
}, function (cb) {
// AGAIN A BUNCH OF STUFF
//
// NOW I'M PREPARING MY NEXT QUERY
//
var myQueryGetFreeRoom =
"SELECT rms.rid FROM rooms_tbl AS rms WHERE NOT EXISTS ("
+ "SELECT NULL FROM classes_tbl AS cls"
+ " WHERE ("
+ "(cls.bis > '" + bisMinus1 + "' AND cls.bis <= '" + realBis + "')"
+ " OR (cls.von > '" + bisMinus1 + "' AND cls.von < '" + realBis + "')"
+ ") AND ("
+ "cls.datum = '" + teacherHour.datum.getFullYear() + "-" + (teacherHour.datum.getMonth() + 1) + "-" + teacherHour.datum.getDate() + "'"
+ ") AND cls.rid = rms.rid";
async.waterfall([
function (cb) {
connection.query(myQueryGetFreeRoom, cb);
},
function(rowRIDS, fields, cb) {
roomIDs = rowRIDS;
console.log("the rowRIDS" + rowRIDS);
//
// MORE STUFF
// HAPPENING
//
if (roomIDs.length > 0) {
//
// PREPARING QUERY NO.3 - WHICH IS WHERE MY ERROR POINTS - TO THE USE OF tid PROPERTY
//
var myQueryBookClass = "INSERT INTO classes_tbl (rid, tid, belegtAnz, datum, von, bis) VALUES ("
+ Math.floor(Math.random() * roomIDs.length)
+ ", " + teacherHour.tid
+ ", 0, '" + teacherHour.datum.getFullYear() + "-" + (teacherHour.datum.getMonth() + 1) + "-" + teacherHour.datum.getDate() + "', '" + bisMinus1 + "', '" + realBis + "')";
console.log("myQueryBookClass: " + myQueryBookClass);
availableHours = 0;
//
// HERE WAS SUPPOSED TO FOLLOW QUERY 3 - myQueryBookClass
//
// BUT SINCE I DONT EVEN GET INSIDE HERE IT IS IN COMMENTS
//
connection.query(myQueryBookClass, function (err, insertRows, fields) {
if (err) {
console.error('error querying: '+err.stack);
return;
}
console.log("Inserted Rows: "+ insertRows);
// Here do whatever you need to do, then call the callback;
cb();
});
} else {
--availableHours;
//
// STUFF HAPPENING
//
cb();
}
}
], function (err) {
if (!err) {
// Notice that you are decrementing the `availableHours` twice here and above.
// Make sure this is what you want.
--availableHours;
}
cb(err);
});
}, done);
}, function (err) {
connection.release(function (err) {
if (err) {
console.error('error disconnecting: ' + err.stack);
return;
}
});
});
}
], function (err) {
conn && pool.release(conn);
err && throw err;
});
Next time please format your code properly for better readability which will help yourself to get answers faster, and split your question text into paragraphs for the same purpose.
Explanation
There are four nested async flows:
async.waterfall
-> async.eachSeries
-> async.whilst
-> async.waterfall
Basically, the async.waterfall library allows you to execute a list of functions in series.
Every next function will be executed only after the previous function has returned the response.
To indicate that a function is completed and the results are available, it has to call the callback, in our case it is cb (you can call it whatever you like, eg. callback). The rule is to call it, otherwise, the next function will never be executed because the previous one doesn't seem to have finished its work.
Once the previous function has completed, it calls the provided cb with the following signature:
cb(err, connection);
If there was an error while requesting for a connection, the entire async.waterfall will interrupt and the final callback function will be executed.
If there was no error, the connection will be provided as a second argument. async module passes all arguments of the previous function to the next function as the first, second, etc. arguments which is why the second function receives the conn as the first argument.
Every next function will receive the callback cb as the last argument, which you must eventually call when the job is done.
Thus, in the first async.waterfall flow:
It requests a new database connection.
Once the connection is available, the next function is executed which sends a query to the database.
Waits for the query results, then once the results are available, it is ready to run the next function which iterates over each row.
async.eachSeries allows to iterate over a gives array of values sequentially.
In the second async.eachSeries flow:
It iterates over each element in the teacherHours array sequentially.
Once each element is processed (however you want), you must call the done callback. Again, you could have called this as cb like in the previous async.waterfall or callback. done is just for clarity that the process is done.
Then we have the async.whilst which provides the same logic as the normal while () {} statement but handles the loop asynchronously.
In this third async.whilst flow:
Calls the first function. Its return value indicates whether it has to continue the loop, i.e. call the second asynchronous function.
If the return value is truthful (availableHours > 0), then the second function is called.
When the async function is done, it must call the provided callback cb to indicate that it is over. Then async module will call the first function to check if it has to continue the loop.
In this asynchronous function inside async.whilst we have another async.waterfall because you need to send queries to the database for each teacherHour.
In this last fourth async.watercall flow:
It sends the SELECT query to the database.
Waits for the response. Once the rowRIDS are available, it calls the second function in the waterfall.
If there are rowRIDS (roomIDs.length > 0), it sends the INSERT query to the database.
Once done, it calls the callback cb.
If there were no rowRIDs, it calls the callback cb, too, to indicate that the job is done.
It is a great thing that JavaScript is asynchronous. It might be difficult at the beginning when you convert from other synchronous languages, but once you get the idea, it will be hard to thing synchronously. It becomes so intuitive that you will start thinking why other languages don't work asynchronous.
I hope I could explain the above code thoroughly. Enjoy JavaScript! It's awesome!