I am working on a student project. My goal is to collect data and store it.
I use nodeJs with express, then I run queries with superagent on ThemoviesDb to collect movies and then store them on neo4j.
Here is my example :
app.get('/', function(req,res){
var hostname = "api.themoviedb.org/3";
var path = "/movie/";
// random id, to store random film
for(i=0;i<39;i++){
randomId = Math.floor(Math.random() * 500) + 80;
console.log(' --------------------- Random id is ---------------------- ' + randomId);
superagent.get(hostname + path + randomId)
.query({ api_key: api_key, append_to_response : 'credits' })
.end((err, res) => {
if (err) {
if(err.status_code = '34') {
return console.log('id : ' + randomId + 'ne correspond pas à un film')
}
else{
return console.log(err);
}
}
cpt ++;
title = res.body.original_title;
if(title){ // Test if title isn't nul, undefined, Nan, empty or 0 to add it
console.log('randomId --> ' + res.body.id + '--> ' + title );
if(!Filmadded.includes(film)){
Filmadded.push(film);
}
}
});
}
console.log('cpt : ' + cpt + '/39');
res.send('it works' + i );
});
I I'm just executing a loop ( 39 because the limit of the api is 40 )
and for each I make a request to get a movie.
The result :
As you can see i first have all the id's displayed then the titles that match the id.
--> I would like to wait until the request is over to move on.
I looked promised but I do not have everything.
I then think that my problem on id / film is due to this.
Thank you for your help and sorry for my english
Your superagent() call is asynchronous. As such, it does not block and your for loop just runs to completion starting all 39 superagent() calls in parallel. You could probably code to run all these in parallel (if the target host allows it), but since you asked for the ability to run them one after another, here's an implementation of your function using the promise capabilities of superagent() and await to serialize the async call inside your for loop so it runs one, waits for it to finish, then runs the next one:
app.get('/', async function(req, res){
let hostname = "api.themoviedb.org/3";
let path = "/movie/";
// random id, to store random film
for (let i = 0; i < 39; i++) {
let randomId = Math.floor(Math.random() * 500) + 80;
console.log(' --------------------- Random id is ---------------------- ' + randomId);
try {
let result = await superagent.get(hostname + path + randomId)
.query({ api_key: api_key, append_to_response : 'credits' });
cpt ++;
let title = result.body.original_title;
if (title) { // Test if title isn't nul, undefined, Nan, empty or 0 to add it
console.log('randomId --> ' + res.body.id + '--> ' + title );
if (!Filmadded.includes(film)) {
Filmadded.push(film);
}
}
} catch(err) {
if(err.status_code = '34') {
console.log('id : ' + randomId + 'ne correspond pas à un film')
}
else{
console.log(err);
}
}
console.log('cpt : ' + cpt + '/39');
}
res.send('it works' + i );
});
In addition, you need to make sure you're declaring all variables you are using so they are accidental globals that can conflict when there are other requests also running.
Other things that don't look correct about this code, but I don't know what you intend:
The variable cpt looks like it needs to be initialized and declared somewhere.
The variable Filmadded probably needs to be scoped locally (not some higher scoped variable that can conflict when multiple requests are running on your server).
It isn't clear what you actually intend to do for error handling here when a superagent() call fails. Here it just logs errors, but you probably need to be able to return an error status if you're getting errors.
As you said, superagent's get function is async, meaning that the event loop doesn't wait for the function to finish before executing the next command. So the loop initiates 40 executions of your loop, which includes creating a random id and then using superagent with that id. So we're talking about two actions - one is synchronous and the second is asynchronous.
Let's look at it in another way. Say we had the following loop:
for(i=0; i<39; i++) {
const randomId = Math.floor(Math.random() * 500) + 80;
console.log("RANDOM IS: ", randomId);
setTimeout(function(){
console.log("PRINT AGAIN: ", randomId);
}, 10000);
}
What you'll have here, is 40 rows of "RANDOM IS: [random_number]" in a consecutive manner, and only after 10 seconds you'll have 40 rows of "PRINT AGAIN: [random_number]", and that's because you set a timeout of 10 seconds for the second logging.
You can compare that setTimeout with 10 seconds to an async function - only in an async function you can't really tell when the function will finish. So basically what you have is similar to the above example - 40 loggings of the random number, and some random-timed promises executions.
So what you might want to consider is promise chaining using the reduce function of js Array, or use es6 async await function notation.
You can use superagent functions as promises, and use then and catch instead. Then, chaining the promises means you wait for one promise to finish and only then executing the following one.
Related
I'm trying to use async/await in my firebase function but I am getting an error. I have marked the function as async but when I try to use await inside of it I get error: Expression has type void. Put it on its own line as a statement.
I think this is strange because I think it each await function call is already in its own line as a statement. So, I'm not sure what to do. Any help would be appreciated.
For clarity, I am making a request with the Cheerio web scrape library, and then trying to make two async function calls during each loop with the .each method.
export const helloWorld = functions.https.onRequest((req, response) => {
const options = {
uri: 'https://www.cbssports.com/nba/scoreboard/',
transform: function (body) {
return cheerio.load(body);
}
};
request(options)
.then(($) => {
$('.live-update').each((i, element) => {
const homeTeamAbbr = $(element).find('tbody').children('tr').eq(0).find('a').html().split("alt/").pop().split('.svg')[0];
const awayTeamAbbr = $(element).find('tbody').children('tr').eq(1).find('a').html().split("alt/").pop().split('.svg')[0];
const homeTeam = $(element).find('tbody').children('tr').eq(0).find('a.team').text().trim();
const awayTeam = $(element).find('tbody').children('tr').eq(1).find('a.team').text().trim();
let homeTeamStatsURL = $(element).find('tbody').children('tr').eq(0).find('td').html();
let awayTeamStatsURL = $(element).find('tbody').children('tr').eq(1).find('td').html();
const gameTime = $(element).find('.pregame-date').text().trim();
homeTeamStatsURL = homeTeamStatsURL.match(/href="([^"]*)/)[1] + "roster";
awayTeamStatsURL = awayTeamStatsURL.match(/href="([^"]*)/)[1] + "roster";
const matchupString = awayTeamAbbr + "#" + homeTeamAbbr;
const URLString = "NBA_" + urlDate + "_" + matchupString;
// var docRef = database.collection('NBASchedule').doc("UpcommingSchedule");
// var boxScoreURL = "www.cbssports.com/nba/gametracker/boxscore/" + URLString;
// var setAda = docRef.set({[URLString]:{
// homeTeam: homeTeam,
// awayTeam: awayTeam,
// date: gameTime,
// homeTeamAbbr: homeTeamAbbr,
// awayTeamAbbr: awayTeamAbbr,
// homeTeamStatsURL: homeTeamStatsURL,
// awayTeamStatsURL: awayTeamStatsURL,
// boxScoreURL: boxScoreURL
// }}, { merge: true });
getTeamPlayers(homeTeamStatsURL, matchupString);
getTeamPlayers(awayTeamStatsURL, matchupString);
console.log("retrieved schedule for "+ matchupString + " on " + urlDate)
});
response.send("retrieved schedule");
})
.catch(function (err) {
console.log("error " + err);
});
});
The function I am calling just makes another request and then I'm trying to log some data.
function getTeamPlayers(playerStatsURL, matchupString) {
const options = {
uri: playerStatsURL,
transform: function (body) {
return cheerio.load(body);
}
};
console.log(playerStatsURL + " stats url");
request(options)
.then(($) => {
console.log('inside cheerio')
$('tbody').children('tr').each(function(i, element){
const playerName = $(element).children('td').eq(1).children('span').eq(1).find('a').text().trim();
const injury = $(element).children('td').eq(1).children('span').eq(1).children('.icon-moon-injury').text().trim();
const news = $(element).children('td').eq(1).children('span').eq(1).children('.icon-moon-news').text().trim();
const playerUrl = $(element).children('td').eq(1).children('span').eq(1).find('a').attr('href');
const playerLogsUrl = "https://www.cbssports.com" + playerUrl.replace('playerpage', 'player/gamelogs/2018');
console.log(playerName + ": Inj: " + injury + " News: " + news);
// database.collection('NBAPlayers').add({[playerName]:{
// '01 playerName': playerName,
// '03 playerLogsUrl': playerLogsUrl,
// '04 inj': injury,
// '05 news': news
// }})
// .then(docRef => {
// console.log("ID " + docRef.id);
// //getPlayerLogs(playerLogsUrl, playerName, docRef.id);
// })
// .catch(error => console.error("Error adding document: ", error));
});
});
}
async/await is supported in the Node version 8 (which is deployable as Cloud Functions for Firebase). Specifically, use 8.6.1 (at the time of this writing).
As for awaiting x2 inside a loop - I don't think this is best practice.
Instead, push all these requests into an array, then Promise.all so as to fetch all in parallel.
just in case someone comes after. As Ron stated before, you shouldn't use async/await inside a traditional for loop if the second call doesn't need the value of the first, as you will increase the run time of the code.
Furthermore you can't use async/await inside of a forEach=> or a map=> loop, as it won't stop and wait for the promise to resolve.
The most efficient way is to use Promise.all([]). Here I leave a great youtube video of a crack that explains async/await and Promises => https://www.youtube.com/watch?v=vn3tm0quoqE&t=6s
As to one of the questions in the comments by DarkHorse:
But all I want is for getTeamPlayers() to execute and write to the database so I don't know why I need to return anything.
In Firebase Functions all functions need to return something before the final response.
For example in this case, he has created an http function. Before he ends the function with response.send("retrieved schedule"); you need to finish each and every function triggered. As Firebase Functions will clean up after the final response, erasing and stoping anything that it is still running. So any function that hasn't finish will be killed before doing its job.
Returning a promise is the way Firebase Functions knows when all executions have finished and can clean up.
Hop it helps :)
I'm just getting started with hapi.js (^17.3.1) and mongodb (^3.0.7), and with asynchronous js code.
Inside a route handler, I'm trying to retrieve data from the database. As a test, I'm storing a string inside a variable "s" built by looping through database collection records. The expected output to the browser is
start dbInfo1 dbInfo2 dbInfoN end
I've tried various versions of this code:
module.exports = {
method: 'GET',
handler: async function (request, reply) {
return await getRoutes();
}
}
async function getRoutes() {
var s = "start";
const mongo = require('mongodb').MongoClient;
const mongoUrl = "mongodb://127.0.0.1:27017/";
return // I'm returning this whole thing because hapi.js says it wants a promise. (500 error)
await mongo.connect(mongoUrl)
.then(function(client) {
client.db("dbName").collection("collectionName")
.find({})
.forEach(function (record) {
console.log(record.item);
s += " | " + record.item;
});
s + " end"; // But I've tried placing "return" here (500 error)
});
// I've also tried ".then(function(s) { return s + 'end' }) here but it seems to only have the same set of options/problems manifest.
// I've also made it so that I place "return s + 'end'" here (displays "start end" with nothing in the middle).
}
I've tried placing the return statement in different places. I either get an http 500 error in the console
Debug: internal, implementation, error
Error: handler method did not return a value, a promise, or throw an error
dbInfo1
dbInfo2
dbInfoN
if I return the promise itself or from inside the promise, or I get
start end
in the browser if I return from outside the promise.
In either case, the console.log statement prints out the dbInfos output.
I've tried different placements, inclusions, and omissions of async and await with pretty much the same results. I've also tried wrapping what is being returned inside getRoutes into an explicit Promise using "new Promise(...". In this case, the console logs the dbInfos, but the browser hangs.
How do I await that "foreach" function before returning the variable s?
Without test, I can say the this is wrong:
return // I'm returning this whole thing because hapi.js says it wants a promise. (500 error)
await mongo.connect(mongoUrl)
.then(function(client) {
client.db("dbName").collection("collectionName")
.find({})
.forEach(function (record) {
console.log(record.item);
s += " | " + record.item;
});
s + " end"; // But I've tried placing "return" here (500 error)
});
return is parsed as return;
return await mongo.connect(mongoUrl)
.then(function(client) {
client.db("dbName").collection("collectionName")
.find({})
.forEach(function (record) {
console.log(record.item);
s += " | " + record.item;
});
s + " end"; // But I've tried placing "return" here (500 error)
});
is the correct way. Any linter would have warned you about it.
Finally! Got it working with this code:
module.exports = {
method: 'GET',
handler: function (request, reply) {
return getRoutes();
}
}
function getRoutes() {
const mongo = require('mongodb').MongoClient;
const mongoUrl = "mongodb://127.0.0.1:27017/";
return mongo.connect(mongoUrl)
.then(async function(client) {
var s = "start";
var documents = await
client.db("dbName").collection("collectionName")
.find()
.toArray();
for (const doc of documents)
s += " | " + await doc.item;
return s + " end";
});
}
The issue was that I thought that since "getRoutes" was marked as "async", the stuff inside ".then" was async as well. But I really needed to mark "function(client)" as "async". I also needed to stop using "forEach" and use a more traditional iteration over the collection.
I had actually marked "function(client)" as "async" before, but it was out of blind trial and error, and so I never used "await" properly. I didn't really start to understand it until I read this blog by Anton Lavrenov.
Though I only asked the question recently, I was working on it before that for a long time. Really happy with where I'm at now. And of course thank you #desoares for pointing out my silly error in the version of code I was working with above.
I have found that write() method of stream.Writable class does not write data sequentially. When I an sending am attachment to the server in chunks, this code assembles data chunks in wrong order if no delay occurs. If I put a debug message like console.log() in the middle of the loop (like to dump the data to watch what is being written, actually), this bug disappears. So, what is the race condition in this code ? Looks like I am enforcing a sequential assembling of the file, so I do not understand what is wrong.
My code:
function join_chunks(company_id,attachment_id,num_chunks) {
var stream;
var file;
var output_filename=ATTACHMENTS_PATH + '/comp' + company_id + '/' + attachment_id + '.data';
var input_filename;
var chunk_data;
var chunk_count=0;
stream=fs.createWriteStream(output_filename,{flags:'w+',mode: 0666});
console.log('joining files:');
for(var i=0;i<num_chunks;i++) {
input_filename=ATTACHMENTS_PATH + '/comp' + company_id + '/' + attachment_id + '-' + (i+1) + '.chunk';
console.log(input_filename);
fs.readFile(input_filename , (err, chunk_data) => {
if (err) throw err;
stream.write(chunk_data,function() {
chunk_count++;
if (chunk_count==num_chunks) {
console.log('join finished. closing stream');
stream.end();
}
});
});
}
}
The console:
joining files:
/home/attachments/comp-2084830518/67-1.chunk
/home/attachments/comp-2084830518/67-2.chunk
/home/attachments/comp-2084830518/67-3.chunk
/home/attachments/comp-2084830518/67-4.chunk
join finished. closing stream
Node version: v6.9.2
stream.write is an asynchronous operation. This means that multiple calls to it may be serviced out of order.
If you want your writes to happen in order, use stream.writeSync, or use the callback argument to stream.write to sequence your writes.
currently trying to develop one of my first alexa skills. trying to build a skill that updates the user on who out of their Steam friends, are online.
Curerntly it's pretty messy and not in the format i hope it to be in the end, but for testing purposes, I'm not yet using intents, im just testing using the launchrequest.
So far I've managed to get a users (mine) friends list from steam and put all the friends ids in a url that should be able to grab the details for these
users.
The issue I'm having is performing the second API call to grab the players details using the steamIds. i keep getting an 'undefined' return and am stumped on what i'm doing wrong.
I'm very much new to JS so there's bound to be mistakes in here, but i can work on tidying up later once i've got it working.
This works fine
/**
* Called when the user invokes the skill without specifying what they want.
*/
function onLaunch(launchRequest, session, callback) {
console.log("onLaunch requestId=" + launchRequest.requestId
+ ", sessionId=" + session.sessionId);
var cardTitle = "Hello, World!"
testGet(function (response) {
var speechOutput = "Here is the result of your query: " + response;
var shouldEndSession = true;
callback(session.attributes,
buildSpeechletResponse(cardTitle, speechOutput, "", true));
});
//var speechOutput = "You can tell Hello, World! to say Hello, World!"
//callback(session.attributes,
// buildSpeechletResponse(cardTitle, speechOutput, "", true));
}
This is the function that is grabbing the details from my friendlist
function testGet(response) {
var http = require('http')
var url = " http://api.steampowered.com/ISteamUser/GetFriendList/v0001/?key=XXXXXXXXXX&steamid=76561198068311091&relationship=friend"
http.get(url, function (res) {
// data is streamed in chunks from the server
// so we have to handle the "data" event
var buffer = "",
data,
friendsList,
i,
address,
textResponse,
route;
res.on("data", function (chunk) {
buffer += chunk;
});
res.on("end", function (err) {
// finished transferring data
// dump the raw data
console.log(buffer);
console.log("\n");
data = JSON.parse(buffer);
friendsList = data.friendslist.friends;
textResponse = isOnline(friendsList);
response("Friends online: " + textResponse);
}).on('error', function (e) {
console.log("Error message: " + e.message);
});
})
}
and this is the final function which i'm having difficulties with.
function isOnline(friendsList){
var http = require('http'),
i,
comma,
friendsIDs = "";
// for loop to get all friends ids in string
for (i = 0; i < friendsList.length; i++) {
// if i equals 0 then it is the start of the loop so no
//comma needed, otherwise add a comma to seperate the ids.
if(i === 0) {comma = ""}
else{comma = ","}
//place the ids in a comma seperate string
friendsIDs += comma + friendsList[i].steamid;
}
var playerurl = "http://api.steampowered.com/ISteamUser/GetPlayerSummaries/v0002/?key=xxxxx&steamids=" + friendsIDs;
// works fine up to this point
// run the api call to get player details
http.get(playerurl, function (response) {
// data is streamed in chunks from the server
// so we have to handle the "data" event
var buffer = "",
playerdata,
returnText,
textResponse,
friendsInformation,
route;
response.on("playerdata", function (chunk) {
buffer += chunk;
});
response.on("end", function (err) {
// finished transferring data
// dump the raw data
console.log(buffer);
console.log("\n");
playerdata = JSON.parse(buffer);
friendsInformation = playerdata.response.players;
for (i = 0; i < friendsInformation.length; i++) {
if(friendsInformation[i].personastate == 1) {
returnText += friendsInformation[i].personaname + " chicken";
}
}
return returnText;
}).on('error', function (e) {
console.log("Error message: " + e.message);
});
});
}
Been going round in circles for hours and feel so close to doing this but have no idea where I'm going wrong?!
thanks
I have managed to solve my problem by using javascript promises. I'm completely new to promises so took some trial and error but have managed to get it to work. here is the simplest video i could find to explain the concept, it definately helped me understand how to rearrange my code.
If you wish to do two API calls in an Alexa skill, using the data from the first api call to construct and inform the second call, you will need to use promises to do them sequentially.
it took the code from the isOnline() and testGet() functions and moved them into a promise, which allowed me to complete 1 api call (the original call to get friends list info) before executing the second call (the second api to get player details based on the results of the friendslist api call)
I can now check with alexa to see which of my steam friends are online! Hopefully i'll be able to build in some more functionality (eg what they are playing, if they are offline or just away/busy/snoozing)
thanks for contributing
I got a pretty awkward problem.
I create a pool, connect to the database, create a connection and query, get the results, do a bunch of stuff, then I have to create another connection and query, but actually it has to be dynamically so I loop over my Array teacherHours containing the data.
Then more Code is happening, and I have to create an extra loop, because certain elements of my teacherHours Array have to try multiple times to get the correct response from the upcoming query.
So another loop follows, which is supposed to loop as long as availableHours > 0. Now, here is where it all goes left.
A buch of code happens inside the second loop, I prepare my second query, call connection.query() and inside of the callback function I prepare my third query (after doing some other stuff) and this is actually where Node kicks me out.
It gives me TypeError: Cannot read property 'tid' of undefined. tid needs to be accessed for my third query, so I try to access it just like I did before but Node doesn't allow it.
I know that the queries return useful data (rows) so it can't be a problem of querying but receiving no data. Actually I console.log("the rowRIDS"+rowRIDS); the result of the second query and I see that it returns 2 rows and just after that it gives me the error.
What is also strange to me, all the console.logs inside my my two loops are being logged, and my console.log of the second query (containing the 2 rows) are being logged after the loops ran through, since the queries are nested shouldn't the returned 2 rows and the error appear within the first iteration of the loop, since the code should access the second query at that point.
BTW, I've tried to set a hardcoded number instead of the tid just to get the next property datum to be an error. I kind of got a feeling as if the variable teacherHours is out of scope, but it is supposed to be a global variable.
To get a better feeling of what I'm talking about I copied the code and uncommented all the javascript code, where I populate and calculate stuff. Any help would be really great, its been almost 7 hours of try & error without any luck. Thank You!
pool.getConnection(function(err, connection){
if (err) throw err;
connection.query('SELECT * FROM teachers_teaching_tbl WHERE fwdid = 1 ', function(err, rows, fields) {
if (err) {
console.error('error querying: ' + err.stack);
return;
}
rowArray=rows;
console.log(rowArray);
//
// HERE HAPPENS
// A LOOOOT OF STUFF
//
// AND teacherHours IS BEING POPULATED
//
// THEN COMES A FOR LOOP
for(var i=0; i<teacherHours.length;i++){
//
// MORE STUFF
//
//AND ANOTHER LOOP
while(availableHours>0){//do{ ORIGINALLY I TRIED WITH A DO WHILE LOOP
//
// AGAIN A BUNCH OF STUFF
//
// NOW I'M PREPARING MY NEXT QUERY
//
var myQueryGetFreeRoom=" SELECT rms.rid FROM rooms_tbl as rms WHERE NOT EXISTS ( ";
myQueryGetFreeRoom+=" SELECT NULL FROM classes_tbl as cls ";
myQueryGetFreeRoom+=" WHERE ( (cls.bis > '"+bisMinus1+"' AND cls.bis <= '"+realBis+"' ) OR ( cls.von > '"+bisMinus1+"' AND cls.von < '"+realBis+"' ) ) AND (cls.datum = '"+teacherHours[i].datum.getFullYear()+"-"+(teacherHours[i].datum.getMonth()+1)+"-"+teacherHours[i].datum.getDate()+"') AND (cls.rid=rms.rid) ) ";
//
//
connection.query(myQueryGetFreeRoom, function(err, rowRIDS, fields) {
if (err) {
console.error('error querying: ' + err.stack);
return;
}
roomIDs=rowRIDS;
console.log("the rowRIDS"+rowRIDS);
//
// MORE STUFF
// HAPPENING
//
if(roomIDs.length>0){
//
// PREPARING QUERY NO.3 - WHICH IS WHERE MY ERROR POINTS - TO THE USE OF tid PROPERTY
//
var myQueryBookClass = " INSERT INTO classes_tbl ( rid , tid , belegtAnz, datum, von , bis ) ";
myQueryBookClass+=" VALUES ( "+Math.floor(Math.random() * roomIDs.length)+", "+teacherHours[i].tid+" , 0, '"+teacherHours[i].datum.getFullYear()+"-"+(teacherHours[i].datum.getMonth()+1)+"-"+teacherHours[i].datum.getDate()+"' , '"+bisMinus1+"' , '"+realBis+"' ) ";
console.log("myQueryBookClass: "+myQueryBookClass);
availableHours = 0;
//
// HERE WAS SUPPOSED TO FOLLOW QUERY 3 - myQueryBookClass
//
// BUT SINCE I DONT EVEN GET INSIDE HERE IT IS IN COMMENTS
//
/*connection.query(myQueryBookClass, function(err, insertRows, fields){
if(err){
console.error('error querying: '+err.stack);
return;
}
console.log("Inserted Rows: "+ insertRows);
}); */
} else {
availableHours= availableHours - 1;
//
// STUFF HAPPENING
//
}
});
availableHours= availableHours - 1;
}//while(availableHours>0);
//
}
connection.release(function(err){
if (err){
console.error('error disconnecting: ' + err.stack);
return;
}
});
});
});
I think you are coming from a non-async language like Python, Java, etc. which is why Node, i.e. JavaScript, seems to screw things up for you, but actually it isn't.
The problem you have in your code is that you execute async functions like query synchronously all at the same time in the same while loop. You need to use a module like async which helps to run and collect results asynchronously.
Here is the updated code.
var async = require('async'),
connection;
async.waterfall([
function (cb) {
pool.getConnection(cb);
},
function (conn, cb) {
connection = conn;
connection.query('SELECT * FROM teachers_teaching_tbl WHERE fwdid = 1', cb);
},
function (rows, fields, cb) {
rowArray = rows;
console.log(rowArray);
// HERE HAPPENS
// A LOOOOT OF STUFF
//
// AND teacherHours IS BEING POPULATED
//
// THEN COMES A FOR LOOP
async.eachSeries(teacherHours, function (teacherHour, done) {
// MORE STUFF
//
//AND ANOTHER LOOP
async.whilst(function () {
return availableHours > 0;
}, function (cb) {
// AGAIN A BUNCH OF STUFF
//
// NOW I'M PREPARING MY NEXT QUERY
//
var myQueryGetFreeRoom =
"SELECT rms.rid FROM rooms_tbl AS rms WHERE NOT EXISTS ("
+ "SELECT NULL FROM classes_tbl AS cls"
+ " WHERE ("
+ "(cls.bis > '" + bisMinus1 + "' AND cls.bis <= '" + realBis + "')"
+ " OR (cls.von > '" + bisMinus1 + "' AND cls.von < '" + realBis + "')"
+ ") AND ("
+ "cls.datum = '" + teacherHour.datum.getFullYear() + "-" + (teacherHour.datum.getMonth() + 1) + "-" + teacherHour.datum.getDate() + "'"
+ ") AND cls.rid = rms.rid";
async.waterfall([
function (cb) {
connection.query(myQueryGetFreeRoom, cb);
},
function(rowRIDS, fields, cb) {
roomIDs = rowRIDS;
console.log("the rowRIDS" + rowRIDS);
//
// MORE STUFF
// HAPPENING
//
if (roomIDs.length > 0) {
//
// PREPARING QUERY NO.3 - WHICH IS WHERE MY ERROR POINTS - TO THE USE OF tid PROPERTY
//
var myQueryBookClass = "INSERT INTO classes_tbl (rid, tid, belegtAnz, datum, von, bis) VALUES ("
+ Math.floor(Math.random() * roomIDs.length)
+ ", " + teacherHour.tid
+ ", 0, '" + teacherHour.datum.getFullYear() + "-" + (teacherHour.datum.getMonth() + 1) + "-" + teacherHour.datum.getDate() + "', '" + bisMinus1 + "', '" + realBis + "')";
console.log("myQueryBookClass: " + myQueryBookClass);
availableHours = 0;
//
// HERE WAS SUPPOSED TO FOLLOW QUERY 3 - myQueryBookClass
//
// BUT SINCE I DONT EVEN GET INSIDE HERE IT IS IN COMMENTS
//
connection.query(myQueryBookClass, function (err, insertRows, fields) {
if (err) {
console.error('error querying: '+err.stack);
return;
}
console.log("Inserted Rows: "+ insertRows);
// Here do whatever you need to do, then call the callback;
cb();
});
} else {
--availableHours;
//
// STUFF HAPPENING
//
cb();
}
}
], function (err) {
if (!err) {
// Notice that you are decrementing the `availableHours` twice here and above.
// Make sure this is what you want.
--availableHours;
}
cb(err);
});
}, done);
}, function (err) {
connection.release(function (err) {
if (err) {
console.error('error disconnecting: ' + err.stack);
return;
}
});
});
}
], function (err) {
conn && pool.release(conn);
err && throw err;
});
Next time please format your code properly for better readability which will help yourself to get answers faster, and split your question text into paragraphs for the same purpose.
Explanation
There are four nested async flows:
async.waterfall
-> async.eachSeries
-> async.whilst
-> async.waterfall
Basically, the async.waterfall library allows you to execute a list of functions in series.
Every next function will be executed only after the previous function has returned the response.
To indicate that a function is completed and the results are available, it has to call the callback, in our case it is cb (you can call it whatever you like, eg. callback). The rule is to call it, otherwise, the next function will never be executed because the previous one doesn't seem to have finished its work.
Once the previous function has completed, it calls the provided cb with the following signature:
cb(err, connection);
If there was an error while requesting for a connection, the entire async.waterfall will interrupt and the final callback function will be executed.
If there was no error, the connection will be provided as a second argument. async module passes all arguments of the previous function to the next function as the first, second, etc. arguments which is why the second function receives the conn as the first argument.
Every next function will receive the callback cb as the last argument, which you must eventually call when the job is done.
Thus, in the first async.waterfall flow:
It requests a new database connection.
Once the connection is available, the next function is executed which sends a query to the database.
Waits for the query results, then once the results are available, it is ready to run the next function which iterates over each row.
async.eachSeries allows to iterate over a gives array of values sequentially.
In the second async.eachSeries flow:
It iterates over each element in the teacherHours array sequentially.
Once each element is processed (however you want), you must call the done callback. Again, you could have called this as cb like in the previous async.waterfall or callback. done is just for clarity that the process is done.
Then we have the async.whilst which provides the same logic as the normal while () {} statement but handles the loop asynchronously.
In this third async.whilst flow:
Calls the first function. Its return value indicates whether it has to continue the loop, i.e. call the second asynchronous function.
If the return value is truthful (availableHours > 0), then the second function is called.
When the async function is done, it must call the provided callback cb to indicate that it is over. Then async module will call the first function to check if it has to continue the loop.
In this asynchronous function inside async.whilst we have another async.waterfall because you need to send queries to the database for each teacherHour.
In this last fourth async.watercall flow:
It sends the SELECT query to the database.
Waits for the response. Once the rowRIDS are available, it calls the second function in the waterfall.
If there are rowRIDS (roomIDs.length > 0), it sends the INSERT query to the database.
Once done, it calls the callback cb.
If there were no rowRIDs, it calls the callback cb, too, to indicate that the job is done.
It is a great thing that JavaScript is asynchronous. It might be difficult at the beginning when you convert from other synchronous languages, but once you get the idea, it will be hard to thing synchronously. It becomes so intuitive that you will start thinking why other languages don't work asynchronous.
I hope I could explain the above code thoroughly. Enjoy JavaScript! It's awesome!