Develop Reference

This code doesn't seem to fire in order?

My problem is that the code does not seem to be running in order, as seen below.
This code is for my discord.js bot that I am creating.
var Discord = require("discord.js");
var bot = new Discord.Client();
var yt = require("C:/Users/username/Documents/Coding/Discord/youtubetest.js");
var youtubetest = new yt();
var fs = require('fs');
var youtubedl = require('youtube-dl');
var prefix = "!";
var vidid;
var commands = {
play: {
name: "!play ",
fnc: "Gets a Youtube video matching given tags.",
process: function(msg, query) {
youtubetest.respond(query, msg);
var vidid = youtubetest.vidid;
console.log(typeof(vidid) + " + " + vidid);
console.log("3");
}
}
};
bot.on('ready', () => {
console.log('I am ready!');
});
bot.on("message", msg => {
if(!msg.content.startsWith(prefix) || msg.author.bot || (msg.author.id === bot.user.id)) return;
var cmdraw = msg.content.split(" ")[0].substring(1).toLowerCase();
var query = msg.content.split("!")[1];
var cmd = commands[cmdraw];
if (cmd) {
var res = cmd.process(msg, query, bot);
if (res) {
msg.channel.sendMessage(res);
}
} else {
let msgs = [];
msgs.push(msg.content + " is not a valid command.");
msgs.push(" ");
msgs.push("Available commands:");
msgs.push(" ");
msg.channel.sendMessage(msgs);
msg.channel.sendMessage(commands.help.process(msg));
}
});
bot.on('error', e => { console.error(e); });
bot.login("mytoken");
The youtubetest.js file:
var youtube_node = require('youtube-node');
var ConfigFile = require("C:/Users/username/Documents/Coding/Discord/json_config.json");
var mybot = require("C:/Users/username/Documents/Coding/Discord/mybot.js");
function myyt () {
this.youtube = new youtube_node();
this.youtube.setKey(ConfigFile.youtube_api_key);
this.vidid = "";
}
myyt.prototype.respond = function(query, msg) {
this.youtube.search(query, 1, function(error, result) {
if (error) {
msg.channel.sendMessage("There was an error finding requested video.");
} else {
vidid = 'http://www.youtube.com/watch?v=' + result.items[0].id.videoId;
myyt.vidid = vidid;
console.log("1");
}
});
console.log("2");
};
module.exports = myyt;
As the code shows, i have an object for the commands that the bot will be able to process, and I have a function to run said commands when a message is received.
Throughout the code you can see that I have put three console.logs with 1, 2 and 3 showing in which order I expect the parts of the code to run. When the code is run and a query is found the output is this:
I am ready!
string +
2
3
1
This shows that the code is running in the wrong order that I expect it to.
All help is very highly appreciated :)
*Update! Thank you all very much to understand why it isn't working. I found a solution where in the main file at vidid = youtubetest.respond(query, msg) when it does that the variable is not assigned until the function is done so it goes onto the rest of my code without the variable. To fix I simply put an if statement checking if the variable if undefined and waiting until it is defined.*

Like is mentioned before, a lot of stuff in javascript runs in async, hence the callback handlers. The reason it runs in async, is to avoid the rest of your code being "blocked" by remote calls. To avoid ending up in callback hell, most of us Javascript developers are moving more and more over to Promises. So your code could then look more like this:
myyt.prototype.respond = function(query, msg) {
return new Promise(function(resolve, reject) {
this.youtube.search(query, 1, function(error, result) {
if (error) {
reject("There was an error finding requested video."); // passed down to the ".catch" statement below
} else {
vidid = 'http://www.youtube.com/watch?v=' + result.items[0].id.videoId;
myyt.vidid = vidid;
console.log("1");
resolve(2); // Resolve marks the promises as successfully completed, and passes along to the ".then" method
}
});
}).then(function(two) {
// video is now the same as myyt.vidid as above.
console.log(two);
}).catch(function(err) {
// err contains the error object from above
msg.channel.sendMessage(err);
})
};
This would naturally require a change in anything that uses this process, but creating your own prototypes seems.. odd.
This promise returns the vidid, so you'd then set vidid = youtubetest.response(query, msg);, and whenever that function gets called, you do:
vidid.then(function(id) {
// id is now the vidid.
});
Javascript runs async by design, and trying to hack your way around that leads you to dark places fast. As far as I can tell, you're also targetting nodeJS, which means that once you start running something synchronously, you'll kill off performance for other users, as everyone has to wait for that sync call to finish.
Some suggested reading:
http://callbackhell.com/
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
https://stackoverflow.com/a/11233849/3646975
I'd also suggest looking up ES6 syntax, as it shortens your code and makes life a hellofalot easier (native promises were only introduced in ES6, which NodeJS 4 and above supports (more or less))

In javascript, please remember that any callback function you pass to some other function is called asynchronously. I.e. the calls to callback function may not happen "in order". "In order" in this case means the order they appear on the source file.
The callback function is simply called on certain event:
When there is data to be processed
on error
in your case for example when the youtube search results are ready,
'ready' event is received or 'message' is received.
etc.

Javascript function doesn't return query result

I am trying to figure out why one of my queries won't return the value from a query...my code looks like this:
var client = new pg.Client(conString);
client.connect();
var query = client.query("SELECT count(*) as count FROM sat_scores")
// Don't use demo key in production. Get a key from https://api.nasa.gov/index.html#apply-for-an-api-key
function getNEO(callback) {
var data = '';
query.on('rows', function(rows) {
console.log("Row count is: %s", rows[0].count)
data += rows[0].count;
});
query.on('end', function() {
callback(data);
});
}
with that, getNEO returns a blank...but if I set var data = '4', then getNEO returns 4....the query should return 128 but it just returns a blank...

First of all, getNEO() doesn't return anything - I'm operating on the assumption that you call getNEO() exactly once for your query, and pass in a callback to handle the data, and that callback is what's not getting the appropriate data?
My typical recommendation for troubleshooting things like this is to simplify your code, and try and get really close to any example code given (for instance):
var client = new pg.Client(conString);
// define your callback here, in theory
client.connect(function (err) {
if (err) throw err;
var query = client.query("SELECT count(*) as count FROM sat_scores"),
function(err, result) {
if (err) throw err;
console.log(result.rows.length);
}
);
});
... I'm doing a couple things here you'll want to note:
It looks like the client.connect() method is asynchronous - you can't just connect and then go run your query, you have to wait until the connection is completed, hence the callback. Looking through the code, it looks like it may emit a connect event when it's ready to send queries, so you don't have to use a callback on the connect() method directly.
I don't see a data event in the documentation for the query object nor do I see one in the code. You could use the row event, or you could use a callback directly on the query as in the example on the main page - that's what I've done here in the interest of simplicity.
I don't see the count property you're using, and row[0] is only going to be the first result - I think you want the length property on the whole rows array if you're looking for the number of rows returned.
I don't know if you have a good reason to use the getNEO() function as opposed to putting the code directly in procedurally, but I think you can get a closer approximation of what you're after like this:
var client = new pg.Client(conString);
// define your callback here, in theory
client.connect();
function getNEO(callback) {
client.on('connect', function () {
var query = client.query("SELECT count(*) as count FROM sat_scores"));
query.on('end', function(result) {
callback(result.rowCount);
});
});
}
... so, you can call your getNEO() function whenever you like, it'll appropriately wait for the connection to be completed, and then you can skip tracking each row as it comes; the end event receives the result object which will give you all the rows and the row count to do with what you wish.

so here is how I was able to resolve the issue....I moved the var query inside of the function
function getNEO(state, callback) {
var conString = "postgres://alexa:al#alexadb2.cgh3p2.us-east-1.redshift.amazonaws.com:5439/alexa";
var client = new pg.Client(conString);
client.connect();
var data = '';
var query = client.query("SELECT avg(Math) as math, avg(Reading) as reading FROM sat_scores WHERE State = '" + state + "'");
console.log("query is: %s", query);
query.on('row', function(row) {
console.log("Row cnt is: %s", row.math);
console.log("row is: " + row)
data += row;
});
console.log("made it");
query.on('end', function() {
callback(data);
});
}

Array returning Undefined because of asynchrony

I am accessing the API Trello, but I came across the following problem:
Trello access the information, getting the id of each existing row, the code is as follows:
var x;
var numberCardsByList = [];
trello.get("/1/boards/[idBoard]/lists/all", function(err, data) {
if (err) throw err;
console.log("Number of list: " + data.length);
for(var i=0; i<data.length; i++){
x = data[i];
findNumberCards(x);
}
});
As you can see, after getting the size, I walk all these queues with is, within the loop, attach each row in a variable x and call a function that aims to get the number of cards that queue. The code for the number of cards is as follows:
function findNumberCards(x){
trello.get("/1/lists/"+x.id+"/cards", function(err, dados){
if(err) throw err;
console.log("Name List: " + x.name + " have " + dados.length + " cards");
numberCardsByList[x.name] = dados.length;
});
}
Until then all right, but when I try to access the vector numberCardsByList after the end of the search in Trello, it returns undefined:
var x;
var numberCardsByList = [];
trello.get("/1/boards/[idBoard]/lists/all", function(err, data) {
if (err) throw err;
console.log("Quantidade de Filas: " + data.length);
for(var i=0; i<data.length; i++){
x = data[i];
findNumberCards(x);
}
});
console.log(numberCardsByList);
I am aware that it is because of asynchrony, however, can not solve.

The problem you're facing has been solved many times before. If you want to know more, search for the keyword "Promise". If you're familiar with jQuery, try and look up: $.whenAll, $.ajax().done, $.ajax().always, etc.
If you want to come up with a light weight solution yourself, here's a pointer:
By the time you get to your console.log(numberCardsByList), your requests triggered by findNumberCards haven't yet completed, making the Array empty. You'll need to make sure you know when all findNumberCards requests have completed and then log them. Alternatively, you could log them every time one of them completes.
There are roughly two approaches:
Keep track of your open requests and call a function when a request is handled.
Observe your numberCardsByList object and call a function whenever items are added (you won't know if they were added async or synchronously)
I'd suggest going with the first approach. Check out this example code and the comments:
var numberCardsByList = {};
// This array will store the url for every open request
var openRequests = [];
var removeRequest = function(url) {
var index = openRequests.indexOf(url);
if (index === -1) return;
// Remove url from array
openRequests = openRequests
.slice(0, index)
.concat(openRequests
.slice(index + 1));
};
// This will be called whenever one request completes
var onComplete = function(url) {
removeRequest(url);
// When all have completed, we can call our callback
if (openRequests.length === 0) {
onAllComplete();
}
});
// This will be called when there are no open requests left
var onAllComplete = function(data) {
console.log(numberCardsByList);
}
trello.get("/1/boards/[idBoard]/lists/all", function(err, data) {
if (err) throw err;
console.log("Number of list: " + data.length);
for (var i = 0; i < data.length; i++) {
x = data[i];
findNumberCards(x);
}
});
function findNumberCards(x) {
var url = "/1/lists/" + x.id + "/cards";
// Before we make the request, we register it:
openRequests.push(url);
trello.get(url, function(err, dados) {
numberCardsByList[x.name] = dados.length;
// When it is completed, we call onComplete
onComplete(url);
});
};
Note that this onAllComplete isn't 100% safe: it might be called multiple times if a request finishes before the next one is started.
Concluding:
If you can, I'd use a library to handle promises. If you want to try and build something yourself, you could try and keep track of the requests and execute a callback when they've all completed.

Keep in mind my above code most likely wont work for you as i dont know whats going on in your code so this is an example / explanation how to deal with your problem.
Since you are unfamiliar with async operation i will assume you dont have a prior knowledge of promises and therefore give you a less optimal solution - however promises are alot better and you should defintely learn them.
You need to execute sequence procedures inside the result of the async code.
First you'll create a function for the second operation for example:
function numberCardsByList (param1,param2){.....}
You will then change fineNumberCards to also accept a callback:
function findNumberCards(x, callback){
trello.get("/1/lists/"+x.id+"/cards", function(err, dados){
if(err) throw err;
console.log("Name List: " + x.name + " have " + dados.length + " cards");
numberCardsByList[x.name] = dados.length;
});
// pass in any params you need.
callback();
}
And then you will pass the newly created function numberCardsByList to findNumberCards or wherever you want it.
trello.get("/1/boards/[idBoard]/lists/all", function(err, data) {
if (err) throw err;
console.log("Number of list: " + data.length);
for(var i=0; i<data.length; i++){
x = data[i];
// and here we are calling findNumberCards and passing in the callback..
findNumberCards(x, numberCardsByList);
}
});
That is generally how you will deal with async operation, you will pass a callback for the next operation to be executed.
update
here is an example of how this is done with another scenario just to demonstrate the point farther.
we start by getting user
service.getUser(userName, function(err,user){
if(user) {
// we get user picture passing getPictureSize as callback
getUserPicture(user.picture, getPictureSize)
}
})
we get the pictureURL
function getUserPicture(picName, cb){
service.getPictureURL(picName, function(err, pictureURL){
if(pictureURL) {
// we then call the callback - the next async operation we want.
cb(pictureURL);
}
});
}
we get picture size - which is the last operation
function getPictureSize(pictureURL){
service.getPictureSize(pictureURL, function(err, pictureSize){
$('.picName').attr('src', picName);
$('.picName').width(pictureSize.width);
$('.picName').height(pictureSize.height);
});
}
I hope that clarify things a little.

Express app.post getting called multiple times

I have been building a youtube video conversion app which streams youtube videos using youtube-dl and saves them, everything was working fine until I attempted to stream a video that was over an hour long. When the task was anywhere between 50% - 100% complete or 40-80seconds in, would be when the entire block of code would get re-run resulting in multiple simultaneous streams occurring. The response can therefor never get sent as it waits for the pipe to finish. Adding next(); outside the stream function allowed the conversion to complete with out any interruption or reruns of the code block, however it resulted in the following error when attempting to send the response:
throw new Error('Can\'t set headers after they are sent.');
This is the Node.js code block in question:
app.post('/convert/', function (req, res, next){
var url = 'http://www.youtube.com/'+req.body.url;
var quality = req.body.quality;
var socketId = req.body.socketId;
stream = youtubedl(url,
['-f ' + quality],
// // Additional options can be given for calling `child_process.execFile()`.
{ cwd: __dirname });
stream.on('info', function(info) {
console.log('Download started');
console.log('filename: ' + info._filename);
console.log('size: ' + info.size);
console.log('format: ' + info.format);
var fileName = info._filename;
var videoId = info.id;
var videoTitle = info.title;
videoTitle = videoTitle.replace(/[^a-zA-Z0-9\s]/g, '');
console.log(videoTitle);
var mp4 = 'downloads/'+videoTitle+'-'+info.format_id+'.mp4';
fs.writeFile(mp4, "file", function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
var stat = fs.statSync(mp4);
var str = progress({
length: info.size,
time: 100
});
str.on('progress', function(progress) {
io.to(global.socketid).emit('progressVideo',{progress: progress.percentage});
console.log(info.size);
console.log(progress.transferred);
console.log(progress.percentage);
console.log(progress.remaining);
});
var pipe = stream.pipe(str).pipe(fs.createWriteStream(mp4));
pipe.on('finish', function () {
console.log("stream finished.");
res.json(videoTitle+'-'+info.format_id+'.mp4');
});
});
});
// next();
});
Called by some Angular code.
// Sends youtube link to backend
$scope.getVideo = function(youtubeLink, resolution){
var cleanedLink = youtubeLink.substring(24);
var url = {url: cleanedLink, quality: resolution};
$http.post('/convert/', url).success(function (response){
// Do some stuff
});
}
Confused as to why it was getting run more then once, I slowly removed more and more code until I was left with this simple test.
app.post('/convert/', function (req, res, next){
console.log('hello!');
});
Which was called by an ng-click event and after waiting a minute or so the console also printed out two and then three hello! statements. I am completely lost as to why this happens. If anyone could shed some light on this for me it would be greatly appreciated.

So after just getting down to a very basic post route and logging out some text but not returning a response, I came to the conclusion the issue resides with node. I decided to record the length of time between the console.log statements which turned out to be every 2 minutes. With this I was able to find out that node has a default timeout of 2 minutes if a response is not sent back to the client.
I was able to set the response to never timeout with the following code:
res.connection.setTimeout(0);
I hope this helps anyone else that needs to hold connections for large periods of times for file conversions/transfers etc...

nodejs handling multiple subprocess together

I have got a situation where i need to create n number of subpocess, for each subprocess i need to provide stdin data and expected output, the result of the subpocess is success, if the expected output is same as that of output produced. If all such subprocess is success then the status need to be send to user. How to do the above in nodejs in a nonblocking way?

Promises!
I personally use Bluebird, and here is an example that uses it too.
I hope you understand it, feel free to ask when you do not :-)
var Promise = require('bluebird')
var exec = require('child_process').exec
// Array with input/output pairs
var data = [
['input1', 'output1'],
['input2', 'output2'],
...
]
var PROGRAM = 'cat'
Promise.some(data.map(function(v) {
var input = v[0]
var output = v[1]
new Promise(function(yell, cry) {
// Yes it is ugly, but exec is just saves many lines here
exec('echo "' + input + '" | ' + PROGRAM, function(err, stdout) {
if(err) return cry(err)
yell(stdout)
})
}).then(function(out) {
if(out !== output) throw new Error('Output did not match!')
})
}), data.length) // Require them all to succeed
.then(function() {
// Send succes to user
}).catch(function() {
// Send failure to the user
})