Develop Reference

Node.js with Express: Push to an empty Array returns an empty Array

I am listing all files from all directories in /home/myComputer/Desktop/Research, and then filtering them with an if statement to only get the .txt files that I would like to read and store into arrays. All works fine, but pushing the data into the arrays is not functioning. When I console log them, they return no value [].
I tried promise as well as call back function, but they didn't work for me because I didn't know how to implement them properly.
app.get('/jsonData', function(req, res) {
/* Define Arrays */
var theFile = [];
var theCategory = [];
var theContent = [];
var walk = function(dir, done) {
var results = [];
fs.readdir(dir, function(err, list) {
if (err) return done(err);
var i = 0;
(function next() {
var file = list[i++];
if (!file) return done(null, results);
file = dir + '/' + file;
fs.stat(file, function(err, stat) {
if (stat && stat.isDirectory()) {
walk(file, function(err, res) {
results = results.concat(res);
next();
});
} else {
results.push(file);
next();
}
});
})();
});
};
//walk(process.env.HOME, function(err, results) {
walk("/home/myComputer/Desktop/Research", function(err, results) {
if (err) throw err;
//console.log(results);
results.map(function(val) {
//Get the filename
var fileName = val.match(/[^\/]+$/).join();
//Get the category
var category = val.substr(48).match(/[^\/]+/);
if (fileName == 'written-speech.txt') {
console.log('FOUND!: ' + fileName + ' Category: ' + category) //this works
fs.readFile(val, 'utf8', function(err, contents) {
console.log(contents); // this works
theFile.push(fileName);
theCategory.push(category);
theContent.push(contents);
});
}
})
});
console.log(theFile); // The problem: This returns an empty Array []
console.log(theCategory); // The problem: This returns an empty Array []
console.log(theContent); // The problem: This returns an empty Array []
});
I expect console.log(theFile); console.log(theCategory); and console.log(theContent); to return the data pushed in them.

The reason for this is that many callbacks in Javascript are asynchronous, which means both fs.readdir and fs.readFile are asynchronous and their callbacks are not called immediately but slightly later (please read about Event Loop in javascript). So at the moment, when you log your arrays they are empty and data to them will be pushed later, e.g. in future. To avoid this you can either use synchronous methods (fs.readdirSync and fs.readFileSync) which is ugly and can cause performance issues if the app has a lot of other asynchronous operations. If in your case it is just a simple script to read some data, it might be fine.
And the other, preferred way is to use promises or some library for managing callbacks, e.g. async. Please read some articles regarding managing async code if these concepts are fully unfamiliar for you, e.g. https://dev.to/mrm8488/from-callbacks-to-fspromises-to-handle-the-file-system-in-nodejs-56p2 to get a basic understanding and see some use case examples.
Regarding your current version, there is no easy way to make it work without a lot of changes. It is better to rewrite it to use the concepts I described earlier.

walk is an asynchronous function because fs.readdir is an asynchronous method and the console.log statements are running (in a synchronous manner) before the callback of fs.readdir getting invoked.
You can console the values of these variables at the end inside the callback of walk.

NodeJS - Need help understanding and converting a synchronous dependant MySQL query code into something usable

this is my second Node project. I am using Node, Express, MySQL.
What I am doing is, I have an array of names of users that have posted something, I then loop over those names and for each of them I do a connection.query to get their posts and I store those into an array(after that I do some other data manipulation to it, but that's not the important part)
The problem is: my code tries to do that data manipulation before it even receives the data from the connection.query!
I google-d around and it seems async await is the thing I need, problem is, I couldn't fit it in my code properly.
// namesOfPeopleImFollowing is the array with the names
namesOfPeopleImFollowing.forEach(function(ele){
connection.query(`SELECT * FROM user${ele}posts`, function(error,resultOfThis){
if(error){
console.log("Error found" + error)
} else {
allPostsWithUsername.push([{username:ele},resultOfThis])
}
})
})
console.log(JSON.stringify(allPostsWithUsername)) // This is EMPTY, it mustn't be empty.
So, how do I convert that into something which will work properly?
(Incase you need the entire code, here it is: https://pastebin.com/dDEJbPfP though I forgot to uncomment the code)
Thank you for your time.

There are many ways to solve this. A simple one would be to wrap your function inside a promise and resolve when the callback is complete.
const allPromises = [];
namesOfPeopleImFollowing.forEach((ele) => {
const myPromise = new Promise((resolve, reject) => {
connection.query(`SELECT * FROM user${ele}posts`, (error, resultOfThis) => {
if (error) {
reject(error);
console.log(`Error found${error}`);
} else {
resolve({ username: ele });
}
});
});
allPromises.push(myPromise);
});
Promise.all(allPromises).then((result) => {
// your code here
})
You can read more about promise.all here

sails.js node.js Parse JSON on controller

In my controller called MapController I'm doing a function to do a parse of remote json files, and from an if-else structure add some values in an array called "parsewebservice", apparently everything is working fine but console.log ( parsewebservice); is not returning the values that were passed to the array "parsewebservice" in the place where it is returning it empty. But when I put it inside the forEach it returns, but everything cluttered and repeated then is not the right way.
I wanted to know why the values that were passed to the array "parsewebservice" are not going along with the variable after populada and what would be the correct way to do it?
Here is my code below:
/**
* MapController
*
* #description :: Server-side logic for managing Maps
* #help :: See http://sailsjs.org/#!/documentation/concepts/Controllers
*/
module.exports = {
index: function(req, res, next) {
Data.find(function foundData(err, datas) {
if (err) return next(err);
var parsewebservice = [];
datas.forEach(function(data, index) {
var req = require("request");
var url = data.address + "?f=pjson";
req(url, function(err, res, retorno) {
if (err) {
console.log(err);
} else {
var camadas = JSON.parse(retorno);
if (camadas.mapName) {
camadas.layers.forEach(function(campo, i) {
if (campo.subLayerIds != null) {
} else if (campo.subLayerIds == null) {
parsewebservice.push([i, "dynamicMapLayer", campo.name, data.address]);
}
});
} else if (camadas.serviceDataType) {
parsewebservice.push([null, "imageMapLayer", camadas.name, data.address]);
} else if (camadas.type) {
parsewebservice.push([null, "featureLayer", camadas.name, data.address]);
}
}
});
});
console.log(parsewebservice);
});
},
};

My first comment has to be that you should not combine function(req, res) with var req = require('request')... you lose your access to the original req object!
So, you need to run a list of async tasks, and do something when they are all complete. That will never be entirely easy, and no matter what, you will have to get used to the idea that your code does not run from top to bottom as you've written it. Your console.log at the bottom runs before any of the callbacks (functions you pass in) you pass to your external requests.
The right way to do this is to use promises. It looks like you are using this request library, whose returned requests can only accept callbacks, not be returned as promises. You can create your own promise wrapper for them, or use an alternative library (several are recommended on the page).
I don't want to write a whole intro-to-promises right here, so what I will do is give you a less pretty, but maybe more understandable way to run some code at the completion of all your requests.
Data.find(function foundData(err, datas) {
if (err) return next(err);
var parsewebservice = [];
// here we will write some code that we will run once per returned data
var processResponse = function(resp) {
parsewebservice.push(resp);
if(parsewebservice.length >= datas.length) {
// we are done, that was the final request
console.log(parsewebservice);
return res.send({data: parsewebservice)}); // or whatever
}
};
datas.forEach(function(data, index) {
var request = require("request");
var url = data.address + "?f=pjson";
request(url, function(err, res, retorno) {
// do some processing of retorno...
// call our function to handle the result
processResponse(retorno);
});
});
console.log(parsewebservice); // still an empty array here
});

I solved the problem.
the "request" module is asynchronous so we need to wait for it to respond and then send the response to the view.
To do this we created a function called "foo" to contain the foreach and the request, we made a callback of that function and finally we made the response (res.view) within that function, so that the controller response would only be sent after the response of the "foo" function to the callback. So we were able to parse.json the data from the "data" collection using foreach and the "request" module and send the objects to the view.
Many thanks to all who have helped me, my sincere thanks.

get object of returned mongoose find

Hi i'm trying to access the elements returned from a find in mongoose and having some trouble with the asynchronous and callback situation.
Here is the code for better understanding.
function retrieveBudgets(email, callback) {
models.User.find({email: email},{budget:true}, function(err,budgets) {
if (err) {
callback(err, null);
} else {
callback(null, budgets);
}
});
};
retrieveBudgets(user.email, function(err, budgets) {
if (err) {
console.log(err);
}
budgets.forEach(function(budget){
console.log(JSON.stringify(budget, null, 4));
});
});
So this line console.log(JSON.stringify(budget, null, 4)); is working correctly and printing the objects to screen in json format but how do I store each to an array of objects from here? if I try to push to an array at this same line I get an error.
I have seen some questions that are similar but i am not getting any headway with them.
EDIT:____________________________________________________________
I did a little hack to get it working, i moved res.render up, so that rendering the page was done at the same time as the callback but I cant see this being the right solution any thoughts
var user=req.session.user;
res.locals.budgets=[];
function retrieveBudgets(email, callback) {
models.User.find({email: email},{budget:true}, function(err, budgets) {
if (err) {
callback(err, null);
} else {
callback(null, budgets);
}
});
};
retrieveBudgets(user.email, function(err, budgets) {
if (err) {
console.log(err);
}
res.locals.budgets = budgets.map((function(b){ return b; });
res.render('budget/budget.jade',{ csrfToken: req.csrfToken() });
});
This works I can access budgets through locals so any feedback on this would be great I doubt its the right way to do it?

the budgets return value that you get from the retrieveBudgets call is already an array.
this is evidenced by your call to budgets.forEach which is a method on arrays.
is there a specific need to create a new array from the items? that can be easily done:
var myNewArray = budgets.map((function(b){ return b; });
this one line of code will map the original budgets array into a new array containing each of the budget items.
there are other methods of creating a new array, depending on what you need to do exactly
update from comments below
what i really want to do to is use the budgets outside of the query so I can pass it to the view
in that case, you need to render the view from within the callback and pass the budgets to the view:
router.get("/foo", function(req, res, next){
retrieveBudgets(user.email, function(err, budgets) {
if (err) { return next(err); }
res.render('budget/budget.jade',{
budgets: budgets,
csrfToken: req.csrfToken()
});
});
});
This is the only, and correct, way to make this work.
If you tried to do it without waiting for the callback to finish, you would not have any data in your budgets array. Therefore, you must wait for the callback to be executed and then render your view with the budgets (or single budget or whatever) passed to the view.
(There are variations of this using promises, but I find callbacks to be the easier way to handle this.)

Turn the line in which you're passing an object to your jade file
res.render('budget/budget.jade',{
csrfToken: req.csrfToken(),
budgets: budgets.map(function(b) {return b;})
});
This will pass budgets to your jade file, and you should be able to access it there.

How do I run an asynchronous 'find' in a loop while incrementing the find parameter so I can generate unique custom id's?

I'm new to mongoose/mongodb and I am trying to do some sort of error handling with my document save.
I am trying to create a stub id to store into the db for easier data retrieval later on (and also to put into the url bar so people can send links to my website to that particular page more easily -- like jsfiddle or codepen).
Basically I want to search for a document with a page_id and if it exists, I want to regenerate that page_id and search until it gets to one that's unused like this:
while(!done){
Model.findOne({'page_id': some_hex}, function (err, doc) {
if(doc){
some_hex = generate_hex();
}
else
{
done = true;
}
});
}
model.page_id = some_hex;
model.save();
However, since mongoose is asynchronous, the while loop will pretty much run indefinitely while the find works in the background until it finds something. This will kill the resources on the server.
I'm looking for an efficient way to retry save() when it fails (with a change to page_id). Or to try and find an unused page_id. I have page_id marked as unique:true in my schema.

Retrying should be performed asynchronously:
var tryToSave = function(doc, callback) {
var instance = new Model(doc);
instance.page_id = generate_hex();
instance.save(function(err) {
if (err)
if (err.code === 11000) { // 'duplicate key error'
// retry
return tryToSave(doc, callback);
} else {
// another error
return callback(err);
}
}
// it worked!
callback(null, instance);
});
};
// And somewhere else:
tryToSave(doc, function(err, instance) {
if (err) ...; // handle errors
...
});