Develop Reference

How to stop code within nested callbacks and a foreach loop?

so currently I'm setting up a little nodejs database and I'm basically trying to get all user info (if the user exists) and do a callback returning the user's info, if the user doesn't exist then I return null. Basically my issue here is that when I do return callback(null) or return callback(userdata) return doesn't stop code execution, resulting in 2 executions of the callback and invalid data. It could be due to the various callbacks that these return callback()'s are nested in or the foreach loop, although if it is, I'm not sure how to fix it. Any help is greatly appreciated, thank you!
Here is the code:
const getUserInfo = (sessionToken, callback) => {
return fs.readdir(__dirname + "/database/users", (err, files) => {
if(err) return callback(null);
return files.forEach((file, index) => {
return fs.readFile(__dirname + "/database/users/" + file, "UTF-8", (err, data) => {
if(err) return callback(null);
try {
const parsedData = JSON.parse(data);
if(parsedData.sessionToken === sessionToken) return callback({ email: parsedData.email, username: parsedData.username, bought: parsedData.bought });
else if(index === files.length - 1) return callback(null);
} catch {
return callback(null);
}
});
});
});
}
A little explanation for the code: this is a function that takes in a sessionToken, which is a random string of characters for the user's current session, and a callback of course. It then reads the database/users directory to get all the users, then for every user in that directory it reads the file and try's to parse the data through JSON, if it doesn't work then of course it returns null, if it does work and the sessionToken matches then it executes the callback with all the necessary user information. If we get to the end of the list and none of the sessions matched then we come to the conclusion the user doesn't exist.

Don't use a .forEach() loop. Use a plain for loop instead. .forEach() provides NO way to stop the loop. When you return from the .forEach() loop, you're just returning from the callback function, not returning from the parent function so there's no way to stop the loop or return from the parent function from within the .forEach() loop.
Using a regular for loop, you can either break or return to stop the loop.
In addition, since you're doing an asynchronous operation inside the loop, you also will need to use let data = await fs.promises.readFile() (and make the parent function async) in order to sequence your file reads one after the other. Without that, all your read operations will proceed in parallel and you will have no control over the order of completion. And, one you do that, you may as well return a promise from your function with the results, rather than use a plain callback function.
Here's an example implementation:
const getUserInfo = async (sessionToken) => {
let files = await fs.promises.readdir(__dirname + "/database/users");
for (let file of files) {
let data = await fs.promises.readFile(__dirname + "/database/users/" + file, "UTF-8");
const parsedData = JSON.parse(data);
if (parsedData.sessionToken === sessionToken) {
return { email: parsedData.email, username: parsedData.username, bought: parsedData.bought };
}
}
return null;
}
// usage
getUserInfo(token).then(val => {
if (val) {
console.log('Got user data', val);
} else {
console.log('User data not found');
}
}).catch(err => {
console.log(err);
})

Trying to wait for list to be populated before iterating through it NodeJS

I have some code that takes an uploaded file, executes it and gets the output. It then compares this output against expected output to check if the script did as expected.
I am now trying to improve this functionality so that an uploaded file will be run several times, each time being checked against a different expected output, or "test case". I then want to push "correct" or "incorrect" onto a results array, so that I can go through that array at the end and check whether there are any "incorrect" (whether the file failed any test case).
I have tried just callbacks within each function.
I have tried using await and async on the getArray as seen below
Using both callbacks and async together.
This is the parent function code that calls for the array to be created, and wants to iterate through it after it has been created.
var resultsArr = await getResults(file.name, files.length, markerDir);
//file.name is the name from the uploaded file object
//files.length is the number of subdirectories (number of testcases to run against)
//markerDir is the str path to where these testcases are stored
if (resultsArr){
for(var i=0;i<resultsArr.length;i++) {
if (resultsArr[i] == "incorrect"){
checkForMarkerCb('incorrect'); //Calls back to frontend to
break; //display result on web app
}
else if (i+1 == resultsArr.length) {
checkForMarkerCb('correct');
}
}
}
The following is inside the getResults function that is called above
for(var i=1; i<=fileLength; i++) {
var sampleOut = markerDir + '/test' + i + '/stdout.txt';
//Grab expected stdout.txt
var markerOut = fs.readFileSync(sampleOut, 'utf-8', function(err){
if (err){
throw err;
};
});
//Run the file and grab the output
executeFile(filename, function(fileOut){
//Compare output with sample stdout
if (markerOut == fileOut){
resultsArr.push('correct');
}
else {
resultsArr.push('incorrect');
}
});
}
//If results array has a response for each testcase
if (resultsArr.length == fileLength) {
return resultsArr;
}
Implementation of executeFile() as requested:
function executeFile(filename, execFileCb){
//pathToUpload is a str path to where the upload is stored
const child = execFile('python', [pathToUpload], (err,stdout,stderr) => {
if (err) {
throw err;
}
execFileCb(stdout); //Callback with output of file
});
}
function executeFileAsync(filename) {
return new Promise(function(resolve,reject){
executeFile(filename, function(err, data){
if (err !== null) reject(err);
else resolve(data);
});
});
}
which was called inside getResults() using
var fileOut = await executeFileAsync(filename)
The initial function that calls getResults().
getResults(): which gets the path to each directory and calls pushes the results of comparing outputs onto a results array.
executeFile(): uses 'child_process' to run a file and calls back with the output.
I expect the code to wait for getResults to return with the resultsArr so that the for loop can iterate through and check for any "incorrect". Instead, getResults returns before resultsArr is populated.
Using some logging, I see that the code for checking markerOut == fileOut is executed at the end after the getResults() for loop has already completed. I tried setting up the call to executeFile() to also be an async/await similar to how getResults() is called but still no change.
I may not be using async/callbacks correctly, any help is greatly appreciated.

Your executeFileAsync function currently calls executeFile with a callback that is expecting two arguments, but executeFile then does call this execFileCb always with only one argument which is interpreted as an error. It also should not use throw in an asynchronous callback.
Instead, merge them into one function:
function executeFile(filename) {
return new Promise(function(resolve,reject){
//pathToUpload is a str path to where the upload is stored
const child = execFile('python', [pathToUpload], (err,stdout,stderr) => {
if (err) reject(err);
else resolve(stdout); //Callback with output of file
});
});
}

sails.js node.js Parse JSON on controller

In my controller called MapController I'm doing a function to do a parse of remote json files, and from an if-else structure add some values in an array called "parsewebservice", apparently everything is working fine but console.log ( parsewebservice); is not returning the values that were passed to the array "parsewebservice" in the place where it is returning it empty. But when I put it inside the forEach it returns, but everything cluttered and repeated then is not the right way.
I wanted to know why the values that were passed to the array "parsewebservice" are not going along with the variable after populada and what would be the correct way to do it?
Here is my code below:
/**
* MapController
*
* #description :: Server-side logic for managing Maps
* #help :: See http://sailsjs.org/#!/documentation/concepts/Controllers
*/
module.exports = {
index: function(req, res, next) {
Data.find(function foundData(err, datas) {
if (err) return next(err);
var parsewebservice = [];
datas.forEach(function(data, index) {
var req = require("request");
var url = data.address + "?f=pjson";
req(url, function(err, res, retorno) {
if (err) {
console.log(err);
} else {
var camadas = JSON.parse(retorno);
if (camadas.mapName) {
camadas.layers.forEach(function(campo, i) {
if (campo.subLayerIds != null) {
} else if (campo.subLayerIds == null) {
parsewebservice.push([i, "dynamicMapLayer", campo.name, data.address]);
}
});
} else if (camadas.serviceDataType) {
parsewebservice.push([null, "imageMapLayer", camadas.name, data.address]);
} else if (camadas.type) {
parsewebservice.push([null, "featureLayer", camadas.name, data.address]);
}
}
});
});
console.log(parsewebservice);
});
},
};

My first comment has to be that you should not combine function(req, res) with var req = require('request')... you lose your access to the original req object!
So, you need to run a list of async tasks, and do something when they are all complete. That will never be entirely easy, and no matter what, you will have to get used to the idea that your code does not run from top to bottom as you've written it. Your console.log at the bottom runs before any of the callbacks (functions you pass in) you pass to your external requests.
The right way to do this is to use promises. It looks like you are using this request library, whose returned requests can only accept callbacks, not be returned as promises. You can create your own promise wrapper for them, or use an alternative library (several are recommended on the page).
I don't want to write a whole intro-to-promises right here, so what I will do is give you a less pretty, but maybe more understandable way to run some code at the completion of all your requests.
Data.find(function foundData(err, datas) {
if (err) return next(err);
var parsewebservice = [];
// here we will write some code that we will run once per returned data
var processResponse = function(resp) {
parsewebservice.push(resp);
if(parsewebservice.length >= datas.length) {
// we are done, that was the final request
console.log(parsewebservice);
return res.send({data: parsewebservice)}); // or whatever
}
};
datas.forEach(function(data, index) {
var request = require("request");
var url = data.address + "?f=pjson";
request(url, function(err, res, retorno) {
// do some processing of retorno...
// call our function to handle the result
processResponse(retorno);
});
});
console.log(parsewebservice); // still an empty array here
});

I solved the problem.
the "request" module is asynchronous so we need to wait for it to respond and then send the response to the view.
To do this we created a function called "foo" to contain the foreach and the request, we made a callback of that function and finally we made the response (res.view) within that function, so that the controller response would only be sent after the response of the "foo" function to the callback. So we were able to parse.json the data from the "data" collection using foreach and the "request" module and send the objects to the view.
Many thanks to all who have helped me, my sincere thanks.

How to assign a variable in callback function in a callback function in javascript

So I have found this question which seems pretty similar but I do not understand the answer at all I tried to implement it but I do not recognize the patterns of the answer in my code. similar question
Now here is my problem, I have this piece of code :
var fs = require('fs');
var index = JSON.parse(fs.readFileSync('../data/7XXX7/index.json', 'utf8'));
window = {};
var indicators = require('./indicators');
var parser = new window.patient.Indicator('tes', 'test');
var i = 0;
function create_indicators() {
var result = [];
fs.readdirSync('../data/7XXX7/files/').forEach(file => {
fs.readFile('../data/7XXX7/files/' + file, 'utf8', function (err, data) {
if (err)
throw err;
let $ = {};
$.poids = parser.poids(data);
$.taille = parser.taille(data);
$.temperature = parser.temperature(data);
$.tension = parser.tension(data);
$.pouls = parser.pouls(data);
$.ps = parser.ps(data);
$.saturation = parser.saturation(data);
for (var j in index.files)
{
if (index.files[j].name === file)
{
$.id = index.files[j].name;
$.date = index.files[j].date;
$.name = index.files[j].IntituleSession;
break;
}
}
if ($.poids || $.taille || $.temperature || $.tension || $.pouls || $.ps || $.saturation)
{
result.push($);
console.log(result); // print the actual state of result
// console.log(i); prints 0 then 1 then ...
i++;
}
});
console.log(i); // prints 0
});
console.log(result); // prints []
return result;
}
let result = create_indicators();
console.log(result); // prints []
And it displays :
[]
Why does the callback function in readFile has it's own variables ? Cause it's asynchronous ? But when I use readFileSync it doesn't work too.
How to make result get all the values I put into it ? when I console log result after result.push($); it works so that's not my parser, i is also properly indented each time.

Your code doesn't wait for the files to get read and have the result pushed to result before moving on. Where you're doing asynchronous operations on items in an array, I would recommend using promises and using Promise.all() to wait for each file to get read and processed before you try using the result. You could do something like this:
function create_indicators() {
const result = fs.readdirSync('../data/7XXX7/files/').map(file =>
new Promise((resolve, reject) => {
fs.readFile('../data/7XXX7/files/' + file, 'utf8', (err, data) => {
if (err) reject(err);
// do whatever
if ($.poids || /* ... */ $.saturation) {
// ...
resolve($); // instead of `result.push($);`
} else {
resolve(); // can't reject for `Promise.all()` to work
}
})
}));
return Promise.all(result).then(items => items.filter(item => item));
}
create_indicators().then(indicators => {
// do something with your list of indicators
}).catch(err => {
// handle error
});
It creates a promise for each file in your directory that resolves when the file has been processed. It resolves with the item if there is one or nothing if your condition is not met, rejecting if there's an error (promise equivalent to throw). Since you only want the items that meet your condition, you can then do a filter on the result of Promise.all() to get rid of any undefined in the array (you could also get rid of the condition checking in the fs.readFile callback and do it instead in the filter if you'd like). This returns a promise that resolves with your filtered list.

Here's your problem:
fs.readFileSync('../data/7XXX7/files/' + file, 'utf8', function (err, data) {
The readFileSync doesn't take a callback as an argument. It returns the data or raises an exception. It is synchronous (as the "Sync" in the name suggests) and you're using it as if it was asynchronous.
See the docs:
https://nodejs.org/api/fs.html

readFileSync doesn't callback. It is synchronous.
use fs.readdir to get the list of files you want to read. See How do you get a list of the names of all files present in a directory in Node.js?
Need to understand how callback works.
readFileSync doesn't callback. It might be helpful to explain how callback works in asynchronous fs.readFile and fs.readdir
When you are doing asynchronous operations, because you don't know when it is going to be finished, you pass in a function (callback) in the parameter, and run it at the end of the operation.
fs.readFile('/etc/passwd', function (err, data) {
if (err) throw err;
console.log(data);
});
fs.readFile in the above code will run the function (err, data) when it finishes executing and pass in the data as the second parameter. If error occurs it will pass in the error as the first parameter.
You can also get a callback function defining what to do when the parsing is over. The callback will need to take error and result. (if you need the error)
Read:
http://fredkschott.com/post/2014/03/understanding-error-first-callbacks-in-node-js/
So your create_indicators function should take a callback function.
fs = require("fs")
function create_indicators(folderPath, callback) {
let result = [];
fs.readdir(folderPath, (err, files) => {
if (err)
callback(err, null); //pass the error to callback if there is any
else {
files.forEach((file, index, filesArray) => {
fs.readFile(file, (err, data) => {
if (err)
callback(err, null); //pass the error to callback if there is any
else {
//.....parse....
result.push(data);
// pass data to callback function when it is the last result
if (result.length == filesArray.length)
callback(null, result);
}
});
});
}
})
}
When you call it, pass in what you want to do with the result and error as a function.
create_indicators(".", function(err,result){
if (err)
console.error("Got error:", err);
else
console.log("Got result:", result);
//do what you want with the final result
})
Once you got the callback working, look into Promise which will make this procedure cleaner and easier. Read: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise

How to return from a looped asynchronous function with callback in Node

I am trying to write a function that:
Takes an array of URLs
Gets files from URLs in parallel (order's irrelevant)
Processes each file
Returns an object with the processed files
Furthermore, I don't need for errors in #2 or #3 to affect the rest of the execution in my application in any way - the app could continue even if all the requests or processing failed.
I know how to fire all the requests in a loop, then once I have all the data, fire the callback to process the files, by using this insertCollection pattern.
However, this is not efficient, as I shouldn't need to wait for ALL files to download before attempting to process them - I would like to process them as each download finishes.
So far I have this code:
const request = require('request');
const urlArray = [urlA, urlB, urlC];
const results = {};
let count = 0;
let processedResult;
const makeRequests = function (urls, callback) {
for (let url of urls) {
request(url, function(error, response, body) {
if (error) {
callback(error);
return;
}
processedResult = callback(null, body)
if (processedResult) {
console.log(processedResult); // prints correctly!
return processedResult;
}
})
}
};
const processResult = function(error, file) {
if (error) {
console.log(error);
results.errors.push(error);
}
const processedFile = file + `<!-- Hello, Dolly! ${count}-->`;
results.processedFiles.push(processedFile);
if (++count === urlArray.length) {
return results;
}
};
const finalResult = makeRequests(urlArray, processResult);
console.log(finalResult); // undefined;
In the last call to processResult I manage to send a return, and makeRequests captures it, but I'm failing to "reign it in" in finalResult after that.
My questions are:
Why is this not working? I can print a well-formed processedResult
on the last iteration of makeRequests, but somehow I cannot return
it back to the caller (finalResult)
How can this be solved, ideally "by hand", without promises or the
help of libraries like async?

The makeRequests function returns undefined to finalResult because that is a synchronous function. Nothing stops the code executing, so it gets to the end of the function and, because there is no defined return statement, it returns undefined as default.