so currently I'm setting up a little nodejs database and I'm basically trying to get all user info (if the user exists) and do a callback returning the user's info, if the user doesn't exist then I return null. Basically my issue here is that when I do return callback(null) or return callback(userdata) return doesn't stop code execution, resulting in 2 executions of the callback and invalid data. It could be due to the various callbacks that these return callback()'s are nested in or the foreach loop, although if it is, I'm not sure how to fix it. Any help is greatly appreciated, thank you!
Here is the code:
const getUserInfo = (sessionToken, callback) => {
return fs.readdir(__dirname + "/database/users", (err, files) => {
if(err) return callback(null);
return files.forEach((file, index) => {
return fs.readFile(__dirname + "/database/users/" + file, "UTF-8", (err, data) => {
if(err) return callback(null);
try {
const parsedData = JSON.parse(data);
if(parsedData.sessionToken === sessionToken) return callback({ email: parsedData.email, username: parsedData.username, bought: parsedData.bought });
else if(index === files.length - 1) return callback(null);
} catch {
return callback(null);
}
});
});
});
}
A little explanation for the code: this is a function that takes in a sessionToken, which is a random string of characters for the user's current session, and a callback of course. It then reads the database/users directory to get all the users, then for every user in that directory it reads the file and try's to parse the data through JSON, if it doesn't work then of course it returns null, if it does work and the sessionToken matches then it executes the callback with all the necessary user information. If we get to the end of the list and none of the sessions matched then we come to the conclusion the user doesn't exist.
Don't use a .forEach() loop. Use a plain for loop instead. .forEach() provides NO way to stop the loop. When you return from the .forEach() loop, you're just returning from the callback function, not returning from the parent function so there's no way to stop the loop or return from the parent function from within the .forEach() loop.
Using a regular for loop, you can either break or return to stop the loop.
In addition, since you're doing an asynchronous operation inside the loop, you also will need to use let data = await fs.promises.readFile() (and make the parent function async) in order to sequence your file reads one after the other. Without that, all your read operations will proceed in parallel and you will have no control over the order of completion. And, one you do that, you may as well return a promise from your function with the results, rather than use a plain callback function.
Here's an example implementation:
const getUserInfo = async (sessionToken) => {
let files = await fs.promises.readdir(__dirname + "/database/users");
for (let file of files) {
let data = await fs.promises.readFile(__dirname + "/database/users/" + file, "UTF-8");
const parsedData = JSON.parse(data);
if (parsedData.sessionToken === sessionToken) {
return { email: parsedData.email, username: parsedData.username, bought: parsedData.bought };
}
}
return null;
}
// usage
getUserInfo(token).then(val => {
if (val) {
console.log('Got user data', val);
} else {
console.log('User data not found');
}
}).catch(err => {
console.log(err);
})
Related
i am building validation for one of form's field serverside (expressjs) and doing following actions for that:
Read data from json file
Get property from it (Array)
Check if it contains every single element of user generated array and nothing more, for example:
[1,2,3,4,5]; (json array)
[1,2,3,4,5,6] (user generated array) //must return false
[1,2,3,4,5];
[1,3,4] //must return true;
[1,2,3,4,5];
[1,2,7] //must return false;
so i am using this code for that:
const contains = (arr1, arr2) => {
arr2.every(v => arr1.indexOf(v) !== -1)
}
var match;
fs.readFile('../tags.json', 'utf8', (err, data)=>{
var JsonData = JSON.parse(data);
var tagsArray = JsonData.tags;
console.log(tagsArray)
console.log(tags)
if(tagsArray instanceof Array){
console.log('tagsArray is array')
}
if(!contains(tagsArray, tags)){
match = false
}
else{
match = true
}
console.log(match + ' blah1')
});
console.log(match + ' blah2')
if(match == false){
return res.status(409).send({
message: 'Do not provide your own tags'
});
}
but it always returns false inside fs.readFile block because it returns undefined outside fs.readFile block, so this means that contains function return undefined (i tested it)
so what is the clue for this?
Thanks!
fs.readFile is asynchronous, so any code that depends on its result (the file being read) needs to go within your callback function. (The callback function is the (err, data) => { ... } part.)
Move the console.log(match + 'blah2') and if(match == false) { ... } parts inside of the callback (after the blah1 line).
You could also look into async or use fs.readFileSync which would allow you to avoid using callback functions.
Another side point, you will want to make sure you always reach a res.send() line, i.e. when match == true in your case. Otherwise your http request will not return when match is true.
Edit:
Here's a really basic structure for express, mostly pseudocode & comments, just to illustrate callbacks:
app.post('/tags', (req, res) => {
// your setup code here
fs.readFile('../tags.json', 'utf8', (err, data) => {
console.log('readFile has finished')
// now you have heard back from readFile
// check the err and send 500 if there was a problem
// otherwise work with the file in the var data
// any other db-related stuff also goes in here, which probably
// has its own callback you need to use
db.save(data, (err) => {
// db call is done, potentially with an error
// Here you can use `res` to send http response
})
// !! again here the db is still doing its work
})
// !! anything you add here will be executed before readFile is done
console.log('readFile is probably still at work')
})
I should also point out that you want contains to return the bool value, i.e. return arr2.every(...)
You can use async/await :
async function read(){
let data = await fs.readFile(<path>);
console.log(data); //You can use data everywhere within this scope
}
Actually I'm not sure that Title of my question is 'correct', if you
have any idea with it, you could leave a comment and I'll rename it.
I am trying to rewrite my old function which make http-requests and insert many object at mongoDB via mongoose. I already have a working version of it, but I face a problem while using it. Basically, because when I'm trying to insertMany 20 arrays from 20+ request with ~50'000 elements from one request it cause a huge memory leak. Even with MongoDB optimization.
Logic of my code:
function main() {
server.find({locale: "en_GB"}).exec(function (err, server) {
for (let i = 0; i < server.length; i++) { //for example 20 servers
rp({url: server[i].slug}).then(response => {
auctions.count({
server: server[i].name,
lastModified: {$gte: response.data.files[0].lastModified}
}).then(function (docs) {
if (docs < 0) {
//We don't insert data if they are already up-to-date
}
else {
//I needed response.data.files[0].url and server[i].name from prev. block
//And here is my problem
requests & insertMany and then => loop main()
})
}
})
}).catch(function (error) {
console.log(error);
})
}
})
}
main()
Actually I have already trying many different things to fix it. First-of-all I was trying to add setInterval after else block like this:
setTimeout(function () {
//request every server with interval, instead of all at once
}, 1000 * (i + 1));
but I create another problem for myself because I needed to recursive my main() function right after. So I can't use: if (i === server[i].length-1) to call garbage collector or to restart main() because not all server skip count validation
Or let's see another example of mine:
I change for (let i = 0; i < server.length; i++) from 3-rd line to .map and move it from 3-rd line close to else block but setTimeout doesn't work with .map version, but as you may already understand script lose correct order and I can't make a delay with it.
Actually I already understand how to fix it at once. Just re-create array via let array_new = [], array_new.push = response.data.files[0].url with use of async/await. But I'm not a big expert in it, so I already waste a couple of hours. So the only problem for now, that I don't know how to return values from else block
As for now I'm trying to form array inside else block
function main() {
--added let array_new = [];
[v1]array_new.url += response.data.files[0].url;
[v2]array_new.push(response.data.files[0].url);
return array_new
and then call array_new array via .then , but not one of these works fine for now. So maybe someone will give me a tip or show me already answered question #Stackoverflow that could be useful in my situation.
Since you are essentially dealing with promises, you can refactor your function logic to use async await as follows:
function async main() {
try {
const servers = await server.find({locale: "en_GB"}).exec()
const data = servers.map(async ({ name, slug }) => {
const response = await rp({ url: slug })
const { lastModified, url } = response.data.files[0]
const count = await auctions.count({
server: name,
lastModified: { $gte: lastModified }
})
let result = {}
if (count > 0) result = { name, url }
return result
}).filter(d => Object.keys(d).length > 0)
Model.insertMany(data)
} catch (err) {
console.error(err)
}
}
Your problem is with logic obscured by your promises. Your main function recursively calls itself N times, where N is the number of servers. This builds up exponentially to eat memory both by the node process and MongoDB handling all the requests.
Instead of jumping into async / await, start by using the promises and waiting for the batch of N queries to complete before starting another batch. You can use [Promise.all] for this.
function main() {
server.find({locale: "en_GB"}).exec(function (err, server) {
// need to keep track of each promise for each server
let promises = []
for (let i = 0; i < server.length; i++) {
let promise = rp({
url: server[i].slug
}).then(function(response) {
// instead of nesting promises, return the promise so it is handled by
// the next then in the chain.
return auctions.count({
server: server[i].name,
lastModified: {
$gte: response.data.files[0].lastModified
}
});
}).then(function (docs) {
if (docs > 0) {
// do whatever you need to here regarding making requests and
// inserting into DB, but don't call main() here.
return requestAndInsert();
}
}).catch(function (error) {
console.log(error);
})
// add the above promise to out list.
promises.push(promise)
}
// register a new promise to run once all of the above promises generated
// by the loop have been completed
Promise.all(promises).then(function () {
// now you can call main again, optionally in a setTimeout so it waits a
// few seconds before fetchin more data.
setTimeout(main, 5000);
})
})
}
main()
So I have found this question which seems pretty similar but I do not understand the answer at all I tried to implement it but I do not recognize the patterns of the answer in my code. similar question
Now here is my problem, I have this piece of code :
var fs = require('fs');
var index = JSON.parse(fs.readFileSync('../data/7XXX7/index.json', 'utf8'));
window = {};
var indicators = require('./indicators');
var parser = new window.patient.Indicator('tes', 'test');
var i = 0;
function create_indicators() {
var result = [];
fs.readdirSync('../data/7XXX7/files/').forEach(file => {
fs.readFile('../data/7XXX7/files/' + file, 'utf8', function (err, data) {
if (err)
throw err;
let $ = {};
$.poids = parser.poids(data);
$.taille = parser.taille(data);
$.temperature = parser.temperature(data);
$.tension = parser.tension(data);
$.pouls = parser.pouls(data);
$.ps = parser.ps(data);
$.saturation = parser.saturation(data);
for (var j in index.files)
{
if (index.files[j].name === file)
{
$.id = index.files[j].name;
$.date = index.files[j].date;
$.name = index.files[j].IntituleSession;
break;
}
}
if ($.poids || $.taille || $.temperature || $.tension || $.pouls || $.ps || $.saturation)
{
result.push($);
console.log(result); // print the actual state of result
// console.log(i); prints 0 then 1 then ...
i++;
}
});
console.log(i); // prints 0
});
console.log(result); // prints []
return result;
}
let result = create_indicators();
console.log(result); // prints []
And it displays :
[]
Why does the callback function in readFile has it's own variables ? Cause it's asynchronous ? But when I use readFileSync it doesn't work too.
How to make result get all the values I put into it ? when I console log result after result.push($); it works so that's not my parser, i is also properly indented each time.
Your code doesn't wait for the files to get read and have the result pushed to result before moving on. Where you're doing asynchronous operations on items in an array, I would recommend using promises and using Promise.all() to wait for each file to get read and processed before you try using the result. You could do something like this:
function create_indicators() {
const result = fs.readdirSync('../data/7XXX7/files/').map(file =>
new Promise((resolve, reject) => {
fs.readFile('../data/7XXX7/files/' + file, 'utf8', (err, data) => {
if (err) reject(err);
// do whatever
if ($.poids || /* ... */ $.saturation) {
// ...
resolve($); // instead of `result.push($);`
} else {
resolve(); // can't reject for `Promise.all()` to work
}
})
}));
return Promise.all(result).then(items => items.filter(item => item));
}
create_indicators().then(indicators => {
// do something with your list of indicators
}).catch(err => {
// handle error
});
It creates a promise for each file in your directory that resolves when the file has been processed. It resolves with the item if there is one or nothing if your condition is not met, rejecting if there's an error (promise equivalent to throw). Since you only want the items that meet your condition, you can then do a filter on the result of Promise.all() to get rid of any undefined in the array (you could also get rid of the condition checking in the fs.readFile callback and do it instead in the filter if you'd like). This returns a promise that resolves with your filtered list.
Here's your problem:
fs.readFileSync('../data/7XXX7/files/' + file, 'utf8', function (err, data) {
The readFileSync doesn't take a callback as an argument. It returns the data or raises an exception. It is synchronous (as the "Sync" in the name suggests) and you're using it as if it was asynchronous.
See the docs:
https://nodejs.org/api/fs.html
readFileSync doesn't callback. It is synchronous.
use fs.readdir to get the list of files you want to read. See How do you get a list of the names of all files present in a directory in Node.js?
Need to understand how callback works.
readFileSync doesn't callback. It might be helpful to explain how callback works in asynchronous fs.readFile and fs.readdir
When you are doing asynchronous operations, because you don't know when it is going to be finished, you pass in a function (callback) in the parameter, and run it at the end of the operation.
fs.readFile('/etc/passwd', function (err, data) {
if (err) throw err;
console.log(data);
});
fs.readFile in the above code will run the function (err, data) when it finishes executing and pass in the data as the second parameter. If error occurs it will pass in the error as the first parameter.
You can also get a callback function defining what to do when the parsing is over. The callback will need to take error and result. (if you need the error)
Read:
http://fredkschott.com/post/2014/03/understanding-error-first-callbacks-in-node-js/
So your create_indicators function should take a callback function.
fs = require("fs")
function create_indicators(folderPath, callback) {
let result = [];
fs.readdir(folderPath, (err, files) => {
if (err)
callback(err, null); //pass the error to callback if there is any
else {
files.forEach((file, index, filesArray) => {
fs.readFile(file, (err, data) => {
if (err)
callback(err, null); //pass the error to callback if there is any
else {
//.....parse....
result.push(data);
// pass data to callback function when it is the last result
if (result.length == filesArray.length)
callback(null, result);
}
});
});
}
})
}
When you call it, pass in what you want to do with the result and error as a function.
create_indicators(".", function(err,result){
if (err)
console.error("Got error:", err);
else
console.log("Got result:", result);
//do what you want with the final result
})
Once you got the callback working, look into Promise which will make this procedure cleaner and easier. Read: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
I am trying to write a function that:
Takes an array of URLs
Gets files from URLs in parallel (order's irrelevant)
Processes each file
Returns an object with the processed files
Furthermore, I don't need for errors in #2 or #3 to affect the rest of the execution in my application in any way - the app could continue even if all the requests or processing failed.
I know how to fire all the requests in a loop, then once I have all the data, fire the callback to process the files, by using this insertCollection pattern.
However, this is not efficient, as I shouldn't need to wait for ALL files to download before attempting to process them - I would like to process them as each download finishes.
So far I have this code:
const request = require('request');
const urlArray = [urlA, urlB, urlC];
const results = {};
let count = 0;
let processedResult;
const makeRequests = function (urls, callback) {
for (let url of urls) {
request(url, function(error, response, body) {
if (error) {
callback(error);
return;
}
processedResult = callback(null, body)
if (processedResult) {
console.log(processedResult); // prints correctly!
return processedResult;
}
})
}
};
const processResult = function(error, file) {
if (error) {
console.log(error);
results.errors.push(error);
}
const processedFile = file + `<!-- Hello, Dolly! ${count}-->`;
results.processedFiles.push(processedFile);
if (++count === urlArray.length) {
return results;
}
};
const finalResult = makeRequests(urlArray, processResult);
console.log(finalResult); // undefined;
In the last call to processResult I manage to send a return, and makeRequests captures it, but I'm failing to "reign it in" in finalResult after that.
My questions are:
Why is this not working? I can print a well-formed processedResult
on the last iteration of makeRequests, but somehow I cannot return
it back to the caller (finalResult)
How can this be solved, ideally "by hand", without promises or the
help of libraries like async?
The makeRequests function returns undefined to finalResult because that is a synchronous function. Nothing stops the code executing, so it gets to the end of the function and, because there is no defined return statement, it returns undefined as default.
I am using [ssh2-sftp-client][1] package to recursively read all the directories inside a given remote path.
Here is the code.
const argv = require('yargs').argv;
const client = require('ssh-sftp-client');
const server = new Client();
const auth = {
host: '192.168.1.11',
username: argv.u,
password: argv.p
};
const serverRoot = '/sites/';
const siteName = 'webmaster.com';
// list of directories on the server will be pushed to this array
const serverPaths = [];
server.connect(auth).then(() => {
console.log(`connected to ${auth.host} as ${auth.username}`);
}).catch((err) => {
if (err) throw err;
});
server.list('/sites/').then((dirs) => {
redursiveDirectorySearch(dirs, `${serverRoot}${siteName}/`);
})
.catch((err) => {
if (err) throw err;
});
function recursiveDirectorySearch(dirs, prevPath) {
let paths = dirs.filter((dir) => {
// returns directories only
return dir.type === 'd';
});
if (paths.length > 0) {
paths.forEach((path) => {
server
.list(`${prevPath}${path.name}`)
.then((dirs) => {
console.log(`${prevPath}${path.name}`);
recursiveDirectorySearch(dirs, `${prevPath}${path.name}`);
serverPaths.push(`${prevPath}${path.name}`);
})
}
}
}
At first, a connection will be made to the server and then list whatever is under '/sites/' directory, which will then be passed to 'recursiveDirectorySearch' function. This function will receive an array of whatever is found under '/sites/' directory on the server as the first parameter, which will be filtered out so it only has directories. If one or more directory was found, a call to the server for each directory in the array will be made in order to retrieve everything under '/sites/'+'name of the directory in the array'. This same function will be called again with whatever is returned by the call to the server until no other directory is found.
Whenever a directory is found, its name in string will be pushed to 'serverPaths' array. As far as I can tell, this search is working and successfully pushing all the directory names to the array.
However, I can't think of a way to detect when this recursive search for all the directories is complete so I can do something with the 'serverPaths' array.
I tried to take advantage of Promise.all() but don't know how to use it when how many function calls are made is unknown.
You're simply lacking a couple of returns, add a Promise.all, and an Array#map and you're done
Note: not using Promise.all on serverPaths, but rather, using the fact that returning a Promise in .then will result in the Promise that is returned by .then taking on the Promise that is returned (hmmm, that isn't very well explained, is it, but it's Promises 101 stuff really!
server.list('/sites/').then((dirs) => {
// added a return here
return recursiveDirectorySearch(dirs, `${serverRoot}${siteName}/`);
})
.then(() => {
// everything is done at this point,
// serverPaths should be complete
})
.catch((err) => {
if (err) throw err;
});
function recursiveDirectorySearch(dirs, prevPath) {
let paths = dirs.filter((dir) => {
// returns directories only
return dir.type === 'd';
});
// added a return, Promise.all and changed forEach to map
return Promise.all(paths.map((path) => {
//added a return here
return server
.list(`${prevPath}${path.name}`)
.then((dirs) => {
console.log(`${prevPath}${path.name}`);
// swapped the next two lines
serverPaths.push(`${prevPath}${path.name}`);
// added a return here, push the path before
return recursiveDirectorySearch(dirs, `${prevPath}${path.name}`);
})
}));
}
One of the main things that is jumping out at me is your initial if statement. (if paths.length > 0) { run recursion } This appears to work really well for the first call because you know that the data coming back will be populated with an array full of directories.
Your function however, does not appear to have logic built out for an array with a length of 0. In this scenario it would be possible for you to get all of the directory names you are looking for. Presented in the manner that you are looking for. It would also mean that your calls on the higher parts of the tree are never able to resolve.
Try to add logic to handle cases for an array with a length of zero | if (paths.length === 0) return; | This would be a hard break out of the recursive calls on the higher parts of the stack.