I have multiple(hundreds) of files in a folder i need to read. Currently I am using a method that reads all the files and then returns the content as an array. But i need to change it so that i read a file and i send it to the client and then i read the second one and so one until the end... to make it more efficient. Any suggestions?
Currently code:
const fse = require("fs-extra");
const path = require("path");
return (
fse
.readdir(direc)
.then((filenames: any) => {
return filenames.map((filename: any) => path.join(direc, filename));
})
.then((filepaths: any) => {
return filepaths.map((filepath: any) =>
fse.readFile(filepath).then((filecontents: any) => {
return JSON.parse(filecontents);
})
);
})
// Promise.all consumes an array of promises, and returns a
// new promise that will resolve to an array of concrete "answers"
.then((filecontents: any) => Promise.all(filecontents))
.then((realcontents: any) => {
return realcontents;
})
);
Instead of putting all the file reading promises in an array and then calling Promise.all, chain them together. I'm not very familiar with Typescript syntax, so I'll provide an example code in plain JS and I hope you can convert whatever's needed:
// ...
.then(filepaths => {
let promise = Promise.resolve();
filepaths.forEach(filepath => {
promise = promise
.then(() => fs.readFile(filePath))
.then(handleFile)
.catch(handleError);
});
return promise;
})
// ...
Then you just need to create a handleFile function that takes in the file contents of a single file and does whatever you want with it (parses it, sends data back over a response, etc.). This way each file will be handled in order. You're going to want your handleError function to return a resolved promise, too, so one failed file read doesn't break the entire chain.
Related
This question already has answers here:
How to return many Promises and wait for them all before doing other stuff
(6 answers)
Closed 2 years ago.
Please, I know this question has been answered before. I have read this answer & this article, but I can't figure out how to fix my code yet.
I have created a function that reads some file's content & returns a new Promise. Here it's the function:
// array of string representing each file's path
const allSpecFiles = [
'/path/to/the/file1.spec.js',
'/path/to/the/file2.spec.js',
'/path/to/the/file3.spec.js'
];
// method to read an individual file content & return a Promise
const readFileContent = file => {
return new Promise((resolve, reject) => {
fs.readFile(file, 'utf8', (err, data) => {
if (err) return reject(err);
return resolve(data);
});
});
};
Now, I'm trying to loop through an array of strings storing each file's path, call the readFileContent() method & pass current loop's value as its param with map() method since I would like to create another array of strings with each file's content.
This is what I have tried:
const allSpecFilesArr = allSpecFiles.map(async file => await readFileContent(file));
console.log(allSpecFilesArr); // I get Promise { <pending> }
I have also tried wrapping the whole script like so:
(async () => {
const allSpecFilesArr = await allSpecFiles.map(file => readFileContent(file));
console.log(allSpecFilesArr); // still prints Promise { <pending> }
})();
What I'm doing wrong?
There's no need to wrap fs.readFile, use fs/promises. Try this out:
const fs = require('fs/promises')
const paths = [ './one', './two' ]
;(async () => {
const contents = await Promise.all(paths.map((p) => fs.readFile(p, 'utf8')))
console.log(contents)
})()
The second solution is partially correct. You're awaiting the result of the map function which is, in this case, an array of promises.
If you removed await in front of the map call and called await Promise.all(allSpecFilesArr) you will get what you need.
You Could do something like this:
async read (paths) {
const promises = [];
for (path in paths) {
promises.push(readFileContent(path));
}
const arrOfContentYouWant = await Promise.all(promises);
return arrOfContentYouWant;
}
What you want is to use Promise.all(allSpecFilesArr) because that is an array of promises. You can await it and you will receive an array of data returned from the inner resolved promises.
const fileContents = await Promise.all(allSpecFilesArr);
I have the following method in which I read from a csv and collect the data in an array of strings. After I do that I want to use that array. I'm new to handling asynchronous calls and I think what's happening in the code below is that reading from the .csv file is asynchronous so the array is empty once I start looping through it. How do I complete the .csv reading so that all the array is filled completely and only until it is, move on to the next task of looping through the array?
static async readAndPopulateGMDevicesToMigrate() {
const bookTitles = [];
await fs.createReadStream('./BookTitles.csv')
.pipe(csv())
.on('data', (data) => bookTitles.push(data.Titles))
.on('error', (error) => loggingService.getDefaultLogger().error(error))
.on('end', () => loggingService.getDefaultLogger().info("Book Titles:" + booksTitles));
console.log(bookTitles);
const booksToAdd = [];
bookTitles.forEach(bookTitle => booksToAdd.push(new Object({
Title: bookTitle}))),
})))
console.log(readDevices);
}
Any help on this would be much appreciated!
A couple of pointers, hoping this can help with getting the approach clear.
createReadStream returns a ReadableStream, not an async/promise to await on.
The returned ReadableStream provides means to react to events, as shown when pipelining the handlers by using on.
Now, if you wrap your code in a Promise where either resolve or reject base on the end or error event respectively.
static async readAndPopulateGMDevicesToMigrate() {
const bookTitles = [];
const logger = loggingService.getDefaultLogger();
/* WRAP in promise to wait */
await new Promise((resolve, reject) => {
fs.createReadStream('./BookTitles.csv')
.pipe(csv())
.on('data', ({ Titles }) => {
logger.info(`Adding ${Titles} Titles`);
bookTitles.push(Titles);
})
.on('error', (error) => {
logger.error(error);
/* REJECT on error, maybe reject with the partial result ? */
reject(error);
})
.on('end', () => {
logger.info("Book Titles:" + booksTitles);
const booksToAdd = bookTitles.map(bookTitle => ({Title: bookTitle}));
/* RESOLVE when the stream was read to the end */
resolve(booksToAdd);
});
})
console.log(bookTitles);
console.log(readDevices);
}
All of my other functions are working fine and this function works with a limited data set, so it's not really a credentials problem. It appears that I am running into the issue describes in these answers:
https://stackoverflow.com/a/58828202/3662110
https://stackoverflow.com/a/58779000/3662110
I'm fetching a pretty large chunk of JSON, so I think the issue is similar in that it's timing out, but I'm unable to add return or await in the same way described in those answers (tried them both, but return does nothing and await causes another error) because I'm looping through the data and writing many documents. This function has worked in the past when I limit the number of results fetched from the API. How do I keep the function alive long enough to write all the data?
const functions = require('firebase-functions');
const admin = require('firebase-admin');
exports.fetchPlayers = functions.pubsub
.schedule('every Tuesday of sep,oct,nov,dec 6:00')
.timeZone('America/New_York')
.onRun(async context => {
const response = fetch(
`<<< the endpoint >>>`,
{<<< the headers >>>}
)
.then(res => {
<<< handle stuff >>>
})
.then(res => res.json());
const resObj = await response;
const players = [];
resObj.dfsEntries[0].dfsRows.forEach(x => {
<<< extract the data >>>
players.push({ id, fName, lName, pos, team, [week]: fp });
})
players.forEach(player =>
admin
.firestore()
.collection('players')
.doc(`${player.id}`)
.set(player, {merge: true})
.then(writeResult => {
// write is complete here
})
);
});
In order to correctly terminate a Cloud Function, you are required to return a promise from the function that resolves when all of the asynchronous work is complete. You can't ignore any promises. Calling then on a promise isn't enough, since then just returns another promise. The problem is that you're ignoring the promises generated inside the forEach loop. You will need to collect all those promises in an array, then use Promise.all() to create a new promise that you can return from the function. To put it briefly:
const promises = players.map(player => {
return admin.firestore....set(...)
})
return Promise.all(promises)
If you don't wait for all the promises to resolve, then the function will terminate and shut down early, before the async work is done.
So the easiest way to explain this is that I am trying to search your package.json for all your packages, then use the npm registry to see if there are new updates. I have most of that done. Accept for the following for loop:
import request from 'request'
export const findAllUpdates = (packageInfo) => {
for(prop in packageInfo) {
request('https://registry.npmjs.org/' + prop, function(err, resp) {
json = JSON.parse(resp.body);
if (json['dist-tags'].latest !== packageInfo[prop].version) {
// Do something, we are not the same version so we should
// update some counter.
}
});
}
}
Your packageInfo is a object of key=>value which represents either a dependency or a dev dependency from the package.json and the package-lock.json or the yarn.lock
The important part is we look at, in the above function, what you have installed, we then use the registry to get that package info and compare the latest version on the registry to that of what you have installed and then we want, ideally, to update the state of the component with the total count.
how ever the issue is that we have entered callback hell, especially with a for loop, looping over every package, making a request.
I cannot create a variable and store the response in there because it cannot be accessed after the request is made. I cannot see how using events here would work because you might have 40 packages installed as either dep or dev and thats a lot of events to fire.
And finally the proper solution might be to use promises, but the whole concept of promises is yet more callbacks with the .then(() => {}), .catch(() => {}) which puts me back at square one.
The goal is to call this in a component and have the state of that component be updated with the total amount of packages that have to be updated (or at least have new versions)
Any ideas? Am I going about this all wrong?
You're right to be looking at promises.
The first thing is to give yourself a promise-enabled version of request. There's a promisify function in util that takes a Node-callback-style function and returns a promise-enabled version. So:
import util from 'util';
// ...
const prequest = util.promisify(request);
(There's also an npm module, promisify, that can do entire APIs all at once.)
Then, do your requests, gathering the promises in an array (and using then on them to do any post-processing; then returns a new promise, so we're still good), and use Promise.all to wait until they all resolve (or any of them rejects);
So all together:
import request from 'request';
import util from 'util';
const prequest = util.promisify(request);
export const findAllUpdates = (packageInfo) => {
const updates = []
for (const prop in packageInfo) {
updates.push(prequest('https://registry.npmjs.org/' + prop).then(resp => {
const json = JSON.parse(resp.body);
if (json['dist-tags'].latest !== packageInfo[prop].version) {
// Do something, we are not the same version so we should
// update some counter.
}
// Perhaps return something here, it will be the resolution
// for the promise for this request; otherwise, the promise
// will resolve with `undefined`.
}));
}
return Promise.all(updates);
};
That overall function's promise will resolve to an array of the results of each promise in the array (in order), or (again) reject if any of them rejects.
I want to be able to suggest async/await, but it doesn't currently bring much to the table for a bunch of parallel promises. (There's been the occasional talk of await.all to do what Promise.all does, but it hasn't caught on [yet?].)
Having said that, though, if we break things up a bit it is a bit nicer with async/await:
import request from 'request';
import util from 'util';
const prequest = util.promisify(request);
const checkOne = async (prop) => {
const resp = await prequest('https://registry.npmjs.org/' + prop);
const json = JSON.parse(resp.body);
if (json['dist-tags'].latest !== packageInfo[prop].version) {
// Do something, we are not the same version so we should
// update some counter.
}
// Perhaps return something here, it will be the resolution
// for the promise for this request; otherwise, the promise
// will resolve with `undefined`.
};
export const findAllUpdates = (packageInfo) => {
const updates = []
for (const prop in packageInfo) {
updates.push(checkOne(prop);
}
return Promise.all(updates);
};
And of course, if all the properties in packageInfo are "own" properties (not inherited), findAllUpdates gets a lot simpler:
export const findAllUpdates = (packageInfo) => {
return Promise.all(Object.keys(packageInfo).map(checkOne));
};
Side note: I've added a few missing declarations above.
I would recommend using Promise. You can make several async calls, and then wait for them using Promise.all.
If you also return a Promise from your "findAllUpdates" method, then it is easy for the caller to just update its own state when the information is available.
In React that would be something like:
findAllUpdates(..)
.then(result => this.setState({
packagesNeedUpdate: result.filter(p => !p.latest).length
}));
I've created a simple example below that uses promises. It fakes the requests, but otherwise should be pretty accurate.
// Fake requests. Will return version '1.0.0' after one second.
const makeRequest = url => {
return new Promise(resolve => {
setTimeout(() => {
resolve(JSON.stringify({
'dist-tags': {
'latest': '1.0.0'
}
}, null, 2));
}, 1000);
});
}
const findAllUpdates = (packageInfo) => {
return new Promise(resolve => {
const promises = Object.keys(packageInfo)
.map(prop => makeRequest('https://registry.npmjs.org/' + prop).then(JSON.parse));
Promise.all(promises) // Wait for all promises
.then(jsons => {
const results = Object.keys(packageInfo)
.map((prop, i) => {
const latestInNpm = jsons[i]['dist-tags'].latest;
const current = packageInfo[prop].version
return {
prop,
latest: latestInNpm === current
}
});
resolve(results); // Return result to caller
});
});
}
function run() {
console.log('Loading..');
const packages = {
'react': {version: '0.0.1'},
'angular': {version: '1.0.0'},
'ember': {version: '0.5.0'},
'mithril': {version: '0.9.0'}
};
findAllUpdates(packages).then(result => {
const needUpdates = result.filter(p => !p.latest).length;
console.log('Need updates: ', needUpdates);
});
}
run();
I'm playing around with Rx.js and I'm looking to run two promises one that creates a directory and one that makes a http request, both take one parameter user. How would I chain these two things together?
const GITHUB_USER = 'reggi'
let repos = Rx.Observable.just(GITHUB_USER)
.map(user => {
return createDir(user)
})
.map(user => {
return getRepos(user)
})
The flatMap operator allows you to 'flatten' an Observable of Promises (ie convert a stream of Promises that resolve something to just a stream of the resolved data):
const GITHUB_USER = 'reggi'
// repos$ is a stream that will resolve the user's repositories
// once the directory is created.
let repos$ = Rx.Observable.just(GITHUB_USER)
.flatMap(user => {
// Return a Promise that resolves the user
return createDir(user).then(() => user);
})
.flatMap(user => {
// Return a Promise that resolves the repos
return getRepos(user)
});
What you could do is cheat a little:
let repos = Rx.Observable.just(GITHUB_USER)
.map(user => {
return createDir(user).then(res_cd => {
return getRepos(user);
});
})
.then(res => {...})
It's not ideal, but it'll work.