I'm looping an array and reading from a CSV file so I can reduce its data, I've implemented something like this. I want to have a Map variable populated from a csv file in memory and get data from it. But I want to populate it once (or when I explicitly do so, but that's other thing).
when running I would like to print like this:
before
after
But I'm getting this:
before
before
before
after
after
after
Apparently when reading from the CSV is not awaiting. Is there any way I can read from CSV asynchronous?
This is my code:
let myMap = new Map();
const parseMap = async(stream) => {
const map = new Map();
return new Promise((resolve, reject) => {
stream
.on('data', elems => {
const key = elems[0];
const value = elems.slice(-1)[0];
map.set(key, value);
})
.on('error', () => reject)
.on('end', () => {
console.log('promise');
resolve(map);
});
});
}
const getMap = async(path_string) => {
const stream = fs.createReadStream(path_string).pipe(parse({delimiter: DELIMITER}));
return await parseMap(stream);
}
const translate = async (key) => {
if (!myMap.size) {
console.log('before');
myMap = await getMap('my-csv.csv');
console.log('after');
}
return myMap.get(key);
};
const guidRankings = await ['a', 'b', 'c'].reduce(async (accumulator, value, idx) => {
const data = await translate(value);
if (data) {
(await accumulator).push({
data,
order: idx
});
}
return accumulator;
}, Promise.resolve([]));
The reduce method can't handle an ascynrous callback.
You need to pass it a function that returns the actual value you care about, not one that returns a promise.
map your array values through translate to get an array of promises
Use await Promise.all to wait for them all to settle and get an array of the translated values
reduce that array (without doing anything asynchronous inside the reducer).
Assuming that the 3 calls can be made at same time it just using map to store the promises, using Promise All to wait for them to complete, and then a map to get your array.
const promises = ['a', 'b', 'c'].map(value) => translate(value));
const results = await Promise.all(promises);
const guidRankings = results.map((data, order) => ({ data, order }));
If for some reason they have to be made one at a time you want to use for await of
Related
I know there is a lot of information on this and I have read several articles, but I am still wandering.
I have an array of HTTP request urls. The order of each element in the array is very important. I want to get a response by fetching these request urls, and put the responses in the same order as the original array.
Below is the code I wrote. I thought it was right to use promise.all. We found promise.all to process arrays in parallel, but ordering is not guaranteed. (Is this correct?)
const getAllImagePaths = async urls => {
let copyPaths = [];
try {
const data = await Promise.all(urls.map(url => fetch(url)));
for (let item of data) {
const { url } = item;
copyPaths.push(url);
}
return copyPaths;
} catch (err) {
const { response } = err;
if (response) alert(MESSAGES.GET_PHOTOS_FAIL);
}
};
So I modified the code by using the for..of statement.
const getAllImagePaths = async urls => {
let copyPaths = [];
try {
for(let url of urls) {
const res = await fetch(url);
const json = await res.json();
const data = json.parse();
const { url } = data;
copyPaths.push(data);
}
return copyPaths;
} catch (err) {
const { response } = err;
if (response) alert(MESSAGES.GET_PHOTOS_FAIL);
}
};
However, when the for of statement is used, the response type of the data is 'cors'.
So to fix this, should I change the data to json and then parse the json again to use the data? I think this is very weird. This is because I only want to contain information called 'url' in the response object in the array.
We found promise.all to process arrays in parallel, but ordering is not guaranteed. (Is this correct?)
No, it isn't correct. The result array you get on fulfillment of the Promise.all promise is in the same order as the input array, regardless of the order in which the input promises settled. Here's an example:
function delayedValue(ms, value) {
return new Promise(resolve => {
setTimeout(() => {
console.log(`Resolving with ${value}`);
resolve(value);
}, ms);
});
}
async function example() {
const results = await Promise.all([
delayedValue(100, 1),
delayedValue(10, 2),
delayedValue(200, 3),
]);
console.log(`results: ${results}`);
}
example();
So you can just use the result directly:
const getAllImagePaths = async urls => {
try {
const data = await Promise.all(urls.map(url => fetch(url)));
return data;
} catch (err) {
const { response } = err;
if (response) alert(MESSAGES.GET_PHOTOS_FAIL);
}
};
I am writing some code that loops over a CSV and creates a JSON file based on the CSV. Included in the JSON is an array named photos, which is to contain the returned urls for the images that are being uploaded to Google Cloud Storage within the function. However, having the promise wait for the uploads to finish has me stumped, since everything is running asynchronously, and finishes off the promise and the JSON compilation prior to finishing the bucket upload and returning the url. How can I make the promise resolve after the urls have been retrieved and added to currentJSON.photos?
const csv=require('csvtojson')
const fs = require('fs');
const {Storage} = require('#google-cloud/storage');
var serviceAccount = require("./my-firebase-storage-spot.json");
const testFolder = './Images/';
var csvFilePath = './Inventory.csv';
var dirArr = ['./Images/Subdirectory-A','./Images/Subdirectory-B','./Images/Subdirectory-C'];
var allData = [];
csv()
.fromFile(csvFilePath)
.subscribe((json)=>{
return new Promise((resolve,reject)=>{
for (var i in dirArr ) {
if (json['Name'] == dirArr[i]) {
var currentJSON = {
"photos" : [],
};
fs.readdir(testFolder+json['Name'], (err, files) => {
files.forEach(file => {
if (file.match(/.(jpg|jpeg|png|gif)$/i)){
var imgName = testFolder + json['Name'] + '/' + file;
bucket.upload(imgName, function (err, file) {
if (err) throw new Error(err);
//returned uploaded img address is found at file.metadata.mediaLink
currentJSON.photos.push(file.metadata.mediaLink);
});
}else {
//do nothing
}
});
});
allData.push(currentJSON);
}
}
resolve();
})
},onError,onComplete);
function onError() {
// console.log(err)
}
function onComplete() {
console.log('finito');
}
I've tried moving the resolve() around, and also tried placing the uploader section into the onComplete() function (which created new promise-based issues).
Indeed, your code is not awaiting the asynchronous invocation of the readdir callback function, nor of the bucket.upload callback function.
Asynchronous coding becomes easier when you use the promise-version of these functions.
bucket.upload will return a promise when omitting the callback function, so that is easy.
For readdir to return a promise, you need to use the fs Promise API: then you can use
the promise-based readdir method and use
promises throughout your code.
So use fs = require('fs').promises instead of fs = require('fs')
With that preparation, your code can be transformed into this:
const testFolder = './Images/';
var csvFilePath = './Inventory.csv';
var dirArr = ['./Images/Subdirectory-A','./Images/Subdirectory-B','./Images/Subdirectory-C'];
(async function () {
let arr = await csv().fromFile(csvFilePath);
arr = arr.filter(obj => dirArr.includes(obj.Name));
let allData = await Promise.all(arr.map(async obj => {
let files = await fs.readdir(testFolder + obj.Name);
files = files.filter(file => file.match(/\.(jpg|jpeg|png|gif)$/i));
let photos = await Promise.all(
files.map(async file => {
var imgName = testFolder + obj.Name + '/' + file;
let result = await bucket.upload(imgName);
return result.metadata.mediaLink;
})
);
return {photos};
}));
console.log('finito', allData);
})().catch(err => { // <-- The above async function runs immediately and returns a promise
console.log(err);
});
Some remarks:
There is a shortcoming in your regular expression. You intended to match a literal dot, but you did not escape it (fixed in above code).
allData will contain an array of { photos: [......] } objects, and I wonder why you would not want all photo elements to be part of one single array. However, I kept your logic, so the above will still produce them in these chunks. Possibly, you intended to have other properties (next to photos) as well, which would make it actually useful to have these separate objects.
The problem is the your code is not waiting in your forEach. I would highly recommend to look for stream and try to do things in parallel as much as possible. There is one library which is very powerful and does that job for you. The library is etl.
You can read rows from csv in parallel and process them in parallel rather than one by one.
I have tried to explain the lines in the code below. Hopefully it makes sense.
const etl = require("etl");
const fs = require("fs");
const csvFilePath = `${__dirname }/Inventory.csv`;
const testFolder = "./Images/";
const dirArr = [
"./Images/Subdirectory-A",
"./Images/Subdirectory-B",
"./Images/Subdirectory-C"
];
fs.createReadStream(csvFilePath)
.pipe(etl.csv()) // parse the csv file
.pipe(etl.collect(10)) // this could be any value depending on how many you want to do in parallel.
.pipe(etl.map(async items => {
return Promise.all(items.map(async item => { // Iterate through 10 items
const finalResult = await Promise.all(dirArr.filter(i => i === item.Name).map(async () => { // filter the matching one and iterate
const files = await fs.promises.readdir(testFolder + item.Name); // read all files
const filteredFiles = files.filter(file => file.match(/\.(jpg|jpeg|png|gif)$/i)); // filter out only images
const result = await Promise.all(filteredFiles).map(async file => {
const imgName = `${testFolder}${item.Name}/${file}`;
const bucketUploadResult = await bucket.upload(imgName); // upload image
return bucketUploadResult.metadata.mediaLink;
});
return result; // This contains all the media link for matching files
}));
// eslint-disable-next-line no-console
console.log(finalResult); // Return arrays of media links for files
return finalResult;
}));
}))
.promise()
.then(() => console.log("finsihed"))
.catch(err => console.error(err));
Here's a way to do it where we extract some of the functionality into some separate helper methods, and trim down some of the code. I had to infer some of your requirements, but this seems to match up pretty closely with how I understood the intent of your original code:
const csv=require('csvtojson')
const fs = require('fs');
const {Storage} = require('#google-cloud/storage');
var serviceAccount = require("./my-firebase-storage-spot.json");
const testFolder = './Images/';
var csvFilePath = './Inventory.csv';
var dirArr = ['./Images/Subdirectory-A','./Images/Subdirectory-B','./Images/Subdirectory-C'];
var allData = [];
// Using nodejs 'path' module ensures more reliable construction of file paths than string manipulation:
const path = require('path');
// Helper function to convert bucket.upload into a Promise
// From other responses, it looks like if you just omit the callback then it will be a Promise
const bucketUpload_p = fileName => new Promise((resolve, reject) => {
bucket.upload(fileName, function (err, file) {
if (err) reject(err);
resolve(file);
});
});
// Helper function to convert readdir into a Promise
// Again, there are other APIs out there to do this, but this is a rl simple solution too:
const readdir_p = dirName => new Promise((resolve, reject) => {
fs.readdir(dirName, function (err, files) {
if (err) reject(err);
resolve(files);
});
});
// Here we're expecting the string that we found in the "Name" property of our JSON from "subscribe".
// It should match one of the strings in `dirArr`, but this function's job ISN'T to check for that,
// we just trust that the code already found the right one.
const getImageFilesFromJson_p = jsonName => new Promise((resolve, reject) => {
const filePath = path.join(testFolder, jsonName);
try {
const files = await readdir_p(filePath);
resolve(files.filter(fileName => fileName.match(/\.(jpg|jpeg|png|gif)$/i)));
} catch (err) {
reject(err);
}
});
csv()
.fromFile(csvFilePath)
.subscribe(async json => {
// Here we appear to be validating that the "Name" prop from the received JSON matches one of the paths that
// we're expecting...? If that's the case, this is a slightly more semantic way to do it.
const nameFromJson = dirArr.find(dirName => json['Name'] === dirName);
// If we don't find that it matches one of our expecteds, we'll reject the promise.
if (!nameFromJson) {
// We can do whatever we want though in this case, I think it's maybe not necessarily an error:
// return Promise.resolve([]);
return Promise.reject('Did not receive a matching value in the Name property from \'.subscribe\'');
}
// We can use `await` here since `getImageFilesFromJson_p` returns a Promise
const imageFiles = await getImageFilesFromJson_p(nameFromJson);
// We're getting just the filenames; map them to build the full path
const fullPathArray = imageFiles.map(fileName => path.join(testFolder, nameFromJson, fileName));
// Here we Promise.all, using `.map` to convert the array of strings into an array of Promises;
// if they all resolve, we'll get the array of file objects returned from each invocation of `bucket.upload`
return Promise.all(fullPathArray.map(filePath => bucketUpload_p(filePath)))
.then(fileResults => {
// So, now we've finished our two asynchronous functions; now that that's done let's do all our data
// manipulation and resolve this promise
// Here we just extract the metadata property we want
const fileResultsMediaLinks = fileResults.map(file => file.metadata.mediaLink);
// Before we return anything, we'll add it to the global array in the format from the original code
allData.push({ photos: fileResultsMediaLinks });
// Returning this array, which is the `mediaLink` value from the metadata of each of the uploaded files.
return fileResultsMediaLinks;
})
}, onError, onComplete);
You are looking for this library ELT.
You can read rows from CSV in parallel and process them in parallel rather than one by one.
I have tried to explain the lines in the code below. Hopefully, it makes sense.
const etl = require("etl");
const fs = require("fs");
const csvFilePath = `${__dirname }/Inventory.csv`;
const testFolder = "./Images/";
const dirArr = [
"./Images/Subdirectory-A",
"./Images/Subdirectory-B",
"./Images/Subdirectory-C"
];
fs.createReadStream(csvFilePath)
.pipe(etl.csv()) // parse the csv file
.pipe(etl.collect(10)) // this could be any value depending on how many you want to do in parallel.
.pipe(etl.map(async items => {
return Promise.all(items.map(async item => { // Iterate through 10 items
const finalResult = await Promise.all(dirArr.filter(i => i === item.Name).map(async () => { // filter the matching one and iterate
const files = await fs.promises.readdir(testFolder + item.Name); // read all files
const filteredFiles = files.filter(file => file.match(/\.(jpg|jpeg|png|gif)$/i)); // filter out only images
const result = await Promise.all(filteredFiles).map(async file => {
const imgName = `${testFolder}${item.Name}/${file}`;
const bucketUploadResult = await bucket.upload(imgName); // upload image
return bucketUploadResult.metadata.mediaLink;
});
return result; // This contains all the media link for matching files
}));
// eslint-disable-next-line no-console
console.log(finalResult); // Return arrays of media links for files
return finalResult;
}));
}))
.promise()
.then(() => console.log("finsihed"))
.catch(err => console.error(err));
I know there are similar questions, but I can't find the answer.
First, please tell me if I'm doing something really wrong
I need to populate my state with data from an API call. This was working fine with code above:
export const GetPlanets = async () => {
const planets = await axios.get(`${BASE_URL}`).catch((e) => {
console.error(e);
})
return planets.data.results
}
But then I needed to make a second call to several links from one json response filed, and I managed to make it work (don't know if it is the correct approach, though)
const GetPlanets = async () => {
let planetas = {}
await axios.get(`${PLANETS_URL}`)
.then((p) => {
planetas = p.data.results
return axios.get(`${FILMS_URL}`)
}).then((f) => {
planetas.films.forEach((v, i) => {
planetas[i].film = f
})
})
})
.catch((e) => {
console.error(e);
})
return planetas
}
This is my component file, where I try to get the object, like I was doing before
useEffect(() => {
const fetchPlanetas = async () => { // ME TRYING...
const planetas = await GetPlanets()
setPlanetas(planetas)
setToShow(planetas[0])
};
fetchPlanetas()
}, [])
But all I get is undefined
You're getting an array of undefined because .map() needs a return value. In both your .map() callbacks, you are not returning anything.
const results = [1, 2, 3, 4]
const results2 = results.map(elem => {
elem = elem + 1
})
console.log(results2)
But, even if you did return something in your .map() callback, GetFilms(f) is asynchronous, so you would not get the results of GetFilms() mapped into the array as you would expect.
You have a couple of options:
If you have access to the API, send the films data along with the rest of the data when you do your first request.
Use async/await and Promise.all() to get responses.
A bit of an ambiguous title, but I'll explain...
I am making a fetch call to 4 different URLs from The Movie Database.
Once these fetch calls retrieve the data, it will then setState and update my initial state. However, I don't want my page to load until all of the data is retrieved, so I am using Promise.all (or attempting to).
My code so far...
state = {
movies: {
trending: {},
topRated: {},
nowPlaying: {},
upcoming: {},
},
};
const allMovieURLs = {
trending: `https://api.themoviedb.org/3/trending/all/day?api_key=${API_KEY}`,
topRated: `https://api.themoviedb.org/3/movie/top_rated?api_key=${API_KEY}&language=en-US&page=1`,
nowPlaying: `https://api.themoviedb.org/3/movie/now_playing?api_key=${API_KEY}&language=en-US&page=1`,
upcoming: `https://api.themoviedb.org/3/movie/upcoming?api_key=${API_KEY}&language=en-US&page=1`
};
//initialize an empty array to Promise.all on
const promiseArr = [];
//loop through movie URLs, call fetch + json()
for (const movies in allMovieURLs) {
//create an object that directly describes the data you get back from the url with a key
const obj = {
[movies]: fetch(allMovieURLs[movies]).then(res => res.json())
};
//push that object into an array so you can use Promise.all
promiseArr.push(obj);
Now what I have here is an array (promiseArr) of objects that have the correct key with the correct Promise stored inside of them.
My next plan was going to be to call Promise.all on them, but I need an array of promises. After I get the actual data back from the URL calls, I wanted to store them back in the object, to the correct corresponding key?
I've been stuck at this part for a couple hours now...any tips would be appreciated!
You can use async/await and Object.entries() to convert a JavaScript plain object to an array of arrays of key, value pairs
(async() => {
for (const [movie, url] of Object.entries(allMovieURLs)) {
try {
allMovieURLs[movie] = await fetch(url).then(res => res.json())
.catch(e => {throw e})
} catch(e) {
console.error(e)
}
}
})()
Don't call then when adding to promise array. Instead call it in promise all
const trending = fetch(trending);
...
Promise.all([trending, topRated, nowplaying, upcoming]).then(([trending, topRated, nowplaying, upcoming]) => {
movies.trending = trending.json();
....
});
You could just also store the promises themselves in another array.
function fetch() {
var time = Math.random() * 1E3;
return new Promise(function (res, rej) {
setTimeout(function () {
res(time);
}, time);
});
}
const API_KEY = "";
const allMovieURLs = {
trending: `https://api.themoviedb.org/3/trending/all/day?api_key=${API_KEY}`,
topRated: `https://api.themoviedb.org/3/movie/top_rated?api_key=${API_KEY}&language=en-US&page=1`,
nowPlaying: `https://api.themoviedb.org/3/movie/now_playing?api_key=${API_KEY}&language=en-US&page=1`,
upcoming: `https://api.themoviedb.org/3/movie/upcoming?api_key=${API_KEY}&language=en-US&page=1`
};
const promiseArr = [];
const promises = [];
for (const movies in allMovieURLs) {
const obj = { [movies]: undefined };
promises.push(fetch(allMovieURLs[movies]).then(res => obj[movies] = res));
promiseArr.push(obj);
}
Promise.all(promises).then(function () {
console.log(promiseArr);
});
If that is not feasible, you can do something like: Promise.all(promiseArr.map(obj => Object.values(obj)[0]))
Background
I am trying to filter an array of objects. Before I filter, I need to convert them to some format, and this operation is asynchronous.
const convert = () => new Promise( resolve => {
setTimeout( resolve, 1000 );
});
So, my first try was to do something like the following using async/await:
const objs = [ { id: 1, data: "hello" }, { id: 2, data: "world"} ];
objs.filter( async ( obj ) => {
await convert();
return obj.data === "hello";
});
Now, as some of you may know, Array.protoype.filter is a function which callback must return either true or false. filter is synchronous. In the previous example, I am returning none of them, I return a Promise ( all async functions are Promises ).
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
So as one can assume, the code before doesn't really work... That assumption is correct.
Problem
To make filter work with an async function, I checked stackoverflow and found this topic:
Filtering an array with a function that returns a promise
Unfortunately, the chosen answer is overly complex and uses classes. This won't do for me. I am instead looking for a more simple solution, using simple functions with a functional approach.
There is one solution at the very end, using a map with a callback to simulate a filter:
https://stackoverflow.com/a/46842181/1337392
But I was hoping to fix my filter function, not to replace it.
Questions
Is there a way to have an async function inside a filter?
If not, what is the simplest replacement I can do?
There is no way to use filter with an async function (at least that I know of).
The simplest way that you have to use filter with a collection of promises is to use Promise.all and then apply the function to your collection of results.
It would look something like this:
const results = await Promise.all(your_promises)
const filtered_results = results.filter(res => //do your filtering here)
Hope it helps.
Adapted from the article How to use async functions with Array.filter in Javascript by Tamás Sallai, you basically have 2 steps:
One that creates the conditions for an object to pass
One that receives the objects and returns true or false according to conditions
Here's an example
const arr = [1, 2, 3, 4, 5];
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
const asyncFilter = async (arr, predicate) => {
const results = await Promise.all(arr.map(predicate));
return arr.filter((_v, index) => results[index]);
}
const asyncRes = await asyncFilter(arr, async (i) => {
await sleep(10);
return i % 2 === 0;
});
console.log(asyncRes);
// 2,4
Use Scramjet fromArray/toArray methods...
const result = await scramjet.fromArray(arr)
.filter(async (item) => somePromiseReturningMethod(item))
.toArray();
as simple as that - here's a ready example to copy/paste:
const scramjet = require('../../');
async function myAsyncFilterFunc(data) {
return new Promise(res => {
process.nextTick(res.bind(null, data % 2));
});
}
async function x() {
const x = await scramjet.fromArray([1,2,3,4,5])
.filter(async (item) => myAsyncFilterFunc(item))
.toArray();
return x;
}
x().then(
(out) => console.log(out),
(err) => (console.error(err), process.exit(3)) // eslint-disable-line
);
Disclamer: I am the author of scramjet. :)
Build a parallel array to your array which you want to call filter on.
Await all of the promises from your filter func, in my eg, isValid.
In the callback in filter, use the 2nd arg, index, to index into your parallel array to determine if it should be filtered.
// ===============================================
// common
// ===============================================
const isValid = async (value) => value >= 0.5;
const values = [0.2, 0.3, 0.4, 0.5, 0.6];
// ===============================================
// won't filter anything
// ===============================================
const filtered = values.filter(async v => await isValid(v));
console.log(JSON.stringify(filtered));
// ===============================================
// filters
// ===============================================
(async () => {
const shouldFilter = await Promise.all(values.map(isValid));
const filtered2 = values.filter((value, index) => shouldFilter[index]);
console.log(JSON.stringify(filtered2));
})();
This behavior makes sense since any Promise instance has a truthy value, but it's not intuitive at a glance.
This answer uses library iter-ops, which handles iterable objects, and supports async filtering:
import {pipeAsync, filter, toAsync} from 'iter-ops';
// your input data:
const objs = [{id: 1, data: 'hello'}, {id: 2, data: 'world'}];
const i = pipeAsync(
objs,
filter(async value => {
await convert(); // any async function
return value.data === 'hello'; // filtering logic
})
);
for await(const a of i) {
console.log(a); // filtered data
}
P.S. I'm the author of iter-ops.
Reduce method can mimic filter and can operate with promises.
const isPositiveNumberAsync = async (number) => number >= 0;
const filterPositiveNumbersAsync = async (numbers) => numbers?.reduce(async (accumulatorPromise, number) => {
const accumulator = await accumulatorPromise;
if (await isPositiveNumberAsync(number)) {
return [...accumulator, number];
}
return accumulator;
}, Promise.resolve([])) || [];
(async () => {
// no numbers argument provided
console.log(await filterPositiveNumbersAsync());
// an empty argument list provided
console.log(await filterPositiveNumbersAsync([]));
// ok, but no positive numbers provided
console.log(await filterPositiveNumbersAsync([-1,-2,-3]));
// ok, positive numbers filtered.
console.log(await filterPositiveNumbersAsync([0,1,-1,-3,2,-2]));
})();
Array.prototype.asyncFilter =function( filterFn) {
const arr = this;
return new Promise(function(resolve){
const booleanArr = [];
arr.forEach(function (e) {
booleanArr.push(filterFn(e))
})
Promise.all(booleanArr).then(function (booleanArr) {
const arr2 = arr.filter(function (e, i) {
return booleanArr[i]
})
resolve(arr2)
})
})
}
/** use it like this**/
const arr=[1,2,3]
arr.asyncFilter(async e=>{}).then(...)
You can use Promise.filter from Bluebird that works similarly to Array.filter but it supports async & await.
Add asyncFilter as an extension to Array:
#available(macOS 10.15.0, *)
extension Array where Element: Any {
public func asyncFilter(closure: (Element) async -> Bool) async -> Array {
var result = [Element]()
for item in self {
if await closure(item) {
result.append(item)
}
}
return result
}
}
Usage:
result = await result.asyncFilter { item in
if <item match> {
return true
}
}