Develop Reference

Promise , Async Javascript Api [duplicate]

This question already has answers here:
async/await always returns promise
(4 answers)
Closed 12 months ago.
Ok, I think I am doing the promise wrong. I have 2 problems:
The results inside the class is working not problem. but when I retaive it at app.js it is showing as a promise.
The results inside the classrooms should be more than one row/object but it is only giving me one result
I am able to get the output I want inside the class, but in the following app.js I get this when I try to retraive the results values :
Promise {}
let results
document.getElementById('crypto').addEventListener('keyup', (e) => {
const cryptoInput = e.target.value;
console.log(cryptoInput);
if (cryptoInput.length > 2) {
results = crypto.getPrice(currency, cryptoInput)
//here I get Promise {<pending>}
console.log(results)
}
})
const table = document.getElementById('resultsTable');
let i = 0;
function showOptions(results) {
console.log(results)
}
the class is working :
class CoinGecko {
constructor() {
}
async list() {
const list1 = await fetch(`https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=250&page=1&sparkline=false`);
const list2 = await fetch(`https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=250&page=2&sparkline=false`);
const list1Json = await list1.json();
const list2Json = await list2.json();
const list = Object.assign(list1Json, list2Json);
//const list = list1Json.concat(list2Json);
return list;
//console.log(list);
}
async getPrice(currency = 'usd', crypto) {
let results=[];
const toutCrypto = await this.list().then(res => {
Object.keys(res).forEach(function (element){
if (res[element].id.startsWith(crypto) ||
res[element].name.startsWith(crypto) ||
res[element].symbol.startsWith(crypto))
{
//console.log();
results.push({
id: res[element].id,
name: res[element].name,
image: res[element].image,
symbol: res[element].symbol,
price: res[element].current_price,
date: res[element].last_updated,
})
console.log(results[0])
}
});
});
console.log(results[0].id)
return {
results
}
}
}

If 'results' in a promise then it won't execute until you trigger it by telling it how to resolve. To do this, you call Promise.prototype.then((resp) => {...})
Alternatively, you can declare your arrow functions and make them async and use await, but I would suggest that since you're new to Promises you do them the simple way and become accustomed to them. Once you understand how they work, you can use them with async/await, but since that is less intuitive you can stick to the basics.
crypto.getPrice(currency, cryptoInput)
.then(data => results = data)
.catch(err => console.error(err));
// simplified Promise chaining with catch method.

Firestore slow queries are causing the entire logic to crash

I am currently designing a todo app with react and firebase without any node js server code. Everything was fine when inserting data and authenticating, but when I tried to query all the tasks where uid equals a certain value(that means all the tasks that belongs to a particular user), somehow the query gets skipped and returns null. But then, after a second or so, the array got retrieved and printed out.
The code:
function retrieveTasksOfUser(uid) {
// get all tasks where the uid equals the given value
return db.collection("tasks").where("uid", "==", uid).get();
}
function retrieveTasks() {
let res = [];
retrieveTasksOfUser(currentUser["uid"])
.then((snapshot) => {
snapshot.forEach((doc) => {
// include the doc id in each returned object
let buffer = doc.data();
buffer["id"] = doc.id;
res.push(buffer);
});
console.log(res);
return res;
})
.catch((err) => {
console.error("Error retrieving tasks: ", err);
return res;
});
}
let tasks = retrieveTasks();
console.log({ tasks });
EDIT:
Inspired by Frank's answer, I modified my code and got:
async function retrieveTasks() {
let res = [];
const snapshot = await retrieveTasksOfUser(currentUser["uid"]);
snapshot.forEach((doc) => {
let buffer = doc.data();
buffer["id"] = doc.id;
res.push(buffer);
});
return res;
}
let tasks = [];
let promise = retrieveTasks();
promise.then((res) => {
console.log("Tasks are");
tasks = res;
console.log(res);
});
console.log(tasks);
But the result turns out to be an empty array
Thanks a bunch in advance.

You're not returning anything from the top-level code in retrieveTasks, so that means that tasks will always be undefined.
The simplest fix is to return the result of the query, which then means that your return res will be bubbled up:
return retrieveTasksOfUser(currentUser["uid"])
...
But this means you're returning a promise, as the data is loaded asynchronously. So in the calling code, you have to wait for that promise to resolve with then:
retrieveTasks().then((tasks) => {
console.log({ tasks });
})
You can make all of this a lot more readable (although it'll function exactly the same) by using the async and await keywords.
With those, the code would become:
async function retrieveTasks() {
let res = [];
const snapshot = await retrieveTasksOfUser(currentUser["uid"]);
snapshot.forEach((doc) => {
let buffer = doc.data();
buffer["id"] = doc.id;
res.push(buffer);
});
return res;
}
let tasks = await retrieveTasks();
console.log({ tasks });
Or with a bit more modern JavaScript magic, we can make it even shorter:
async function retrieveTasks() {
const snapshot = await retrieveTasksOfUser(currentUser["uid"]);
return snapshot.docs.map((doc) => { ...doc.data, id: doc.id });
}
let tasks = await retrieveTasks();
console.log({ tasks });

Unable to use async in forEach loop inside an API call

I have a get API call which looks like this
router.get('/review', async (req, res) => {
try {
const entity = await Entity.find();
const entityId = [];
Object.keys(entity).forEach((key) => {
entityId.push(entity[key]._id);
});
const results = [];
Object.keys(entityId).forEach(async (key) => {
const reviews = await Review.find({ entityId: entityId[key] });
results.push(reviews);
});
res.send(results);
} catch (e) {
res.status(500).send();
}
});
In entityId array it has a list of all the id i need and till there it works. Now what I want to do is iterate over each id of entityId and find the corresponding review that the entity has, push those review into a results array and return results.
review has an entityId field which is same as id of entity.
I also looked at - Using async/await with a forEach loop
which suggested to use for loop but it gave the following error.
iterators/generators require regenerator-runtime, which is too heavyweight for this guide to allow them. Separately, loops should be avoided in favor of array iterations.eslintno-restricted-syntax
How can I solve this?

forEach does not respect await therefore it may result in unintentional behaviour
you can use map to return array of promises from Review.find() and wrap those in await Promise.all({promises array here}) and use await on Promise.all.
This will result in parallel calls instead of doing them sequentially one after another.
const promisesWoAwait = this.entityIds.map(entityId => Review.find({ entityId }));
const result = await Promise.all(promisesWoAwait);

use promises instead of foreach.
Thy this
const data = async () => {
const entity = {
a: { _id: "1231" },
b: { _id: "1232" },
c: { _id: "1233" }
};
const entityId = [];
Object.keys(entity).forEach(key => {
entityId.push(entity[key]._id);
});
const promise = [];
Object.keys(entityId).forEach(async key => {
const reviews = Review({ entityId: entityId[key] });
promise.push(reviews);
});
const results = await Promise.all(promise);
};
const Review = (option) => {
return true;
};
data();

Promise Resolving before Google Cloud Bucket Upload

I am writing some code that loops over a CSV and creates a JSON file based on the CSV. Included in the JSON is an array named photos, which is to contain the returned urls for the images that are being uploaded to Google Cloud Storage within the function. However, having the promise wait for the uploads to finish has me stumped, since everything is running asynchronously, and finishes off the promise and the JSON compilation prior to finishing the bucket upload and returning the url. How can I make the promise resolve after the urls have been retrieved and added to currentJSON.photos?
const csv=require('csvtojson')
const fs = require('fs');
const {Storage} = require('#google-cloud/storage');
var serviceAccount = require("./my-firebase-storage-spot.json");
const testFolder = './Images/';
var csvFilePath = './Inventory.csv';
var dirArr = ['./Images/Subdirectory-A','./Images/Subdirectory-B','./Images/Subdirectory-C'];
var allData = [];
csv()
.fromFile(csvFilePath)
.subscribe((json)=>{
return new Promise((resolve,reject)=>{
for (var i in dirArr ) {
if (json['Name'] == dirArr[i]) {
var currentJSON = {
"photos" : [],
};
fs.readdir(testFolder+json['Name'], (err, files) => {
files.forEach(file => {
if (file.match(/.(jpg|jpeg|png|gif)$/i)){
var imgName = testFolder + json['Name'] + '/' + file;
bucket.upload(imgName, function (err, file) {
if (err) throw new Error(err);
//returned uploaded img address is found at file.metadata.mediaLink
currentJSON.photos.push(file.metadata.mediaLink);
});
}else {
//do nothing
}
});
});
allData.push(currentJSON);
}
}
resolve();
})
},onError,onComplete);
function onError() {
// console.log(err)
}
function onComplete() {
console.log('finito');
}
I've tried moving the resolve() around, and also tried placing the uploader section into the onComplete() function (which created new promise-based issues).

Indeed, your code is not awaiting the asynchronous invocation of the readdir callback function, nor of the bucket.upload callback function.
Asynchronous coding becomes easier when you use the promise-version of these functions.
bucket.upload will return a promise when omitting the callback function, so that is easy.
For readdir to return a promise, you need to use the fs Promise API: then you can use
the promise-based readdir method and use
promises throughout your code.
So use fs = require('fs').promises instead of fs = require('fs')
With that preparation, your code can be transformed into this:
const testFolder = './Images/';
var csvFilePath = './Inventory.csv';
var dirArr = ['./Images/Subdirectory-A','./Images/Subdirectory-B','./Images/Subdirectory-C'];
(async function () {
let arr = await csv().fromFile(csvFilePath);
arr = arr.filter(obj => dirArr.includes(obj.Name));
let allData = await Promise.all(arr.map(async obj => {
let files = await fs.readdir(testFolder + obj.Name);
files = files.filter(file => file.match(/\.(jpg|jpeg|png|gif)$/i));
let photos = await Promise.all(
files.map(async file => {
var imgName = testFolder + obj.Name + '/' + file;
let result = await bucket.upload(imgName);
return result.metadata.mediaLink;
})
);
return {photos};
}));
console.log('finito', allData);
})().catch(err => { // <-- The above async function runs immediately and returns a promise
console.log(err);
});
Some remarks:
There is a shortcoming in your regular expression. You intended to match a literal dot, but you did not escape it (fixed in above code).
allData will contain an array of { photos: [......] } objects, and I wonder why you would not want all photo elements to be part of one single array. However, I kept your logic, so the above will still produce them in these chunks. Possibly, you intended to have other properties (next to photos) as well, which would make it actually useful to have these separate objects.

The problem is the your code is not waiting in your forEach. I would highly recommend to look for stream and try to do things in parallel as much as possible. There is one library which is very powerful and does that job for you. The library is etl.
You can read rows from csv in parallel and process them in parallel rather than one by one.
I have tried to explain the lines in the code below. Hopefully it makes sense.
const etl = require("etl");
const fs = require("fs");
const csvFilePath = `${__dirname }/Inventory.csv`;
const testFolder = "./Images/";
const dirArr = [
"./Images/Subdirectory-A",
"./Images/Subdirectory-B",
"./Images/Subdirectory-C"
];
fs.createReadStream(csvFilePath)
.pipe(etl.csv()) // parse the csv file
.pipe(etl.collect(10)) // this could be any value depending on how many you want to do in parallel.
.pipe(etl.map(async items => {
return Promise.all(items.map(async item => { // Iterate through 10 items
const finalResult = await Promise.all(dirArr.filter(i => i === item.Name).map(async () => { // filter the matching one and iterate
const files = await fs.promises.readdir(testFolder + item.Name); // read all files
const filteredFiles = files.filter(file => file.match(/\.(jpg|jpeg|png|gif)$/i)); // filter out only images
const result = await Promise.all(filteredFiles).map(async file => {
const imgName = `${testFolder}${item.Name}/${file}`;
const bucketUploadResult = await bucket.upload(imgName); // upload image
return bucketUploadResult.metadata.mediaLink;
});
return result; // This contains all the media link for matching files
}));
// eslint-disable-next-line no-console
console.log(finalResult); // Return arrays of media links for files
return finalResult;
}));
}))
.promise()
.then(() => console.log("finsihed"))
.catch(err => console.error(err));

Here's a way to do it where we extract some of the functionality into some separate helper methods, and trim down some of the code. I had to infer some of your requirements, but this seems to match up pretty closely with how I understood the intent of your original code:
const csv=require('csvtojson')
const fs = require('fs');
const {Storage} = require('#google-cloud/storage');
var serviceAccount = require("./my-firebase-storage-spot.json");
const testFolder = './Images/';
var csvFilePath = './Inventory.csv';
var dirArr = ['./Images/Subdirectory-A','./Images/Subdirectory-B','./Images/Subdirectory-C'];
var allData = [];
// Using nodejs 'path' module ensures more reliable construction of file paths than string manipulation:
const path = require('path');
// Helper function to convert bucket.upload into a Promise
// From other responses, it looks like if you just omit the callback then it will be a Promise
const bucketUpload_p = fileName => new Promise((resolve, reject) => {
bucket.upload(fileName, function (err, file) {
if (err) reject(err);
resolve(file);
});
});
// Helper function to convert readdir into a Promise
// Again, there are other APIs out there to do this, but this is a rl simple solution too:
const readdir_p = dirName => new Promise((resolve, reject) => {
fs.readdir(dirName, function (err, files) {
if (err) reject(err);
resolve(files);
});
});
// Here we're expecting the string that we found in the "Name" property of our JSON from "subscribe".
// It should match one of the strings in `dirArr`, but this function's job ISN'T to check for that,
// we just trust that the code already found the right one.
const getImageFilesFromJson_p = jsonName => new Promise((resolve, reject) => {
const filePath = path.join(testFolder, jsonName);
try {
const files = await readdir_p(filePath);
resolve(files.filter(fileName => fileName.match(/\.(jpg|jpeg|png|gif)$/i)));
} catch (err) {
reject(err);
}
});
csv()
.fromFile(csvFilePath)
.subscribe(async json => {
// Here we appear to be validating that the "Name" prop from the received JSON matches one of the paths that
// we're expecting...? If that's the case, this is a slightly more semantic way to do it.
const nameFromJson = dirArr.find(dirName => json['Name'] === dirName);
// If we don't find that it matches one of our expecteds, we'll reject the promise.
if (!nameFromJson) {
// We can do whatever we want though in this case, I think it's maybe not necessarily an error:
// return Promise.resolve([]);
return Promise.reject('Did not receive a matching value in the Name property from \'.subscribe\'');
}
// We can use `await` here since `getImageFilesFromJson_p` returns a Promise
const imageFiles = await getImageFilesFromJson_p(nameFromJson);
// We're getting just the filenames; map them to build the full path
const fullPathArray = imageFiles.map(fileName => path.join(testFolder, nameFromJson, fileName));
// Here we Promise.all, using `.map` to convert the array of strings into an array of Promises;
// if they all resolve, we'll get the array of file objects returned from each invocation of `bucket.upload`
return Promise.all(fullPathArray.map(filePath => bucketUpload_p(filePath)))
.then(fileResults => {
// So, now we've finished our two asynchronous functions; now that that's done let's do all our data
// manipulation and resolve this promise
// Here we just extract the metadata property we want
const fileResultsMediaLinks = fileResults.map(file => file.metadata.mediaLink);
// Before we return anything, we'll add it to the global array in the format from the original code
allData.push({ photos: fileResultsMediaLinks });
// Returning this array, which is the `mediaLink` value from the metadata of each of the uploaded files.
return fileResultsMediaLinks;
})
}, onError, onComplete);

You are looking for this library ELT.
You can read rows from CSV in parallel and process them in parallel rather than one by one.
I have tried to explain the lines in the code below. Hopefully, it makes sense.
const etl = require("etl");
const fs = require("fs");
const csvFilePath = `${__dirname }/Inventory.csv`;
const testFolder = "./Images/";
const dirArr = [
"./Images/Subdirectory-A",
"./Images/Subdirectory-B",
"./Images/Subdirectory-C"
];
fs.createReadStream(csvFilePath)
.pipe(etl.csv()) // parse the csv file
.pipe(etl.collect(10)) // this could be any value depending on how many you want to do in parallel.
.pipe(etl.map(async items => {
return Promise.all(items.map(async item => { // Iterate through 10 items
const finalResult = await Promise.all(dirArr.filter(i => i === item.Name).map(async () => { // filter the matching one and iterate
const files = await fs.promises.readdir(testFolder + item.Name); // read all files
const filteredFiles = files.filter(file => file.match(/\.(jpg|jpeg|png|gif)$/i)); // filter out only images
const result = await Promise.all(filteredFiles).map(async file => {
const imgName = `${testFolder}${item.Name}/${file}`;
const bucketUploadResult = await bucket.upload(imgName); // upload image
return bucketUploadResult.metadata.mediaLink;
});
return result; // This contains all the media link for matching files
}));
// eslint-disable-next-line no-console
console.log(finalResult); // Return arrays of media links for files
return finalResult;
}));
}))
.promise()
.then(() => console.log("finsihed"))
.catch(err => console.error(err));

properly using async and await

The function below calls several asynchronous functions in a for loop. It's parsing different CSV files to build a single JavaScript object. I'd like to return the object after the for loop is done. Its returning the empty object right away while it does the asynchronous tasks. Makes sense, however I have tried various Promise / async /await combinations hopes of running something once the for loop has completed. I am clearly not understanding what is going on. Is there a better pattern to follow for something like this or am I thinking about it incorrectly?
async function createFormConfig(files: string[]): Promise<object>
return new Promise(resolve => {
const retConfig: any = {};
for (const file of files) {
file.match(matchFilesForFormConfigMap.get('FIELD')) ?
parseCsv(file).then(parsedData => {
retConfig.fields = parsedData.data;
})
: file.match(matchFilesForFormConfigMap.get('FORM'))
? parseCsv(file).then(parsedData => retConfig.formProperties = parsedData.data[0])
: file.match(matchFilesForFormConfigMap.get('PDF'))
? parseCsv(file).then(parsedData => retConfig.jsPdfProperties = parsedData.data[0])
: file.match(matchFilesForFormConfigMap.get('META'))
? parseCsv(file).then(parsedData => {
retConfig.name = parsedData.data[0].name;
retConfig.imgType = parsedData.data[0].imgType;
// console.log(retConfig); <- THIS CONSOLE WILL OUTPUT RETCONFIG LOOKING LIKE I WANT IT
})
: file.match(matchFilesForFormConfigMap.get('PAGES'))
? parseCsv(file).then(parsedData => retConfig.pages = parsedData.data)
: console.log('there is an extra file: ' + file);
}
resolve(retConfig); // <- THIS RETURNS: {}
});
This is the code I'm using to call the function in hopes of getting my 'retConfig' filled with the CSV data.
getFilesFromDirectory(`${clOptions.directory}/**/*.csv`)
.then(async (files) => {
const config = await createFormConfig(files);
console.log(config);
})
.catch(err => console.error(err));
};

First, an async function returns a Promise, so you dont have to return one explicitely.Here is how you can simplify your code:
async function createFormConfig(files: string[]): Promise<object> {
// return new Promise(resolve => { <-- remove
const retConfig: any = {};
// ...
// The value returned by an async function is the one you get
// in the callback passed to the function `.then`
return retConfig;
// }); <-- remove
}
Then, your function createFormConfig returns the config before it has finished to compute it. Here is how you can have it computed before returning it:
async function createFormConfig(files: string[]): Promise<object> {
const retConfig: any = {};
// Return a Promise for each file that have to be parsed
const parsingCsv = files.map(async file => {
if (file.match(matchFilesForFormConfigMap.get('FIELD'))) {
const { data } = await parseCsv(file);
retConfig.fields = data;
} else if (file.match(matchFilesForFormConfigMap.get('FORM'))) {
const { data } = await parseCsv(file);
retConfig.formProperties = data[0];
} else if (file.match(matchFilesForFormConfigMap.get('PDF'))) {
const { data } = await parseCsv(file);
retConfig.jsPdfProperties = data[0];
} else if (file.match(matchFilesForFormConfigMap.get('META'))) {
const { data } = await parseCsv(file);
retConfig.name = data[0].name;
retConfig.imgType = data[0].imgType;
} else if (file.match(matchFilesForFormConfigMap.get('PAGES'))) {
const { data } = await parseCsv(file);
retConfig.pages = data;
} else {
console.log('there is an extra file: ' + file);
}
});
// Wait for the Promises to resolve
await Promise.all(parsingCsv)
return retConfig;
}

async functions already return promises, you don't need to wrap the code in a new one. Just return a value from the function and the caller will receive a promise that resolves to the returned value.
Also, you have made an async function, but you're not actually using await anywhere. So the for loop runs through the whole loop before any of your promises resolve. This is why none of the data is making it into your object.
It will really simplify your code to only use await and get rid of the then() calls. For example you can do this:
async function createFormConfig(files: string[]): Promise<object> {
const retConfig: any = {};
for (const file of files) {
if (file.match(matchFilesForFormConfigMap.get('FIELD')){
// no need for the then here
let parsedData = await parseCsv(file)
retConfig.field = parsedData.data
}
// ...etc
At the end you can just return the value:
return retConfig