Develop Reference

Asynchronous processing of arrays (Promise.all or for..of)

I know there is a lot of information on this and I have read several articles, but I am still wandering.
I have an array of HTTP request urls. The order of each element in the array is very important. I want to get a response by fetching these request urls, and put the responses in the same order as the original array.
Below is the code I wrote. I thought it was right to use promise.all. We found promise.all to process arrays in parallel, but ordering is not guaranteed. (Is this correct?)
const getAllImagePaths = async urls => {
let copyPaths = [];
try {
const data = await Promise.all(urls.map(url => fetch(url)));
for (let item of data) {
const { url } = item;
copyPaths.push(url);
}
return copyPaths;
} catch (err) {
const { response } = err;
if (response) alert(MESSAGES.GET_PHOTOS_FAIL);
}
};
So I modified the code by using the for..of statement.
const getAllImagePaths = async urls => {
let copyPaths = [];
try {
for(let url of urls) {
const res = await fetch(url);
const json = await res.json();
const data = json.parse();
const { url } = data;
copyPaths.push(data);
}
return copyPaths;
} catch (err) {
const { response } = err;
if (response) alert(MESSAGES.GET_PHOTOS_FAIL);
}
};
However, when the for of statement is used, the response type of the data is 'cors'.
So to fix this, should I change the data to json and then parse the json again to use the data? I think this is very weird. This is because I only want to contain information called 'url' in the response object in the array.

We found promise.all to process arrays in parallel, but ordering is not guaranteed. (Is this correct?)
No, it isn't correct. The result array you get on fulfillment of the Promise.all promise is in the same order as the input array, regardless of the order in which the input promises settled. Here's an example:
function delayedValue(ms, value) {
return new Promise(resolve => {
setTimeout(() => {
console.log(`Resolving with ${value}`);
resolve(value);
}, ms);
});
}
async function example() {
const results = await Promise.all([
delayedValue(100, 1),
delayedValue(10, 2),
delayedValue(200, 3),
]);
console.log(`results: ${results}`);
}
example();
So you can just use the result directly:
const getAllImagePaths = async urls => {
try {
const data = await Promise.all(urls.map(url => fetch(url)));
return data;
} catch (err) {
const { response } = err;
if (response) alert(MESSAGES.GET_PHOTOS_FAIL);
}
};

Chaining unknown number promises in a recursive function

I'm trying to find the best way to go about this service call where I retain all the data in a single object. The call returns an object that has a property of next_page_url. If there is a next_page_url the function should keep chaining. Since I don't know what the url is until the next call resolves I need to call these in order and resolve them in order. I'm also collecting data from each call. I haven't been able to figure out what the structure should be
what I have so far
getDataFromAllPages = (url) => {
waniKaniAxios.get(url).then(object => {
if(object.data.pages.next_url){
return this.getDataFromAllPages(object.data.pages.next_url.replace(waniKaniAxios.defaults.baseURL, ''));
}
});
}
getWanikaniData = () => {
this.getDataFromAllPages('/subjects?types=vocabulary').then(result => {
console.log(result);
});
}

Abstract away the wanikaniaxios.get in another function to make recursion clearer.
Here's my badly formatted code (don't know how SF editor works) , feel to ask any questions if you have any. Happy coding.
getWanikaniData = () => {
this.getDataFromAllPages("/subjects?types=vocabulary")
.then((result) => {
console.log(result);
})
.catch((err) => {
console.log(err); // always put a .catch when you're using prmomises.
});
};
getDataFromAllPages = async (url) => {
// using async await;
try {
let arr = []; // i am assuming you'll improve upon what data structure you might want to return. Linked list seems best to me.
const object = await waniKaniAxios.get(url);
if (object.data.pages.next_url) {
const laterData = await this.getDataFromAllPages(
object.data.pages.next_url.replace(waniKaniAxios.defaults.baseURL, "")
);
arr = [...arr, ...laterData];
} else {
arr = [...arr, object];
}
Promise.resolve(arr);
} catch (err) {
Promise.reject(new Error(`Oops new wanikani error, ${err}`));
}
};
FINAL UPDATE
Using part of the answer below I managed to get it working. Had to partially give up on the recursion aspect because I didn't how to make the promise resolve into data
Here's the final solution that I came up with
getDataFromAllPages = async (url) => {
let results = {};
try {
//getting intial data
const initialData = await waniKaniAxios.get(url);
//using the intial data and mapping out the levels then saving it into results object
results = this.mapOutLevels(initialData.data, results);
//get the next page url
let nextPageUrl = initialData.data.pages.next_url;
//while there is a next page url keep calling the service and adding it to the results object
while (nextPageUrl) {
const laterData = await waniKaniAxios.get(nextPageUrl);
nextPageUrl = laterData.data.pages.next_url;
results = this.mapOutLevels(laterData.data, results);
}
} catch (err) {
Promise.reject(new Error(`Opps new wanikani error, ${err}`));
}
return Promise.resolve(results);
};
getWanikaniData = () => {
this.getDataFromAllPages("/subjects?types=vocabulary")
.then((result) => {
console.log(result);
})
.catch((err) => {
console.log(err);
});
};

fs.readdir recursive search with depth=1

I have to write a code which takes one parameter i.e. path to directory, fetch files from the given directory and again does the same for the directories inside the given directory.The whole search should be wrapped in a promise.
But the depth of recursive search is 1.
Final array should look like: [file1, file2, file3, [file1inDir1, file2inDir1, Dir1inDir1, file3inDir1, Dir2inDir1], file4, file5]
My code is:
const fs = require("fs");
const path = require("path");
function checkfile(files){
let result = [];
for(let i=0 ; i<files.length ;i++){
let newpath = path.join(__dirname,files[i]);
fs.stat(newpath, (err,stats)=>{
if(stats.isDirectory()){
fs.readdir(newpath, (error,files)=>{result.push(files)})
}
else{result.push(files[i])}
})
}
return result;
}
let test = (filepath) => {
return new Promise((resolve, reject) =>{
fs.readdir(filepath, (error,files) => {
if (error) {
reject("Error occured while reading directory");
} else {
resolve(checkfile(files));
}
});
}
)}
test(__dirname)
.then(result => {
console.log(result);
})
.catch(er => {
console.log(er);
});
When I run it I get the following output: []
How do I correct this?

test correctly returns a promise, but checkfile does not, thus all the async operations happen after the yet empty result array was synchronously returned.
Fortunately NodeJS already provides utilities that return promises instead of taking callbacks docs, with them writing that code without callbacks going wrong is easy:
async function checkfile(files){
const result = [];
for(let i=0 ; i<files.length ;i++){
let newpath = path.join(__dirname,files[i]);
const stats = await fs.promises.stat(newpath);
if(stats.isDirectory()){
const files = await fs.promises.readdir(newpath);
result.push(files);
} else result.push(files[i]);
}
return result;
}
async function test(filepath) {
const files = await fs.promises.readdir(filepath);
return checkfile(files);
}

Why is my apolloFetch call returning an empty query when called from within a promise.all?

I'm trying to use apolloFetch inside a Promise.all in my Node.js microservice but keep getting an error that the query is empty. The reason for using apolloFetch is to call another micro service and pass it an array of queries. Can someone give me some direction? My code is as follows:
const uri = "dsc.xxx.yyyy.com/abc/def/graphql";
const apolloFetch = CreateApolloFetch({uri});
const QryAllBooks = {
type: new GraphQLList(BookType),
args: {},
resolve() {
return new Promise((resolve, reject) => {
let sql = singleLineString`
select distinct t.bookid,t.bookname,t.country
from books_tbl t
where t.ship_status = 'Not Shipped'
`;
pool.query(sql, (err, results) => {
if (err) {
reject(err);
}
resolve(results);
const str = JSON.stringify(results);
const json = JSON.parse(str);
const promises = [];
for (let p = 0; p < results.length; p++) {
const book_id = json[p].bookid;
const query = `mutation updateShipping
{updateShipping
(id: ${book_id}, input:{
status: "Shipped"
})
{ bookid
bookname }}`;
promises.push(query);
}
//Below is the Promise.all function with the
//apolloFetch that calls another graphql endpoint
//an array of queries
Promise.all(promises.map(p => apolloFetch({p}))).then((result) => {
//this is the problem code^^^^^^^^^^^^^^^^^^^^^
resolve();
console.log("success!");
}).catch((e) => {
FunctionLogError(29, "Error", e);
});
});
});
}
};
module.exports = {
QryAllBooks,
BookType
};

It looks like apolloFetch requires query - you are passing p
change
Promise.all( promises.map(p=>apolloFetch({p})) )
to
Promise.all( promises.map(query=>apolloFetch({query})) )
You also call resolve twice
To resolve all errors or success
const final_results = []
Promise.all(promises.map(query => apolloFetch({
query,
}))).then((result) => {
final_results.push(result)
}).catch((e) => {
final_results.push(e)
}).then(() => {
resolve(final_results)
});

You immediately resolve or rejects once the pool.query() callback starts:
if(err){ reject(err);}resolve(results);
So unless the query fails, you never resolve with the results from the apolloFetch calls, since the promise is already resolved with the pool.query() results. I guess you're missing an else block:
if( err ) {
reject();
}
else {
const promises = ...
}
PS: you can try using node.js' util.promisify() to turn pool.query() into a promise as well so you can just write something resembling: query(...).then(results=>results.map(apolloFetch) instead of ahving to mix callbacks and promises.

Using loops and promises in transactions in Sequelize

I am currently building a Nodejs, Express, Sequelize (w. PostgreSQL) app, and have run into a few problems with using promises together with transactions and loops.
I am trying to figure out how to use a for loops in a transaction. I am trying to loop through a list of members and create a new user in the database for each of them.
I know the following code is wrong but it shows what I am trying to do.
Can anyone point me in the right direction?
var members = req.body.members;
models.sequelize.transaction(function (t) {
for (var i = 0; i < members.length; i++) {
return models.User.create({'firstname':members[i], 'email':members[i], 'pending':true}, {transaction: t}).then(function(user) {
return user.addInvitations([group], {transaction: t}).then(function(){}).catch(function(err){return next(err);});
})
};
}).then(function (result) {
console.log("YAY");
}).catch(function (err) {
console.log("NO!!!");
return next(err);
});

You should use a Promise.all
var members = req.body.members;
models.sequelize.transaction(function (t) {
var promises = []
for (var i = 0; i < members.length; i++) {
var newPromise = models.User.create({'firstname':members[i], 'email':members[i], 'pending':true}, {transaction: t});
promises.push(newPromise);
};
return Promise.all(promises).then(function(users) {
var userPromises = [];
for (var i = 0; i < users.length; i++) {
userPromises.push(users[i].addInvitations([group], {transaction: t});
}
return Promise.all(userPromises);
});
}).then(function (result) {
console.log("YAY");
}).catch(function (err) {
console.log("NO!!!");
return next(err);
});
I don't believe you need to catch within sequelize transactions as I think it jumps out to the catch on the transaction
Sorry for formatting. On mobile.
Promise.all will wait for all promises to return (or fail) before running the .then, and the .then callback will be all the promise data from each array

You'll need to use the built in looping constructs of bluebird which ships with sequelize:
var members = req.body.members;
models.sequelize.transaction(t =>
Promise.map(members, m => // create all users
models.User.create({firstname: m, email: m, 'pending':true}, {transaction: t})
).map(user => // then for each user add the invitation
user.addInvitations([group], {transaction: t}) // add invitations
)).nodeify(next); // convert to node err-back syntax for express

Depending on your implementation of Node.js this may help. I have the same setup using express, POSTGRES and sequelize.
Personally I'd prefer the async/await (ES6) implementation over then/catch as it is easier to read. Also creating a function that can be called externally improves re-usability.
async function createMemeber(req) {
let members = req.body.members;
for (var i = 0; i < members.length; i++) {
// Must be defined inside loop but outside the try to reset for each new member;
let transaction = models.sequelize.transaction();
try {
// Start transaction block.
let user = await models.User.create({'firstname':members[i], 'email':members[i], 'pending':true}, {transaction});
await user.addInvitations([group], {transaction}));
// if successful commit the record. Else in the catch block rollback the record.
transaction.commit();
// End transaction block.
return user;
} catch (error) {
console.log("An unexpected error occurred creating user record: ", error);
transaction.rollback();
// Throw the error back to the caller and handle it there. i.e. the called express route.
throw error;
}
}
}

if someone is looking for a solution with typescript v4.0.5 using async and await here is what worked out for me. Maybe you can use it on your javascript aplication too but it will depend on the version of it.
const array = ['one','two','three'];
const createdTransaction = sequelize.transaction();
const promises = array.map(async item => {
await model.create({
name: item,
},
{ transaction: createdTransaction },
);
});
Promise.all(promises).then(async values => {
await createdTransaction.commit();
});

First: https://caolan.github.io/async/docs.html
So, easily:
// requiring...
const async = require('async');
// exports...
createAllAsync: (array, transaction) => {
return new Promise((resolve, reject) => {
var results = [];
async.forEachOf(array, (elem, index, callback) => {
results.push(models.Model.create(elem, {transaction}));
callback();
}, err => {
if (err) {
reject(err);
}
else {
resolve(results);
}
});
});
}