Develop Reference

Knexjs how to handle multiple updates

I´m quite unsure on how to handle multiple updates / inserts in knex and return whatever it was successfull on the end or not.
I´m passing an array through req.body loop through it and trigger actions based on informations inside the array.
Example:
const data = [...req.body]
for(let i = 0; i < data.length; i++) {
data[i].totals.length
for(let y = 0; y < data[i].totals.length; y++) {
if(data[i].totals[y].info === "Holiday") {
calcHoliday(data[i].totals[y].total, data[i].id)
} else if(data[i].totals[y].info === "ZA") {
calcZA(data[i].totals[y].total, data[i].id)
}
}
calcOvertime(data[i].totalSum, data[i].id)
if(i === data.length -1) {
res.json("Success")
}
}
The Array I´m passing in looks like this:
[
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
},
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
}
]
Function Example which gets called in for loop:
const calcHoliday = (hours, userid) => {
knex.transaction(trx => {
trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
.then(() => {
return trx("hours")
.decrement("holiday_hours", hours)
}).then(trx.commit)
.catch(trx.rollback)
}).then(() => console.log("WORKED"))
.catch(err => console.log(err))
}
This is working perfectly fine but I can´t figure out how to gather the results from each table update in order to respond if everything worked or an error appeared. If I call e.g. after one calcHoliday call .then(resp => res.json(resp) I receive only the response from the first operation.
In short I need a way on how to res.json if everything succeeded or an error appeared somewhere.
Thanks in advance!

TLDR;
Turning your insert calls into an array of promises and then using await and a Promise.all() / Promise.allSettled() structure might solve this problem, but there are some UX decisions to make on what to rollback and how to return errors.
Error Handling Choices:
Any error --> all insertions in all loop iterations should be rolled back
Do you want partial success? The way the code is written now, rollback only applies to items in one function call. If one of the hour-decrement calls fails, it will roll back one log insert, but not any that succeeded for previous data in the loop. If you want the whole dataset to rollback, you'd need to pass the txn through each function call or do a bulk insert of all of your rows in one function call, which might be nice for performance reasons anyway depending on the use case.
Partial success --> commits successes, rolls back single loop iterations that fail, sends detailed list of errors and successes
You'd want to use Promise.allSettled(), which aggregates the successes and errors as an array from all promises in the loop.
Partial success --> commits the successes, rolls back single loop iterations that fail, sends just one error
Opinion: This can be a misleading UX unless the error is "some of the insertions were unsuccessful" and the endpoint is idempotent
This looks closest to what you're describing you want. If this is the case, you'd want to use Promise.all(), which throws an error as soon as one promise in the array errors.
Example Implementation:
Since the original code is incomplete, this is a loose, incomplete example of what option 2/3 might look like. This could easily be transformed into option 1.
First, it might help to modify all of your functions with asynchronous calls to be fulfillable as promises. Async/await helps avoid .then() trees that are hard to reason about.
const calcHoliday = async (hours, userid) => {
try {
const result = await knex.transaction(async(trx) => {
await trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
return trx("hours").decrement("holiday_hours", hours)
}
return result
} catch(err) {
console.log("It didn't work.")
throw new Error(`Error: Failure to insert for user ${userid}:`, err)
}
}
Here are some utilities to get the data transformed, and to get the appropriate unfulfilled promise to supply to the map in Promise.all/allSettled.
/*
Here's an example of how you might transform the original data with maps in order to avoid nested for-loops:
[
{ id: 1, info: 'Holiday', total: 4 },
{ id: 1, info: 'Holiday', total: 4 }
]
*/
const flatData = data.map(item => {
return item.totals.map(total => ({
id: item.id,
...total
}))
}).flat()
// Returns the appropriate promise based on data
const getTotalsPromises = (row) => {
const {info, id, total} = row
if(info === "Holiday") {
return calcHoliday(total, id)
} else if(info === "ZA") {
return calcZA(total, id)
}
}
const getcalcOvertimePromises = (rowInData) = {
// work left to reader
return calcOvertime(rowInData.correctData, rowInData.otherData)
}
If you want option 2:
// Replaces the loop
// Fulfills *all* the promises, creating an array of errors and successes
const responses = await Promise.allSettled([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
// insert loop here to do something with errors if you want
res.send(responses)
OR Option 3
Create an array of all of the promises you want to run, run them, and process up to one error.
// Replaces the loop
// Runs the promises and waits for them all to finish or the first error.
try {
const responses = await Promise.all([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
res.send(responses)
} catch(err){
// Reached if one of the rows errors
res.send(err)
}
Docs:
Promise.allSettled
Promise.all

Fetching documents from Mongoose inside a map function returns Promise pending as well as the data

So I have this array of strings which I'm using as filter to fetch documents from MongoDB using Mongoose. I'm successfully getting the data back from the db, but it is also returning Promise { <pending> } on the console even though I'm wrapping the map function with Promise.all.
Please note that this map function is nested inside two level deep for in loops which I'm leaving out for simplicities sake.
Suppose this is the array of strings:
const subCategories = [ 'Politics', 'Entertainment' ];
And this is the map function:
const subCategoryID = await Promise.all( subCategories.map( async item => {
try {
const data = await SubCategory.findOne( { subCategory: item } );
return data;
} catch( e ) {
throw err;
}
}));
console.log( subCategoryID );
Promises make my head spin. I have tried learning Promises more than 5-6 times in the last 1-2 years and it still confuses me. So if you can, please provide a solution with Async/Await.

If you only want the result, Try this approach:
const subCategories = [ 'Politics', 'Entertainment' ];
const queries = subCategories.map((i) => {
return { subCategory: i };
});
const subCategoryID = await SubCategory.find({
$or: queries,
});
console.log(subCategoryID);
using $or operator is explained in mongodb manual:
The $or operator performs a logical OR operation on an array of two or more <expressions> and selects the documents that satisfy at least one of the <expressions>.
more about it here.
In mongoose, you can use it in any find methods.
for explanation about your promise, sorry I can't help.
Please mark answered if this solves your problem.

Skipping top level of JSON data and retrieving data below it via JavaScript

Via a microservice, I retrieve several packages of JSON data and spit them out onto a Vue.js-driven page. The data looks something like this:
{"data":{"getcompanies":
[
{"id":6,"name":"Arena","address":"12 Baker Street","zip":"15090"},
{"id":7,"name":"McMillan","address":null,"zip":"15090"},
{"id":8,"name":"Ball","address":"342 Farm Road","zip":"15090"}
]
}}
{"data":{"getusers":
[{"id":22,"name":"Fred","address":"Parmesean Street","zip":"15090"},
{"id":24,"name":"George","address":"Loopy Lane","zip":"15090"},
{"id":25,"name":"Lucy","address":"Farm Road","zip":"15090"}]}}
{"data":{"getdevices":
[{"id":2,"name":"device type 1"},
{"id":4,"name":"device type 2"},
{"id":5,"name":"device type 3"}]}}
...and I successfully grab them individually via code like this:
getCompanies() {
this.sendMicroServiceRequest({
method: 'GET',
url: `api/authenticated/function/getcompanies`
})
.then((response) => {
if(response.data) {
this.dataCompanies = response.data.getcompanies
} else {
console.error(response)
}
}).catch(console.error)
}
...with getUsers() and getDevices() looking respectively the same. getCompanies() returns:
[{"id":6,"name":"Arena","address":"12 Baker Street","zip":"15090"},
{"id":7,"name":"McMillan","address":null,"zip":"15090"},
{"id":8,"name":"Ball","address":"342 Farm Road","zip":"15090"}]
...which I relay to the Vue template in a table, and this works just fine and dandy.
But this is obviously going to get unwieldy if I need to add more microservice calls down the road.
What I'm looking for is an elegant way to jump past the response.data.*whatever* and get to those id-records with a re-useable call, but I'm having trouble getting there. response.data[0] doesn't work, and mapping down to the stuff I need either comes back undefined, or in bits of array. And filtering for response.data[0].id to return just the rows with ids keeps coming back undefined.
My last attempt (see below) to access the data does work, but looks like it comes back as individual array elements. I'd rather not - if possible - rebuild an array into a JSON structure. I keep thinking I should be able to just step past the next level regardless of what it's called, and grab whatever is there in one chunk, as if I read response.data.getcompanies directly, but not caring what 'getcompanies' is, or needing to reference it by name:
// the call
this.dataCompanies = this.getFullData('companies')
getFullData(who) {
this.sendMicroServiceRequest({
method: 'GET',
url: 'api/authenticated/function/get' + who,
})
.then((response) => {
if(response) {
// attempt 1 to get chunk below 'getcompanies'
Object.keys(response.data).forEach(function(prop) {
console.log(response.data[prop])
})
// attempt 2
// for (const prop in response.data) {
// console.log(response.data[prop])
// }
let output = response.data[prop] // erroneously thinking this is in one object
return output
} else {
console.error(response)
}
}).catch(console.error)
}
...outputs:
(63) [{…}, {…}, {…}] <-- *there are 63 of these records, I'm just showing the first few...*
0: {"id":6,"name":"Arena","address":"12 Baker Street","zip":"15090"}
1: {"id":7,"name":"McMillan","address":null,"zip":"15090"},
2: {"id":8,"name":"Ball","address":"342 Farm Road","zip":"15090"}...
Oh, and the return above comes back 'undefined' for some reason that eludes me at 3AM. >.<
It's one of those things where I think I am close, but not quite. Any tips, hints, or pokes in the right direction are greatly appreciated.

I feel it's better to be explicit about accessing the object. Seems like the object key is consistent with the name of the microservice function? If so:
getData(functionName) {
return this.sendMicroServiceRequest({
method: 'GET',
url: "api/authenticated/function/" + functionName
})
.then( response => response.data[functionName] )
}
getCompanies(){
this.getData("getcompanies").then(companies => {
this.dataCompanies = companies
})
}

let arrResponse = {data: ['x']};
let objResponse = {data: {getcompanies: 'x'}};
console.log(arrResponse.data[0]);
console.log(Object.values(objResponse.data)[0]);
response.data[0] would work if data was an array. To get the first-and-only element of an object, use Object.values(response.data)[0] instead. Object.values converts an object to an array of its values.
Its counterparts Object.keys and Object.entries likewise return arrays of keys and key-value tuples respectively.
Note, order isn't guaranteed in objects, so this is only predictable in your situation because data has exactly a single key & value. Otherwise, you'd have to iterate the entry tuples and search for the desired entry.

firstValue
Let's begin with a generic function, firstValue. It will get the first value of an object, if present, otherwise it will throw an error -
const x = { something: "foo" }
const y = {}
const firstValue = t =>
{ const v = Object.values(t)
if (v.length)
return v[0]
else
throw Error("empty data")
}
console.log(firstValue(x)) // "foo"
console.log(firstValue(y)) // Error: empty data
getData
Now write a generic getData. We chain our firstValue function on the end, and be careful not to add a console.log or .catch here; that is a choice for the caller to decide -
getData(url) {
return this
.sendMicroServiceRequest({ method: "GET", url })
.then(response => {
if (response.data)
return response.data
else
return Promise.reject(response)
})
.then(firstValue)
}
Now we write getCompanies, getUsers, etc -
getCompanies() {
return getData("api/authenticated/function/getcompanies")
}
getUsers() {
return getData("api/authenticated/function/getusers")
}
//...
async and await
Maybe you could spruce up getData with async and await -
async getData(url) {
const response =
await this.sendMicroServiceRequest({ method: "GET", url })
return response.data
? firstValue(response.data)
: Promise.reject(response)
}
power of generics demonstrated
We might even suggest that these get* functions are no longer needed -
async getAll() {
return {
companies:
await getData("api/authenticated/function/getcompanies"),
users:
await getData("api/authenticated/function/getusers"),
devices:
await getData("api/authenticated/function/getdevices"),
// ...
}
}
Above we used three await getData(...) requests which happen in serial order. Perhaps you want all of these requests to run in parallel. Below we will show how to do that -
async getAll() {
const requests = [
getData("api/authenticated/function/getcompanies"),
getData("api/authenticated/function/getusers"),
getData("api/authenticated/function/getdevices")
]
const [companies, users, devices] = Promise.all(requests)
return { companies, users, devices }
}
error handling
Finally, error handling is reserved for the caller and should not be attempted within our generic functions -
this.getAll()
.then(data => this.render(data)) // some Vue template
.catch(console.error)

Is a promise within a promise the best solution? asynchronous node file read within for loop

The Node.js function below takes:
an object, shop which contains a regular expression
an array of filenames
The function will read each csv file listed in the array and test a cell in the first row with a regular expression, returning a new array of matching filenames.
function matchReport(shop, arr) {
return promise = new Promise(resolve => {
var newArray = [];
for(var j=0;j<arr.length;++j) {
let filename = arr[j];
csv()
.fromFile(filename)
.then(reportData => {
if (reportData[0]['Work'].match(shop.productRegex)) {
newArray.push(filename);
}
if (j === arr.length) {
resolve(newArray);
}
});
}
}).then(matches => {
return {
'shop' : shop.name,
'reports' : matches
}
}).catch(e => {
console.log(e);
});
}
Very rarely the function will return with the correct behavior which is this:
{ shop: 'shop1',
reports: [ '../artist-sales-report-2020-11-12(1).csv' ] }
{ shop: 'shop2',
reports:
[ '../artist-sales-report-2020-12-03.csv',
'../artist-sales-report-2020-09-01.csv' ] }
More often it returns with reports missing, like below:
{ shop: 'shop1',
reports: [ '../artist-sales-report-2020-11-12(1).csv' ] }
{ shop: 'shop2',
reports: [ '../artist-sales-report-2020-12-03.csv' ] }
I understand where the problem is taking place, within the csv reportData block. I understand that it is an asynchronous issue and I have tried to write more elaborate if..then or switch statements as a hack solution with no luck. It seems a little sloppy and cluttered to me to create more promises inside of this promise but I have been unsuccessful at that as well.

Using async/await and your disliked nested promises you could simplify your code to something like this, which should always await all results. I made the assumption that your problem is the fromFile method, which feels like it is itself asynchronous since it uses a then that you are not awaiting.
async function matchReport(shop, arr) {
const matches = await Promise.all(arr.map(async filename => {
const reportData = await csv().fromFile( filename );
if( reportData[0]['Work'].match(shop.productRegex) ){
return filename;
}
}));
return {
'shop': shop.name,
'reports': matches.filter( Boolean )
};
}

Nesting API calls in React Redux

I am having difficulty figuring out what is happening (and not happening) in my action creator.
I need to make a call to one API endpoint, get the ids and names of all the items returned, then for each of those ids, make another call. I want to store the return of the last call and the ids/names from the first call in an object and dispatch it.
{
category: name //name of category
subcategory: [] //array of categories in the category above.
}
Right now, my reducer does end up having what I want, but when I attempt to log that particular prop in the component it is empty. (below I am using OpenSocialAPI or osapi. This is just a basic wrapper for an ajax request. Allows for me to not have to authenticate as it sees I am already authenticated.)
export function fetchCategories(id){
let categories = []
return function(dispatch){
dispatch(requestCategories(id))
return osapi.core.get({
v: "v3",
href: "/places/" + id + "/places"
}).execute(function(response) {
response.content.list.forEach(function(category) {
osapi.core.get({
v: "v3",
href: "/places/" + category.id+ "/categories"
}).execute(function(response) {
categories.push({
category: category.name,
subcategories: response.content.list.map(category => category.name)
})
})
})
console.log("Category response: ", categories)
dispatch(receiveCategories(id, categories))
})
}
}
export function receiveCategories(id,array){
return {
type: RECEIVE_CATEGORIES,
id,
categories: array,
recievedAt: new Date(Date.now()),
isFetching: false
}
}
And in my app I am dispatching the action creator in componentDidMount
componentDidMount() {
const { dispatch } = this.props
dispatch(fetchCategoriesIfNeeded(id))
}
Right now when I console log in my Category component and in the execute above, it is empty. But looking at my state in my logger, when recieveCategories is completed, I have the array of objects I want
[{category:...,
subcategories:[...]
},
{category:...,
subcategories:[...]
}]
I suspect this is because of something asynchronous but I'm unsure how to proceed.
I attempted to create my own wrapper for the call that was promise based, but I had similar issues, probably more so because I'm not sure if resolve(response) is what I want.
function httpService(path){
return new Promise(function(resolve, reject){
osapi.core.get({
v: 'v3',
href: path
}).execute(function(response, error){
if(error){
return reject(new Error("Error: ", error))
}
resolve(response)
})
})
}
export function fetchCategories(spaceId) {
let categories = []
return function(dispatch) {
dispatch(requestCategories(id))
return httpService("/places/" + id + "/places")
.then(function(response) {
response.content.list.forEach(function(category) {
fetchSubCategories("/places/" + category.id + "/categories")
.then(function(response) {
categories.push({
category: category.name,
subcategories: response
})
})
})
console.log("CATEGORIES: ", categories)
dispatch(receiveCategories(id, categories))
})
}
}
function fetchSubCategories(url){
return httpService(url)
.then(function(response){
return response.content.list.map(category => category.name)
})
}
Can you look at this and give guidance? Also, is me dispatching an array that I built based on the API responses a valid way of doing things or is there someway better? Thank you
I was only able to find 1 other question with similar use case but they are using bluebird or something similar. I'd really like to keep this without anything extra besides Redux.

It looks like you just need to dispatch your categories inside your .execute() callback, not outside of it. You're doing osapi.core.get().execute((response) => but then outside of that execute callback, you dispatch receiveCategories, which will execute long before your Promise resolves, and dispatch the empty array you initialize.
You also need to use Promise.all to get the response of all of your nested GET requests.
There's also no reason to keep a mutating array around.

I guess osapi.core.get is some kind of promise based fetch library? And .execute is called when the get succeeds?
If so, then what you're missing is that you're not waiting for all asynchronous calls to finish.
I'm going to show a solution based on generic fetch and native Promises so you can understand the solution and adopt it based on your specific libraries.
const promises = [];
response.content.list.forEach(function(category) {
const promise = fetch("/places/" + category.id+ "/categories");
promises.push(promise);
})
Promise.all(promises).then(responses => {
categories = responses.map(response => ({/* do your object mapping here */}))
// now you can dispatch with all received categories
dispatch(receiveCategories(id, categories))
});
Also, you're using the same variable in your nested functions - while this may work and the computers may understand it, it makes it super hard for any human to figure out which response belongs to which scope. So you may want to take a second look at your variable names as well.