Develop Reference

Knexjs how to handle multiple updates

I´m quite unsure on how to handle multiple updates / inserts in knex and return whatever it was successfull on the end or not.
I´m passing an array through req.body loop through it and trigger actions based on informations inside the array.
Example:
const data = [...req.body]
for(let i = 0; i < data.length; i++) {
data[i].totals.length
for(let y = 0; y < data[i].totals.length; y++) {
if(data[i].totals[y].info === "Holiday") {
calcHoliday(data[i].totals[y].total, data[i].id)
} else if(data[i].totals[y].info === "ZA") {
calcZA(data[i].totals[y].total, data[i].id)
}
}
calcOvertime(data[i].totalSum, data[i].id)
if(i === data.length -1) {
res.json("Success")
}
}
The Array I´m passing in looks like this:
[
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
},
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
}
]
Function Example which gets called in for loop:
const calcHoliday = (hours, userid) => {
knex.transaction(trx => {
trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
.then(() => {
return trx("hours")
.decrement("holiday_hours", hours)
}).then(trx.commit)
.catch(trx.rollback)
}).then(() => console.log("WORKED"))
.catch(err => console.log(err))
}
This is working perfectly fine but I can´t figure out how to gather the results from each table update in order to respond if everything worked or an error appeared. If I call e.g. after one calcHoliday call .then(resp => res.json(resp) I receive only the response from the first operation.
In short I need a way on how to res.json if everything succeeded or an error appeared somewhere.
Thanks in advance!

TLDR;
Turning your insert calls into an array of promises and then using await and a Promise.all() / Promise.allSettled() structure might solve this problem, but there are some UX decisions to make on what to rollback and how to return errors.
Error Handling Choices:
Any error --> all insertions in all loop iterations should be rolled back
Do you want partial success? The way the code is written now, rollback only applies to items in one function call. If one of the hour-decrement calls fails, it will roll back one log insert, but not any that succeeded for previous data in the loop. If you want the whole dataset to rollback, you'd need to pass the txn through each function call or do a bulk insert of all of your rows in one function call, which might be nice for performance reasons anyway depending on the use case.
Partial success --> commits successes, rolls back single loop iterations that fail, sends detailed list of errors and successes
You'd want to use Promise.allSettled(), which aggregates the successes and errors as an array from all promises in the loop.
Partial success --> commits the successes, rolls back single loop iterations that fail, sends just one error
Opinion: This can be a misleading UX unless the error is "some of the insertions were unsuccessful" and the endpoint is idempotent
This looks closest to what you're describing you want. If this is the case, you'd want to use Promise.all(), which throws an error as soon as one promise in the array errors.
Example Implementation:
Since the original code is incomplete, this is a loose, incomplete example of what option 2/3 might look like. This could easily be transformed into option 1.
First, it might help to modify all of your functions with asynchronous calls to be fulfillable as promises. Async/await helps avoid .then() trees that are hard to reason about.
const calcHoliday = async (hours, userid) => {
try {
const result = await knex.transaction(async(trx) => {
await trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
return trx("hours").decrement("holiday_hours", hours)
}
return result
} catch(err) {
console.log("It didn't work.")
throw new Error(`Error: Failure to insert for user ${userid}:`, err)
}
}
Here are some utilities to get the data transformed, and to get the appropriate unfulfilled promise to supply to the map in Promise.all/allSettled.
/*
Here's an example of how you might transform the original data with maps in order to avoid nested for-loops:
[
{ id: 1, info: 'Holiday', total: 4 },
{ id: 1, info: 'Holiday', total: 4 }
]
*/
const flatData = data.map(item => {
return item.totals.map(total => ({
id: item.id,
...total
}))
}).flat()
// Returns the appropriate promise based on data
const getTotalsPromises = (row) => {
const {info, id, total} = row
if(info === "Holiday") {
return calcHoliday(total, id)
} else if(info === "ZA") {
return calcZA(total, id)
}
}
const getcalcOvertimePromises = (rowInData) = {
// work left to reader
return calcOvertime(rowInData.correctData, rowInData.otherData)
}
If you want option 2:
// Replaces the loop
// Fulfills *all* the promises, creating an array of errors and successes
const responses = await Promise.allSettled([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
// insert loop here to do something with errors if you want
res.send(responses)
OR Option 3
Create an array of all of the promises you want to run, run them, and process up to one error.
// Replaces the loop
// Runs the promises and waits for them all to finish or the first error.
try {
const responses = await Promise.all([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
res.send(responses)
} catch(err){
// Reached if one of the rows errors
res.send(err)
}
Docs:
Promise.allSettled
Promise.all

mongoosejs - find() using nested objects

question is possibly a duplicate but I haven't found anything that provides an appropriate answer to my issue.
I have an ExpressJS server which is used to provide API requests to retrieve data from a MongoDB database. I am using mongoosejs for the MongoDB connection to query/save data.
I am building a route that will allow me to find all data that matches some user input but I am having trouble when doing the query. I have spent a long while looking online for someone with a similar issue but coming up blank.
I will leave example of the code I have at the minute below.
code for route
// -- return matched data (GET)
router.get('/match', async (req, res) => {
const style_data = req.query.style; // grab url param for style scores ** this comes in as a string **
const character_data = req.query.character; // grab url param for character scores ** this comes in as a string **
// run matcher systems
const style_matches = style_match(style_data);
res.send({
response: 200,
data: style_matches
}); // return data
});
code for the query
// ---(Build the finder)
const fetch_matches_using = async function(body, richness, smoke, sweetness) {
return await WhiskyModel.find({
'attributes.body': body,
'attributes.richness': richness,
'attributes.smoke': smoke,
'attributes.sweetness': sweetness
});
}
// ---(Start match function)---
const style_match = async function (scores_as_string) {
// ---(extract data)---
const body = scores_as_string[0];
const richness = scores_as_string[1];
const smoke = scores_as_string[2];
const sweetness = scores_as_string[3];
const matched = [];
// ---(initialise variables)---
let match_count = matched.length;
let first_run; // -> exact matches
let second_run; // -> +- 1
let third_run; // -> +- 2
let fourth_run; // -> +- 3
// ---(begin db find loop)---
first_run = fetch_matches_using(body, richness, smoke, sweetness).then((result) => {return result});
matched.push(first_run);
// ---(return final data)---
return matched
}
example of db object
{
_id: mongoid,
meta-data: {
pagemd:{some data},
name: whiskyname
age: whiskyage,
price: price
},
attributes: {
body: "3",
richness: "3",
smoke: "0",
sweetness: "3",
some other data ...
}
}
When I hit the route in postman the JSON data looks like:
{
response: 200,
data: {}
}
and when I console.log() out matched from within the style match function after I have pushed the it prints [ Promise(pending) ] which I don't understand.
if I console.log() the result from within the .then() I get an empty array.
I have tried using the populate() method after running the find which does technically work, but instead of only returning data that matches it returns every entry in the collection so I think I am doing something wrong there, but I also don't see why I would need to use the .populate() function to access the nested object.
Am I doing something totally wrong here?
I should also mention that the route and the matching functions are in different files just to try and keep things simple.
Thanks for any answers.

just posting an answer as I seem to have fixed this.
Issue was with my .find() function, needed to pass in the items to search by and then also a call back within the function to return error/data. I'll leave the changed code below.
new function
const fetch_matches_using = async function(body, richness, smoke, sweetness) {
const data = await WhiskyModel.find({
'attributes.body': body,
'attributes.richness': richness,
'attributes.smoke': smoke,
'attributes.sweetness': sweetness
}, (error, data) => { // new ¬
if (error) {
return error;
}
if (data) {
console.log(data)
return data
}
});
return data; //new
}
There is still an issue with sending the found results back to the route but this is a different issue I believe. If its connected I'll edit this answer with the fix for that.

Child functions and async await

In one of my API endpoints I fetch a json resource (1) from the web and edit it to fit my needs. In the "lowest" or "deepest" part of the tree I'm trying to fetch another resource and add it to the final json object. I'm relatively new to async/await but am trying to move away from the "old" Promises since I see the advantage (or the gain) of using async/await.
The object from (1) looks like;
const json = {
date,
time,
trips: [{
name,
legs: [{
id
},
{
id
}
]
}]
};
Here's how I "reformat" and change the json object;
{
date,
time,
trips: json.trips.map(trip => formatTrip(trip))
};
function formatTrip(trip) {
return {
name,
legs: trip.legs.map(leg => formatLeg(leg))
};
};
async function formatLeg(leg) {
const data = await fetch();
return {
id,
data
};
};
The problem with this is that after I've "reformatted/edited" the original json to look how I want it (and ran through all format... functions) the legs objects are empty {}.
I figured this might be due to the async/await promises not finishing. I've also read that if a child-function uses async/await all the higher functions has to use async/await as well.
Why? How can I rewrite my code to work and look good? Thanks!
EDIT:
I updated my code according to Randy's answer. getLegStops(leg) is still undefined/empty.
function formatLeg(leg) {
return {
other,
stops: getLegStops(leg)
};
};
function getLegStops(leg) {
Promise.all(getLegStopRequests(leg)).then(([r1, r2]) => {
/* do stuff here */
return [ /* with data */ ];
});
};
function getLegStopRequests(leg) {
return [ url1, url2 ].map(async url => await axios.request({ url }));
};

Two things lead you to want to nest these Promises:
The old way of thinking about callbacks and then Promises
Believing the software process must match the data structure
It appears you only need to deal with the Promises once if I understand correctly.
Like this:
async function getLegs(){
return trip.legs.map(async leg => await fetch(...)); // produces an array of Promises
}
const legs = Promise.all(getLegs());
function formatLegs(legs) {
// do something with array of legs
};
function formatTrip(){
//format final output
}
EDIT: per your comment below, this snippet represents what I've demonstrated and what your goal should look like. Please review your code carefully.
const arr = [1, 2, 3, ];
const arrPromises = arr.map(async v => await new Promise((res) => res(v)));
const finalPromise = Promise.all(arrPromises);
console.log(finalPromise.then(console.log));

Nesting API calls in React Redux

I am having difficulty figuring out what is happening (and not happening) in my action creator.
I need to make a call to one API endpoint, get the ids and names of all the items returned, then for each of those ids, make another call. I want to store the return of the last call and the ids/names from the first call in an object and dispatch it.
{
category: name //name of category
subcategory: [] //array of categories in the category above.
}
Right now, my reducer does end up having what I want, but when I attempt to log that particular prop in the component it is empty. (below I am using OpenSocialAPI or osapi. This is just a basic wrapper for an ajax request. Allows for me to not have to authenticate as it sees I am already authenticated.)
export function fetchCategories(id){
let categories = []
return function(dispatch){
dispatch(requestCategories(id))
return osapi.core.get({
v: "v3",
href: "/places/" + id + "/places"
}).execute(function(response) {
response.content.list.forEach(function(category) {
osapi.core.get({
v: "v3",
href: "/places/" + category.id+ "/categories"
}).execute(function(response) {
categories.push({
category: category.name,
subcategories: response.content.list.map(category => category.name)
})
})
})
console.log("Category response: ", categories)
dispatch(receiveCategories(id, categories))
})
}
}
export function receiveCategories(id,array){
return {
type: RECEIVE_CATEGORIES,
id,
categories: array,
recievedAt: new Date(Date.now()),
isFetching: false
}
}
And in my app I am dispatching the action creator in componentDidMount
componentDidMount() {
const { dispatch } = this.props
dispatch(fetchCategoriesIfNeeded(id))
}
Right now when I console log in my Category component and in the execute above, it is empty. But looking at my state in my logger, when recieveCategories is completed, I have the array of objects I want
[{category:...,
subcategories:[...]
},
{category:...,
subcategories:[...]
}]
I suspect this is because of something asynchronous but I'm unsure how to proceed.
I attempted to create my own wrapper for the call that was promise based, but I had similar issues, probably more so because I'm not sure if resolve(response) is what I want.
function httpService(path){
return new Promise(function(resolve, reject){
osapi.core.get({
v: 'v3',
href: path
}).execute(function(response, error){
if(error){
return reject(new Error("Error: ", error))
}
resolve(response)
})
})
}
export function fetchCategories(spaceId) {
let categories = []
return function(dispatch) {
dispatch(requestCategories(id))
return httpService("/places/" + id + "/places")
.then(function(response) {
response.content.list.forEach(function(category) {
fetchSubCategories("/places/" + category.id + "/categories")
.then(function(response) {
categories.push({
category: category.name,
subcategories: response
})
})
})
console.log("CATEGORIES: ", categories)
dispatch(receiveCategories(id, categories))
})
}
}
function fetchSubCategories(url){
return httpService(url)
.then(function(response){
return response.content.list.map(category => category.name)
})
}
Can you look at this and give guidance? Also, is me dispatching an array that I built based on the API responses a valid way of doing things or is there someway better? Thank you
I was only able to find 1 other question with similar use case but they are using bluebird or something similar. I'd really like to keep this without anything extra besides Redux.

It looks like you just need to dispatch your categories inside your .execute() callback, not outside of it. You're doing osapi.core.get().execute((response) => but then outside of that execute callback, you dispatch receiveCategories, which will execute long before your Promise resolves, and dispatch the empty array you initialize.
You also need to use Promise.all to get the response of all of your nested GET requests.
There's also no reason to keep a mutating array around.

I guess osapi.core.get is some kind of promise based fetch library? And .execute is called when the get succeeds?
If so, then what you're missing is that you're not waiting for all asynchronous calls to finish.
I'm going to show a solution based on generic fetch and native Promises so you can understand the solution and adopt it based on your specific libraries.
const promises = [];
response.content.list.forEach(function(category) {
const promise = fetch("/places/" + category.id+ "/categories");
promises.push(promise);
})
Promise.all(promises).then(responses => {
categories = responses.map(response => ({/* do your object mapping here */}))
// now you can dispatch with all received categories
dispatch(receiveCategories(id, categories))
});
Also, you're using the same variable in your nested functions - while this may work and the computers may understand it, it makes it super hard for any human to figure out which response belongs to which scope. So you may want to take a second look at your variable names as well.

Why am I getting back "undefined" from my Axios call, when it should return me objects?

Problem Intro:
In my React app I'm making some API calls to Github to fetch some user data. In another function I call these functions and wait for it with Axios's .all() method.
I want to fetch the data and do something with it using .then() but the returned value is just an array with 2 times undefined
What is the expected thing to happen:
It should return me 2 player objects with profile infos in the profile key of the object and a score value in the score key of the object.
What is my app doing in a nutshell?
It fetches the data from 2 usernames and they can "battle" each other. It just fetches the score and the followers and returns a sum. At the end it (should) returns an array with 2 player objects already sorted with the winner in the first place (exampleArray[0]).
General information's
It's an react app using components. It's really about one very tiny component and the helper functions in another file.
Here is where I call my custom function (the one returning undefined):
componentDidMount() {
const players = queryString.parse(this.props.location.search); //<== the usernames
const playersArray = [players.playerOneName, players.playerTwoName];
console.log(playersArray); // <== this logs the output as expected (an array with 2 usernames)
battle(playersArray).then((data) => { // <== the function it's all about
console.log(data); // <== data is => [undefined, undefined];
})
}
Next is the battle function from above that uses 2 other functions:
battle
export function battle(players) { // <== players aray with 2 usernames as string
return axios.all(players.map(getUserData)) // <== look 1 function below
.then(sortPlayers) // <== two functions below
.catch(handleError)
}
getUserData
let getUserData = (player) => {
axios.all([
getProfile(player),
getRepos(player)
]).then((data) => {
return {
profile: data[0],
score: calculateScore(data[0], data[1])
}
})
}
sortPlayers
let sortPlayers = (players) => {
return players.sort((a, b) => {
return b.score - a.score;
})
}
Ok so they also use other functions but they are really not too complicated. Let me know when you need examples from the other little helpers too.
I tried it with placing the debugger in different spots in the code and console logged different things, but I can't come through (first time I'm really working with promises). Sitting now 2 hours in front of this tiny problem and I can't figure it out.
I think the problem lies somewhere in battle function itself or getUserData
At the end a little screenshot, what the output of my console.log looks: http://prntscr.com/hz5abq
Thanks in advance

You don't have anything being returned in getUserData . Either add a return or remove the {} wrapping axios.all
let getUserData = (player) => {
return axios.all([
getProfile(player),
getRepos(player)
]).then((data) => {
return {
profile: data[0],
score: calculateScore(data[0], data[1])
}
})
}

getUserData needs to return the promise that it creates. At the moment it's not returning anything, so an implicit undefined is returned, and thus players.map(getUserData) results in an array of [undefined, undefined]
Ie, do this:
let getUserData = (player) => {
// VVV added return statement
return axios.all([
getProfile(player),
getRepos(player)
]).then((data) => {
return {
profile: data[0],
score: calculateScore(data[0], data[1])
}
})
}