I'm developing a cloud based billing system and I have two tables in my database namely bill_history and sold_items. I want to store the Bill number, date, customer name, phone number and total amount and then to return the bill number from bill_history and store the array of objects containing item no, item name, price, quantity, amount with the returned bill no in sold_items. I'm using the following code:
app.post('/billed', (req, res) => {
const { items, total, date } = req.body;
console.log(items, total, date);
db.transaction(trx => {
db.insert({
total: total,
date: date,
}).into('billhead')
.transacting(trx)
.returning('billno')
.then(num => {
for (var i = 0; i < items.length; i++) {
trx.insert({
billno: num,
prodname: items[i].name,
quantity: items[i].quantity,
netprice: items[i].amount
}).into('billdetails')
}).then(trx.commit())
.catch(trx.rollback())
})
})
Now Entries are found in bill_history but not entered in sold_items. I can't find the mistake! Help me with this error. The console and terminal shows No Error
Important thing to remember when working with knex queries: they are promises and they will only execute if:
You call then on the knex object itself
You return the knex query inside a promise chain, and call then somewhere down the chain
Inside your for loop, you have only stated what the knex object should do and because of syntax errors didn't call then on the knex object itself.
.into('billdetails').then(inserts => { /// })
It does work if you return trx.insert()...
That being said, it wouldn't suit your use case, as when inserting multiple values inside a transaction, you need to make sure all inserts have been succesfull. Using for loops the way you did in async fashion is dangerous and won't guarantee all individual inserts have completed without errors and that it's safe to commit the transaction.
One way of achieving this in a safe manner would be modifying this section of your code:
// ...
.returning('billno')
.then(num => {
// We create an array of individual inserts
// Each element in the array will be a single knex
// object/promise that inserts one row into the database
const billDetailInserts = items.map(item => trx.insert({
billno: num,
prodname: item.name,
quantity: item.quantity,
netprice: item.amount
).into('billdetails')
})
// we utilize the Promise.all method that will resolve when
// all individual inserts have completed succesfully
return Promise.all(billDetailInserts);
})
.then(inserts => {
// ... commits, rollbacks, logging etc
Related
I am working with a PostgreSQL database using Prisma. I have a bulk update command which I want to fail if any of the records have changed since my last read.
My schema:
model OrderItem {
id String #id #default(uuid()) #db.Uuid
quantity Int
lastUpdated DateTime #updatedAt #map("last_updated")
##map("order_item")
}
I have written a query which works, but I built the query manually rather than using Prisma's safe query builder tools.
My query:
type OrderItemType = {
id: string;
quantity: number;
lastUpdated: Date;
}
type OrderItemUpdateDataType = {
quantity: number;
}
const updateByIds = async (
orderItemIdLastUpdatedTuples: ([OrderItemType['id'], OrderItemType['lastUpdated']])[],
orderItemUpdateData: OrderItemUpdateDataType,
) => {
// Optimistic concurrency - try updating based on last known "last updated" state. If mismatch, fail.
await prisma.$transaction(async (prisma) => {
// TODO: Prefer prisma.$queryRaw. Prisma.join() works on id[], but not on [id, lastUpdated][]
const idLastUpdatedPairs = orderItemIdLastUpdatedTuples
.map(([id, lastUpdated]) => `(uuid('${id}'), '${lastUpdated.toISOString()}')`)
.join(', ');
const query = `SELECT * FROM order_item WHERE (id, last_updated) in ( ${idLastUpdatedPairs} )`;
const items = await prisma.$queryRawUnsafe<OrderItem[]>(query);
// If query doesn't match expected update count then another query has outraced and updated since last read.
const itemIds = orderItemIdLastUpdatedTuples.map(([id]) => id);
if (items.length !== orderItemIdLastUpdatedTuples.length) {
throw new ConcurrentUpdateError(`Order Items ${itemIds.join(', ')} were stale. Failed to update.`);
}
await prisma.orderItem.updateMany({
where: { id: { in: itemIds } },
data: orderItemUpdateData,
});
});
};
This function wants to update a set of items. It accepts a list of tuples - id/lastUpdated pairs. It starts an explicit transaction, then performs an unsafe SELECT query to confirm the items to affect haven't been updated, then updates. This is following the guidance of Prisma's docs here - https://www.prisma.io/docs/concepts/components/prisma-client/transactions#interactive-transactions-in-preview
I was hoping to achieve the same results using prisma.$queryRaw rather than prisma.$queryRawUnsafe or even using implicit transactions rather than an explicit transaction wrapper. I wasn't able to find a syntax for expressing "where in tuple" using either of these approaches, though.
I am able to express what I want using implicit transactions when updating a single record. An example here would look like:
const { count } = await prisma.orderItem.updateMany({
where: { id, lastUpdated },
data: orderItemUpdateData,
});
and when using an explicit, safe query I stumbled on joining the array of tuples properly.
From the Prisma documentation, https://www.prisma.io/docs/concepts/components/prisma-client/raw-database-access#tagged-template-helpers, there exists a Prisma.join command (which happens implicitly when using their tagged template helper syntax) but I wasn't able to generate a valid output when feeding it an array of tuples.
Did I miss anything? Does Prisma support joining a tuple using their safe query template syntax?
I´m quite unsure on how to handle multiple updates / inserts in knex and return whatever it was successfull on the end or not.
I´m passing an array through req.body loop through it and trigger actions based on informations inside the array.
Example:
const data = [...req.body]
for(let i = 0; i < data.length; i++) {
data[i].totals.length
for(let y = 0; y < data[i].totals.length; y++) {
if(data[i].totals[y].info === "Holiday") {
calcHoliday(data[i].totals[y].total, data[i].id)
} else if(data[i].totals[y].info === "ZA") {
calcZA(data[i].totals[y].total, data[i].id)
}
}
calcOvertime(data[i].totalSum, data[i].id)
if(i === data.length -1) {
res.json("Success")
}
}
The Array I´m passing in looks like this:
[
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
},
{
"id": 1,
"totals": [
{
"info": "Holiday",
"total": 4
}
]
}
]
Function Example which gets called in for loop:
const calcHoliday = (hours, userid) => {
knex.transaction(trx => {
trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
.then(() => {
return trx("hours")
.decrement("holiday_hours", hours)
}).then(trx.commit)
.catch(trx.rollback)
}).then(() => console.log("WORKED"))
.catch(err => console.log(err))
}
This is working perfectly fine but I can´t figure out how to gather the results from each table update in order to respond if everything worked or an error appeared. If I call e.g. after one calcHoliday call .then(resp => res.json(resp) I receive only the response from the first operation.
In short I need a way on how to res.json if everything succeeded or an error appeared somewhere.
Thanks in advance!
TLDR;
Turning your insert calls into an array of promises and then using await and a Promise.all() / Promise.allSettled() structure might solve this problem, but there are some UX decisions to make on what to rollback and how to return errors.
Error Handling Choices:
Any error --> all insertions in all loop iterations should be rolled back
Do you want partial success? The way the code is written now, rollback only applies to items in one function call. If one of the hour-decrement calls fails, it will roll back one log insert, but not any that succeeded for previous data in the loop. If you want the whole dataset to rollback, you'd need to pass the txn through each function call or do a bulk insert of all of your rows in one function call, which might be nice for performance reasons anyway depending on the use case.
Partial success --> commits successes, rolls back single loop iterations that fail, sends detailed list of errors and successes
You'd want to use Promise.allSettled(), which aggregates the successes and errors as an array from all promises in the loop.
Partial success --> commits the successes, rolls back single loop iterations that fail, sends just one error
Opinion: This can be a misleading UX unless the error is "some of the insertions were unsuccessful" and the endpoint is idempotent
This looks closest to what you're describing you want. If this is the case, you'd want to use Promise.all(), which throws an error as soon as one promise in the array errors.
Example Implementation:
Since the original code is incomplete, this is a loose, incomplete example of what option 2/3 might look like. This could easily be transformed into option 1.
First, it might help to modify all of your functions with asynchronous calls to be fulfillable as promises. Async/await helps avoid .then() trees that are hard to reason about.
const calcHoliday = async (hours, userid) => {
try {
const result = await knex.transaction(async(trx) => {
await trx.insert({
created_at: convertedTime,
info: "Booking Holiday - Hours: " + hours,
statuscode: 200
}).into("logs")
return trx("hours").decrement("holiday_hours", hours)
}
return result
} catch(err) {
console.log("It didn't work.")
throw new Error(`Error: Failure to insert for user ${userid}:`, err)
}
}
Here are some utilities to get the data transformed, and to get the appropriate unfulfilled promise to supply to the map in Promise.all/allSettled.
/*
Here's an example of how you might transform the original data with maps in order to avoid nested for-loops:
[
{ id: 1, info: 'Holiday', total: 4 },
{ id: 1, info: 'Holiday', total: 4 }
]
*/
const flatData = data.map(item => {
return item.totals.map(total => ({
id: item.id,
...total
}))
}).flat()
// Returns the appropriate promise based on data
const getTotalsPromises = (row) => {
const {info, id, total} = row
if(info === "Holiday") {
return calcHoliday(total, id)
} else if(info === "ZA") {
return calcZA(total, id)
}
}
const getcalcOvertimePromises = (rowInData) = {
// work left to reader
return calcOvertime(rowInData.correctData, rowInData.otherData)
}
If you want option 2:
// Replaces the loop
// Fulfills *all* the promises, creating an array of errors and successes
const responses = await Promise.allSettled([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
// insert loop here to do something with errors if you want
res.send(responses)
OR Option 3
Create an array of all of the promises you want to run, run them, and process up to one error.
// Replaces the loop
// Runs the promises and waits for them all to finish or the first error.
try {
const responses = await Promise.all([
...flatData.map(getTotalsPromises),
...data.map(getCalcOvertimePromises)
])
res.send(responses)
} catch(err){
// Reached if one of the rows errors
res.send(err)
}
Docs:
Promise.allSettled
Promise.all
I'm fetching tasks from multiple APIs, namely Notion, Todoist and DevOps for now, and trying to merge them into a single collection of tasks so that I can sync them all.
First off, I normalize the tasks so they all have a similar signature like this for Notion:
return {
title: task.properties.Name.title[0].text.content,
status: status(), // Status enum mapping
priority: priority(), // Priority enum mapping
integrations: {
'Notion': {
id: task.id,
original: task
},
'Todoist': {
id: task.properties['Todoist'].number ?? undefined
},
'DevOps': {
id: task.properties['DevOps'].number ?? undefined
}
}
}
And I fetch them all into different arrays:
const fetchedTasks = Promise.all(integrations.map(integration => await integration.getTasks()))
But I don't know how to merge those arrays into a single one based on matching integration ids in an efficient manner. The only solution I could think of would be to loop over every item of every array, adding them to a new collection and checking that entire collection for a matching integration every time, but it just has too many loops to be the right solution.
Something like:
const merged = []
for (const collection of tasks) {
for (const task of collection) {
for (const integration of Object.keys(task.integrations)) {
const mergedTask = merged.find(mergedTask => mergedTask.integrations[integration].id === task.integrations[integration].id)
if (mergedTask)
mergedTask.integrations[integration].original = task
else
merged.push(task)
}
}
}
So that at the end I have a list of tasks with the same signature as the normalized task but with all existing integrations having their original value set.
As as sidenote, are there patterns for syncing similar data from multiple sources? I coulnd't find anything that applied here.
I have the following Schema:
const SubmitDebtSchema = new Schema ({
balance: [{
balanceDate: Date,
newBalance: Number
}],
});
I am attempting to loop through my database entries, pull the 'newBalance' out from each object in the balance array, and then reduce / sum them together.
However, it is returning 'NaN' - and I can't figure out why.
Here is my Axios call to get the data:
componentDidMount() {
axios.get("/api/fetch/fetchDebtCards")
.then((response) => {
this.setState({
debts: response.data
})
console.log(this.state.debts.balance.newBalance)
})
}
The console log in there successfully retrieves the database entries.
And here is my reduce function:
const sumBalance = this.state.debts.reduce(function(previousValue, currentValue) {
return (
previousValue + currentValue.balance.newBalance
)
}, 0)
You can see, I'm attempting to tap into 'balance.newBalance' to access the newBalance within each of the balance objects.
Can anyone point out what I'm doing wrong?
EDIT: My console log, with two entries. What I want to do is get the newBalance array.length -1 from these, and sum them together by reducing.
[Log] Array
0 Object
balance: [{_id: "5fbbddd1077c56000828973c", balanceDate:
"2020-11-23T16:05:36.124Z", newBalance: 400}]
1 Object
balance: [{_id: "5fbc06f58b2f98000865df54", balanceDate:
"2020-11-23T19:01:07.789Z", newBalance: 300}] (1)
if "console.log(this.state.debts.balance.newBalance)" works then debts is not an array, how are you using map on debts then? Map can only be used on arrays.
I'm not sure how your debts object/array is exactly. Maybe a log of it would be helpful.
If it is an array, then this might work.
const sumBalance = this.state.debts.map(x => x.balance).flat().reduce((a,b) => a + b.newBalance, 0)
whereas if it's an object, then this might work
const sumBalance = this.state.debts.balance.reduce((a,b) => a+b.newBalance, 0)
If none of these work, just log "this.state.debts" and let us see what you have there.
Edit: Ok so you only need the last values of the balance arrays (the latest balance), something like this?
const sumBalance = this.state.debts.map(x => x.balance[x.balance.length-1].newBalance).reduce((a,b) => a + b, 0)
Im using Firebase Firestore and want to update an array field under a userprofile with the latest chat thread's id.. Im guessing that I have to pull the entire array (if it exists) from the chat node under that user, then I need to append the new id (if it doesnt exist) and update the array.. It works when theres only 1 value in the array then it fails after that with the following error:
Transaction failed: { Error: Cannot convert an array value in an array value.
at /user_code/node_modules/firebase-admin/node_modules/grpc/src/node/src/client.js:554:15 code: 3, metadata: Metadata { _internal_repr: {} } }
and here is my firebase cloud function, can anyone tell me where im going wrong ?
exports.updateMessages = functions.firestore.document('messages/{messageId}/conversation/{msgkey}').onCreate( (event) => {
/// console.log('function started');
const messagePayload = event.data.data();
const userA = messagePayload.userA;
const userB = messagePayload.userB;
// console.log("userA " + userA);
// console.log("userB " + userB);
// console.log("messagePayload " + JSON.stringify(messagePayload, null, 2) );
const sfDocRef = admin.firestore().doc(`users/${userB}`);
return admin.firestore().runTransaction( (transaction) => {
return transaction.get(sfDocRef).then( (sfDoc) => {
const array = [];
array.push(...[event.params.messageId, sfDoc.get('chats') ]);
transaction.update(sfDocRef, { chats: array } );
});
}).then( () => {
console.log("Transaction successfully committed!");
}).catch( (error) => {
console.log("Transaction failed: ", error);
});
});
You're nesting arrays in your code here:
const array = [];
array.push(...[event.params.messageId, sfDoc.get('chats') ]);
This leads to an array with two values, the first one being the new messageId and the second value contains an array all of your previous values, e.g.
[ "new message id", ["previous id", "older id"] ]
This type of nested array is something that Firestore (apparently) doesn't allow to be stored.
The solution is simple:
const array = [event.params.messageId, ...sfDoc.get('chats')];
The fact that you have to first load the array to then add a single element to it is one of reasons Firebasers recommend not storing data in arrays. Your current data looks like it'd be better off as a set, as shown in the Firestore documenation:
{
"new message id": true,
"previous id": true,
"older id": true
}
That way adding a chat ID is as simple as:
sfDocRef.update({ "chats."+event.params.messageId, true })
I have looked further into the matter, and I would follow the advice that Frank gave you in his post; allocate the data in collections rather than with arrays as they have greater versatility for Firebase 1. Researching under the examples listed in the Firebase website looking for anything related to a chat, I’ve found the data structure and code for messages that are used by Firechat as they might be of use for you.
In the source code, they use a collection for the their message-id -userId pair with the following topology 2 :
The exact way how the saving is executed at the repository is 3 :
It executes an append of the message into the Room-id collection. Instead of this structure, you could use an userID - messageID pair as it might fit you better.