I'm fetching tasks from multiple APIs, namely Notion, Todoist and DevOps for now, and trying to merge them into a single collection of tasks so that I can sync them all.
First off, I normalize the tasks so they all have a similar signature like this for Notion:
return {
title: task.properties.Name.title[0].text.content,
status: status(), // Status enum mapping
priority: priority(), // Priority enum mapping
integrations: {
'Notion': {
id: task.id,
original: task
},
'Todoist': {
id: task.properties['Todoist'].number ?? undefined
},
'DevOps': {
id: task.properties['DevOps'].number ?? undefined
}
}
}
And I fetch them all into different arrays:
const fetchedTasks = Promise.all(integrations.map(integration => await integration.getTasks()))
But I don't know how to merge those arrays into a single one based on matching integration ids in an efficient manner. The only solution I could think of would be to loop over every item of every array, adding them to a new collection and checking that entire collection for a matching integration every time, but it just has too many loops to be the right solution.
Something like:
const merged = []
for (const collection of tasks) {
for (const task of collection) {
for (const integration of Object.keys(task.integrations)) {
const mergedTask = merged.find(mergedTask => mergedTask.integrations[integration].id === task.integrations[integration].id)
if (mergedTask)
mergedTask.integrations[integration].original = task
else
merged.push(task)
}
}
}
So that at the end I have a list of tasks with the same signature as the normalized task but with all existing integrations having their original value set.
As as sidenote, are there patterns for syncing similar data from multiple sources? I coulnd't find anything that applied here.
Related
I am working with a PostgreSQL database using Prisma. I have a bulk update command which I want to fail if any of the records have changed since my last read.
My schema:
model OrderItem {
id String #id #default(uuid()) #db.Uuid
quantity Int
lastUpdated DateTime #updatedAt #map("last_updated")
##map("order_item")
}
I have written a query which works, but I built the query manually rather than using Prisma's safe query builder tools.
My query:
type OrderItemType = {
id: string;
quantity: number;
lastUpdated: Date;
}
type OrderItemUpdateDataType = {
quantity: number;
}
const updateByIds = async (
orderItemIdLastUpdatedTuples: ([OrderItemType['id'], OrderItemType['lastUpdated']])[],
orderItemUpdateData: OrderItemUpdateDataType,
) => {
// Optimistic concurrency - try updating based on last known "last updated" state. If mismatch, fail.
await prisma.$transaction(async (prisma) => {
// TODO: Prefer prisma.$queryRaw. Prisma.join() works on id[], but not on [id, lastUpdated][]
const idLastUpdatedPairs = orderItemIdLastUpdatedTuples
.map(([id, lastUpdated]) => `(uuid('${id}'), '${lastUpdated.toISOString()}')`)
.join(', ');
const query = `SELECT * FROM order_item WHERE (id, last_updated) in ( ${idLastUpdatedPairs} )`;
const items = await prisma.$queryRawUnsafe<OrderItem[]>(query);
// If query doesn't match expected update count then another query has outraced and updated since last read.
const itemIds = orderItemIdLastUpdatedTuples.map(([id]) => id);
if (items.length !== orderItemIdLastUpdatedTuples.length) {
throw new ConcurrentUpdateError(`Order Items ${itemIds.join(', ')} were stale. Failed to update.`);
}
await prisma.orderItem.updateMany({
where: { id: { in: itemIds } },
data: orderItemUpdateData,
});
});
};
This function wants to update a set of items. It accepts a list of tuples - id/lastUpdated pairs. It starts an explicit transaction, then performs an unsafe SELECT query to confirm the items to affect haven't been updated, then updates. This is following the guidance of Prisma's docs here - https://www.prisma.io/docs/concepts/components/prisma-client/transactions#interactive-transactions-in-preview
I was hoping to achieve the same results using prisma.$queryRaw rather than prisma.$queryRawUnsafe or even using implicit transactions rather than an explicit transaction wrapper. I wasn't able to find a syntax for expressing "where in tuple" using either of these approaches, though.
I am able to express what I want using implicit transactions when updating a single record. An example here would look like:
const { count } = await prisma.orderItem.updateMany({
where: { id, lastUpdated },
data: orderItemUpdateData,
});
and when using an explicit, safe query I stumbled on joining the array of tuples properly.
From the Prisma documentation, https://www.prisma.io/docs/concepts/components/prisma-client/raw-database-access#tagged-template-helpers, there exists a Prisma.join command (which happens implicitly when using their tagged template helper syntax) but I wasn't able to generate a valid output when feeding it an array of tuples.
Did I miss anything? Does Prisma support joining a tuple using their safe query template syntax?
I'm creating a StencilJS app (no framework) with a Google Firestore backend, and I want to use the RxFire and RxJS libraries as much as possible to simplify data access code. How can I combine into a single observable stream data coming from two different collections that use a reference ID?
There are several examples online that I've read through and tried, each one using a different combination of operators with a different level of nested complexity. https://www.learnrxjs.io/ seems like a good resource, but it does not provide line-of-business examples that make sense to me. This question is very similar, and maybe the only difference is some translation into using RxFire? Still looking at that. Just for comparison, in SQL this would be a SELECT statement with an INNER JOIN on the reference ID.
Specifically, I have a collection for Games:
{ id: "abc000001", name: "Billiards" },
{ id: "abc000002", name: "Croquet" },
...
and a collection for Game Sessions:
{ id: "xyz000001", userId: "usr000001", gameId: "abc000001", duration: 30 },
{ id: "xyz000002", userId: "usr000001", gameId: "abc000001", duration: 45 },
{ id: "xyz000003", userId: "usr000001", gameId: "abc000002", duration: 55 },
...
And I want to observe a merged collection of Game Sessions where gameId is essentially replace with Game.name.
I current have a game-sessions-service.ts with a function to get sessions for a particular user:
import { collectionData } from 'rxfire/firestore';
import { Observable } from 'rxjs';
import { GameSession } from '../interfaces';
observeUserGameSesssions(userId: string): Observable<GameSession[]> {
let collectionRef = this.db.collection('game-sessions');
let query = collectionRef.where('userId', '==', userId);
return collectionData(query, 'id);
}
And I've tried variations of things with pipe and mergeMap, but I don't understand how to make them all fit together properly. I would like to establish an interface GameSessionView to represent the merged data:
export interface GameSessionView {
id: string,
userId: string,
gameName: string,
duration: number
}
observeUserGameSessionViews(userId: string): Observable<GameSessionView> {
this.observeUserGameSessions(userId)
.pipe(
mergeMap(sessions => {
// What do I do here? Iterate over sessions
// and embed other observables for each document?
}
)
}
Possibly, I'm just stuck in a normalized way of thinking, so I'm open to suggestions on better ways to manage the data. I just don't want too much duplication to keep synchronized.
You can use the following code (also available as Stackblitz):
const games: Game[] = [...];
const gameSessions: GameSession[] = [...];
combineLatest(
of(games),
of(gameSessions)
).pipe(
switchMap(results => {
const [gamesRes, gameSessionsRes] = results;
const gameSessionViews: GameSessionView[] = gameSessionsRes.map(gameSession => ({
id: gameSession.id,
userId: gameSession.userId,
gameName: gamesRes.find(game => game.id === gameSession.gameId).name,
duration: gameSession.duration
}));
return of(gameSessionViews);
})
).subscribe(mergedData => console.log(mergedData));
Explanation:
With combineLatest you can combine the latest values from a number of Obervables. It can be used if you have "multiple (..) observables that rely on eachother for some calculation or determination".
So assuming you lists of Games and GameSessions are Observables, you can combine the values of each list.
Within the switchMap you create new objects of type GameSessionView by iterating over your GameSessions, use the attributes id, userId and duration and find the value for gameName within the second list of Games by gameId. Mind that there is no error handling in this example.
As switchMap expects that you return another Observable, the merged list will be returned with of(gameSessionViews).
Finally, you can subscribe to this process and see the expected result.
For sure this is not the only way you can do it, but I find it the simplest one.
I'm developing a cloud based billing system and I have two tables in my database namely bill_history and sold_items. I want to store the Bill number, date, customer name, phone number and total amount and then to return the bill number from bill_history and store the array of objects containing item no, item name, price, quantity, amount with the returned bill no in sold_items. I'm using the following code:
app.post('/billed', (req, res) => {
const { items, total, date } = req.body;
console.log(items, total, date);
db.transaction(trx => {
db.insert({
total: total,
date: date,
}).into('billhead')
.transacting(trx)
.returning('billno')
.then(num => {
for (var i = 0; i < items.length; i++) {
trx.insert({
billno: num,
prodname: items[i].name,
quantity: items[i].quantity,
netprice: items[i].amount
}).into('billdetails')
}).then(trx.commit())
.catch(trx.rollback())
})
})
Now Entries are found in bill_history but not entered in sold_items. I can't find the mistake! Help me with this error. The console and terminal shows No Error
Important thing to remember when working with knex queries: they are promises and they will only execute if:
You call then on the knex object itself
You return the knex query inside a promise chain, and call then somewhere down the chain
Inside your for loop, you have only stated what the knex object should do and because of syntax errors didn't call then on the knex object itself.
.into('billdetails').then(inserts => { /// })
It does work if you return trx.insert()...
That being said, it wouldn't suit your use case, as when inserting multiple values inside a transaction, you need to make sure all inserts have been succesfull. Using for loops the way you did in async fashion is dangerous and won't guarantee all individual inserts have completed without errors and that it's safe to commit the transaction.
One way of achieving this in a safe manner would be modifying this section of your code:
// ...
.returning('billno')
.then(num => {
// We create an array of individual inserts
// Each element in the array will be a single knex
// object/promise that inserts one row into the database
const billDetailInserts = items.map(item => trx.insert({
billno: num,
prodname: item.name,
quantity: item.quantity,
netprice: item.amount
).into('billdetails')
})
// we utilize the Promise.all method that will resolve when
// all individual inserts have completed succesfully
return Promise.all(billDetailInserts);
})
.then(inserts => {
// ... commits, rollbacks, logging etc
If a collection have a list of dogs, and there is duplicate entries on some races. How do i remove all, but a single specific/non specific one, from just one query?
I guess it would be possible to get all from a Model.find(), loop through every index except the first one and call Model.remove(), but I would rather have the database handle the logic through the query. How would this be possible?
pseudocode example of what i want:
Model.remove({race:"pitbull"}).where(notFirstOne);
To remove all but one, you need a way to get all the filtered documents, group them by the identifier, create a list of ids for the group and remove a single id from
this list. Armed with this info, you can then run another operation to remove the documents with those ids. Essentially you will be running two queries.
The first query is an aggregate operation that aims to get the list of ids with the potentially nuking documents:
(async () => {
// Get the duplicate entries minus 1
const [doc, ...rest] = await Module.aggregate([
{ '$match': { 'race': 'pitbull'} },
{ '$group': {
'_id': '$race',
'ids': { '$push': '$_id' },
'id': { '$first': '$_id' }
} },
{ '$project': { 'idsToRemove': { '$setDifference': [ ['$id'], '$ids' ] } } }
]);
const { idsToRemove } = doc;
// Remove the duplicate documents
Module.remove({ '_id': { '$in': idsToRemove } })
})();
if purpose is to keep only one, in case of concurrent writes, may as well just write
Module.findOne({race:'pitbull'}).select('_id')
//bla
Module.remove({race:'pitbull', _id:{$ne:idReturned}})
If it is to keep the very first one, mongodb does not guarantee results will be sorted by increasing _id (natural order refers to disk)
see Does default find() implicitly sort by _id?
so instead
Module.find({race:'pitbull'}).sort({_id:1}).limit(1)
Is there a way to sort by a given array?
something like this:
const somes = await SomeModel.find({}).sort({'_id': {'$in': [ObjectId('sdasdsd), ObjectId('sdasdsd), ObjectId('sdasdsd)]}}).exec()
What i looking for is a way to get a solution, to get all document of the collection and sort by if the document's _id match with one of the given array.
An example:
we have albums collection and songs collection. In albums collection we store the ids of the songs that belongs to the albums.
I want to get the songs, but if the song is in the album take them front of the array.
I solved this as follow, but its looks a bit hacky:
const songs = await SongMode.find({}).skipe(limit * page).limit(limit).exec();
const album = await AlbumModel.findById(id).exec();
if(album) {
songArr = album.songs.slice(limit * page);
for(let song of album.songs) {
songs.unshift(song);
songs.pop();
}
}
This cannot be accomplished using an ordinary .find().sort(). Instead, you will need to use the MongoDB aggregation pipeline (.aggregate()). Specifically, you will need to do the following:
Perform a $projection such that if the _id is $in the array, your new sort_field is given the value 1, otherwise it's given a value of 0.
Perform a $sort such that you're doing a descending sort on the new sort_field.
If you're using MongoDB version 3.4 or greater, then this is easy because of the $addFields operator:
const your_array_of_ids = [
ObjectId('objectid1'),
ObjectId('objectid2'),
ObjectId('objectid3')
];
SomeModel.aggregate([
{ '$addFields': {
'sort_field': { '$cond': {
'if': { '$in': [ '$_id', your_array_of_ids ] },
'then': 1,
'else': 0
}}
}},
{ '$sort': {
'sort_field': -1
}}
]);
If you're using an older version of MongoDB, then the solution is similar, but instead of $addFields you will be using $project. Additionally, you will need to explicitly include all of the other fields you want included, otherwise they will be excluded from the results.