MongoDB: Fetching documents from different collections in one query - javascript

I have two different collections for two different type of products. Now, I want to fetch all documents from both collections for a particular user.
I know I can do that with 2 queries for each collection, merging them on the server side and sending the result to the user. Downside of this is that I have to fetch all documents for a user from both collections, which is not good for pagination. That is why I want to do it in one query, so I can leave a pagination logic to MongoDB as well.
Here is the example of collections and expected result:
Products_type_1
[
{
"name": "product_1",
"user": "user_1",
...
},
{
"name": "product_2",
"user": "user_2",
...
}
]
Products_type_2
[
{
"name": "product_3",
"user": "user_1",
...
},
{
"name": "product_4",
"user": "user_2",
...
}
]
The expected result:
[
{
"type": "Products_type_1",
"name": "product_1",
"user": "user_1",
...
},
{
"type": "Products_type_2",
"name": "product_3",
"user": "user_1",
...
}
]

You can use aggregation framework with $unionWith stage:
db.Products_type_1.aggregate([
{
"$match": {
"user": "user_1"
}
},
{
$unionWith: {
coll: "Products_type_2",
pipeline: [
{
"$match": {
"user": "user_1"
}
}
]
}
}
])
Playground: https://mongoplayground.net/p/v0dKCwiKsZU
If you want to use pagination you will need to add sort stage to ensure consistent order of the documents in the result.

Firstly I would query the logic of having a different collection for the different 'product_type_x'. If you had a single collection with an added field...
{ "productType" : 1,
...
},
That way that issue has just been resolved, everything to do with Procts is now accessible in a single collection. Aggregation of your data now becomes simple (by comparison)

Related

mongo aggregate based on conditions to filter the document for versioning

I am working on versioning, We have documents based on UUIDs andjobUuids, andjobUuids are the documents associated with the currently working user. I have some aggregate queries on these collections which I need to update based on the job UUIDs,
The results fetched by the aggregate query should be such that,
if the current usersjobUuid document does not exist then the master document with jobUuid: "default" will be returned(The document without any jobUuid),
if job uuid exists then only the document is returned.
I have a$match used to get these documents based on certain conditions, from those documents I need to filter out the documents based on the above conditions, and an example is shown below,
The data looks like this:
[
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"name": "adam",
"jobUuid": "default",
},
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "adam"
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"name": "eve",
"jobUuid": "default",
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "eve"
},
{
"uuid": "26cba689-7eb6-4a9e-a04e-24ede0309e50",
"name": "john",
"jobUuid": "default",
}
]
Results for "jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12" should be:
[
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "adam"
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "eve"
},
{
"uuid": "26cba689-7eb6-4a9e-a04e-24ede0309e50",
"name": "john",
"jobUuid": "default",
}
]
Based on the conditions mentioned above, is it possible to filter the document within the aggregate query to extract the document of a specific job uuid?
Edit 1: I got the following solution, which is working fine, I want a better solution, eliminating all those nested stages.
Edit 2: Updated the data with actual UUIDs and I just included only the name as another field, we do have n number of fields which are not relevant to include here but needed at the end (mentioning this for those who want to use the projection over all the fields).
Update based on comment:
but the UUIDs are alphanumeric strings, as shown above, does it have
an effect on these sorting, and since we are not using conditions to
get the results, I am worried it will cause issues.
You could use additional field to match the sort order to be the same order as values in the in expression. Make sure you provide the values with default as the last value.
[
{"$match":{"jobUuid":{"$in":["d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12","default"]}}},
{"$addFields":{ "order":{"$indexOfArray":[["d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12","default"], "$jobUuid"]}}},
{"$sort":{"uuid":1, "order":1}},
{
"$group": {
"_id": "$uuid",
"doc":{"$first":"$$ROOT"}
}
},
{"$project":{"doc.order":0}},
{"$replaceRoot":{"newRoot":"$doc"}}
]
example here - https://mongoplayground.net/p/wXiE9i18qxf
Original
You could use below query. The query will pick the non default document if it exists for uuid or else pick the default as the only document.
[
{"$match":{"jobUuid":{"$in":[1,"default"]}}},
{"$sort":{"uuid":1, "jobUuid":1}},
{
"$group": {
"_id": "$uuid",
"doc":{"$first":"$$ROOT"}
}
},
{"$replaceRoot":{"newRoot":"$doc"}}
]
example here - https://mongoplayground.net/p/KrL-1s8WCpw
Here is what I would do:
match stage with $in rather than an $or (for readability)
group stage with _id on $uuid, just as you did, but instead of pushing all the data into an array, be more selective. _id is already storing $uuid, so no reason to capture it again. name must always be the same for each $uuid, so take only the first instance. Based on the match, there are only two possibilities for jobUuid, but this will assume it will be either "default" or something else, and that there can be more than one occurrence of the non-"default" jobUuid. Using "$addToSet" instead of pushing to an array in case there are multiple occurrences of the same jobUuid for a user, also, before adding to the set, use a conditional to only add non-"default" jobUuids, using $$REMOVE to avoid inserting a null when the jobUuid is "default".
Finally, "$project" to clean things up. If element 0 of the jobUuids array does not exist (is null), there is no other possibility for this user than for the jobUuid to be "default", so use "$ifNull" to test and set "default" as appropriate. There could be more than 1 jobUuid here, depending if that is allowed in your db/application, up to you to decide how to handle that (take the highest, take the lowest, etc).
Tested at: https://mongoplayground.net/p/e76cVJf0F3o
[{
"$match": {
"jobUuid": {
"$in": [
"1",
"default"
]
}
}
},
{
"$group": {
"_id": "$uuid",
"name": {
"$first": "$name"
},
"jobUuids": {
"$addToSet": {
"$cond": {
"if": {
"$ne": [
"$jobUuid",
"default"
]
},
"then": "$jobUuid",
"else": "$$REMOVE"
}
}
}
}
},
{
"$project": {
"_id": 0,
"uuid": "$_id",
"name": 1,
"jobUuid": {
"$ifNull": [{
"$arrayElemAt": [
"$jobUuids",
0
]
},
"default"
]
}
}
}]
I was able to solve this problem with the following aggregate query,
We are first extracting the results matching only the jobUuid provided by the user or the "default" in the match section.
Then the results are grouped based on the uuid, using a group stage and we are counting the results as well.
Using the conditions in replaceRoot first we are checking the length of the grouped document,
If the grouped document length is greater than or equal to 2, we are
filtering the document that matches the provided jobUuid.
If it's less or equal to the 1, then we are checking if it's matching the default jobUuid and returning it.
The Query is below:
[
{
$match: {
$or: [{ jobUuid:1 },{ jobUuid: 'default'}]
}
},
{
$group: {
_id: '$uuid',
count: {
$sum: 1
},
docs: {
$push: '$$ROOT'
}
}
},
{
$replaceRoot: {
newRoot: {
$cond: {
if: {
$gte: [
'$count',
2
]
},
then: {
$arrayElemAt: [
{
$filter: {
input: '$docs',
as: 'item',
cond: {
$ne: [
'$$item.jobUuid',
'default'
]
}
}
},
0
]
},
else: {
$arrayElemAt: [
{
$filter: {
input: '$docs',
as: 'item',
cond: {
$eq: [
'$$item.jobUuid',
'default'
]
}
}
},
0
]
}
}
}
}
}
]

Is it appropriate to use Apollo local state to store a different representation of returned data

I have an Apollo Client that I'm using to request data from a service. I want to use the data I get in response to create a network of nodes and links e.g.
// Response data:
{
"Team": [
{
"name": "Example Team",
"members": [
{ "name": "Bob" },
{ "name": "Alice" }
]
}
]
}
// Network data:
{
"nodes": [
{ "name": "Example Team" }
{ "name": "Bob" },
{ "name": "Alice" }
],
"links": [
{ "source": "Example Team", "target": "Bob" },
{ "source": "Example Team", "target": "Alice" }
]
}
Historically, before using GraphQL, I would have used Redux to store the munged API response in state and read from there.
Is it appropriate to take a GraphQL result from Apollo and immediately save it back to Apollo local state in a different form so it can be queried by components in that format?
The main problem I foresee is that I think I'd have to query to check if the data I want exists in local state, then make another query if it didn't. In a Redux-world this would be wrapped up inside my store which would then only make the request off to the API if it didn't have the data it needed which 'feels' much cleaner.
In my case this could be solved using Afterware in Apollo Client, see this answer for more information.
It would allow me to munge the returned data into the form I need, and return it in the response alongside the original data.

Counting relationships in many to many table between joined tables in Sequelize.js

I am building a project using sequelize.js that includes a Tags table and a Stories table. They have a many to many relationship, which I created in sequelize with a through table of StoryTag. This all works perfectly so far, but I want to get a list of most popluar tags, as in how many stories they are associated with in the StoryTag table, and order them by the number of stories that use this tag.
This is the MySQL syntax of what I am trying to do. This works perfectly in MySQL Workbench:
SELECT tagName, COUNT(StoryTag.TagId)
FROM Tags
LEFT JOIN StoryTag on Tags.id = StoryTag.TagId
GROUP BY Tags.tagName ORDER BY COUNT(StoryTag.TagId) DESC;
This is what works in sequelize.js. It's a raw query, which is not ideal, but since this doesn't handle any sensitive information, it's not a huge worry, just very inelegant.
//DIRECT QUERY METHOD (TEST)
app.get("/api/directags", function (req, res) {
db.sequelize.query("select tags.id, tags.TagName, COUNT(stories.id) as num_stories
from tags left join storytag on storytag.TagId = tags.id
left join stories on storytag.StoryId = stories.id
group by tags.id order by num_stories desc;", {
type: db.Sequelize.QueryTypes.SELECT
}).then(function(result) {
res.send(result);
});
});
This outputs
[
{
"id": 3,
"TagName": "fiction",
"num_stories": 3
},
{
"id": 5,
"TagName": "Nursery Rhyme",
"num_stories": 2
},
...
{
"id": 4,
"TagName": "nonfiction",
"num_stories": 0
}
]
As it should. What doesn't quite work is:
//Sequelize count tags
//Known issues: will not order by the count
//Includes a random 'storytag' many-to-many table row for some reason
app.get("/api/sequelizetags", function (req, res) {
db.Tag.findAll({
attributes: ["id","TagName"],
include: [{
model: db.Story,
attributes: [[db.sequelize.fn("COUNT", "stories.id"), "Count_Of_Stories"]],
duplicating: false
}],
group: ["id"]
}).then(function (dbExamples) {
res.send(dbExamples);
});
});
Which outputs:
[
{
"id": 1,
"TagName": "horror",
"Stories": [
{
"Count_Of_Stories": 1,
"StoryTag": {
"createdAt": "2018-11-29T21:09:46.000Z",
"updatedAt": "2018-11-29T21:09:46.000Z",
"StoryId": 1,
"TagId": 1
}
}
]
},
{
"id": 2,
"TagName": "comedy",
"Stories": []
},
{
"id": 3,
"TagName": "fiction",
"Stories": [
{
"Count_Of_Stories": 3,
"StoryTag": {
"createdAt": "2018-11-29T21:10:04.000Z",
"updatedAt": "2018-11-29T21:10:04.000Z",
"StoryId": 1,
"TagId": 3
}
}
]
},
{
"id": 4,
"TagName": "nonfiction",
"Stories": []
},
...
{
"id": 8,
"TagName": "Drama",
"Stories": [
{
"Count_Of_Stories": 1,
"StoryTag": {
"createdAt": "2018-11-30T01:13:56.000Z",
"updatedAt": "2018-11-30T01:13:56.000Z",
"StoryId": 3,
"TagId": 8
}
}
]
},
{
"id": 9,
"TagName": "Tragedy",
"Stories": []
}
]
This is not in order, and the count of stories is buried. This seems like the sort of thing that would be a common and frequent request from a database, but I am at a loss of how to do this correctly with sequelize.js.
Resources that have failed me:
Sequelize where on many-to-many join
Sequelize Many to Many Query Issue
How to query many-to-many relationship data in Sequelize
Select from many-to-many relationship sequelize
The official documentation for sequelize: http://docs.sequelizejs.com/manual/tutorial/
Some less official and more readable documentation for sequelize: https://sequelize.readthedocs.io/en/v3/docs/querying/
Here's what finally worked, in case anyone else has this question. We also added a where to the include Story, but that's optional.
This resource is easier to understand than the official sequelize docs: https://sequelize-guides.netlify.com/querying/
I also learned that being familiar with promises is really helpful when working with sequelize.
db.Tag.findAll({
group: ["Tag.id"],
includeIgnoreAttributes:false,
include: [{
model: db.Story,
where: {
isPublic: true
}
}],
attributes: [
"id",
"TagName",
[db.sequelize.fn("COUNT", db.sequelize.col("stories.id")), "num_stories"],
],
order: [[db.sequelize.fn("COUNT", db.sequelize.col("stories.id")), "DESC"]]
}).then(function(result){
return result;
});
Please, use the same name if you mean the same thing (num_stories - Count_Of_Stories, etc.).
For ordering use order option.
Include count in top level attributes for get it on top level of instance.
I can't find include[].duplicating option in doc.
Your case:
db.Tag.findAll({
attributes: [
"id",
"TagName",
[db.sequelize.fn("COUNT", "stories.id"), "Count_Of_Stories"]
],
include: [{
model: db.Story,
attributes: [],
duplicating: false
}],
group: ["id"],
order: [
[db.sequelize.literal("`Count_Of_Stories`"), "DESC"]
]
});
Use through: {attributes: []} in options

DocumentDB: How to filter document on array within array?

Let's say I have the following document:
{
"Id": "1",
"Properties": [
{
"Name": "Name1",
"PropertyTypes": [
"Type1"
]
},
{
"Name": "Name2",
"PropertyTypes": [
"Type1",
"Type2",
"Type3"
]
}
]
}
When I use the following SQL:
SELECT c.Id FROM c
JOIN p in c.Properties
WHERE ARRAY_CONTAINS(p.PropertyTypes,"Type1")
I get as return:
[
{
"Id": "1"
},
{
"Id": "1"
}
]
How do I change my query so that it only returns distinct documents?
As far as I know, Distinct hasn't supported by Azure Cosmos DB yet.
It seems that there is no way to remove the repeat data in the query SQL level.
You could handle with your query result set in the loop locally.
However, if your result data is large,I suggest you using a stored procedure to handle with result data in Azure Cosmos DB to release the pressure on your local server.
You could refer to the official tutorial about SP.

How to create Backbone collection from following json array?

I am getting a following JSON(/users.json) which contains users:
[
[
{ "id": "43343", "project_id": "1", "username": "Amy" }
{ "id": "34244", "project_id": "1", "username": "Tommy" }
],
[
{ "id": "76575", "project_id": "2", "username": "Izzy" }
{ "id": "13322", "project_id": "2", "username": "Sam" }
],
{ "id": "09983", "project_id": "3", "username": "Max" }
]
When project has one user I get one user hash which is not in array.
I would like to build a Backbone collection with all users. How to do that?
You provide an array of arrays of users. So to fetch all users in the init method, you can give it data but as an array of users, i.e. you will flatten this original array of arrays once with underscore flatten method :
data = _(data).flatten(true);
Then the collection constructor will natively understand your json array.
But maybe you already do this transformation in the fetching method and this is not the problem you are facing..
If you have defined a collection (say userCollection) with a user model you should be able to simply do something like this:
var col;
$.getJSON("/users.json", function(data) {
col = new userCollection(data);
});
This would more likely be done in the fetch function of the collection, but the principle here is that you can pass an array of objects to a collection and it will marshal all from json to backbone models.

Categories

Resources