I'm wondering how I can compare arrays of (nested) objects in Mongoose.
Considering the data below, I would like to get results when the name properties match. Could anyone help me with this?
Organisation.find( {
$or: [
{ "category_list": { $in: cat_list } },
{ "place_topics.data": { $in: place_tops } }
]
}
)
Let's say that this is the data stored in my MongoDB:
"category_list": [
{
"id": "197750126917541",
"name": "Pool & Billiard Hall"
},
{
"id": "197871390225897",
"name": "Cafe"
},
{
"id": "218693881483234",
"name": "Pub"
}
],
"place_topics": {
"data": [
{
"name": "Pool & Billiard Hall",
"id": "197750126917541"
},
{
"name": "Pub",
"id": "218693881483234"
}
]
}
And let's say that these are the arrays I want to compare against (almost the same data):
let cat_list = [
{
"id": "197750126917541",
"name": "Pool & Billiard Hall"
},
{
"id": "197871390225897",
"name": "Cafe"
},
{
"id": "218693881483234",
"name": "Pub"
}
]
let place_tops = [
{
"name": "Pool & Billiard Hall",
"id": "197750126917541"
},
{
"name": "Pub",
"id": "218693881483234"
}
]
When there are "multiple conditions" required for each array element is when you actually use $elemMatch, and in fact "need to" otherwise you don't match the correct element.
So to apply multiple conditions, you would rather make an array of conditions for $or instead of shortcuts with $in:
Organizations.find({
"$or": [].concat(
cat_list.map( c => ({ "category_list": { "$elemMatch": c } }) ),
place_tops.map( p => ({ "place_topics": { "$elemMatch": p } }) )
)
})
However, if you take a step back and think logically about it, you actually named one of the properties "id". This would generally imply in all good practice that the value is in fact ""unique".
Therefore, all you really should need to do is simply extract those values and stick with the original query form:
Organizations.find({
"$or": [
{ "category_list.id": { "$in": cat_list.map(c => c.id) } },
{ "place_topics.id": { "$in": place_tops.map(p => p.id) } }
]
})
So simply mapping both the values and the property to "match" onto the "id" value instead. This is a simple "dot notation" form that generally suffices when you have one condition per array element to test/match.
That is generally the most logical approach given the data, and you should apply which one of these actually suits the data conditions you need. For "multiple" use $elemMatch. But if you don't need multiple because there is a singular match, then simply do the singular match
Related
I am trying to make a healthcheck on references in one of my collections. so to see if objects referenced to still exist and if not I want to delete that _id in the array
I haven't found anything to that so my idea is to get the reversed result of a $lookup
Is it possible to get the reversed result of a lookup in MongoDB?
Here is an example of a collection and its taskList with references to the tasks collection.
Now I want to delete all the id's in there that do not have an existing result in the tasks collection.
How I solve it right now which is tons of queries:
get all the ids from taskList
Send a query for every single one of them to see if there is no match with the task collection
Send a query to pull that empty reference out of the array
I think this does what you want, its ok even if you have big collections.
But its not an update you can do after that a $merge stage, to the tasklists (if match on _id replace)(requires MongoDB >= 4.4) or you can do a $out stage to another collection, and replace the tasklist collection.
Test code here
Data in
db={
"tasklists": [
{
"_id": 1,
"tasklist": [
1,
2,
3,
4
]
},
{
"_id": 2,
"tasklist": [
5,
6,
7
]
}
],
"tasks": [
{
"_id": 1
},
{
"_id": 2
},
{
"_id": 3
},
{
"_id": 5
}
]
}
db.tasklists.aggregate([
{
"$lookup": {
"from": "tasks",
"let": {
"tasklist": "$tasklist"
},
"pipeline": [
{
"$match": {
"$expr": {
"$in": [
"$_id",
"$$tasklist"
]
}
}
}
],
"as": "valid"
}
},
{
"$addFields": {
"valid": {
"$map": {
"input": "$valid",
"as": "v",
"in": "$$v._id"
}
}
}
},
{
"$addFields": {
"tasklist": {
"$filter": {
"input": "$tasklist",
"as": "t",
"cond": {
"$in": [
"$$t",
"$valid"
]
}
}
}
}
},
{
"$unset": [
"valid"
]
}
])
Results (tasks 4,6,7 wasnt found in the task collection,and removed)
[
{
"_id": 1,
"tasklist": [
1,
2,
3
]
},
{
"_id": 2,
"tasklist": [
5
]
}
]
Edit
If you want to use index to do the $lookup you can try this
Test code here
Tasks have index on _id so no need to make one, if you dont join on _id make one.
db.tasklists.aggregate([
{
"$unwind": {
"path": "$tasklist"
}
},
{
"$lookup": {
"from": "tasks",
"localField": "tasklist",
"foreignField": "_id",
"as": "joined"
}
},
{
"$match": {
"$expr": {
"$gt": [
{
"$size": "$joined"
},
0
]
}
}
},
{
"$unset": [
"joined"
]
},
{
"$group": {
"_id": "$_id",
"tasklist": {
"$push": "$tasklist"
},
"afield": {
"$first": "$afield"
}
}
}
])
After that you can do $out or $merge with replace option.
But both lose the updated data if any while this was happening.
Only solution for this(if it is a problem) $merge with pipeline,
You need to keep also in the pipeline above an extra array with the initial tasklist, so you remove the valid ones, to have the invalid ones, and then on merge with pipeline to filter the array, and just removed those invalid. (this is safe, from data loss)
I think the best approach instead of doing all those is to have an index on tasklist(multikey index) and when an _id is deleted from tasks,to delete the _id from the array in tasklist.With index it will be fast, so you dont need to check for invalid _ids.
Afaik there's no other way than you described in order to achieve the desired outcome, but you can greatly simplify the second step to find the non-matching items. In fact it's the set difference between the taskList-ids and the existing task-ids.
So you could use the $setDifference-operator to calculate that difference:
db.tasks.aggregate([
{
$group: {
_id: "null",
ids: {
"$addToSet": "$_id"
}
}
},
{
$project: {
nonMatchingTaskIds: {
$setDifference: [
[
"taskId1",
"taskId2",
"taskId7",
"taskId8"
],
"$ids"
]
}
}
}
])
Assuming your tasks collection contains taskId1, task2 (and other documents), but not taskId7 and taskId8, the query will result in nonMatchingTaskIds containing taskId7 and taskId8.
Here's an example on mongoplayground: https://mongoplayground.net/p/75BpiGBJi3Q
So what I came to do now is a few stepped method.
This is quite fast but sicne the taskIds collected from Sets are currently way smaller than the entire amount of sets I imagine working with the $setDifference operator mentioned by eol will be faster once I get that many references.
let taskIdsInSets = []
// Get all referenced task ids
const result = await this.setSchema.aggregate([
{
'$project': {
'taskList': 1
}
}
])
// Map all elements in one row
result.forEach(set => taskIdsInSets.push(...set.taskList.map(x=> x.toString())))
// Delete duplicates of taskIds here
taskIdsInSets.filter((item, index) => taskIdsInSets.indexOf(item) != index)
// Get the existing task ids that are referenced in a Set
const result2 = await this.taskSchema.aggregate([
{
'$match': {
'_id': {
'$in': [...taskIdsInSets.map(x => Types.ObjectId(x.toString()))]
}
}
}, {
'$project': {
'_id': 1
}
}
])
let existingIdsInTasks = []
// Getting ids from result2 Object into
result2.forEach(set => existingIdsInTasks.push(set._id.toString()))
// Filtering out the ids that don't actually exist
let nonExistingTaskIds = taskIdsInSets.filter(x => existingIdsInTasks.indexOf(x) === -1);
// Deleting the ids that don't actually exist but are in Sets
const finalResult = await this.setSchema.updateMany(
{
$pullAll: {
taskList: [...nonExistingTaskIds.map(x => Types.ObjectId(x.toString()))]
}
})
console.log(finalResult)
return finalResult // returns the information how much got changed. unfortunately in mongoose there isn't the option to use findAndModify with `{new:true}` or atleast I didn't manage to make it work.
for some reason what the database returns neither matches the Mongo ObjectId nor strings so I have to do some castings there.
I am working on versioning, We have documents based on UUIDs andjobUuids, andjobUuids are the documents associated with the currently working user. I have some aggregate queries on these collections which I need to update based on the job UUIDs,
The results fetched by the aggregate query should be such that,
if the current usersjobUuid document does not exist then the master document with jobUuid: "default" will be returned(The document without any jobUuid),
if job uuid exists then only the document is returned.
I have a$match used to get these documents based on certain conditions, from those documents I need to filter out the documents based on the above conditions, and an example is shown below,
The data looks like this:
[
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"name": "adam",
"jobUuid": "default",
},
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "adam"
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"name": "eve",
"jobUuid": "default",
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "eve"
},
{
"uuid": "26cba689-7eb6-4a9e-a04e-24ede0309e50",
"name": "john",
"jobUuid": "default",
}
]
Results for "jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12" should be:
[
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "adam"
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "eve"
},
{
"uuid": "26cba689-7eb6-4a9e-a04e-24ede0309e50",
"name": "john",
"jobUuid": "default",
}
]
Based on the conditions mentioned above, is it possible to filter the document within the aggregate query to extract the document of a specific job uuid?
Edit 1: I got the following solution, which is working fine, I want a better solution, eliminating all those nested stages.
Edit 2: Updated the data with actual UUIDs and I just included only the name as another field, we do have n number of fields which are not relevant to include here but needed at the end (mentioning this for those who want to use the projection over all the fields).
Update based on comment:
but the UUIDs are alphanumeric strings, as shown above, does it have
an effect on these sorting, and since we are not using conditions to
get the results, I am worried it will cause issues.
You could use additional field to match the sort order to be the same order as values in the in expression. Make sure you provide the values with default as the last value.
[
{"$match":{"jobUuid":{"$in":["d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12","default"]}}},
{"$addFields":{ "order":{"$indexOfArray":[["d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12","default"], "$jobUuid"]}}},
{"$sort":{"uuid":1, "order":1}},
{
"$group": {
"_id": "$uuid",
"doc":{"$first":"$$ROOT"}
}
},
{"$project":{"doc.order":0}},
{"$replaceRoot":{"newRoot":"$doc"}}
]
example here - https://mongoplayground.net/p/wXiE9i18qxf
Original
You could use below query. The query will pick the non default document if it exists for uuid or else pick the default as the only document.
[
{"$match":{"jobUuid":{"$in":[1,"default"]}}},
{"$sort":{"uuid":1, "jobUuid":1}},
{
"$group": {
"_id": "$uuid",
"doc":{"$first":"$$ROOT"}
}
},
{"$replaceRoot":{"newRoot":"$doc"}}
]
example here - https://mongoplayground.net/p/KrL-1s8WCpw
Here is what I would do:
match stage with $in rather than an $or (for readability)
group stage with _id on $uuid, just as you did, but instead of pushing all the data into an array, be more selective. _id is already storing $uuid, so no reason to capture it again. name must always be the same for each $uuid, so take only the first instance. Based on the match, there are only two possibilities for jobUuid, but this will assume it will be either "default" or something else, and that there can be more than one occurrence of the non-"default" jobUuid. Using "$addToSet" instead of pushing to an array in case there are multiple occurrences of the same jobUuid for a user, also, before adding to the set, use a conditional to only add non-"default" jobUuids, using $$REMOVE to avoid inserting a null when the jobUuid is "default".
Finally, "$project" to clean things up. If element 0 of the jobUuids array does not exist (is null), there is no other possibility for this user than for the jobUuid to be "default", so use "$ifNull" to test and set "default" as appropriate. There could be more than 1 jobUuid here, depending if that is allowed in your db/application, up to you to decide how to handle that (take the highest, take the lowest, etc).
Tested at: https://mongoplayground.net/p/e76cVJf0F3o
[{
"$match": {
"jobUuid": {
"$in": [
"1",
"default"
]
}
}
},
{
"$group": {
"_id": "$uuid",
"name": {
"$first": "$name"
},
"jobUuids": {
"$addToSet": {
"$cond": {
"if": {
"$ne": [
"$jobUuid",
"default"
]
},
"then": "$jobUuid",
"else": "$$REMOVE"
}
}
}
}
},
{
"$project": {
"_id": 0,
"uuid": "$_id",
"name": 1,
"jobUuid": {
"$ifNull": [{
"$arrayElemAt": [
"$jobUuids",
0
]
},
"default"
]
}
}
}]
I was able to solve this problem with the following aggregate query,
We are first extracting the results matching only the jobUuid provided by the user or the "default" in the match section.
Then the results are grouped based on the uuid, using a group stage and we are counting the results as well.
Using the conditions in replaceRoot first we are checking the length of the grouped document,
If the grouped document length is greater than or equal to 2, we are
filtering the document that matches the provided jobUuid.
If it's less or equal to the 1, then we are checking if it's matching the default jobUuid and returning it.
The Query is below:
[
{
$match: {
$or: [{ jobUuid:1 },{ jobUuid: 'default'}]
}
},
{
$group: {
_id: '$uuid',
count: {
$sum: 1
},
docs: {
$push: '$$ROOT'
}
}
},
{
$replaceRoot: {
newRoot: {
$cond: {
if: {
$gte: [
'$count',
2
]
},
then: {
$arrayElemAt: [
{
$filter: {
input: '$docs',
as: 'item',
cond: {
$ne: [
'$$item.jobUuid',
'default'
]
}
}
},
0
]
},
else: {
$arrayElemAt: [
{
$filter: {
input: '$docs',
as: 'item',
cond: {
$eq: [
'$$item.jobUuid',
'default'
]
}
}
},
0
]
}
}
}
}
}
]
I am trying in Javascript, using PUG template (if possible), to compare two arrays and when I find a correspondance in IDs, display some particular elements.
// First Array : I iterate over "hearts" object
// Called in PUG : - const user
[
{
"hearts": [
"5e70c63a94b27b164c9b897f",
"5e723c75e4bfdf4f58c55e32"
],
"_id": "5e6bb1189978fd5afc98c57a",
"email": "catherine#catherine.com",
"name": "Catherine",
"photo": "0121b7fe-b2ae-4e75-979d-7dea1a432855.jpeg",
"__v": 0
},
{
"hearts": [
"5e723c75e4bfdf4f58c55e32"
],
"_id": "5e6bc41f5915e3d2980a5174",
"email": "marc#marc.com",
"name": "Marc",
"photo": "4caa7bfb-6408-4893-a78b-fa6e8e5b03e7.png",
"__v": 0
}
]
// Second array : I iterate over "author.hearts" object
// Called in PUG : - const store
[{
"product": {
"categories": [
1,
2
]
},
"_id": "5e6bcc76c4022eae00e22af6",
"date": "2222-02-20T21:22:00.000Z",
"author": {
"hearts": [
"5e723c75e4bfdf4f58c55e32",
"5e70c63a94b27b164c9b897f"
],
"_id": "5e6bb1189978fd5afc98c57a",
"__v": 0
},
"created": "2020-03-13T18:09:58.086Z",
"id": "5e6bcc76c4022eae00e22af6"
}]
I want to loop over the first array, find the first ID (here 5e70c63a94b27b164c9b897f), loop over the second array and see if this ID is present within the "author.hearts" object. If it is not, carry on with the second ID and if it is present, display all the keys (tags, photos, _id, date...) from the object where the ID was found.
In my example, I have just one object in my array, but I'll be having much more later on.
Many thanks for your help
If I'm understanding correctly you can do something like this. Loop through all your users and when you find their id in author.hearts stop the loop there and return the object the user's _id was found in.
var resultFound = undefined;
try {
user.forEach((el) => {
const id = el._id;
const result = store.find(el => el.author.hearts.includes(id));
if (result) {
resultFound = result;
throw resultFound;
}
});
} catch (e) {
if (e !== resultFound) {
throw e;
}
}
My object is like:
"resources":[
{
"date": "2019-04-17",
"values":[
{"Customer":"C_name"},
{"Environment":66.5},
{"Other": {"goods": "feature", "present": 12}
}
]
I want to know how to validate values, I mean the values array should contain at least one customer object, an Environment object, and Other Objects. How to use #hapi/joi framework to validate this array and it should contain different objects in this array?
Using a combination of map, every and some:
const resources = [{
"date": "2019-04-17",
"values": [{
"Customer": "C_name"
},
{
"Environment": 66.5
},
{
"Other": {
"goods": "feature",
"present": 12
}
}
]
}]
function valid(arr) {
return arr
.map(x => x.values)
.every(y =>
y.some(z => z.hasOwnProperty("Customer"))
&& y.some(z => z.hasOwnProperty("Environment"))
&& y.some(z => z.hasOwnProperty("Other"))
)
}
console.log(valid(resources));
I'm creating a JSON object from an array and I want to dynamically push data to this JSON object based on the values from array. See my code for a better understanding of my problem...
for(i=0;i<duplicates.length; i++) {
var request = {
"name": duplicates[i].scope,
"id": 3,
"rules":[
{
"name": duplicates[i].scope + " " + "OP SDR Sync",
"tags": [
{
"tagId": 1,
"variables":[
{
"variable": duplicates[i].variable[j],
"matchType": "Regex",
"value": duplicates[i].scopeDef
}
],
"condition": false,
},
{
"tagId": 1,
"condition": false,
}
],
"ruleSetId": 3,
}
]
}
}
I take object properties from the duplicates array that can have the following elements:
[{scopeDef=.*, scope=Global, variable=[trackingcode, v1, v2]}, {scopeDef=^https?://([^/:\?]*\.)?delta.com/products, scope=Products Section, variable=[v3]}]
As you can see, an object contain variable element that can have multiple values. I need to push to the JSON object all those values dynamically (meaning that there could be more than 3 values in an array).
For example, after I push all the values from the duplicates array, my JSON object should look like this:
name=Products Section,
rules=
[
{
name=Products Section OP SDR Sync,
tags=[
{
variables=
[
{
matchType=Regex,
variable=v3,
value=^https?://([^/:\?]*\.)?delta.com/products
},
{
matchType=Regex,
variable=trackingcode,
value=.*
},
{
matchType=Regex,
variable=v1,
value=.*
},
{
matchType=Regex,
variable=v2,
value=.*
}
],
condition=false,
},
{
condition=false,
tagId=1
}
],
ruleSetId=3
}
]
}
I tried the following code but without success:
for(var j in duplicates[i].variable) {
var append = JSON.parse(request);
append['variables'].push({
"variable":duplicates[i].variable[j],
"matchType": "Regex",
"value": duplicates[i].scopeDef
})
}
Please let me know if I need to provide additional information, I just started working with JSON objects.
First of all, you dont need to parse request, you already create an object, parse only when you get JSON as string, like:
var json='{"a":"1", "b":"2"}';
var x = JSON.parse(json);
Next, you have any property of object wrapped in arrays. To correctly work with it you should write:
request.rules[0].tags[0].variables.push({
"variable":duplicates[i].variable[j],
"matchType": "Regex",
"value": duplicates[i].scopeDef
})
If you want to use your code snippet, you need some changes in request:
var request = {
"name": duplicates[i].scope,
"id": 3,
"variables":[
{
"variable": duplicates[i].variable[j],
"matchType": "Regex",
"value": duplicates[i].scopeDef
}
],
"rules":[
{
"name": duplicates[i].scope + " " + "OP SDR Sync",
"tags": [
{
"tagId": 1,
"condition": false,
},
{
"tagId": 1,
"condition": false,
}
],
"ruleSetId": 3,
}
]
}
}
To understand JSON remember basic rule: read JSON backward. It means:
property
object.property
arrayOfObfects['id'].object.property
mainObject.arrayOfObfects['id'].object.property
and so on. Good luck!