I currently have this schema
var dataSchema = new Schema({
hid: { type: String },
sensors: [{
nid: { type: String },
sid: { type: String },
data: {
param1: { type: String },
param2: { type: String },
data: { type: String }
},
date: { type: Date, default: Date.now }
}],
actuators: [{
nid: { type: String },
aid: { type: String },
control_id: { type: String },
data: {
param1: { type: String },
param2: { type: String },
data: { type: String }
},
date: { type: Date, default: Date.now }
}],
status: [{
nid: {type: String},
status_code: {type: String},
date: { type: Date, default: Date.now }
}],
updated: { type: Date, default: Date.now },
created: { type: Date }
});
The query that I'm trying to build should search the schema by "hid", then only pick from the "sensors", "actuators" and "status" arrays the objects that match the nid that i provide and then also limit the result to 10 element for each array.
Actually, don't use multiple pipeline phases when "one" will do. Every pipeline stage you include is effectively adding "time" to processing since it's another pass through the data.
So what logically works in a "single" stage, should stay in a "single" stage:
Data.aggregate([
{ "$match": { "hid": hid } },
{ "$project": {
"sensors": {
"$slice": [
{ "$filter": {
"input": "$sensors",
"as": "sensor",
"cond": { "$eq": [ "$$sensor.nid", nid ] }
}},
-10
]
},
"actuators": {
"$slice": [
{ "$filter": {
"input": "$actuators",
"as": "actuator",
"cond": { "$eq": [ "$$actuator.nid", nid ] }
}},
-10
]
},
"status": {
"$slice": [
{ "$filter": {
"input": "$status",
"as": "status",
"cond": { "$eq": [ "$$status.nid", nid ] }
}},
-10
]
},
"updated": 1,
"created": 1
}}
])
Also, it's not necessary to use "_id": 1 in inclusions since "_id" is always included unless explicitly "excluded".
The main case is to try not to create unnecessary stages, since it's bad for performance. Good indicators of this are:
$project followed by $project, usually means you can do this in one stage.
$project followed by $group is probably going to get compacted by the "optimizer" anyway, but you "should" get in the habit of of combining the two.
$project followed by $match, should indicate that you probably should be using $redact in a single stage instead. Since you likely did the $project to produce fields based on calculations, that were then considered in the $match.
And finally:
$unwind before a $match, really should have been a $filter in a $project "before" the $unwind. Since it does in fact work a lot faster to "filter" within the document, and also saves cost on processing the $unwind, due to less output from already filtered content.
But one more:
All of the "array" operations such as $map, $filter, $slice and even the "set operators" should always be used "in-line" with each other, in cases where those are manipulating the same array content. The same now even applies to wrapping with $sum or $max now that these can work directly on arrays themselves, and often with the strange looking but effective:
{ "$sum": { "$sum": "$array" } }
structure within a $group stage.
This can be achieved with the aggregation framework.
The query look like this:
Data.aggregate([
{ "$match": { "hid": hid } },
{ "$project": {
"_id": 1,
"sensors": {
"$filter": { "input": "$sensors", "as": "sensor", "cond": { "$eq": [ "$$sensor.nid", nid ] } }
},
"actuators": {
"$filter": { "input": "$actuators", "as": "actuator", "cond": { "$eq": [ "$$actuator.nid", nid ] } }
},
"status": {
"$filter": { "input": "$status", "as": "state", "cond": { "$eq": [ "$$state.nid", nid ] } }
},
"updated": 1,
"created": 1
}},
{ "$project": {
"_id": 1,
"sensors": {
"$slice": [ "$sensors", -10 ]
},
"actuators": {
"$slice": [ "$actuators", -10 ]
},
"status": {
"$slice": [ "$status", -10 ]
},
"updated": 1,
"created": 1
}}
]).exec(function(err,data) {
});
It uses the $match to find the schema, the $filter to pick from the array only the elements that match the provided nid and then uses the $slice to pick the last 10 elements from the filtered array
Related
I have an object as follows.
cityConfiguration -
{
_id: 62e135519567726de42421c2,
configType: 'cityConfig'
'16': {
cityName: 'Delhi',
metro: true,
tier1: true
},
'17': {
cityName: 'Indore',
metro: false,
tier1: false
}
}
I have another table called BusDetails with fields like - [cityId, revenue, tax...]. Example document -
{
"busNumber": "KA25D5057",
"cityId": "17",
"revenue": 46594924,
"totalCollection": 3456342
"tax": "2906",
"passengerCount": 40
......
}
I want to do an aggregation ($project) on the BusDetails table such that I take the cityId from that BusDetails document and if that cityId is a metro city (according to the cityConfig object) then I will return 0 or 1. I have designed the below code but its not working. How to fix it.
return await BsDetails.aggregate([
{
$match: { busNumber: 'KA25D5057' }
},
{
$project: {
totalCollection: 1,
passengerCount: 1,
.
.
.
metro: {"$cond": [ cityPassConfig["$cityId"].metro == true, 1, 0]},
}
}
]);
So for ex, in the above code, the cityId is 17 which is a non-metro city. So the "metro" field should store 0. Here I am facing problem in
cityPassConfig["$cityId"].metro
this particular statement.
Using objects with dynamic field name is generally considered as anti-pattern and should be avoided since it introduces unnecessary complexity into queries. Nevertheless, you can use $objectToArray to achieve what you want.
db.BusDetails.aggregate([
{
$match: {
busNumber: "KA25D5057"
}
},
{
"$addFields": {
// put your object here
"cityConfiguration": {
_id: ObjectId("62e135519567726de42421c2"),
configType: "cityConfig",
"16": {
cityName: "Delhi",
metro: true,
tier1: true
},
"17": {
cityName: "Indore",
metro: false,
tier1: false
}
}
}
},
{
"$addFields": {
// convert into array of k-v tuples
"cityConfiguration": {
"$objectToArray": "$cityConfiguration"
}
}
},
{
"$addFields": {
"cityConfiguration": {
// iterate through the array and get the matched cityConfiguration entry
"$reduce": {
"input": "$cityConfiguration",
"initialValue": null,
"in": {
"$cond": {
"if": {
$eq: [
"$$this.k",
"$cityId"
]
},
"then": "$$this.v",
"else": "$$value"
}
}
}
}
}
},
{
$project: {
totalCollection: 1,
passengerCount: 1,
metro: {
"$cond": {
"if": "$cityConfiguration.metro",
"then": 1,
"else": 0
}
}
}
}
])
Here is the Mongo playground for your reference.
I have a questions collection with _id and name and other fields(..), and a tests collection with _id, name and array of questions.
I'm trying to get all the questions with their fields and adding a field "usedIn" which counts the number of tests that the specific questions is present in.
questions = await Question.aggregate([
/*{
"$match": {
params
}
},*/
{
"$lookup": {
"from": "tests",
"let": {"questionId": "$_id"},
pipeline: [
{
"$match": {
"$expr": {
"$in": ["$$questionId", "$questions"],
},
},
},
],
as: "tests"
}
},
{
"$addFields": {
"usedIn": {
"$size": "tests"
}
}
},
{
"$project": fieldsObject
},
])
This code is giving me this error:
Error: Failed to optimize pipeline :: caused by :: The argument to $size must be an array, but was of type: string
What Am I doing wrong ?
You can do it like this:
db.questions.aggregate([
{
"$lookup": {
"from": "tests",
"localField": "_id",
"foreignField": "questions",
"as": "usedIn"
}
},
{
"$project": {
"usedIn": {
"$size": "$usedIn"
},
"name": 1
}
}
])
Working example
I have a collection in database that I am trying to retrieve some data from it , the query is working fine when $orderID has string elements , but is failing when $orderID has some numbers in array , and it is throwing
query failed: (Location40395) PlanExecutor error during aggregation :: caused by :: $arrayToObject requires an array of key-value pairs, where the key must be of type string. Found key type: double
I think there must be some old data when we were saving orderID as a number so that is why it is failing from some range of dates
Query
{
"Order_Details": {
"$map": {
"input": {
"$objectToArray": {
"$arrayToObject": {
"$zip": {
"inputs": [
"$orderID",
"$total_value_of_order"
]
}
}
}
},
"as": "el",
"in": {
"orderID": "$$el.k",
"total_value_of_order": "$$el.v"
}
}
}
}
I am trying to typecast el.k to string I am using $toString but can't seem to work , the way I am trying it is
{
"as": "el",
"in": {
"orderID": {
"$toString": "$$el.k"
},
"total_value_of_order": "$$el.v"
}
}
Example collection
[
{
"_id": ObjectId("5e529ee5f8647eb59e5620a2"),
"visitID": "dVmy7flXFHzzkn9HiMt8IoWvthoTZW",
"date": ISODate("2022-02-08T16:29:13.413Z"),
"control": true,
"orderID": [
122343242
],
"target": "test",
"total_value_of_order": [
60
]
}
]
You are close, the approach is fine. you just have a couple of syntax issues.
The major thing that needs to change is the input for $arrayToObject, currently your input looks like this:
[[number, number], [number, number]]
However $arrayToObject expects input in a certain format:
[{k: string, v: value}]
So this it what we'll add, like so:
db.collection.aggregate([
{
$project: {
"Order_Details": {
"$map": {
"input": {
"$objectToArray": {
"$arrayToObject": {
$map: {
input: {
"$zip": {
"inputs": [
"$orderID",
"$total_value_of_order"
]
}
},
in: {
k: {
$toString: {
"$arrayElemAt": [
"$$this",
0
]
}
},
v: {
"$arrayElemAt": [
"$$this",
1
]
}
}
}
}
}
},
"as": "el",
"in": {
"orderID": "$$el.v",
"total_value_of_order": "$$el.k"
}
}
}
}
}
])
Mongo Playground
Notice the "orderid" format changes to string which affects it's structure, I recommend just switching between the k and v in the pipeline, like this
I have stacked in a nested object. here is my collection.
{
"key": 1,
"subKey": ""
},
{
"key": 2,
"subKey": 1
},
{
"key": 3,
"subKey": 2
},
{
"key": 4,
"subKey": 3
}
I want to query Key:4, which gives me result
{
"key": 4,
"subKey": 3
}
after getting result i want to query "subKey": 3 as a key:"$subKey" and i want to run a loop, until i find a empty subKey in our case It is Key:1. and whenever i found an empty subKey i want it document as a parent.
In the end, I want the result
{
"key": 4,
"parent":{"key":1,"subKey":"",....}
}
or similar.
Is it possible by using MongoDB built-in function? if not available how do I achieve this goal?
also, I want an alternative solution for it if there is.
You can achieve using $graphLookup
play
db.collection.aggregate([
{
$graphLookup: {
from: "collection",
startWith: "$key",
connectFromField: "subKey",
connectToField: "key",
as: "keys"
}
}
])
If you want a match filter add it,
play
db.collection.aggregate([
{
$match: {
key: 4
}
},
{
$graphLookup: {
from: "collection",
startWith: "$key",
connectFromField: "subKey",
connectToField: "key",
as: "keys"
}
}
])
Important consideration:
The $graphLookup stage must stay within the 100 MiB memory limit. If allowDiskUse: true is specified for the aggregate() operation, the $graphLookup stage ignores the option
To transform the data, you cannot have duplicate keys in parent object. So parent should be an array
play
db.collection.aggregate([
{
$match: {
key: 4
}
},
{
$graphLookup: {
from: "collection",
startWith: "$key",
connectFromField: "subKey",
connectToField: "key",
as: "keys"
}
},
{
"$addFields": {
"parent": {
"$map": {
"input": "$keys",
"as": "res",
"in": {
"key": "$$res.key",
"subKey": "$$res.subKey"
}
}
},
"key": "$key",
}
},
{
$project: {
keys: 0
}
}
])
I need to extract all the attributes which are of numeric types. For example, if the different attributes are
{
age: 32
gender: "female"
year: 2020
name: "Abc"
}
My query should return ["age","year"]
I think the below query should help you out.
db.test.aggregate([
// Remove this `$limit` stage if your Collection schema is dynamic and you want to process all the documents instead of just one
{
"$limit": 1
},
{
"$project": {
"arrayofkeyvalue": {
"$filter": {
"input": {"$objectToArray":"$$ROOT"},
"as": "keyValPairs",
"cond": {
"$in": [{"$type": "$$keyValPairs.v"}, ["double", "int", "long"]],
// Change the above line to the following to get only `int` keys instead of `int, double` and `long`:
// "$eq": [{"$type": "$$keyValPairs.v"}, "int"],
}
}
}
}
},
{
"$group": {
"_id": null,
"unique": {"$addToSet": "$arrayofkeyvalue.k"}
}
},
{
"$project": {
"_id": 0,
"intKeyNames": {
"$reduce": {
input: "$unique",
initialValue: [],
in: {$setUnion : ["$$value", "$$this"]}
}
}
}
},
])
The above query result will be something like this:
{
"intKeyNames" : [
"_id",
"abc",
"paymentMonth",
"paymentYear",
"value"
]
}