Hi i have a mongo scheme called "payments" that has a 2 keys that optional:
userId or representativeId (if userId exist representativeId not exist and same about representativeId).
When i found the payments scheme based on cheque scheme that using the $match to filter the result based on my data, I bring all payments in the first lookup that match for my query , in the second and third lookup I want to bring the user data / representative data.
Maybe it will be mix of them its ok, its what I want to see if one of them does not exist the other must to be exist.
I want to get the final array that include the user or the representative or mix of them in the same array.
I am using aggregate to implement this.
the problem its give me back a empty array when 2 of the lookup show(user and representative),
but when I comment the lookup of the user or the representative, and i left with 2 lookup one for payment and after that last lookup user / representative and its work me like I want but just for user / representative just if i remove one of the lookup.
I want its bring my array with two of them.
const userAndRepData = await ChequeDB.aggregate<{[key: string]: any}>([
{
$match: {
$and: [
{
'chequeNumber': {
$in: chequeData.map(c => c.chequeNumber)
},
'accountNumber': {
$in: chequeData.map(c => c.accountNumber)
}
}
]
}
},
{
$lookup: {
from: 'payments',
localField: 'paymentId',
foreignField: '_id',
as: 'payment'
}
},
{
$unwind:'$payment'
},
{
$lookup: {
from: 'representatives',
localField: 'payment.representativeId',
foreignField: '_id',
as: 'representative'
}
},
{
$unwind: '$representative'
},
{
$lookup: {
from: 'users',
localField: 'payment.userId',
foreignField: '_id',
as: 'user'
}
},
{
$unwind: '$user'
},
{
$project: {
user: 1,
_id: 0,
representative: 1
}
}
]);
Unfortunately, you can't use data from one $lookup in the next one since the data from $lookup still hasn't been loaded at the moment query is executing, i.e. payment.representativeId will be null.
You can write resource intensive queries that would work around this, but the easiest and best (performance-wise) way to execute this query would be to have representativeId and userId stored on the Cheque collection.
Related
How to filter products by deep nested populated fields. catalogProduct is an ObjectId (ref to catalog product). category is an ObjectId inside catalogProduct (ref to categories). Categories is an array of category ids.
products = await StorageModel
.find({"catalogProduct.category": {$in: categories }})
.skip((page-1)*8)
.limit(8)
.populate({path: "catalogProduct", populate: {path: "category", select: "name"}})
.select('-__v')
.sort({_id: -1});
You'll need to do a $lookup on the catelogProduct collection so that you can access the catelogProduct data in the query.
Unfortunately that's only available when using Mongo Aggregation, however aggregation is very powerful and is perfect for this sort of thing. You could do something like this:
const products = await StorageModel.aggregate([
{ $lookup: { // Replace the Catelog Product ID with the Catelog Product
from: "catelogProduct",
localField: "catelogProduct",
foreignField: "_id",
as: "catelogProduct"
} },
{ $lookup: { // Replace the Category ID with the Category
from: "categories",
localField: "catelogProduct.category",
foreignField: "_id",
as: "catelogProduct.category"
} },
{ $addFields: { // Replace the Category with its name
"catelogProduct.category": "$catelogProduct.category.name"
} },
{ $match: {
"catalogProduct.category": { $in: categories }
} },
{ $sort: { _id: -1 } },
{ $skip: (page - 1) * 8 },
{ $limit: 8 }
]);
Ideally you wouldn't do the $lookup until you've paginated the results (using $skip and $limit), but in this case it makes sense to do the $lookup first. Make sure you've got an index on catelogProduct._id and categories._id to optimize the query.
For more info on $lookup, look at this article. For more info on Mongo Aggregation, look at this article.
I am trying to develop a personal project, a website that functions in a similair way to Stack Exchange, a user can ask a question which can receive multiple answers. Each question and answer can have multiple comments.
I am using nodeJS for my backend.
How can I fetch all comments for all the answers on a particular question in a single mongoDB / mongoose query?
It would be even more helpful if you could tell me how to fetch all comments for all the answers on a particular question as well as all the comments for the question in a single mongoDB / mongoose query?
Mongoose Schemas:
const questionSchema = new mongoose.Schema({
title: String,
content: String
})
const answerSchema = new mongoose.Schema({
questionId: String,
content: String,
})
const commentSchema = new mongoose.Schema({
idQuestion: String, // nullable
idAnswer: String, // nullable
content: String
})
Currently, I am performing a mongoose query to find all the answers for a particular questions. Then, using forEach, performing a mongoose query on each answer to find all the comments for each answer. I believe this is very taxing, performance wise and is not an ideal way to do what I would like to achieve.
You can try below aggregation. Match on question id followed by join to lookup all the answers ids with question id followed by lookup to pull in all comments.
db.questions.aggregate([
{"$match":{"_id":input_question_id}},
{"$lookup":{
"from":"answers",
"localField":"_id",
"foreignField":"questionId",
"as":"answers"
}},
{"$lookup":{
"from":"comments",
"let":{"ids":{"answers_id":"$answers._id","question_id":"$_id"}},
"pipeline":[
{"$match":{"$expr":{
"$or":[
{"$eq":["$idQuestion","$$ids.question_id"]},
{"$in":["$idAnswer","$$ids.answers_id"]}
]
}}}
],
"as":"comments"
}},
{"$project":{"comments":"$comments.content"}}
])
Working example here - https://mongoplayground.net/p/qBlKqk-JsxA
You can try,
$match your conditions questionId
$lookup join with comments
db.answers.aggregate([
{ $match: { questionId: 1 } },
{
$lookup: {
from: "comments",
localField: "_id",
foreignField: "idAnswer",
as: "comments"
}
}
])
Playground
Second approach, if you want to select questions with all answers and comment then try,
$match your conditions
$lookup with pipeline join with answers collection
pipeline field can allow to add all stages of pipeline that we are using in root level
$match questionId and get answers
$lookup join with comments collection
db.questions.aggregate([
{ $match: { _id: 1 } }, // this is optional if you want to select all questions then remove this
{
$lookup: {
from: "answers",
let: { questionId: "$_id" },
pipeline: [
{ $match: { $expr: { $eq: ["$$questionId", "$questionId"] } } },
{
$lookup: {
from: "comments",
localField: "_id",
foreignField: "idAnswer",
as: "comments"
}
}
],
as: "answers"
}
}
])
Playground
Show or hide extra fields, you can use $project operator at the end of above query,
You can show fields as per your needs
{
$project: {
_id: 1,
content: 1,
"comments._id": 1,
"comments.content": 1
}
}
Playground
Suggestions:
I am not sure, you have already done or not, but try to define object id type in reference field instead of string type, like I have updated your schema, this will add a default index in object id and this will increase a speed of fetching data,
const answerSchema = new mongoose.Schema({
questionId: mongoose.Types.ObjectId,
content: String,
})
const commentSchema = new mongoose.Schema({
idQuestion: mongoose.Types.ObjectId, // nullable
idAnswer: mongoose.Types.ObjectId, // nullable
content: String
})
I've two collections called user and subscription, every subscription has user_id which is _id of user collection. How can I join these two collections by where condition with is_account_active = 1.
Please check the below code which I'm using:
const users = await User.find({ is_account_active: 1 });
This will get me all users which have is_account_active flag as 1 but at the same time, I want subscription details also with respective user ids.
You can below query.
const users = await User.aggregate([
{
$match: {
your_condition
}
},
{
$lookup: {
from: 'subscriptions', // secondary db
localField: '_id',
foreignKey: 'user_id',
as: 'subscription' // output to be stored
}
}
]);
But instead of using _id as a foreign it should be better if you can use a new
field like user_id in primary collection and can use auto increment on that which will now automatically insert new data with new unique id, and you can create index on it for faster query execution.
You can use for example aggregate function.
If you keep user_id as string and you have mongo db version >= 4.0 then you can make _id conversion to string (because _id is an ObjectId type):
const users = await User.aggregate([
{
$match: {
is_account_active: 1
}
},
{
$project: {
"_id": {
"$toString": "$_id"
}
}
},
{
$lookup: {
from: 'subscriptions', //collection name
localField: '_id',
foreignKey: 'user_id',
as: 'subscription'. //alias
}
}
]);
But it is a better idea to store user_id in Subscription schema as Object id
user_id: {
type: mongoose.Schema.Types.ObjectId,
ref:'User'
}
so then
const users = await User.aggregate([
{
$match: {
is_account_active: 1
}
},
{
$lookup: {
from: 'subscriptions', //collection name
localField: '_id',
foreignKey: 'user_id',
as: 'subscription'. //alias
}
}
]);
More about ObjectId
More about Aggregate function
I'm using Mongodb right now with Mathon's excellent answer. I don't have the reputation points to state this in the comments: I believe there is a stray period after the 'as' and the argument foreignKey should be foreignField - at least Mongodd 6.0.3 is presenting an error with it and NodeJS. It works for me with those changes as shown below.
const users = await User.aggregate([
{
$match: {
is_account_active: 1
}
},
{
$project: {
"_id": {
"$toString": "$_id"
}
}
},
{
$lookup: {
from: 'subscriptions', //collection name
localField: '_id',
foreignField: 'user_id',
as: 'subscription' //alias
}
}
]);
I'm using a Mongo DB. Have a two collections main_hikanshou and main_kokyaku. main_hikanshou contains only one column - phone_number. main_kokyaku contains about 10 columns where one is same phone_number. This two collections have about 150000 values, I would like to compare this two collections and get .csv output on matched values in computational-light way...
I'm using this js to just to print it, but it takes too long for using it this way...
var list = []
db.main_hikanshou.find().forEach(function(doc1){
var doc2 = db.main_kokyaku.findOne({phone_number: doc1.phone_number});
if (doc2) {
list.push(doc1);
}
});
print(list);
You can use $in
Try this:
db.main_hikanshou.find({},{phone_number:1} ).toArray(function (_err, docs) {//get all phone phone numbers
let phone_numberList = docs.map(v => { return v.phone_number });
var doc2 = db.main_kokyaku.find({ phone_number: { "$in": phone_numberList } });//search phone
print(doc2);
})
Join the matching phone_number to the main_hikanshou collection using aggregate
db.collection('main_hikanshou').aggregate([
{ $lookup:
{
from: 'main_kokyaku',
localField: 'phone_number',
foreignField: 'phone_number',
as: 'phoneNumber'
}
}
])
I hit an API which follows 50 members' data in a game once a day, and use mongoose to convert the JSON into individual documents in a collection. Between days there is data which is consistent, for example each member's tag (an id for the member in game), but there is data which is different (different scores etc.). Each document has a createdAt property.
I would like to find the most recent document for each member, and thus have an array with each member's tag.
I an currently using the following query to find all documents where tags match, however they are returning all documents, not just one. How do I sort/limit the documents to the most recent one, whilst keep it as one query (or is there a more "mongodb way")?
memberTags = [1,2,3,4,5];
ClanMember.find({
'tag': {
$in: memberTags
}
}).lean().exec(function(err, members) {
res.json(members);
});
Thanks
You can query via the aggregation framework. Your query would involve a pipeline that has stages that process the input documents to give you the desired result. In your case, the pipeline would have a $match phase which acts as a query for the initial filter. $match uses standard MongoDB queries thus you can still query using $in.
The next step would be to sort those filtered documents by the createdAt field. This is done using the $sort operator.
The preceding pipeline stage involves aggregating the ordered documents to return the top document for each group. The $group operator together with the $first accumulator are the operators which make this possible.
Putting this altogether you can run the following aggregate operation to get your desired result:
memberTags = [1,2,3,4,5];
ClanMember.aggregate([
{ "$match": { "tag": { "$in": memberTags } } },
{ "$sort": { "tag": 1, "createdAt: -1 " } },
{
"$group": {
"_id": "$tag",
"createdAt": { "$first": "$createdAt" } /*,
include other necessary fields as appropriate
using the $first operator e.g.
"otherField1": { "$first": "$otherField1" },
"otherField2": { "$first": "$otherField2" },
...
*/
}
}
]).exec(function(err, members) {
res.json(members);
});
Or tweak your current query using find() so that you can sort on two fields, i.e. the tag (ascending) and createdAt (descending) attributes. You can then select the top 5 documents using limit, something like the following:
memberTags = [1,2,3,4,5];
ClanMember.find(
{ 'tag': { $in: memberTags } }, // query
{}, // projection
{ // options
sort: { 'createdAt': -1, 'tag': 1 },
limit: memberTags.length,
skip: 0
}
).lean().exec(function(err, members) {
res.json(members);
});
or
memberTags = [1,2,3,4,5];
ClanMember.find({
'tag': {
$in: memberTags
}
}).sort('-createdAt tag')
.limit(memberTags.length)
.lean()
.exec(function(err, members) {
res.json(members);
});
Ok, so, first, let's use findOne() so you get only one document out of the request
Then to sort by the newest document, you can use .sort({elementYouWantToSort: -1}) (-1 meaning you want to sort from newest to oldest, and 1 from the oldest to the newest)
I would recommend to use this function on the _id, which already includes creation date of the document
Which gives us the following request :
ClanMember.findOne({
'tag': {
$in: memberTags
}
}).sort({_id: -1}).lean().exec(function(err, members) {
res.json(members);
});