I have a Cheques and a Payees collection, every cheque has its corresponding Payee ID.
What I'm trying to do is to write some queries on cheques, but I need to preform the searching after populating the payee (to get the name)
const search = req.query.search || "";
const cheques = await Cheque
.find({
isCancelled: false,
dueDate: { $gte: sinceDate, $lte: tillDate }
})
.select("_id serial dueDate value payee")
.skip(page * limit)
.limit(limit)
.sort({ dueDate: -1, serial: 1 })
.populate({
path: "payee",
select: "name"
})
I guess what I'm trying do is fit this somewhere in my code,
match: {
name: { $regex: search, $options: "i" }
},
I have tried to put the match within the populate, but then it will still find all cheques even if they don't satisfy the population match but populate as null.
I hate this answer and I hope someone is going to post a better one, but I've surfed the web for that with no luck.
The only method I was able to find is to use the $lookup method in aggregation.
So you'll have to change your code from calling .find() to .aggregate().
It's not the sad news, it's great, stable and no problems at all.
but I hated that because it's going to change some patterns you might be following in your code.
const search = req.query.search || "";
const cheques = await Cheque
.aggregate([
{
$lookup: { // similar to .populate() in mongoose
from: 'payees', // the other collection name
localField: 'payee', // the field referencing the other collection in the curent collection
foreignField: '_id', // the name of the column where the cell in the current collection can be found in the other collection
as: 'payee' // the field you want to place the db response in. this will overwrite payee id with the actual document in the response (it only writes to the response, not on the database, no worries)
},
{ // this is where you'll place your filter object you used to place inside .find()
$match: {
isCancelled: false,
dueDate: { $gte: sinceDate, $lte: tillDate }
'payee.branch': 'YOUR_FILTER', // this is how you access the actual object from the other collection after population, using the dot notation but inside a string.
}
},
{ // this is similar to .select()
$project: {_id: 1, serial: 1, dueDate: 1, value: 1, payee: 1}
},
{
$unwind: '$payee' // this picks the only object in the field payee: [ { payeeDoc } ] --> { payeeDoc }
}
])
.skip(page * limit)
.limit(limit)
.sort({ dueDate: -1, serial: 1 })
Notice how you can no longer chain .select() and .populate() on the model query the way you used to do it on .find(), because now you're using .aggregate() which returns a different class instance in mongoose.
you can call .projcet() instead of doing it inside the aggregation array if you want to, but as far as I know, you can't use the .select() method.
My opinion-based solution to this problem is to include the payee information you need for filtering in the Cheque collection.
In my senario, this happen when I was filtering for the sake of my users-roles and permissions, so someone can not see what another one is seeing.
It's up to you, but this makes it easier later when you want to generate the reports (I assume you're working on a payment service).
The populate() feature provided by mongoose first fetches all the Cheques with given conditions and then makes another query to payee with _ids to populate the fields you wanted.
https://mongoosejs.com/docs/api.html#query_Query-populate
by putting match in populate you're filtering which cheques need to be populated but not the cheques themselves.
A simple solution for this is to filter the cheques which are populated as null and return them for your use.
If you see more queries of this sort and/or the collection is huge, it's better you add the payee name in the Cheques collection itself if that fits your purpose.
Related
My mongodb collection login is structured as so:
email: "John.doe#gmail.com"
password: "password"
full_name: "John Doe"
list_of_docs_bought: Array
0: "g81h8"
1: "fursr"
now, in my code, what I want to do is insert a new id into the list_of_docs_bought array. My code is as follows:
let collection = database.collection('login')
let doc = collection.updateOne({ email: req.session.username }, {$set: { list_of_docs_bought: '2xx729' }})
however, this isn't going to work because I need to essentially insert a new id into there, not update it. But if I use .insert(), I am not sure if that is going to work because it will insert a totally new record instead of just into the array. Can anyone offer some help?
You can use upsert which does exactly what you need.
Explanation below is taken from the official docs.
Creates a new document if no documents match the filter. For more details see upsert behavior.
Updates a single document that matches the filter.
So you would need to change your code to that:
let doc = collection.updateOne({ email: req.session.username }, {$push: { list_of_docs_bought: '2xx729' }}, { upsert: true })
Edit: You need to use the $push operator in order to add an element in an array. The $set operator will just overwrite it.
our current setup is: SPA frontend, Azure functions with mongoose middleware, MongoDB
(Maybe first read the question***)
Since we have a lot of documents in our DB and our customer wants to query them we are facing the following problem:
The user is assigned to his organization. He wants to search for Doc1s he has not responded to.
Doc1
{
_id
organization -> partitionKey
content
}
By creating doc2 with reference to doc1 he can respond.
Doc2
{
_id
organization -> partitionKey
Doc1ref
content
}
We have a 1:n relationship.
At the moment we filter just by query criteria of doc1 with limit and skip options.
But the new requirement is to filter the same way by referring doc2s.
I was thinking of:
Doing it in my code => Problem: after we have read with limit=100 and I filter it by my code, the result is not 100 anymore.
Extending doc1 by doc2 arrays => Must be the last option
Dynamic aggregation, Prepared in the code and executed at runtime => Don't want to user dynamic aggregations and the benefits of mongoose are almost lost.
Create a MongoDB view with lookup aggregation (populating doc1 by doc1.respondedOrganizations) => Problem is see here is the performance. When searching a lot of documents and then joining them by a non partitionKey.
*** So, I come to my question:
Is it possible to pass a virtual (not existing) query criteria...
doc1.find({ alreadyResponded : my.organization } )
...and use it as input variable in an aggregation
{
$lookup: {
from: Doc2s,
localField: _id,
foreignField: Doc1ref,
as: < output array field >
pipeline: [{
$match: {
$organization: {
$eq: $$alreadyResponded
}]
}
}
It would reduce query performance extremly.
Thanks
I am attempting to add a help request system which allows the requestor to make only one request for help on each topic from an expert. If the expert lists multiple topics which they can help, I want to limit each requestor to one help request per topic per expert.
I am using node.js and mongoose.js with a self-hosted mongodb instance
I have tried using the $and operator to find the ._id of the expert as long as they don't already have an existing request from the same requestor on the same topic. It works for one update but after the experts document has a subdocument inserted with either the topic_id or the requestor_id the filter is applied and no expert is returned.
// Schema
ExpertSchema = new mongoose.Schema({
expert_id: String,
helpRequests: [
requestor_id: String,
topic_id: String
]
});
//query
const query = {
$and:[
{expert_id: req.body.expert_id},
{'helpRequests.requestor_id': {$ne: req.body.requestor_id}},
{'helpRequests.topic_id': {$ne: req.body.topic_id}}
]
};
// desired update
const update = {
$push: {
helpRequests: {
requestor_id: req.body.requestor_id,
topic_id: req.body.topic_id
}
}
Expert.findOneAndUpdate(query, update, {new: true}, (err, expert) =>{
// handle return or error...
});
The reason you are not getting any expert is condition inside your query.
Results always returned based on the condition of your query if your condition inside query get satisfied you will get your result as simple as that.
Your query
{'helpRequests.requestor_id': {$ne: req.body.requestor_id}},
{'helpRequests.topic_id': {$ne: req.body.topic_id}}
you will get your expert only if requestor_id and topic_id is not exists inside helpRequests array. thats you are querying for.
Solution
As per you schema if helpRequests contains only requestor_id and topic_id then you can achieve what you desire by below query.
Expert.findOneAndUpdate(
{
expert_id: req.body.expert_id,
}, {
$addToSet: {
helpRequests: {
requestor_id: req.body.requestor_id,
topic_id: req.body.topic_id
}
}
}, { returnNewDocument: true });
The structure of the table is:
chats
--> randomId
-->--> participants
-->-->--> 0: 'name1'
-->-->--> 1: 'name2'
-->--> chatItems
etc
What I am trying to do is query the chats table to find all the chats that hold a participant by a passed in username string.
Here is what I have so far:
subscribeChats(username: string) {
return this.af.database.list('chats', {
query: {
orderByChild: 'participants',
equalTo: username, // How to check if participants contain username
}
});
}
Your current data structure is great to look up the participants of a specific chat. It is however not a very good structure for looking up the inverse: the chats that a user participates in.
A few problems here:
you're storing a set as an array
you can only index on fixed paths
Set vs array
A chat can have multiple participants, so you modelled this as an array. But this actually is not the ideal data structure. Likely each participant can only be in the chat once. But by using an array, I could have:
participants: ["puf", "puf"]
That is clearly not what you have in mind, but the data structure allows it. You can try to secure this in code and security rules, but it would be easier if you start with a data structure that implicitly matches your model better.
My rule of thumb: if you find yourself writing array.contains(), you should be using a set.
A set is a structure where each child can be present at most once, so it naturally protects against duplicates. In Firebase you'd model a set as:
participants: {
"puf": true
}
The true here is really just a dummy value: the important thing is that we've moved the name to the key. Now if I'd try to join this chat again, it would be a noop:
participants: {
"puf": true
}
And when you'd join:
participants: {
"john": true,
"puf": true
}
This is the most direct representation of your requirement: a collection that can only contain each participant once.
You can only index known properties
With the above structure, you could query for chats that you are in with:
ref.child("chats").orderByChild("participants/john").equalTo(true)
The problem is that this requires you to define an index on `participants/john":
{
"rules": {
"chats": {
"$chatid": {
"participants": {
".indexOn": ["john", "puf"]
}
}
}
}
}
This will work and perform great. But now each time someone new joins the chat app, you'll need to add another index. That's clearly not a scaleable model. We'll need to change our data structure to allow the query you want.
Invert the index - pull categories up, flattening the tree
Second rule of thumb: model your data to reflect what you show in your app.
Since you are looking to show a list of chat rooms for a user, store the chat rooms for each user:
userChatrooms: {
john: {
chatRoom1: true,
chatRoom2: true
},
puf: {
chatRoom1: true,
chatRoom3: true
}
}
Now you can simply determine your list of chat rooms with:
ref.child("userChatrooms").child("john")
And then loop over the keys to get each room.
You'll like have two relevant lists in your app:
the list of chat rooms for a specific user
the list of participants in a specific chat room
In that case you'll also have both lists in the database.
chatroomUsers
chatroom1
user1: true
user2: true
chatroom2
user1: true
user3: true
userChatrooms
user1:
chatroom1: true
chatroom2: true
user2:
chatroom1: true
user2:
chatroom2: true
I've pulled both lists to the top-level of the tree, since Firebase recommends against nesting data.
Having both lists is completely normal in NoSQL solutions. In the example above we'd refer to userChatrooms as the inverted index of chatroomsUsers.
Cloud Firestore
This is one of the cases where Cloud Firestore has better support for this type of query. Its array-contains operator allows filter documents that have a certain value in an array, while arrayRemove allows you to treat an array as a set. For more on this, see Better Arrays in Cloud Firestore.
I have a blogs collection that contains title, body and agrregate rating that the users have given to them. Another collection 'Ratings' whose schema has reference to the blog, user who rated(if at all he rates them) it in the form of their ObjectIds and the rating they have given ie., +1 or -1.
When a particular user browses through blogs in the 'latest first' order (say 40 of them per page. Call them an array of blogs[0] to blogs[39]) I have to retrieve the rating documents related to this particular user and those 40 blogs if at all the user rated them and notify him of what ratings he has given those blogs.
I tried to extract all rating documents of a particular user in which blog reference objectIds lie between blogs[0]._id and blogs[39]._id which returns empty list in my case. May be objectIds cant be compared using $lt and $gt queries. In that case how should I go about it? Should I redesign my schemas to fit to this scenario?
I am using mongoosejs driver for this case. Here are the relevant parts of the code which differ a bit in execution but youu get the idea.
Schemas:
Client= new mongoose.Schema({
ip:String
})
Rates = new mongoose.Schema({
client:ObjectId,
newsid:ObjectId,
rate:Number
})
News = new mongoose.Schema({
title: String,
body: String,
likes:{type:Number,default:0},
dislikes:{type:Number,default:0},
created:Date,
// tag:String,
client:ObjectId,
tag:String,
ff:{type:Number,default:20}
});
models:
var newsm=mongoose.model('News', News);
var clientm=mongoose.model('Client', Client);
var ratesm=mongoose.model('Rates', Rates);
Logic:
newsm.find({tag:tag[req.params.tag_id]},[],{ sort:{created:-1},limit: buffer+1 },function(err,news){
ratesm.find({client:client._id,newsid:{$lte:news[0]._id,$gte:news.slice(-1)[0]._id}},function(err,ratings){
})
})
Edit:
While implementing the below said schema, I had to do this query in mongoose.js
> db.blogposts.findOne()
{ title : "My First Post", author: "Jane",
comments : [{ by: "Abe", text: "First" },
{ by : "Ada", text : "Good post" } ]
}
> db.blogposts.find( { "comments.by" : "Ada" } )
How do I do this query in mongoose?
A good practice with MongoDB (and other non-relational data stores) is to model your data so it is easy to use/query in your application. In your case, you might consider denormalizing the structure a bit and store the rating right in the blog collection, so a blog might look something like this:
{
title: "My New Post",
body: "Here's my new post. It is great. ...",
likes: 20,
dislikes: 5,
...
rates: [
{ client_id: (id of client), rate: 5 },
{ client_id: (id of another client), rate: 3 },
{ client_id: (id of a third client), rate: 10 }
]
}
The idea being that the objects in the rates array contains all the data you'll need to display the blog entry, complete with ratings, right in the single document. If you also need to query the rates in another way (e.g. find all the ratings made by user X), and the site is read-heavy, you may consider also storing the data in a Rates collection as you're doing now. Sure, the data is in two places, and it's harder to update, but it may be an overall win after you analyze your app and how it accesses your data.
Note that you can apply indexes deep into a document's structure, so for example you can index News.rates.client_id, and then you can quickly find any documents in the News collection that a particular user has rated.