Develop Reference

Query for only a single document based on key name in MongoDB

I am working on a MERN project. I have created a collection in MongoDB having different types of document. Is it an accepted practice to have different structure documents in a single collection? Secondly i need to fetch only a single document from the collection using the key name. My documents are
[{
"_id": {
"$oid": "6333f72822dc0acc4bea17bd"
},
"designation": [
{
"name": "Chairman",
"level": 17
},
{
"name": "Director",
"level": 13
},
{
"name": "Secretary ",
"level": 13
},
{
"name": "Account Officer",
"level": 9
},
{
"name": "Data Entry Operator-GR B",
"level": 5
}
]
},
{
"_id": {
"$oid": "6334313b22dc0acc4bea17c2"
},
"storeRole": ["manager", "approver", "accepter", "firstsignatory"]
},
{
"_id": {
"$oid": "63369d2083a7cc2e818990dd"
},
"designationSuffix": ["I","II", "III"]
}]
How do I get any of the three documents if I only know the key name i.e(designation, storeRole, designationSuffix). I dont want to use ID value.

Welcome to SO.
First, yes it is an accepted practice and indeed, a powerful feature of MongoDB to have different shapes of data in a single collection.
There are two important things to remember when querying for data:
Matching on fields that don't even exist in a document is OK; the document will simply be skipped. This permits you, for example, to query for storeRole and ignore the other documents with designation, etc. -- unless of course you wish to look for those too using an $or expression.
Matching (using $match) for elements in an array will return the whole array, not just the elements that match.
To illustrate this point, let's expand your input data slightly:
{"designation": [
{"name": "Chairman","level": 17},
{"name": "Director", "level": 13}
]
},
{"designation": [
{"name": "Secretary","level": 13}
]
},
We will use dot notation to reach into the structures in the designation array to find those docs where at least one of the name fields is Chairman:
db.foo.aggregate([
{$match: {"designation.name": "Chairman"}}
]);
{
"_id" : 0,
"designation" : [
{
"name" : "Chairman",
"level" : 17
},
{
"name" : "Director",
"level" : 13
}
]
}
The query eliminated the document with name = Secretary as expected but properly returned the whole document (and the whole array) where name = Chairman. Very often the goal is to fetch only the matching items in the array; this is accomplished with the $filter operator:
db.foo.aggregate([
{$match: {"designation.name": "Chairman"}},
{$project: {
// Assigning the output of $filter to the same name as input:
designation: {$filter: {
input: "$designation",
as: "zz",
cond: {$eq: ['$$zz.name','Chairman']}
}}
}}
]);
{
"_id" : 0,
"designation" : [
{
"name" : "Chairman",
"level" : 17
}
]
}
An alternative approach which is useful when query conditions yield null or empty arrays instead of eliminating the document altogether is to $filter first, then match only on results where the array has a length > 1. We must use the $ifNull function to protect $size from being passed a null by turning it into an empty (but not null) array:
db.foo.aggregate([
{$project: {
// Assigning the output of $filter to the same name as input:
designation: {$filter: {
input: "$designation",
as: "zz",
cond: {$eq: ['$$zz.name','Chairman']}
}}
}},
{$match: {$expr: {$gt:[{$size: {$ifNull:["$designation",[] ]}}, 0]}} }
]);
Try commenting out the $match to see what $filter returns when a document has the target array field but no matches vs. when the document does not have the field.

mongo aggregate based on conditions to filter the document for versioning

I am working on versioning, We have documents based on UUIDs andjobUuids, andjobUuids are the documents associated with the currently working user. I have some aggregate queries on these collections which I need to update based on the job UUIDs,
The results fetched by the aggregate query should be such that,
if the current usersjobUuid document does not exist then the master document with jobUuid: "default" will be returned(The document without any jobUuid),
if job uuid exists then only the document is returned.
I have a$match used to get these documents based on certain conditions, from those documents I need to filter out the documents based on the above conditions, and an example is shown below,
The data looks like this:
[
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"name": "adam",
"jobUuid": "default",
},
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "adam"
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"name": "eve",
"jobUuid": "default",
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "eve"
},
{
"uuid": "26cba689-7eb6-4a9e-a04e-24ede0309e50",
"name": "john",
"jobUuid": "default",
}
]
Results for "jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12" should be:
[
{
"uuid": "5cdb5a10-4f9b-4886-98c1-31d9889dd943",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "adam"
},
{
"uuid": "b745baff-312b-4d53-9438-ae28358539dc",
"jobUuid": "d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12",
"name": "eve"
},
{
"uuid": "26cba689-7eb6-4a9e-a04e-24ede0309e50",
"name": "john",
"jobUuid": "default",
}
]
Based on the conditions mentioned above, is it possible to filter the document within the aggregate query to extract the document of a specific job uuid?
Edit 1: I got the following solution, which is working fine, I want a better solution, eliminating all those nested stages.
Edit 2: Updated the data with actual UUIDs and I just included only the name as another field, we do have n number of fields which are not relevant to include here but needed at the end (mentioning this for those who want to use the projection over all the fields).

Update based on comment:
but the UUIDs are alphanumeric strings, as shown above, does it have
an effect on these sorting, and since we are not using conditions to
get the results, I am worried it will cause issues.
You could use additional field to match the sort order to be the same order as values in the in expression. Make sure you provide the values with default as the last value.
[
{"$match":{"jobUuid":{"$in":["d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12","default"]}}},
{"$addFields":{ "order":{"$indexOfArray":[["d275781f-ed7f-4ce4-8f7e-a82e0e9c8f12","default"], "$jobUuid"]}}},
{"$sort":{"uuid":1, "order":1}},
{
"$group": {
"_id": "$uuid",
"doc":{"$first":"$$ROOT"}
}
},
{"$project":{"doc.order":0}},
{"$replaceRoot":{"newRoot":"$doc"}}
]
example here - https://mongoplayground.net/p/wXiE9i18qxf
Original
You could use below query. The query will pick the non default document if it exists for uuid or else pick the default as the only document.
[
{"$match":{"jobUuid":{"$in":[1,"default"]}}},
{"$sort":{"uuid":1, "jobUuid":1}},
{
"$group": {
"_id": "$uuid",
"doc":{"$first":"$$ROOT"}
}
},
{"$replaceRoot":{"newRoot":"$doc"}}
]
example here - https://mongoplayground.net/p/KrL-1s8WCpw

Here is what I would do:
match stage with $in rather than an $or (for readability)
group stage with _id on $uuid, just as you did, but instead of pushing all the data into an array, be more selective. _id is already storing $uuid, so no reason to capture it again. name must always be the same for each $uuid, so take only the first instance. Based on the match, there are only two possibilities for jobUuid, but this will assume it will be either "default" or something else, and that there can be more than one occurrence of the non-"default" jobUuid. Using "$addToSet" instead of pushing to an array in case there are multiple occurrences of the same jobUuid for a user, also, before adding to the set, use a conditional to only add non-"default" jobUuids, using $$REMOVE to avoid inserting a null when the jobUuid is "default".
Finally, "$project" to clean things up. If element 0 of the jobUuids array does not exist (is null), there is no other possibility for this user than for the jobUuid to be "default", so use "$ifNull" to test and set "default" as appropriate. There could be more than 1 jobUuid here, depending if that is allowed in your db/application, up to you to decide how to handle that (take the highest, take the lowest, etc).
Tested at: https://mongoplayground.net/p/e76cVJf0F3o
[{
"$match": {
"jobUuid": {
"$in": [
"1",
"default"
]
}
}
},
{
"$group": {
"_id": "$uuid",
"name": {
"$first": "$name"
},
"jobUuids": {
"$addToSet": {
"$cond": {
"if": {
"$ne": [
"$jobUuid",
"default"
]
},
"then": "$jobUuid",
"else": "$$REMOVE"
}
}
}
}
},
{
"$project": {
"_id": 0,
"uuid": "$_id",
"name": 1,
"jobUuid": {
"$ifNull": [{
"$arrayElemAt": [
"$jobUuids",
0
]
},
"default"
]
}
}
}]

I was able to solve this problem with the following aggregate query,
We are first extracting the results matching only the jobUuid provided by the user or the "default" in the match section.
Then the results are grouped based on the uuid, using a group stage and we are counting the results as well.
Using the conditions in replaceRoot first we are checking the length of the grouped document,
If the grouped document length is greater than or equal to 2, we are
filtering the document that matches the provided jobUuid.
If it's less or equal to the 1, then we are checking if it's matching the default jobUuid and returning it.
The Query is below:
[
{
$match: {
$or: [{ jobUuid:1 },{ jobUuid: 'default'}]
}
},
{
$group: {
_id: '$uuid',
count: {
$sum: 1
},
docs: {
$push: '$$ROOT'
}
}
},
{
$replaceRoot: {
newRoot: {
$cond: {
if: {
$gte: [
'$count',
2
]
},
then: {
$arrayElemAt: [
{
$filter: {
input: '$docs',
as: 'item',
cond: {
$ne: [
'$$item.jobUuid',
'default'
]
}
}
},
0
]
},
else: {
$arrayElemAt: [
{
$filter: {
input: '$docs',
as: 'item',
cond: {
$eq: [
'$$item.jobUuid',
'default'
]
}
}
},
0
]
}
}
}
}
}
]

Unable to send Firebase QueryDocumentSnapshot over API Call

I'm currently making my first network call in a paginated series of calls on the server side. Prior to doing this I was making all of the calls client side and stored the last document in the collection call as an offset.
The offset was then sent as a .startAfter call for the same collection. The offset document looked like this:
exists: (...)
id: (...)
metadata: (...)
ref: (...)
_document: Document {key: DocumentKey, version: SnapshotVersion, data: ObjectValue, proto: {…}, hasLocalMutations: false, …}
_firestore: Firestore {_queue: AsyncQueue, INTERNAL: {…}, _config: FirestoreConfig, _databaseId: DatabaseId, _dataConverter: UserDataConverter, …}
_fromCache: false
_hasPendingWrites: false
_key: DocumentKey {path: ResourcePath}
__proto__: DocumentSnapshot
When I make the call on the server I'm currently able to see the document looks the same, but when I send it over the network it seems to be stripped or at least looks very different by the time it's sent over JSON and parsed back.
Sent like this:
res.json(offset)
then parsed like this:
feedData = await dataWithOffset.json();
After the parsing it looks like this:
{ _ref:
{ _firestore:
{ _settings: [Object],
_settingsFrozen: true,
_serializer: [Object],
_projectId: '***-prod',
_lastSuccessfulRequest: 1566308918946,
_preferTransactions: false,
_clientPool: [Object] },
_path:
{ segments: [Array],
projectId: '***-prod',
databaseId: '(default)' } },
_fieldsProto:
{ lastModified: { timestampValue: [Object], valueType: 'timestampValue' },
...,
_serializer: { timestampsInSnapshots: true },
_readTime: { _seconds: 1566308918, _nanoseconds: 909566000 },
_createTime: { _seconds: 1565994031, _nanoseconds: 304997000 },
_updateTime: { _seconds: 1565994031, _nanoseconds: 304997000 } }
Any idea why it is losing its shape and what I can do to fix it so it returns to working as a proper offset? Should I not be converting to JSON and back, as that may strip some important things?

Solution (tested)
So I went ahead and made a sandbox to test the implementation, and like I mentioned in the comments .startAfter expects a DocumentSnapshot not a DocumentReference
Given that I created a collection named cars with 4 records like this:
[
{
"name": "car 1",
"hp": 5
},
{
"name": "car 2",
"hp": 10
},
{
"name": "car 2.5",
"hp": 10
},
{
"name": "car 3",
"hp": 15
}
]
Then I configured an express endpoint:
router.get('/cars/:lastDocId', (req, res) => {
const query = db.collection('cars')
.orderBy('hp')
.limit(2);
const handleQueryRes = (snap) => {
return res.send({
docs: snap.docs.map(doc => doc.data()),
last: snap.docs.length > 0 ? snap.docs[snap.docs.length - 1].id : null
});
}
if(req.params.lastDocId != -1) {
// this makes an "auxiliar" read from the DB to transform the given ID in a DocumentSnapshot needed for startAfter
return db.collection('cars').doc(req.params.lastDocId).get().then(snap => {
return query.startAfter(snap).get();
}).then(handleQueryRes).catch(console.log);
} else {
return query.get().then(handleQueryRes).catch(console.log);
}
});
The path parameter lastDocId is either -1 for first page or the last document fetched in the previous page
When I make a GET request to /cars/-1 it returns this:
{
"docs": [
{
"hp": 5,
"name": "car 1"
},
{
"name": "car 2",
"hp": 10
}
],
"last": "9sRdLOvV8REwEpHDjEw7"
}
Now if I grab the last prop in the response and use it in my next GET like so /cars/9sRdLOvV8REwEpHDjEw7 I get my next 2 cars:
{
"docs": [
{
"name": "car 2.5",
"hp": 10
},
{
"name": "car 3",
"hp": 15
}
],
"last": "tZPF7Wav7jZuchoCp6zM"
}
Even though I only have 4 records and the second request should be the last the function does not know the length of the collection so it returns another last ID
If you make the last request /cars/tZPF7Wav7jZuchoCp6zM it returns:
{
"docs": [],
"last": null
}
Thus giving indication that is the last page.
I'm not too fond of having to read the document again to transform it into a DocumentSnapshot but I guess that's a firebase limitation
Hope it helped
Answer with steps
When converting to JSON and back you lose the DocumentReference prototype
The solution to pass that type of data through network requests is:
Client sends a plain-text ref to your endpoint ( or doesn't for first page)
Have the server side code re-create the DocumentReference with the received ref
Use the re-created DocumentReference to paginate query
Return from server the requested page's documents alongside with a plain-text refto get the next page
Initial answer (obsolete) but for future reference
You need to send offset.data() instead of sending offset alltogether
One is the document's data (probably what you're after) the other is a DocumentReference which has methods that cannot be represented in JSON.
If you need anything else from the document (like it's id or reference) you're better off building a new object:
res.json({
data: offset.data(),
ref: offset.ref,
id: offset.id
});

Create ID with data of other fields

I'm new to mongodb, and I'm using mongoose to validate and order the data (I'm open to change it to MySQL if this doesn't work).
The app will be an e-shop, to buy merchandising related to movies, games, ext.
My schema is as follows:
var productSchema = {
id: {
type: String,
required: true
},
name: {
type: String,
required: true
},
img: {
type: String,
required: true
},
price: {
type: Number,
required: true
},
stock: {
type: Number,
required: true
},
category: {
object: {
type: String,
required: true
},
group: {
type: String,
required: true
},
name: {
type: String,
required: true
}
}
};
This is what I would like to do:
If I have the following data in category:
category.object = "ring"
category.group = "movies"
category.name= "lord of the rings"
I want the id to be made of the first letters of every field in category and a number (the number of the last item added plus 1). In this case, It would be RMLOTR1.
What I'm doing right now
I'm adding a lot of data at the same time, so every time I do it, I made a function that iterates through all the items added and does what I want but...
My question is
Is there a built-in way to do this with mongodb or mongoose, adding the data and creating the id at the same time? I know I can do a virtual, but I want the data to be stored.
Extras
If it's not posible to do this with mongodb, is there a way to do this with MySQL?
Is doing this kind of thing considered a correct/wrong approach?

You are basically looking for a "pre" middleware hook on the "save" event fired by creating new documents in the collection. This will inspect the current document content and extract the "strings" from values in order to create your "prefix" value for _id.
There is also another part, where the "prefix" needs the addition of the numeric counter when there is already a value present for that particular "prefix" to make it distinct. There is a common technique in MongoDB used to "Generate an auto-incrementing sequence field", which basically involves keeping a "counters" collection and incrementing the value each time you access it.
As a complete and self contained demonstration, you combine the techniques as follows:
var async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost/warehouse');
var counterSchema = new Schema({
"type": { "type": String, "required": true },
"prefix": { "type": String, "required": true },
"counter": Number
});
counterSchema.index({ "type": 1, "prefix": 1 },{ "unique": true });
counterSchema.virtual('nextId').get(function() {
return this.prefix + this.counter;
});
var productSchema = new Schema({
"_id": "String",
"category": {
"object": { "type": String, "required": true },
"group": { "type": String, "required": true },
"name": { "type": String, "required": true }
}
},{ "_id": false });
productSchema.pre('save', function(next) {
var self = this;
if ( !self.hasOwnProperty("_id") ) {
var prefix = self.category.object.substr(0,1).toUpperCase()
+ self.category.group.substr(0,1).toUpperCase()
+ self.category.name.split(" ").map(function(word) {
return word.substr(0,1).toUpperCase();
}).join("");
mongoose.model('Counter').findOneAndUpdate(
{ "type": "product", "prefix": prefix },
{ "$inc": { "counter": 1 } },
{ "new": true, "upsert": true },
function(err,counter) {
self._id = counter.nextId;
next(err);
}
);
} else {
next(); // Just skip when _id is already there
}
});
var Product = mongoose.model('Product',productSchema),
Counter = mongoose.model('Counter', counterSchema);
async.series(
[
// Clean data
function(callback) {
async.each([Product,Counter],function(model,callback) {
model.remove({},callback);
},callback);
},
function(callback) {
async.each(
[
{
"category": {
"object": "ring",
"group": "movies",
"name": "lord of the rings"
}
},
{
"category": {
"object": "ring",
"group": "movies",
"name": "four weddings and a funeral"
}
},
{
"category": {
"object": "ring",
"group": "movies",
"name": "lord of the rings"
}
}
],
function(data,callback) {
Product.create(data,callback)
},
callback
)
},
function(callback) {
Product.find().exec(function(err,products) {
console.log(products);
callback(err);
});
},
function(callback) {
Counter.find().exec(function(err,counters) {
console.log(counters);
callback(err);
});
}
],
function(err) {
if (err) throw err;
mongoose.disconnect();
}
)
This gives you output like:
[ { category: { name: 'lord of the rings', group: 'movies', object: 'ring' },
__v: 0,
_id: 'RMLOTR1' },
{ category:
{ name: 'four weddings and a funeral',
group: 'movies',
object: 'ring' },
__v: 0,
_id: 'RMFWAAF1' },
{ category: { name: 'lord of the rings', group: 'movies', object: 'ring' },
__v: 0,
_id: 'RMLOTR2' } ]
[ { __v: 0,
counter: 2,
type: 'product',
prefix: 'RMLOTR',
_id: 57104cdaa774fcc73c1df0e8 },
{ __v: 0,
counter: 1,
type: 'product',
prefix: 'RMFWAAF',
_id: 57104cdaa774fcc73c1df0e9 } ]
To first understand the Counter schema and model, you are basically defining something where you are going to look up a "unique" key and also attach a numeric field to "increment" on match. For convenience this just has a two fields making up the unique combination and a compound index defined. This could just also be a compound _id if so wanted.
The other convenience is the virtual method of nextId, which just does a concatenation of the "prefix" and "counter" values. It's also best practice here to include something like "type" here since your Counter model can be used to service "counters" for use in more than one collection source. So here we are using "product" whenever accessing in the context of the Product model to differentiate it from other models where you might also keep a similar sequence counter. Just a design point that is worthwhile following.
For the actual Product model itself, we want to attach "pre save" middleware hook in order to fill the _id content. So after determining the character portion of the "prefix", the operation then goes off and looks for that "prefix" with the "product" type data in combination in the Counter model collection.
The function of .findOneAndUpdate() is to look for a document matching the criteria in the "counters" collection and then where a document is found already it will "increment" the current counter value by use of the $inc update operator. If the document was not found, then the "upsert" option means that a new document will be created, and at any rate the same "increment" will happen in the new document as well.
The "new" option here means that we want the "modified" document to be returned ( either new or changed ) rather than what the document looked like before the $inc was applied. The result is that "counter" value will always increase on every access.
Once that is complete and a document for Counter is either incremented or created for it's matching keys, then you now have something you can use to assign to the _id in the Product model. As mentioned earlier you can use the virtual here for convenience to get the prefix with the appended counter value.
So as long as your documents are always created by either the .create() method from the model or by using new Product() and then the .save() method, then the methods attached to your "model" in your code are always executed.
Note here that since you want this in _id, then as a primary key this is "immutable" and cannot change. So even if the content in the fields referenced was later altered, the value in _id cannot be changed, and therefore why the code here makes no attempt when an _id value is already set.

Change MongoDB find results dynamically

I'm using mongoose with node.js.
Let's say I have 'Posts' DB where each document in it is a post.
Each post has a 'ReadBy' array which holds names of users that had read this post.
When I'm searching for documents in this DB, I want to "change" the 'ReadBy' value to show by Boolean value if the user that is searching for it is in this array or not.
For example, let's say these are 2 documents that are in this DB:
{ "PostName": "Post Number 1", "ReadBy": ["Tom", "John", "Adam"] }
{ "PostName": "Post Number 2", "ReadBy": ["John", "Adam"] }
If I'm user 'Tom', I want to get the results like this:
[
{
"PostName": "Post Number 1",
"ReadBy": true,
},
{
"PostName": "Post Number 2",
"ReadBy": false,
}
]
Now, I know that I can get the documents and go over each one of them with forEach function, and then use forEach again on the "ReadBy" array and change this field.
I'm asking if there is more efficient way to do it in the mongoDB query itself, or some other way in the code.
If there is another way with mongoose - even better.

Using mongoDb $setIntersection in aggregation you get the result like this :
db.collectionName.aggregate({
"$project": {
"ReadBy": {
"$cond": {
"if": {
"$eq": [{
"$setIntersection": ["$ReadBy", ["Tom"]]
},
["Tom"]
]
},
"then": true,
"else": false
}
},
"PostName": 1
}
})
So above working first like this
{ $setIntersection: [ [ "Tom", "John", "Adam"], [ "Tom"] ] }, return [ "Tom"]
{ $setIntersection: [ [ "John", "Adam"], [ "Tom"] ] }, return [ ]
and $eq to check whether setIntersection results matched with ["Tom"] if yes then return true else false

You can try something similar to
var unwind = {"$unwind": "$ReadBy"}
var eq = {$eq: ["$ReadBy", "Bob"]}
var project = {$project: {PostName: 1, seen: eq}}
db.posts.aggregate([unwind, project])
Just notice that you solution is highly inefficient. Both for storing the data ( growing array) and for searching.