Develop Reference

Mongo not creating all docs in array and not giving errors

I have a long array of docs to create. When I create them I get no errors.
const docsJson =[some array of json of docs to create]
const orders = await MySchema.create(ordersJSON);
// orders.length returns the same number of docs as docsJson
But when I search for the new docs, only some were created.
const actualOrdersCreated = await MySchema.find({ _id: { $in: orders.map((p) => p._id) } });
// actualOrdersCreated.length returns less docs than in docsJson
What's causing this?

I think your data is to large.
The maximum BSON document size is 16 megabytes.
Reference: https://www.mongodb.com/docs/manual/reference/limits/

This was due to having a ttl (time to live) index on one mongo database and not the other. I was copying docs over from a database. The index on the first database was:
$ mongo "mongodb+srv://....database-1-url"
>> db.myschema.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_"
},
{
"v" : 2,
"key" : {
"paidOn" : 1
},
"name" : "paidOn_1",
"background" : true
}
]
But the database I was working with had expireAfterSeconds.
$ mongo "mongodb+srv://....database-2-url"
>> db.myschema.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_"
},
{
"v" : 2,
"key" : {
"expireAt" : 1
},
"name" : "expireAt_1",
"background" : true,
"expireAfterSeconds" : 86400
},
{
"v" : 2,
"key" : {
"paidOn" : 1
},
"name" : "paidOn_1",
"background" : true
}
]
So mongo was deleting the new docs where the expireAt field had an old date.
To fix it I ran await Order.syncIndexes(); in a script. This cleared the index to [ { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_" } ]. This is fine for my purpose. But the index isn't the same as the 1st database. The paidOn key is no longer indexes.
What I thought helped but didn't
At first I thought the issue was due to the large size of the jsonDocs.
I had objects with fields that had large base64 strings for images. These were placeholders, supposed to be replaced with http urls for the images.
After I removed the base64 strings I was able to upload the documents. I thought this helped but it was just speeding things up. It takes 1 minute for mongo to check up on expired docs.

Rrules and dayllght savings

I'm using rrule to create and store events in my database.
All was working until I found that my recurring events had an one hour difference past the 31st march.
In France, it's the day when we do a daylight saving time change.
Actually, my events are stored in a mongo database, with the start date and the duration of the event, + the eventuals rrules (all events aren't recurring events) like this :
{
"_id" : ObjectId("5c8e4706703df43859aabbe7"),
"duration" : 2879,
"type" : "unavailability",
"title" : "Weekend",
"description" : "C'est le weekend",
"rrules" : [
{
"until" : ISODate("2021-03-22T23:00:00.000Z"),
"dtstart" : ISODate("2019-03-11T23:00:00.000Z"),
"byweekday" : [
{
"weekday" : 5
},
{
"weekday" : 6
}
],
"interval" : 1,
"freq" : 2
}
],
"__v" : 0
}
When the frontend search for a date in the calendar, it will search with this args :
?from=2019-03-10T23:00:00.000Z&to=2019-03-17T23:00:00.000Z
It works well with this date, because no daylight savings are occuring in between. If I have this object :
normalizedDates = { from: 2019-03-10T23:00:00.000Z, to: 2019-03-17T23:00:00.000Z }
and this rule :
{ until: 2021-03-22T23:00:00.000Z,
dtstart: 2019-03-11T23:00:00.000Z,
byweekday: [ { weekday: 5 }, { weekday: 6 } ],
interval: 1,
freq: 2 }
Running :
const recurringDays = rruleSet.between(normalizedDates.from, normalizedDates.to)
shows, indeed :
recurringDays [ 2019-03-23T23:00:00.000Z ]
But if y use :
normalizedDates = { from: 2019-03-31T22:00:00.000Z, to: 2019-04-07T22:00:00.000Z }
Rrules returns :
recurringDays [ 2019-03-31T23:00:00.000Z, 2019-04-06T23:00:00.000Z ]
while I'm expecting :
recurringDays [ 2019-04-06T22:00:00.000Z ]
Do you know how I could handle this ?

If you want a recurrence rule to observe daylight saving time for a particular time zone, then you must schedule using this time zone. In your example, the schedule is based on UTC.
RRule provides time zone support. You should use that, and specify tzid: 'Europe/Paris'.
Also, you might consider using the toString and fromString functions to work with iCalendar formatted strings, and store that in your MongoDB instance instead of serializing the RRule as JSON.

MongoDB - Query on the last object of an array?

Is there any way in MongoDB in which I can specifically query on last object's key value in an array in a single db query.
For eg: This is my doc in a collection.
{
"com" : [
{ "ts" : 1510830164203, "com" : "com1" },
{ "ts" : 1511242569673, "com" : "connected" },
{ "ts" : 1511244832741, "com" : "vb" }
],
"status" : [
{ "ts" : 1510857000000, "stat" : 3 }
]
}
So as you can see there are multiple objects in com.
How can I query on last object's ts(timestamp) or I want to check is last com inserted in between today's date or not.
I have already gone through this link. But didn't find the appropriate solution.
Any help can be appreciated.

You can use $arrayElemAt to get the last element and then match applied in aggregation. To get the last element using $arrayElemAt use second value -1 that indicate last element of an array. $arrayElemAt: ["arrayName", -1]. code will be like
db.collectionName.aggregate([
{
$project: {
status: 1,
com: {$arrayElemAt: ["$com", -1]}
}
},
{
$match: {"com.ts": 15115465465}
}
])
N.B: if you want to compare like less than or greater than then use like : $lt, $lte, $gt, or $gte any one that you need
$match: {"com.ts": {$lt: 15115465465}}

var d1 = new Date( parseInt( "Today at 12:00 AM", 16 ) * 1000 )
var d2 = new Date( parseInt( "Tomorrow at 12:00 AM", 16 ) * 1000 )
db.table.find(
{ com: { $elemMatch: {ts:{ $gte: d1, $lt: d2 } } } })
)

db.collection.aggregate(
// Pipeline
[
// Stage 1
{
$match: {
"_id" : ObjectId("5a197a3bde472b16ed9fc28d")
}
},
// Stage 2
{
$unwind: {
path : "$com"
}
},
// Stage 3
{
$sort: {
'com.ts':-1
}
},
// Stage 4
{
$limit: 1
}
]
);

You can use the below project query if you only need to find the last element.
db.collection.find({},{status: 1, com:{$slice: -1}})
More discussion on the similar topic here

Robmongo - aggregate values distinct by other value\cloumn

I'm new to robmongo and I received an assignment to write some queries.
let say I have a collection that each key has some values for example value of "userId" and value of "deviceModel".
I need to write a query that shows for each device model how many users has this device.
this is what I got so far:
db.device_data.aggregate([ {"$group" : {_id:"$data.deviceModel", count:{$sum:1}}}])
The problem is that this aggregate for each device the number of keys it appears.
{
"_id" : { "$binary" : "AN6GmE7Thi+Sd/dpLRjIilgsV/4AAAg=", "$type" : "00" },
"auditVersion" : "1.0",
"currentTime" : NumberLong(1479301118381),
"data" : {
"deviceDesign" : "bullhead",
"loginType" : "GOOGLE",
"source" : "SDKLoader",
"systemUptimeMillis" : 137652880.0,
"simCountryIso" : "il",
"networkOperatorName" : "Cellcom",
"hasPhonePermission" : true,
"deviceIdentifier" : "353627074839559",
"sdkVersion" : "0.7.939.2016-11-14.masterDev",
"brand" : "google",
"osVersion" : "7.0",
"osVersionIncremental" : "3239497",
"deviceModel" : "Nexus 5X",
"deviceSDKVersion" : 24.0,
"manufacturer" : "LGE",
"sdkShortBuildDate" : "2016-11-14",
"sdkFullBuildDate" : "Mon Nov 14 22:16:40 IST 2016",
"product" : "bullhead"
},
"timezone" : "Asia/Jerusalem",
"collectionAlias" : "DEVICE_DATA",
"shortDate" : 17121,
"userId" : "00DE86984ED3862F9277F7692D18C88A#1927cc81cfcf7a467e9d4f4ac7a1534b"}
this is an example of how one key locks like.

The below query should give you distinct count of userId for a deviceModel. I meant if a same userId present for a deviceModel multiple items, it will be counted only once.
db.collection.aggregate([ {"$group" : {_id:"$data.deviceModel", userIds:{$addToSet: "$userId"}}
},
{
$unwind:"$userIds"
},
{
$group: { _id: "$_id", userIdCount: { $sum:1} }
}])
Unwind:-
Deconstructs an array field from the input documents to output a
document for each element.
In the above solution, it deconstructs the userId array formed on the first pipeline.
addToSet:-
Returns an array of all unique values that results from applying an
expression to each document in a group of documents that share the
same group by key.
This function ensures that only unique values are added to an array. In the above case, the userId is added to an array in the first pipeline.

Match Documents where all array members do not contain a value

MongoDB selectors become quickly complicated, especially when you come from mySQL using JOIN and other fancy keywords. I did my best to make the title of this question as clear as possible, but failed miserably.
As an example, let a MongoDB collection have the following schema for its documents:
{
_id : int
products : [
{
qte : int
status : string
},
{
qte : int
status : string
},
{
qte : int
status : string
},
...
]
}
I'm trying to run a db.collection.find({ }) query returning documents where all products do not have the string "finished" as status. Please note that the products array has a variable length.
We could also say we want all documents that has at least one product with a status that is not "finished".
If I were to run it as a Javascript loop, we would have something like the following :
// Will contain queried documents
var matches = new Array();
// The documents variable contains all documents of the collection
for (var i = 0, len = documents.length; i < len; i++) {
var match = false;
if (documents[i].products && documents[i].products.length !== 0) {
for (var j = 0; j < documents[i].products; j++) {
if (documents[i].products[j].status !== "finished") {
match = true;
break;
}
}
}
if (match) {
matches.push(documents[i]);
}
}
// The previous snippet was coded directly in the Stack Overflow textarea; I might have done nasty typos.
The matches array would contain the documents I'm looking for. Now, I wish there would be a way of doing something similar to collection.find({"products.$.status" : {"$ne":"finished"}}) but MongoDB hates my face when I do so.
Also, documents that do not have any products need to be ignored, but I already figured this one out with a $and clause. Please note that I need the ENTIRE document to be returned, not just the product array. If a document has products that are not "finished", then the entire document should be present. If a document has all of its products set at "finished", the document is not returned at all.
MongoDB Version: 3.2.4
Example
Let's say we have a collection that contains three documents.
This one would match because one of the status is not "finished".
{
_id : 1,
products : [
{
qte : 10,
status : "finished"
},
{
qte : 21,
status : "ongoing"
},
]
}
This would not match because all statuses are set to "finished"
{
_id : 2,
products : [
{
qte : 35,
status : "finished"
},
{
qte : 210,
status : "finished"
},
{
qte : 2,
status : "finished"
},
]
}
This would also not match because there are no products. It would also not match if the products field was undefined.
{
_id : 3,
products : []
}
Again, if we ran the query in a collection that had the three documents in this example, the output would be:
[
{
_id : 1,
products : [
{
qte : 10,
status : "finished"
},
{
qte : 21,
status : "ongoing"
},
]
}
]
Only the first document gets returned because it has at least one product that doesn't have a status of "finished", but the last two did not make the cut since they either have all their products' statuses set as "finished", or don't have any products at all.

Try following query. It's fetching documents where status is not equals to "finished"
Note: This query will work with MongoDB 3.2+ only
db.collection.aggregate([
{
$project:{
"projectid" : 1,
"campname" : 1,
"campstatus" : 1,
"clientid" : 1,
"paymentreq" : 1,
products:{
$filter:{
input:"$products",
as: "product",
cond:{$ne: ["$$product.status", "finished"]}
}
}
}
},
{
$match:{"products":{$gt: [0, {$size:"products"}]}}
}
])

You need .aggregate() rather than .find() here. That is the only way to determine if ALL elements actually don't contain what you want:
// Sample data
db.products.insertMany([
{ "products": [
{ "qte": 1 },
{ "status": "finished" },
{ "status": "working" }
]},
{ "products": [
{ "qte": 2 },
{ "status": "working" },
{ "status": "other" }
]}
])
Then the aggregate operation with $redact:
db.products.aggregate([
{ "$redact": {
"$cond": {
"if": {
"$anyElementTrue": [
{ "$map": {
"input": "$products",
"as": "product",
"in": {
"$eq": [ "$$product.status", "finshed" ]
}
}}
]
},
"then": "$$PRUNE",
"else": "$$KEEP"
}
}}
])
Or alternately you can use the poorer and slower cousin with $where
db.products.find(function(){
return !this.products.some(function(product){
return product.status == "finished"
})
})
Both return just the one sample document:
{
"_id" : ObjectId("56fb4791ae26432047413455"),
"products" : [
{
"qte" : 2
},
{
"status" : "working"
},
{
"status" : "other"
}
]
}
So the $anyElementTrue with the $map input or the .some() are basically doing the same thing here and evaluating if there was any match at all. You use the "negative" assertion to "exclude" documents that actually find a match.