I have a long array of docs to create. When I create them I get no errors.
const docsJson =[some array of json of docs to create]
const orders = await MySchema.create(ordersJSON);
// orders.length returns the same number of docs as docsJson
But when I search for the new docs, only some were created.
const actualOrdersCreated = await MySchema.find({ _id: { $in: orders.map((p) => p._id) } });
// actualOrdersCreated.length returns less docs than in docsJson
What's causing this?
I think your data is to large.
The maximum BSON document size is 16 megabytes.
Reference: https://www.mongodb.com/docs/manual/reference/limits/
This was due to having a ttl (time to live) index on one mongo database and not the other. I was copying docs over from a database. The index on the first database was:
$ mongo "mongodb+srv://....database-1-url"
>> db.myschema.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_"
},
{
"v" : 2,
"key" : {
"paidOn" : 1
},
"name" : "paidOn_1",
"background" : true
}
]
But the database I was working with had expireAfterSeconds.
$ mongo "mongodb+srv://....database-2-url"
>> db.myschema.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_"
},
{
"v" : 2,
"key" : {
"expireAt" : 1
},
"name" : "expireAt_1",
"background" : true,
"expireAfterSeconds" : 86400
},
{
"v" : 2,
"key" : {
"paidOn" : 1
},
"name" : "paidOn_1",
"background" : true
}
]
So mongo was deleting the new docs where the expireAt field had an old date.
To fix it I ran await Order.syncIndexes(); in a script. This cleared the index to [ { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_" } ]. This is fine for my purpose. But the index isn't the same as the 1st database. The paidOn key is no longer indexes.
What I thought helped but didn't
At first I thought the issue was due to the large size of the jsonDocs.
I had objects with fields that had large base64 strings for images. These were placeholders, supposed to be replaced with http urls for the images.
After I removed the base64 strings I was able to upload the documents. I thought this helped but it was just speeding things up. It takes 1 minute for mongo to check up on expired docs.
Related
I am new to queries in mongodb. I have a document like this -
{
"_id" : ObjectId("5eb0f70f88cd051e7839325c"),
"id" : "1",
"arrayInfo" : [ {"color":"red"}, {"color":"black"}, {"color":"cyan"} ]
}
There are many documents in this format with changing ids and colors inside arrayInfo. I want to do something like -
Find record with id "1" -> Display object inside array info with {"color" : "cyan"}
I believe I have to chain queries after finding like this -
db.collection('Records').findOne({id:"1"}).**something**
Any help will be appreciated thanks.
if(id===1){
res.arrayInfo.map(item => console.log(item.color))
}
db.inventory.find( { "instock": { $elemMatch: { qty: 5, warehouse: "A" } } } )
enter link description here
If no operator is specified, MongoDB by default performs array element matching when the document stores an array. Thus you can simply do:
MongoDB Enterprise ruby-driver-rs:PRIMARY> db.foo.findOne({id:'1',arrayInfo:{color:'cyan'}})
{
"_id" : ObjectId("5eb0f70f88cd051e7839325c"),
"id" : "1",
"arrayInfo" : [
{
"color" : "red"
},
{
"color" : "black"
},
{
"color" : "cyan"
}
]
}
To match one field in the array instead of the complete array element, use $elemMatch.
I have a MongoDB collection with the following structure:
/* 1 */
{
"_id" : ObjectId("5cdb24b41a40ae58e6d690fd"),
"versions" : [
ObjectId("5cdb24b41a40ae58e6d690fe")
],
"releases" : [],
"monetization" : [],
"owner" : "testuser",
"name" : "test-repo-2",
"repoAddress" : "/testuser/test-repo-2",
"repoKey" : null,
"__v" : 0
}
/* 2 */
{
"_id" : ObjectId("5cdb23cb1a40ae58e6d690fa"),
"versions" : [
ObjectId("5cdb23cb1a40ae58e6d690fb"),
ObjectId("5cdda9c54e6d0b795a007960")
],
"releases" : [
ObjectId("5cdda9c54e6d0b795a00795c")
],
"monetization" : [],
"owner" : "testuser",
"name" : "test-repo-1",
"repoAddress" : "/testuser/test-repo-1",
"repoKey" : null,
"__v" : 2,
"createdAt" : ISODate("2019-05-16T18:19:49.159Z"),
"updatedAt" : ISODate("2019-05-16T18:19:49.252Z")
}
I need to loop though all the documents in the collection as well as they array of versions to look for a specific to match it to the project. I need to do this with NodeJS, but for now I'm trying it from mongoshell. I'm trying to use forEach() and $in operator to do this.
db.projects.find().forEach(
function () {
{
versions: {
$in: ['5cdb24b41a40ae58e6d690fe']
}
}
}
);
But each time I get the following response: Script executed successfully, but there are no results to show. Am I doing this correctly?
There are various solutions you could try, for example:
1) You could add a filter to your find-query:
db.getCollection('debug').find({"versions":{"$in":[ObjectId("5cdb24b41a40ae58e6d690fe")]}})
This would directly return the object you're looking for.
2) If you don't want to pass a filter and really query all documents and filter them yourself, you could try the following:
db.getCollection('debug').find({}).forEach(doc => {
doc.versions.forEach(v => {
if(v.toString() === "ObjectId(\"5cdb24b41a40ae58e6d690fe\")") {
print("match");
}
});
});
I'm new to robmongo and I received an assignment to write some queries.
let say I have a collection that each key has some values for example value of "userId" and value of "deviceModel".
I need to write a query that shows for each device model how many users has this device.
this is what I got so far:
db.device_data.aggregate([ {"$group" : {_id:"$data.deviceModel", count:{$sum:1}}}])
The problem is that this aggregate for each device the number of keys it appears.
{
"_id" : { "$binary" : "AN6GmE7Thi+Sd/dpLRjIilgsV/4AAAg=", "$type" : "00" },
"auditVersion" : "1.0",
"currentTime" : NumberLong(1479301118381),
"data" : {
"deviceDesign" : "bullhead",
"loginType" : "GOOGLE",
"source" : "SDKLoader",
"systemUptimeMillis" : 137652880.0,
"simCountryIso" : "il",
"networkOperatorName" : "Cellcom",
"hasPhonePermission" : true,
"deviceIdentifier" : "353627074839559",
"sdkVersion" : "0.7.939.2016-11-14.masterDev",
"brand" : "google",
"osVersion" : "7.0",
"osVersionIncremental" : "3239497",
"deviceModel" : "Nexus 5X",
"deviceSDKVersion" : 24.0,
"manufacturer" : "LGE",
"sdkShortBuildDate" : "2016-11-14",
"sdkFullBuildDate" : "Mon Nov 14 22:16:40 IST 2016",
"product" : "bullhead"
},
"timezone" : "Asia/Jerusalem",
"collectionAlias" : "DEVICE_DATA",
"shortDate" : 17121,
"userId" : "00DE86984ED3862F9277F7692D18C88A#1927cc81cfcf7a467e9d4f4ac7a1534b"}
this is an example of how one key locks like.
The below query should give you distinct count of userId for a deviceModel. I meant if a same userId present for a deviceModel multiple items, it will be counted only once.
db.collection.aggregate([ {"$group" : {_id:"$data.deviceModel", userIds:{$addToSet: "$userId"}}
},
{
$unwind:"$userIds"
},
{
$group: { _id: "$_id", userIdCount: { $sum:1} }
}])
Unwind:-
Deconstructs an array field from the input documents to output a
document for each element.
In the above solution, it deconstructs the userId array formed on the first pipeline.
addToSet:-
Returns an array of all unique values that results from applying an
expression to each document in a group of documents that share the
same group by key.
This function ensures that only unique values are added to an array. In the above case, the userId is added to an array in the first pipeline.
When I try to find specific object in array using find({query}) I always get all elements from array.
Activities array stores activities (it would be a thousands of them) as you can see in the following snippet:
This is my collection:
{
"_id" : ObjectId("58407140755324d04db2ce95"),
"owner" : 103429326776572,
"activities" : [
{
"name" : "test1",
"startTime" : ISODate("2016-08-11T17:41:54Z"),
"type" : "te1",
"lat" : 1,
"lon" : 1,
"creator" : 126212904493088,
"coverPhoto" : {
"name" : "test1",
"path" : "c:\\Users\\Francis\\Desktop\\dusk\\public\\coverPhotos\\SJ9tpP6Mx.jpg"
},
"identifier" : "H1g9F6vpGl",
"users" : [
1,
2,
3
],
"hashTags" : [
"some",
"hashtags"
]
},
{
"name" : "test2",
"startTime" : ISODate("2016-08-11T17:41:53Z"),
"type" : "te2",
"lat" : 1,
"lon" : 1,
"creator" : 103312904493090,
"coverPhoto" : {
"name" : "test2",
"path" : "c:\\Users\\Francis\\Desktop\\dusk\\public\\coverPhotos\\Hy8qpvafe.jpg"
},
"identifier" : "rJlU5TvpMx",
"users" : [
1,
2,
3
],
"hashTags" : [
"some",
"hashtags"
]
}
]
}
I need to get for example an activity that has specific identifier.
I tried to use queries like:
1) db.myCollection.find({'activities.identifier' : "rJlU5TvpMx"})
2) db.myCollection.find({'activities' : { $elemMatch : { "identifier" : "rJlU5TvpMx", "creator" : 103312904493090 } })
And all combinations with '' or "" signs
I found above queries at mongodb docs in equal documents schema as mine is.
Can you tell me what am I doing wrong ?
You can try either use single match or multiple match based on your need. This makes use of $elemMatch(projection)
db.myCollection.find({"_id" : ObjectId("58407140755324d04db2ce95")},
{activities: {$elemMatch: { identifier: "rJlU5TvpMx"}}})
db.myCollection.find( {"_id" : ObjectId("58407140755324d04db2ce95")},
{activities: {$elemMatch: {creator : 103312904493090, identifier: "rJlU5TvpMx" }}})
You are looking for the projection object which gets passed as an argument in your query. It allows the return of specific fields from your search rather than the entire document. http://mongoosejs.com/docs/api.html#model_Model.find
I would also suggest looking at the response to this question here: Mongoose Query: Find an element inside an array which makes use of the unwind operator to enter the array as it seems to be relevant to your needs.
In the collection you are searching in, you have just one Document(Object). If you apply method find() to your collection and the query inside matches the value in activities.identifier it will return the only Document(object).
To have a better understanding of what I am talking about check example on mongoose API doc
And query result here.
Try check this out https://docs.mongodb.com/v3.0/reference/operator/projection/elemMatch/#proj._S_elemMatch instead
So I store the ObjectId of a post inside a comment BSON. An example of my BSON schema is as follows:
{ "_id" : ObjectId("570175e6c002e46edb92aaa5"), "userid" : "56f3f70d4de8c74a69d1d5e1", "postid" : "5700edfe03fcdb000347beaa" }
Now, I want to find all the posts below the postid 5700edfe03fcdb000347bebc.
So I go:
"$match" : {
"postid" : {
"$lt" : "5700edfe03fcdb000347bebc"
}
}
However, the $lt aggregation operator is comparing a String ID instead of an ObjectID and the results returned is still accurate. So, if this is a list of my postids:
{ "_id" : "5700edfe03fcdb000347beda" }
{ "_id" : "5700edfe03fcdb000347bebf" }
{ "_id" : "5700edfe03fcdb000347bebc" }
{ "_id" : "5700edfe03fcdb000347bebb" }
{ "_id" : "5700edfe03fcdb000347beaa" }
{ "_id" : "5700edfb03fcdb000347bebe" }
{ "_id" : "5700edfb03fcdb000347beba" }
It will return:
{ "_id" : "5700edfe03fcdb000347bebb" }
{ "_id" : "5700edfe03fcdb000347beaa" }
{ "_id" : "5700edfb03fcdb000347bebe" }
{ "_id" : "5700edfb03fcdb000347beba" }
However, I wonder if this will always be true or was the above just luck?
I understand that sorting ObjectID will always return an accurate result because inside an ObjectId is a timestamp so it can use the timestamp to sort ObjectId accurately. Inside a string, there is no timestamp so will it sort accurately?
Mongo uses ASCII-betically string comparison, which is case sensitive. In this case, that will work, but thinking in performance area - it is slow.
As objectId field is a 12bit object, so it string equivalent in string will take more space, having two documents
{
"_id" : ObjectId("575f2aa031dcfb59af388e1f"),
"a" : 2.0,
"id" : "575f2a5731dcfb59af388e1e"
},{
"_id" : ObjectId("575f2ab031dcfb59af388e21"),
"a" : 3.0,
"id" : ObjectId("575f2ab031dcfb59af388e20")
}
and checking size:
Object.bsonsize(db.test.findOne({a:2})) // returns 66
Object.bsonsize(db.test.findOne({a:3})) // returns 49
gives 17 bytes less when using objeciId.
Is the game worth a deal? Yes, when field is indexed - index file is smaller => faster, less reads, more performance