I'm looking in ways to improve the upsert performance of my mongoDB application. In my test program I have a 'user' collection which has an 'id' (type - Number) and a 'name' (type - string) property. There is an unique index on the 'id'.
The Problem:
When performing a bulk write (ordered: false) - It seems that updateOne or replaceOne with upsert enabled is about 6 to 8 times slower than 'insertOne'.
My Index:
await getDb().collection('user').createIndex({
id: 1
}, {
unique: true,
name: "id_index"
});
Example replaceOne (Take 8.8 seconds) for 100,000 users:
operations.push({
replaceOne: {
filter: {id: 1},
replacement: {id: 1, name: "user 1"},
upsert: true
}
})
Example updateOne (Take 8.4 seconds) 100,000 users:
operations.push({
updateOne: {
filter: {id: 1},
update: {$set:{name: "user 1"}},
upsert: true
}
})
Example insertOne (Take 1.3 seconds) 100,000 users:
operations.push({
insertOne: {id: 1, name: "user 1"}
})
NOTE - each time I preformed these tests, the collection was emptied, and index was recreated.
Is that to be expected?
Is there anything else I can do to improve upsert performance? I have modified writeConcern on bulkWrite with little to no impact.
I was 'following' this question to see what might come of it. Seeing no answers over the week, I'll offer my own.
Without further information, I think the evidence that you provided yourself is reasonably strong evidence to suggest that the answer is 'yes, this is expected'. While we don't know details such as how many updates versus inserts were performed by your test or what version of the database you are using, there doesn't seem to be anything blatantly wrong with the comparison. The upsert documentation suggest that the database first checks for the existence of documents that would be updated by the command before performing the insert. This is further suggested by the following text a little bit lower on the same page (emphasis added):
If all update() operations finish the query phase before any client successfully inserts data, and there is no unique index on the name field, each update() operation may result in an insert, creating multiple documents with name: Andy.
Based on all of this, I think it is perfectly reasonable to surmise that the update portion of an upsert operation has a noticeable overhead on the operation that is absent for direct insert operations.
Given this information, I think it raises a few additional questions which are important to consider:
What was your goal in knowing this information? Just to make sure you had configured things optimally, or were you not currently achieving some performance targets?
Depending on a variety of factors, perhaps an alternative approach here would be just attempt the insert (or update) and deal with the exceptions separately afterwards?
Perhaps out of curiosity, what's the purpose of having a separate unique index on id when there is already one present for the _id field? Certainly each new index (unique or not) adds some overhead, so perhaps it would be best to just repurpose the required _id field and index to use your particular needs?
Related
I currently have a few issues with my Firestore querying technique. As per this stackoverflow post I made recently, Querying with two array with firestore security rules
The answer proposed to add the the "ids" into a object, with the key as the id, and the value simply being "true". I have completed this, and now my structure looks like so:
This leaves me with this query:
db.collection('Depots')
.where(`products.${productId}`, '==', true)
.where(`users.${userId}`, '==', true)
.where('created', '>', 1585998560500)
.orderBy('created', 'asc')
.get();
This query leaves me with throwing an error, asking to create an index:
The query requires an index. You can create it here: ...
However, this tries to index the specific object key, i.e. QXooVYGBIFWKo6C so products.QXooVYGBIFWKo6C. Which is certianly not what I want, as this query changes, and can have an infinite number of possibilities, which means I would have to create another index for each key entry in order to query it.
Is there any way to solve this issue? I am assuming it needs to index this query due to the different operators used in the query, so I was wondering if there were any workarounds to this issue.
Thank you very much in advance.
What you have here is a map field, for which indexes should usually be created automatically.
That indeed means that you'll have as many indexes as you have products, which means:
You are limited in how many products you can have, as there is a maximum of 40,000 index entries per document.
You pay more per document, as you pay for the storage of each index.
If these are not what you want, you'll have to switch back to your original model, with the query limitations you had there. There doesn't seem to be a solution that fits both of your requirements.
After our discussion in chat, this is the starting point I would suggest. Who knows what the end architecture would look like, but I think this or very close to this. You say that a user can exist in multiple depots at the same time and multiple depots can contain the same products, also at the same time. You also said that a depot can never have more than 40 users at a given time, so an array of 40 users would certainly not encroach on Firestore's document limit of 1,048,576 bytes.
[collection]
<documentId>
- field: value
[depots]
<UUID>
- depotId: string "depot456"
- productCount: num 5,000
<UUID>
- depotId: string "depot789"
- productCount: num 4,500
[products]
<UUID>
- productId: string "lotion123"
- depotId: string "depot456"
- users: [string] ["user10", "user27", "user33"]
<UUID>
- productId: string "lotion123"
- depotId: string "depot789"
- users: [string] ["user10", "user17", "user50"]
[users]
<userId>
- depots: [string] ["depot456", "depot999"]
<userId>
- depots: [string] ["depot333", "depot999"]
In NoSQL, storage is cheap and computation isn't so denormalize your data as much as you need to make your queries possible and efficient (fast and cheap).
To find all depots in a single query where user10 and lotion123 are both true, query the products collection where productId equals x and users array-contains y and collect the depotId values from those results. If you want to preserve the array-contains operation for something else, you'd have to denormalize your data further (replace the array for a single user). Or you could split this query into two separate queries.
With this model, when a user leaves a depot, get all products where users array-contains that user and remove that userId from the array. And when a user joins a depot, get all products where depotId equals x and append that userId to the array.
Watch this video, and others by Rick, to get a solid handle on NoSQL: https://www.youtube.com/watch?v=HaEPXoXVf2k
#danwillm If you are not sure about the number of users and products then your DB structure seems unfit for this situation because there are size and length limitations of the firestore document.
You should rather create a separate collection for products and users i.e normalize your data and have a reference for the user in the product collection.
User :
{
userId: documentId,
name: John,
...otherInfo
}
Product :
{
productId: documentId,
createdBy: userId,
createdOn:date,
productName:"exa",
...otherInfo
}
This way you there will be the size of the document would be limited, i.e try avoiding using maps/arrays in firestore if you are not sure about there size.
Also, in this case, the number of queries would be increased but you don't need many indexes in this case.
This question already has answers here:
Mongodb inserting doc without _id field
(6 answers)
Closed 8 years ago.
If I have an array of objects, say
var myArray = [
{ name: 'Fred', state: 'CA' },
{ name: 'Jane', state: 'DE' }
];
and insert them into a mongo collection (I'm using node.js and the "standard" npm mongo driver, npmjs.org/package/mongodb)
someCollection.insert(myArray, errCallback);
I was slightly surprised that the resulting objects (in memory - looking with the debugger) now have the added _id field.
Is there an option to avoid this? In some ways having the _id is a good thing, in fact, I wrote some extra code cause I didn't think it would be there! But I want to print them out as JSON etc. and the end user probably doesn't want to see stuff like
"name" : "Fred",
"_id": "52caeb50559518c01ab1dda5"
(plus possible security issue)
Obviously, I could adjust my printing code, but for the future it might be nice to avoid this, or at least know how to. If it cant be avoided, I will definitely have to write my own ID generator.
Looking at the docs (http://mongodb.github.io/node-mongodb-native/api-generated/collection.html) I see a forceServerObjectId but that's not what I want.
I'm not familiar with the node.js driver, but I can attest to the default behavior of the default MongoDB shell. When you do a normal db.collection.find( {query}, {fields}) query, the _id field is automatically included as part of the output. To remove the _id field, you need to explicitly state _id : 0 in the field selection for the output. The behavior could be similar in the node.js driver.
I want to implement pagination on top of a MongoDB. For my range query, I thought about using ObjectIDs:
db.tweets.find({ _id: { $lt: maxID } }, { limit: 50 })
However, according to the docs, the structure of the ObjectID means that "ObjectId values do not represent a strict insertion order":
The relationship between the order of ObjectId values and generation time is not strict within a single second. If multiple systems, or multiple processes or threads on a single system generate values, within a single second; ObjectId values do not represent a strict insertion order. Clock skew between clients can also result in non-strict ordering even for values, because client drivers generate ObjectId values, not the mongod process.
I then thought about querying with a timestamp:
db.tweets.find({ created: { $lt: maxDate } }, { limit: 50 })
However, there is no guarantee the date will be unique — it's quite likely that two documents could be created within the same second. This means documents could be missed when paging.
Is there any sort of ranged query that would provide me with more stability?
It is perfectly fine to use ObjectId() though your syntax for pagination is wrong. You want:
db.tweets.find().limit(50).sort({"_id":-1});
This says you want tweets sorted by _id value in descending order and you want the most recent 50. Your problem is the fact that pagination is tricky when the current result set is changing - so rather than using skip for the next page, you want to make note of the smallest _id in the result set (the 50th most recent _id value and then get the next page with:
db.tweets.find( {_id : { "$lt" : <50th _id> } } ).limit(50).sort({"_id":-1});
This will give you the next "most recent" tweets, without new incoming tweets messing up your pagination back through time.
There is absolutely no need to worry about whether _id value is strictly corresponding to insertion order - it will be 99.999% close enough, and no one actually cares on the sub-second level which tweet came first - you might even notice Twitter frequently displays tweets out of order, it's just not that critical.
If it is critical, then you would have to use the same technique but with "tweet date" where that date would have to be a timestamp, rather than just a date.
Wouldn't a tweet "actual" timestamp (i.e. time tweeted and the criteria you want it sorted by) be different from a tweet "insertion" timestamp (i.e. time added to local collection). This depends on your application, of course, but it's a likely scenario that tweet inserts could be batched or otherwise end up being inserted in the "wrong" order. So, unless you work at Twitter (and have access to collections inserted in correct order), you wouldn't be able to rely just on $natural or ObjectID for sorting logic.
Mongo docs suggest skip and limit for paging:
db.tweets.find({created: {$lt: maxID}).
sort({created: -1, username: 1}).
skip(50).limit(50); //second page
There is, however, a performance concern when using skip:
The cursor.skip() method is often expensive because it requires the server to walk from the beginning of the collection or index to get the offset or skip position before beginning to return result. As offset increases, cursor.skip() will become slower and more CPU intensive.
This happens because skip does not fit into the MapReduce model and is not an operation that would scale well, you have to wait for a sorted collection to become available before it can be "sliced". Now limit(n) sounds like an equally poor method as it applies a similar constraint "from the other end"; however with sorting applied, the engine is able to somewhat optimize the process by only keeping in memory n elements per shard as it traverses the collection.
An alternative is to use range based paging. After retrieving the first page of tweets, you know what the created value is for the last tweet, so all you have to do is substitute the original maxID with this new value:
db.tweets.find({created: {$lt: lastTweetOnCurrentPageCreated}).
sort({created: -1, username: 1}).
limit(50); //next page
Performing a find condition like this can be easily parallellized. But how to deal with pages other than the next one? You don't know the begin date for pages number 5, 10, 20, or even the previous page! #SergioTulentsev suggests creative chaining of methods but I would advocate pre-calculating first-last ranges of the aggregate field in a separate pages collection; these could be re-calculated on update. Furthermore, if you're not happy with DateTime (note the performance remarks) or are concerned about duplicate values, you should consider compound indexes on timestamp + account tie (since a user can't tweet twice at the same time), or even an artificial aggregate of the two:
db.pages.
find({pagenum: 3})
> {pagenum:3; begin:"01-01-2014#BillGates"; end:"03-01-2014#big_ben_clock"}
db.tweets.
find({_sortdate: {$lt: "03-01-2014#big_ben_clock", $gt: "01-01-2014#BillGates"}).
sort({_sortdate: -1}).
limit(50) //third page
Using an aggregate field for sorting will work "on the fold" (although perhaps there are more kosher ways to deal with the condition). This could be set up as a unique index with values corrected at insert time, with a single tweet document looking like
{
_id: ...,
created: ..., //to be used in markup
user: ..., //also to be used in markup
_sortdate: "01-01-2014#BillGates" //sorting only, use date AND time
}
The following approach wil work even if there are multiple documents inserted/updated at same millisecond even if from multiple clients (which generates ObjectId). For simiplicity, In following queries I am projecting _id, lastModifiedDate.
First page, fetch the result Sorted by modifiedTime (Descending), ObjectId (Ascending) for fist page.
db.product.find({},{"_id":1,"lastModifiedDate":1}).sort({"lastModifiedDate":-1, "_id":1}).limit(2)
Note down the ObjectId and lastModifiedDate of the last record fetched in this page. (loid, lmd)
For sencod page, include query condition to search if (lastModifiedDate = lmd AND oid > loid ) OR (lastModifiedDate < loid)
db.productfind({$or:[{"lastModifiedDate":{$lt:lmd}},{"_id":1,"lastModifiedDate":1},{$and:[{"lastModifiedDate":lmd},{"_id":{$gt:loid}}]}]},{"_id":1,"lastModifiedDate":1}).sort({"lastModifiedDate":-1, "_id":1}).limit(2)
repeat same for subsequent pages.
ObjectIds should be good enough for pagination if you limit your queries to the previous second (or don't care about the subsecond possibility of weirdness). If that is not good enough for your needs then you will need to implement an ID generation system that works like an auto-increment.
Update:
To query the previous second of ObjectIds you will need to construct an ObjectID manually.
See the specification of ObjectId http://docs.mongodb.org/manual/reference/object-id/
Try using this expression to do it from a mongos.
{ _id :
{
$lt : ObjectId(Math.floor((new Date).getTime()/1000 - 1).toString(16)+"ffffffffffffffff")
}
}
The 'f''s at the end are to max out the possible random bits that are not associated with a timestamp since you are doing a less than query.
I recommend during the actual ObjectId creation on your application server rather than on the mongos since this type of calculation can slow you down if you have many users.
I have build a pagination using mongodb _id this way.
// import ObjectId from mongodb
let sortOrder = -1;
let query = []
if (prev) {
sortOrder = 1
query.push({title: 'findTitle', _id:{$gt: ObjectId('_idValue')}})
}
if (next) {
sortOrder = -1
query.push({title: 'findTitle', _id:{$lt: ObjectId('_idValue')}})
}
db.collection.find(query).limit(10).sort({_id: sortOrder})
I'm starting off with Meteor and need some help with Mongo. I have a collection of names that I'm displaying on a list and want to be able to update one variable of certain entries in the database based on other criteria. Basically what I want to do is:
For every entry where characteristic A = true and B = true, change characteristic C to be false.
So far, I've been trying to figure out how Mongo can handle a "for each" loop over the elements of the collection, and for each element check if conditions A and B hold, and then collection.update(element, {C: false}). This is proving to be a lot more problematic than I thought. I want to do something like this (using dummy variable names):
for (i = 0; i < collection.find().count(); i++){
if (collection[i].A===true && collection[i].B===true)
collection.update(collection[i], {$set: {C: false}});
};
I've been changing this base code around, but am starting to sense that I'm missing something basic about indexing/how collections work in Mongo. Can you index a collection like this (and if so, is this even the most convenient way to do what I'm trying to do?)?
Of course I figure out how to do this right after posting, and of course it's suggested in the Meteor documentation!
And, of course, it's a simple solution:
collection.update({A: true, B: true}, {$set: {C:false}});
As already suggested in comments, the correct answer is:
collection.update({A: true, B: true}, {$set: {C:false}}, {multi: true});
(At least in pure MongoDB, see there).
Without multi: true it will change only one document matching the criteria.
In Meteor it is a bit more tricky, as you are not allowed to do client-side updates other than by matching it (so no possibility for various criteria, no possibility for multi), see http://docs.meteor.com/#update.
You can iterate over all finds, but it would be better to run such code server-side.
I am having some trouble when trying to filter nested documents in Meteor, and I don't want to use MongoDB Aggregation to unwind my documents.
An example of my Users document
{
Publications:[{
type:1
},{
type:2
}]
}
When I do a find to get only type 1, I get the expected result - they return only the User documents who have a type 1 publication, but they also return the publications with type 2 too, because they are in the publications array of that User.
I can make a loop in the results to remove these publications manually before publishing, but I don't think this is the right approach.
Can someone please help guide me to a better solution?
Edited
The Sean answer is good but not the right yet, the projection dont work on meteor. That work on robomongo, but in the project still returning the 2 publications types of the user.
Give this query a shot.
Meteor.users.find(
{ 'Publications.type': 1 },
{ fields: { 'Publications.$': 1 } }
);
This should find users with publications of type 1 and will only include those matching publications in the output. You can include other fields in the output as well by including them in the projection parameter (e.g., { 'Publications.$': 1, profile: 1 }).
One thing to be aware of though. I think queries that use the { 'Publications.$': 1 } projection can only be done on the server side. The minimongo implementation used on the client does not support it.
More info about the $ projection can be found here.