I want to implement pagination on top of a MongoDB. For my range query, I thought about using ObjectIDs:
db.tweets.find({ _id: { $lt: maxID } }, { limit: 50 })
However, according to the docs, the structure of the ObjectID means that "ObjectId values do not represent a strict insertion order":
The relationship between the order of ObjectId values and generation time is not strict within a single second. If multiple systems, or multiple processes or threads on a single system generate values, within a single second; ObjectId values do not represent a strict insertion order. Clock skew between clients can also result in non-strict ordering even for values, because client drivers generate ObjectId values, not the mongod process.
I then thought about querying with a timestamp:
db.tweets.find({ created: { $lt: maxDate } }, { limit: 50 })
However, there is no guarantee the date will be unique — it's quite likely that two documents could be created within the same second. This means documents could be missed when paging.
Is there any sort of ranged query that would provide me with more stability?
It is perfectly fine to use ObjectId() though your syntax for pagination is wrong. You want:
db.tweets.find().limit(50).sort({"_id":-1});
This says you want tweets sorted by _id value in descending order and you want the most recent 50. Your problem is the fact that pagination is tricky when the current result set is changing - so rather than using skip for the next page, you want to make note of the smallest _id in the result set (the 50th most recent _id value and then get the next page with:
db.tweets.find( {_id : { "$lt" : <50th _id> } } ).limit(50).sort({"_id":-1});
This will give you the next "most recent" tweets, without new incoming tweets messing up your pagination back through time.
There is absolutely no need to worry about whether _id value is strictly corresponding to insertion order - it will be 99.999% close enough, and no one actually cares on the sub-second level which tweet came first - you might even notice Twitter frequently displays tweets out of order, it's just not that critical.
If it is critical, then you would have to use the same technique but with "tweet date" where that date would have to be a timestamp, rather than just a date.
Wouldn't a tweet "actual" timestamp (i.e. time tweeted and the criteria you want it sorted by) be different from a tweet "insertion" timestamp (i.e. time added to local collection). This depends on your application, of course, but it's a likely scenario that tweet inserts could be batched or otherwise end up being inserted in the "wrong" order. So, unless you work at Twitter (and have access to collections inserted in correct order), you wouldn't be able to rely just on $natural or ObjectID for sorting logic.
Mongo docs suggest skip and limit for paging:
db.tweets.find({created: {$lt: maxID}).
sort({created: -1, username: 1}).
skip(50).limit(50); //second page
There is, however, a performance concern when using skip:
The cursor.skip() method is often expensive because it requires the server to walk from the beginning of the collection or index to get the offset or skip position before beginning to return result. As offset increases, cursor.skip() will become slower and more CPU intensive.
This happens because skip does not fit into the MapReduce model and is not an operation that would scale well, you have to wait for a sorted collection to become available before it can be "sliced". Now limit(n) sounds like an equally poor method as it applies a similar constraint "from the other end"; however with sorting applied, the engine is able to somewhat optimize the process by only keeping in memory n elements per shard as it traverses the collection.
An alternative is to use range based paging. After retrieving the first page of tweets, you know what the created value is for the last tweet, so all you have to do is substitute the original maxID with this new value:
db.tweets.find({created: {$lt: lastTweetOnCurrentPageCreated}).
sort({created: -1, username: 1}).
limit(50); //next page
Performing a find condition like this can be easily parallellized. But how to deal with pages other than the next one? You don't know the begin date for pages number 5, 10, 20, or even the previous page! #SergioTulentsev suggests creative chaining of methods but I would advocate pre-calculating first-last ranges of the aggregate field in a separate pages collection; these could be re-calculated on update. Furthermore, if you're not happy with DateTime (note the performance remarks) or are concerned about duplicate values, you should consider compound indexes on timestamp + account tie (since a user can't tweet twice at the same time), or even an artificial aggregate of the two:
db.pages.
find({pagenum: 3})
> {pagenum:3; begin:"01-01-2014#BillGates"; end:"03-01-2014#big_ben_clock"}
db.tweets.
find({_sortdate: {$lt: "03-01-2014#big_ben_clock", $gt: "01-01-2014#BillGates"}).
sort({_sortdate: -1}).
limit(50) //third page
Using an aggregate field for sorting will work "on the fold" (although perhaps there are more kosher ways to deal with the condition). This could be set up as a unique index with values corrected at insert time, with a single tweet document looking like
{
_id: ...,
created: ..., //to be used in markup
user: ..., //also to be used in markup
_sortdate: "01-01-2014#BillGates" //sorting only, use date AND time
}
The following approach wil work even if there are multiple documents inserted/updated at same millisecond even if from multiple clients (which generates ObjectId). For simiplicity, In following queries I am projecting _id, lastModifiedDate.
First page, fetch the result Sorted by modifiedTime (Descending), ObjectId (Ascending) for fist page.
db.product.find({},{"_id":1,"lastModifiedDate":1}).sort({"lastModifiedDate":-1, "_id":1}).limit(2)
Note down the ObjectId and lastModifiedDate of the last record fetched in this page. (loid, lmd)
For sencod page, include query condition to search if (lastModifiedDate = lmd AND oid > loid ) OR (lastModifiedDate < loid)
db.productfind({$or:[{"lastModifiedDate":{$lt:lmd}},{"_id":1,"lastModifiedDate":1},{$and:[{"lastModifiedDate":lmd},{"_id":{$gt:loid}}]}]},{"_id":1,"lastModifiedDate":1}).sort({"lastModifiedDate":-1, "_id":1}).limit(2)
repeat same for subsequent pages.
ObjectIds should be good enough for pagination if you limit your queries to the previous second (or don't care about the subsecond possibility of weirdness). If that is not good enough for your needs then you will need to implement an ID generation system that works like an auto-increment.
Update:
To query the previous second of ObjectIds you will need to construct an ObjectID manually.
See the specification of ObjectId http://docs.mongodb.org/manual/reference/object-id/
Try using this expression to do it from a mongos.
{ _id :
{
$lt : ObjectId(Math.floor((new Date).getTime()/1000 - 1).toString(16)+"ffffffffffffffff")
}
}
The 'f''s at the end are to max out the possible random bits that are not associated with a timestamp since you are doing a less than query.
I recommend during the actual ObjectId creation on your application server rather than on the mongos since this type of calculation can slow you down if you have many users.
I have build a pagination using mongodb _id this way.
// import ObjectId from mongodb
let sortOrder = -1;
let query = []
if (prev) {
sortOrder = 1
query.push({title: 'findTitle', _id:{$gt: ObjectId('_idValue')}})
}
if (next) {
sortOrder = -1
query.push({title: 'findTitle', _id:{$lt: ObjectId('_idValue')}})
}
db.collection.find(query).limit(10).sort({_id: sortOrder})
Related
I currently have a few issues with my Firestore querying technique. As per this stackoverflow post I made recently, Querying with two array with firestore security rules
The answer proposed to add the the "ids" into a object, with the key as the id, and the value simply being "true". I have completed this, and now my structure looks like so:
This leaves me with this query:
db.collection('Depots')
.where(`products.${productId}`, '==', true)
.where(`users.${userId}`, '==', true)
.where('created', '>', 1585998560500)
.orderBy('created', 'asc')
.get();
This query leaves me with throwing an error, asking to create an index:
The query requires an index. You can create it here: ...
However, this tries to index the specific object key, i.e. QXooVYGBIFWKo6C so products.QXooVYGBIFWKo6C. Which is certianly not what I want, as this query changes, and can have an infinite number of possibilities, which means I would have to create another index for each key entry in order to query it.
Is there any way to solve this issue? I am assuming it needs to index this query due to the different operators used in the query, so I was wondering if there were any workarounds to this issue.
Thank you very much in advance.
What you have here is a map field, for which indexes should usually be created automatically.
That indeed means that you'll have as many indexes as you have products, which means:
You are limited in how many products you can have, as there is a maximum of 40,000 index entries per document.
You pay more per document, as you pay for the storage of each index.
If these are not what you want, you'll have to switch back to your original model, with the query limitations you had there. There doesn't seem to be a solution that fits both of your requirements.
After our discussion in chat, this is the starting point I would suggest. Who knows what the end architecture would look like, but I think this or very close to this. You say that a user can exist in multiple depots at the same time and multiple depots can contain the same products, also at the same time. You also said that a depot can never have more than 40 users at a given time, so an array of 40 users would certainly not encroach on Firestore's document limit of 1,048,576 bytes.
[collection]
<documentId>
- field: value
[depots]
<UUID>
- depotId: string "depot456"
- productCount: num 5,000
<UUID>
- depotId: string "depot789"
- productCount: num 4,500
[products]
<UUID>
- productId: string "lotion123"
- depotId: string "depot456"
- users: [string] ["user10", "user27", "user33"]
<UUID>
- productId: string "lotion123"
- depotId: string "depot789"
- users: [string] ["user10", "user17", "user50"]
[users]
<userId>
- depots: [string] ["depot456", "depot999"]
<userId>
- depots: [string] ["depot333", "depot999"]
In NoSQL, storage is cheap and computation isn't so denormalize your data as much as you need to make your queries possible and efficient (fast and cheap).
To find all depots in a single query where user10 and lotion123 are both true, query the products collection where productId equals x and users array-contains y and collect the depotId values from those results. If you want to preserve the array-contains operation for something else, you'd have to denormalize your data further (replace the array for a single user). Or you could split this query into two separate queries.
With this model, when a user leaves a depot, get all products where users array-contains that user and remove that userId from the array. And when a user joins a depot, get all products where depotId equals x and append that userId to the array.
Watch this video, and others by Rick, to get a solid handle on NoSQL: https://www.youtube.com/watch?v=HaEPXoXVf2k
#danwillm If you are not sure about the number of users and products then your DB structure seems unfit for this situation because there are size and length limitations of the firestore document.
You should rather create a separate collection for products and users i.e normalize your data and have a reference for the user in the product collection.
User :
{
userId: documentId,
name: John,
...otherInfo
}
Product :
{
productId: documentId,
createdBy: userId,
createdOn:date,
productName:"exa",
...otherInfo
}
This way you there will be the size of the document would be limited, i.e try avoiding using maps/arrays in firestore if you are not sure about there size.
Also, in this case, the number of queries would be increased but you don't need many indexes in this case.
This post will be quite long, if you are a mongo god and you want to help me anyway, I thank you from the bottom of my heart. I try to be as exaustive as possible, with all the data I collected.
I'm experiencing some strange behaviour in my MongoDD database, and I'm questionning mongodb's write order.
I have logged bugs, that only happen at the time of the execution, that made me think that we have a timming problem here, but the requesting is so slow, that I don't understand how It can happen in suck conditions
Starting points:
All ObjectIds are auto-given by mongo, I never set them on my side
This objectId is indexed (obviously) and unique
I use the timestamp in those requests ids to query the DB, using this request:
db.getCollection('eventStore').find({
'_id': {
'$gt': ObjectId("5d285c784460c502cc66ff9b"),
'$lte': ObjectId("5d285cf7856cda0266215c77")
}
})
The results of this request are then streamed using the basic Node.js client possibilies:
collection.find({
'_id': {
...(lower ? { '$gt': lower } : {}),
'$lte': higher
}
}).sort({ _id: 1 }).stream({
transform: (element) => {
logger.info(`Exiting Get events by range::${JSON.stringify(lower)}::${JSON.stringify(higher)}`)
logger.info(`Parse event::${JSON.stringify(element)}}`)
return // PARSED EVENT
}
})
At this point, I know that the .sort({ _id: 1 }) is probably useless, but I keep it here anyway, just in case.
The requesting procedure is quite slow: I query the upperbound in order to have the eventStore's offset, then the view's offset, then I query the store in order to have all events between these boundaries.
The sample of mongo's data:
/* 1 */
{
"_id" : ObjectId("5d285cf77f6482027108c15c"),
"events" : [
// Some events
]
}
/* 2 */
{
"_id" : ObjectId("5d285cf77f6482027108c15d"),
"events" : [
// Some events
]
}
/* 3 */
{
"_id" : ObjectId("5d285cf7856cda0266215c77"),
"events" : [
// Some events
]
}
Expected Mongo Behaviour:
Mongo is self assigning the Ids, so I expect that when the ID
ObjectId("5d285cf7856cda0266215c77") is stored in base, all the IDs with a older timestamp are stored safe in the base already. ==> No backpedaling
The timestamp encoded in the ObjectId (when it's self given by mongo) is the one when it's written and persisted in the DB, not the one when it's received by Mongo.
Actual behaviour (as logged in the logger's function):
Only the first and the last _ids have been seen in the logger's function
{"message":"Exiting Get events by range::\"5d285c784460c502cc66ff9b\"::\"5d285cf7856cda0266215c77\"","level":"info","timestamp":"2019-07-12 10:12:07"}
{"message":"Parse event::{\"_id\":\"5d285cf77f6482027108c15c\",\"events\":[ // Data ]}}","level":"info","timestamp":"2019-07-12 10:12:07"}
{"message":"Exiting Get events by range::\"5d285c784460c502cc66ff9b\"::\"5d285cf7856cda0266215c77\"","level":"info","timestamp":"2019-07-12 10:12:07"}
{"message":"Parse event::{\"_id\":\"5d285cf7856cda0266215c77\",\"events\":[ // Data ]}}","level":"info","timestamp":"2019-07-12 10:12:07"}
As it seems:
Or the event _id: ObjectId("5d285cf77f6482027108c15d") was not in base at the time of the call, but the older _id: ObjectId("5d285cf7856cda0266215c77") was.
Or my request or stream is invalid at the time of the call (but now, returns the correct answer)
Something else ?
Mongo is self assigning the Ids, so I expect that when the ID ObjectId("5d285cf7856cda0266215c77") is stored in base, all the IDs with a older timestamp are stored safe in the base already. ==> No backpedaling
Mongo _ids are ObjectIds which are:
a 4-byte value representing the seconds since the Unix epoch,
a 5-byte random value, and
a 3-byte counter, starting with a random value.
These are often generated in application driver code (on the server that's sending data to mongo).
This means that:
Network delays can create out of order records
Application servers with clock drift can create out of order records
5 byte random values don't necessarily move forward within the same second (which can create out of order records, even if mongo is assigning the _ids)
NTP updates can create out of order records (even on mongo -- it doesn't do any smearing when updating the clock)
Leap seconds can create out of order records
If you take a look at the _ids that you shared, the first 4 bytes (8 characters) of 5d285cf77f6482027108c15d & 5d285cf7856cda0266215c77 (5d285cf7) both share the same timestamp because they happened in the same second after the epoch.
I'm using MiniMongo through Meteor, and I'm trying to create a frequency table based off of a dynamic set of queries.
I have two main fields, localHour and localDay. I expect many overlaps, and I'd like to determine where the most overlaps occur. My current method of doing this is so.
if(TempStats.findOne({
localHour: hours,
localDay: day
})){//checks if there is already some entry on the same day/hour
TempStats.update({//if so, we just increment frequency
localHour: hours,
localDay: day
},{
$inc: {freq: 1}
})
} else {//if nothing exists yet, we put in a new entry
TempStats.insert({
localHour: hours,
localDay: day,
freq: 1
});
}
Essentially, this code runs every time I have new data I want to insert. It works fine at the moment, in that, after all data is inserted, I can sort by frequency to find what set of hours & days occurs the most often (TempStats.find({}, {sort: {freq: -1}}).fetch()).
However, I'm looking more for a way to search by frequency for any key. For instance, searching for the day which everything occurs on the most often as opposed to both the date and hour. With my current way of doing this, I would need to have multiple databases and different methods of inserting for each, which is a bit ridiculous. Is there a Mongo (specifically MiniMongo) solution to do frequency maps based on keys?
Thanks!
It looks like miniMongo does not in fact support aggregation, which makes this kind of operation difficult. One way to go about it would be aggregating yourself at the end of each day and inserting that aggregate record into your db (without the hour field or with it set to something like -1). Conversely as wastefully you could also update that record at the time of each insert. This would allow you to use the same collection for both and is fairly common in other dbs.
Also you should consider #nickmilon's first suggestion since the use of an upsert statement with the $inc operator would reduce your example to a single operation per data point.
a small note on your code: the part that comes as an else statement is not really required your update will do the complete job if you combine it with the option upsert=true it will insert a new document and $inc will set the freq field to 1 as desired see: here and here
for alternative ways to count your frequencies: assuming you store the date as a datetime object I would suggest to use an aggregation (I am not sure if they added support for aggregation yet in minimongo) but there are solutions then with aggregation you can use datetime operators as
$hour, $week, etc for filtering and $count to count the frequencies without you having to keep counts in the database.
This is basically a simple map-reduce problem.
First, don't separate the derived data into 2 fields. This violates DB best practices. If the data comes to you this way, use it to create a Date object. I assume you have a bunch of collections that are being subscribed to and then you aggregate all those into this temporary local collection. This is the mapping of the map-reduce pattern. At this point, since your query in unknown, it's a waste of CPU (even though it's your client) to aggregate. Map first, reduce second. What you should have is a collection full of datetimes. call it TempMapCollection if you wish. Now, use a forEach() and pass in your reduce function (by day, by hour, etc).
You can reduce into another local collection, or into a javascript object. I like using collections, but if the objects are complex, you'll get EJSON errors all up in there. Since your objects are nothing more than a datetime, let's use collections.
so you've got something like:
TempMapCollection.find().forEach(function(doc) {
var date = doc.dateTime.getDate();
TempReduceCollection.upsert({timequery: hours}, {$inc: {freq: 1}});
})
Now query your reduce collection. This has the added benefit that you won't have to re-map if you want to do 2 unique queries.
In my node js app Im using Mongo DB , and I have an Issue when Inserting something in database. When I add new record to collection objects beign reordered . Does anyone knows why is this happening ? Im doing insert like this
collection.insert(object, {safe:true}, function(err, result) {
if (err) {
res.send({'error':'An error has occurred'});
} else {
// Error
}
});
Actually , any operation on collection change the order of objects , does anyone knows why is this happening ?
MongoDB documents have padding space that is used for updates. If you make small changes to a document like adding/updating a small field there is a good chance that the size of the updated document will increase but will still fit into allocated space because it can use that padding. If the updated document does not fit in that space Mongo will move it to a new place on disk. Thus your documents might move a lot in the beginning until Mongo learns how much padding you would usually need to prevent such moves. You can also set higher padding to avoid documents being moved in the first place.
In either case you can't really rely on the insertion order to get sorted list of documents. If you want guaranteed order you need to sort. In your case you can sort by _id because it's a monotonically increasing counter which contains date and time details:
// in the order of insertion unless you got `_id` value externally
// (in your code, not auto assigned by Mongo) and doc with such ID
// was inserted much later.
// Sharding might also introduce tiny ordering mismatches
db.collection.find().sort( { '_id': 1 } );
// most recently inserted items first
db.collection.find().sort( { '_id': -1 } );
If you use capped collection the order of inserts will be preserved always, i.e. Mongo will never move documents. In that case you can use natural order sorting:
db.collection.find().sort( { $natural: 1 } )
which is equivalent to sorting by _id as shown above.
Do not use natural order sorting with non-capped collections (regular collections) because it will not be reliable in presence of updates.
I'm using what seems to be a common trick for creating a join view:
// a Customer has many Orders; show them together in one view:
function(doc) {
if (doc.Type == "customer") {
emit([doc._id, 0], doc);
} else if (doc.Type == "order") {
emit([doc.customer_id, 1], doc);
}
}
I know I can use the following query to get a single customer and all related Orders:
?startkey=["some_customer_id"]&endkey=["some_customer_id", 2]
But now I've tied my query very closely to my view code. Is there a value I can put where I put my "2" to more clearly say, "I want everything tied to this Customer"? I think I've seen
?startkey=["some_customer_id"]&endkey=["some_customer_id", {}]
But I'm not sure that {} is certain to sort after everything else.
Credit to cmlenz for the join method.
Further clarification from the CouchDB wiki page on collation:
The query startkey=["foo"]&endkey=["foo",{}] will match most array keys with "foo" in the first element, such as ["foo","bar"] and ["foo",["bar","baz"]]. However it will not match ["foo",{"an":"object"}]
So {} is late in the sort order, but definitely not last.
I have two thoughts.
Use timestamps
Instead of using simple 0 and 1 for their collation behavior, use a timestamp that the record was created (assuming they are part of the records) a la [doc._id, doc.created_at]. Then you could query your view with a startkey of some sufficiently early date (epoch would probably work), and an endkey of "now", eg date +%s. That key range should always include everything, and it has the added benefit of collating by date, which is probably what you want anyways.
or, just don't worry about it
You could just index by the customer_id and nothing more. This would have the nice advantage of being able to query using just key=<customer_id>. Sure, the records won't be collated when they come back, but is that an issue for your application? Unless you are expecting tons of records back, it would likely be trivial to simply pluck the customer record out of the list once you have the data retrieved by your application.
For example in ruby:
customer_records = records.delete_if { |record| record.type == "customer" }
Anyways, the timestamps is probably the more attractive answer for your case.
Rather than trying to find the greatest possible value for the second element in your array key, I would suggest instead trying to find the least possible value greater than the first: ?startkey=["some_customer_id"]&endkey=["some_customer_id\u0000"]&inclusive_end=false.
CouchDB is mostly written in Erlang. I don't think there would be an upper limit for a string compound/composite key tuple sizes other than system resources (e.g. a key so long it used all available memory). The limits of CouchDB scalability are unknown according to the CouchDB site. I would guess that you could keep adding fields into a huge composite primary key and the only thing that would stop you is system resources or hard limits such as maximum integer sizes on the target architecture.
Since CouchDB stores everything using JSON, it is probably limited to the largest number values by the ECMAScript standard.All numbers in JavaScript are stored as a floating-point IEEE 754 double. I believe the 64-bit double can represent values from - 5e-324 to +1.7976931348623157e+308.
It seems like it would be nice to have a feature where endKey could be inclusive instead of exclusive.
This should do the trick:
?startkey=["some_customer_id"]&endkey=["some_customer_id", "\uFFFF"]
This should include anything that starts with a character less than \uFFFF (all unicode characters)
http://wiki.apache.org/couchdb/View_collation