When adding a field called date in mongo db, I can do just:
date: {
type: Date,
default: Date.now
}
and it will automatically add date field to my new collection when it was created. Is there some way to add a self-incrementing index (id) to my collections?
Note: I tried to do it on client side, however with every time I push collection with (id) field, its being deleted from the collection and replaced with _id which is a long string with random characters. Way to long!
Looking for every hints.
Edit: code responsible for adding use to db
app.post("/users", function (req, res) {
createUser(req.body, function (err, user) {
if (err) {
return res.json(err);
}
return res.json(user);
});
});
MongoDB automatically makes unique ids for each object in the database, resulting in each entry having a unique _id field being an ObjectId. You don't really need to worry about specifying custom ids.
You can sort by _id if you want objects roughly in the order they were created, or you could add a date field which is set on creation and sort by that.
Other than that, I'm not sure what you'd gain by having auto incrementing ids
There are multiple ways to implement an auto-increment index but it is not considered a good practice.
Detailed information here: Auto increment in MongoDB to store sequence of Unique User ID
Check Rizwan Siddiquee answer about how to implement it with a stored javascript function.
Another way would be to implement it on application layer using any kind of ODM but this is obviously dangerous and not so trustable for serious applications.
Related
So I'm working on a personal project to learn react-native and Firestore.
I have a DB like this:
And I want my code to add a new battery in the array batteries.
The elements in the array are just a map{string, string}
The problem is that when I update the array with a new brand that's work but if I want to update it with the same brand again have,
so having by the end
batteries[0]: {'brand': 'cestmoi'}
batteries[1]: {'brand': 'cestmoi'}
The DB doesn't update, doesn't have any error or so.
I don't understand why and I followed their tutorial. Here is my code:
async function addData(collection, doc, value) {
console.log(`Add data ${value.brand}`)
try {
const result = await firestore()
.collection(collection)
.doc(doc)
.set({
batteries: firestore.FieldValue.arrayUnion(value)
})
console.log(result);
return result;
} catch (error) {
return error;
}
}
I use try-catch by habit but I don't know if the then...catch is better or not.
As already #windowsill mentioned in his answer, there is no way you can add duplicate elements in an array using client-side code. If your application requires that, then you have to read the entire array, add the duplicates and then write the document back to Firestore.
However, if you want to update an existing element in an array of objects (maps) then you have to use arrayUnion with the entire object. If you want to understand the mechanism better, you can read the following article which is called:
How to update an array of objects in Firestore?
arrayUnion says that it "adds elements to an array but only elements not already present". Maybe it does a stringify or something to check equality and therefore doesn't add the new element. I think you'll have to 1. get the current list, 2. add your element, 3. set the batteries field to the updated list.
As far as I know, firebase assigns automatically an unique ID to every new entry in the database. However - these ids are really long and not good looking.
Whats more - I have to refer to them somehow, so currently when Im doing a get request, e.g. to get one entry Im doing something like:
/getEntry/L4Cu7UOENIivnB2bgt
And it's fine, since user doesn't see it anyways.
Hovewer, when making routes to every entry in my app, again I have to refer to specific entry by it's id. So e.g. if Im on route of specified element, e.g.:
http://myapp.com/users/L4Cu7UOENIivnB2bgt - it doesn't look very well if not ugly. If I would make my db in e.g. SQL or NoSQL, I would be able to assign an id by myself so it would increase from 1 and so on.
Q: Am I able to change these long id's somehow? It has to be fixable somehow... Thanks.
Yes you can set your own unique key. Say you have unique usernames for each user then you can do
firebase.database().ref('users/' + userName).set({
firstName: name,
email: email,
profile_picture : imageUrl
});
or you can create your own unique ids and use instead. But there is no auto incremental ids.
Using set() overwrites data at the specified location, including any child nodes.
I'm using Firebase and Vuejs to create an database element, which has object array inside.
That's how the field looks, and I want to add tasks through the form into the 'moreTasks' as an array.
I tried using this, but it just creates new entity in the database.
db.collection('Tasks').add({
tasker: this.tasker.taskerName
})
I also tried checking API but I couldnt understand the refs, because I was using different methods to achieve that goal.
creatTask() {
db.collection('Tasks').add({
task_id: this.task_id,
name: this.name,
What would be correct way to approach this problem?
You can append an item to an array using FieldValue.arrayUnion() as described in the documentation. For example:
// Atomically add a new region to the "regions" array field.
washingtonRef.update({
regions: firebase.firestore.FieldValue.arrayUnion("greater_virginia")
});
The accepted answer used to be correct but is now wrong. Now there is an atomic append operation using the arrayUnion method:
https://firebase.google.com/docs/firestore/manage-data/add-data#update_elements_in_an_array
This is true as long as you are using firestore and not real time db (which I assume is the case from the tags)
Is there any methods or packages, that can help me add auto increments to existing collection? Internet full of information, about how to add AI before you create collection, but I did not find information on how to add AI when collection already exist...
MongoDB does not have an inbuilt auto-increment functionality.
Create a new collection to keep track of the last sequence value used for insertion:
db.createCollection("counter")
It will hold only one record as:
db.counter.insert({_id:"mySequence",seq_val:0})
Create a JavaScript function as:
function getNextSequenceVal(seq_id){
// find record with id seq_id and update the seq_val by +1
var sequenceDoc = db.counter.findAndModify({
query:{_id: seq_id},
update: {$inc:{seq_val:1}},
new:true
});
return sequenceDoc.seq_val;
}
To update all the already existing values in your existing collection, this should work (For the empty {}, you can place your conditions if you want to update some documents only):
db.myCollection.update({},
{$set:{'_id':getNextSequenceVal("mySequence")}},{multi:true})
Now you can insert new records into your existing collection as:
db.myCollection.insert({
"_id":getNextSequenceVal("mySequence"),
"name":"ABC"
})
MongoDB reserves the _id field in the top level of all documents as a primary key. _id must be unique, and always has an index with a unique constraint. It is an auto-incrementing field. However, it is possible to define your own auto-incrementing field following the tutorial in the MongoDB documentation.
Tutorial link: https://docs.mongodb.com/v3.0/tutorial/create-an-auto-incrementing-field/
I want to implement pagination on top of a MongoDB. For my range query, I thought about using ObjectIDs:
db.tweets.find({ _id: { $lt: maxID } }, { limit: 50 })
However, according to the docs, the structure of the ObjectID means that "ObjectId values do not represent a strict insertion order":
The relationship between the order of ObjectId values and generation time is not strict within a single second. If multiple systems, or multiple processes or threads on a single system generate values, within a single second; ObjectId values do not represent a strict insertion order. Clock skew between clients can also result in non-strict ordering even for values, because client drivers generate ObjectId values, not the mongod process.
I then thought about querying with a timestamp:
db.tweets.find({ created: { $lt: maxDate } }, { limit: 50 })
However, there is no guarantee the date will be unique — it's quite likely that two documents could be created within the same second. This means documents could be missed when paging.
Is there any sort of ranged query that would provide me with more stability?
It is perfectly fine to use ObjectId() though your syntax for pagination is wrong. You want:
db.tweets.find().limit(50).sort({"_id":-1});
This says you want tweets sorted by _id value in descending order and you want the most recent 50. Your problem is the fact that pagination is tricky when the current result set is changing - so rather than using skip for the next page, you want to make note of the smallest _id in the result set (the 50th most recent _id value and then get the next page with:
db.tweets.find( {_id : { "$lt" : <50th _id> } } ).limit(50).sort({"_id":-1});
This will give you the next "most recent" tweets, without new incoming tweets messing up your pagination back through time.
There is absolutely no need to worry about whether _id value is strictly corresponding to insertion order - it will be 99.999% close enough, and no one actually cares on the sub-second level which tweet came first - you might even notice Twitter frequently displays tweets out of order, it's just not that critical.
If it is critical, then you would have to use the same technique but with "tweet date" where that date would have to be a timestamp, rather than just a date.
Wouldn't a tweet "actual" timestamp (i.e. time tweeted and the criteria you want it sorted by) be different from a tweet "insertion" timestamp (i.e. time added to local collection). This depends on your application, of course, but it's a likely scenario that tweet inserts could be batched or otherwise end up being inserted in the "wrong" order. So, unless you work at Twitter (and have access to collections inserted in correct order), you wouldn't be able to rely just on $natural or ObjectID for sorting logic.
Mongo docs suggest skip and limit for paging:
db.tweets.find({created: {$lt: maxID}).
sort({created: -1, username: 1}).
skip(50).limit(50); //second page
There is, however, a performance concern when using skip:
The cursor.skip() method is often expensive because it requires the server to walk from the beginning of the collection or index to get the offset or skip position before beginning to return result. As offset increases, cursor.skip() will become slower and more CPU intensive.
This happens because skip does not fit into the MapReduce model and is not an operation that would scale well, you have to wait for a sorted collection to become available before it can be "sliced". Now limit(n) sounds like an equally poor method as it applies a similar constraint "from the other end"; however with sorting applied, the engine is able to somewhat optimize the process by only keeping in memory n elements per shard as it traverses the collection.
An alternative is to use range based paging. After retrieving the first page of tweets, you know what the created value is for the last tweet, so all you have to do is substitute the original maxID with this new value:
db.tweets.find({created: {$lt: lastTweetOnCurrentPageCreated}).
sort({created: -1, username: 1}).
limit(50); //next page
Performing a find condition like this can be easily parallellized. But how to deal with pages other than the next one? You don't know the begin date for pages number 5, 10, 20, or even the previous page! #SergioTulentsev suggests creative chaining of methods but I would advocate pre-calculating first-last ranges of the aggregate field in a separate pages collection; these could be re-calculated on update. Furthermore, if you're not happy with DateTime (note the performance remarks) or are concerned about duplicate values, you should consider compound indexes on timestamp + account tie (since a user can't tweet twice at the same time), or even an artificial aggregate of the two:
db.pages.
find({pagenum: 3})
> {pagenum:3; begin:"01-01-2014#BillGates"; end:"03-01-2014#big_ben_clock"}
db.tweets.
find({_sortdate: {$lt: "03-01-2014#big_ben_clock", $gt: "01-01-2014#BillGates"}).
sort({_sortdate: -1}).
limit(50) //third page
Using an aggregate field for sorting will work "on the fold" (although perhaps there are more kosher ways to deal with the condition). This could be set up as a unique index with values corrected at insert time, with a single tweet document looking like
{
_id: ...,
created: ..., //to be used in markup
user: ..., //also to be used in markup
_sortdate: "01-01-2014#BillGates" //sorting only, use date AND time
}
The following approach wil work even if there are multiple documents inserted/updated at same millisecond even if from multiple clients (which generates ObjectId). For simiplicity, In following queries I am projecting _id, lastModifiedDate.
First page, fetch the result Sorted by modifiedTime (Descending), ObjectId (Ascending) for fist page.
db.product.find({},{"_id":1,"lastModifiedDate":1}).sort({"lastModifiedDate":-1, "_id":1}).limit(2)
Note down the ObjectId and lastModifiedDate of the last record fetched in this page. (loid, lmd)
For sencod page, include query condition to search if (lastModifiedDate = lmd AND oid > loid ) OR (lastModifiedDate < loid)
db.productfind({$or:[{"lastModifiedDate":{$lt:lmd}},{"_id":1,"lastModifiedDate":1},{$and:[{"lastModifiedDate":lmd},{"_id":{$gt:loid}}]}]},{"_id":1,"lastModifiedDate":1}).sort({"lastModifiedDate":-1, "_id":1}).limit(2)
repeat same for subsequent pages.
ObjectIds should be good enough for pagination if you limit your queries to the previous second (or don't care about the subsecond possibility of weirdness). If that is not good enough for your needs then you will need to implement an ID generation system that works like an auto-increment.
Update:
To query the previous second of ObjectIds you will need to construct an ObjectID manually.
See the specification of ObjectId http://docs.mongodb.org/manual/reference/object-id/
Try using this expression to do it from a mongos.
{ _id :
{
$lt : ObjectId(Math.floor((new Date).getTime()/1000 - 1).toString(16)+"ffffffffffffffff")
}
}
The 'f''s at the end are to max out the possible random bits that are not associated with a timestamp since you are doing a less than query.
I recommend during the actual ObjectId creation on your application server rather than on the mongos since this type of calculation can slow you down if you have many users.
I have build a pagination using mongodb _id this way.
// import ObjectId from mongodb
let sortOrder = -1;
let query = []
if (prev) {
sortOrder = 1
query.push({title: 'findTitle', _id:{$gt: ObjectId('_idValue')}})
}
if (next) {
sortOrder = -1
query.push({title: 'findTitle', _id:{$lt: ObjectId('_idValue')}})
}
db.collection.find(query).limit(10).sort({_id: sortOrder})