Develop Reference

Best way to batch create if not exists in firestore

I am working with a project where we create a bunch of entries in firestore based on results from an API endpoint we do not control, using a firestore cloud function. The API endpoint returns ids which we use for the document ids, but it does not include any timestamp information. Since we want to include a createdDate in our documents, we are using admin.firestore.Timestamp.now() to set the timestamp of the document.
On subsequent runs of the function, some of the documents will already exist so if we use batch.commit with create, it will fail since some of the documents exist. However, if we use batch.commit with update, we will either not be able to include a timestamp, or the current timestamp will be overwritten. As a final requirement, we do update these documents from a web application and set some properties like a state, so we can't limit the permissions on the documents to disallow update completely.
What would be the best way to achieve this?
I am currently using .create and have removed the batch, but I feel like this is less performant, and I occasionally do get the error Error: 4 DEADLINE_EXCEEDED on the firestore function.
First prize would be a batch that can create or update the documents, but does not edit the createdDate field. I'm also hoping to avoid reading the documents first to save a read, but I'd be happy to add it in if it's the best solution.
Thanks!
Current code is something like this:
const createDocPromise = docRef
.create(newDoc)
.then(() => {
// success, do nothing
})
.catch(err => {
if (
err.details &&
err.details.includes('Document already exists')
) {
// doc already exists, ignore error
} else {
console.error(`Error creating doc`, err);
}
});

This might not be possible with batched writes as set() will overwrite the existing document, update() will update the timestamp and create() will throw an error as you've mentioned. One workaround would be to use create() for each document with Promise.allSettled() that won't run catch() if any of the promise fails.
const results = [] // results from the API
const promises = results.map((r) => db.doc(`col/${r.id}`).create(r));
const newDocs = await Promise.allSettled(promises)
// either "fulfilled" or "rejected"
newDocs.forEach((result) => console.log(result.status))
If any documents exists already, create() will throw an error and status for that should be rejected. This way you won't have to read the document at first place.
Alternatively, you could store all the IDs in a single document or RTDB and filter out duplicates (this should only cost 1 read per invocation) and then add the data.

Since you prefer to keep the batch and you want to avoid reading the documents, a possible solution would be to store the timestamps in a field of type Array. So, you don't overwrite the createdDate field but save all the values corresponding to the different writes.
This way, when you read one of the documents you sort this array and take the oldest value: it is the very first timestamp that was saved and corresponds to the document creation.
This way you don't need any extra writes or extra reads.

deleting a collection that includes a nested sub collection in firebase?

I have the following batch, where I delete three documents in a collection,
the data-set-c collection however has one nested collection for massages where each message is a doc each.
the problem is that the (data-set-c) collection never gets deleted, I don't know if it is due to the nesting taking place? will deleting this way affect also sub-collection? or is it the rules that are blocking it since my rules are specific to the deepest level endpoint, or should use the cloud function instead since this doc will be massive later on, and each day is massages collection but the main issue here is how to delete this one level nested structure.
could you please take a look and see what I am doing wrong.
// Batch
const batch = writeBatch(db);
// data-set-a/{Id}/sub-set-a/{subSetId} ---> the direct document
const colA = collection(db, data-set-a,_authContext.currentUser.uid, sub-set-a);
// data-set-b/{Id}/sub-set-b/{subSetId} ---> the direct document
const colB = collection(db, data-set-b, _authContext.currentUser.uid, sub-set-b);
// data-set-c/{Id}/sub-set-c/{subSetId}/masseges/{mgsId} /*nested collection*/
const colC = collection(db, 'data-set-c', _authContext.currentUser.uid, sub-set-c);
const docA = doc(colA, subSetId);
const docB = doc(colB, subSetId);
const docC = doc(colC, subSetId);
const docsArr = [docA, docB, docC];
docsArr.forEach((col: any) => {
batch.delete(col)
});
await batch.commit();
// sub-set-a + sub-set-b SECURITY RULES
match /data-set-a/Id}sub-set-a/{subId} {
allow delete: if request.auth != null
&& request.auth.token.email_verified
&& request.auth.uid == Id
}
// sub-set-c SECURITY RULES
match /data-set-c/{Id}/sub-set-c/{subSetId}/masseges/{mgsId} {
allow delete: if request.auth != null
&& request.auth.token.email_verified
&& request.auth.uid == Id
}

the problem is that the (data-set-c) collection never gets deleted.
That's the expected behavior. If you delete a document, doesn't mean that you delete all sub-collections that exist within that document. Besides that, that document ID will continue to exist and the Firebase Console will display it in italic.
will deleting this way affect also sub-collection?
No, it won't.
or is it the rules that are blocking it since my rules are specific to the deepest level endpoint.
If it was about the security rules, then you would have received an error indicating that.
or should use cloud function instead since this doc will be massive later on
Yes, you can indeed use a Cloud Function to delete all documents that exist in all sub-collection of the document that was deleted.
How to delete this one-level nested structure.
You can achieve this by iterating each sub-collection, and by deleting each document. But don't do it on the client!

Pushing String into database utilizing nodeJS and mongoDB

I am new to JS and I am utilizing the MEAN stack to create a place where students can add classes to their user profile. I already have a session store in my database, so "req.user" will return the currently logged in user information and specifically "req.user.id" will return the currently logged in user's id. Also, I have figured out how to search a course in my database from my application. Ultimately, my goal is that when the user makes the post request to search in the database, I also want those "values" to be pushed into the classes "key". I have provided two options, both of which do not add the respective strings to the database. Thank you for any help!
Portion of Search.JS Option #1
router.post('/', ensureAuthenticated, function (req,res,next) {
var query = {course: req.body.coursename};
db.collection('courses').find(query).toArray()
.then(db.collection('DefaultUser').update({_id: req.user.id}, {$push: {classes: req.body.coursename}}));
res.render('search', {title: 'search'})
});
Portion of Search.JS Option #2
router.post('/', ensureAuthenticated, function(req,res,next) {
var query = {course: req.body.coursename};
db.collection('courses').find(query).toArray((err, results) => {
if (err) {
throw err;
} else {
db.collection('DefaultUser').updateOne({_id: '5c17161d3347e79410ff29ba'}, {
$push: {
classes: req.body.coursename
}
})
console.log(results)
res.render('search', {courses: results, title: 'search'})
}
})
});

Some tips may help:
req.body will hold nothing if you forget to use app.use(express.urlencoded()) to parse the posted data.
You may use ObjectId('<string>') (in option #2) for finding and updating queries, not just the string, because mongodb stores _id as ObjectId type by default.
You may use $addToSet instead of $push modifier to add a unique element to an array, or you may get one student subscribed with two more same class, if he repeatedly submit the form.
In your code, you find in the courses collection first and then update, but since you did nothing with the find result, it is not necessary (empty result does not throw any error). Checking the data is valid is good practice, if you would like to do so, in both options, you need to use findOne instead of find to make sure the course is existed, and .then(course => {/* check if the course is null and then throws an error */} ).
I don't have the full code of your project so I can only guess there may be the problems listed above, wish you good luck!

Does modelObject.save() only update an existing database document when the modelObject was obtained from the database itself?

To use an example that demonstrates the question, assume I have a User model defined by the following schema:
var UserSchema = new Schema({
username: String,
email: String
}
mongoose.model('User', UserSchema);
I know that to update a user using the save method, I could query the user and then save changes like so:
User.findOne({username: usernameTofind}, function(err, user) {
//ignore errors for brevity
user.email = newEmail;
user.save(function(err) { console.log('User email updated') });
});
But if I try to create a new User object with the exact same field values (including the _id) is there any possibility of overwriting the database document? I would assume not, because in theory this would mean that a malicious user could exploit an insecure api and overwrite existing documents (for instance using a 'Create a New Account' request, which wouldn't/couldn't rely on the user already being authenticated) , but more importantly, when I try to do this using a request tool (I'm using Postman, but I'm sure a similar curl command would suffice), I get a duplicate _id error
MongoError: insertDocument :: caused by :: 11000 E11000 duplicate key error index
So I just want to clarify that the only way to update an existing document is to query for the document, modify the returned instance, then call the save method on that instance, OR use the static update(). Both of these could be secured by requiring authentication.
If it helps, my motivation for this question is mentioned above, in that I want to make sure a user is not able to overwrite an existing document if a method such as the following is exposed publicly:
userCtrl.create = function(req, res, next) {
var user = new User(req.body);
user.save(function(err) {
if (err) {
return next(err);
} else {
res.json(user);
}
});
};
Quick Edit: I just realized, if this is the case, then how does the database know the difference between the queried instance and a new User object with the exact same keys and properties?

Does modelObject.save() only update an existing database document when
the modelObject was obtained from the database itself?
Yes, it does. There is a flag that indicates if the document is new or not. If it is new, Mongoose will insert the document. If not, then it will update the document. The flag is Document#isNew.
When you find a document:
User.findOne({username: usernameTofind}, function(err, user) {
//ignore errors for brevity
console.log(user.isNew) // => will return false
});
When you create a new instance:
var user = new User(req.body);
console.log(user.isNew) // => will return true
So I just want to clarify that the only way to update an existing
document is to query for the document, modify the returned instance,
then call the save method on that instance, OR use the static
update(). Both of these could be secured by requiring authentication.
There are other ways you can update documents, using Model#update, Model.findOneAndUpdate and others.
However, you can't update an _id field. MongoDB won't allow it even if Mongoose didn't already issue the proper database command. If you try it you will get something like this error:
The _id field cannot be changed from {_id: ObjectId('550d93cbaf1e9abd03bf0ad2')} to {_id: ObjectId('550d93cbaf1e9abd03bf0ad3')}.
But assuming you are using the last piece of code in your question to create new users, Mongoose will issue an insert command, so there is no way someone could overwrite an existing document. Even if it passes an _id field in the request body, MongoDB will throw a E11000 duplicate key error index error.
Also, you should filter the fields a user can pass as payload before you use them to create the user. For example, you could create a generic function that receives an object and an array of allowed parameters:
module.exports = function(object, allowedParams) {
return Object.keys(object).reduce(function(newObject, param) {
if (allowedParams.indexOf(param) !== -1)
newObject[param] = object[param];
return newObject;
}, {});
}
And then you only require and use the function to filter the request body:
var allow = require('./parameter-filter');
function filter(params) {
return allow(params, ["username", "email"]);
}
var user = new User(filter(req.body));

Mongoosejs refresh a document

Suppose I have a document for example: var doc = Model.findOne({name:"name"});
Now if the document gets edited trough another connection the the database, doc doesn't hold the right information. I do need it, so I have to "refresh" or "redownload" it from the database. Is there any way to do this with only the object "doc"?

Assuming doc contains the document instance to refresh, you can do this to generically refresh it:
doc.model(doc.constructor.modelName).findOne({_id: doc._id},
function(err, newDoc) {
if (!err) {
doc = newDoc;
}
}
);
However, it's better to not persist/cache Mongoose document instances beyond your immediate need for them. Cache the immutable _id of docs you need to quickly access, not the docs themselves.

Sorry I know this is old but I have a better solution in case anyone else is still looking around.
What you can do in this situation of concurrent access to the same document is perform a normal save.
Mongoose uses the __v property of an object to make sure old instances don't overwrite the newest ones.
So if you have 2 instances of doc.__v = 0, and A saves first, the db now has doc.__v = 1. So that when B saves, it'll error and because B.__v = 0, not 1. So what you can do is then catch the error (it should be a specific mongoose error) and handle it by refreshing the object, (which should bump it up to __v = 1) then either SAVE it or not SAVE it depending on what you want to do.
so in code
doc.save(function (err, _doc) {
if (err) {
if (err.some_error_property_telling_you_version_was_wrong) {
// Do your business here
// Refresh your object
// Model.findById(doc._id, function (err, _refreshed_doc) {});
// doc = _refreshed_doc;
// doc.set('property', value); set the value again
// Then re-save? or maybe make this a recursive function,
// so it can keep trying to save the object?
} else {
// Handle it like a normal error
}
}
});

If it's defined with let you can just do
await thing.update(stuff);
thing = await Thing.findById(thing._id);
or if it was with const then do it immutably:
await thing.update(stuff);
const updatedThing = await Thing.findById(thing._id);