Develop Reference

How and when to write entity relationships with Firebase Database [duplicate]

I've read the Firebase docs on Stucturing Data. Data storage is cheap, but the user's time is not. We should optimize for get operations, and write in multiple places.
So then I might store a list node and a list-index node, with some duplicated data between the two, at very least the list name.
I'm using ES6 and promises in my javascript app to handle the async flow, mainly of fetching a ref key from firebase after the first data push.
let addIndexPromise = new Promise( (resolve, reject) => {
let newRef = ref.child('list-index').push(newItem);
resolve( newRef.key()); // ignore reject() for brevity
});
addIndexPromise.then( key => {
ref.child('list').child(key).set(newItem);
});
How do I make sure the data stays in sync in all places, knowing my app runs only on the client?
For sanity check, I set a setTimeout in my promise and shut my browser before it resolved, and indeed my database was no longer consistent, with an extra index saved without a corresponding list.
Any advice?

Great question. I know of three approaches to this, which I'll list below.
I'll take a slightly different example for this, mostly because it allows me to use more concrete terms in the explanation.
Say we have a chat application, where we store two entities: messages and users. In the screen where we show the messages, we also show the name of the user. So to minimize the number of reads, we store the name of the user with each chat message too.
users
so:209103
name: "Frank van Puffelen"
location: "San Francisco, CA"
questionCount: 12
so:3648524
name: "legolandbridge"
location: "London, Prague, Barcelona"
questionCount: 4
messages
-Jabhsay3487
message: "How to write denormalized data in Firebase"
user: so:3648524
username: "legolandbridge"
-Jabhsay3591
message: "Great question."
user: so:209103
username: "Frank van Puffelen"
-Jabhsay3595
message: "I know of three approaches, which I'll list below."
user: so:209103
username: "Frank van Puffelen"
So we store the primary copy of the user's profile in the users node. In the message we store the uid (so:209103 and so:3648524) so that we can look up the user. But we also store the user's name in the messages, so that we don't have to look this up for each user when we want to display a list of messages.
So now what happens when I go to the Profile page on the chat service and change my name from "Frank van Puffelen" to just "puf".
Transactional update
Performing a transactional update is the one that probably pops to mind of most developers initially. We always want the username in messages to match the name in the corresponding profile.
Using multipath writes (added on 20150925)
Since Firebase 2.3 (for JavaScript) and 2.4 (for Android and iOS), you can achieve atomic updates quite easily by using a single multi-path update:
function renameUser(ref, uid, name) {
var updates = {}; // all paths to be updated and their new values
updates['users/'+uid+'/name'] = name;
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.once('value', function(snapshot) {
snapshot.forEach(function(messageSnapshot) {
updates['messages/'+messageSnapshot.key()+'/username'] = name;
})
ref.update(updates);
});
}
This will send a single update command to Firebase that updates the user's name in their profile and in each message.
Previous atomic approach
So when the user change's the name in their profile:
var ref = new Firebase('https://mychat.firebaseio.com/');
var uid = "so:209103";
var nameInProfileRef = ref.child('users').child(uid).child('name');
nameInProfileRef.transaction(function(currentName) {
return "puf";
}, function(error, committed, snapshot) {
if (error) {
console.log('Transaction failed abnormally!', error);
} else if (!committed) {
console.log('Transaction aborted by our code.');
} else {
console.log('Name updated in profile, now update it in the messages');
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.on('child_added', function(messageSnapshot) {
messageSnapshot.ref().update({ username: "puf" });
});
}
console.log("Wilma's data: ", snapshot.val());
}, false /* don't apply the change locally */);
Pretty involved and the astute reader will notice that I cheat in the handling of the messages. First cheat is that I never call off for the listener, but I also don't use a transaction.
If we want to securely do this type of operation from the client, we'd need:
security rules that ensure the names in both places match. But the rules need to allow enough flexibility for them to temporarily be different while we're changing the name. So this turns into a pretty painful two-phase commit scheme.
change all username fields for messages by so:209103 to null (some magic value)
change the name of user so:209103 to 'puf'
change the username in every message by so:209103 that is null to puf.
that query requires an and of two conditions, which Firebase queries don't support. So we'll end up with an extra property uid_plus_name (with value so:209103_puf) that we can query on.
client-side code that handles all these transitions transactionally.
This type of approach makes my head hurt. And usually that means that I'm doing something wrong. But even if it's the right approach, with a head that hurts I'm way more likely to make coding mistakes. So I prefer to look for a simpler solution.
Eventual consistency
Update (20150925): Firebase released a feature to allow atomic writes to multiple paths. This works similar to approach below, but with a single command. See the updated section above to read how this works.
The second approach depends on splitting the user action ("I want to change my name to 'puf'") from the implications of that action ("We need to update the name in profile so:209103 and in every message that has user = so:209103).
I'd handle the rename in a script that we run on a server. The main method would be something like this:
function renameUser(ref, uid, name) {
ref.child('users').child(uid).update({ name: name });
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.once('value', function(snapshot) {
snapshot.forEach(function(messageSnapshot) {
messageSnapshot.update({ username: name });
})
});
}
Once again I take a few shortcuts here, such as using once('value' (which is in general a bad idea for optimal performance with Firebase). But overall the approach is simpler, at the cost of not having all data completely updated at the same time. But eventually the messages will all be updated to match the new value.
Not caring
The third approach is the simplest of all: in many cases you don't really have to update the duplicated data at all. In the example we've used here, you could say that each message recorded the name as I used it at that time. I didn't change my name until just now, so it makes sense that older messages show the name I used at that time. This applies in many cases where the secondary data is transactional in nature. It doesn't apply everywhere of course, but where it applies "not caring" is the simplest approach of all.
Summary
While the above are just broad descriptions of how you could solve this problem and they are definitely not complete, I find that each time I need to fan out duplicate data it comes back to one of these basic approaches.

To add to Franks great reply, I implemented the eventual consistency approach with a set of Firebase Cloud Functions. The functions get triggered whenever a primary value (eg. users name) gets changed, and then propagate the changes to the denormalized fields.
It is not as fast as a transaction, but for many cases it does not need to be.

Meteor: trying to add another field to the user profile Accounts.onCreateUser();

Basically trying to modify the user that was just created by giving it an extra field called sid in it's profile object. I'm running this on server.js (the server code)
Accounts.onCreateUser(function (options, user) {
Meteor.users.update({_id: user._id}, {$set: {"user.profile.sid": [post.content]}});
});
console.log(JSON.stringify(user));
However, the user object does not show the sid field in it's output. Am I doing this in the wrong location or is my code wrong?

From the docs
The function you pass will be called with two arguments: options and user. The options argument comes from Accounts.createUser for password-based users or from an external service login flow. options may come from an untrusted client so make sure to validate any values you read from it. The user argument is created on the server and contains a proposed user object with all the automatically generated fields required for the user to log in, including the _id.
The function should return the user document (either the one passed in or a newly-created object) with whatever modifications are desired. The returned document is inserted directly into the Meteor.users collection.
So your code should be:
Accounts.onCreateUser(function (options, user) {
user.profile.sid = [post.content];
return user;
});
However be aware that anything in the user.profile object can be changed by your users.
profile: an Object which the user can create and update with any data. Do not store anything on profile that you wouldn't want the user to edit unless you have a deny rule on the Meteor.users collection.

Try this instead
Accounts.onCreateUser(function (options, user) {
user.profile.sid = [post.content];
return user;
});
From the documentation it reads (http://docs.meteor.com/#/full/accounts_oncreateuser):
The user argument is created on the server and contains a proposed user object...
So at this point it looks like the user does not actually exist in the database yet.

Does modelObject.save() only update an existing database document when the modelObject was obtained from the database itself?

To use an example that demonstrates the question, assume I have a User model defined by the following schema:
var UserSchema = new Schema({
username: String,
email: String
}
mongoose.model('User', UserSchema);
I know that to update a user using the save method, I could query the user and then save changes like so:
User.findOne({username: usernameTofind}, function(err, user) {
//ignore errors for brevity
user.email = newEmail;
user.save(function(err) { console.log('User email updated') });
});
But if I try to create a new User object with the exact same field values (including the _id) is there any possibility of overwriting the database document? I would assume not, because in theory this would mean that a malicious user could exploit an insecure api and overwrite existing documents (for instance using a 'Create a New Account' request, which wouldn't/couldn't rely on the user already being authenticated) , but more importantly, when I try to do this using a request tool (I'm using Postman, but I'm sure a similar curl command would suffice), I get a duplicate _id error
MongoError: insertDocument :: caused by :: 11000 E11000 duplicate key error index
So I just want to clarify that the only way to update an existing document is to query for the document, modify the returned instance, then call the save method on that instance, OR use the static update(). Both of these could be secured by requiring authentication.
If it helps, my motivation for this question is mentioned above, in that I want to make sure a user is not able to overwrite an existing document if a method such as the following is exposed publicly:
userCtrl.create = function(req, res, next) {
var user = new User(req.body);
user.save(function(err) {
if (err) {
return next(err);
} else {
res.json(user);
}
});
};
Quick Edit: I just realized, if this is the case, then how does the database know the difference between the queried instance and a new User object with the exact same keys and properties?

Does modelObject.save() only update an existing database document when
the modelObject was obtained from the database itself?
Yes, it does. There is a flag that indicates if the document is new or not. If it is new, Mongoose will insert the document. If not, then it will update the document. The flag is Document#isNew.
When you find a document:
User.findOne({username: usernameTofind}, function(err, user) {
//ignore errors for brevity
console.log(user.isNew) // => will return false
});
When you create a new instance:
var user = new User(req.body);
console.log(user.isNew) // => will return true
So I just want to clarify that the only way to update an existing
document is to query for the document, modify the returned instance,
then call the save method on that instance, OR use the static
update(). Both of these could be secured by requiring authentication.
There are other ways you can update documents, using Model#update, Model.findOneAndUpdate and others.
However, you can't update an _id field. MongoDB won't allow it even if Mongoose didn't already issue the proper database command. If you try it you will get something like this error:
The _id field cannot be changed from {_id: ObjectId('550d93cbaf1e9abd03bf0ad2')} to {_id: ObjectId('550d93cbaf1e9abd03bf0ad3')}.
But assuming you are using the last piece of code in your question to create new users, Mongoose will issue an insert command, so there is no way someone could overwrite an existing document. Even if it passes an _id field in the request body, MongoDB will throw a E11000 duplicate key error index error.
Also, you should filter the fields a user can pass as payload before you use them to create the user. For example, you could create a generic function that receives an object and an array of allowed parameters:
module.exports = function(object, allowedParams) {
return Object.keys(object).reduce(function(newObject, param) {
if (allowedParams.indexOf(param) !== -1)
newObject[param] = object[param];
return newObject;
}, {});
}
And then you only require and use the function to filter the request body:
var allow = require('./parameter-filter');
function filter(params) {
return allow(params, ["username", "email"]);
}
var user = new User(filter(req.body));

Meteor - Allow multiple users to edit a post

I'm not able to use the node server debugger so I'm posting here to see if I can get a nudge in the right direction.
I am trying to allow multiple users to edit documents created by any of the users within their specific company. My code is below. Any help would be appreciated.
(Server)
ComponentsCollection.allow({
// Passing in the user object (has profile object {company: "1234"}
// Passing in document (has companyId field that is equal to "1234"
update: function(userObject, components) {
return ownsDocument(userObject, components);
}
});
(Server)
// check to ensure user editing document created/owned by the company
ownsDocument = function(userObject, doc) {
return userObject.profile.company === doc.companyId;
}
The error I'm getting is: Exception while invoking method '/components/update' TypeError: Cannot read property 'company' of undefined
I'm trying to be as secure as possible, though am doing some checks before presenting any data to the user, so I'm not sure if this additional check is necessary. Any advice on security for allowing multiple users to edit documents created by the company would be awesome. Thanks in advance. -Chris
Update (solution):
// check that the userId specified owns the documents
ownsDocument = function(userId, doc) {
// Gets the user form the userId being passed in
var userObject = Meteor.users.findOne(userId);
// Checking if the user is associated with the company that created the document being modified
// Returns true/false respectively
return doc.companyId === userObject.profile.companyId;
}

Looking at the docs, it looks like the first argument to the allow/deny functions is a user ID, not a user document. So you'll have to do Meteor.users.findOne(userId) to get to the document first.
Do keep in mind that users can write to their own profile subdocument, so if you don't disable that, users will be able to change their own company, allowing them to edit any post. You should move company outside of profile.
(If you can't use a proper debugger, old-fashioned console.log still works. Adding console.log(userObject) to ownsDocument probably would have revealed the solution.)

Meteor: Adding Fields on createAccount

I'm trying to use the Meteor Roles package: https://github.com/alanning/meteor-roles
to obviously create a new field in user model.
The user is created no problem but the 'roles' field I'm trying to define isn't created. I can add things like 'Profile' and details within that too. But for some reason I can't make a roles field. Here's my form:
Template.signup.events({
'submit #signup-form' : function(e, t) {
e.preventDefault();
var roles = ['admin'],
email = t.find('#email').value,
password = t.find('#password').value;
Accounts.createUser({email: email, password : password, roles: roles}, function(err){
if (err) {
alert("User Not Added")
} else {
console.log("User Added.")
}
});
}
});
Eventually I'll need to publish this to the client but for right now I just want the field to show in MongoDb, which it's not.
3 things:
I feel like the code above should work but I'm clearly missing something
In the package docs it mentions this Roles.addUsersToRoles which I
tried but no luck
Or do I need to possibly update the record, after it's been created?
I did go into the DB and manually added the field and associated string to update it (with $set) and it worked. But from the form itself though, no luck.
Any pointers would be much appreciated. Thank you.

The Accounts.createUser function only lets you add arbitrary user properties via the profile option which is where they end up getting stored in mongo. That is why Meteor is ignoring the roles: roles part of your Accounts.createUser call.
It is true that the meteor-roles package stores the list of roles assigned to a user directly in the users collection, but that is almost just an implementation detail and you are probably best off sticking to the API that meteor-roles provides for adding users to a role:
Roles.addUsersToRoles(<userId>,[<list of roles>])
The userId passed to Roles.addUsersToRoles is the value returned by Accounts.createUser when its called on the server which is probably where you want to be doing this as that feels way more secure.

The Accounts.createUser function only takes username, email, password and profile as params for the user object. See the documentation here. So, to add another field to a new user object, you need to add it in a second step:
var uid = Accounts.createUser({email: email, password: password});
Meteor.users.update(uid, {$set: {roles: roles}});