Develop Reference

Efficient way making "Like" function without duplication?

I am making a "Like" function (facebook or instagram kind of), but not sure what is the right way to do it.
I could think of 2 ways.... (User cannot like same article twice)
A. "User" data has an array of "Article" IDs...
// simplified user schema MongoDB
const UserSchema ={
id:ObjectID,
username:String,
likes:[{type:ObjectID,ref:"Article"}]
}
// simplified article schema
const AriticleSchema = {
id:ObjectID,
title:String,
content:String,
likes:Number,
}
B. "Article" data has an array of "User" IDs...
// simplified user schema MongoDB
const UserSchema ={
id:ObjectID,
username:String,
}
// simplified article schema
const AriticleSchema = {
id:ObjectID,
title:String,
content:String,
likes:[{type:ObjectID,ref:"User"}],
}
I tried both ways and they all worked fine when I only have few users and few articles.
but What if I have thousands of "User"s and thousands of "Article"s? I am worrying that everytime I request "User" data or "Article" data(let's say several at a time), I also have to bring arrays of thousands? I think there must be better way to do this...
Do you know how people or companies do this? I want to know the concept of how "Like" function works.
Thank you.
** Adding some details **
I want "User" can login for reading articles, and press "like button" to like it. Article "like" will go up by 1 every time unique user likes it (no duplicate). Somebody who already liked the " article" they can "unlike" it or "user" will see they already liked the "article", which means we gotta know that "user" like this "article" or not. Other people dosen't need to know.

I'll extrapolate on a solution by going through the requirements you elaborated step by step.
User can login
This will require some system for authorization. You could perhaps use another Mongo table dedicated to this sort of thing - at the minimum it should link some authorization token to a user id.
const AuthSchema = {
user_id: ObjectID,
auth_token: string,
}
The way you get this auth token is through various means - really depends on how you auth your users, e.g. phone auth or username/password. All that's a bit beyond the scope of this answer.
Article "like" will go up by 1 every time unique user likes it (no duplicate). Somebody who already liked the " article" can "unlike" it or "user" will see they already liked the "article"
A Redis Set would model this quite well. Every time a user likes an article, add their user_id into a Redis Set for that article. The key for such set could look like this:
article:${article_id}:likes
Get the number of people who liked the article by taking the size of its set (full of unique user_ids). Whenever someone unlikes an article, remove their user_id from the set.
Further reading:
Redis Sets and other Redis datatypes
Redis sadd command.

How and when to write entity relationships with Firebase Database [duplicate]

I've read the Firebase docs on Stucturing Data. Data storage is cheap, but the user's time is not. We should optimize for get operations, and write in multiple places.
So then I might store a list node and a list-index node, with some duplicated data between the two, at very least the list name.
I'm using ES6 and promises in my javascript app to handle the async flow, mainly of fetching a ref key from firebase after the first data push.
let addIndexPromise = new Promise( (resolve, reject) => {
let newRef = ref.child('list-index').push(newItem);
resolve( newRef.key()); // ignore reject() for brevity
});
addIndexPromise.then( key => {
ref.child('list').child(key).set(newItem);
});
How do I make sure the data stays in sync in all places, knowing my app runs only on the client?
For sanity check, I set a setTimeout in my promise and shut my browser before it resolved, and indeed my database was no longer consistent, with an extra index saved without a corresponding list.
Any advice?

Great question. I know of three approaches to this, which I'll list below.
I'll take a slightly different example for this, mostly because it allows me to use more concrete terms in the explanation.
Say we have a chat application, where we store two entities: messages and users. In the screen where we show the messages, we also show the name of the user. So to minimize the number of reads, we store the name of the user with each chat message too.
users
so:209103
name: "Frank van Puffelen"
location: "San Francisco, CA"
questionCount: 12
so:3648524
name: "legolandbridge"
location: "London, Prague, Barcelona"
questionCount: 4
messages
-Jabhsay3487
message: "How to write denormalized data in Firebase"
user: so:3648524
username: "legolandbridge"
-Jabhsay3591
message: "Great question."
user: so:209103
username: "Frank van Puffelen"
-Jabhsay3595
message: "I know of three approaches, which I'll list below."
user: so:209103
username: "Frank van Puffelen"
So we store the primary copy of the user's profile in the users node. In the message we store the uid (so:209103 and so:3648524) so that we can look up the user. But we also store the user's name in the messages, so that we don't have to look this up for each user when we want to display a list of messages.
So now what happens when I go to the Profile page on the chat service and change my name from "Frank van Puffelen" to just "puf".
Transactional update
Performing a transactional update is the one that probably pops to mind of most developers initially. We always want the username in messages to match the name in the corresponding profile.
Using multipath writes (added on 20150925)
Since Firebase 2.3 (for JavaScript) and 2.4 (for Android and iOS), you can achieve atomic updates quite easily by using a single multi-path update:
function renameUser(ref, uid, name) {
var updates = {}; // all paths to be updated and their new values
updates['users/'+uid+'/name'] = name;
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.once('value', function(snapshot) {
snapshot.forEach(function(messageSnapshot) {
updates['messages/'+messageSnapshot.key()+'/username'] = name;
})
ref.update(updates);
});
}
This will send a single update command to Firebase that updates the user's name in their profile and in each message.
Previous atomic approach
So when the user change's the name in their profile:
var ref = new Firebase('https://mychat.firebaseio.com/');
var uid = "so:209103";
var nameInProfileRef = ref.child('users').child(uid).child('name');
nameInProfileRef.transaction(function(currentName) {
return "puf";
}, function(error, committed, snapshot) {
if (error) {
console.log('Transaction failed abnormally!', error);
} else if (!committed) {
console.log('Transaction aborted by our code.');
} else {
console.log('Name updated in profile, now update it in the messages');
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.on('child_added', function(messageSnapshot) {
messageSnapshot.ref().update({ username: "puf" });
});
}
console.log("Wilma's data: ", snapshot.val());
}, false /* don't apply the change locally */);
Pretty involved and the astute reader will notice that I cheat in the handling of the messages. First cheat is that I never call off for the listener, but I also don't use a transaction.
If we want to securely do this type of operation from the client, we'd need:
security rules that ensure the names in both places match. But the rules need to allow enough flexibility for them to temporarily be different while we're changing the name. So this turns into a pretty painful two-phase commit scheme.
change all username fields for messages by so:209103 to null (some magic value)
change the name of user so:209103 to 'puf'
change the username in every message by so:209103 that is null to puf.
that query requires an and of two conditions, which Firebase queries don't support. So we'll end up with an extra property uid_plus_name (with value so:209103_puf) that we can query on.
client-side code that handles all these transitions transactionally.
This type of approach makes my head hurt. And usually that means that I'm doing something wrong. But even if it's the right approach, with a head that hurts I'm way more likely to make coding mistakes. So I prefer to look for a simpler solution.
Eventual consistency
Update (20150925): Firebase released a feature to allow atomic writes to multiple paths. This works similar to approach below, but with a single command. See the updated section above to read how this works.
The second approach depends on splitting the user action ("I want to change my name to 'puf'") from the implications of that action ("We need to update the name in profile so:209103 and in every message that has user = so:209103).
I'd handle the rename in a script that we run on a server. The main method would be something like this:
function renameUser(ref, uid, name) {
ref.child('users').child(uid).update({ name: name });
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.once('value', function(snapshot) {
snapshot.forEach(function(messageSnapshot) {
messageSnapshot.update({ username: name });
})
});
}
Once again I take a few shortcuts here, such as using once('value' (which is in general a bad idea for optimal performance with Firebase). But overall the approach is simpler, at the cost of not having all data completely updated at the same time. But eventually the messages will all be updated to match the new value.
Not caring
The third approach is the simplest of all: in many cases you don't really have to update the duplicated data at all. In the example we've used here, you could say that each message recorded the name as I used it at that time. I didn't change my name until just now, so it makes sense that older messages show the name I used at that time. This applies in many cases where the secondary data is transactional in nature. It doesn't apply everywhere of course, but where it applies "not caring" is the simplest approach of all.
Summary
While the above are just broad descriptions of how you could solve this problem and they are definitely not complete, I find that each time I need to fan out duplicate data it comes back to one of these basic approaches.

To add to Franks great reply, I implemented the eventual consistency approach with a set of Firebase Cloud Functions. The functions get triggered whenever a primary value (eg. users name) gets changed, and then propagate the changes to the denormalized fields.
It is not as fast as a transaction, but for many cases it does not need to be.

MongoDB + Node JS + Role Based Access Control (RBAC)

Am currently learning MEAN stack, developing a simple TODO's app and want to implement Role Based Access Control (RBAC) for that. How do i set up roles & permission on MongoDB.
I want 3 roles (roles may look funny but this is purely to learn) :
GOD
SUPER HERO
MAN
GOD - similar to super admin, can do anything in the application. C,R,U,D permissions for TODO's and for other users too. Can Create a TODO & assign it to any SUPER HERO or MAN directly. Update or Delete either a TODO or a User at any point in time.
SUPER HERO - similar to admin, has super power to do anything on his personal Data - C,R,U,D for TODO's. Can't create any users. Can only Read & add comments for TODO's created by GOD & assigned to him/her.
MAN - Can only Read and add comments to TODO's assigned to him/her.
To sum it up :
GOD - C,R,U,D [Global Level]
SUPER HERO - C,R,U,D [Private] + R,U [Assigned to him]
MAN - R,U [Assigned to him]
I understand that i need to have USERS & ROLES collections. Where ROLES inturn should have PERMISSIONS etc. How do i wire them all ?

I like names given to roles - GOD, SUPER HERO & MAN, easy to understand.
As you are using MEAN stack and much of routes validation happens on node, i would prefer keeping roles table simple.
Roles :
{
_id : 1,
name : GOD,
golbalPerms : true
},
{
_id : 2,
name : SUPER HERO,
privatePerms : true
},
{
_id : 3,
name : MAN
}
Users :
{
_id : 111,
name : Jesus,
roleId : 1
},
{
_id : 222,
name : BatMan,
roleId : 2
},
{
_id : 333,
name : Jack,
roleId : 3
}
When user logs in and sending user object back to client, make sure to replace roleId with corresponding role object from DB.
Coming to code on Node JS :
By completely understanding your usecase we can divide them into following methods -
CreateUser
CreateTodo
DeleteTodo
ReadTodo
UpdateTodo
CommentTodo
AssignTodo
Lets go step by step, CreateUser.
Routes code snippet :
app.all('/users', users.requiresLogin);
// Users Routes
app.route('/users')
.post(users.hasPerms('globalPerms'), users.create);
In your Controller you can validate based on the input globalPerms, if validated allow to create user by calling next() else return with corresponding error message.
Now CreateTodo && DeleteTodo :
Both of them pretty much work on same logic with a small trick.
Routes code snippet :
app.all('/todos', users.requiresLogin);
// Users Routes
app.route('/todos')
.post(users.hasPerms('globalPerms','privatePerms'), todos.create);
.delete(users.hasPerms('globalPerms','privatePerms'), todos.delete);
For creating a Todo, globalPerms are with GOD & privatePerms are with SUPER HERO, both of them can be allowed.
Trick here will be in todos.delete method, just ensure user.id === todos.createById else SUPER HERO may go on to delete Todos created by GOD.
ReadTodo :
When a TODO is created it should have a createById stored likewise when a TODO is assigned to someone then assignedTo and assignedBy should be recorded too.
This makes lot of other operations easy to handle.
user.role.globalPerms - give GOD all TODO's data.
user.role.privatePerms - give TODO's either createdBy him/her or assigned to him/her.
user.role.globalPerms === undefined && user.role.privatePerms === undefined - its MAN and give TODO's which are only assignedTo him.
UpdateTodo & CommentTodo :
This is exact replica of what ReadTODO does so DIY
Last one, AssignTodo :
Simple one, loggedInUser.id === todos.createdById then he can assign it to anyone.
Two things to keep in mind here :
As assigning part mostly happens on your UI (Angular) front, i have given that approach of checking loggedInUser.id === todos.createdById. Logged in user any ways will see all TODO's by read operation and can assign it to anyone he/she likes.
Make sure a SUPER HERO can only assign a TODO to himself or other SUPER HERO or to a MAN but not to GOD. How you show Assign to options on UI front is out of scope of this question. This is just a heads up.
Hope this was clear.
NOTE : There was no necessity to give permissions to MAN in Roles collection & we managed all possible operations with out that.

This is a very broad question which can be solved in many ways.
You have added that you are using MEAN stack therefore I'll restrict my question to that.
One thing that you haven't included in the whole question is what kind of authentication architecture are you using. Let's say you are using token based authentication, generally people these days use it.
We have 3 types of users.
You have different options available to differentiate between type of tokens as well.
Different Collection (mongoDB) or Redis sets where they will be stored
The encrypted token will have type of the user as well etc.. (This will come in handy if you don't need to store tokens on the backend, you can just decrypt and check)
It will completely depend on use case.
Now, before allowing any user's entry to user specific route make sure that you are checking the token first.
Example
app.post('/godlevelroute', godtokencheck, callrouteandfunction);
app.post('/superherolevelroute', superheroroute, callrouteandfunction);
You must send token in header from angular and then you can take the data out from the header and then you can check if that specific user has permission to go through that route or not.
Let's say a god level user is logged in then he'll have the godleveltoken with him and we'll check that first before allowing him to access that route or else you can just show error message.
This can be your sample token checking function on server end
function checkToken(req, res, next) {
var token = req.headers['accesstoken']; //access token from header
//now depending upon which system you are following you can run a check
};
Node Module Suggestion : https://www.npmjs.com/package/jsonwebtoken
Now coming to frontend part. You are using angular based on what you have written, you can intercept the token before showing any page.
You can go through this blog to get a pictorial representation of what I have tried to explain. Click Here

Possible approach-> have role embedded in user collection/schema:
users document shall have the following:
{
_id : "email#mail.com",
name: "lorem ipsum",
role: "MAN"
}
As far as your post describes, only god can make and assign TODOs.
Roles Collection may hold the following:
{
_id : "MAN",
globalPerm: [],
privatePerm: [],
assignedPerm: ["r","u"],
},
{
_id : "SUPER_HERO",
globalPerm: [],
privatePerm: ["c","r","u","d"],
assignedPerm: ["c","r","u","d"],
},
{
_id : "GOD",
globalPerm: ["c","r","u","d"],
privatePerm: ["c","r","u","d"],
assignedPerm: ["c","r","u","d"],
}
Node JS Middlewares
After getting correct permission values for a user, you might want to use middlewares.
Sample express HTTP request route:
app.post('/updateTodo', permissions.check('privatePerm', 'c'), function (req, res) {
// do stuff
};
permissions.check is called before actually executing function body to update TODO.
Hence if a user tries to update a todo, it will first verify the corresponding permissions.

Best way to manage Chat channels in Firebase

In my main page I have a list of users and i'd like to choose and open a channel to chat with one of them.
I am thinking if use the id is the best way and control an access of a channel like USERID1-USERID2.
But of course, user 2 can open the same channel too, so I'd like to find something more easy to control.
Please, if you want to help me, give me an example in javascript using a firebase url/array.
Thank you!

A common way to handle such 1:1 chat rooms is to generate the room URL based on the user ids. As you already mention, a problem with this is that either user can initiate the chat and in both cases they should end up in the same room.
You can solve this by ordering the user ids lexicographically in the compound key. For example with user names, instead of ids:
var user1 = "Frank"; // UID of user 1
var user2 = "Eusthace"; // UID of user 2
var roomName = 'chat_'+(user1<user2 ? user1+'_'+user2 : user2+'_'+user1);
console.log(user1+', '+user2+' => '+ roomName);
user1 = "Eusthace";
user2 = "Frank";
var roomName = 'chat_'+(user1<user2 ? user1+'_'+user2 : user2+'_'+user1);
console.log(user1+', '+user2+' => '+ roomName);
<script src="https://getfirebug.com/firebug-lite-debug.js"></script>
A common follow-up questions seems to be how to show a list of chat rooms for the current user. The above code does not address that. As is common in NoSQL databases, you need to augment your data model to allow this use-case. If you want to show a list of chat rooms for the current user, you should model your data to allow that. The easiest way to do this is to add a list of chat rooms for each user to the data model:
"userChatrooms" : {
"Frank" : {
"Eusthace_Frank": true
},
"Eusthace" : {
"Eusthace_Frank": true
}
}
If you're worried about the length of the keys, you can consider using a hash codes of the combined UIDs instead of the full UIDs.
This last JSON structure above then also helps to secure access to the room, as you can write your security rules to only allow users access for whom the room is listed under their userChatrooms node:
{
"rules": {
"chatrooms": {
"$chatroomid": {
".read": "
root.child('userChatrooms').child(auth.uid).child(chatroomid).exists()
"
}
}
}
}

In a typical database schema each Channel / ChatGroup has its own node with unique $key (created by Firebase). It shouldn't matter which user opened the channel first but once the node (& corresponding $key) is created, you can just use that as channel id.
Hashing / MD5 strategy of course is other way to do it but then you also have to store that "route" info as well as $key on the same node - which is duplication IMO (unless Im missing something).

We decided on hashing users uid's, which means you can look up any existing conversation,if you know the other persons uid.
Each conversation also stores a list of the uids for their security rules, so even if you can guess the hash, you are protected.

Hashing with js-sha256 module worked for me with directions of Frank van Puffelen and Eduard.
import SHA256 from 'crypto-js/sha256'
let agentId = 312
let userId = 567
let chatHash = SHA256('agent:' + agentId + '_user:' + userId)

How to write denormalized data in Firebase

I've read the Firebase docs on Stucturing Data. Data storage is cheap, but the user's time is not. We should optimize for get operations, and write in multiple places.
So then I might store a list node and a list-index node, with some duplicated data between the two, at very least the list name.
I'm using ES6 and promises in my javascript app to handle the async flow, mainly of fetching a ref key from firebase after the first data push.
let addIndexPromise = new Promise( (resolve, reject) => {
let newRef = ref.child('list-index').push(newItem);
resolve( newRef.key()); // ignore reject() for brevity
});
addIndexPromise.then( key => {
ref.child('list').child(key).set(newItem);
});
How do I make sure the data stays in sync in all places, knowing my app runs only on the client?
For sanity check, I set a setTimeout in my promise and shut my browser before it resolved, and indeed my database was no longer consistent, with an extra index saved without a corresponding list.
Any advice?

Great question. I know of three approaches to this, which I'll list below.
I'll take a slightly different example for this, mostly because it allows me to use more concrete terms in the explanation.
Say we have a chat application, where we store two entities: messages and users. In the screen where we show the messages, we also show the name of the user. So to minimize the number of reads, we store the name of the user with each chat message too.
users
so:209103
name: "Frank van Puffelen"
location: "San Francisco, CA"
questionCount: 12
so:3648524
name: "legolandbridge"
location: "London, Prague, Barcelona"
questionCount: 4
messages
-Jabhsay3487
message: "How to write denormalized data in Firebase"
user: so:3648524
username: "legolandbridge"
-Jabhsay3591
message: "Great question."
user: so:209103
username: "Frank van Puffelen"
-Jabhsay3595
message: "I know of three approaches, which I'll list below."
user: so:209103
username: "Frank van Puffelen"
So we store the primary copy of the user's profile in the users node. In the message we store the uid (so:209103 and so:3648524) so that we can look up the user. But we also store the user's name in the messages, so that we don't have to look this up for each user when we want to display a list of messages.
So now what happens when I go to the Profile page on the chat service and change my name from "Frank van Puffelen" to just "puf".
Transactional update
Performing a transactional update is the one that probably pops to mind of most developers initially. We always want the username in messages to match the name in the corresponding profile.
Using multipath writes (added on 20150925)
Since Firebase 2.3 (for JavaScript) and 2.4 (for Android and iOS), you can achieve atomic updates quite easily by using a single multi-path update:
function renameUser(ref, uid, name) {
var updates = {}; // all paths to be updated and their new values
updates['users/'+uid+'/name'] = name;
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.once('value', function(snapshot) {
snapshot.forEach(function(messageSnapshot) {
updates['messages/'+messageSnapshot.key()+'/username'] = name;
})
ref.update(updates);
});
}
This will send a single update command to Firebase that updates the user's name in their profile and in each message.
Previous atomic approach
So when the user change's the name in their profile:
var ref = new Firebase('https://mychat.firebaseio.com/');
var uid = "so:209103";
var nameInProfileRef = ref.child('users').child(uid).child('name');
nameInProfileRef.transaction(function(currentName) {
return "puf";
}, function(error, committed, snapshot) {
if (error) {
console.log('Transaction failed abnormally!', error);
} else if (!committed) {
console.log('Transaction aborted by our code.');
} else {
console.log('Name updated in profile, now update it in the messages');
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.on('child_added', function(messageSnapshot) {
messageSnapshot.ref().update({ username: "puf" });
});
}
console.log("Wilma's data: ", snapshot.val());
}, false /* don't apply the change locally */);
Pretty involved and the astute reader will notice that I cheat in the handling of the messages. First cheat is that I never call off for the listener, but I also don't use a transaction.
If we want to securely do this type of operation from the client, we'd need:
security rules that ensure the names in both places match. But the rules need to allow enough flexibility for them to temporarily be different while we're changing the name. So this turns into a pretty painful two-phase commit scheme.
change all username fields for messages by so:209103 to null (some magic value)
change the name of user so:209103 to 'puf'
change the username in every message by so:209103 that is null to puf.
that query requires an and of two conditions, which Firebase queries don't support. So we'll end up with an extra property uid_plus_name (with value so:209103_puf) that we can query on.
client-side code that handles all these transitions transactionally.
This type of approach makes my head hurt. And usually that means that I'm doing something wrong. But even if it's the right approach, with a head that hurts I'm way more likely to make coding mistakes. So I prefer to look for a simpler solution.
Eventual consistency
Update (20150925): Firebase released a feature to allow atomic writes to multiple paths. This works similar to approach below, but with a single command. See the updated section above to read how this works.
The second approach depends on splitting the user action ("I want to change my name to 'puf'") from the implications of that action ("We need to update the name in profile so:209103 and in every message that has user = so:209103).
I'd handle the rename in a script that we run on a server. The main method would be something like this:
function renameUser(ref, uid, name) {
ref.child('users').child(uid).update({ name: name });
var query = ref.child('messages').orderByChild('user').equalTo(uid);
query.once('value', function(snapshot) {
snapshot.forEach(function(messageSnapshot) {
messageSnapshot.update({ username: name });
})
});
}
Once again I take a few shortcuts here, such as using once('value' (which is in general a bad idea for optimal performance with Firebase). But overall the approach is simpler, at the cost of not having all data completely updated at the same time. But eventually the messages will all be updated to match the new value.
Not caring
The third approach is the simplest of all: in many cases you don't really have to update the duplicated data at all. In the example we've used here, you could say that each message recorded the name as I used it at that time. I didn't change my name until just now, so it makes sense that older messages show the name I used at that time. This applies in many cases where the secondary data is transactional in nature. It doesn't apply everywhere of course, but where it applies "not caring" is the simplest approach of all.
Summary
While the above are just broad descriptions of how you could solve this problem and they are definitely not complete, I find that each time I need to fan out duplicate data it comes back to one of these basic approaches.

To add to Franks great reply, I implemented the eventual consistency approach with a set of Firebase Cloud Functions. The functions get triggered whenever a primary value (eg. users name) gets changed, and then propagate the changes to the denormalized fields.
It is not as fast as a transaction, but for many cases it does not need to be.