I have a problematic here:
I'm builing a mobile app with ionic frmaework who needs to be able to work offline.
I'd like to adopt the CouchDB / PouchDB solution. But to do that i need to know how to put my data in the noSQl datatbase (MySQL user before ...). So, nosql is new to me but it seems interesting.
So my app has a connection part so a user database. And each user has documents who are attached to him. But many users can have many documents (sharing documents). And I want to replicate the data of one user (so his information + his documents on the mobile app).
What I thought is this:
One database per. One database for all Document with a server filtering to send only the documents that belongs to the user.
And on the client side I'd juste have to call :
var localDB = new PouchDB("myuser");
var remoteDB = new PouchDB("http://128.199.48.178:5984/myuser");
localDB.sync(remoteDB, {
live: true
});
And like that on the client side I'd have something like that :
{
username: "myuser",
birthday : "Date",
documents : [{
"_id": "2",
"szObject": "My Document",
},
{
"_id": "85",
"szObject": "My Document",
}]
}
Do you think something like that is possible using Couchdb and pouchdb, and if yes, am I thinking about it the right way?
I read it's not a problem to have one database per document, but I don't know if the replication will work like I imagine it
Plain CouchDB doesn't have any per-document access options, but these could be your solutions:
A. Create a View, then sync Pouch-To-Couch with a filter. But although this will only sync the documents that the user is supposed to see, anyone with enough knowledge could alter the code and view someone else's documents or just do anything with the database actually (probably not what you're looking for).
B. Create a master DB with all documents, then a database for each user, and a filtered replication between the master & per-user-dbs. Probably the simplest and most proper way to handle this.
C. Unfortunately there isn't a validate_doc_read (as there is a validate_doc_update) but perhaps you could make a simple HTTP proxy, which would parse out incoming JSON, check if a particular user can view it and if not, throw a 403 Forbidden. Well you'd also have to catch any views that query with include_docs=true.
(late reply, I hope it's still useful - or if not, that you found a good solution for your problem)
Related
I have a subscriber which pushes data into queues. Now the messages looks this
{
"Content": {
"_id" ""5ceya67bbsbag3",
"dataset": {
"upper": {},
"lower": {}
}
}
Now a new message can be pushed with same content id but data will be different. So in that i want to delete the old message with same id or replaece the message those id is same & retain only latest message.
I have not found a direct solution for this in rabbitmq. Please guide me how we can do this ?
I have already gone through some posts.
Post 1
Post 2
What you are trying to achieve cannot be trivially solved with RabbitMQ (or rather the AMQP protocol).
RabbitMQ queues are simple FIFO stacks and don't offer any mean of access to the elements beyond publishing at their top and consuming from their bottom.
Therefore, the only way to "update" an already existing message without relying on an another service would be to fetch all the messages until you find the one you are interested in, discard it, and publish the new one with the other messages you fetched together with it.
Overall, the recommendation when using RabbitMQ in regards of message duplication is to make their consumption idempotent. In other words, the consumption of 2 messages deemed to be the same should lead to the same outcome.
One way to achieve idempotency is to rely on a secondary cache where you store the message identifiers and their validity. Once a consumer fetches a new message from RabbitMQ, it would check the cache to see if it's a valid message or not and act accordingly.
I think this is a slightly wrong way to use rabbitMQ.
only immutable (not intended to change) tasks should be put into queues which a worker shall consume.
An alternative way to implement your particular task is
just push immutable data into queue { "content" : { "_id" : "5ceya67bbsbag3"} .. }
store mutable data in db (mongo) or in-mem db (something like redis is suggested here).
whenever update needed, update in db
let your worker fetch required data using your "_id" ref from the db
I am not sure if removing a message is a good idea. If your requirement is to update the data as it comes so that always latest data is maintained for same Id.
Other thing is as messages are getting consumed always the last message data will get updated. So I don't see a issue here in Rabbit MQ.
Background: Using MEAN stack to build a web app, I am still learning.
The Issue: I find the following confusing. Say I have a user logged in (I am using Passport.js). From Angular I can retrieve it querying my Node.js server.
What I am doing now is something similar to:
app.get('/userLogged',function(req,res){
res.json({req.user});
});
This does not sound safe to me. I might be a novice, but I have seen this in many tutorials. With a console.log in the browser I can print all the info about the user, including the hashed password. My guess is that I should send a minimal set of information to the browser, filtering out the rest.
My Question: is this safe at all, or I am just leaving the door open to hackers?
Take a look at the concept of ViewModel. It represents the data you want to share publicly with an external user of the system.
What can be achieved in your case, is implementing the right view model out of the data model you store internally. A simplistic example illustrating this concept would be to create a view model for your user object that will pick the data you would like to send back :
// This function will return a different version
// of the `user` object having only a `name`
// and an `email` attribute.
var makeViewModel = function (user) {
return _.pick(user, ['name', 'email']);
}
You will then be able to construct the right view model on demand :
app.get('/user',function (req,res){
res.json(makeViewModel(req.user));
});
I am using Parse.com with my iPhone app.
I ran into a problem earlier where I was trying to add the currently logged in user to another user's PFRelation key/column called "friendsRelation" which is basically the friends list.
The only problem, is that you are not allowed to save changes to any other users besides the one that is currently logged in.
I then learned, that there is a workaround you can use, using the "master key" with Parse Cloud Code.
I ended up adding the code here to my Parse Cloud Code: https://stackoverflow.com/a/18651564/3344977
This works great and I can successfully test this and add an NSString to a string column/key in the Parse database.
However, I do not know how to modify the Parse Cloud Code to let me add a user to another user's PFRelation column/key.
I have been trying everything for the past 2 hours with the above Parse Cloud Code I linked to and could not get anything to work, and then I realized that my problem is with the actual cloud code, not with how I'm trying to use it in xcode, because like I said I can get it to successfully add an NSString object for testing purposes.
My problem is that I do not know javascript and don't understand the syntax, so I don't know how to change the Cloud Code which is written in javascript.
I need to edit the Parse Cloud Code that I linked to above, which I will also paste below at the end of this question, so that I can add the currently logged in PFUser object to another user's PFRelation key/column.
The code that I would use to do this in objective-c would be:
[friendsRelation addObject:user];
So I am pretty sure it is the same as just adding an object to an array, but like I said I don't know how to modify the Parse Cloud Code because it's in javascript.
Here is the Parse Cloud Code:
Parse.Cloud.define('editUser', function(request, response) {
var userId = request.params.userId,
newColText = request.params.newColText;
var User = Parse.Object.extend('_User'),
user = new User({ objectId: userId });
user.set('new_col', newColText);
Parse.Cloud.useMasterKey();
user.save().then(function(user) {
response.success(user);
}, function(error) {
response.error(error)
});
});
And then here is how I would use it in xcode using objective-c:
[PFCloud callFunction:#"editUser" withParameters:#{
#"userId": #"someuseridhere",
#"newColText": #"new text!"
}];
Now it just needs to be modified for adding the current PFUser to another user's PFRelation column/key, which I am pretty sure is technically just adding an object to an array.
This should be fairly simple for someone familiar with javascript, so I really appreciate the help.
Thank you.
I would recommend that you rethink your data model, and extract the followings out of the user table. When you plan a data model, especially for a NoSQL database, you should think about your queries first and plan your structure around that. This is especially true for mobile applications, as server connections are costly and often introduces latency issues if your app performs lots of connections.
Storing followings in the user class makes it easy to find who a person is following. But how would you solve the task of finding all users who follow YOU? You would have to check all users if you are in their followings relation. That would not be an efficient query, and it does not scale well.
When planning a social application, you should build for scalabilty. I don't know what kind of social app you are building, but imagine if the app went ballistic and became a rapidly growing success. If you didn't build for scalability, it would quickly fall apart, and you stood the chance of losing everything because the app suddenly became sluggish and therefore unusable (people have almost zero tolerance for waiting on mobile apps).
Forget all previous prioities about consistency and normalization, and design for scalability.
For storing followings and followers, use a separate "table" (Parse class) for each of those two. For each user, store an array of all usernames (or their objectId) they follow. Do the same for followers. This means that when YOU choose to follow someone, TWO tables need to be updated: you add the other user's username to the array of who you follow (in the followings table), and you also add YOUR username to the array of the other user's followers table.
Using this method, getting a list of followers and followings is extremely fast.
Have a look at this example implementation of Twitter for the Cassandra NoSQL database:
https://github.com/twissandra/twissandra
Is there any way to drop a Mongo Database Collection from within the server side JavaScript code with Meteor? (really drop the whole thing, not just Meteor.Collection.remove({}); it's contents)
In addition, is there also a way to drop a Meteor.Collection from within the server side JavaScript code without dropping the corresponding database collection?
Why do that?
Searching in the subdocuments (subdocuments of the user-document, e.g. userdoc.mailbox[12345]) with underscore or similar turns out quiet slow (e.g. for large mailboxes).
On the other hand, putting all messages (in context of the mailbox-example) of all users in one big DB and then searching* all messages for one or more particular messages turns out to be very, very slow (for many users with large mailboxes), too.
There is also the size limit for Mongo documents, so if I store all messages of a user in his/her user-document, the mailbox's maximum size is < 16 MB together with all other user-data.
So I want to have a database for each of my user to use it as a mailbox, then the maximum size for one message is 16 MB (very acceptable) and I can search a mailbox using mongo queries.
Furthemore, since I'm using Meteor, it would be nice to then have this mongo db collection be loaded as Meteor.Collection whenever a user logs in. When a user deactivates his/her account, the db should of course be dropped, if the user just logs out, only the Meteor.Collection should be dropped (and restored when he/she logs in again).
To some extent, I got this working already, each user has a own db for the mailbox, but if anybody cancels his/her account, I have to delete this particular Mongo Collection manually. Also, I have do keep all mongo db collections alive as Meteor.Collections at all times because I cannot drop them.
This is a well working server-side code snippet for one-collection-per-user mailboxes:
var mailboxes = {};
Meteor.users.find({}, {fields: {_id: 1}}).forEach(function(user) {
mailboxes[user._id] = new Meteor.Collection("Mailbox_" + user._id);
});
Meteor.publish("myMailbox", function(_query,_options) {
if (this.userId) {
return mailboxes[this.userId].find(_query, _options);
};
});
while a client just subscribes with a certain query with this piece of client-code:
myMailbox = new Meteor.Collection("Mailbox_"+Meteor.userId());
Deps.autorun(function(){
var filter=Session.get("mailboxFilter");
if(_.isObject(filter) && filter.query && filter.options)
Meteor.subscribe("myMailbox",filter.query,filter.options);
});
So if a client manipulates the session variable "mailboxFilter", the subscription is updated and the user gets a new bunch of messages in the minimongo.
It works very nice, the only thing missing is db collection dropping.
Thanks for any hint already!
*I previeously wrote "dropping" here, which was a total mistake. I meant searching.
A solution that doesn't use a private method is:
myMailbox.rawCollection().drop();
This is better in my opinion because Meteor could randomly drop or rename the private method without any warning.
You can completely drop the collection myMailbox with myMailbox._dropCollection(), directly from meteor.
I know the question is old, but it was the first hit when I searched for how to do this
Searching in the subdocuments...
Why use subdocuments? A document per user I suppose?
each message must be it's own document
That's a better way, a collection of messages, each is id'ed to the user. That way, you can filter what a user sees when doing publish subscribe.
dropping all messages in one db turns out to be very slow for many users with large mailboxes
That's because most NoSQL DBs (if not all) are geared towards read-intensive operations and not much with write-intensive. So writing (updating, inserting, removing, wiping) will take more time.
Also, some online services (I think it was Twitter or Yahoo) will tell you when deactivating the account: "Your data will be deleted within the next N days." or something that resembles that. One reason is that your data takes time to delete.
The user is leaving anyway, so you can just tell the user that your account has been deactivated, and your data will be deleted from our databases in the following days. To add to that, so you can respond to the user immediately, do the remove operation asynchronously by sending it a blank callback.
I'm currently using couchdb hosted on cloudant (thanks cloudant guys! You're great!).
I've many users in my database (which is here), and now I manage them maually: I've a script on my linux box which generates users/pass json object, like that
{
"_id": "org.couchdb.user:MY_USER",
"_rev": "1-7aXXXXXXXXXXXXXXXXXXXXXXXXXX2783",
"name": "MY_USER",
"type": "user",
"roles": ["admin"],
"password_sha": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"salt": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
(removed real string and substituded with 'XXX...' of course).
I then paste this object in the _user database and everything works fine.
However, as it happens to frequently to remove, update user information, I was wandering if anybody created a couchdb GUI for user management.
Something really simple, like the phpmyadmin GUI:
I googled and stackoverflowed around to search for something like this, but found nothing.
Does something like that already exist?
(and if not, do you think there is a special reason? Couchdb is widespread and I don't think I'm the first one who had to face this issue...).
Currently the most easy and common way to edit and maintain user information is from the couchdb admin gui called futon. You can access it at
http://localhost:5984/_utils/database.html?_users
While it is not an ideal user management tool, it gets the job done.
I have not seen a nice gui for doing the user management. It is a good idea. We are currently in the process of working on the next admin gui for couchdb called 'fauxton' were we will try and make this more user friendly.