I'm using Firebase for an app and the built-in real-time capabilities seem well suited for instant messaging. I'm just having a hard time working out in my head how the database should be set up. Ideally, it's something like this:
messages: {
<messageId>: {
from: <userId>,
to: <userId>,
text: <String>,
dateSent: <Date>
dateRead: <Date>
}
}
And that's all fine for sending messages, but reading message threads becomes difficult. I need to query the (potentially huge) list of messages for messages that match the current thread's sender and receiver, and then order those by dateSent. If that is possible with Firebase's new querying API, then I have yet to figure out exactly how to do it.
Querying a huge list of messages is never a good idea. If you want a fast-performing Firebase/NoSQL application, you'll need to model the data to allow fast look up.
In a chat scenario that typically means that you'll model your chat rooms into the data structure. So instead of storing one long list of messages, store the messages for each chat "room" separately.
messages
<roomId>
<messageId1>: "..."
<messageId2>: "..."
<messageId3>: "..."
Now you can access the messages for the chat without a query, just ref.child(roomId).on(....
If you want a persistent mapping that ensures the same two users end up in the same room, have a look at Best way to manage Chat channels in Firebase
I have a problematic here:
I'm builing a mobile app with ionic frmaework who needs to be able to work offline.
I'd like to adopt the CouchDB / PouchDB solution. But to do that i need to know how to put my data in the noSQl datatbase (MySQL user before ...). So, nosql is new to me but it seems interesting.
So my app has a connection part so a user database. And each user has documents who are attached to him. But many users can have many documents (sharing documents). And I want to replicate the data of one user (so his information + his documents on the mobile app).
What I thought is this:
One database per. One database for all Document with a server filtering to send only the documents that belongs to the user.
And on the client side I'd juste have to call :
var localDB = new PouchDB("myuser");
var remoteDB = new PouchDB("http://128.199.48.178:5984/myuser");
localDB.sync(remoteDB, {
live: true
});
And like that on the client side I'd have something like that :
{
username: "myuser",
birthday : "Date",
documents : [{
"_id": "2",
"szObject": "My Document",
},
{
"_id": "85",
"szObject": "My Document",
}]
}
Do you think something like that is possible using Couchdb and pouchdb, and if yes, am I thinking about it the right way?
I read it's not a problem to have one database per document, but I don't know if the replication will work like I imagine it
Plain CouchDB doesn't have any per-document access options, but these could be your solutions:
A. Create a View, then sync Pouch-To-Couch with a filter. But although this will only sync the documents that the user is supposed to see, anyone with enough knowledge could alter the code and view someone else's documents or just do anything with the database actually (probably not what you're looking for).
B. Create a master DB with all documents, then a database for each user, and a filtered replication between the master & per-user-dbs. Probably the simplest and most proper way to handle this.
C. Unfortunately there isn't a validate_doc_read (as there is a validate_doc_update) but perhaps you could make a simple HTTP proxy, which would parse out incoming JSON, check if a particular user can view it and if not, throw a 403 Forbidden. Well you'd also have to catch any views that query with include_docs=true.
(late reply, I hope it's still useful - or if not, that you found a good solution for your problem)
I'm using the dead letter exchange feature on rabbitmq to perform scheduled rpc calls, but after the queue is dead lettered it dropped the replyto property that was in the original queue. Is there anyway to declare the replyto property in a way that it will be retained in the "dead queue"?
I'm doing this with amqplib in node.js by the way.
Unfortunately, RabbitMQ will only preserve the properties that are listed on the Dead Letter Exchange page:
queue - the name of the queue the message was in before it was dead-lettered,
reason - reason for DLX being used
time - the date and time the message was dead lettered as a 64-bit AMQP format timestamp,
exchange - the exchange the message was published to
routing-keys - the routing keys the message was published with,
count - how many times this message was dead-lettered in this queue for this reason, and
original-expiration - the original expiration property of the message.
There are 2 ways to solve the problem you're seeing, I think.
1) Put the reply-to in your own header or property field, and read it from there / replace it when it's not in the usual spot
2) Don't use the reply-to field. Instead, use a well-known exchange for the reply at a later point in time.
Using the reply-to field typically implies a request/response or RPC scenario. These scenarios usually need a response fairly quickly. If a response does not come quickly, the system can usually move forward without it - even if it's just a message to the user saying "X is not available right now".
You say you're using a DLX to do scheduled RPC calls... delayed messages is a common use case for a DLX - nothing wrong with that. But delaying the RPC response can run in to some significant challenges beyond what you're already seeing.
For example, what happens when your system has a hiccup and the code that made the original request is no longer there to listen for the response? The answer to this depends on whether or not you really need the response to be handled. If you do need it to be handled - the system will run in to serious trouble if it isn't - then RPC can be dangerous.
Instead of relying on RPC and implying a temporal need for a given response, it's often better to use two-way messaging via separate queues. I've written about this in both my managing long running processes post and in my RabbitMQ Patterns email course / ebook.
The gist of it is that you can avoid the need for a reply-to queue by having the original message publisher also be a subscriber with a queue for the responses.
An example from the long running process post:
var DrinkRequestSender = new Sender(/* ... details ... */);
var DrinkRequestReceiver = new Receiver(/* ... details ... */);
var DrinkStation = {
make: function(drink){
DrinkRequestReceiver.receive((response) => {
var drinkResponse = response.body;
this.trigger("drinkup", drinkResponse);
});
var drinkData = drink.toJSON();
DrinkRequestSender.send(drinkData);
}
};
In this example, the code is sending out a "request" and later receiving a "response" - but not using a standard RPC setup. It is using a dedicated queue for the response, with the code on the other end sending the reply back via an exchange that routes to that queue.
This allows you to better handle failure scenarios, very long running processes and more.
This style of 2-way messaging does add some additional challenges, though. For one, you'll have to build in the ability to reconstruct the object that made the original request.
You can find this detailed in the long running process post, and there's a bit more info in RMQ Patterns, as well (along with a lot of other patterns).
Hope that helps!
I am developing a webapp in which I'd need one client, associated with the admin, to trigger an event (e.g., a new value selected in a dropdown list) which in turns will tell all the other connected clients to change the subscription, possibly using a parameter, i.e., the new selected value.
Something along the lines of
Template.bid.events
"change .roles": (e, tpl) ->
e.preventDefault()
role = tpl.$("select[name='role']").val()
Meteor.subscribe role
Of course this works for the current client only.
One way I thought would be keeping a separate collection that points a the current collection to be used, so the clients can programmatically act on that. It feels cumbersome, thou.
Is there a Meteor-way to achieve this?
Thanks
In meteor, whenever you have a problem that sounds like: "I need to synchronize data across clients", you should use a collection. I realize it seems like overkill just to send one piece of data, but I assure you it's currently the path of least resistance.
There are ways you can expose pseudo-collections which don't actually write to mongo, but for your use case that really sounds like overkill - new Mongo.Collection is the way to go.
You can use streams to setup a simple line of communication between connected clients and the server. It doesn't store data in MongoDB. Just let all connected clients listen to a stream and switch subscriptions when a new message comes in with the subscription name. Make sure only your client associated to your admin can push messages to the stream.
Available package: https://atmospherejs.com/lepozepo/streams
Examples: http://arunoda.github.io/meteor-streams/
I need users to be able to post data from a single page browser application (SPA) to me, but I can't put server-side code on the host.
Is there a web service that I can use for this? I looked at Amazon SQS (simple queue service) but I can't call their REST APIs from within the browser due to cross origin policy.
I favour ease of development over robustness right now, so even just receiving an email would be fine. I'm not sure that the site is even going to catch on. If it does, then I'll develop a server-side component and move hosts.
Not only there are Web Services, but nowadays there are robust systems that provide a way to server-side some logic on your applications. They are called BaaS or Backend as a Service providers, usually to provide some backbone to your front end applications.
Although they have multiple uses, I'm going to list the most common in my opinion:
For mobile applications - Instead of having to learn an API for each device you code to, you can use an standard platform to store logic and data for your application.
For prototyping - If you want to create a slick application, but you don't want to code all the backend logic for the data -less dealing with all the operations and system administration that represents-, through a BaaS provider you only need good Front End skills to code the simplest CRUD applications you can imagine. Some BaaS even allow you to bind some Reduce algorithms to calls your perform to their API.
For web applications - When PaaS (Platform as a Service) came to town to ease the job for Backend End developers in order to avoid the hassle of System Administration and Operations, it was just logic that the same was going to happen to the Backend. There are many clones that showcase the real power of this strategy.
All of this is amazing, but I have yet to mention any of them. I'm going to list the ones that I know the most and have actually used in projects. There are probably many, but as far as I know, this one have satisfied most of my news, whether it's any of the previously ones mentioned.
Parse.com
Parse's most outstanding features target mobile devices; however, nowadays Parse contains an incredible amount of API's that allows you to use it as full feature backend service for Javascript, Android and even Windows 8 applications (Windows 8 SDK was introduced a few months ago this year).
How does a Parse code looks in Javascript?
Parse works through classes and objects (ain't that beautiful?), so you first create a specific class (can be done through Javascript, REST or even the Data Browser manager) and then you add objects to specific classes.
First, add up Parse as a script tag in javascript:
<script type="text/javascript" src="http://www.parsecdn.com/js/parse-1.1.15.min.js"></script>
Then, through a given Application ID and a Javascript Key, initialize Parse.
Parse.initialize("APPLICATION_ID", "JAVASCRIPT_KEY");
From there, it's all object manipulation
var Person = Parse.Object.extend("Person"); //Person is a class *cof* uppercase *cof*
var personObject = new Person();
personObject.save({name: "John"}, {
success: function(object) {
console.log("The object with the data "+ JSON.stringify(object) + " was saved successfully.");
},
error: function(model, error) {
console.log("There was an error! The following model and error object were provided by the Server");
console.log(model);
console.log(error);
}
});
What about authentication and security?
Parse has a User based authentication system, which pretty much allows you to store a base of users that can manipulate the data. If map the data with User information, you can ensure that only a given user can manipulate specific data. Plus, in the settings of your Parse application, you can specify that no clients are allowed to create classes, to ensure innecesary calls are performed.
Did you REALLY used in a web application?
Yes, it was my tool of choice for a medium fidelity prototype.
Firebase.com
Firebase's main feature is the ability to provide Real Time to your application without all the hassle. You don't need a MeteorJS server in order to bring Push Notifications to your software. If you know Javascript, you are half way through to bring Real Time magic to your users.
How does a Firebase looks in Javascript?
Firebase works in a REST fashion, and I think they do an amazing job structuring the Glory of REST. As a good example, look at the following Resource structure in Firebase:
https://SampleChat.firebaseIO-demo.com/users/fred/name/first
You don't need to be a rocket scientist to know that you are retrieve the first name of the user "Fred", giving there's at least one -usually there should be a UUID instead of a name, but hey, it's an example, give me a break-.
In order to start using Firebase, as with Parse, add up their CDN Javascript
<script type='text/javascript' src='https://cdn.firebase.com/v0/firebase.js'></script>
Now, create a reference object that will allow you to consume the Firebase API
var myRootRef = new Firebase('https://myprojectname.firebaseIO-demo.com/');
From there, you can create a bunch of neat applications.
var USERS_LOCATION = 'https://SampleChat.firebaseIO-demo.com/users';
var userId = "Fred"; // Username
var usersRef = new Firebase(USERS_LOCATION);
usersRef.child(userId).once('value', function(snapshot) {
var exists = (snapshot.val() !== null);
if (exists) {
console.log("Username "+userId+" is part of our database");
} else {
console.log("We have no register of the username "+userId);
}
});
What about authentication and security?
You are in luck! Firebase released their Security API about two weeks ago! I have yet to explore it, but I'm sure it fills most of the gaps that allowed random people to use your reference to their own purpose.
Did you REALLY used in a web application?
Eeehm... ok, no. I used it in a Chrome Extension! It's still in process but it's going to be a Real Time chat inside a Chrome Extension. Ain't that cool? Fine. I find it cool. Anyway, you can browse more awesome examples for Firebase in their examples page.
What's the magic of these services? If you read your Dependency Injection and Mock Object Testing, at some point you can completely replace all of those services for your own through a REST Web Service provider.
Since these services were created to be used inside any application, they are CORS ready. As stated before, I have successfully used both of them from multiple domains without any issue (I'm even trying to use Firebase in a Chrome Extension, and I'm sure I will succeed soon).
Both Parse and Firebase have Data Browser managers, which means that you can see the data you are manipulating through a simple web browser. As a final disclaimer, I have no relationship with any of those services other than the face that James Taplin (Firebase Co-founder) was amazing enough to lend me some Beta access to Firebase.
You actually CAN use SQS from the browser, even without CORS, as long as you only need the browser to send messages, not receive them. Warning: this is a kludge that would make my CS professors cry.
When you perform a GET request via javascript, the browser will always perform the request, however, you'll only get access to the response if it was from the same origin (protocol, host, port). This is your ticket to ride, since messages can be posted to an SQS queue with just a GET, and who really cares about the response anyways?
Assuming you're using jquery, your queue is https://sqs.us-east-1.amazonaws.com/71717171/myqueue, and allows anyone to post a message, the following will post a message with the body "HITHERE" to the queue:
$.ajax({
url: 'https://sqs.us-east-1.amazonaws.com/71717171/myqueue' +
'?Action=SendMessage' +
'&Version=2012-11-05' +
'&MessageBody=HITHERE'
})
The'll be an error in the console saying that the request failed, but the message will show up in the queue anyways.
Have you considered JSONP? That is one way of calling cross-domain scripts from javascript without running into the same origin policy. You're going to have to set up some script somewhere to send you the data, though. Javascript just isn't up to the task.
Depending in what kind of data you want to send, and what you're going to do with it, one way of solving it would be to post the data to a Google Spreadsheet using Ajax. It's a bit tricky to accomplish though.Here is another stackoverflow question about it.
If presentation isn't that important you can just have an embedded Google Spreadsheet Form.
What about mailto:youremail#goeshere.com ? ihihi
Meantime, you can turn on some free hostings like Altervista or Heroku or somenthing else like them .. so you can connect to their server , if i remember these free services allows servers p2p, so you can create a sort of personal web services and push ajax requests as well, obviously their servers are slow for free accounts, but i think it's enought if you do not have so much users traffic, else you should turn on some better VPS or Hosting or Cloud solution.
Maybe CouchDB can provide what you're after. IrisCouch provides free CouchDB instances. Lock it down so that users can't view documents and have a sensible validation function and you've got yourself an easy RESTful place to stick your data in.