I have a PostgreSQL 9.5 & meteor 1.4.
I want to select data from db and show it in the browser console.
For meteor & SQL I use this package https://github.com/storeness/meteor-postgres.
I want to show data of db in the browser console in the real time.
e.g my db is [9, 5, ..., 5].
After user came to site we add some data [9, 5, ..., 5, 6,7].
So in the console we have [6,7]
I'm newbie, how can I do that?
Many thanks.
I assume the question is "how do I get reactivity using Postgres"? I will try my best to answer this based on the knowledge I have, but please keep in mind:
I do not work for MDG so the depth of my knowledge on what they are working on is limited to what they have publicized.
I am basing this on MY knowledge, which comes from hands-on experience with a few Meteor-based projects.
Things in the JS frameworks world have been changing SUPER fast, so there may be things out there that I just didn't know about. I am relying on others to fill in here where I may have a lack of knowledge.
So, the short answer: Under current Meteor, as far as I am aware, it is not possible to have reactivity out-of-the-box with anything other than MongoDB.
Now, long answer is a little more complicated. You can write your own code to monitor the SQL server's operations and setups up the pub/sub through DDP yourself. I have some demo code in git that I did for a Meteor Meet-Up in Tampa that dove into how to hack at DDP through Pub/Sub. It's really not super-hard to do the DDP part. The hard part would be to get the reactivity setup on your SQL server of choice...in this case Postgres.
To map things out, here is the basics of what you would need to do:
Implement some sort of custom code that gets events triggered when Postgres gets updated. This code should be intelligent enough to know what tables are updated and which ones are being used in your pub/sub, so that it will update only when needed.
Use the events from #1 to drive your publication. Here is where you could either get nifty and track the data rows OR simply send all rows upon each update. If you choose the first option you will need to somehow reconcile the changes to what's been sent so you can publish the deltas. If you choose the second option then you will still need to track what results you sent to remove them all through DDP. Either way, it seems daunting. Joins and sub-queries might be near impossible to handle
From the client, we would need to simply subscribe to the publication. The client would use mini-Mongo same as before.
No matter how you slice it, it's a tough process to code. MongoDB was used in Meteor mostly because it's Oplog made tracking changes very easy, and since it's not a relational DB they didn't have to worry about the joins/subqueries issues.
The big complication comes in the form of Apollo, which is a HUGE project from MDG to re-vamp how Meteor does reactivity. Apollo is said to decouple Meteor from DDP and MongoDB, and be less pub/sub-ie and rely on GraphQL. I haven't really looked into it much, so I can't really give you details as to what any of that really means in the grand scheme of things, but the big take-away is that if you can wait it might be able to solve some, if not all, of your problems. On the flip-side, many people would say "don't count your chickens before they hatch", i.e. don't expect something that hasn't been released/delivered to solve real problems you have now.
Related
I'm developing a websocket real time browser game ( socketio nodejs) and wanted to start implementing my database and would like to know which type of database (nosql, sql .. ) would fit my situation best.
A small description of how i intend to use it :
I want to have game rooms with all their options stored ( which can be changed frequently)
a list of the connected users and their linked accounts( if a user is a guest he won't have a linked account, however if he does it would be linked with another table containing some extra data like lvl or something like that)
a list of user thema they like ( so for example when a game starts, it will take the themas in common for all users to use the game) all typical themas are stored in another big table that has a lot of data that won't be updated in a while.
So basically I have some tables that need a lot of quick and concurrent access which get updated and deleted frequently and some that don't need it and have a lot of permanent data.
I was thinking about using mongodb but honestly don't want to commit on something i don't have experience with ( i do have sql knowledge). I need suggestions thanks
TD;LR: I would personally go with PostgreSQL.
Recently, I had the same question as you, but with the only difference, I was not looking to build game rooms. After some research, I was convinced that PostgreSQL is awesome and suitable for every project. Well, tech giants use it too, including Facebook, Uber, Netflix... (https://stackshare.io/postgresql). It is scalable, easy to set one DB up, great community with lots of tutorials (both videos and articles) and it is also extensible, as it supports JSON, which is great!
You can use a great ORM for Node.js, such as TypeORM or Sequelize. Furthermore, I suggest you take a look at GraphQL, an API that has subscriptions (real-time operations). As I guess it will be a web application, I highly recommend you to go with React.js for the front-end functionalities, which interact great with GraphQL & Node.js. Last but not least, lots of developers and companies use both MongoDB and PostgreSQL for different functionalities and purposes for each. For user's tables use PostgreSQL, for multiple "big" JSON objects use MongoDB.
I hope that helps and lets you understand what you want in a more clear way. Good luck with your project!
I'm currently working on a new google polymer web application and wondered if I should use firebase as the backend/db. I took a look at the project, made some test applications and really liked it! But to fully convince me, that firebase is the way to go I need the following questions answered:
I'm a little bit concerned about security: So, I know, that firebase uses read, write and validate to implement server side security. From the samples, I noticed that the validation basically is a one-line JS script, that represents a 'if'. As I'm planning to build a web e-commerce application I need to validate quite some inputs. Is there a possibility, to outsource the validation in a separate file, to make it more readable? Also I wondered, if there is a possibility, to test these server side validations, with for example unit tests?
I'm not 100% sure at the moment, that firebase can cover all of our use cases. Would it be possible/a good solution to use a "normal" backend for some critical functions and then persist the data from the backend in firebase?
I saw some nice polymer elements for firebase. Is firebase 100% supported in polymer/web components?
Is there an other way (like Java approach) to implement server business logic?
Is there a way, to define update scripts, so that new releases can easily be pushed to production?
Thanks & kind regards
Marc
So, I asked the firebase supprt and got the following answer:
Great to meet you.
I'm a little bit concerned about security: So, I know, that firebase uses read, write and validate to implement server side security. From the samples, I noticed that the validation basically is a one-line JS script, that represents a 'if'. As I'm planning to build a web e-commerce application I need to validate quite some inputs. Is there a possibility, to outsource the validation in a separate file, to make it more readable? Also I wondered, if there is a possibility, to test these server side validations, with for example unit tests?
You can implement extremely complex and effective rules using our security rules language. You can deploy security rules as part of your hosting deploy process, or via the REST API. It's not possible to break the contents into multiple files on the server, but you could certainly build your own process for merging multiple files into a single JSON result.
I'm not 100% sure at the moment, that firebase can cover all of our use cases. Would it be possible/a good solution to use a "normal" backend for some critical functions and then persist the data from the backend in firebase?
Generally speaking, synchronizing Firebase and a SQL back end is not very practical and they don't translate well. It's probably entirely redundant as well.
I saw some nice polymer elements for firebase. Is firebase 100% supported in polymer/web components?
I don't know what 100% supported means in this context. We offer a JavaScript SDK so they should play fine together.
Is there an other way (like Java approach) to implement server business logic?
We offer official SDKs in Java, Objective-C/Swift, Android, Node.js, JavaScript, and a REST API for use with other languages.
Is there a way, to define update scripts, so that new releases can easily be pushed to production?
I'm not sure what this means. Most likely the answer is no, since we don't provide a build process or any tools to release your software.
I hope that helps!
I responded:
Thank you for the information, it helped me very much! After reading your response on question number 5 one further question popped into my mind:
…
5. Is there a way, to define update scripts, so that new releases can easily be pushed to production?
I'm not sure what this means. Most likely the answer is no, since we don't provide a build process or any tools to release your software.
Is there like a best practice on how to handle the database schema? I only have one web application (without apps, etc.) in my case... I expect, that the database will change drastically over time and releases. Should I write JS logic, that checks the current database version and update it, if it's necessary? Maybe this would make a nice feature...
For example: I deployed Version 1.0 of my application and everything works fine. After 3 months of programming I notice, that the user data needs a further attribute: address, which is a 'not null' attribute. I now deploy Version 2.0 of my application and every new registered user has a address, but the old users (from Version 1.0) do not have this field or a value.
How should I handle this?
Support responded:
Hi Marc,
There’s no best practice here, but your ideas seem fairly sound. You probably don’t need to check in your JavaScript. You can probably store a version number in the user’s profiles, and when they upgrade to the latest software, you can upgrade that in their profile data.
Then your validation rules could use something like the following:
{
"user": {
".write": "newData.hasChild('address') || newData.child('appVersion') < 4",
"address": {
".validate": "newData.isString() && newData.val().length < 1000"
}
}
}
So if you are concerned about versioning, this could be used to deal with legacy releases.
Another popular approach I’ve seen from devs is to do intermediate upgrades by duplicating data. Thus, you release an intermediate version that writes to the old path and to the new path with the updated data structure (which keeps the app working for old users till they upgrade). Once a reasonable percent of clients are upgraded, then release a final version that no longer does a dual write to the old structure and newer structure.
Of course, flattening data, while it makes joining and fetching data bit more of a pain, will make upgrades much easier as the modular data structure adapts more easily to changes. And, naturally, a pragmatic design where you wrap the various records in a class (e.g. the UserProfile class with getter/setter methods) makes transitions simpler as you can easily hack in versioning at one place.
Hope this helps someone :)
I am currently using meteor.js for a social networking application that I am building. Currently I am struggling to find the best way to design a newsfeed for my users. Users are able to follow each other and create posts that should theoretically go to their followers. I have come up with two possible concepts that would work theoretically.
I. When a users makes a post the post goes into the global database which every user is subscribed and listening to, and whenever a post is added the user checks to see if the parent if they are following the parent of the post, and if they are they display it onto their newsfeed.
This way seems horribly inefficient, but it is a way.
II. The other way would be for each user to have their own little "hub" or each user would just subscribe to and listen for change in the user's hub.
This way seems better but I would have to do a for-loop on startup and subscribe to each user manually. Which may cause a slight delay on startup.
I am thinking that option two will work best, but I am wondering if anyone has any suggestions on how to accomplish this task. Or if there is a standard way to go about this. Thanks.
Look at the reywood:publish-composite package. I'm doing something similar in my app and am using this package to essentially perform a join. In your case people are following people so when creating a publication that returns the set of users that a person is following you can at the same time return the union of all their posts.
Building scalable feed technology is quite a hard problem to solve. With Meteor it is even harder because MongoDB doesn't lend itself very well for the storage of relational data, since it is a non-relational database, and creating publications that publish 'joined' data isn't straightforward. A lot of articles have been written about the problem of building scalable feed technology, have you seen my article on using Meteor in combination with Stream to build a solution that fits your needs?
Another route would be to use an open-source framework to create the backend for your feed technology. Have a look at Stream-Framework, the repositories README also has a great list of resources on building scalable feed technology.
I tried to implement a simple login system with Mongodb and Node.js, but I found it to be too much of a hassle (everyone is building things on a hundred of frameworks and not explaining half of the code). I probably could do it, but it would require too much work and so on, as I only want to make a simple multiplayer game.
My alternative to databases are plaintext files. I do plan to hash the passwords, of course. The first problem that came to my mind was that the file could potentially get too big, and that the server would slow down completely trying to read it (in a hypothetical situations, I doubt my game will have that many users). The solution would be to create a users/ folder on the server, and split the big file into many smaller ones. Their names would correspond with the usernames, and there hashed passwords and emails would be held. This solves the speed issue, and I can't see how I still need databases!
Am I missing something? Are there other advantages to using databases? Also, I would be very pleased if someone answered with a login Mongodb example, that would solve all my problems. Just make it as simple as possible.
There are a ton of reasons to use a real database instead of some hacked, half-baked file implementation. Some big ones:
ACID (this is really 4 reasons)
Performance & stability (databases are some of the most mature pieces of software still in use)
Standardization
If you want "lightweight," I'd recommend that you start with SQLite.
Related:
Why use SQL database?
Why use MySQL over flatfiles?
Let's say you have an authentication information as: UserName(10), Password(32). These makes 42 characters. Let's say each character is 4 bytes (consisting all characters in the universe), each user consumes 42 * 4 = 168 bytes. Even if you have 100.000 users, it would only require 16 MBs of memory. It can fit the whole RAM, you only need to wirte changes to disk, and read them all in the boot of the program. Of course it would have a litle overhead, but not a big deal.
However, handling files may become an issue, you do not have to re invent the wheel. This is what databases are for. I agree, using Mongo, MySQL can be a overkill sometiems, however If you only plan to use database for authentication, you may go with more lightweight solution, you can use SQLite, which has only a file, a ~270KB client and a nice NodeJS library. Have a look at this nice post
If you are working with mongo I would strongly recommend working with the Mongoose library. It provides an excellent ORM for Mongo. If you do end up deciding to work with that ORM there is an established authentication library called mongoose-auth. I used the default password example for my own site.
If you are using NodeJS mongoDB would work well for what you want, there is also the option of redis,couch etc... MongoDB is as good a place as any to start and f you have been playing with it a little might as well stick with it.
Have a look to this repo which using a cloudfoundry or Mongolab (both free) mongoDb server.
https://github.com/daithiw44/Node_Authenticated_MongoDB
The demo has a button which when clicks hits the db to get a value. Substitute the hardcoded value for values in username and password textfields and one could authenticated a user and set up an expressjs session.
This was written with an older version of node but a tweak to the package.json should give you a good place to get started.
Hope that helps.
Me and my startup app company is working on a turn based multiplayer iPhone application. Let it be said that neither one of us have any database, or server, knowledge whatsoever. Though, we are willing to learn.
The flow of the game will be similar to such games as: WordFeud, WordsWithFriends, Rumble etc.
Let me start of by where a lot of searching on the web has gotten us:
We have decided to use CouchDB as a tool for storing information about users, game sessions and other stuff. CouchDB is an open source noSQL database system. The reason is that we have been taught, that it should support a lot off concurring users. Besides, that it scales - we are hoping to go big, of course.
Our CouchDB, is hosted on IrisCouch. IrisCouch is an "in cloud" hosting service designed for running CouchDb.
So, we've got a CouchDB server up and running, and we know the basics on how to query data from the server.
Our biggest confusion right now, is how we should set up the system to work according to best practices. Right now we are at the point where we are able to receive and submit data to the server.
Our game is supposed to have Facebook integration, so that the users can register via our app or through Facebook. After that they can play with random matched opponents, or play with friends. After a match is started, one player will get a set of question to be asked, after he has answered, the other player should be notified, through push notification, that it's their turn. After a few rounds the game is finished.
At this point, we think this might be the best solution for the flow of the application:
A user connects to another user -> a game session is opened as a
document in a database called "games".
The newly created document contains both player names, question,
answers etc
A field named "whos_turn" decides which of the two players turn it is.
After the game has ended, the session is erased.
Again, and as you may see, we are in the dark as of how to really do it, but this is the general idea.
So, my questions goes as following:
Is it best to query the data directly from the iPhone application, or through a web service?
What is the best way to set up the database, to best manage the flow of the application?
Any information, that could lead us closer will be gladly appreciated :)
In advance, thank you!
Olav Gundersen
EDIT#1 : Our Objective-C programmer managed to connect two iPhone devices using CouchDb. The iPhone application consists of a table view, that has a concurrent connection with the database, so that when someone POST to the database, it shows up on the tableview of all the other connected phones. Behold: a severely ineffective chat system.
If is a multiplayer you would need to have the app to communicate to the remote iriscouch.db but I am concerned by the point where you state that neither you or your friends have any database experience. You are willing to learn so I think the best place to start is:
http://guide.couchdb.org/editions/1/en/index.html
There are several issue you might find with scalability if you plan to erase documents continuously. DB Size can be considerable on couchdb and you will need to compact &cleanup the db regularly. But I don't think is a major issue for now as this is at a start up level.
The question "best way to setup the database and best manage the flow of the application" should be addressed by your team. If you do not have someone with any database experience you should try to find someone willing to help you. It should be someone with extensive experience in databases. You might find some fairly reasonable professionals at http://www.odesk.com
In total honestly I don't think you will be successful if you don't have such a figure - either as a freelancer or contributor - to help you having a solid database logic in the game that will ensure a great user experience.
For example: have you considered the latency-delay issue by using a db based in the USA (Iriscouch) vs. where your users are located?
For this reason you might want to do as much as possible client side (embedded database like sqlite or touchDB that is essentially couchDB for iPhone)
For an iPhone application you might want to try TouchDB that is made exactly for that
https://github.com/couchbaselabs/TouchDB-iOS (caveat: being that you need connectivity to check turns etc this might not be the ideal solution but it could work to store some information locally).
To lay this out you would need someone with experience with couchdb to set up a proper, usable application. There is nothing wrong in being enthusiast about your idea but to make it a success you need a technical mind in the database side. Of course you might be well capable to learn this yourself. After reading the CouchDB book you should be in position to create a basic flow to fit your needs.
Of course other more experienced users might come with a more comprehensive answer or a sample layout but I don't think would be the best approach. Even if someone posts a full layout of the doc structure and how to query it how are you going to service the app if something goes awry e.g. sessions don't get deleted, conflicts etc. ? hence my sincere advice to get some ad-hoc expertise for your case.
This might also result in analyzing suitable alternatives. I don't think you should buy into the idea that CouchDB can scale and hence is the best/only option for you (of course this might well be the case and if you feel that is a good option..go for it). For example twitter, google adwords and many other online apps are using mysql to store their data so for sure CouchDB is not the only database that can scale!
I think this demo app could be a good example to follow: iOS Couchbase Demo