I'm building a NodeJS platform that consist of several core 'parts' (users, messages and trading signals).
So my idea is to create microservices for each of them, and it works pretty well.. But I can't get my head around how to 'join' data between the microservices (I'm a frontender originally.....)
For example, I have 3 microservices.. Each with its own MongoDB, on its own machine complete isolated.. Imagine the common situation where the messages is retrieved from a single microservice, the message has a 'user_id' and I need to get the username and profilePicture to be combined in the retrieved message object..?
I read a lot about using Redis, but it seems like a 'messaging' service to me, not much of a 'combine' service.. Can anyone help me through the darkness??
Thanks!!
I know its a very general question... But I just can't get a grip of what the 'best practice' is when combining data of multiple micro services..
I believe this can be done using kafka streams, it is a pub/sub model where you can join streams based on a common key. You can either subscribe to a topic or add a data transformation service which joins multiple streams and pushes the joined/transformed data into a new stream. All services who wants to use this data can subscribe to this new stream. I recommend you go through this link
Related
I have created a mySQL database at a Host IP, but now wish to use GraphQL to make queries easier from the front-end. I've know how to set up a GraphQL server from scratch, but was unsure how to access my pre-existing tables at the back-end, and where to define the schema to use them. How would I connect to the database from a GraphQL server?
Prisma was an option I considered, but the service doesn't allow connections to mySQL databases which have pre-existing data.
Thanks for the help!
Prisma and other ORMs are a good option if you want a relatively easy/cheap way to expose CRUD operations for your entire database.
If you only need to expose specific aspects of your data or just need to start iterating quickly, you can define your GraphQL API schema at the server level and write resolvers that connect to your database as needed. Your schema does not need to reflect your entire database, but only the data you'd like to expose to clients.
In my experience with GraphQL APIs, I've found that manually writing query schemas and creating resolvers as needed for servicing the client is both faster and easier to maintain for smaller applications.
You can use a SQL client like https://github.com/mysqljs/mysql to interface with your database. The resolvers you write for your schema would then query your database for any data needed to serve the client's request, even if it spans multiple tables.
The GraphQL spec learning site graphql.org has a good description of this process https://graphql.org/learn/execution/#root-fields-resolvers
You can try a new open source tool called SwitchQL (github.com/SwitchQL/SwitchQL). I've been working on the project for a while.
You pass it your connection string and it returns everything you need to run a graphql server on top of an existing database. It also returns Apollo compliant client mutation and queries.
We only support Postgres at the moment. If you end up trying it out, please let me know what you think!
I'm making a list of tasks to learn how to use PouchDB / CouchDB, the application is quite simple, would have authentication and the user would create their tasks.
My question is regarding how to store each user's information in the database. Should I create a database for each user with their tasks? Or is there a way to put all of the tasks of all users into a database called "Tasks" and somehow filter the synchronization so that PouchDB does not synchronize the whole database (including other users' tasks) that is in the server?
(I have read the pouchdb documentation a few times and I have not been able to define this, if it is documented, please inform me where.)
You can use both approaches to fulfill your use case:
Database per user
A database per user, is the db-per-user pattern in CouchDB. CouchDB can handle the database creation/deletion each time a user is created/deleted in CouchDB. In this case each PouchDB client will replicate the complete user database.
You can enable it in the server config
This is a proper approach if the users data is isolated and you don't need to share information between users. In this case you can have some scalability issues if you need you sync many user databases with another one in CouchDB. See this post.
Single database for every user
You need to use the filtered-replication feature in CouchDB/PouchDB. This post explains how to use it.
With this approach you can replicate a subset of the CouchDB database in PouchDB
As you have a single database is easier to share info between users
But, this approach has some performance problems. The filtering process is very inefficient. As it has to process the whole dataset, including the deleted documents to determine the set of documents to be included in the replication. This filtering is done in a couchdb external process in the server which add more cost to the process.
If you need to use the filtering approach it is better to use a Mango Selector for this purpose as it is evaluated in the CouchDB main process and it could be indexed. See options.selector in the PouchDB replication filtering options.
Conclusion
Which is better? depends on your use case... In any case you should consider the scalability issues in both cases:
In the case of filtered replication, you will face some issues as the number of documents grow if you have to filter the complete dataset. This is reported to be 10x faster when using mango selectors.
In the case of db-per-user, you will have some issues if you need to consolidate the different user databases in a single one when the number of users grow.
Both pattern are valid. The only difference is that in order to use the filtered replication, you need to provide access to the main database.
Since it's in javascript, it's easy to get credentials and then access the main database. This would give users the ability to see everyone's data.
A more secure approach would be to use a database-per-user pattern. Each database will be protected by the user's credentials.
I have built a web application using AngularJS (front-end) and PHP/MySQL (back-end).
I was wondering if there is a way to "watch" the MySQL database (without Node.js), so if one user adds some data to it, the changes are synced to other users too.
E.g. I know Firebase does that, but it's object oriented database and I am unable to do the advanced queries there like I do with SQL.
I was thinking to use $interval and $http and do ajax requests, so that way I could detect changes in the database. Well, that's possible, but it'll then do thousands of http requests to the server everyday and plus interpret php on each request.
I believe nothing is impossible, I just need an idea to do this, which I don't have, so that's why I am asking for a help here.
If you want a form of "real-time communication" you'll likely have to incorporate some form of long-polling from the client. Unless you use web sockets, but that's a big post about a bunch of different things. You're right to be concerned about bandwidth and demand on the DB though. So here's my suggestion:
If you don't have experience with web sockets then log your events in a separate table/view and use the pub/sub method to subscribe entities to an event, and broadcast that event to the table. Then long-poll against the watcher view to see when changes may have occurred. If one did occur then you query for the exact value.
Another option would be to use some query system with "deciders" that hold messages. Take a look at Amazon's SQS platform for a better explanation of how this could work. Basically you have a queue that holds messages and a decider chooses where to store the message using some hash or sorting method (to reduce run time). When the client requests an update, the decider finds any messages that would apply based on the hash/sort and returns them. Then you just have to decide how and when to destruct the messages.
The second option would require a lot more tinkering though, so it's really about your preference. I think what you'll find the difficulty to be is that most solutions have to deal with the fact that the message has to be delivered 1 or More times and you'll need to track when someone received the message and if it can now be deleted from the queue/event table or if you still need to wait. Otherwise you'll consume a lot of memory.
I come from a LAMP stack background but lately I am been interested in Node.js/Angluarjs combo. Now I am wanting to get Mongodb into the mix but I am having the hardest time getting it set up. Every tutorial I find using a different kind of stack, some are using Express.js, others Mongoose, or something else. How do I go about using angularjs HTTP service to connect to Mongo? Do I need an intermediate library like Mongoose or can I do it directly with Angular. I had set this up a while back as an experiment. I am trying to do what I am doing in this page. In this, I am storing the data in a JSON file. I want to replace that JSON file with a connection to the Mongodb. I have MongoDb installed globally for Node and I have created a data collection through the console. You can see my experiment here - http://monkbunker.com/saas/#/
And the code for the http connection I have for this can be seen here starting on line 14 https://github.com/seanandersonmke/saas/blob/master/js/main.js
Is this similar to how I can work with MongoDB? How to set up an initial connection? I have sat down several times now for hours and ended up stumped every time. Please help.
thanks.
Have a look on the following links:
Querying MongoDB with JSON / HTTP / REST Interface
Does MongoDB have a native REST interface?
There is a full python proxy solution:
http://python-eve.org/
Maybe this is helpful for you.:)
I'm writing a simple web app that requires a poll - using Backbone which interfaces with a RESTful Node API and a Mongo DB.
The premise is simple: there are 5 topics, the web user can select one and lodge a vote. This will insert a document into Mongo which can then be counted for total votes.
Should I use BB models for the actual lodging of the vote or is just directly firing an API call to Node sufficient? I know that when modifying BB models you would use .save() but in this situation I'm not creating another model for the front-end - I'm just inserting a document into the DB.
Any ideas?
Yes, I would recommend you use models on the server side, primarily for validation. Although it's technically very easy to insert straight JSON from the client, you shouldn't ever blindly trust data that a user sent you because it may very well be malicious. This is especially true if you're going to resurface that data to other users, but still a good practice regardless.