I am very new in GraphQL and trying to do a simple join query. My sample tables look like below:
{
phones: [
{
id: 1,
brand: 'b1',
model: 'Galaxy S9 Plus',
price: 1000,
},
{
id: 2,
brand: 'b2',
model: 'OnePlus 6',
price: 900,
},
],
brands: [
{
id: 'b1',
name: 'Samsung'
},
{
id: 'b2',
name: 'OnePlus'
}
]
}
I would like to have a query to return a phone object with its brand name in it instead of the brand code.
E.g. If queried for the phone with id = 2, it should return:
{id: 2, brand: 'OnePlus', model: 'OnePlus 6', price: 900}
TL;DR
Yes, GraphQL does support a sort of pseudo-join. You can see the books and authors example below running in my demo project.
Example
Consider a simple database design for storing info about books:
create table Book ( id string, name string, pageCount string, authorId string );
create table Author ( id string, firstName string, lastName string );
Because we know that Author can write many Books that database model puts them in separate tables. Here is the GraphQL schema:
type Query {
bookById(id: ID): Book
}
type Book {
id: ID
title: String
pageCount: Int
author: Author
}
type Author {
id: ID
firstName: String
lastName: String
}
Notice there is no authorId on the Book type but a type Author. The database authorId column on the book table is not exposed to the outside world. It is an internal detail.
We can pull back a book and it's author using this GraphQL query:
{
bookById(id:"book-1"){
id
title
pageCount
author {
firstName
lastName
}
}
}
Here is a screenshot of it in action using my demo project:
The result nests the Author details:
{
"data": {
"book1": {
"id": "book-1",
"title": "Harry Potter and the Philosopher's Stone",
"pageCount": 223,
"author": {
"firstName": "Joanne",
"lastName": "Rowling"
}
}
}
}
The single GQL query resulted in two separate fetch-by-id calls into the database. When a single logical query turns into multiple physical queries we can quickly run into the infamous N+1 problem.
The N+1 Problem
In our case above a book can only have one author. If we only query one book by ID we only get a "read amplification" against our database of 2x. Imaging if you can query books with a title that starts with a prefix:
type Query {
booksByTitleStartsWith(titlePrefix: String): [Book]
}
Then we call it asking it to fetch the books with a title starting with "Harry":
{
booksByTitleStartsWith(titlePrefix:"Harry"){
id
title
pageCount
author {
firstName
lastName
}
}
}
In this GQL query we will fetch the books by a database query of title like 'Harry%' to get many books including the authorId of each book. It will then make an individual fetch by ID for every author of every book. This is a total of N+1 queries where the 1 query pulls back N records and we then make N separate fetches to build up the full picture.
The easy fix for that example is to not expose a field author on Book and force the person using your API to fetch all the authors in a separate query authorsByIds so we give them two queries:
type Query {
booksByTitleStartsWith(titlePrefix: String): [Book] /* <- single database call */
authorsByIds(authorIds: [ID]) [Author] /* <- single database call */
}
type Book {
id: ID
title: String
pageCount: Int
}
type Author {
id: ID
firstName: String
lastName: String
}
The key thing to note about that last example is that there is no way in that model to walk from one entity type to another. If the person using your API wants to load the books authors the same time they simple call both queries in single post:
query {
booksByTitleStartsWith(titlePrefix: "Harry") {
id
title
}
authorsByIds(authorIds: ["author-1","author-2","author-3") {
id
firstName
lastName
}
}
Here the person writing the query (perhaps using JavaScript in a web browser) sends a single GraphQL post to the server asking for both booksByTitleStartsWith and authorsByIds to be passed back at once. The server can now make two efficient database calls.
This approach shows that there is "no magic bullet" for how to map the "logical model" to the "physical model" when it comes to performance. This is known as the Object–relational impedance mismatch problem. More on that below.
Is Fetch-By-ID So Bad?
Note that the default behaviour of GraphQL is still very helpful. You can map GraphQL onto anything. You can map it onto internal REST APIs. You can map some types into a relational database and other types into a NoSQL database. These can be in the same schema and the same GraphQL end-point. There is no reason why you cannot have Author stored in Postgres and Book stored in MongoDB. This is because GraphQL doesn't by default "join in the datastore" it will fetch each type independently and build the response in memory to send back to the client. It may be the case that you can use a model that only joins to a small dataset that gets very good cache hits. You can then add caching into your system and not have a problem and benefit from all the advantages of GraphQL.
What About ORM?
There is a project called Join Monster which does look at your database schema, looks at the runtime GraphQL query, and tries to generate efficient database joins on-the-fly. That is a form of Object Relational Mapping which sometimes gets a lot of "OrmHate". This is mainly due to Object–relational impedance mismatch problem.
In my experience, any ORM works if you write the database model to exactly support your object API. In my experience, any ORM tends to fail when you have an existing database model that you try to map with an ORM framework.
IMHO, if the data model is optimised without thinking about ORM or queries then avoid ORM. For example, if the data model is optimised to conserve space in classical third normal form. My recommendation there is to avoid querying the main data model and use the CQRS pattern. See below for an example.
What Is Practical?
If you do want to use pseudo-joins in GraphQL but you hit an N+1 problem you can write code to map specific "field fetches" onto hand-written database queries. Carefully performance test using realist data whenever any fields return an array.
Even when you can put in hand written queries you may hit scenarios where those joins don't run fast enough. In which case consider the CQRS pattern and denormalise some of the data model to allow for fast lookups.
Update: GraphQL Java "Look-Ahead"
In our case we use graphql-java and use pure configuration files to map DataFetchers to database queries. There is a some generic logic that looks at the graph query being run and calls parameterized sql queries that are in a custom configuration file. We saw this article Building efficient data fetchers by looking ahead which explains that you can inspect at runtime the what the person who wrote the query selected to be returned. We can use that to "look-ahead" at what other entities we would be asked to fetch to satisfy the entire query. At which point we can join the data in the database and pull it all back efficiently in the a single database call. The graphql-java engine will still make N in-memory fetches to our code. The N requests to get the author of each book are satisfied by simply lookups in a hashmap that we loaded out of the single database call that joined the author table to the books table returning N complete rows efficiently.
Our approach might sound a little like ORM yet we did not make any attempt to make it intelligent. The developer creating the API and our custom configuration files has to decide which graphql queries will be mapped to what database queries. Our generic logic just "looks-ahead" at what the runtime graphql query actually selects in total to understand all the database columns that it needs to load out of each row returned by the SQL to build the hashmap. Our approach can only handle parent-child-grandchild style trees of data. Yet this is a very common use case for us. The developer making the API still needs to keep a careful eye on performance. They need to adapt both the API and the custom mapping files to avoid poor performance.
GraphQL as a query language on the front-end does not support 'joins' in the classic SQL sense.
Rather, it allows you to pick and choose which fields in a particular model you want to fetch for your component.
To query all phones in your dataset, your query would look like this:
query myComponentQuery {
phone {
id
brand
model
price
}
}
The GraphQL server that your front-end is querying would then have individual field resolvers - telling GraphQL where to fetch id, brand, model etc.
The server-side resolver would look something like this:
Phone: {
id(root, args, context) {
pg.query('Select * from Phones where name = ?', ['blah']).then(d => {/*doStuff*/})
//OR
fetch(context.upstream_url + '/thing/' + args.id).then(d => {/*doStuff*/})
return {/*the result of either of those calls here*/}
},
price(root, args, context) {
return 9001
},
},
Related
I have a JSON file which is 3000+ lines. What I'd like to do is create a NoSQL database with the same structure (it has embedded documents between 3-5 levels deep). But I want to add information to each level and create a schema for each item, so that I can go back at a later stage and update the information fields, and even have users log-in and change their own values.
I am using JavaScript to write a script that will iterate through the file and upload to MongoDB the schema that I want, based on the information at each level. But I'm struggling to write the code that does this efficiently. At this stage, I'm just wasting too much time trying this and that, and want to move on to the next step of my site.
Below is an example of the file. Basically, it's a bunch of embedded documents, and then at the final level (which will be at a different depth depending on which document it's in), there is an array where each of the fields is a string.
How can I use this data to create a MongoDB database while adding a schema to each item, but keeping the hierarchical nature of the documents? I want all of the documents to have one schema, and then each of the strings at the final depth to have their own, separate schema as well. I can't think of an efficient way to iterate through.
Example from the JSON file:
{
"Applied Sciences": {
"Agriculture": {
"Agricultural Economics": [
"Agricultural Environment And Natural Resources",
"Developmental Economics",
"Food And Consumer Economics",
"Production Economics And Farm Management"
],
"Agronomy": [
"Agroecology",
"Biotechnology",
"Plant Breeding",
"Soil Conservation",
"Soil Science",
"Theoretical Modeling"
],
Here's my schema for all but the strings at the end:
name: String,
completed: Boolean,
category: "Field",
items: {
type: Array
},
description: String,
resources: {
type: Array
}
};
And my rough code which at this stage just iterates through. I'm trying to use the same function call to create the Arrays in the schema, but I'm just not up to that stage yet because I can't even iterate properly through:
function createDatabase(data){
for (field in data){
items = {};
for (field in data){
if (typeof data[field] == "object");
items[field] = createDatabase(data[field]);
};
return items;
}
I've had a google for this, but can't seem to find the answer.
I have a GraphQL type that looks like this:
type Ticket {
id: Int!
bandID: Int!
band: Band
ticketURL: String!
price: Int!
date: String!
}
I'd like to be able to return something like this from MSSQL, GraphQL and JS:
[
{
id: 1,
ticketURL: "https://example.co.uk",
price: 50,
date: "2019/01/01",
band: {
id: 1,
name: "Band name"
}
}
]
What would be the most efficient way of returning a data structure like this? The first thing that comes to mind is something like the below, but it seems so inefficient and wrong.
// Call SQL to get all tickets: "SELECT * FROM Ticket"
// For each ticket
// Call SQL to get the Band
// Merge with the ticket obj
Generally speaking, I would encourage you not to shoot for what you're describing. The designing principles of the GraphQL spec encourage you to make sure that your API logic is specifically for your client's wants, and that if something better-suited to your needs comes along to replace GraphQL, you should be able to remove the GraphQL layer and replace it with whatever the new thing is without having to rewrite your logic. This specific request is, according to the GraphQL inventors, too tightly-coupled to the API. For most people, the latency to the database isn't a big enough bottleneck to need this kind of optimization. Instead, I would encourage you to use dataloaders (for caching and for bulk requests) and just to write your resolvers to call your ORM or whatever to fetch these data points.
I am working to solve a problem not dissimilar to the discussion present at the following blog post. This is wishing to publish two related data sets in Meteor, with a 'reactive join' on the server side.
https://www.discovermeteor.com/blog/reactive-joins-in-meteor/
Unfortunately for me, however, the related collection I wish to join to, will not be joined using the "_id" field, but using another field. Normally in mongo and meteor I would create a 'filter' block where I could specify this query. However, as far as I can tell in the PWR package, there is an implicit assumption to join on '_id'.
If you review the example given on the 'publish-with-relations' github page (see below) you can see that both posts and comments are being joined to the Meteor.users '_id' field. But what if we needed to join to the Meteor.users 'address' field ?
https://github.com/svasva/meteor-publish-with-relations
In the short term I have specified my query 'upside down' (as luckily I m able to use the _id field when doing a reverse join), but I suspect this will result in an inefficient query as the datasets grow, so would rather be able to do a join in the direction planned.
The two collections we are joining can be thought of as like a conversation topic/header record, and a conversation message collection (i.e. one entry in the collection for each message in the conversation).
The conversation topic in my solution is using the _id field to join, the conversation messages have a "conversationKey" field to join with.
The following call works, but this is querying from the messages to the conversation, instead of vice versa, which would be more natural.
Meteor.publishWithRelations({
handle: this,
collection: conversationMessages,
filter: { "conversationKey" : requestedKey },
options : {sort: {msgTime: -1}},
mappings: [{
//reverse: true,
key: 'conversationKey',
collection: conversationTopics,
filter: { startTime: { $gt : (new Date().getTime() - aLongTimeAgo ) } },
options: {
sort: { createdAt: -1 }
},
}]
});
Can you do a join without an _id?
No, not with PWR. Joining with a foreign key which is the id in another table/collection is nearly always how relational data is queried. PWR is making that assumption to reduce the complexity of an already tricky implementation.
How can this publish be improved?
You don't actually need a reactive join here because one query does not depend on the result of another. It would if each conversation topic held an array of conversation message ids. Because both collections can be queried independently, you can return an array of cursors instead:
Meteor.publish('conversations', function(requestedKey) {
check(requestedKey, String);
var aLongTimeAgo = 864000000;
var filter = {startTime: {$gt: new Date().getTime() - aLongTimeAgo}};
return [
conversationMessages.find({conversationKey: requestedKey}),
conversationTopics.find(requestedKey, {filter: filter})
];
});
Notes
Sorting in your publish function isn't useful unless you are using a limit.
Be sure to use a forked version of PWR like this one which includes Tom's memory leak fix.
Instead of conversationKey I would call it conversationTopicId to be more clear.
I think this could be now much easier solved with the reactive-publish package (I am one of authors). You can make any query now inside an autorun and then use the results of that to publish the query you want to push to the client. I would write you an example code, but I do not really understand what exactly do you need. For example, you mention you would like to limit topics, but you do not explain why would they be limited if you are providing requestedKey which is an ID of a document anyway? So only one result is available?
I'm building a site at the moment where there are many relational links between data. As an example, users can make bookings, which will have booker and bookee, along with an array of messages which can be attached to a booking.
An example json would be...
booking = {
id: 1,
location: 'POST CDE',
desc: "Awesome stackoverflow description."
booker: {
id: 1, fname: 'Lawrence', lname: 'Jones',
},
bookee: {
id: 2, fname: 'Stack', lname: 'Overflow',
},
messages: [
{ id: 1, mssg: 'For illustration only' }
]
}
Now my question is, how would you model this data in your angular app? And, while very much related, how would you pull it from the server?
As I can see it I have a few options.
Pull everything from the server at once
Here I would rely on the server to serialize the nested data and just use the given json object. Downsides are that I don't know what users will be involved when requesting a booking or similar object, so I can't cache them and I'll therefore be pulling a large chunk of data every time I request.
Pull the booking with booker/bookee as user ids
For this I would use promises for my data models, and have the server return an object such as...
booking = {
id: 1,
location: 'POST CDE',
desc: "Awesome stackoverflow description."
booker: 1, bookee: 2,
messages: [1]
}
Which I would then pass to a Booking constructor, which would resolve the relevant (booker,bookee and message) ids into data objects via their respective factories.
The disadvantages here are that many ajax requests are used for a single booking request, though it gives me the ability to cache user/message information.
In summary, is it better practise to rely on a single ajax request to collect all the nested information at once, or rely on various requests to 'flesh out' the initial response after the fact.
I'm using Rails 4 if that helps (maybe Rails would be more suited to a single request?)
I'm going to use a system where I can hopefully have the best of both worlds, by creating a base class for all my resources that will be given a custom resolve function, that will know what fields in that particular class may require resolving. A sample resource function would look like this...
class Booking
# other methods...
resolve: ->
booking = this
User
.query(booking.booker, booking.bookee)
.then (users) ->
[booking.booker, booking.bookee] = users
Where it will pass the value of the booker and bookee fields to the User factory, which will have a constructor like so...
class User
# other methods
constructor: (data) ->
user = this
if not isNaN(id = parseInt data, 10)
User.get(data).then (data) ->
angular.extend user, data
else angular.extend this, data
If I have passed the User constructor a value that cannot be parsed into a number (so this will happily take string ids as well as numerical) then it will use the User factorys get function to retrieve the data from the server (or through a caching system, implementation is obviously inside the get function itself). If however the value is detected to be non-NaN, then I'll assume that the User has already been serialized and just extend this with the value.
So it's invisible in how it caches and is independent of how the server returns the nested objects. Allows for modular ajax requests and avoids having to redownload unnecessary data via its caching system.
Once everything is up and running I'll write some tests to see whether the application would be better served with larger, chunked ajax requests or smaller modular ones like above. Either way this lets you pass all model data through your angular factories, so you can rely on every record having inherited any prototype methods you may want to use.
I have a blogs collection that contains title, body and agrregate rating that the users have given to them. Another collection 'Ratings' whose schema has reference to the blog, user who rated(if at all he rates them) it in the form of their ObjectIds and the rating they have given ie., +1 or -1.
When a particular user browses through blogs in the 'latest first' order (say 40 of them per page. Call them an array of blogs[0] to blogs[39]) I have to retrieve the rating documents related to this particular user and those 40 blogs if at all the user rated them and notify him of what ratings he has given those blogs.
I tried to extract all rating documents of a particular user in which blog reference objectIds lie between blogs[0]._id and blogs[39]._id which returns empty list in my case. May be objectIds cant be compared using $lt and $gt queries. In that case how should I go about it? Should I redesign my schemas to fit to this scenario?
I am using mongoosejs driver for this case. Here are the relevant parts of the code which differ a bit in execution but youu get the idea.
Schemas:
Client= new mongoose.Schema({
ip:String
})
Rates = new mongoose.Schema({
client:ObjectId,
newsid:ObjectId,
rate:Number
})
News = new mongoose.Schema({
title: String,
body: String,
likes:{type:Number,default:0},
dislikes:{type:Number,default:0},
created:Date,
// tag:String,
client:ObjectId,
tag:String,
ff:{type:Number,default:20}
});
models:
var newsm=mongoose.model('News', News);
var clientm=mongoose.model('Client', Client);
var ratesm=mongoose.model('Rates', Rates);
Logic:
newsm.find({tag:tag[req.params.tag_id]},[],{ sort:{created:-1},limit: buffer+1 },function(err,news){
ratesm.find({client:client._id,newsid:{$lte:news[0]._id,$gte:news.slice(-1)[0]._id}},function(err,ratings){
})
})
Edit:
While implementing the below said schema, I had to do this query in mongoose.js
> db.blogposts.findOne()
{ title : "My First Post", author: "Jane",
comments : [{ by: "Abe", text: "First" },
{ by : "Ada", text : "Good post" } ]
}
> db.blogposts.find( { "comments.by" : "Ada" } )
How do I do this query in mongoose?
A good practice with MongoDB (and other non-relational data stores) is to model your data so it is easy to use/query in your application. In your case, you might consider denormalizing the structure a bit and store the rating right in the blog collection, so a blog might look something like this:
{
title: "My New Post",
body: "Here's my new post. It is great. ...",
likes: 20,
dislikes: 5,
...
rates: [
{ client_id: (id of client), rate: 5 },
{ client_id: (id of another client), rate: 3 },
{ client_id: (id of a third client), rate: 10 }
]
}
The idea being that the objects in the rates array contains all the data you'll need to display the blog entry, complete with ratings, right in the single document. If you also need to query the rates in another way (e.g. find all the ratings made by user X), and the site is read-heavy, you may consider also storing the data in a Rates collection as you're doing now. Sure, the data is in two places, and it's harder to update, but it may be an overall win after you analyze your app and how it accesses your data.
Note that you can apply indexes deep into a document's structure, so for example you can index News.rates.client_id, and then you can quickly find any documents in the News collection that a particular user has rated.