I have using nodeJS and mongoose/mongoDB to build an API. Consider the following model for a generic Model (Gmodel):
{
name: "teleni",
type: "person",
data: mixed
}
The data field in this document is a mixed type so it can be anything, but I intend for it to be a subdocument/object in most cases (depending on the value of the type field). If I wanted to find all the documents that were of type person I would execute the following code:
Gmodel.find({type: "person"}, callbackFunction)
Now consider another instance of the Model:
{
name: "teleni",
type: "person",
data: {
data_type: "event",
date: "Jan 30 2019"
}
}
My issue is that I want to use mongoose to find all documents where the data field is an object/document where the data_type field is "event". I have tried the following code:
Gmodel.find({data: {data_type: "event"}}, callbackFunction)
However nothing is returned when I execute this code because find is looking for Documents where the data field is exactly {data_type: "event"}. So how do I use find to retrieve Documents whose subdocuments (of mixed type) partially match a criteria, similar to how find works on top level fields.
You can use the dot syntax data.data_type to specify a field in subdocument. This way it would match on the basis on this field only, not the entire subdocument as in your original code.
Gmodel.find({data.data_type: "event"}, callbackFunction)
Related
I am trying to create a Mongoose schema which contains a field that should be an array that contains different types/objects.
For example:
myArrayField: [
{ type: 'type1', type1Property: 'blabla', type1Property2: 'blabla'},
{ type: 'type2', type2Property2: 'blablo' }
]
I have taken a look at discriminators without any success and I have found mongoose.Schema.Types.Mixed. However, Mixed seems to be very open in terms that Mongo will not check anything about it. I would like to know if I can specify the different types the array can have, something like:
myArrayField: [ SchemaType1, SchemaType2]
So items can follow SchemaType1 OR SchemaType2.
Thank you in advance and regards
Only one type can be assigned to a field. May be you should create one schema with two fields that has different types and required attributes are set to false which is default.
Let's say my graphql server wants to fetch the following data as JSON where person3 and person5 are some id's:
"persons": {
"person3": {
"id": "person3",
"name": "Mike"
},
"person5": {
"id": "person5",
"name": "Lisa"
}
}
Question: How to create the schema type definition with apollo?
The keys person3 and person5 here are dynamically generated depending on my query (i.e. the area used in the query). So at another time I might get person1, person2, person3 returned.
As you see persons is not an Iterable, so the following won't work as a graphql type definition I did with apollo:
type Person {
id: String
name: String
}
type Query {
persons(area: String): [Person]
}
The keys in the persons object may always be different.
One solution of course would be to transform the incoming JSON data to use an array for persons, but is there no way to work with the data as such?
GraphQL relies on both the server and the client knowing ahead of time what fields are available available for each type. In some cases, the client can discover those fields (via introspection), but for the server, they always need to be known ahead of time. So to somehow dynamically generate those fields based on the returned data is not really possible.
You could utilize a custom JSON scalar (graphql-type-json module) and return that for your query:
type Query {
persons(area: String): JSON
}
By utilizing JSON, you bypass the requirement for the returned data to fit any specific structure, so you can send back whatever you want as long it's properly formatted JSON.
Of course, there's significant disadvantages in doing this. For example, you lose the safety net provided by the type(s) you would have previously used (literally any structure could be returned, and if you're returning the wrong one, you won't find out about it until the client tries to use it and fails). You also lose the ability to use resolvers for any fields within the returned data.
But... your funeral :)
As an aside, I would consider flattening out the data into an array (like you suggested in your question) before sending it back to the client. If you're writing the client code, and working with a dynamically-sized list of customers, chances are an array will be much easier to work with rather than an object keyed by id. If you're using React, for example, and displaying a component for each customer, you'll end up converting that object to an array to map it anyway. In designing your API, I would make client usability a higher consideration than avoiding additional processing of your data.
You can write your own GraphQLScalarType and precisely describe your object and your dynamic keys, what you allow and what you do not allow or transform.
See https://graphql.org/graphql-js/type/#graphqlscalartype
You can have a look at taion/graphql-type-json where he creates a Scalar that allows and transforms any kind of content:
https://github.com/taion/graphql-type-json/blob/master/src/index.js
I had a similar problem with dynamic keys in a schema, and ended up going with a solution like this:
query lookupPersons {
persons {
personKeys
person3: personValue(key: "person3") {
id
name
}
}
}
returns:
{
data: {
persons: {
personKeys: ["person1", "person2", "person3"]
person3: {
id: "person3"
name: "Mike"
}
}
}
}
by shifting the complexity to the query, it simplifies the response shape.
the advantage compared to the JSON approach is it doesn't need any deserialisation from the client
Additional info for Venryx: a possible schema to fit my query looks like this:
type Person {
id: String
name: String
}
type PersonsResult {
personKeys: [String]
personValue(key: String): Person
}
type Query {
persons(area: String): PersonsResult
}
As an aside, if your data set for persons gets large enough, you're going to probably want pagination on personKeys as well, at which point, you should look into https://relay.dev/graphql/connections.htm
I am using Kuzzle in order to build a simple slack-like application. I have a simple collection with documents only containing other documents ids.
I wanted to query a list of ids from this collection and got stuck with an empty hits array in the response, no matter what id values I tried in the query. It got me to read about Elasticsearch query syntax, mapping, and I found out I needed to specify a mapping.
When the collection is created without specifying a mapping, document.search:
returns an empty array of hits if given a "query" body property such as the following:
{"query": {"terms": {"id": <ids array>}}}
throws the following if given a "sort":
// {"sort": [{"id": {"order": "desc"}}]}
Error: No mapping found for [id] in order to sort on
and returns an empty bucket list for an "aggregations" like this :
"aggregations": {
"groupById": {
"terms": {
"field": "id"
}
}
}
I don't understand what is happening behind this. Is it normal that only one case is being treated as an error? I was working on the query part, and it was not obvious that the error didn't come from the content of my ids array!
Adding a simple mapping on "id" with the "keyword" type makes all cases work as intended, but I read about dynamic mappings in Elasticsearch. Is there a way to use them? Or is there another solution for querying a (the only) parameter in my document?
I'm using kuzzle-sdk v7.1.4
Many thanks!
Elasticsearch mappings are a way to tell the underlying engine how to index the fields of your documents.
For example, a text field will not be indexed the same way as a keyword field and you won't be able to make the same kind of query.
keyword fields are indexed "as is", so you can do very fast search on exact terms (with the term query) but text fields are analyzed with various method to be able to do a fuzzy match query for example.
If there is no declared mapping for a field, Elasticsearch won't be able to access it in any query.
You can read our Elasticsearch Cookbook for more informations.
About the mappings dynamic policy, by default this policy is set to false, meaning that ES will not infer newly introduced fields types.
We choose to have this default value because you cannot modify the type of a field so it's better to define it yourself than having poorly accurate types inferred by ES.
You can change this policy for the entire mapping but also for only a nested object for example (which is considered as a best practice)
{
"dynamic": "false",
"properties": {
"id": { "type": "keyword" },
"metadata": {
"dynamic": "true",
"properties": {
// field types will be automatically inferred in the "metadata" object
}
}
}
}
Then if you create the following document:
{
"id": "your-unique-uuid",
"name": "aschen",
"metadata": {
"avatar": "http://url.com"
}
}
id field was already declared as keyword, you can make query
on this field.
name field was not declared and the dynamic policy is false
so you cannot make query on this field.
metadata.avatar field was not declared but the dynamic policy
for this nested object is true so ES will add a text type for
this field, you can make query on this field.
One last thing, you are using an id field inside your document but Elasticsearch already generate unique identifier (_id) for every documents so you may want to use this instead.
I am a developer on Kuzzle core
If you had a collection as such.
user {
name: String,
email: String,
information: String
}
You would do something like so to get a list of all John's that have information of doctor.
db.user.find({name: "John", information: "doctor" });
Now this makes the code redundant when having variable inputs. Such as having permutations of fields to filter. I'm trying to make my query generic, such as this bad broken example.
Therefore I might want to be able to explicitly state fields that can match any value. The following examples should return the same documents in theory.
Example:
Un-Explicit (normal) db.user.find({});
Explicit (weird) db.user.find({name: {$ANY}});
Or make it even more complex.
db.user.find({name: {$ANY}, information: "doctor"});
This would not work, but the intention is to get all the documents that are doctors but have ANY sort of value on the name field, not just for John's. Maybe even something more complex like so.
db.user.find({name: function(){
if(req.query.name)//check if empty
{ return req.query.name; }
else { return $ANY; }
}, information: "doctor"});
This would then be generic enough to use a single query instance for dynamic request behavior.As far as I know there isn't anything like this implemented.
Thank you in advance.
To get all the documents that are doctors but have ANY sort of value on the name field, you need the $exists and $ne operators in your query, since the $exists operator matches the documents that contain the field, including documents where the field value is null and the $ne operator selects the documents where the name field is not null:
db.user.find({"name": { "$exists": true, "$ne": null }, "information": "doctor"});
If you want to find all the documents with information = 'doctor', why not just query without the name?
db.user.find({information: "doctor" });
I am learning nodejs along with mongodb currently, and there are two things that confuse me abit.
(1),
When a new Schema and model name are used (not in db), the name is changed into its plural form. Example:
mongoose.model('person', personSchema);
in the database, the table will be called "people" instead.
Isn't this easy to confuse new developer, why has they implemented it this way?
(2),
Second thing is that whenever I want to refer to an existing model in mongoDb (assume that in db, a table called people exists). Then in my nodejs code, I still have to define a Schema in order to create a model that refer to the table.
personSchema = new mongoose.Schema({});
mongoose.model('person',personSchema);
The unusual thing is, it does not seem to matter how I define the schema, it can just be empty like above, or fill with random attribute, yet the model will always get the right table and CRUD operations performs normally.
Then what is the usage of Schema other than defining table structure for creating new table?
Many thanks,
Actually two questions, you usually do better asking one, just for future reference.
1. Pluralization
Short form is that it is good practice. In more detail, this is generally logical as what you are referring to is a "collection" of items or objects rather. So the general inference in a "collection" is "many" and therefore a plural form of what the "object" itself is named.
So a "people" collection implies that it is in fact made up of many "person" objects, just as "dogs" to "dog" or "cats" to "cat". Not necessarily "bovines" to "cow", but generally speaking mongoose does not really deal with Polymorphic entities, so there would not be "bull" or "bison" objects in there unless just specified by some other property to "cow".
You can of course change this if you want in either of these forms and specify your own name:
var personSchema = new Schema({ ... },{ "collection": "person" });
mongoose.model( "Person", personSchema, "person" );
But a model is general a "singular" model name and the "collection" is the plural form of good practice when there are many. Besides, every SQL database ORM I can think of also does it this way. So really this is just following the practice that most people are already used to.
2. Why Schema?
MongoDB is actually "schemaless", so it does not have any internal concept of "schema", which is one big difference from SQL based relational databases which hold their own definition of "schema" in a "table" definition.
While this is often actually a "strength" of MongoDB in that data is not tied to a certain layout, some people actually like it that way, or generally want to otherwise encapsulate logic that governs how data is stored.
For these reasons, mongoose supports the concept of defining a "Schema". This allows you to say "which fields" are "allowed" in the collection (model) this is "tied" to, and which "type" of data may be contained.
You can of course have a "schemaless" approach, but the schema object you "tie" to your model still must be defined, just not "strictly":
var personSchema = new Schema({ },{ "strict": false });
mongoose.model( "Person", personSchema );
Then you can pretty much add whatever you want as data without any restriction.
The reverse case though is that people "usually" do want some type of rules enforced, such as which fields and what types. This means that only the "defined" things can happen:
var personSchema = new Schema({
name: { type: String, required: true },
age: Number,
sex: { type: String, enum: ["M","F"] },
children: [{ type: Schema.Types.ObjectId, ref: "Person" }],
country: { type: String, default: "Australia" }
});
So the rules there break down to:
"name" must have "String" data in it only. Bit of a JavaScript idiom here as everything in JavaScript will actually stringify. The other thing on here is "required", so that if this field is not present in the object sent to .save() it will throw a validation error.
"age" must be numeric. If you try to .save() this object with data other than numeric supplied in this field then you will throw a validation error.
"sex" must be a string again, but this time we are adding a "constraint" to say what the valid value are. In the same way this also can throw a validation error if you do not supply the correct data.
"children" actually an Array of items, but these are just "reference" ObjectId values that point to different items in another model. Or in this case this one. So this will keep that ObjectId reference in there when you add to "children". Mongoose can actually .populate() these later with their actual "Person" objects when requested to do so. This emulates a form of "embedding" in MongoDB, but used when you actually want to store the object separately without "embedding" every time.
"country" is again just a String and requires nothing special, but we give it a default value to fill in if no other is supplied explicitly.
There are many other things you can do, I would suggest really reading through the documentation. Everything is explained in a lot of detail there, and if you have specific questions then you can always ask, "here" (for example).
So MongoDB does things differently to how SQL databases work, and throws out some of the things that are generally held in "opinion" to be better implemented at the application business logic layer anyway.
Hence in Mongoose, it tries to "put back" some of the good things people like about working with traditional relational databases, and allow some rules and good practices to be easily encapsulated without writing other code.
There is also some logic there that helps in "emulating" ( cannot stress enough ) "joins", as there are methods that "help" you in being able to retrieve "related" data from other sources, by essentially providing definitions where which "model" that data resides in within the "Schema" definition.
Did I also not mention that "Schema" definitions are again just objects and re-usable? Well yes they are an can in fact be tied to "many" models, which may or may not reside on the same database.
Everything here has a lot more function and purpose than you are currently aware of, the good advice here it to head forth and "learn". That is the usual path to the realization ... "Oh, now I see, that's what they do it that way".