Develop Reference

How to model the rows in getData() since i have nested JSON data?

I want to display these fields :name, age, addresses_id, addresses_city, addresses_primary for each person into data studio.
My JSON data
{
"data": [
{
"name": "Lio",
"age": 30,
"addresses": [
{
"id": 7834,
"city": "ML",
"primary": 1
},
{
"id": 5034,
"city": "MM",
"primary": 1
}
]
},
{
"name": "Kali",
"age": 41,
"addresses": [
{
"id": 3334,
"city": "WK",
"primary": 1
},
{
"id": 1730,
"city": "DC",
"primary": 1
}
]
},
...
]
}
there is no problem if i don't render the addresses field
return {
schema: requestedFields.build(),
rows: rows
};
//rows:
/*
"rows": [
{
"values": ["Lio", 30]
},
{
"values": ["Kali", 41]
},
...
]
*/
The problem is
I'm not able to model the nested JSON data in Google Data Studio. I
have the problem exactly in the "addresses" field.
Could anyone tell me what format should be for the rows in this case?

As you already know, for each name of your dataset, you clearly have more than one row (one person has multiple addresses). Data Studio only accepts a single data for each field, since arrays are not supported at all. So you need to work on this.
There are some ways to solve this, but always keep in mind that:
getSchema() should return all available fields for your connector (the order doesn't really matter, since Data Studio always sort alphabetically the available fields)
getData() should return a list of values. But here the order is relevant: it should be the same as the parameter passed to getData() (which means the results should be dynamic, sometimes you'll return all values, sometimes not, and the order may change).
Solution 1: Return multiple rows per record
Since you can produce multiple rows for each name, just do it.
To achieve this, your field definition (=getSchema()) should include fields address_id, address_city and address_primary (you can also add address_order if you need to know the position of the address in the list).
Supposing getData() is called with all fields in the same order they were discribed, rows array should look like this:
"rows": [
{
"values": ["Lio", 30, "7834", "ML", 1]
},
{
"values": ["Lio", 30, "5034", "MM", 1]
},
{
"values": ["Kali", 41, "3334", "WK", 1]
},
{
"values": ["Kali", 41, "1730", "DC", 1]
},
...
]
IMO, this is the best solution for your data.
Solution 2: Return one address only, ignoring others
If you prefer one row per person, you can get one of the addresses and display only it (usually the main/primary address, or the first one).
To achieve this, your field definition (=getSchema()) should include fields address_id, address_city and address_primary.
Supposing getData() is called with all fields in the same order they were discribed, rows array should look like this:
"rows": [
{
"values": ["Lio", 30, "7834", "ML", 1]
},
{
"values": ["Kali", 41, "3334", "WK", 1]
},
...
]
Solution 3: Return all addresses, serialized in a field
This is helpful if you really need all information but do not want a complex scheme.
Just create a field called addresses in your field definition (=getSchema()) and write the JSON there as a string (or any other format you want).
Supposing getData() is called with all fields in the same order they were discribed, rows array should look like this:
"rows": [
{
"values": ["Lio", 30, "[{\"id\": 7834, \"city\": "ML", \"primary\": 1}, {\"id\": 5034, \"city\": \"MM\", \"primary\": 1}]"]
},
{
"values": ["Kali", 41, "[{\"id\": 3334, \"city\": \"WK\", \"primary\": 1}, {\"id\": 1730, \"city\": \"DC\", \"primary\": 1}]"]
},
...
]
This solution may appear senseless, but it is possible to interact with this data later in DataStudio using REGEX if really needed.
Solution 4: Create a different field for each address
If you're sure all records has a maximum number of addresses (in you example, both names have 2 addresses, for example), you can create multiple fields.
Your field definition (=getSchema()) should include fields address_id1, address_city1, address_primary1, address_id2, ... address_primaryN.
I wouldn't explain how rows should look like in this situation, but it is not hard to guess with the other examples.

Map dependencies in JSON structure

I am currently building a web tool which enables the user to generate a package of options in the form of a String. To select which options he wants he uses a form with different inputs (radio, checkbox) which is generated from a dictionary.json that currently holds all available options and their codes in the following format (subject to change):
[
{
"id": 0001,
"title":"foo",
"type":"radio",
"options":[
{
"bar":"",
"foo":"489",
"foobar":"489+490"
}
]
},
{
"id": 0002,
"title":"something",
"type":"check",
"options":[
{
"everything":"M016",
"evenmore":"M139"
}
]
},
[...]
As you can see it is basically a small database. The problem is that the options depend on each other so if foo is foobar it might determine that something is definitely evenmore and can NOT be changed to everything. How would I map these dependencies in the dictionary.json so that the generated form can reliably grey out options that are determined by other choices?
The structure has to be flexible so new dependencies can be inserted and would generate the new form reliably or validate existing outputs against them. There could also be options that depend on multiple other options. I can't think of a smart way of saving these dependencies and I wonder if JSON is the right format to go with here.
Any tips or ideas are welcome. Thanks!

You could try to save every option as one object which stores all the options which will be excluded if that option is selected.
So your JSON could look like the following:
[
{
"id": 0001,
"title":"foo",
"type":"radio",
"options":[
{
"bar":"",
"excludes": []
},
{
"foo":"489",
"excludes": []
},
{
"foobar":"489+490",
"excludes": [
{
"id": 0002,
"options": [
"everything"
],
},
{
"id": 0003,
"options": [
"apple",
"cherry"
],
},
]
}
]
},
{
"id": 0002,
"title":"something",
"type":"check",
"options":[
{
"everything":"M016",
"excludes": []
},
{
"evenmore":"M139",
"excludes": []
}
]
},
[...]
Everytime an option is selected you would have to check their excludes list and diable all those options for the specific fields.
To improve the usability you could check there is only one option left for a field, select this option and then disable the whole field.
EDIT:
Additionally you could save a isExcludedBy field to each of the options.
The everything option of id 0002 would then look like this:
"isExcludedBy": [
"id": 0001,
"options": [
"foobar"
]
]
This would be kind of redundant, but depending on what you want your UI to show, it could save you some computing time.

A possible simple solution (which answers your question):
// dictionary.json
{
"options": [
{
"id": 0001,
"title":"foo",
"type":"radio",
"options":[
{
"bar":"",
"foo":"489",
"foobar":"489+490"
}
]
}
// etc.; same as before
],
// this is it:
"dependencies": [
[["0001", "foobar"], ["0002", "evenmore"]],
]
}
dependencies here consist of pairs of [path to option in options that implies another option, path to the implied option].
You could make a Map data structure out of this directly (the implying options are keys, the implied are values).
This assumes that one option can imply only one other option (but it still allows for options that depend on multiple other options).
You could of course easily extend that like so:
[["0001", "foobar"], [["0002", "evenmore"], ["0003", "namaste"]]]
This would mean that "0001"/"foobar" implies both "0002"/"evenmore" and "0003"/"namaste". But perhaps YAGNI. :)

One way to approach this is to model the domain you're actually expressing, and generate the form based on that. For example, we know that apartments have street numbers, and apartment numbers, whereas houseboats don't even have streets.
{
"dwelling": {
"type": "houseboat",
"latitude": null,
"longitude": null,
}
}
or
{
"dwelling": {
"type": "apartment",
"street": "Beech St.",
"street_number": 123,
"apartment_number": 207,
}
}
By modelling the domain rather than the form, you can write rules that apply beyond the form, and you won't have to develop a mini-language for expressing form dependencies.

MongoDb: How to get a field (sub document) from a document?

Consider this example collection:
{
"_id:"0,
"firstname":"Tom",
"children" : {
"childA":{
"toys":{
'toy 1':'batman',
'toy 2':'car',
'toy 3':'train',
}
"movies": {
'movie 1': "Ironman"
'movie 2': "Deathwish"
}
},
"childB":{
"toys":{
'toy 1':'doll',
'toy 2':'bike',
'toy 3':'xbox',
}
"movies": {
'movie 1': "Frozen"
'movie 2': "Barbie"
}
}
}
}
Now I would like to retrieve ONLY the movies from a particular document.
I have tried something like this:
movies = users.find_one({'_id': 0}, {'_id': 0, 'children.ChildA.movies': 1})
However, I get the whole field structure from 'children' down to 'movies' and it's content. How do I just do a query and retrieve only the content of 'movies'?
To be specific I want to end up with this:
{
'movie 1': "Frozen"
'movie 2': "Barbie"
}

The problem here is your current data structure is not really great for querying. This is mostly because you are using "keys" to actually represent "data points", and while it might initially seem to be a logical idea it is actually a very bad practice.
So rather than do something like assign "childA" and "childB" as keys of an object or "sub-document", you are better off assigning these are "values" to a generic key name in a structure like this:
{
"_id:"0,
"firstname":"Tom",
"children" : [
{
"name": "childA",
"toys": [
"batman",
"car",
"train"
],
"movies": [
"Ironman"
"Deathwish"
]
},
{
"name": "childB",
"toys": [
"doll",
"bike",
"xbox",
],
"movies": [
"Frozen",
"Barbie"
]
}
]
}
Not the best as there are nested arrays, which can be a potential problem but there are workarounds to this as well ( but later ), but the main point here is this is a lot better than defining the data in "keys". And the main problem with "keys" that are not consistently named is that MongoDB does not generally allow any way to "wildcard" these names, so you are stuck with naming and "absolute path" in order to access elements as in:
children -> childA -> toys
children -> childB -> toys
And that in a nutshell is bad, and compared to this:
"children.toys"
From the sample prepared above, then I would say that is a whole lot better approach to organizing your data.
Even so, just getting back something such as a "unique list of movies" is out of scope for standard .find() type queries in MongoDB. This actually requires something more of "document manipulation" and is well supported in the aggregation framework for MongoDB. This has extensive capabilities for manipulation that is not present in the query methods, and as a per document response with the above structure then you can do this:
db.collection.aggregate([
# De-normalize the array content first
{ "$unwind": "$children" },
# De-normalize the content from the inner array as well
{ "$unwind": "$children.movies" },
# Group back, well optionally, but just the "movies" per document
{ "$group": {
"_id": "$_id",
"movies": { "$addToSet": "$children.movies" }
}}
])
So now the "list" response in the document only contains the "unique" movies, which corresponds more to what you are asking. Alternately you could just $push instead and make a "non-unique" list. But stupidly that is actually the same as this:
db.collection.find({},{ "_id": False, "children.movies": True })
As a "collection wide" concept, then you could simplify this a lot by simply using the .distinct() method. Which basically forms a list of "distinct" keys based on the input you provide. This playes with arrays really well:
db.collection.distinct("children.toys")
And that is essentially a collection wide analysis of all the "distinct" occurrences for each"toys" value in the collection, and returned as a simple "array".
But as for you existing structure, it deserves a solution to explain, but you really must understand that the explanation is horrible. The problem here is that the "native" and optimized methods available to general queries and aggregation methods are not available at all and the only option available is JavaScript based processing. Which even though a little better through "v8" engine integration, is still really a complete slouch when compared side by side with native code methods.
So from the "original" form that you have, ( JavaScript form, functions have to be so easy to translate") :
db.collection.mapReduce(
// Mapper
function() {
var id this._id;
children = this.children;
Object.keys(children).forEach(function(child) {
Object.keys(child).forEach(function(childKey) {
Object.keys(childKey).forEach(function(toy) {
emit(
id, { "toys": [children[childkey]["toys"][toy]] }
);
});
});
});
},
// Reducer
function(key,values) {
var output = { "toys": [] };
values.forEach(function(value) {
value.toys.forEach(function(toy) {
if ( ouput.toys.indexOf( toy ) == -1 )
output.toys.push( toy );
});
});
},
{
"out": { "inline": 1 }
}
)
So JavaScript evaluation is the "horrible" approach as this is much slower in execution, and you see the "traversing" code that needs to be implemented. Bad news for performance, so don't do it. Change the structure instead.
As a final part, you could model this differently to avoid the "nested array" concept. And understand that the only real problem with a "nested array" is that "updating" a nested element is really impossible without reading in the whole document and modifying it.
So $push and $pull methods work fine. But using a "positional" $ operator just does not work as the "outer" array index is always the "first" matched element. So if this really was a problem for you then you could do something like this, for example:
{
"_id:"0,
"firstname":"Tom",
"childtoys" : [
{
"name": "childA",
"toy": "batman"
}.
{
"name": "childA",
"toy": "car"
},
{
"name": "childA",
"toy": "train"
},
{
"name": "childB",
"toy": "doll"
},
{
"name": "childB",
"toy": "bike"
},
{
"name": "childB",
"toy": "xbox"
}
],
"childMovies": [
{
"name": "childA"
"movie": "Ironman"
},
{
"name": "childA",
"movie": "Deathwish"
},
{
"name": "childB",
"movie": "Frozen"
},
{
"name": "childB",
"movie": "Barbie"
}
]
}
That would be one way to avoid the problem with nested updates if you did indeed need to "update" items on a regular basis rather than just $push and $pull items to the "toys" and "movies" arrays.
But the overall message here is to design your data around the access patterns you actually use. MongoDB does generally not like things with a "strict path" in the terms of being able to query or otherwise flexibly issue updates.

Projections in MongoDB make use of '1' and '0' , not 'True'/'False'.
Moreover ensure that the fields are specified in the right cases(uppercase/lowercase)
The query should be as below:
db.users.findOne({'_id': 0}, {'_id': 0, 'children.childA.movies': 1})
Which will result in :
{
"children" : {
"childA" : {
"movies" : {
"movie 1" : "Ironman",
"movie 2" : "Deathwish"
}
}
}
}

How to search and select an object in JSON

Consider a JSON like this:
[{
"type": "person",
"name": "Mike",
"age": "29"
},
{
"type": "person",
"name": "Afshin",
"age": "21"
},
{
"type": "something_else",
"where": "NY"
}]
I want to search in the JSON value with a key (for example type='person') and then select a whole object of matched item in JSON. For example when I search for type='person' I expect this value:
[{
"type": "person",
"name": "Mike",
"age": "29"
},
{
"type": "person",
"name": "Afshin",
"age": "21"
}]
Because it's a really big JSON value, I don't want to do a brute-force search in all nodes, so I think the only way is using Regular Expressions but I don't know how can I write a Regex to match something like above.
I'm using NodeJs for the application.

Using underscore.js#where:
var results = _(yourObject).where({ type: 'person' })
If your data set is very very big [e.g. 10k or so], consider filtering / paginating stuff server side.

Plain javascript :
var results = dataset.filter(function(p) {
if(p.type == 'person')
return true;
});

If the requirement is to scan multiple times through the collection, the following one time construction overhead might be of worth.
Use hashing based on values of type.Convert the current data structure to hash map.
var hashMap ={
};
hashMap['person'] =[{},{}];
Hope this helps you.

Use
$.grep(jsonarrayobj,function(n, i){
if(n.type==="person")
{}
})

Backbone.js + Relational + AMD ... bootstrapping relationals

I feel sort of lost and overlooking something, but i am not sure how to approach to this and even not very much sure how to ask...
First of all, i am using AMD approach (with curl.js library), which makes this probably more difficult, but i am not giving up on AMD because of this problem.
I have this structure of bootstrap data from the server, stored in 'window.bootstrap' property.
Departments = [
{"Id": 1, "Name": "Early Collections" },
{"Id": 2, "Name": "Collections" }
]
Blocks = [
{"Id": 1, "Code": "K", "Department": 1 },
{"Id": 2, "Code": "A", "Department": 2 }
]
Now i am confused about approach to this. Here is my 'DataModel/Block' module:
define [
'Collection/DepartmentCollection'
'DataModel/Department'
], (DepartmentCollection, Department) ->
Backbone.RelationalModel.extend
relations: [
type: Backbone.HasOne
key: 'Department'
relatedModel: Department
collectionType: DepartmentCollection
]
Module 'DataModel/Department' is just plain RelationalModel without any relations. Also every mentioned Collection here is also plain without anything but reference to Model like this:
define ['DataModel/Department'] , (Department) ->
Backbone.Collection.extend
model: Department
And finally, here goes Bootstrap module, which looks like this:
define [
'DataModel/Department'
'Collection/DepartmentCollection'
'DataModel/Block'
'Collection/BlockCollection'
] , (Department, DepartmentCollection, Block, BlockCollection) ->
model = Backbone.RelationalModel.extend
relations: [
type: Backbone.HasMany
key: 'Departments'
relatedModel: Department
collectionType: DepartmentCollection
,
type: Backbone.HasMany
key: 'Blocks'
relatedModel: Block
collectionType: BlockCollection
]
data = window.bootstrap || {}
boot = new model
boot.get('Departments').reset data.Departments || []
boot.get('Blocks').reset data.Blocks || []
return boot
I would expect from this, that it would find Departments for those Blocks and assign models there, but calling
console.debug ins.get('Blocks').at(0).get('Department')
...gets me undefined.
But this is not the end. I will be having other entities from server with relation to Department too. And i would like to see, it automatically attaches Department from that bootstrap, so i can use it transparently.
I don't know if i had just misunderstood this relational library, or it's not AMD ready. Any help is appreciated.

Potential scoping / name resolution problem? What output do you get for console.debug(window.Block, window.Department)? If you do get the model type, it might help to give the relatedModel as a string, e.g. relatedModel: "Department".

Looks it's solved. Problem was in one line of code...
Backbone.Model.prototype.idAttribute = "Id"
I forgot i am using PascalCase identifiers for object properties. Everything looks ok for now.