merge few json files in to one - javascript

After days of searching on google, I finally pushed myself to ask the question over here.
I have few json files.
localhost/feed01.json
localhost/feed02.json
localhost/feed03.json
all the json file structures similar to this one. but random data including random, id, date type etc...
[
{
"id":1,
"date": "12/12/2011",
"type": "Run",
"distance": "3 miles",
"comments": "This was really hard",
"minutes": 36
},
{
"id":2,
"date": "12/11/2011",
"type": "Bike",
"distance": "6 miles",
"comments": "All down hill...felt like nothing",
"minutes": 30
},
{
"id":3,
"date": "12/10/2011",
"type": "Walk",
"distance": "2.5 miles",
"comments": "Shouldn't have taken the dog",
"minutes": 45
}
]
Now I want to merge all this json files to a one and sort by date by default (on initialization). What is the best way to merge and sort using backbone.js using collections?

Backbone lets you override some of backbone.sync through options
you can pass a target url as an option to collection.fetch to override the default url
and passing add: true appends the models to the collection instead of resetting it
which means you can build a collection with
var c = new Backbone.Collection();
c.fetch({
url: ('localhost/feed01.json'),
add: true
});
c.fetch({
url: ('localhost/feed02.json'),
add: true
});
...
To invoke a callback when all fetches have completed, you can build a jQuery deferred object
var files = ['localhost/feed01.json', 'localhost/feed02.json', 'localhost/feed03.json'];
var c=new Backbone.Collection();
var dfds = [], dfd, i, l;
for (i=0, l=files.length; i<l;i++) {
dfd = c.fetch({
url: (files[i]),
add: true
});
dfds.push(dfd);
}
$.when.apply(null, dfds).done(function() {
console.log(c.pluck('id'));
});

To merge data just create a collection and all the arrays with json objects/model representations to this collection. To achieve the ability you can override the fetch method and make it do 3 ajax calls and using jquery deferreds wait until all three of them are resolved and added returned data to collection before you resolve the fetch request.
To alter the sorting logics you need to overwrite the collections comparator method to sort based on your data.
you could specify it as for example as this
comparator: function(model) {
return +new Date(model.get('date'));
}
and it would sort it by date.
If you would for whatever reason specify the comparator after adding the models to the collection you'd need to call the sort method so that the models get sorted according to the rule specified by comparator

Related

JSON Format is off

The keys and values are separated in the Json object that I get from an api call. I have tried finding a solution It looks like the following:
{
"range": "'1'!A1:AM243",
"majorDimension": "ROWS",
"values":
[
"DeptID",
"DeptDescr",
"VP Area",
"VP Descr",
"HR Category",
"Employee Relations1",
"ER1Title",
"ER1Phone",
"ER1Email",
"Employee Relations2",
"ER2Title",
"ER2Phone",
"ER2Email",
"Compensation1",
"Comp1Title",
"Comp1Phone",
"Comp1Email",
"Compensation2",
"Comp2Title",
"Comp2Phone",
"Comp2Email",
"Employment1",
"E1Title",
"E1Phone",
"E1Email",
"Employment2",
"E2Title",
"E2Phone",
"E2Email",
"Employee Pay Services1",
"EPS1Title",
"EPS1Phone",
"EPS1Email",
"Employee Pay Services2",
"EPS2Title",
"EPS2Phone",
"EPS2Email"
],
[
"20734",
"Academic Success Centers",
"VES",
"VP Enroll Mgmt & Student Aff",
"Administrative",
"Brian Schmidt",
" Employee Relations Consultant",
"(928)523-6139",
"Brian.Schmidt#nau.edu",
"Marcia Warden",
"Assistant Director, Employee Relations",
"(928)523-9624",
"Marcia.Warden#nau.edu",
"Nicole Christian",
"Employment & Compensation Analyst",
"(928)523-6127",
" Nicole.Christian#nau.edu",
"Cathy Speirs",
"Associate Director",
"(928)523-6136",
"Cathy.Speirs#nau.edu",
"Nicole Christian",
"Employment & Compensation Analyst",
"(928)523-6127",
" Nicole.Christian#nau.edu",
"Cathy Speirs",
"Associate Director",
"(928)523-6136",
"Cathy.Speirs#nau.edu",
"Katherine Kurpierz",
"Payroll Specialist",
"(928)523-6129",
"Katherine.Kurpierz#nau.edu",
"Cheryl Brothers",
"Assistant Director - HR Payroll Services",
"(928)523-6085",
"Cheryl.Brothers#nau.edu"
], etc.
But I need it to look like:
[
{
"DeptID": 20734,
"DeptDescr": "Academic Success Centers",
"VP Area": "VES",
"VP Descr": "VP Enroll Mgmt & Student Aff",
"HR Category": "Administrative",
"Employee Relations1": "Brian Schmidt",
"Employee Relations2": "Marcia Warden",
"Compensation1": "Nicole Christian",
"Compensation2": "Cathy Speirs",
"Employment1": "Nicole Christian",
"Employment3": "Cathy Speirs",
"Employee Pay Services1": "Katherine Kurpierz",
"Employee Pay Services2": "Cheryl Brothers"
},etc
I am trying to use the data to populate a drop down using javascript and ajax. Any help is really appreciated.
The object your API returns is not a valid JSON. Was that API made by you or can you get that fixed somehow?
There are 2 things you could do to make it work
-One is change it to return exactly what you want;
-Two is to fix what it returns so that it is a valid JSON;
Going for what is wrong with the file you initially posted, let's remove the contents of the arrays so it's easier to spot the problem:
Your original data looks roughly like this:
{ "range": "'1'!A1:AM243",
"majorDimension": "ROWS",
"values": [],[]
}
To be valid you would need it to look like this:
{ "range": "'1'!A1:AM243",
"majorDimension": "ROWS",
"values": {
"keys": [],
"data": [],
}
}
Notice that I wrapped the two arrays of "values" with { } because it has to be an object if you want it to contain two arrays in it.
Then I gave each array a key with which you can call them. With that you'd be able to get what you want from your "values", so that for each item in the "keys" array you have something in that "data" array.
Hope this helps.
Well let's have a look;
Suppose this is a short version of the response data you got:
var res = `
{
"range": "'1'!A1:AM243",
"majorDimension": "ROWS",
"values": [
"DeptID",
"DeptDescr",
"VP Area"
],
[
"20734",
"Academic Success Centers",
"VES"
],
[
"345543",
"Academic Fails Centers",
"OK"
]
}
`;
As we can see by the first data, looks like a dump from a spreadsheet of sorts, and someone maybe scripted a way to export this data in a JSON-ish way. The values "Array" are the rows of this "spreadsheet".
We will clean it up, and get only the chunks that looks like ["value", "another value", "etc"]
// clean tabs and returns
res = res.replace(/\t/g, '').replace(/\n/g, '');
// get the array-ish chunks
rows = res.match(/\[(((["'])(?:(?=(\\?))\4.)*?\3),*)+\]/gm)
now let's make them real arrays:
var data = rows.map(function (row) {
return JSON.parse(row);
});
Now we have an array of arrays of strings. that means, an array of "rows" that contains the values of the "cells". The first one looks like the header row (the one with the names of the fields)
Lets make objects using each row of data except the first one. The first will serve us as the keys (we match the position (index) of the value from rows[n] from the value on rows[0] to get a key-value pair)
// Here we will define an object to store data
var data_object = { values: [] };
// for each row except the first
for(var i = 1; i < data.length; i++) {
var my_data = {};
//for each element of this row
for(var j = 0; j < data[i].length; j++) {
my_data[data[0][j]] = data[i][j];
}
data_object.values.push(my_data);
}
We have our object, let's suppose you need it in JSON format now:
var json_data = JSON.stringify(data_object);
// let's look what we have here
console.log('json_data:', json_data);
We will look at something like this as a result:
json_data: {"values":[{"DeptID":"20734","DeptDescr":"Academic Success Centers","VP Area":"VES"},{"DeptID":"345543","DeptDescr":"Academic Fails Centers","VP Area":"OK"}]}
NOW A WARNING:
This is what you DON'T want to do if you can fix the API you are getting this data from first. If any inconsistency appears, things will break. and in this example i'm not managing any edge case or exception, neither checking boundaries of arrays or wrapping things in try-catch blocks.

String manipulation and aggregation

I have a log collection in MongoDB that has a structure that looks like this:
{
url : "http://example.com",
query : "name=blah,;another_param=bleh",
count : 5
}
where the "query" field is the query parameters in the requested url.
I want to compute a total of count grouped by the query parameter "name". For example, for this collection:
[{
url : "http://example.com",
query : "name=blah,;another_param=bleh",
count : 3
},
{
url : "http://example.com",
query : "name=blah,;another_param=xyz",
count : 4
},
{
url : "http://example.com",
query : "name=another_name,;another_param=bleh",
count : 3
}]
I need this output:
[{
key : "blah",
count : 7
},
{
key : "another_name",
count : 3
}]
It doesnt look like I can do this string manipulation using the aggregation framework. I can do this via map-reduce, but can a map-reduce operation be part of the aggregation pipeline?
The aggregation framework does not have the string manipulation operators necessary to dissect the string content and break this up into the key/value pairs you need for this operation. The only string manipulation currently available is $substr, which is not going to help unless you are dealing with fixed length data.
So the only server side way to do this at present is with mapReduce since you can just the JavaScript functions available to do the right manipulation. Something like this:
For the mapper:
function() {
var obj = {};
this.query.split(/,;/).forEach(function(item) {
var temp = item.split(/=/);
obj[temp[0]] = temp[1];
});
if (obj.hasOwnProperty('name')
emit(obj.name,this.count);
}
And the reducer:
function(key,values) {
return Array.sum( values );
}
Which is the basic structure of the JavaScript functions required to split out the "name" parameters and use them as the "keys" for aggregation, or general counting of the "key" occurrences.
So the aggregation framework cannot execute any JavaScript itself, as it just runs native code operators over the data.
It would be a good idea though to look at changing how your data is stored, so that the elements are broken down into a an "object" representation rather than a string when the documents are inserted to MongoDB. This allows native query forms that don't rely on JavaScript execution to manipulate the data:
[{
"url": "http://example.com",
"query": {
"name": "blah",
"another_param": "bleh"
},
"count": 3
},
{
"url": "http://example.com",
"query": {
"name": "blah",
"another_param": "xyz"
},
"count": 4
},
{
"url": "http://example.com",
"query": {
"name": "another_name",
"another_param": "bleh"
},
"count": 3
}]
This makes a $group pipeline stage quite simple as the data is now organized in a form that can be natively processed:
{ "$match": { "query.name": { "$exists": true } },
{ "$group": {
"_id": "$query.name",
"count": { "$sum": "$count" }
}}
So use mapReduce for now, but ultimately consider changing your recording of the data to split the "tokens" from the query string and represent this as structured data, optionally keeping the original string in another field.
The aggregation framework will process this much faster than mapReduce can, so this would be the better ongoing option.

Ember.js REST adapter handling different JSON structure

I'm using REST adapter, when I call App.Message.find() Ember.js makes call to the /messages to retrieve all messages and expect to see JSON structure like this:
{
"messages": [] // array contains objects
}
However API I have to work with response always with:
{
"data": [] // array contains objects
}
I only found the way1 to change namespace or URL for the API. How to tell REST adapter to look for data instead of messages property?
If this is not possible how to solve this problem? CTO said we can adapt API to use with REST adapter as we want, but from some reason we can't change this data property which will be on each response.
Assuming you are ok with writing your own adapter to deal with the difference, in the success callback you can simply modify the incoming name from "data" to your specific entity -in the case above "messages"
I do something like this to give you and idea of what if possible in a custom adapter
In the link below I highlighted the return line from my findMany
The json coming back from my REST api looks like
[
{
"id": 1,
"score": 2,
"feedback": "abc",
"session": 1
},
{
"id": 2,
"score": 4,
"feedback": "def",
"session": 1
}
]
I need to transform this before ember-data gets it to look like this
{
"sessions": [
{
"id": 1,
"score": 2,
"feedback": "abc",
"session": 1
},
{
"id": 2,
"score": 4,
"feedback": "def",
"session": 1
}
]
}
https://github.com/toranb/ember-data-django-rest-adapter/blob/master/packages/ember-data-django-rest-adapter/lib/adapter.js#L56-57
findMany: function(store, type, ids, parent) {
var json = {}
, adapter = this
, root = this.rootForType(type)
, plural = this.pluralize(root)
, ids = this.serializeIds(ids)
, url = this.buildFindManyUrlWithParent(store, type, ids, parent);
return this.ajax(url, "GET", {
data: {ids: ids}
}).then(function(pre_json) {
json[plural] = pre_json; //change the JSON before ember-data gets it
adapter.didFindMany(store, type, json);
}).then(null, rejectionHandler);
},

How to create Backbone collection from following json array?

I am getting a following JSON(/users.json) which contains users:
[
[
{ "id": "43343", "project_id": "1", "username": "Amy" }
{ "id": "34244", "project_id": "1", "username": "Tommy" }
],
[
{ "id": "76575", "project_id": "2", "username": "Izzy" }
{ "id": "13322", "project_id": "2", "username": "Sam" }
],
{ "id": "09983", "project_id": "3", "username": "Max" }
]
When project has one user I get one user hash which is not in array.
I would like to build a Backbone collection with all users. How to do that?
You provide an array of arrays of users. So to fetch all users in the init method, you can give it data but as an array of users, i.e. you will flatten this original array of arrays once with underscore flatten method :
data = _(data).flatten(true);
Then the collection constructor will natively understand your json array.
But maybe you already do this transformation in the fetching method and this is not the problem you are facing..
If you have defined a collection (say userCollection) with a user model you should be able to simply do something like this:
var col;
$.getJSON("/users.json", function(data) {
col = new userCollection(data);
});
This would more likely be done in the fetch function of the collection, but the principle here is that you can pass an array of objects to a collection and it will marshal all from json to backbone models.

Traversing a JSON object

The data which I fetch from PHP page is like:
[{
"id": "1",
"name": null,
"startdate": "2012-07-20",
"starttime": "09:53:02",
"enddate": "2012-07-20",
"endtime": "09:54:10",
"duration": "01:00:00",
"feedbacks": [{
"id": "1",
"type": "1",
"content": "cont"
}],
"conditions": [{
"id": "1",
"dev_id": "1",
"mod_id": "2",
"sub_id": "3",
"to_be_compared_value": "1",
"comparison_type": "1"
}],
"actions": [{
"id": "1",
"dev_id": "1",
"mod_id": "1",
"sub_id": "1",
"target_action": "1"
}]
}]
Which way is easy, efficent and elegant to traverse this object? I used this two until this time. Can you tell me which one must be my choice, or can you give me an alternative? And why? I have a running version of my application and I'm reviewing now my own code, and I want to take some advices from you all.
Thanks in advance,
Methods I use before:
$.map
for(var i in obj)
One more to go, I will create a table from this data.
I would use jQuery's each() (or map() if I wanted to change the data)
I should add that you should also create a function which returns an object (possibly even with some utility methods), since your data isn't very JS-friendly right now. Those dates and times, those ID's as strings.
Example:
function cleanMyObject(object){
var cleanFeedbacks = function(feedbacks){
/* ... */
return feedback;
};
object.start = /* transform date and time strings to datetime object ...*/
object.end = /*...*/
/*...*/
$.map(object.feedbacks,cleanFeedbacks);
/* cleanup the remaining objects... */
return object;
}
$.map(receivedData, cleanMyObject);
// cleanMyObject() returns the modified object so $.map will clean everything in your array.
I prefer to use http://underscorejs.org/ for things like this. It has a lot of useful functions for objects, collections etc.
If the data you are recieving doesn't change, just parse the object and use the keys you need.
All browsers I'm aware of have a function called JSON.parse to convert a JSON string into a JS object.
What I'm trying to say is: Don't be lazy, you aren't gaining any benefits from writing a "general" function if your object will always provide the same data, and there is little to no chance you can use that function again with a different object.
var myobj= JSON.parse(phpJSONstring);
var feedbacks= myobj["feedbacks"];
//do something with feedbacks
var conditions= myobj["conditions"];
//do something with conditions
etc
You can transform the json string in a javascript object, and then access the object like this:
var obj = jQuery.parseJSON(jsonString);
alert('Id='+obj.id);
var feedbackList = obj.feedbacks;
for (var i=0; i<feedbackList.length; i++) {
...
}
Reference to jQuery.parseJSON: http://api.jquery.com/jQuery.parseJSON/

Categories

Resources