My task: given a set of data in CSV format, display a sankey chart using D3.
Given data format: (I'm unable to change this)
Uses,Types,Feedback
Use1,Type1,Feedback1
Use2,Type1,Feedback1
Use2,Type2,Feedback1
...
Required format for D3 Sankey plugin:
{ "nodes": [
{"name": "Use1"},
{"name": "Use2"},
{"name": "Type1"},
...
], "links": [
{"source":"Use1", "target":"Type1", "value":1},
...
}
My problem: Convert CSV data into JSON needed for the Sankey chart. I can not change the initial data given to me, so I must build the JSON dynamically.
My research led me here, but the only example of massaging the CSV data (that does not already included values, only sources and targets) was through MySQL. As I do not have access to a database on my project, I have resorted to using Underscore.js to help me in the conversion (within a Backbone.js application)
Here is what I have so far, which works as intended.
// buildJSON is a method of a Backbone View that oversees the creation of the diagram
buildJSON: function( csv ) {
var json = {
nodes: [], // unique nodes found in data
links: [] // links between nodes
};
// get unique nodes!
var uniqueNodes = _.chain(csv).map(_.values).flatten().unique().value().sort();
uniqueNodes.forEach(function( node ) {
json.nodes.push({ name: node });
});
// map colors to nodes
this.color.domain(uniqueNodes);
// map links
var links = [];
var rMap = {};
var keys = _.keys(csv[0]);
for ( var i = 0; i < csv.length; i++ ) {
for ( var j = 0; j < keys.length - 1; j++ ) {
var relationship = csv[i][keys[j]] + '-' + csv[i][keys[j + 1]];
rMap[relationship] = ++rMap[relationship] || 1;
}
}
// create links from the linkmap
for ( var r in rMap ) {
if ( rMap.hasOwnProperty(r) ) {
var rel = r.split('-');
links.push({
source: rel[0],
target: rel[1],
value: rMap[r]
});
}
}
var nodeMap = {};
json.nodes.forEach(function( node ) { nodeMap[node.name] = node; });
json.links = links.map(function( link ) {
return {
source: nodeMap[link.source],
target: nodeMap[link.target],
value: link.value
};
});
return json;
}
This wouldn't be such an issue with a small data set, but the data may contain thousands of rows and possibly up to ~10 columns.
So, long story short, my question comes in two parts:
Are there any obvious performance gains I can achieve, and
Is there a better (more efficient) way of massaging data for a Sankey Chart in D3?
I realize this is a particularly narrow issue, so I appreciate any and all help with this!
Related
I have been able to export data from google sheets into a pretty awesome JSON file, just not in the format I need (oops:) ).
Basically, I have data where the column name is the value header (in this case: location, infected, deaths, recovered, etc.) and row name are the values respectively.
Is there a way I can reformat this into multiple objects so I have:
var death_data = {
cn: "132",
th: "0",
mo: "0",
au: "0",
sg: "0",
};
var infected_data = {
cn: "4415",
th: "8",
mo: "7",
au: "5",
sg: "5",
};
(The 2 letters are the Country Codes. CN: 3554+277+296+..other china values+...152+108+84...+13+6.)(132=125+2+1+1+1+1+1)
I found a similar resource here, however it is in R. Is there a similar method for JavaScript?
In case your wondering how I have tried to generate the objects:
function doGet() {
var result = {};
var infected = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Data').getDataRange().getValues();
var death = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Data').getDataRange().getValues();
result = makeObject(infected);
// result.death = makeObject(death);
//Logger.log(result);
return ContentService.createTextOutput(JSON.stringify(result)).setMimeType(ContentService.MimeType.JSON);
}
function makeObject(multiArr) {
var obj = {};
var headers = multiArr.shift();
for(var i = 0; i < headers.length; i++){
obj[headers[i]] = multiArr.map(function(app) {
return app[i];
});
}
return obj;
}
Using the above code, the JSON looks like this (Using JSON Viewer chrome extension):
Where each key then has values, like this:
and this
Maybe there is a better way than trying to convert the formats after the fact, and instead just generate them correctly the first time.
Sorry in advance for my basic knowledge of JavaScript, i'm just getting started.
Any help would be greatly appreciated.
Kind Regards,
Camden
You could create such a object by looping over the array and adding relevant values:
Sample script:
function makeObject(multiArr) {
var obj = {deaths:{}, infected:{}};
var headers = multiArr.shift();
multiArr.forEach(function(row){
var [_,country,_,infected,deaths,_] = row;
obj.deaths[country] = obj.deaths[country] || 0;
obj.infected[country] = obj.infected[country] || 0;
obj.deaths[country] = obj.deaths[country] + Number(deaths);
obj.infected[country] = obj.infected[country] + Number(infected);
})
return obj;
}
A javascript data object (JSON notation) has been created with the following content:
"[
{"range":"Shape","values":[{"idx":0,"val":"Random"},{"idx":1,"val":"Line"},{"idx":2,"val":"Square"},{"idx":3,"val":"Circle"},{"idx":4,"val":"Oval"},{"idx":5,"val":"Egg"}]},
{"range":"Color","values":[{"idx":0,"val":"Red"},{"idx":1,"val":"Blue"},{"idx":2,"val":"Yellow"},{"idx":3,"val":"Green"},{"idx":4,"val":"Cyan"}]}
]"
In a next step the index of an ordinal value has to be found in this object. The function should find the index of the value 'Blue' in the range 'Color'.
So the function should have the meta scripting form
f("Color")("Blue")=1
What is the most elegant form to create such a function in the context of D3 and javascript?
Depending on your use case, it might make sense to convert the data structure to a different structure more suitable for direct access. E.g. you could convert your structure to
var data = {
Shape: ['Random', 'Line', ...],
// ...
};
and access it with
data['Shape'].indexOf('Line') // or data.Shape.indexOf('Line')
Or go even one step further and convert to
var data = {
Shape: {
Random: 0,
Line: 1,
// ...
},
// ...
};
and access it with
data['Shape']['Line'] // or data.Shape.Line
What the best solution is depends on the actual use case.
Converting the structure dynamically is pretty straight forward. Here is an example to convert it to the first suggestion:
var newData = {};
data.forEach(function(item) {
newData[item.range] =
item.values.map(function(value) { return value.val; });
});
This would also reduce redundancy (e.g. idx seems to correspond with the element index).
Would this work for you ?
var dataJson = '[ \
{"range":"Shape","values":[{"idx":0,"val":"Random"},{"idx":1,"val":"Line"},{"idx":2,"val":"Square"},{"idx":3,"val":"Circle"},{"idx":4,"val":"Oval"},{"idx":5,"val":"Egg"}]},\
{"range":"Color","values":[{"idx":0,"val":"Red"},{"idx":1,"val":"Blue"},{"idx":2,"val":"Yellow"},{"idx":3,"val":"Green"},{"idx":4,"val":"Cyan"}]}\
]';
var data = JSON.parse(dataJson);
for (each in data){
if ( (data[each].range) === 'Color'){
for (eachVal in data[each].values){
if (data[each].values[eachVal].val === 'Blue'){
alert(data[each].values[eachVal].idx);
}
}
} ;
}
And here is the JSFiddle for you too.
UPDATE - Thanks for all the great answers and incredibly fast response. I've learned a great deal from the suggested solutions. I ultimately chose the answer I did because the outcome was exactly as I asked, and I was able to get it working in my application with minimal effort - including the search function. This site is an invaluable resource for developers.
Probably a simple task, but I can't seem to get this working nor find anything on Google. I am a Javascript novice and complex JSON confuses the hell out of me. What I am trying to do is make a PhoneGap Application (Phone Directory) for our company. I'll try to explain my reasoning and illustrate my attempts below.
I have JSON data of all of our employees in the following format:
[
{
"id":"1",
"firstname":"John",
"lastname":"Apple",
"jobtitle":"Engineer"
},
{
"id":"2",
"firstname":"Mark",
"lastname":"Banana",
"jobtitle":"Artist"
},
... and so on
]
The mobile framework (Framework 7) that I am using offers a "Virtual List" solution which I need to take advantage of as our directory is fairly large. The virtual list requires you to know the exact height of each list item, however, you can use a function to set a dynamic height.
What I am trying to do is create "headers" for the alphabetical listing based on their last name. The JSON data would have to be restructured as such:
[
{
"title":"A"
},
{
"id":"1",
"firstname":"John",
"lastname":"Apple",
"jobtitle":"Engineer"
},
{
"title":"B"
},
{
"id":"2",
"firstname":"Mark",
"lastname":"Banana",
"jobtitle":"Artist"
},
... and so on
]
I've been able to add key/value pairs to existing objects in the data using a for loop:
var letter, newLetter;
for(var i = 0; i < data.length; i++) {
newLetter = data[i].lastname.charAt(0);
if(letter != newLetter) {
letter = newLetter
data[i].title = letter;
}
}
This solution changes the JSON, thus outputting a title bar that is connected to the list item (the virtual list only accepts ONE <li></li> so the header bar is a div inside that bar):
{
"id":"1",
"firstname":"John",
"lastname":"Apple",
"jobtitle":"Engineer",
"title":"A"
},
{
"id":"1",
"firstname":"Mike",
"lastname":"Apricot",
"jobtitle":"Engineer",
"title":""
}
This solution worked until I tried implementing a search function to the listing. When I search, it works as expected but looks broken as the header titles ("A", "B", etc...) are connected to the list items that start the particular alphabetical section. For this reason, I need to be able to separate the titles from the existing elements and use them for the dynamic height / exclude from search results.
The question: How can I do a for loop that inserts [prepends] a NEW object (title:letter) at the start of a new letter grouping? If there is a better way, please enlighten me. As I mentioned, I am a JS novice and I'd love to become more efficient programming web applications.
var items = [
{ "lastname":"Apple" },
{ "lastname":"Banana" },
{ "lastname":"Box" },
{ "lastname":"Bump" },
{ "lastname":"Can" },
{ "lastname":"Switch" }
];
var lastC = null; //holds current title
var updated = []; //where the updated array will live
for( var i=0;i<items.length;i++) {
var val = items[i]; //get current item
var firstLetter = val.lastname.substr(0,1); //grab first letter
if (firstLetter!==lastC) { //if current title does not match first letter than add new title
updated.push({title:firstLetter}); //push title
lastC = firstLetter; //update heading
}
updated.push(val); //push current index
}
console.log(updated);
Well right now you have an array of objects - prefixing the title as its own object may be a bit confusing - a better structure may be:
[
{
title: "A",
contacts: [
{
"id":"1",
"firstname":"John",
"lastname":"Apple",
"jobtitle":"Engineer",
"title":"A"
}
]
Given your current structure, you could loop and push:
var nameIndexMap = {};
var newContactStructure = [];
for (var i = 0; i < data.length; i++) {
var letter = data[i].lastname.charAt(0);
if (nameIndexMap.hasOwnProperty(letter)) {
//push to existing
newContactStructure[nameIndexMap[letter]].contacts.push(data[i])
} else {
//Create new
nameIndexMap[letter] = newContactStructure.length;
newContactStructure.push({
title: letter,
contacts: [
data[i]
]
});
}
}
newContactStructure will now contain your sorted data.
Demo: http://jsfiddle.net/7s50k104/
Simple for loop with Array.prototype.splice will do the trick:
for (var i = 0; i < data.length; i++) {
if (i == 0 || data[i-1].lastname[0] !== data[i].lastname[0]) {
data.splice(i, 0, {title: data[i].lastname[0]});
i++;
}
}
Demo. Check the demo below.
var data = [
{"lastname":"Apple"},
{"lastname":"Banana"},
{"lastname":"Bob"},
{"lastname":"Car"},
{"lastname":"Christ"},
{"lastname":"Dart"},
{"lastname":"Dog"}
];
for (var i = 0; i < data.length; i++) {
if (i == 0 || data[i-1].lastname[0] !== data[i].lastname[0]) {
data.splice(i, 0, {title: data[i].lastname[0]});
i++;
}
}
alert(JSON.stringify( data, null, 4 ));
I'm working on some realtime graphs built with NVD3.js. I currently refresh each chart with the following:
function reDraw(c) {
d3.json(c.qs, function(data) {
d3.select(c.svg)
.datum(data)
.transition().duration(500)
.call(c.chart);
});
}
c looks like:
function Chart(svg, qs, chart) {
this.qs = qs;
this.svg = svg;
this.ylabel;
this.chart;
}
This works fairly well, but with each refresh I am fetching the whole time series again. It would be more efficient to only grab recent elements and update each graph. There are examples of doing this by appending elements (This answer NVD3 line chart with realtime data and this tutorial for example) , but this isn't ideal for me since some recent elements might be updated that are not the most recent element.
So what I'm looking to do is grab say the most recent minute (by setting query string (.qs) to only get the most recent minute, then take that result and do the following:
Overwrite any existing elements that have the same x value for each series with the most recent data
Append and elements when there are new x values from the update in each series
Expire elements past a certain age
Update the NVD3.js script with the new data. Maybe still use datum with the new merged object?
Can anyone suggest an elegant way to perform the above Merge operation? The existing data object looks like the following:
> d3.select(perf.svg).data()[0]
[
Object
key: "TrAvg"
values: Array[181]
__proto__: Object
,
Object
key: "RedisDurationMsAvg"
values: Array[181]
__proto__: Object
,
Object
key: "SqlDurationMsAvg"
values: Array[181]
__proto__: Object
]
> d3.select(perf.svg).data()[0][0]['values'][0]
Object {x: 1373979220000, y: 22, series: 0}
> d3.select(perf.svg).data()[0][1]['values'][0]
Object {x: 1373979650000, y: 2, series: 1}
The object returned would look something like the following (Except will only be maybe 6 elements or so for each object):
> d3.json(perf.qs, function(data) { foo = data })
Object {header: function, mimeType: function, response: function, get: function, post: function…}
> foo
[
Object
,
Object
,
Object
]
> foo[0]
Object {key: "TrAvg", values: Array[181]}
> foo[0]['values'][0]
Object {x: 1373980220000, y: 49}
In this newer object the series value is missing - maybe that needs to get added or perhaps D3 can do it?
For the time being I used linq.js to perform this operation, and then use .datum() to bind a new dataset each time. The solution isn't very elegant but it does seem to function:
function reDraw2(c, partial) {
if (partial) {
qs = c.qs.replace(/minago=[0-9]+/, "minago=1")
} else {
qs = c.qs
}
d3.json(qs, function(new_data) {
chart = d3.select(c.svg)
if (c.ctype == "ts" && partial) {
thirtyminago = new Date().getTime() - (60*30*1000);
old_data = chart.data()[0]
var union = new Array();
for (var i = 0; i < old_data.length; i++) {
var existing = Enumerable.From(old_data[i]['values']);
var update = Enumerable.From(new_data[i]['values']);
oldfoo = existing
newfoo = update
var temp = {}
temp['key'] = old_data[i]['key']
temp['values'] = update.Union(existing, "$.x").Where("$.x >" + thirtyminago).OrderBy("$.x").Select().ToArray();
union[i] = temp
}
chart.datum(union)
} else {
chart.datum(new_data)
}
chart.call(c.chart)
});
After reading and testing since few days (And already post here but with a wrong question) I really need you because I fail again and again...
My goal : having many series on same charts (and many charts in the future)
My data source : a mysql and a json output :
"[{"name":"Station 1","data":"1360191600,398.625"},{"name":"Station 1","data":"1360192500,398.625"},{"name":"Station 1","data":"1360193400,398.25"},{"name":"Station 1","data":"1360194300,397.375"},{"name":"Station 1","data":"1360195200,397.5"},{"name":"Station 1","data":"1360196100,397.5"},{"name":"Station 1","data":"1360199700,396.75"},{"name":"Station 1","data":"1360200600,397"}...
These data are an example because in normal time I have many station and some with only the timestamp data.
My big fail at this moment is to send these information to the series:[] option.
I have try some loops like this :
data=JSON.parse(data) ;
var series = [];
series.data = [];
$.each(data,function(i,ligne) {
var string = JSON.stringify(ligne);
var obj = $.parseJSON(string);
//var index_serie = obj.name.slice(0,1) ;
console.log(obj) ;
points=(obj.data) ;
series.name=obj.name ;
series.data.push(points) ;
}) ;
options.series.push(series) ;
console.log(options.series) ;
var chart = new Highcharts.Chart(options);
The options of the chart are defined before the ajax call.
I have try with a other format of json like ["name":"station1","data":"[1321654,10],[5465... but I have difficulties to add the [] in my sql quesry and the GROUP_CONCAT have some limitation (2014 character)
So help me to create a nice loop in order to render multiple series with their name etc
Thanks for your help...
Can't you change that "almost" JSON to "another" JSON? Maybe something like this:
[{
"name":'station 1',
"data": [ [360191600,398.625], [360191600,398.625], [360191600,398.625] ... [360191600,398.625] ]
}, {
"name":'station 2',
"data": [ [360191600,398.625], [360191600,398.625], [360191600,398.625] ... [360191600,398.625] ]
}]
If not, you have to add some parsing for your values, example:
data = JSON.parse(data);
var names = [];
$.each(data, function (i, ligne) {
var ind = names.indexOf(ligne.name),
splited = ligne.data.split(','),
x = parseFloat(splited[0]),
y = parseFloat(splited[1]);
if (ind == -1) {
/*series name spotted first time need to add new series */
ind = names.push(ligne.name) - 1;
options.series.push({
data: [],
name: ligne.name
});
}
if(!isNaN(x) && !isNaN(y)){
options.series[ind].data.push([x,y]);
}
});
And working jsfiddle: http://jsfiddle.net/3bQne/40/