Develop Reference

Issues sorting Row Chart in Y axis. Dc-js RowChart

This is my first question here, I'm going mad with a problem.
I'm using DC.js ( lib on top of D3 ) and I'm trying to add my own data to one of them. It sorts the data just fine when it's like 10 rows. But after that is just all over the place.
I want to group the data by price (Kurs) and add the volume together. Then sort it from low to high price.
This code runs just fine on "dc.barChart" but on rowChart I don't scale right.
I have been using the example code, but with my own CSV.
https://dc-js.github.io/dc.js/examples/row.html
var chart = dc.rowChart("#test");
d3.csv("tickdata.csv").then(function(experiments) {
experiments.forEach(function(x) {
x.Volym = +x.Volym;
x.Kurs = +(roundNumber(x.Kurs,0));
});
var ndx = crossfilter(experiments),
runDimension = ndx.dimension(function(d) {return +d.Kurs;}),
speedSumGroup = runDimension.group().reduceSum(function(d) {return +d.Volym;});
chart
.width(1024)
.height(600)
.margins({top: 20, right: 20, bottom: 20, left: 20})
.ordering(function(y){return -y.value.Kurs})
.elasticX(true)
.dimension(runDimension)
.group(speedSumGroup)
.renderLabel(true);
chart.on('pretransition', function() {
chart.select('y.axis').attr('transform', 'translate(0,10000)');
chart.selectAll('line.grid-line').attr('y2', chart.effectiveHeight());
});
chart
.render();
});
And the csv looks like this:
Tid,Volym,Volym_fiat,Kurs
2018-06-27 09:46:00,5320,6372,1515.408825
2018-06-27 09:47:00,3206,4421,1515.742652
2018-06-27 09:48:00,2699,4149,1515.013167
2018-06-27 09:49:00,3563,4198,1515.175342
And I want to sort the Y axis by "Kurs" - value. I can make this work in Bar chart but it does not work in RowChart. Please help!

It would be easier to test this with a fiddle, but it looks like in your ordering function, you assume that the reduced value will have a Kurs field (y.value.Kurs).
However, when you use group.reduceSum(), just a simple numeric value will be produced.
So this should work
.ordering(function(y){return -y.value})
This is the default behavior in dc.js 2.1+ so you might not need the line at all.
Incidentally, if you ever have problems with guessing the right shape of the reduced data, the way to troubleshoot any accessor is to put a breakpoint or console.log inside. You should see what is going wrong pretty quick with a case like this.

dc.js lineChart performance issue with 8k+ items

This is my second question on the dc.js/d3.js/crossfilter.js topic. I am trying to realize a basic personal dashboard and I started by creating a very simple lineChart (with a rangeChart associated) that outputs metrics over time.
The data I have is saved as json (it will be stored in a mongoDb instance at a later stage, so for now I used JSON that also keep datetime format) and looks like this:
[
{"date":1374451200000,"prodPow":0.0,"consPow":0.52,"toGridPow":0.0,"fromGridPow":0.52,"prodEn":0.0,"consEn":0.0,"toGridEn":0.0,"fromGridEn":0.0},
{"date":1374451500000,"prodPow":0.0,"consPow":0.34,"toGridPow":0.0,"fromGridPow":0.34,"prodEn":0.0,"consEn":0.0,"toGridEn":0.0,"fromGridEn":0.0},
{"date":1374451800000,"prodPow":0.0,"consPow":0.42,"toGridPow":0.0,"fromGridPow":0.42,"prodEn":0.0,"consEn":0.0,"toGridEn":0.0,"fromGridEn":0.0},
...
]
I have around 22000 entries like this and I am experiencing lot of performance issues when opening the dashboard. Even if I try to slice the data in a set of 8000 records, the performance are still pretty bad (but at least the rendering finishes after some time) and the interaction with the data is awful.
I am guessing that my code has some pitfall that makes it under-perform since I'd expect dc.js and crossfilter.js to struggle with 100k+ entries and more than one dimension!
Nevertheless, profiling with chrome and reading online didn't help much (more details on what I tried to change later).
Here is my graph.js code:
queue()
.defer(d3.json, "/data")
.await(makeGraphs);
function makeGraphs(error, recordsJson) {
// Clean data
var records = recordsJson;
// Slice data to avoid browser deadlock
records = records.slice(0, 8000);
// Crossfilter instance
ndx = crossfilter(records);
// Define Dimensions
var dateDim = ndx.dimension(function(d) { return d.date; });
// Define Groups
var consPowByDate = dateDim.group().reduceSum(function (d) { return d.consPow; });
var prodPowByDate = dateDim.group().reduceSum(function (d) { return d.prodPow; });
// Min and max dates to be used in the charts
var minDate = dateDim.bottom(1)[0]["date"];
var maxDate = dateDim.top(1)[0]["date"];
// Charts instance
var chart = dc.lineChart("#chart");
var volumeChart = dc.barChart('#volume-chart');
chart
.renderArea(true)
/* Make the chart as big as the bootstrap grid by not setting ".width(x)" */
.height(350)
.transitionDuration(1000)
.margins({top: 30, right: 50, bottom: 25, left: 40})
.dimension(dateDim)
/* Grouped data to represent and label to use in the legend */
.group(consPowByDate, "Consumed")
/* Function to access grouped-data values in the chart */
.valueAccessor(function (d) {
return d.value;
})
/* x-axis range */
.x(d3.time.scale().domain([minDate, maxDate]))
/* Auto-adjust y-axis */
.elasticY(true)
.renderHorizontalGridLines(true)
.legend(dc.legend().x(80).y(10).itemHeight(13).gap(5))
/* When on, you can't visualize values, when off you can filter data */
.brushOn(false)
/* Add another line to the chart; pass (i) group, (ii) legend label and (iii) value accessor */
.stack(prodPowByDate, "Produced", function(d) { return d.value; })
/* Range chart to link the brush extent of the range with the zoom focus of the current chart. */
.rangeChart(volumeChart)
;
volumeChart
.height(60)
.margins({top: 0, right: 50, bottom: 20, left: 40})
.dimension(dateDim)
.group(consPowByDate)
.centerBar(true)
.gap(1)
.x(d3.time.scale().domain([minDate, maxDate]))
.alwaysUseRounding(true)
;
// Render all graphs
dc.renderAll();
};
I Used chrome dev tools to do some CPU profiling and as a summary these are the results:
d3_json parsing at the top takes around 70ms (independent from #records)
with 2000 records:
make_graphs takes slightly under 1s;
dimensions aggregated take around 11ms;
groups aggregated take around 8ms;
dc.lineChart take around 16ms;
dc.barChart take around 8ms;
rendering takes around 700ms (450ms for lineChart);
data interaction is not super smooth but it is still good enough.
with 8000 records:
make_graphs takes around 6s;
dimensions aggregated take around 80ms;
groups aggregated take around 55ms;
dc.lineChart take around 25ms;
dc.barChart take around 15ms;
rendering takes around 5.3s (3s for lineChart);
data interaction is awful and filtering takes lot of time.
with all records the browser stalls and I need to stop the script.
After reading this thread I thought it could have been an issue with dates so I tried to modified the code to use numbers instead of dates. Here is what I modified (I will write down only the changes):
// Added before creating the crossfilter to coerce a number date
records.forEach(function(d) {
d.date = +d.date;
});
// In both the lineChart and barChart I used a numeric range
.x(d3.scale.linear().domain([minDate, maxDate]))
Unfortunately nothing noticeable changed performance-wise.
I have no clue on how to fix this and actually I would like to add more groups, dimensions and charts to the dashboard...
Edit:
Here is a github link if you want to test my code by yourself.
I used python3 and flask for the server side, so you just have to install flask:
pip3 install flask
run the dashboard:
python3 dashboard.py
and then go with your browser to:
localhost:5000

It's hard to tell without trying it out but probably what is happening is that there are too many unique dates, so you end up with a huge number of DOM objects. Remember that JavaScript is fast, but the DOM is slow - so dealing with up to half a gigabyte of data should be fine, but you can only have a few thousand DOM objects before the browser chokes up.
This is exactly what crossfilter was designed to deal with, however! All you need to do is aggregate. You're not going to be able to see 1000s of points; they will only get lost, since your chart is (probably) only a few hundred pixels wide.
So depending on the time scale, you could aggregate by hour:
var consPowByHour = dateDim.group(function(d) {
return d3.time.hour(d);
}).reduceSum(function (d) { return d.consPow; });
chart.group(consPowByHour)
.xUnits(d3.time.hours)
or similarly for minutes, days, years, whatever. It may be more complicated than you need, but this example shows how to switch between time intervals.
(I'm not going to install a whole stack to try this - most examples are JS only so it's easy to try them out in jsfiddle or whatever. If this doesn't explain it, then adding a screenshot might also be helpful.)
EDIT: I also notice that your data is integers but your scale is time-based. Maybe this causes objects to be built all the time. Please try :
records.forEach(function(d) {
d.date = new Date(+d.date);
});

What is the logic behind d3.js nice() ticks

I have generated some charts in d3.js. I use the following code to calculate the values to put in my y axis which works like a charm.
var s = d3.scale.linear().domain([minValue, maxValue]);
var ticks = s.nice().ticks(numberOfPoints);
However I now have to write python code using pycairo which generates clones of the d3.js charts on the server side.
My question is does anybody know the logic used in the above code, or something that can give similar results, so that I can replicate it in python to get nice values to use in my y axis?

D3 is open source, so you can look up the implementation. It basically boils down to rounding the extreme values:
domain[i0] = nice.floor(x0);
domain[i1] = nice.ceil(x1);

Dynamic overiding of measure axes in dimple.js causing floating droplines

eI am creating charts in dimple.js using a dynamic data set. To do this I am using addMeasureAxis for both x and y.
My problem is that I want to change the range of these axes since having the axis cross at the origin often leave my points all in the corner of the graph. To solve this I try set the x/y axis minima to the lowest value of my data by axis.overrideMin.
This gives me a graph scaled better to my data but the axes minima are still not what I had set, instead are slightly lower. As such when I mouseover the drop-lines do not reach the axis, rather they stop at my overrideMin value. Am I overriding incorrectly or can I extend where the drop-lines go to.
$scope.svg = new dimple.newSvg("dimple-chart", 800, 600);
$scope.chart = new dimple.chart($scope.svg, $scope.chartData);
$scope.chart.data = $scope.chartData;
$scope.chart.setBounds(60, 30, 500, 330);
var x,y,dataSeries;
x = $scope.chart.addMeasureAxis("x", xStatProperty);
y = $scope.chart.addMeasureAxis("y", yStatProperty);
$scope.chartData = _.sortBy($scope.chartData, xStatProperty); \\Sorts data
x.overrideMin = $scope.chartData[0][xStatProperty]; \\Overides to min value
$scope.chartData = _.sortBy($scope.chartData, yStatProperty);
y.overrideMin = $scope.chartData[0][yStatProperty];
dataSeries = $scope.chart.addSeries("Team", dimple.plot.bubble);
$scope.chart.draw();

There is nothing wrong with the approach you are using to override, it sounds like you might be using an old version of the library. Updating to version 2.1 should fix this problem.
Edit:
Following your comment below I've done some more investigation and this is caused by axis value rounding. Here is an example of the problem:
http://jsbin.com/ricud/2/edit?js,output
In order to ensure that the axes finish on a labelled point, dimple will round the axes to the next ticked value, this probably shouldn't be the case for overridden axes so please create an issue in Git Hub for that.
It's tricky to work around in a flexible way because you need to know the granularity of the axis. The example above could be fixed by using a floor calculation:
http://jsbin.com/ricud/3/edit?js,output
But that wouldn't work if the axis looked like this:
http://jsbin.com/ricud/4/edit?js,output
The only workaround I can think of relies on some internal API knowledge so is pretty hacky but I think this will work for all cases:
chart.draw();
if (x.overrideMin > 0) {
x._origin = chart._xPixels();
}
if (y.overrideMin > 0) {
y._origin = chart._yPixels() + chart._heightPixels();
}
http://jsbin.com/ricud/7/edit?js,output

How to graph dates on X axis in Rickshaw

I have a set of data for dates. What value should I provide the X axis values? How do I make Rickshaw display the X data values as dates?
I looked around the docs and examples and cannot find anything.

I've just started using Rickshaw and was in the exact situation.
But, before I go any further, Rickshaw documentation is virtually nonexistent which is very upsetting because the performance of Rickshaw compared to other JS graphing libraries is outstanding.
The best way to find examples is to dig into the source code and example code on their github page try to make sense of things (not the way documentation should be).
That being said, let's try and build a strong base of questions/answers here on StackOverflow!
So, back to the question :) It looks like you've already found your own solution to the question, but I'll provide my solution as well.
Rather than using Rickshaw.Graph.Axis.Time, I've used Rickshaw.Graph.Axis.X and set the tickFormat accordingly.
var data = [ { x: TIME_SINCE_EPOCH_IN_SECONDS, y: VALUE },
{ x: NEXT_TIME_SINCE_EPOCH_IN_SECONDS, y: NEXT_VALUE } ]
var xAxis = new Rickshaw.Graph.Axis.X({
graph: graph,
tickFormat: function(x){
return new Date(x * 1000).toLocaleTimeString();
}
})
xAxis.render();
toLocaleTimeString() can be any of the Javascript date functions, such as toLocaleString(), toLocaleDateString(), toTimeString(), or toUTCString(). Obviously, because the tickFormat takes a function as an argument one can supply their own formatter.
Koliber, I'd be interested to understand your answer if you could provide more detail as well.

Additional to Lars' reply, I found by default Rickshaw is calling
.toUTCString(x.value*1000) //(just ctrl+F to find where =) ).
In my case, I saw different time label on X between Graphite and Rickshaw for this reason, and it works beautifully once I changed it to
.toLocaleString(x.value*1000).
Plus, you may need modify this in two places : Rickshaw.Graph.Axis.Time and the ...HoverDetails

I have finally figured out that the X axis values should be epoch time values. Then, using the code from the examples I was able to show a proper time scale.
I still have a problem because I would like to show the tick marks on weeks on the X axis. However, setting timeUnit to 'week' causes JavaScript errors. It works with other time units though.

None of this worked for me. What worked with angularjs was:
'x' : d3.time.format.iso.parse(date).getTime(), 'y' : 10