Develop Reference

Map an arbitrary value from an array to a color that exists in another array

I want to map an arbitrary value to a color scale, for use in a heat map.
I already have the ability to generate a color spectrum between 2 extreme colors as an array, like this:
create_color_spectrum("#b33939", "#ff5252", 20)
...which gives outputs like this:
["#fb5050", "#f74f4f", "#f34e4e", "#ef4d4d", "#ec4b4b", "#e84a4a", "#e44949", "#e04848", "#dc4646", "#d94545", "#d54444", "#d14343", "#cd4141", "#c94040", "#c63f3f", "#c23e3e", "#be3c3c", "#ba3b3b", "#b63a3a", "#b23838"]
Now say I have another array of values (values array) like this (same length as the colors array):
[0.020565500406834823, 0.0006918573709419904, 0.03614457831325302, 0.014884840151254727, 0.9638554216867471, 0.005208333333333333, 0.0006248326341158618, 0.14285714285714285, 0.004872900466547537, 0.8571428571428577, 0, 0.2142857142857144, 0, 0.2499999999999991, 0.5000000004656613, 0.45534591194968543, 0.6349489795918367, 0.25, 0.15218156916454706, 0]
How can I extract an appropriate color from the colors array for each value in the values array? The idea is that smaller values are towards the beginning of the colors array, and larger values are towards the end of the colors array.
I need to read in a value from the values array and map it to an index in the colors array.
I could attempt to rescale the values array, but this wouldn't produce distinct integer values, which would be needed to index the colors array.

Here I build an array of small objects to contain the value and the original index in the values array. Then I sort the new array by value, assign a color to each element by its new index, and re-sort it by original index.
const colors = ["#fb5050", "#f74f4f", "#f34e4e", "#ef4d4d", "#ec4b4b", "#e84a4a", "#e44949", "#e04848", "#dc4646", "#d94545", "#d54444", "#d14343", "#cd4141", "#c94040", "#c63f3f", "#c23e3e", "#be3c3c", "#ba3b3b", "#b63a3a", "#b23838"];
const values = [0.020565500406834823, 0.0006918573709419904, 0.03614457831325302, 0.014884840151254727, 0.9638554216867471, 0.005208333333333333, 0.0006248326341158618, 0.14285714285714285, 0.004872900466547537, 0.8571428571428577, 0, 0.2142857142857144, 0, 0.2499999999999991, 0.5000000004656613, 0.45534591194968543, 0.6349489795918367, 0.25, 0.15218156916454706, 0];
// keep track of the old index and sort by value
const valueMap = values.map((val, i) => ({oldIndex: i, value: val})).sort((a,b) => a.value - b.value);
// add the appropriate color to each element of the sorted array
valueMap.forEach((val, i) => val.color = colors[i]);
//check this console.log to verify it sorted/assigned properly
//console.log(JSON.stringify(valueMap));
// resort it by oldIndex
valueMap.sort((a,b) => a.oldIndex - b.oldIndex);
// get just the array of colors
const newColors = valueMap.map(val => val.color);
console.log(newColors);

You need to map each of your values to an index in your colour map. To do this, you need to know the minimum value you expect to see (that looks like 0 in your example), the maximum value you expect to see (I'm guessing 1) and the number of colours available (20 here).
// You can hard code these if you know them and want a constant representation
// The lowest possible value
const valueFloor = Math.min(...values);
// The range between the lowest and highest possible value
const valueRange = Math.max(...values) - valueFloor;
const maxColorIdx = colors.length - 1;
for (const value of values) {
// Normalize your value to something between 0 and 1
const normalizedValue = (value - valueFloor) / valueRange;
// Scale your normalized value to an integerrepresenting a color index
const colorIdx = Math.round(normalizedValue * maxColorIdx);
console.debug(colors[colorIdx]);
}
This will print out the "heatmap" colour for each of your values.

how to return to start when iterating over array and iterator become larger than length (in a graphics render loop)

I'm trying to make an animation recorder that records x,y positions into an array and allow the animation to be recalled. I specifically have p5.js in mind as the graphics lib, but any should work. since this is just array work.
in p5.js to return the value of Sin() or Cos() you can pass them an angle, that angle can be ever incrementing since 2PI == 4PI (in terms of the direction the rotation is facing) etc. I'm looking to replicate this kind of function but to return the data stored in an array.
so for example you've got an array like the following
let demo = ['297', '298', '299', '300']
It would be easy to loop over the array once since it has 4 items, but I'd like to write a function where if we passed in 4, it would return index 0, '297' or if we fed in 11, it would return '300' or if we fed in 22 it would return '299'
this way the function could continually be fed in an ever increasing value that moves up each frame we could return the values of the array in a loop.
let survey = 0;
let demo = ['297', '298', '299', '300']
//a rendering loop
function draw(){
survey ++
let xPos = getPosition(survey) //this getPosition function is the one in question
ellipse(xPos,100,50)
}
I feel like this is some modulo math, but I cant quite get it sorted.
thanks for taking a look!

The solution to your problem is the modulus (%) operator. This operator will return the remainder of the division.
E.g. 11 % 4 = 3
const positions = [297, 298, 299, 300];
function getPosition(positions, i) {
return positions[i % positions.length];
}
console.log(getPosition(positions, 4)); // 297
console.log(getPosition(positions, 11)); // 300
console.log(getPosition(positions, 22)); // 299

Stack a matrix in d3 without remapping to json

The docs for d3's stacking function d3.stack show an example with an array of objects (each json object representing the ensemble of points for whatever the x-axis is measuring). Eg:
var data = [
{month: new Date(2015, 0, 1), apples: 3840, bananas: 1920, cherries: 960},
{month: new Date(2015, 1, 1), apples: 1600, bananas: 1440, cherries: 720}
]
I'm trying to produce a stacked histogram with a matrix of data series ([ [], [], [], etc ]). It's easy enough to iterate through the rows and get a series of histogram bins (having pre-defined the x scale and domain elsewhere):
for(let i=0; i<data.length; i++){
bins[i] = d3.histogram()
.domain(x.domain())
.thresholds(x.ticks(10))
(data[i]);
}
And create groups for each data series inside another loop:
let bars = this.svg.selectAll(".series" + i)
.data(this.bins[i])
.enter().append("g")
.classed("series" + i, true)
But of course doing it like that I get stuck here. How am I supposed to bars.append("rect") at the correct x,y coords for that particular series? Stated differently, I have a really useful array of bins at the moment, looking something like:
[
[[1,2,3,3], [5,8,9], [10], ... etc], //series0 grouping by bins of 5
[[1,3], [7,7,9,9], [11], ... etc], //series1
[[2,3,3], [8,9], [10,12], ... etc], //series2
...etc
]
Is there a way to invoke stack without munging all the data into json key,value pairs?

I took a glance at the source and no comments + single char variables = me understanding that it's not going to happen without munging. I present therefore my shoddy attempt at saving someone else some time:
/*
* Static helper method to transform an array of histogram bins into an array of objects
* suitable for feeding into the d3.stack() function.
* Args:
* bins (array): an array of d3 histogram bins
*/
static processBins(bins){
let temp = {}; // the keys for temp will be the bin name (i.e. the bin delimiter value)
// now create an object with a key for each bin, and an empty object as a placeholder for the data
bins[0].map( (bin) => { temp[bin.x0] = {}});
for(let i=0; i<bins.length; i++){
//traverse each series
bins[i].map( bin => {
temp[bin.x0]["series"+i] = bin.length; //push the frequency counts for each series
});
}
/* now we have an object whose top-level keys are the bins:
{
binName0: { series0: freqCount0, series1: freqCount1, ...},
binName1: {...},
...
}
now, finally we're going to make an arrays of objects containing all the series' freqencies for that bin
*/
let result = [];
for(let binName in temp){ // iterate through the bin objects
let resultRow = {};
if(temp.hasOwnProperty(binName)){
resultRow["bin"] = binName; //put the bin name key/value pair into the result row
for(let seriesName in temp[binName]){ //iterate through the series keys
if(temp[binName].hasOwnProperty([seriesName])){
resultRow[seriesName] = temp[binName][seriesName];
}
}
}
result.push(resultRow);
}
return result;
}
Call like:
let stack = d3.stack().keys( bins.map( (d,i)=>{return "series"+i})); //stack based on series name keys
let layers = stack(MyCoolHistogram.processBins(bins));
//and now your layers are ready to enter() into a d3 selection.
Edit:
I note that the stack data third argument in anonymous functions seems to be the array of elements. I.e. it's no longer the stack layer index. Eg, when grouping bars side-by-side: http://bl.ocks.org/mbostock/3943967
This breaks grouping functions that rely on this index number to calculate the x position:
rect.attr("x", (d,i,j) => { return x(d.data.bin) + j*barWidth/numberOfSeries});
I guess it's telling that Mike's gist still uses v3, despite being updated long after v4 came out.
To get the layer index you have to use the layer.index attribute directly. So when grouping you would translate the entire layer (which screws up bar-by-bar animations, of course... sigh).
let layers = d3.stack(yourData);
let layer = this.svg.selectAll(".layer")
.data(layers)
layer.transition()
.attr("transform", d => { return "translate(" + d.index*barWidth/numberOfSeries + ",0)"; });

Server side marker clustering PHP & Mongodb

I have implemented a server side marker clustering by following what says in this link
http://www.appelsiini.net/2008/introduction-to-marker-clustering-with-google-maps
This works perfect for markers less than 5,000. But when my markers increased to 17,000
it causes all the memory to exhaust as there are very big loops running.
I am using mongodb for storing all my records with lat n long,
Can i make use of the mongodb's spatial query feature for clustering ?
Some how i want server load to be very less calculating the clusters each time user drags the map,
So far i'm doing the clustering as follows
while (count($markers)) {
$marker = array_pop($markers);
$cluster = array();
/* Compare against all markers which are left. */
foreach ($markers as $key => $target) {
$pixels = $this->pixelDistance($marker['lat'], $marker['long'],
$target['lat'], $target['long'],
$zoom);
if ($distance > $pixels && $zoom < 18) {
unset($markers[$key]);
$cluster[] = $target;
}
if (count($cluster) > 0) {
$cluster[] = $marker;
$clustered[] = $cluster;
} else {
$clustered[] = $marker;
}
}
$newarray = array();
foreach($clustered as $key => $cluster) {
$centroid = array('lat' => 0, 'long' => 0, 'count' => 0);
if(isset($cluster[0]) && is_array($cluster[0])){
foreach($cluster as $marker) {
//echo "{$key} =>"; printArray($marker);
//if($key != 10){
$centroid['lat'] += $marker['lat']; // Sum up the Lats
$centroid['long'] += $marker['long']; // Sum up the Lngs
$centroid['count']++;
//}
}
//if($centroid['count'] != 0){
$centroid['lat'] /= $centroid['count']; // Average Lat
$centroid['long'] /= $centroid['count']; // Average Lng
$clustered[$key] = $centroid; // Overwrite the cluster with the single point.
//}
}
}
return $clustered;
Any help is greatly appreciated.
Thanks in advance

You can use a bounding box to narrow the search. A longitude is 111 km: http://en.m.wikipedia.org/wiki/Longitude. The tile is calculated with 3 variables x,y,z. It uses a 2 dimensional grid with the x and y axis and the zoom level z. Read here:http://msdn.microsoft.com/en-us/library/bb259689.aspx. Basically you need to convert the lat-lng pair to pixel coordinates. Then you can get the tile-number from it. The maximum pixel is a power of 2 number. Hence the big number at maximum zoom level. Because you insist from the bing tiling system:
To optimize the indexing and storage of tiles, the two-dimensional tile XY coordinates are combined into one-dimensional strings called quadtree keys, or “quadkeys” for short. Each quadkey uniquely identifies a single tile at a particular level of detail, and it can be used as an key in common database B-tree indexes. To convert tile coordinates into a quadkey, the bits of the Y and X coordinates are interleaved, and the result is interpreted as a base-4 number (with leading zeros maintained) and converted into a string. For instance, given tile XY coordinates of (3, 5) at level 3, the quadkey is determined as follows:
tileX = 3 = 011 2
tileY = 5 = 101 2
quadkey = 100111 2 = 213 4 = “213”
Quadkeys have several interesting properties. First, the length of a quadkey (the number
of digits) equals the level of detail of the corresponding tile. Second, the quadkey of > any tile starts with the quadkey of its parent tile (the containing tile at the
previous level).
It's very similar to a quadtree or r-tree and it should be a good exercise for the reader but you already have the bing tile code.

Display vertical line on intersection point

I'm trying to represent a Pareto chart with Highcharts, as you can see here.
The horizontal line shows the 80% value, but now I wanted to display a vertical line where that horizontal 80% line intersects with the "Acumulated" chart series.
This is an example of what I'm trying to achieve:
Is there a way to do it?
Another option would be to get the "x" value of the "Acumulated" spline where it's "y" value is "80", that way I could then draw the line manually.
Is this even possible with the Highcharts API?
I know that it's possible to get the values of a point in a series, but that isn't enough in this case:
var point = chart.get('accumulated').data[2];

I have find it for 80:20 calculation.
First I have find the first value in series from Spline data which greater than or equal to 80.
i.e. >= 80
Suppose it is DataX
Then find out the that index in array plus one for DataX.
i.e. DataX location is DataIndex = index+1
(as array start from 0th calculation need plus one)
formula is
DataX : DataIndex :: 80: ?
let the question mark is xIndexOf80
then xIndexOf80 = (DataIndex *80)/(DataX ).
xIndexOf80 is nothing but position of 80 on X axis.
which gives you exact marks on X-Axis
function findInetrSectionPoint(arrSplineData) {
var intLen = arrSplineData.length;
for (var index = 0; index < intLen; index++) {
if (arrSplineData[index] >= 80) {
interSectPoint = ((index + 1) * 80) / arrSplineData[index] - 1;
break;
}
}
return interSectPoint;
}
Here is the Plunker

You can calculate position of 80% point and then use http://api.highcharts.com/highstock#Renderer rect. Apart from that you can also check this option http://api.highcharts.com/highstock#Axis.addPlotLine() / http://api.highcharts.com/highstock#yAxis.plotLines