Develop Reference

Can the built-in heatmap layer in Google Maps API be optimized for animation?

Context:
The built-in heatmap layer in Google Maps API doesn't have native support for animations, so I've created my own animations using requestAnimationFrame with calls to the Maps API to set the next frame's data. Each frame uses ~2500 data points to render the heatmap and maxes out around 4000 data points.
The problem:
My animation has jank every 5-10 cycles. I don't need the animation to render 60fps, but I do want the framerate to be consistent and each step to be <=100ms. In Devtools, the janky animation frames give me this error at the top of call stack: "Warning: Recurring handler took # ms."
My proposed cause of the problem:
The dev console shows memory leaks,* minor GC, major GC, and DOM GC correlating to Maps API calls. The DOM GC in particular is correlated to the animation jank. (*JS Heap, Nodes, and Listeners increase linearly).
Why I'm asking the question:
I'm fairly new to Google Maps API, requestAnimationFrame, and memory management in JavaScript, so first of all I want to make sure I'm understanding the problem correctly. Secondly, I want to see if there is a way to optimize the heatmap layer for animations, or if my approach is misguided.
Most relevant resources I've consulted:
Google Maps API documentation
Google's guide to web performance
StackOverflow question that asks about very basic, short Maps API heatmap animations
Code
All within a d3.csv('data').
Establish recursive requestAnimationFrame:
const periodDuration = 100; // ms
const timeStart = performance.now();
const frameTick = timestamp => {
if (timestamp - timeStart >= periodDuration) {
timeStart = timestamp;
transitionToNextPeriod(); // shown in next code snippet
}
requestAnimationFrame(frameTick);
};
requestAnimationFrame(frameTick);
transitionToNextPeriod increments time, then filters data, and ends by setting heatmap with updated data:
function transitionToNextPeriod() {
let batchedData = [],
currPeriodData = [],
addMin = 0;
prevMin = min,
prevDate = date,
currMin = 0;
// batch the next 'x' # of mins for one batchedData call after `while` loop
while (++addMin <= 7){
if (prevMin + addMin === 60){
if (hour + 1 === 24){
hour = 0;
date++;
} else {
hour++
}
}
// Now with current minute, filter data and add to batchedData
currMin = ((prevMin + addMin) % 60);
let newTime = `${hour}:${zeroPad(currMin)}`;
// Filter based on current time
let currFilter = dayData.filter( d => d.StartTime.indexOf( ` ${newTime}` ) !== -1);
prevPeriodData = prevPeriodData.filter( d => d.EndTime.indexOf( ` ${newTime}` ) === -1 );
batchedData = [...batchedData, ...currFilter ];
} // End while
batchedData = [...batchedData, ...prevPeriodData];
prevPeriodData = batchedData;
min = currMin;
// Update the maps.LatLng data with new data, then set heatmap data
batchedData.forEach( (d,i) => {
// Check for last datum in current period
if (i + 1 === batchedData.length){
heatmap.setData(currPeriodData); // PROBLEM: about every 7th call to setData results in jank
} else {
currPeriodData.push(new google.maps.LatLng( d.Lat , d.Long ));
}
})
} // End transitionToNextPeriod()

on which parameters depend requestAnimationFrame? -- slowness

because of my amount of data, I try to display them in few times, by smallest amount, thanks to requestAnimationFrame.
I'm new with this method and have some issue with it.
It works well for small database, with less than 1000 entries, that is smooth. But when I try it with bigger databases, the loop isn't smooth anymore.
I don't understand this slowness because normally render does the same thing, regardless of the size of "data".
function paths(data, ctx, count) {
var n = data.length,
i = 0,
reset = false;
var lastRun=0;
var fps;
function render() {
var max = d3.min([i+60, n]);
data.slice(i,max).forEach(function(d) {
d3.select(".foreground")
.append("path")
.attr("d", function(p){ return path(d);
})
.attr("stroke", "steelblue");
});
i = max;
console.log("end render");
};
(function animloop(){
console.log("animloop");
if (i >= n || count < brush_count) return;
lastRun = new Date().getTime();
requestAnimationFrame(animloop);
render();
})();
};
// Returns the path for a given data point.
function path(d) {
return line(dimensions.map(function(p) {
return [position(p), y[p](d[p])]; }));
}
I tried to see where the slowness comes from, thanks to console.log(), but actually the lapse is after render. On the console are printed blocks of "end render - animloop" / a lapse / "end render - animloop". I don't understand this lapse...
When I try to use the debugger step by step, I can't see any difference between the cases "few data" and "big data".
If someone sees a problem in my code or knows the origin of the problem, I'll be very grateful.
PS: speedness now : 50fps for 500 entries, 15fps for 7,000, 5fps for 20,000 (I don't need 60fps but 5 is really not enough).

If you don’t need your animation’s frame rate to be equal to display’s frame rate (or cannot provide acceptable performance), then consider skipping some frames (to prevent performing time-consuming computations at each frame) based on the time passed to the requestAnimationFrame()’s callback in the DOMHighResTimeStamp format.

Efficiently detect missing dates in array and inject a null (highcharts and jquery)

I'm using highcharts.js to visualize data series from a database. There's lots of data series and they can potantially change from the database they are collected from with ajax. I can't guarantee that they are flawless and sometimes they will have blank gaps in the dates, which is a problem. Highcharts simply draws a line through the entire gap to the next available date, and that's bad in my case.
The series exists in different resolutions. Hours, Days and Weeks. Meaning that a couple of hours, days or weeks can be missing. A chart will only show 1 resolution at a time on draw, and redraw if the resolution is changed.
The 'acutal' question is how to get highcharts to not draw those gaps in an efficient way that works for hous, days and weeks
I know highcharts (line type) can have that behaviour where it doesn't draw a single line over a gap if the gap begins with a null.
What I tried to do is use the resolution (noted as 0, 1, 2 for hour day or week), to loop through the array that contains the values for and detect is "this date + 1 != (what this date + 1 should be)
The code where I need to work this out is here. Filled with psudo
for (var k in data.values) {
//help start, psudo code.
if(object-after-k != k + resolution){ //The date after "this date" is not as expected
data.values.push(null after k)
}
//help end
HC_datamap.push({ //this is what I use to fill the highchart later, so not important
x: Date.parse(k),
y: data.values[k]
});
}
the k objects in data.values look like this
2015-05-19T00:00:00
2015-05-20T00:00:00
2015-05-21T00:00:00
...and more dates
as strings. They can number in thousands, and I don't want the user to have to wait forever. So performance is an issue and I'm not an expert here either
Please ask away for clarifications.

I wrote this loop.
In my case my data is always keyed to a date (12am) and it moves either in intervals of 1 day, 1 week or 1 month. Its designed to work on an already prepared array of points ({x,y}). Thats what dataPoints is, these are mapped to finalDataPoints which also gets the nulls. finalDataPoints is what is ultimately used as the series data. This is using momentjs, forwardUnit is the interval (d, w, or M).
It assumes that the data points are already ordered from earliest x to foremost x.
dataPoints.forEach(function (point, index) {
var plotDate = moment(point.x);
finalDataPoints.push(point);
var nextPoint = dataPoints[index+1];
if (!nextPoint) {
return;
}
var nextDate = moment(nextPoint.x);
while (plotDate.add(1, forwardUnit).isBefore(nextDate)) {
finalDataPoints.push({x: plotDate.toDate(), y: null});
}
});

Personally, object with property names as dates may be a bit problematic, I think. Instead I would create an array of data. Then simple loop to fill gaps shouldn't be very slow. Example: http://jsfiddle.net/4mxtvotv/ (note: I'm changing format to array, as suggested).
var origData = {
"2015-05-19T00:00:00": 20,
"2015-05-20T00:00:00": 30,
"2015-05-21T00:00:00": 50,
"2015-06-21T00:00:00": 50,
"2015-06-22T00:00:00": 50
};
// let's change to array format
var data = (function () {
var d = [];
for (var k in origData) {
d.push([k, origData[k]]);
}
return d;
})();
var interval = 'Date'; //or Hour or Month or Year etc.
function fillData(data, interval) {
var d = [],
now = new Date(data[0][0]), // first x-point
len = data.length,
last = new Date(data[len - 1][0]), // last x-point
iterator = 0,
y;
while (now <= last) { // loop over all items
y = null;
if (now.getTime() == new Date(data[iterator][0]).getTime()) { //compare times
y = data[iterator][1]; // get y-value
iterator++; // jump to next date in the data
}
d.push([now.getTime(), y]); // set point
now["set" + interval](now.getDate() + 1); // jump to the next period
}
return d;
}
var chart = new Highcharts.StockChart({
chart: {
renderTo: 'container'
},
series: [{
data: fillData(data, interval)
}]
});
Second note: I'm using Date.setDay() or Date.setMonth(), of course if your data is UTC-based, then should be: now["setUTC" + interval].

Show gap of missing data with Highstock

Using Highstock to chart a sorted time serie: [[timestamp, value], ...]
The datasource is sampled at irregular intervals. As result the distances between two points (in the time axis) varies.
If two adjacent points are separated for more than 5 minutes I want to show a gap in the chart.
Using the gapSize option doesn't work, because it doesn't allows to specify the 'size' of the gap as a function of time.
Showing gaps is already a part of Highstock, I just need a way to specify it as a fixed amount of time (5 minutes). Ideas?
Btw, beside that the plot works great.

Here's a slightly unclean way to "manipulate" gapSize to work so that it's value is the amount of milliseconds required to create a gap.
(function (H) {
// Wrap getSegments to change gapSize functionality to work based on time (milliseconds)
H.wrap(H.Series.prototype, 'getSegments', function (proceed) {
var cPR = this.xAxis.closestPointRange;
this.xAxis.closestPointRange = 1;
proceed.apply(this, Array.prototype.slice.call(arguments, 1));
this.xAxis.closestPointRange = cPR;
});
}(Highcharts));
This utilizes that gapSize is only used within the getSegments function (see source), and it works based on the closestPointRange of the axis. It wraps the getSegments, sets closestPointRange to 1, calls the original method and then resets closestPointRange to its original value.
With the code above you could do gaps for 5 minutes like this:
plotOptions: {
line: {
gapSize: 300000 // 5 minutes in milliseconds
}
}
See this JSFiddle demonstration of how it may work.

Halvor Strand function wrapper did not work for me as long as getSegments is not part of highstock source code anymore to calculate that gap. Anyway, you can find an approximation to solve the problem combining this other topic and the previows answer like this:
(function(H) {
H.wrap(H.Series.prototype, 'gappedPath', function(proceed) {
var gapSize = this.options.gapSize,
xAxis = this.xAxis,
points = this.points.slice(),
i = points.length - 1;
if (gapSize && i > 0) { // #5008
while (i--) {
if (points[i + 1].x - points[i].x > gapSize) { // gapSize redefinition to be the real threshold instead of using this.closestPointRange * gapSize
points.splice( // insert after this one
i + 1,
0, {
isNull: true
}
);
}
}
}
return this.getGraphPath(points);
});
}(Highcharts))
setting gapSize in plotOptions to the desired size (in ms) like Halvor said:
plotOptions: {
line: {
gapSize: 300000 // 5 minutes in milliseconds
}
}

In case anyone comes across this and is spending hours trying to figure out why gapSize is not working like me. Make sure your time series data is sorted, only then will the gaps appear in the graph.
Another issue I ran into was my data series was in this format
[
{x: 1643967900000, y: 72},
{x: 1643967600000, y: 72},
{x: 1643967300000, y: 72}
]
However this does not seem to work with gapSize and needs to be in the format below
[
[1643967900000, 72],
[1643967600000, 91],
[1643967300000, 241]
]

how can i crunch numbers in a non-blocking way?

I have a lineCollection of models that represent lines of a graph. Each line crunches through a dataCollection and generates averages once. For one line, its very snappy.
After the dataCollection is fetched, I have a listener to crunch the data for each line.
lineCollection.invoke('setData')
However, this kind of freezes the browser. My next approach was to try to set a timeout to hopefully stop blocking the ui:
lineCollection.each(function(model) {
setTimeout(model.setData, Math.round(Math.random() * 20));
});
This still sorta freezes the whole process. How can I tell the browser to crunch the data in the background?
Edit
"crunch the data" in my case is to zip variable sets of ~3600 length arrays, average each slice, and format ~3600 date objects.
// x axis
var xs = _.map(times, this.formatTime);
// y axis
var values = _.map(allSamples, function(samples) {
return _.pluck(samples, 'value');
});
var avgs = _.map(_.zip.apply(_, values), _.avg);
var ys = _.compact(avgs);
// graph data
var data = { x: xs.slice(xs.length - ys.length), y: ys };

I would definitely look into worker threads.