Develop Reference

How to count prediction (forecasting) in highchart only with data behind the last plotline?

I have a code for standard area-spline highchart where I made a function for prediction based on previous data. Prediction / forecasting is showing next 4 values in a trendline where the first prediction point is counting with real data, second point with real data plus first prediction point and the next two points about the same plus previous predictions. Otherwise it would be only a line with same values. But my data are increasing so the forecasting must be about the same.
Data in highchart are connected to Microsoft SQL Server.
Prediction is based on this code, just with few changes:
Forecast formula from Excel in Javascript
The highchart is now showing increasing data (values of resistance) as a decimal in yAxis and datetime in xAxis. Prediction is working as well but the thing is that not all data are relevant.There is always a point which is very different (lower than previous) and thats the place from where I need to count new prediction. The plotline is generated in the last high value, then starting the new (low) - and that is from where I need to count.
So here is what I do have:
-the function 'average' is counting average from values
-second function 'forecast' as you can expect is counting the forecasting (based on the code from link above)
-third function 'test' is already putting all together
-the 'results' (2,3,4) are the counted points of prediction
-'vlastnidata' are the data from mssql
-'datumek' is for the date
Now the condition for plotline (as you can see there) is that the previous point in chart must be 20 higher and then the plotline is generated - this is also working, just need to find a way how to count only with new data behind the plotline.
And here you can reach the connection function to mssql in php - if needed
Is there a way how to dynamically create a plotline in highchart when the value is lower than previous one?
As I said everything is working, plotlines and the prediction. But to see clear prediction I need to count only with relevant data.
Hope everything is clear. Thank you in advance for any recommendations.
function average(ar)
{
var r=0;
for (i=0;i<ar.length;i++)
{
r = r+ar[i];
}
return r/ar.length;
}
function forecast(x, ky, kx)
{
var i=0, nr=0, dr=0,ax=0,ay=0,a=0,b=0, result=0;
ax=average(kx);
ay=average(ky);
for (i=0;i<kx.length;i++)
{
nr = nr + ((kx[i]-ax) * (ky[i]-ay));
dr = dr + ((kx[i]-ax)*(kx[i]-ax))
}
b=nr/dr;
a=ay-b*ax;
result = (a+b*x);
return result;
}
function test(container,nazev,rtop,vlastnidata,colorSeries)
{
var result = 0, result2 = 0, result3 = 0, result4 = 0, datumek=[],hodnoty=[];
for (a=0;a<vlastnidata.length;a++)
{
datumek[a]=vlastnidata[a][0];
hodnoty[a]=vlastnidata[a][1];
}
kalkulace=(datumek[vlastnidata.length-1]-datumek[vlastnidata.length-2]);
hodnoty_nove=hodnoty;
datumek_nove=datumek;
result = forecast((datumek[vlastnidata.length-1]+kalkulace), hodnoty, datumek);
hodnoty_nove[hodnoty_nove.length]=result;
datumek_nove[datumek_nove.length]=(datumek[vlastnidata.length-1]+kalkulace);
result2 = forecast((datumek[vlastnidata.length-1]+2*kalkulace), hodnoty_nove, datumek_nove);
hodnoty_nove[hodnoty_nove.length]=result2;
datumek_nove[datumek_nove.length]=(datumek[vlastnidata.length-1]+2*kalkulace);
result3 = forecast((datumek[vlastnidata.length-1]+3*kalkulace), hodnoty_nove, datumek_nove);
hodnoty_nove[hodnoty_nove.length]=result3;
datumek_nove[datumek_nove.length]=(datumek[vlastnidata.length-1]+3*kalkulace);
result4 = forecast((datumek[vlastnidata.length-1]+4*kalkulace), hodnoty_nove, datumek_nove);
Highcharts.chart(container, {chart: {type: 'areaspline',
events: {
load:function(){
let points = this.series[0].points;
let plotLines = [];
console.log(this)
let previousPoint = points[0];
points.forEach(function(point) {
if(point.y < previousPoint.y/100*80) {
plotLines.push({
value: previousPoint.x,
color: 'red',
width: 3
});
}
previousPoint = point;
});
this.xAxis[0].update({
plotLines: plotLines
})
}
}
},
title: {text: 'Average of resistance per month, '+rtop},
legend: {layout: 'vertical',
align: 'left',
verticalAlign: 'top',
x: 150,
y: 100,
floating: true,
borderWidth: 1,
backgroundColor: (Highcharts.theme && Highcharts.theme.legendBackgroundColor) || '#FFFFFF'},
xAxis: { type: 'datetime'},
yAxis: {title: {text: 'Resistance: in ohms'}},
tooltip: {shared: true,valueSuffix: ' ohms',valueDecimals: 2},
credits: {enabled: false},
plotOptions: {areaspline: {fillOpacity: 0.5}},
series: [{ name: nazev,
color: colorSeries,
data: vlastnidata},
{name: 'Prediction',
color: '#001a33',
data: [[(datumek[vlastnidata.length-1]+kalkulace),result], [(datumek[vlastnidata.length-1]+2*kalkulace),result2], [(datumek[vlastnidata.length-1]+3*kalkulace),result3], [(datumek[vlastnidata.length-1]+4*kalkulace),result4]]
}]
});
}

Chart.js - consolidating days to month totals along the x (time) axis?

I'm feeding two daily statistics datasets into my chart. As you can see, each element represents the value for a particular day.
"data":[[{"y":"1", "x":"2018-04-01T04:00:00Z"},
{"y":"14", "x":"2018-04-02T04:00:00Z"},
{"y":"5", "x":"2018-04-03T04:00:00Z"},
{"y":"7", "x":"2018-04-04T04:00:00Z"},
...
The x axis is defined as follows:
xAxes: [{
type: 'time',
distribution: 'series',
time: {
unit: 'month'
}
}]
I (naively?) thought that the chart would be rolling up (summing) the day values into the appropriate month buckets but that's not what I got. Instead, I got monthly tick marks along the x-axis but data points are plotted within the chart at daily precision. (See screenshot.)
Before I go ahead and reprocess my dataset to manually roll up days into their respective month buckets, I'd like to hear whether the chart can in fact do this for me but I'm just setting this up wrong, or whether I do in fact need to take care of this summarization myself, before supplying the dataset to the chart for plotting.
Thanks for your advice!

I solved this by doing the rollup myself during the assembly of the underlying dataset which is then supplied to the chart.
var dayDate = new Date($scope.insights.locationMetrics[lm].metricValues[metric].dimensionalValues[dim].timeDimension.timeRange.startTime);
var monthDate = dayDate.getFullYear() + "-" + (dayDate.getMonth() + 1);
var hitCount = {
y: $scope.safeNumber($scope.insights.locationMetrics[lm].metricValues[metric].dimensionalValues[dim].value),
x: monthDate
}
var alreadyRecorded = hits[labelIdx].findIndex(obj => obj.x == hitCount.x)
if (alreadyRecorded > -1) {
hits[labelIdx][alreadyRecorded].y += Number(hitCount.y);
}
else {
hits[labelIdx].push(hitCount);
}
Extract the date from the underlying data source
Extract yyyy-mm from the date
Create the hitCount object
Check if the hitCount object is already in the array
If the object is already in the array then increment the hitCount (y) within the array.
Otherwise, push the object into the array.

Reduce daily data to monthly using Google Earth Engine

I am looking at precipitation data (both GPM and CHIRPS) for different provinces in Indonesia using Google Earth Engine. GPM is sub-daily (every 30 minutes) and CHIRPS is daily. I am only interested in getting the monthly values. Unlike here and here I am not interested in getting the multi-annual monthly values, but simply the average of each month and make a time series.
Here I found a way to create a list of values containing the mean of each month.
Edit: Thanks to Nicholas Clinton's answer I managed to get it to work:
var fc = ee.FeatureCollection('ft:1J2EbxO3zzCLggEYc57Q4mzItFFaaPCAHqe1CBA4u') // Containing multiple polygons
.filter(ee.Filter.eq('name', 'bangka')); // Here I select my ROI
Map.addLayer(fc, {}, 'area');
Map.centerObject(fc, 7);
var aggregate_array = fc.aggregate_array('name');
print('Name of area: ', aggregate_array, 'Selected data in FeatureCollection:', fc);
var month_mean = ee.List.sequence(0, 16*12).map(function(n) { // .sequence: number of years from starting year to present
var start = ee.Date('2002-01-01').advance(n, 'month'); // Starting date
var end = start.advance(1, 'month'); // Step by each iteration
return ee.ImageCollection("UCSB-CHG/CHIRPS/DAILY")
.filterDate(start, end)
.mean()
.set('system:time_start', start.millis());
});
print(month_mean);
var collection = ee.ImageCollection(month_mean);
print(collection);
// Plotting
var area_name = fc.aggregate_array('name').getInfo();
var title = 'CHIRPS [mm/hr] for ' + area_name;
var TimeSeries = ui.Chart.image.seriesByRegion({
imageCollection: collection,
regions: fc,
reducer: ee.Reducer.mean(),
scale: 5000,
xProperty: 'system:time_start',
seriesProperty: 'label'
}).setChartType('ScatterChart')
.setOptions({
title: title,
vAxis: {title: '[mm/hr]'},
lineWidth: 1,
pointSize: 1,
});
print('TimeSeries of selected area:', TimeSeries);

Have not tested, but should be something like this (or set some other date property):
return ee.ImageCollection("UCSB-CHG/CHIRPS/DAILY")
.filterDate(start, end)
.sum()
.set('system:time_start', start.millis());

aggregate_prob function in pkg_trend, works just like aggregate in R language.
var imgcol_all = ee.ImageCollection('NASA/GPM_L3/IMERG_V05');
function add_date(img){
var date = ee.Date(img.get('system:time_start'));
var date_daily = date.format('YYYY-MM-dd');
return img.set('date_daily', date_daily);
}
var startdate = ee.Date.fromYMD(2014,3,1);
var enddate = ee.Date.fromYMD(2014,4,1);
var imgcol = imgcol_all
.filter(ee.Filter.date(startdate,enddate)).select('precipitationCal')
.map(add_date);
// imgcol = pkg_trend.imgcol_addSeasonProb(imgcol);
print(imgcol.limit(3), imgcol.size());
var pkg_trend = require('users/kongdd/public:Math/pkg_trend.js');
var imgcol_daily = pkg_trend.aggregate_prop(imgcol, "date_daily", 'sum');
print(imgcol_daily);
Map.addLayer(imgcol_daily, {}, 'precp daily');
The GEE link is https://code.earthengine.google.com/2e04ad4a4bee6789af23bfac42f63025

Update Chart JS with date range selector

I am trying to update a chart depending on the dates and and shifts selected using ajax. My ajax call returns an array like this:
0
date "2017-11-20"
shift "Day Shift"
availability 100
1
date "2017-11-21"
shift "Day Shift"
availability 63.63636363636363
2
date "2017-11-22"
shift "Day Shift"
availability 63.63636363636363
3
date "2017-11-23"
shift "Day Shift"
availability 63.63636363636363
4
date "2017-11-24"
shift "Day Shift"
availability 14.285714285714285
5
date "2017-11-20"
shift "Night Shift"
availability 67.56756756756756
6
date "2017-11-21"
shift "Night Shift"
availability 67.56756756756756
7
date "2017-11-22"
shift "Night Shift"
availability 67.56756756756756
8
date "2017-11-23"
shift "Night Shift"
availability 67.56756756756756
my javascript looks like this:
// on change event
var request;
$('input').on('change', function(event) {
console.log('changed');
event.preventDefault();
// Abort any pending request
if (request) {
request.abort();
}
var rangeStart = moment($('#daterange').data('daterangepicker').startDate).unix();
var rangeEnd = moment($('#daterange').data('daterangepicker').endDate).unix();
var shift = 'all';
request = $.ajax({
url: "report_availability.php",
type: "post",
data: {
rangeStart: rangeStart,
rangeEnd: rangeEnd,
shift: shift
}
});
request.done(function (response, textStatus){
drawChart(response);
});
request.fail(function (textStatus, errorThrown){
// Log the error to the console
console.error(
"The following error occurred: "+
textStatus, errorThrown
);
});
request.always(function () {
console.log('request finished');
});
});
function drawChart(data) {
var dates = [];
var shift1Score = [];
var shift2Score = [];
for(var i in data) {
var found = jQuery.inArray(data[i].date, dates);
if (found < 0) {
dates.push(data[i].date);
}
if(data[i].shift == 'Day Shift' ) {
shift1Score.push(data[i].availability);
} else {
shift2Score.push(data[i].availability);
}
}
// Destroy the chart if it already exists
// NOT WORKING
if(myChart!=null){
myChart.destroy();
console.log('destroy');
}
var ctx = document.getElementById("myChart").getContext('2d');
ctx.canvas.height = 50;
var myChart = new Chart(ctx, {
type: 'bar',
data: {
labels: dates,
datasets: [
{
label: "Day Shift",
backgroundColor: "#3e95cd",
data: shift1Score
}, {
label: "Night Shift",
backgroundColor: "#8e5ea2",
data: shift2Score
}
]
},
options: {
scales: {
yAxes: [{
ticks: {
beginAtZero:true
}
}]
},
responsive: true,
maintainAspectRatio: false
}
});
}
I have 2 problems with this:
problem 1: The chart is not being destroyed so when I redraw it, it just redraws on top of the old chart which causes problems with the hover events. I have tried to use chart.update() to resolve this issue however this seems to just add to the original data instead of replacing it.
EDIT:- I have solved problem 1 by removing the canvas and then creating a new canvas:-
$('#myChart').remove();
$('#chartBar').append('<canvas id="myChart"></canvas>');
Problem 2: There is more than 2 shifts.. sometime upto 5 and I do not want to hard code all of these. I would like to draw each dataset depending on how many shifts are returned in the array, I am open to changing the array structure in php or javascript but just cannot seem to figure out the correct array structrue or how to build a dynamic dataset array for the chart.
the chart should output like this for 2 shifts:
Any help would be great thanks

I had the same problem that you have.
In order to update the graph i am using two functions:
Remove all datasets in current graph:
function removeDataset(chart) {
chart.data.datasets = [];
};
Add a new dataset to the graph:
function addDataset(chart, name, data, background, border, fill) {
var newDataset = {
label: name,
data: [],
backgroundColor: background,
borderColor: border,
fill: fill
};
for (var index = 0; index < data.length; ++index) {
newDataset.data.push(data[index]);
}
chart.data.datasets.push(newDataset);
};
So, when i have the ajax done, i use like this:
removeDataset(ctx1);
addDataset(ctx1, "Dataset Name", resp.dataset_data_json, "#007bff", "#007bff", true);
ctx1.update();
To declare the graph, i use:
var ctx1;
$(document).ready(function () {
var canvas1 = $("#canvas1")[0];
canvas1.height = "300";
ctx1 = new Chart(canvas1, config_ctx1);
});
I hope it helps.

Efficiently detect missing dates in array and inject a null (highcharts and jquery)

I'm using highcharts.js to visualize data series from a database. There's lots of data series and they can potantially change from the database they are collected from with ajax. I can't guarantee that they are flawless and sometimes they will have blank gaps in the dates, which is a problem. Highcharts simply draws a line through the entire gap to the next available date, and that's bad in my case.
The series exists in different resolutions. Hours, Days and Weeks. Meaning that a couple of hours, days or weeks can be missing. A chart will only show 1 resolution at a time on draw, and redraw if the resolution is changed.
The 'acutal' question is how to get highcharts to not draw those gaps in an efficient way that works for hous, days and weeks
I know highcharts (line type) can have that behaviour where it doesn't draw a single line over a gap if the gap begins with a null.
What I tried to do is use the resolution (noted as 0, 1, 2 for hour day or week), to loop through the array that contains the values for and detect is "this date + 1 != (what this date + 1 should be)
The code where I need to work this out is here. Filled with psudo
for (var k in data.values) {
//help start, psudo code.
if(object-after-k != k + resolution){ //The date after "this date" is not as expected
data.values.push(null after k)
}
//help end
HC_datamap.push({ //this is what I use to fill the highchart later, so not important
x: Date.parse(k),
y: data.values[k]
});
}
the k objects in data.values look like this
2015-05-19T00:00:00
2015-05-20T00:00:00
2015-05-21T00:00:00
...and more dates
as strings. They can number in thousands, and I don't want the user to have to wait forever. So performance is an issue and I'm not an expert here either
Please ask away for clarifications.

I wrote this loop.
In my case my data is always keyed to a date (12am) and it moves either in intervals of 1 day, 1 week or 1 month. Its designed to work on an already prepared array of points ({x,y}). Thats what dataPoints is, these are mapped to finalDataPoints which also gets the nulls. finalDataPoints is what is ultimately used as the series data. This is using momentjs, forwardUnit is the interval (d, w, or M).
It assumes that the data points are already ordered from earliest x to foremost x.
dataPoints.forEach(function (point, index) {
var plotDate = moment(point.x);
finalDataPoints.push(point);
var nextPoint = dataPoints[index+1];
if (!nextPoint) {
return;
}
var nextDate = moment(nextPoint.x);
while (plotDate.add(1, forwardUnit).isBefore(nextDate)) {
finalDataPoints.push({x: plotDate.toDate(), y: null});
}
});

Personally, object with property names as dates may be a bit problematic, I think. Instead I would create an array of data. Then simple loop to fill gaps shouldn't be very slow. Example: http://jsfiddle.net/4mxtvotv/ (note: I'm changing format to array, as suggested).
var origData = {
"2015-05-19T00:00:00": 20,
"2015-05-20T00:00:00": 30,
"2015-05-21T00:00:00": 50,
"2015-06-21T00:00:00": 50,
"2015-06-22T00:00:00": 50
};
// let's change to array format
var data = (function () {
var d = [];
for (var k in origData) {
d.push([k, origData[k]]);
}
return d;
})();
var interval = 'Date'; //or Hour or Month or Year etc.
function fillData(data, interval) {
var d = [],
now = new Date(data[0][0]), // first x-point
len = data.length,
last = new Date(data[len - 1][0]), // last x-point
iterator = 0,
y;
while (now <= last) { // loop over all items
y = null;
if (now.getTime() == new Date(data[iterator][0]).getTime()) { //compare times
y = data[iterator][1]; // get y-value
iterator++; // jump to next date in the data
}
d.push([now.getTime(), y]); // set point
now["set" + interval](now.getDate() + 1); // jump to the next period
}
return d;
}
var chart = new Highcharts.StockChart({
chart: {
renderTo: 'container'
},
series: [{
data: fillData(data, interval)
}]
});
Second note: I'm using Date.setDay() or Date.setMonth(), of course if your data is UTC-based, then should be: now["setUTC" + interval].