Develop Reference

How to apply a code built for arrays to an image collection after using .toArray()?

I’ve found a great code which is built for finding peaks in arrays, which print the result in the console (Finding peaks and troughs in time series data in a 1D array - Javascript) . I would like to apply it on each pixel of an 8-day NDVI image collection. The result should be an image with the peak number in each pixel.
I tried to apply the .toArray() function to the image collection to obtain an array. Then I created a function “findPeaks” that should add a peak number Band to the image. Then I tried to apply this function to the array image using .map(findPeaks), but I get an error “array.map is not a function”
Here you find the code https://code.earthengine.google.com/32e78cc57f87c05a76665ed0e8b6c720.
var aoi =
ee.Geometry.Polygon(
[[[11.111455811313702, 46.3205838600638],
[11.111455811313702, 46.31527834569152],
[11.11800040131004, 46.31527834569152],
[11.11800040131004, 46.3205838600638]]], null, false);
var ndvi_IC=ee.ImageCollection("LANDSAT/LC08/C01/T1_8DAY_NDVI")
.filterDate('2020-04-01','2020-10-01')
.filter(ee.Filter.bounds(aoi))
var array=ndvi_IC.toArray()
print(array)
//code from https://stackoverflow.com/questions/43567335/finding-peaks-and-troughs-in-time-series-data-in-a-1d-array-javascript
//I would like to obtain a new image presenting the number of peaks found for each pixel.
var findPeaks=function(array) {
var start = 1; // Starting index to search
var end = array.length - 2; // Last index to search
var obj = { peaks: [], troughs: [] };// Object to store the indexs of peaks/thoughs
for(var i = start; i<=end; i++)
{
var current = array[i];
var last = array[i-1];
var next = array[i+1];
if(current > next && current > last)
obj.peaks.push(i);
else if(current < next && current < last)
obj.troughs.push(i);
}
var peaksNum=obj.peaks.size()
return array.addBands(peaksNum);
}
var arraywithpeaksBand=array.map(findPeaks)
print(arraywithpeaksBand)
Thank you for your help, I’m new to GEE and coding.
Davide

You cannot use javascript for loops, conditionals or even math on Earth Engine images. All the Earth Engine stuff happens somewhere else on a server, and all the javascript stuff happens in your browser. You have to essentially vectorize the code to work with earth engine operators.
What you want to know is, for each point in time, if it is higher than its previous or next neighbor. Then you count how many times that's true. You can do this with a technique called forward-differencing. See this article and this presentation.
// Count the number of peaks in a 1D array image.
var countPeaks = function(image) {
// Compute the forward and backwards difference.
// Note: these arrays are 1 smaller than the input
// because the first/last pt doesn't have a prev/next neighbor.
var left = image.arraySlice(0, 0, -1)
var right = image.arraySlice(0, 1)
var fwd = left.subtract(right)
var back = right.subtract(left)
// Test if each position is greater than its next/prev neighbor?
var next = fwd.gt(0)
var prev = back.gt(0)
// Test if the first/last point is itself a peak
var first = image.arrayGet([0]).gt(image.arrayGet([1])).toArray()
var last = image.arrayGet([-1]).gt(image.arrayGet([-2])).toArray()
// Reattach the end points.
next = next.arrayCat(last, 0)
prev = first.arrayCat(prev, 0)
// Count how many times both next and prev are greater than 0 (or an end peak)
// and get the result.
var peaks = next.and(prev).arrayReduce(ee.Reducer.sum(), [0]).arrayGet([0])
return peaks
}
var array = ee.Array([113,112,115,120,119,102,101,100,103,105,110,109,105,107])
var image = ee.Image.constant(array)
Map.addLayer(countPeaks(image))
https://code.earthengine.google.com/e3265400b32b3ade1d4df2a7f920d8a5

d3.max() not returning the max value in an array?

I've been teaching myself d3 overnight and can't seem to figure out this one issue I am having. I am dynamically feeding the d3.max() method an array of numbers (I have tried parsing them with parseInt()/parseFloat() as well as not parsing them) through an HTML <select> element. I am using the console to check the values as my visualizations are becoming skewed. However, sometimes it does return the max value, and sometimes it just doesn't.
var localityElectricityConsumption = localities[localityName].totalElectricityConsumption;
maxConsumption = d3.max(localityElectricityConsumption);
r = d3.scale.linear().domain([0, maxConsumption]).range([120, 0]);
console.log("array being fed to d3.max: " + localityElectricityConsumption);
console.log("what d3.max() is returning: " + d3.max(localityElectricityConsumption));
console.log("what parseInt(d3.max()) is returning: " + d3.max(localityElectricityConsumption));
You can see the live visualization and check the console here. Try using the top-most graph to switch from the North American locality to the Mexico locality. d3.max() returns the max value for North America, but not Mexico, and therefore doesn't graph the data correctly for Mexico. I am officially stumped. Any help would be appreciated.
EDIT:
Here is the code that is used to create localities[localityName].totalElectricityConsumption:
d3.csv("data/" + dataset, function (data) {
for (var i = 0; i < data.length; i++) {
var record = data[i];
if (dataset === "total_electricity_consumption.csv") {
currentDataset = listOfDatasets[0].name;
record.totalElectricityConsumption = [];
// loop through all years, from 1980 to 2012
for (var year = 1980; year <= 2012; year++) {
var value = record[year];
// deal with missing data points
if (value === '--') {
value = 0;
}
else if (value === 'NA') {
value = 0;
}
else if (value === '(s)') {
value = 0;
}
else if (value === 'W') {
value = 0;
}
// add record of current total electricity consumption
record.totalElectricityConsumption.push(value);
}
}
// create a list of country names
localities[record.Locality] = record;
listOfLocalities.push(record.Locality);
}
}
Sorry, formatting is not perfect as I had to take some stuff out to make the code more readable for Stack Overflow.

As expected, d3.max() is indeed returning the maximum value... however, it is returning the maximum value among strings, not among numbers.
The explanation is that, by default, d3.csv() will load all the values as strings, even if they are numbers. Therefore, when you do:
var value = record[year];
value is a string, not a number. And in JavaScript the maximum for an array of strings is different from the maximum for an array of numbers. For instance, using numbers:
var myArray = [1, 65, 9, 32, 42];
console.log(d3.max(myArray));
<script src="https://d3js.org/d3.v4.min.js"></script>
That's the expected value. But look what happens if we use strings:
var myArray = ["1", "65", "9", "32", "42"];
console.log(d3.max(myArray));
<script src="https://d3js.org/d3.v4.min.js"></script>
Solution: change your values to numbers:
var value = +record[year];

Javascript sorting array twice (different sort methods)

I have an array that at page load sorts the array how I want, depending on the current date it shows the objects that have a future date and then shows the objects with past dates, lets call this datesToPlay.
I have two radio buttons to call on the method to sort the same array, and at page load it sorts the array perfectly fine how I mentioned above. The problem is when I sort the array with the other sorting method, createdAt which simply sorts the array by the date of creation. This method sorts it fine, but when I press the other radio to sort by datesToPlay it doesn't sorts the array.
function SortByPlayTime(a, b){
var currentDate = new Date();
if(lastDateOfObjectsInArray > currentDate){
if(a.attributes.start >= currentDate){
if(a.attributes.start > b.attributes.start && b.attributes.start >= currentDate)
return 1;
else
return -1;
} else if(b.attributes.start >= currentDate){
if(b.attributes.start > a.attributes.start)
return -1;
else
return 1;
}
else{
if(a.attributes.start > b.attributes.start)
return 1;
else
return -1;
}
} else{
if(a.attributes.start > b.attributes.start)
return -1;
else
return 1;
}
function SortByCreation(a, b){
if(a.attributes.createdAt > b.attributes.createdAt)
return 1;
else
return -1;
Basically what Im doing is I have an array with all the objects im trying to sort, this array varies in size and can be like 1000 or more objects.
In function loadArrayToShowFilters() what Im doing is to prepare a new array that will be shown in a table (see comments above for screens). This is cause Im mimicking a table but actually Im doing all the work with arrays; this array will always be 100 length or less.
function loadArrayToShowFilters() {
//array = the array of objects Im trying to filter
var sizeArrayToShow;
if(array.length < 100)
sizeArrayToShow = array.length;
else
sizeArrayToShow = 100;
arrayTableToShow = [];
//arrayTableToShow = This is the array that holds the objects that are shown on the table
for (var i = 0; i < sizeArrayToShow; i++) {
arrayTableToShow[i] = array[i];
};
Events that trigger the sorts: event click on two radio buttons.
Code that performs the actual sort: on the event click of each radio button, I just do array.sort(SortByCreation) and like so respectively.
Sample data: {"start":{"__type":"Date","iso":"2018-02-01T11:00:00.000Z"},"end":{"__type":"Date","iso":"2018-02-01T12:00:00.000Z"},"createdAt":"2018-01-29T20:37:51.477Z","updatedAt":"2018-02-23T03:12:15.968Z","objectId":"dSVZXFAIyf"}
It's basically just an array with objects, each object with a variable attributes with includes variables start and createdAt which are used to do the sorting
Events:
'click #lastCreated'(event){
orderArregloCreatedAt();
},
'click #nextToPlay'(event){
orderArregloPlayTime();
}
function orderArregloCreatedAt() {
array.sort(SortByCreation);
loadArrayToShowFilters();
}
function orderArregloPlayTime() {
array.sort(SortByPlayTime);
loadArrayToShowFilters();
}

Information needed to resolve this issue is still sorely lacking from this question. But I think the fault lies in the sorting algorithm of SortByPlayTime.
This is a bit of a wild shot, but please can you tell me if the following change delivers:
expected sort order at page load?
expected sort order after clicking the radio button?
the same results for both, even if it's wrong?
Code change:
function SortByPlayTime(a, b) {
var currentDate = new Date();
// a is future, b is past, so a shows above b = -1
if (a.attributes.start >= currentDate && b.attributes.start < currentDate) return -1;
// a is past, b is future, so a shows below b = 1
if (a.attributes.start < currentDate && b.attributes.start >= currentDate) return 1;
// both a and b are similarly future or past so compare on their values alone, in ascending order
// if a > b then it should show below b, = positive result
return a.attributes.start - b.attributes.start;
}

How to sort elements of array in natural order with mixed (letters and numbers) elements

i am trying to create google-form which is used to register students agreements on practice. Every agreement is registered and got agreement number which format is Last to digits of current year-T-number of agreement at this year/M. For example for now it is 17-T-11/M. The number of agreement currently is written by person which is responsible for practice.
Here is code of script below:
function onChange(e)
{
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheets()[1];
var range = sheet.getDataRange();
var values = range.getValues();
var comboValues = ['16-T-105/M'];
// removing titles from 0 column and 1 line (titles)
for (var i = 1; i <= values.length; i++) {
var v = values[i] && values[i][0];
v && comboValues.push(v)
}
// Sort the values
comboValues.sort(
function(a, b) {
if (b.toLowerCase() < a.toLowerCase()) return -1;
if (b.toLowerCase() > a.toLowerCase()) return 1;
return 0;
}
);
Logger.log(comboValues);
// google-form modification
var form = FormApp.openById('1SHgVIosoE34m9cny9EQySljvgnRpzffdFEZe-kzNOzA');
var items = form.getItems();
for (i = 4; i < items.length; i++) {
Logger.log("ID: " + items[i].getId(), ': ' + items[i].getType());
}
form.getItemById('2087613006').asListItem().setChoiceValues(comboValues);
I got issue which is related with lexicographical order. Person which register agreement choose from list last registered agreement number: i tryed to do that last registered agreement number will always be at list top. As time when i started this everything was fine (it started with number 16-T-105/M), but new year come and soon after 17-T-10/M agreement was registered i got issue, that 17-T-10/M was not on list top. Soon i realised that this happens because script use lexicographical order and "thinks" that 2 is more than 10. So i understood that i somehow will have to change that order and do that 2 is less than 10, 11 is less than 101 and so on.
My question is how to do that? I guess that i need to sort array elements in natural order - but i do not have idea how to do this.
I tryed to google how to do it , but result was not satisfactory - maybe my knowledge of coding is pretty limited (i am PhD student of Psychology, not Informatics) :)
Maybe someone will help how to solve that problem.
Updates:
Link to spreadsheet: https://docs.google.com/spreadsheets/d/1FH5qYTrLUNI2SCrcaqlwgu8lzAylaTkZsiALg0zIpCM/edit#gid=1620956794
Link to google-form (Copy of actual form): https://docs.google.com/forms/d/e/1FAIpQLSerJfkv1dgHexUwxppXNyhb46twOZgvEMOIVXSOJoED3SLmyQ/viewform

You should adjust the sorting method to account of the peculiarities of the data. Here is one way to do this: the function splitConvert processes each string, splitting it by non-word characters and then converting what can be converted to integers (and lowercasing the rest). Then the comparison goes through this array one by one.
comboValues.sort(
function(a, b) {
var as = splitConvert(a);
var bs = splitConvert(b);
for (var i = 0; i < as.length; i++) {
if (bs[i] < as[i]) return -1;
if (bs[i] > as[i]) return 1;
}
return 0;
}
);
function splitConvert(str) {
return str.split(/\W/).map(function(part) {
var x = parseInt(part, 10);
return isNaN(x) ? part.toLowerCase() : x;
});
}
This is not the most performance-oriented solution: the split-parse function will be repeatedly called on the same strings as they are being sorted. If this becomes an issue (I don't really think so), one can optimize by having one run of conversion, creating an array of arrays, and then sorting that.

Is there a way to tell crossfilter to treat elements of array as separate records instead of treating whole array as single key?

I have data set where some of the field values are arrays and I'd like to use crossfilter and d3.js or dc.js to display histogram of how many times each of those values was present in the dataset.
Here's an example:
var data = [
{"key":"KEY-1","tags":["tag1", "tag2"]},
{"key":"KEY-2","tags":["tag2"]},
{"key":"KEY-3","tags":["tag3", "tag1"]}];
var cf = crossfilter(data);
var tags = cf.dimension(function(d){ return d.tags;});
var tagsGroup = tags.group();
dc.rowChart("#chart")
.renderLabel(true)
.dimension(tags)
.group(tagsGroup)
.xAxis().ticks(3);
dc.renderAll();
And JSFiddle http://jsfiddle.net/uhXf5/2/
When I run that code it produces graph like this:
But what I want is something like this:
To make things even more complicated it would be awesome to be able to click on any of the rows and filter dataset by the tag that was clicked.
Anyone has any ideas how to achieve that?
Thanks,
Kostya

Solved it myself, here's fiddle with working code http://jsfiddle.net/uhXf5/6/
Here's code in case someone will came across similar problem:
function reduceAdd(p, v) {
v.tags.forEach (function(val, idx) {
p[val] = (p[val] || 0) + 1; //increment counts
});
return p;
}
function reduceRemove(p, v) {
v.tags.forEach (function(val, idx) {
p[val] = (p[val] || 0) - 1; //decrement counts
});
return p;
}
function reduceInitial() {
return {};
}
var data = [
{"key":"KEY-1","tags":["tag1", "tag2"], "date":new Date("10/02/2012")},
{"key":"KEY-2","tags":["tag2"], "date": new Date("10/05/2012")},
{"key":"KEY-3","tags":["tag3", "tag1"], "date":new Date("10/08/2012")}];
var cf = crossfilter(data);
var tags = cf.dimension(function(d){ return d.tags;});
var tagsGroup = tags.groupAll().reduce(reduceAdd, reduceRemove, reduceInitial).value();
// hack to make dc.js charts work
tagsGroup.all = function() {
var newObject = [];
for (var key in this) {
if (this.hasOwnProperty(key) && key != "all") {
newObject.push({
key: key,
value: this[key]
});
}
}
return newObject;
}
var dates = cf.dimension(function(d){ return d.date;});
var datesGroup = dates.group();
var chart = dc.rowChart("#chart");
chart
.renderLabel(true)
.dimension(tags)
.group(tagsGroup)
.filterHandler(function(dimension, filter){
dimension.filter(function(d) {return chart.filter() != null ? d.indexOf(chart.filter()) >= 0 : true;}); // perform filtering
return filter; // return the actual filter value
})
.xAxis().ticks(3);
var chart2 = dc.barChart("#chart2");
chart2
.width(500)
.transitionDuration(800)
.margins({top: 10, right: 50, bottom: 30, left: 40})
.dimension(dates)
.group(datesGroup)
.elasticY(true)
.elasticX(true)
.round(d3.time.day.round)
.x(d3.time.scale())
.xUnits(d3.time.days)
.centerBar(true)
.renderHorizontalGridLines(true)
.brushOn(true);
dc.renderAll();

The example above is a great approach. You can take it one step further though.
In the solution above, it will only filter based on the first selection you make. Any subsequent selections are ignored.
If you want it to respond to all selections, you would create a filterHandler as follows:
barChart.filterHandler (function (dimension, filters) {
dimension.filter(null);
if (filters.length === 0)
dimension.filter(null);
else
dimension.filterFunction(function (d) {
for (var i=0; i < d.length; i++) {
if (filters.indexOf(d[i]) >= 0) return true;
}
return false;
});
return filters;
}
);
Working sample here:
http://jsfiddle.net/jeffsteinmetz/cwShL/

I'd like to try to provide some context for the approach listed by Jeff and Kostya.
You'll notice that the tagsGroup uses groupAll unlike the typical group method. Crossfilter tells us that "The returned object is similar to a standard grouping, except it has no top or order methods. Instead, use value to retrieve the reduce value for all matching records." Kostya called the ".value()" method to retrieve the single object that represents the entire group.
var tagsGroup = tags.groupAll().reduce(reduceAdd, reduceRemove, reduceInitial).value();
This object won't work well with dc.js because dc.js expects the group object to have an all method. Kostya patched that object to have an "all" method like so:
// hack to make dc.js charts work
tagsGroup.all = function() {
var newObject = [];
for (var key in this) {
if (this.hasOwnProperty(key) && key != "all") {
newObject.push({
key: key,
value: this[key]
});
}
}
return newObject;
}
This will work with a simple dc.js chart, but you won't be able to use all dc.js functionality since not all of the group functions are present. For example, you won't be able to use the "cap" method on your chart because the cap method expects the group object to have a "top" method. You could also patch the top method like so:
topicsGroup.top = function(count) {
var newObject = this.all();
newObject.sort(function(a, b){return b.value - a.value});
return newObject.slice(0, count);
};
This will enable your chart to use the cap method:
barChart
.renderLabel(true)
.height(200)
.dimension(topicsDim)
.group(topicsGroup)
.cap(2)
.ordering(function(d){return -d.value;})
.xAxis().ticks(3);
An updated example is available at http://jsfiddle.net/djmartin_umich/m7V89/#base

Jeff's answer does work, but there is no need to keep track of the "found" variable or continue the loop if an item was found. If X is in [X,Y,Z], this has already cut the amount of iterations in 1/3.
else
dimension.filterFunction(function (d) {
for (var i=0; i < d.length; i++) {
if (filters.indexOf(d[i]) >= 0) return true;
}
return false;
});
Alternatively, you could patch dc.js filterFunction method and that would handle all cases.

This is much easier now, since crossfilter and dc support dimensions with arrays. See this question for context and example: Using dimensions with arrays in dc.js/crossfilter