Merging two datasets in D3/JS - javascript

I am trying to create a simple "plug-n-play" map template, that allows user to put a csv file with geoids and values and then see the values as a choropleth.
Right now I am merging two datasets (map and values) using double loop, but wondering if there is any other option:
This chunk of code stays within the function that loads geodata (fresh_ctss) :
d3.csv("data/communities_pop.csv", function(error, comms)
{
csv = comms.map(function(d)
{
//each d is one line of the csv file represented as a json object
// console.log("Label: " + d.CTLabel)
return {"community": d.community, "population" :d.population,"label": d.tract} ;
})
csv.forEach(function(d, i) {
fresh_ctss.forEach(function(e, j) {
if (d.label === e.properties.geoid) {
e.properties.community = parseInt(d.community)
e.properties.population = parseInt(d.population)
}
})
})

You'll definitely need two loops (or a nested loop) - the most optimal way would be to just limit how much iteration needs to happen. Right now, the first loop goes through every csv row. The following nested loop goes through every csv row (as new different object) and then, as many times as there are rows in the csv, through every item in fresh_ctss.
If you mapped the rows into an object instead of an array, you could iterate through the rows once (total) and then once through the elements of fresh_ctss (again, total). Code below assumes that there are no tract duplicates in comms:
all_comms = {}
comms.forEach(function(d) {
all_comms[d.tract] = {"community": d.community, "population": d.population}
})
fresh_ctss.forEach(function(e) {
comm = all_comms[e.properties.geoid]
e.properties.community = parseInt(comm.community)
e.properties.population = parseInt(comm.population)
}

Related

How to Extract data based on the values in one array after matching the corresponding values from another array in JavaScript?

This is the URL from GeoServer to get feature info
{"type":"FeatureCollection","features":[{"type":"Feature","id":"weather_warning_day_1.fid--418ec0da_178b69d5dfc_-715c","geometry":null,"properties":{"issue_date":"2021-04-09","updated_at":"2021-04-09T09:26:33+05:30","utc_time":0,"state_name":"Odisha","state_id":21,"district_name":"MAYURBHANJ","district_id":232,"api_district_name":"MAYURBHANJ","day_1":"6,9,10","day1_color":3}}],"totalFeatures":"unknown","numberReturned":1,"timeStamp":"2021-04-09T15:38:19.536Z","crs":null}
the data I want to extract is of variable: "day_1":"6,9,10"
which I got from the layer and stored it in the variable as
var warning_day_1 = weather_warning_layer_data.features[0].properties.day_1
so basically the input is "day_1":"6,9,10"
which I have stored in the array as
[{"warning":"6"},{"warning":"9"},{"warning":"10"}]
and corresponding output should be Dust Storm, Heat Wave, Hot Day
Dust Storm, Heat Wave, Hot Day
or if the input was "day_1":"2,5"
then output should have been Heavy Rain, Hailstorm
or if the input was "day_1":"1"
then output should have been No Warning
After reading the data of the string and creating its array, I have to compare it with another array and extract the key values (display) corresponding to the key values (warning) in the 1st array.
var warning_data_split = warning_day_1.split(/[ ,]+/);
var warning_data_from_api_array = new Array;
warning_data_from_api_array.push(warning_data_split);
for (var i = 0; i < warning_data_from_api_array.length; i++) {
var item_in_array_to_compare = warning_data_from_api_array[i];
if(warning_data_from_api_array[item_in_array_to_compare.warning_data_from_api_array])
{warning_data_from_api_array[item_in_array_to_compare.warning_data_from_api_array].push(item_in_array_to_compare);}
else {
warning_data_from_api_array[item_in_array_to_compare.warning_data_from_api_array] = [item_in_array_to_compare];}}
let final_array_to_compare = item_in_array_to_compare
final_array_to_compare = final_array_to_compare.map(x => ({warning: x}));
/// this is the first array ////////////
The values in this array are not static in length, as it keeps on changing like, sometimes the array has value [1] or [1,2], [2,5,8], [4,7,12], etc
so I have to extract the corresponding values of display from the lookup array given below
var warning_code_meaning_list = [
{ warning:"1", display:"No Warning"},
{ warning:"2", display:"Heavy Rain"},
{ warning:"3", display:"Heavy Snow"},
{ warning:"4", display:"Thunderstorm & Lightning, Squall etc"},
{ warning:"5", display:"Hailstorm"},
{ warning:"6", display:"Dust Storm"},
{ warning:"7", display:"Dust Raising Winds"},
{ warning:"8", display:"Strong Surface Winds"},
{ warning:"9", display:"Heat Wave"},
{ warning:"10", display:"Hot Day"},
{ warning:"11", display:"Warm Night"},
{ warning:"12", display:"Cold Wave"},
{ warning:"13", display:"Cold Day"},
{ warning:"14", display:"Ground Frost"},
{ warning:"15", display:"Fog"}
]
The data which I am getting in warning_day_1 (in the very first line of the code) is a string (this couldn’t be saved as float/integer in the database column because sometimes there are more than 1 warning for a specific place, so I have stored this as a text in the database)
Which I’m converting to an array after reading it from the API
Now this string, which I am fetching from API has variable data,
Some time single digit like: 1
Sometime multiple : 1,2,3
And each of the integer present in this array corresponds to the specific text shown in the next array like if the warning is 2 it means the heavy rainfall,
but if the string (later converted to an array, with “warning” as a key) has 2,5 as value, it means: heavy rainfall & Hailstorm
I want that the values which come up in array 1 (the dynamic one) got match with the 2nd array ( a sort of lookup array) and fetch its display value as output.
How to do so?
You could use an object to map your warnings to messages.
Try this:
const data = {"type":"FeatureCollection","features":[{"type":"Feature","id":"weather_warning_day_1.fid--418ec0da_178b69d5dfc_-715c","geometry":null,"properties":{"issue_date":"2021-04-09","updated_at":"2021-04-09T09:26:33+05:30","utc_time":0,"state_name":"Odisha","state_id":21,"district_name":"MAYURBHANJ","district_id":232,"api_district_name":"MAYURBHANJ","day_1":"6,9,10","day1_color":3}}],"totalFeatures":"unknown","numberReturned":1,"timeStamp":"2021-04-09T15:38:19.536Z","crs":null}
var warning_code_meaning_list = {
"1":"No Warning",
"2":"Heavy Rain",
"3":"Heavy Snow",
"4":"Thunderstorm & Lightning, Squall etc",
"5":"Hailstorm",
"6":"Dust Storm",
"7":"Dust Raising Winds",
"8":"Strong Surface Winds",
"9":"Heat Wave",
"10":"Hot Day",
"11":"Warm Night",
"12":"Cold Wave",
"13":"Cold Day",
"14":"Ground Frost",
"15":"Fog",
};
results = data["features"].map(feature => {
return feature.properties.day_1.split(',').map(code => {
return warning_code_meaning_list[code];
});
});
That gives you an array of arrays of the displays:
[ [ 'Dust Storm', 'Heat Wave', 'Hot Day' ] ]

How to compare a 2-D array and 1-D Array and to Store common Data in Another Array in Java Script

I have 2 Arrays and one is 2 dimensional and another is 1 dimensional. I need to compare both and need to store there common data in another array. I tried the below approach:-
tw.local.listtodisplayNW = new tw.object.listOf.listtodisplayNWBO();
//if(tw.local.SQLResults[0].rows.listLength >
// tw.local.virtualServers.listLength)
var k=0;
for (var i=0;i<tw.local.SQLResults[0].rows.listLength;i++)
{
log.info("Inside SQLResults loop - For RuntimeID: "
+tw.local.SQLResults[0].rows[i].data[3]);
for(var j=0;j<tw.local.virtualServers.listLength;j++)
{
log.info("Inside API loop - For RuntimeID: "
+tw.local.virtualServers[j].runtimeid);
if(tw.local.SQLResults[0].rows[i].data[3] ==
tw.local.virtualServers[j].runtimeid)
{
tw.local.listtodisplayNW[k] = new tw.object.listtodisplayNWBO();
tw.local.listtodisplayNW[k].vsysName =
tw.local.virtualServers[j].virtualSystemName;
tw.local.listtodisplayNW[k].vsysID =
tw.local.virtualServers[j].virtualSystemId;
tw.local.listtodisplayNW[k].serverName =
tw.local.virtualServers[j].serverName;
tw.local.listtodisplayNW[k].serverID =
tw.local.virtualServers[j].serverId;
tw.local.listtodisplayNW[k].runtimeID =
tw.local.virtualServers[j].runtimeid;
//tw.local.listtodisplayNW[k].IPAddress =
tw.local.virtualServers[j].nics[j].ipAddress;
log.info("VsysName:
"+tw.local.listtodisplayNW[k].vsysName+"RuntimeID:
"+tw.local.listtodisplayNW[k].runtimeID);
//tw.local.listtodisplayNW[k] = new
tw.object.listtodisplayNWBO();
tw.local.listtodisplayNW[k].currentSpeed =
tw.local.SQLResults[0].rows[i].data[5];
log.info("VsysName:
"+tw.local.listtodisplayNW[k].vsysName+"RuntimeID:
"+tw.local.listtodisplayNW[k].runtimeID+"CurrentSpeed:
"+tw.local.listtodisplayNW[k].currentSpeed);
if(tw.local.listtodisplayNW[k].currentSpeed != "100 Mbps")
{
tw.local.listtodisplayNW[k].desiredSpeed = "100 Mbps";
}
else
{
tw.local.listtodisplayNW[k].desiredSpeed = "1 Gbps";
}
log.info("DesiredSpeed:
"+tw.local.listtodisplayNW[k].desiredSpeed);
k++;
}
}
log.info("Length of
listtodisplayNW"+tw.local.listtodisplayNW.listLength);
}
In above code SQLResults is a 2-d array and virtualServers is a 1-D array.
I need to compare both these array and common data need to be store in another array. Here performance is not good. Is there any other way to do this efficiently. Please make a needful favour and Thanks in advance.
Assuming integer data, the following example works on the theme of array implementation of set intersection, which will take care of performance.
Convert 2D array to 1D.
var 2DtoIDArray = 2DArray.join().split(",");
Create an array named marker whose purpose is to serve as a lookup that element.
This needs to be done as follows.
Iterate through the smaller array, say 1DArray and keep setting marker as follows throughout iteration.
marker[1DArray[counter]]='S1';
Now iterate through 2Dto1DArray array(you may use nested loop iteration if you dont want to convert it to 1 dimesnional) and for each element
of this array check if its marked as 'S1' in the marker lookup array.
If yes, keep adding the elements in the commonElementsArray.
Follow this simple approach
Since the matching condition is only one between the two large arrays, create two maps (one for each array) to map each record against that attribute which is to be matched
For SQLResults
var map1 = {};
tw.local.SQLResults[0].rows.each( function(row){
map1[ row.data[3] ] = row;
});
and similarly for virtual servers
var map2 = {};
tw.local.virtualServers.each( function(vs){
map2[ vs.runtimeid ] = vs;
});
Now iterate these two maps wrt to their keys and set the values in new array
new array being tw.local.listtodisplayNW
tw.local.listtodisplayNW = [];
Object.keys( map1 ).forEach( function( key ){
if( map2[ key ] )
{
//set the values in tw.local.listtodisplayNW
}
})
Complexity of the approach is simply O(n) since there is no nested loops.

d3: skipping lines when reading in csv file

I'm trying to read in a csv file with D3 and I'm a little stuck. The way my csv file is formatted is that the first line is a merged cell containing a year then the next line will contain the data descriptions (name, age etc).
Currently I have the following:
var resourceList = [{description: "All Yearly Data",
name: "yearlyData",
path: "data.csv"};
d3.csv(resourceInfo.path, function(error, d) {
theData.resources[resourceInfo.name].processed = true;
theData.resources[resourceInfo.name].error = error;
theData.resources[resourceInfo.name].data = d;
theData.numProcessed += 1;
});
This reads the first line in as the data descriptions and then the following lines as actual data. What I want to do is have an multidimensional array which I could go through by year. Is it possible to skip lines while parsing to make sure I can manage that or no?
Thanks!
one way of getting at this would be to use filter:
d3.csv(resourceInfo.path, function(error, d) {
var newData = d.filter(function(obs) { return INSERT YOUR FILTER CONDITION HERE;});
...
see also:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter?redirectlocale=en-US&redirectslug=JavaScript%2FReference%2FGlobal_Objects%2FArray%2Ffilter

Recursively (or iteratively) make a nested html table with d3.js?

I have an array of nested JSON structures where they have varying depth and not the same set of keys everywhere:
[
{
"name":"bob",
"salary":10000,
"friends":[
{
"name": "sarah",
"salary":10000
},
{
"name": "bill",
"salary":5000
}
]
},
{
"name":"marge",
"salary":10000,
"friends":[
{
"name": "rhonda",
"salary":10000
},
{
"name": "mike",
"salary":5000,
"hobbies":[
{
"name":"surfing",
"frequency":10
},
{
"name":"surfing",
"frequency":15
}
]
}
]
},
{
"name":"joe",
"salary":10000,
"friends":[
{
"name": "harry",
"salary":10000
},
{
"name": "sally",
"salary":5000
}
]
}
]
I wanted to use D3 to render this as nested html tables. For example the friends column will have tables showing the name, and salary of the friends of the individual referenced in the row. Sometimes one of these tables will have another level of a sub table.
I imagine the way to do this is by recursively creating tables. I wrote a python program that takes a JSON structure like this, and renders tables within tables, and the easiest way to do that was recursively. I see on the d3.js documentation there is a .each() thing you can call, which I am sure is what I need, I just need a little boost getting there (https://github.com/mbostock/d3/wiki/Selections#wiki-each).
So is there a nice way to do this in D3? I found this great example for rendering a 2d matrix of data as a table Creating a table linked to a csv file. With that tutorial I was able to get the outer most level of this data-structure rendered as a table, but I am stuck on how to go into levels recursively as needed, as of now they just show up as "Object" in the table since I am not treating them differently from normal strings and numbers.
Also I found this other question/answer that is similar to my question, but I really don't understand javascript well enough to see where/how the recursion is happening and readapt the solution to fit my needs: How do I process data that is nested multiple levels in D3?. Any advice or pointers to tutorials on recursively or iteratively processing nested tree like JSON data-structures in D3 would be much appreciated!
A recursive function would probably be good approach. See code below for one possible implementation (assuming your data is stored in jdata). See the comments in the code for some explanation and see this Gist for a live version: http://bl.ocks.org/4085017
d3.select("body").selectAll("table")
.data([jdata])
.enter().append("table")
.call(recurse);
function recurse(sel) {
// sel is a d3.selection of one or more empty tables
sel.each(function(d) {
// d is an array of objects
var colnames,
tds,
table = d3.select(this);
// obtain column names by gathering unique key names in all 1st level objects
// following method emulates a set by using the keys of a d3.map()
colnames = d // array of objects
.reduce(function(p,c) { return p.concat(d3.keys(c)); }, []) // array with all keynames
.reduce(function(p,c) { return (p.set(c,0), p); }, d3.map()) // map with unique keynames as keys
.keys(); // array with unique keynames (arb. order)
// colnames array is in arbitrary order
// sort colnames here if required
// create header row using standard 1D data join and enter()
table.append("thead").append("tr").selectAll("th")
.data(colnames)
.enter().append("th")
.text(function(d) { return d; });
// create the table cells by using nested 2D data join and enter()
// see also http://bost.ocks.org/mike/nest/
tds = table.append("tbody").selectAll("tr")
.data(d) // each row gets one object
.enter().append("tr").selectAll("td")
.data(function(d) { // each cell gets one value
return colnames.map(function(k) { // for each colname (i.e. key) find the corresponding value
return d[k] || ""; // use empty string if key doesn't exist for that object
});
})
.enter().append("td");
// cell contents depends on the data bound to the cell
// fill with text if data is not an Array
tds.filter(function(d) { return !(d instanceof Array); })
.text(function(d) { return d; });
// fill with a new table if data is an Array
tds.filter(function(d) { return (d instanceof Array); })
.append("table")
.call(recurse);
});
}

How to collect all the VELatLong objects iterating through a list of polygons?

this question should be quite simple, but I haven't been able to figure it out.
I want to collect all the VELatLong objects iterating through a list of polygons in JavaScript in order to use the SetMapView() method.
So far I have been able to do with just 2 polygons and my code looks like this:
var points = [];
// Getting the points for first polygon
map.AddShape(shapeOne);
points = shape.GetPoints();
// Getting the points for second polygon and Concatenating "points" with "pointsTwo".
map.AddShape(shapeTwo);
pointsTwo = shape.GetPoints();
points.concat(pointsTwo);
map.SetMapView(points);
But I would like help to how I can do the same thing iterating through a list of polygons?
My iteration code works fine, it looks like this:
function btnPolygons_Click()
{
$.post
(
"/Search/GetPolygons",
null,
function (items) {
$.each
(
items,
function (i, polygonItem) {
var wktShape = polygonItem.PolygonWKT
// Create a VEShape from the WKT representation
var shape = VirtualEarthWKT.ShapeFromWKT(wktShape);
// Add VEShape to Map
map.AddShape(shape);
}
);
},
"json"
);
}
Can you tell me what to add to my iteration code in order to collect all the VELatLong objects iterating through the list of polygons?
This solved my problem:
// Getting the points of the first iteration.
if (i == 0) {
points = shape.GetPoints();
}
// Concatenating the points of the first iteration to the following iterations.
else {
pointsTwo = shape.GetPoints();
points = points.concat(pointsTwo);
}
// Setting the map view in the last iteration.
if (i == items.length - 1) {
map.SetMapView(points);
}

Categories

Resources