d3: skipping lines when reading in csv file - javascript

I'm trying to read in a csv file with D3 and I'm a little stuck. The way my csv file is formatted is that the first line is a merged cell containing a year then the next line will contain the data descriptions (name, age etc).
Currently I have the following:
var resourceList = [{description: "All Yearly Data",
name: "yearlyData",
path: "data.csv"};
d3.csv(resourceInfo.path, function(error, d) {
theData.resources[resourceInfo.name].processed = true;
theData.resources[resourceInfo.name].error = error;
theData.resources[resourceInfo.name].data = d;
theData.numProcessed += 1;
});
This reads the first line in as the data descriptions and then the following lines as actual data. What I want to do is have an multidimensional array which I could go through by year. Is it possible to skip lines while parsing to make sure I can manage that or no?
Thanks!

one way of getting at this would be to use filter:
d3.csv(resourceInfo.path, function(error, d) {
var newData = d.filter(function(obs) { return INSERT YOUR FILTER CONDITION HERE;});
...
see also:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter?redirectlocale=en-US&redirectslug=JavaScript%2FReference%2FGlobal_Objects%2FArray%2Ffilter

Related

Adding Numbers from csv

I am trying to add all numbers from an column to an variable. The Problem is my code is adding the String to which results into NaN.
var csvData=[];
let test = 0;
var parser = parse({delimiter: ','}, function(err, data){
});
fs.createReadStream(__dirname+'/test2.csv','utf16le').pipe(parser)
.on('data', function(csvrow) {
csvData.push(csvrow);
test = test + (csvrow[2]);
})
.on('end',function() {
console.log(test)
});
gives me : "0Daily Device Installs00001000101100" and if I add parseInt(csvrow[2]) I will get NaN for test.
My goal is to add all numbers after Daily Device Installs, what am I missing?
I did a bit research on Node.js CSV package.
Use the header
If your CSV file contains a header row as supposed in comment by GrafiCode, like in this example:
"Day","Daily Device Installs"
"2021-09-15",1
"2021-09-16",1
Then CSV Parser has a feature to use the header row with column-names.
See the columns option.
Benefit:
log the header
map the column-names (for simple use in code)
use it to make your code clean & expressive
defend against changes of column-order inside the input CSV
var csvData=[];
let test = 0;
// options: use default delimiter comma and map header
let parser = parse({
columns: header =>
header.map( column => {
console.log(column);
// could also map (e.g. similar to Snake_Case)
return column.replace(/ /g,"_");
})
}
function addToCounter(value) {
if (!isNaN(value))
console.log("WARN: not a number: ", value)
return;
test += value;
}
// read from file
fs.createReadStream(__dirname+'/test2.csv','utf16le').pipe(parser)
.on('data', function(csvrow) {
csvData.push(csvrow);
addToCounter(csvrow.Daily_Device_Installs); // the column name as mapped with underscore
})
.on('end',function() {
console.log(test)
});
Note:
I extracted the counter-increment to a function.
Your csvData array now contains for each row an object (with column-names as keys) instead an array of columns.
try
if (!isNaN(csvrow[2])) test += +csvrow[2];

How can I parse a .frd file with Javascript?

A .frd file is a type of multi-column numeric data table used for storing information about the frequency response of speakers. A .frd file looks something like this when opened in a text editor:
2210.4492 89.1 -157.7
2216.3086 88.99 -157.7
2222.168 88.88 -157.6
2228.0273 88.77 -157.4
Using javascript, is there a way that I can parse this data in order to return each column separately?
For example, from the .frd file above, I would need to return the values like so:
var column1 = [2210.4492, 2216.3086, 2222.168, 2228.0273];
var column2 = [89.1, 88.99, 88.88, 88.77];
var column3 = [-157.7, -157.7, -157.6, -157.4];
I'm not exactly sure where to begin in trying to achieve this, so any step in the right direction would be helpful!
I found the following description of the FRD file format and I will follow it.
Let's assume that the content of your .frd file is in the variable called content (the following example is for Node.js):
const fs = require('fs');
const content = fs.readFileSync('./input.frd').toString();
Now if content has your FRD data, it means it's a set of lines, each line contains exactly three numbers: a frequency (Hz), a level (dB), and a phase (degrees). To split your content into lines, we can just literally split it:
const lines = content.split(/\r?\n/);
(normally, splitting just by '\n' would've worked, but let's explicitly support Windows-style line breaks \r\n just in case. The /\r?\n/ is a regular expression that says "maybe \r, then \n")
To parse each line into three numbers, we can do this:
const values = line.split(/\s+/);
If the file can contain empty lines, it may make sense to double check that the line has exactly three values:
if (values.length !== 3) {
// skip this line
}
Given that we have three values in values, as strings, we can assign the corresponding variables:
const [frequency, level, phase] = values.map(value => Number(value));
(.map converts all the values in values from strings to Number - let's do this to make sure we store the correct type).
Now putting all those pieces together:
const fs = require('fs');
const content = fs.readFileSync('./input.frd').toString();
const frequencies = [];
const levels = [];
const phases = [];
const lines = content.split(/\r?\n/);
for (const line of lines) {
const values = line.split(/\s+/);
if (values.length !== 3) {
continue;
}
const [frequency, level, phase] = values.map(value => Number(value));
frequencies.push(frequency);
levels.push(level);
phases.push(phase);
}
console.log(frequencies);
console.log(levels);
console.log(phases);
The main code (the one that works with content) will also work in browser, not just in Node.js, if you need that.
This code can be written in a tons of different ways, but I tried to make it easier to explain so did something very straightforward.
To use it in Node.js (if your JavaScript file is called index.js):
$ cat input.frd
2210.4492 89.1 -157.7
2216.3086 88.99 -157.7
2222.168 88.88 -157.6
2228.0273 88.77 -157.4
$ node index.js
[ 2210.4492, 2216.3086, 2222.168, 2228.0273 ]
[ 89.1, 88.99, 88.88, 88.77 ]
[ -157.7, -157.7, -157.6, -157.4 ]

Merging two datasets in D3/JS

I am trying to create a simple "plug-n-play" map template, that allows user to put a csv file with geoids and values and then see the values as a choropleth.
Right now I am merging two datasets (map and values) using double loop, but wondering if there is any other option:
This chunk of code stays within the function that loads geodata (fresh_ctss) :
d3.csv("data/communities_pop.csv", function(error, comms)
{
csv = comms.map(function(d)
{
//each d is one line of the csv file represented as a json object
// console.log("Label: " + d.CTLabel)
return {"community": d.community, "population" :d.population,"label": d.tract} ;
})
csv.forEach(function(d, i) {
fresh_ctss.forEach(function(e, j) {
if (d.label === e.properties.geoid) {
e.properties.community = parseInt(d.community)
e.properties.population = parseInt(d.population)
}
})
})
You'll definitely need two loops (or a nested loop) - the most optimal way would be to just limit how much iteration needs to happen. Right now, the first loop goes through every csv row. The following nested loop goes through every csv row (as new different object) and then, as many times as there are rows in the csv, through every item in fresh_ctss.
If you mapped the rows into an object instead of an array, you could iterate through the rows once (total) and then once through the elements of fresh_ctss (again, total). Code below assumes that there are no tract duplicates in comms:
all_comms = {}
comms.forEach(function(d) {
all_comms[d.tract] = {"community": d.community, "population": d.population}
})
fresh_ctss.forEach(function(e) {
comm = all_comms[e.properties.geoid]
e.properties.community = parseInt(comm.community)
e.properties.population = parseInt(comm.population)
}

ScatterChart in NVD3 – Reading the data from csv file

I am trying to read in data from csv file and want to visualise this data with a scatterChart in NVD3.
I would have linked to a JSfiddle or something similar but I don't know how to include a csv file in these online JavaScript IDEs. Is that possible?
The csv file has the following format:
country,y,x
Algeria,91.8,15.7
Bahrain,98.2,49.3
Jordan,99.1,55.0
Kuwait,98.6,57.4
Lebanon,98.7,58.6
My best guess for the code to read the csv file with is:
var scatterdata = [
{
key : "Group1",
values : []//{x:"",y:""}
}
];
d3.csv("literacyScatterCountrynames.csv", function (error, csv) {
if (error) return console.log("there was an error loading the csv: " + error);
console.log("there are " + csv.length + " elements in my csv set");
scatterdata[0].values["x"] = csv.map(function(d){return [+d["x"] ]; });
scatterdata[0].values["y"] = csv.map(function(d){return [+d["y"] ]; });
I see my data in the DOM and it looks about right but the chart is not shown and instead it says 'No Data Available.' in bold letters where the chart should be.
Neither here at StockOverflow, nor in the NVD3 documentation on Github, nor in the helpful website on NVD3 charts by cmaurer on GitHub could I find more information on how to do this.
Turning your csv into JSON would work, but isn't necessary. You've just got your data formatting methods inside-out.
You seem to be expecting an object containing three arrays, one for each column in your table. The D3 methods create (and the NVD3 methods expect) an array of objects, one for each row.
When you do
scatterdata[0].values["y"] = csv.map(function(d){return [+d["y"] ]; });
You're creating named properties of the values array object (all Javascript arrays are also objects), but not actually adding content using array methods, so the length of that array is still zero and NVD3 sees it as an empty array -- and gives you the "no data" warning.
Instead of using the mapping function as you have it, you can use a single mapping function to do number formatting on the data array, and then set the result directly to be your values array.
Like so:
var scatterdata = [
{
key : "Group1",
values : []//{x:"",y:""}
}
];
d3.csv("literacyScatterCountrynames.csv", function (error, csv) {
if (error) return console.log("there was an error loading the csv: " + error);
console.log("there are " + csv.length + " elements in my csv set");
scatterdata[0].values = csv.map(function(d){
d.x = +d.x;
d.y = +d.y;
return d;
});
console.log("there are " + scatterdata[0].values.length + " elements in my data");
//this should now match the previous log statement
/* draw your graph using scatterdata */
}
The mapping function takes all the elements in the csv array -- each one of which represents a row from your csv file -- and passes them to the function, then takes the returned values from the function and creates a new array out of them. The function replaces the string-version of the x and y properties of the passed in object with their numerical version, and then returns the correctly formatted object. The resulting array of formatted objects becomes the values array directly.
Edit
The above method creates a single data series containing all the data points. As discussed in the comments, that can be a problem if you want a category name to show up in the tooltip -- the NVD3 tooltip automatically shows the series name as the tooltip value. Which in the above code, would mean that every point would have the tooltip "Group1". Not terribly informative.
To format the data to get useful tooltips, you need each point as its own data series. The easiest way to make that happen, and have the output in the form NVD3 expects, is with d3.nest. Each "nested" sub-array will only have one data point in it, but that's not a problem for NVD3.
The code to create each point as a separate series would be:
var scatterdata;
//Don't need to initialize nested array, d3.nest will create it.
d3.csv("literacyScatterCountrynames.csv", function (error, csv) {
if (error) return console.log("there was an error loading the csv: " + error);
console.log("there are " + csv.length + " elements in my csv set");
var nestFunction = d3.nest().key(function(d){return d.country;});
//create the function that will nest data by country name
scatterdata = nestFunction.entries(
csv.map(function(d){
d.x = +d.x;
d.y = +d.y;
return d;
})
); //pass the formatted data array into the nest function
console.log("there are " + scatterdata.length + " elements in my data");
//this should still match the previous log statement
//but each element in scatterdatta will be a nested object containing
//one data point
/* draw your graph using scatterdata */
}
You could place the data into a variable, as Mike describes here:
name value
Locke 4
Reyes 8
Ford 15
Jarrah 16
Shephard 23
Kwon 42
is represented this way:
var data = [
{name: "Locke", value: 4},
{name: "Reyes", value: 8},
{name: "Ford", value: 15},
{name: "Jarrah", value: 16},
{name: "Shephard", value: 23},
{name: "Kwon", value: 42}
];

d3 - return only data without header

I have a csv data.
DATA :
Time,Count
1377973800,293
1377975600,212
1377977400,129
1377979200,89
1377981000,54
1377982800,21
1377984600,15
I want to return the data in this format.
{
"946705035":4,
"946706692":4,
"946707210":0,
"946709243":2,
"946710714":5,
"946712907":3,
"946713183":4,
"946719001":0
}
I do not want the header Time and Count to be appeared in the json format.
Tried using d3.nest() but the result I got like it starts with key variable which i don't want.
Someone please help me in getting the data in that format.
I believe a code similar to this would do the job:
d3.csv("data.csv", function(error, data) {
var myObject = {};
for (var i=0; i < data.length; i++)
myObject[data[i].Time] = data[i].Count;
};
This gives you data about counts as strings, and if you want numbers, you can just add a "+", which will trigger conversion from string to number:
myObject[data[i].Time] = +data[i].Count;
EDIT: Here is related question on creating object properties dynamically, maybe you can find something useful there too.

Categories

Resources