I am trying to sort sets of associated key value pairs. They look like this:
{"word":"a","votes":9326,"userMade":"FALSE","limiter":"FALSE"},
But organized into labeled subsets of preferably a string or perhaps an index if necessary.
The data-set is a vote-per-use table of most used english words being parsed into pages.
I will be appending them as text to other html elements due to the constraints my use case, makes it a bit tricky, however, for an example I could work with a simple console.log of the page value followed by the console.log of every word value stored within that page. I need the order preserved. so probably indexed. I will also need to be able to sort each page by the votes value, but I think I can figure the rest out for that.
I have found tutorials on how to search through key-value pairs, but I cannot find how to do all of the following with one solution:
A: access the value of word
B: maintain the order of the data-set, allowing me to append them to the matching html element
C: allows me the opportunity to change which set of elements I am appending to when i have finished looping through a single member of the parent index (the one recording the page)
I imagine it is some combination of for/of and for/in, but I'm getting a headache. Please help?
addl info:
function would run at app startup or when the dataset being examined is changed.
function would take a large dataset filled with around 200 page number values, each with 60+ sets of data like the one listed above, the contents of a single page index for example:
{"word":"a","votes":9326,"userMade":"FALSE","limiter":"FALSE"},
{"word":"aaron","votes":4129,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abandoned","votes":1289,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abc","votes":5449,"userMade":"FALSE","limiter":"FALSE"},
{"word":"aberdeen","votes":641,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abilities","votes":2210,"userMade":"FALSE","limiter":"FALSE"},
{"word":"ability","votes":7838,"userMade":"FALSE","limiter":"FALSE"},
{"word":"able","votes":8649,"userMade":"FALSE","limiter":"FALSE"},
{"word":"aboriginal","votes":1837,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abortion","votes":3232,"userMade":"FALSE","limiter":"FALSE"},
{"word":"about","votes":9295,"userMade":"FALSE","limiter":"FALSE"},
{"word":"above","votes":8818,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abraham","votes":867,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abroad","votes":4969,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abs","votes":2415,"userMade":"FALSE","limiter":"FALSE"},
{"word":"absence","votes":4934,"userMade":"FALSE","limiter":"FALSE"},
{"word":"absent","votes":2937,"userMade":"FALSE","limiter":"FALSE"},
{"word":"absolute","votes":5251,"userMade":"FALSE","limiter":"FALSE"},
{"word":"absolutely","votes":5936,"userMade":"FALSE","limiter":"FALSE"},
{"word":"absorption","votes":285,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abstract","votes":7946,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abstracts","votes":1907,"userMade":"FALSE","limiter":"FALSE"},
{"word":"abuse","votes":7238,"userMade":"FALSE","limiter":"FALSE"},
{"word":"academic","votes":7917,"userMade":"FALSE","limiter":"FALSE"},
{"word":"academics","votes":1706,"userMade":"FALSE","limiter":"FALSE"},
{"word":"academy","votes":6755,"userMade":"FALSE","limiter":"FALSE"},
{"word":"acc","votes":6469,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accent","votes":1020,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accept","votes":7547,"userMade":"FALSE","limiter":"FALSE"},
{"word":"acceptable","votes":4907,"userMade":"FALSE","limiter":"FALSE"},
{"word":"acceptance","votes":7273,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accepted","votes":7684,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accepting","votes":1789,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accepts","votes":1535,"userMade":"FALSE","limiter":"FALSE"},
{"word":"access","votes":9031,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accessed","votes":2932,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accessibility","votes":5702,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accessible","votes":5662,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accessing","votes":2096,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accessories","votes":8875,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accessory","votes":5661,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accident","votes":5664,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accidents","votes":2991,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accommodate","votes":1807,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accommodation","votes":8059,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accommodations","votes":3885,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accompanied","votes":2532,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accompanying","votes":664,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accomplish","votes":1070,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accomplished","votes":2419,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accordance","votes":6434,"userMade":"FALSE","limiter":"FALSE"},
{"word":"according","votes":8282,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accordingly","votes":3003,"userMade":"FALSE","limiter":"FALSE"},
{"word":"account","votes":8996,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accountability","votes":3029,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accounting","votes":7459,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accounts","votes":7507,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accreditation","votes":1605,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accredited","votes":3027,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accuracy","votes":6779,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accurate","votes":6427,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accurately","votes":1493,"userMade":"FALSE","limiter":"FALSE"},
{"word":"accused","votes":2853,"userMade":"FALSE","limiter":"FALSE"},
{"word":"acdbentity","votes":1389,"userMade":"FALSE","limiter":"FALSE"},
and the output would ultimately append the value paired with each word to a specific button through iteration, but also sorted by the page value.
each page is a set of buttons in a 3d object that looks like this:
the text is appended to each button which in turn is a 3d object embeded in an html object using aframe. I can make the appending code.
You can use Object.entries() to get the key value pairs of an object.
var words = [
{"word":"a","votes":9326,"userMade":"FALSE","limiter":"FALSE"},
{"word":"aaron","votes":4129,"userMade":"FALSE","limiter":"FALSE"}
];
words.forEach((wordEntry) => {
var keyValuePairs = Object.entries(wordEntry);
keyValuePairs.forEach((kv) => {
console.log(`key: ${kv[0]} value: ${kv[1]}`);
});
});
my latest attempt looks like this:
for (let p=1; p<129; p++){
for (let b=1; b<68; b++){
let pTpl = (p).toLocaleString(undefined, {minimumIntegerDigits: 3});
let bDbl = (b).toLocaleString(undefined, {minimumIntegerDigits: 2});
var `#fCont${pTpl}${bDbl}` = document.createElement('a-text');
`fCont${pTpl}${bDbl}`.setAttribute('value', 'engWordLib[p,b,0]');
`fCont${pTpl}${bDbl}`.setAttribute('votes', 'engWordLib[p,b,1]');
`fCont${pTpl}${bDbl}`.setAttribute('userMade', 'engWordLib[p,b,2]');
`fCont${pTpl}${bDbl}`.setAttribute('limiter', 'engWordLib[p,b,3]');
`fCont${pTpl}${bDbl}`.setAttribute('visible', 'false');
`fBtn${bDbl}`.appendChild(`#fCont${pTpl}${bDbl}`)
}
}
please note that I havent checked this for errors. I still think this code is to WET and I would prefer the key names for the properties be preserved in the datastructure rather than tacked on when it's appended to the page. I guess I could add a dimension to the array.... seems kind of messy when an object property value has the key value pairs right in it. cant get the iteration of objects in an array down though.... Will continue to persue a cleaner method.
I'm looking for some advice on how to effectively use large amounts of data with d3.js. Lets say for instance, I have this data set taken from a raw .csv file (converted from excel);
EA
,Jan_2016,Feb_2016,Mar_2016
Netherlands,11.7999,15.0526,13.2411
Belgium,25.7713,24.1374
France,27.6033,23.6186,20.2142
EB
,Jan_2016,Feb_2016,Mar_2016
Netherlands,1.9024,2.9456,4.0728
Belgium,-,6.5699,7.8894
France,5.3284,4.8213,1.471
EC
,Jan_2016,Feb_2016,Mar_2016
Netherlands,3.1499,3.1139,3.3284
Belgium,3.0781,4.8349,5.1596
France,16.3458,12.6975,11.6196
Using csv I guess the best way to represent this data would be something like;
Org,Country,Month,Score
EA,Netherlands,Jan,11.7999
EA,Belgium,Jan,27.6033
EA,France,Jan,20.2142
EA,Netherlands,Feb,15.0526
EA,Belgium,Feb,25.9374
EA,France,Feb,23.6186
EA,Netherlands,Mar,13.2411
EA,Belgium,Mar,24.1374
EA,France,Mar,20.2142
This seems very long winded to me, and would use up a lot of time. I was wondering if there was an easier way to do this?
From what I can think of, I assume that JSON may be the more logical choice?
And for context of what kind of chart this data would go into, I would be looking to create a pie chart which can update the data depending on the country/month selected and comparing the three organisations scores each time.
(plnk to visualise)
http://plnkr.co/edit/P3loEGu4jMRpsvTOgCMM?p=preview
Thanks for any advice, I'm a bit lost here.
I would say the intermediary step you propose is a good one for keeping everything organized in memory. You don't have to go through a csv file though, you can just load your original csv file and turn it into an array of objects. Here is a parser:
d3.text("data.csv", function(error, dataTxt) { //import data file as text first
var dataCsv=d3.csv.parseRows(dataTxt); //parseRows gives a 2D array
var group=""; // the current group header ("organization")
var times=[]; //the current month headers
var data=[]; //the final data object, will be filled up progressively
for (var i=0;i<dataCsv.length;i++) {
if (dataCsv[i].length==1 ) { //group name
if ( dataCsv[i][0] == "")
i++; //remove empty line
group = dataCsv[i][0]; //get group name
i++;
times = dataCsv[i];//get list of time headings for this group
times.shift(); // (shift out first empty element)
} else {
country=dataCsv[i].shift(); //regular row: get country name
dataCsv[i].forEach(function(x,j){ //enumerate values
data.push({ //create new data item
Org: group,
Country: country,
Month: times[j],
Score: x
})
})
}
}
This gives the following data array:
data= [{"Org":"EA","Country":"Netherlands","Month":"Jan_2016","Score":"11.7999"},
{"Org":"EA","Country":"Netherlands","Month":"Feb_2016","Score":"15.0526"}, ...]
This is IMO the most versatile structure you can have. Not the best for memory usage though.
A simple way to nest this is the following:
d3.nest()
.key(function(d) { return d.Month+"-"+d.Country; })
.map(data);
It will give a map with key-values such as:
"Jan_2016-Netherlands":[{"Org":"EA","Country":"Netherlands","Month":"Jan_2016","Score":"11.7999"},{"Org":"EB","Country":"Netherlands","Month":"Jan_2016","Score":"1.9024"},{"Org":"EC","Country":"Netherlands","Month":"Jan_2016","Score":"3.1499"}]
Use entries instead of mapto have an array instead of a map, and use a rollup function if you want to simplify the data by keeping only the array of scores. At this point it is rather straightforward to plug it into any d3 drawing tool.
PS: a Plunker with the running code of this script. Everything is shown in the console.
I am somewhat new to JavaScript, but I am reasonably experienced with programming in general. I suspect that my problem may have something to do with scoping, or the specifics of passing arrays as parameters, but I am uncertain.
The high-level goal is to have live plotting with several 'nodes', each of which generates 50 points/sec. I have gotten this working running straight into an array and rendered by dygraphs and C3.js and quickly realized that this is too much data to continually live render. Dygraphs seems to start impacting the user experience after about 30s and C3.js seems to choke at around 10s.
The next attempt is to decimate the plotted data based on zoom level.
I have data saved into an 'object' which I am using somewhat like a dictionary in other languages. This is going well using AJAX requests. The idea is to create a large data buffer using AJAX requests and use the keys to store the data generated by units according to the serial number as the keys. This is working well and the object is being populated as expected. I feel that it is informative to know the 'structure' of this object before I get to my question. It is as follows:
{
1: [[x10,y10], [x11,y11], [...], [x1n, y1n]],
2: [[x20,y20], [x21,y21], [...], [x2n, y2n]],
... : [ ... ]
a: [[xa0,ya0], [xa1,ya1], [...], [xan, yan]]
}
Periodically, a subset of that data will be used to generate a dygraphs plot. I am decimating the stored data and creating a 'plot buffer' to hold a subset of the actual data.
The dygraphs library takes data in several ways, but I would like to structure it 'natively', which is just an array of arrays. Each array within the array is a 'row' of data. All rows must have the same number of elements in order to line up into columns. The data generated may or may not be at the same time. If the data x values perfectly match, then the resulting data would look like the following for only two nodes since x10 = x20 = xn0:
[
[x10, y10, y20],
[x11, y11, y21],
[ ... ],
[xan, yan, yan]
]
Note that this is just x and y in rows. In reality, the times for each serial number may not line up, so it may be much closer to:
[
[x10, y10, null],
[x20, null, y20],
[x11, y11, y21],
[ ... ],
[xan, yan, yan]
]
Sorry for all of the background. We can get to the code tha tI'm having trouble with. I'm periodically attempting to create the plot buffer using the following code:
window.intervalId = setInterval(
function(){
var plotData = formatData(nodeData, 45000, 49000, 200);
/* dygraphs stuff here */
},
500
);
function formatData(dataObject, start, end, stride){
var smallBuffer = [];
var keys = Object.keys(dataObject);
keys.forEach(
function(key){
console.log('key: ', key);
mergeArrays(dataObject[key], smallBuffer, start, end, stride);
}
);
return smallBuffer;
}
function mergeArrays(sourceData2D, destDataXD, startInMs, endInMs, strideInMs){
/* ensure that the source data isn't undefined */
if(sourceData2D){
/* if the destDataXD is empty, then basically copy the
* sourceData2D into it as-is taking the stride into account */
if(destDataXD.length == 0){
/* does sourceData2D have a starting point in the time range? */
var startIndexSource = indexNear2D(sourceData2D, startInMs);
var lastTimeInMs = sourceData2D[startIndexSource][0];
for(var i=startIndexSource; i < sourceData2D.length; i++){
/* start to populate the destDataXD based on the stride */
if(sourceData2D[i][0] >= (lastTimeInMs + strideInMs)){
destDataXD.push(sourceData2D[i]);
lastTimeInMs = sourceData2D[i][0];
}
/* when the source data is beyond the time, then break the loop */
if(sourceData2D[i][0] > endInMs){
break;
}
}
}else{
/* the destDataXD already has data in it, so this needs to use that data
* as a starting point to merge the new data into the destination array */
var finalColumnCount = destDataXD[0].length + 1;
console.log('final column count: ', finalColumnCount);
/* add the next column to each existing row as 'null' */
destDataXD.forEach(
function(element){
element.push(null);
}
);
/* TODO: move data into destDataXD from sourceData2D */
}
}
}
To add some information since it probably isn't self-explanatory without some effort. I create two functions, 'formatData' and 'mergeArrays'. These could have been done in a single function, but it was easier for me to separate out the 'object' domain and the 'array' domain conceptually. The 'formatData' function simply iterates through all of the data stored in each key, calling the 'mergeArray' routine each time through. The 'mergeArray' routine is not yet complete and is where I'm having my issue.
The first time through, formatData should be creating an empty array - smallBuffer - into which data is merged using mergeArrays. The first time executing 'mergeArrays' I see that the smallBuffer is indeed being created and is an empty array. This empty array is supplied as a parameter to 'mergeArrays' and - the first time through - this works perfectly. The next time through, the 'smallBuffer' array is no longer empty, so the second case in 'mergeArrays' gets executed. The first step for me was to calculate the number of columns so that I could pad each row appropriately. This worked fine, but helped point out the problem. The next step was to simply append an empty column of 'null' values to each row. This is where things got weird. After the 1st time through 'mergeData', the destDataXD still contained 'null' data from the previous executions. In essence, it appears that the 'var smallBuffer = [];' doesn't actually clear and retains something. That something is not apparent until near the end. I can't explain exactly what is going on b/c I don't fully understand it, but destDataXD continually grows 'nulls' at the end without ever being properly reset as expected.
Thank you for the time and I look forward to hearing your thoughts, j
Quickly reading through the code, the danger point I see is where you first add an element to destDataXD.
destDataXD.push(sourceData2D[i]);
Note that you are not pushing a copy of the array. You are adding a reference to that array. destDataXD and sourceData2D are now sharing the same data.
So, of course, when you push any null values onto an array in destDataXD, you are also modifying sourceData2D.
You should use the javascript array-copying method slice
destDataXD.push(sourceData2D[i].slice());