Develop Reference

Working with 2D array in D3

I'd like to visualize data pulled in from the census data API, specifically from the ACS survey.
The data is returned in a non-standard version of JSON, as a two-dimensional array. It basically looks like this:
[
[
“POPULATION”,
“DATE”,
“ANOTHERTHING”
],
[
“ALABAMA”,
“2000”,
“MORESTUFF”
],
[
“ALASKA”,
“2000”,
“OTHERSTUFF”
],
…
]
I'm unfamiliar with working with this kind of JSON data, which almost looks more like a CSV, where the keys are written in the first line, and the values in every line after the first.
Is anyone familiar with how to parse and work with this data in D3, without having to go convert it first (i.e. https://gist.github.com/sarfarazansari/7e8ae05168b80b36016eb1c561a82f73)? (I'd like to draw from the data API directly).
Any help or guidance would be much appreciated.

First of all: do not use smart quotes (“) in a JSON (or in your code, whatever it is). I reckon that your real JSON is correct, this just happened because you used a text editor, like MS Word, to write this question. Also, there is no such a thing like non-standard JSON, because there is no standard JSON to begin with. You just have a JSON which is an array of arrays, nothing special here.
That being said, you can use that exact data structure to create your charts... However, specially if you're a beginner, it's a good idea to stick with the most common (here we can say standard) way to organise the data in D3 codes, which is an array of objects.
We can easily convert that array of arrays in an array of objects, which will be more comfortable to you.
First, supposing that your array is named data, let's extract the headers:
var keys = data.shift();
That not only creates a headers array, but it will remove the first inner array from the data array.
Now, we iterate over each inner array, creating an object with all the key/value pairs we need:
var newData = data.map(function(d) {
var obj = {};
d.forEach(function(e, i) {
obj[keys[i]] = e;
});
return obj;
});
There are shorter ways to do this, the one above is quite verbose but it is more didactic.
Here is a demo using the data you shared:
var data = [
[
"POPULATION",
"DATE",
"ANOTHERTHING"
],
[
"ALABAMA",
"2000",
"MORESTUFF"
],
[
"ALASKA",
"2000",
"OTHERSTUFF"
]
];
var keys = data.shift();
var newData = data.map(function(d) {
var obj = {};
d.forEach(function(e, i) {
obj[keys[i]] = e;
});
return obj;
});
console.log(newData)
<script src="https://d3js.org/d3.v5.min.js"></script>

Using a functional approach to pull data out from series of arrays

I need to do some additional filtering of the data I'm getting, so that I'm only getting the data that match certain criteria. I'm trying to find a combination of operators I can use to accomplish this via a functional programming approach. I'm thinking perhaps a combination of filter and reduce might work? Could I simply chain a combination of them together to get the result I'm looking for?
My data structure looks like this:
"data": [
{
"_id": "53545",
"services": [
{
"service": "service1",
"history": [
{
"_id": "6546",
"status": "stage1",
"completed": true
},
{
**"_id": "6547",
"status": "stage2",
"completed": false**
}
],
{
"service": "service2",
"history": [
{
"_id": "6743",
"status": "stage3",
"completed": false
},
{
"_id": "3742",
"status": "stage2",
"completed": true
}
]
},
{
"service": "service3",
"history": [
{
"_id": "6448",
"status": "stage4",
"completed": false
},
{
"_id": "4443",
"status": "stage1",
"completed": true
}
]
}
]
To be clear, here data, services and history are ALL arrays. And in the case of the above data, only one object within "history" should pass the tests, as there is only one object where "status:stage2" and "completed:false". BUT that means that, within the "data" array, this particular object, the one with the "_id": "53545" SHOULD pass, as the tests returned true for it. And ultimately, I'm wanting to return an array, from within the objects within the "data" array, that match like this one does.
Very specifically, in my case, there are always 3 services. Within each of those 3, there might be 1 or as many as 30 or more history objects. I need to look through all 3 "services", and find the history objects where the status is "stage2" and "completed:false". So this should return true if ANY of the three services have a history object with "status:stage2" and "completed:false".
I'm getting tripped up iterating over arrays within arrays.
I could get it directly in this manner:
let stage2Data = data.filter(data => data.services[0].history[1].status === 'stage2' && data.services[0].history[1].completed === false);
However, I obviously don't know what item number in these various arrays will contain the value I'm looking for. So what's the best way to tackle this? I suppose a series of for-of loops could work as well, but I'd like to use a more functional approach, and was thinking a combination of "filter" and "reduce", or perhaps "map" could work here. How would I only return the objects within the "data" array where ""status": "stage2" and "completed": false is true for at least one of the objects within the "history" array?

We've been going back and forth in the comments to try to get a valid data structure in your question. Let me give you a simple example of how you can make this much easier for everyone.
First, use the snippet button at the top of the editor box to create a snippet. It's the icon that looks like a typical "folded corner" document icon with <> inside it.
This will open up a popup box with places for HTML/CSS/JavaScript code. Ignore the HTML and CSS boxes and paste your data object into the JavaScript box.
Because it's not JavaScript "as is", wrap it inside an assignment statement to get a JavaScript variable that you can work with.
Then add a console.log statement at the end to display the data. Use the Run button to make sure the code actually parses and runs and displays the data correctly. Once you do that, it will be much easier for people to help you.
Here is a simplified example:
const input = {
"data": [
{
"one": "two"
},
{
"three": "four"
}
]
};
console.log( JSON.stringify( input, null, 4 ) );
Click the Run code snippet button above to see this example run. Notice how it prints out the same data structure that I used above.
If you do the same in your question, this will force you to clean up your data so that it is actually valid. Then you are giving people something they can work with.
One other note: don't be hung up on trying to do this in a "functional" manner. It's very likely that you will get more understandable code by creating an empty array with let output = [];, then using .forEach() on each of your nested arrays (or for...of loops if you are targeting the latest browsers), and using output.push() on each matching element.
To illustrate, assuming I understand your data structure correctly (and assuming you wrap it inside an input variable as in my simplified example) it might look something like this:
let output = [];
input.data.forEach( function( dataItem ) {
dataItem.services.forEach( function( service ) {
service.history.forEach( function( historyItem ) {
if( ! historyItem.completed && historyItem.status == "stage2" ) {
output.push( historyItem );
}
});
});
});
Or using for...of loops:
let output = [];
for( let dataItem of input.data ) {
for( let service of dataItem.services ) {
for( let historyItem of service.history ) {
if( ! historyItem.completed && historyItem.status == "stage2" ) {
output.push( historyItem );
}
}
}
}
Both of these are untested, of course. And either way it's not fancy, but it seems pretty easy to understand.

Depending on how you want the result data, here's a start for you. You can brute-force this with a few nested filters:
result = data.filter((datum) => {
return datum.services.filter((service) => {
return service.history.filter((h) => {
return h.completed && h.status === "stage2"
}).length > 0;
}).length > 0;
});
tl;dr - Filter the main list by any datum that has services that have at least one history record that was completed in stage 2.
Working fiddle: http://jsbin.com/jozujezidu/edit?js,console,output
If you want your resultant array to only have the history records that pass the filter, you'll have to modify the code. Plus, this is an On^3, so if this data can be enormous, you should probably modify the approach. But this should get you going.
Don't worry about being militant functional. If you can come to a working solution that people can understand and maintain, then that's a good start.

A filter operation will be based on a predicate matching your values.
If you are wanting a list of the service objects try this approach:
let result = data[0].services.filter((item) => item.history.some((h) => h.status=='stage2' && h.completed==false));

Doing a push and update of array in javascript

What I am attempting to do is get unique values stored in the array and if a element gets pushed to the JSON string with the same MAC and IP I would like to make it replace the existing record with the new one it could replace it all or just the signal, distance and frequency I don't have a solid piece of code I am currently just experimenting trying to get what I need. Bellow is an example of the json output from my javascript console in chrome I know it's not formatted the best but it is valid.
{"prey":["{\"distance\": 8.686924173343307, \"signal\": \"-59\", \"frequency\": 2447, \"mac\": \"00:00:00:00:00:00\", \"ip\": \"192.168.43.27\"}"]}
What I have tried do far is use a for loop to check for duplicates this did not work as javascript was returning the wrong length of the array for some reason and it just got stuck in a infinite loop because I kept adding values to the array and it wasn't working in the unique way it should.
This is the code I currently have as I just keep hacking away at it trying everything I can think of this does not work in any way I have been playing with trying to get unique array but it just doesn't work the output of this is just the same as the actual output it does not get the unique array. I got the unique function from a stack overflow question
function arrayLastUnique(array) {
return array.filter(function (a, b, c) {
// keeps last occurrence
return c.indexOf(a, b + 1) < 0;
});
}
jsonSpy['prey'].push(msg);
$('#lines').append($('<p>').text(msg));
console.log(arrayLastUnique(jsonSpy.prey));
spyList = JSON.stringify(jsonSpy);
drawPositions(ctx, spyList);

var yourJSON = {"prey":
[
{ "distance": 8.686924173343307,
"signal": "-59",
"frequency": 2447,
"mac": "00:00:00:00:00:00",
"ip": "192.168.43.27"
}
]
};
var newDistance = 8.69;
if (yourJSON["prey"][0]["mac"] && yourJSON["prey"][0]["ip"]) {
yourJSON["prey"][0]["distance"] = newDistance;
}
If the values for "mac" and "ip" exist i.e. they are not an empty string, false, 0, null, undefined, the condition is satisfied. In the condition block you can assign a new value. This is just a very simple example. You have to adapt it and expand it according to your needs.

Creating meaningful suggestions based on an array of tags

I have a project where there is a catalogue of items, each with an array of tags. I would like to present items that are similar, based on these tags.
Something like this (but with a much larger data set):
{
"item": {
"description":"thing",
"tags": ["a","b","c","e","f"]
},
"item": {
"description":"thing",
"tags": ["a","b"]
},
"item": {
"description":"thing",
"tags": ["a","c"]
},
"item": {
"description":"thing",
"tags": ["b","c"]
}
}
Two things I have tried so far:
First was a straight intersection between the tags on the individual item, and other items that have one or more of the same tags. This works well, but in cases where a tag is somewhat generic (think, tagged with something like "music" where they are all musical items), the number of returned items is huge.
The second one was a slightly crazy idea, where I turned the array of tags into a string, and calculated the levenshtein difference. This works for items that have a length that is approximately the same or larger, but is clunky. Still, it did trim off a lot of the fat that first approach returned. It is not the right way, but wanted to show what I am aiming for. Implemented it like this:
// snip: this is inside a BB collection
getSimilarByTag: function(tags, ignore){
var hits = [];
if (tags) {
this.filter(function(item){
if (item.get('cat') === ignore){
return; // no need to include
};
var itemTags = item.get('tags');
var result = _.intersection(tags, itemTags);
if (result.length) {
// calc levenshtein distance between the intersection and the search array
var dist = _.str.levenshtein(result.join(' '), tags.join(' '));
if (Math.log(dist) < 1.5) { // this value needs tuning?
hits.push(item.toJSON());
} else {
// based on our magic number above, ignore this
}
};
});
}
return hits;
}
I'm doing all my code in javascript, using backbone and underscore. However, the language is not so important - just curious about what kind of technique or algorithm might give a better set of results.

A simple routine for most applicable data could be, return matches in order of size of tag intersection, with a limited return count. If you could weight certain tags as being more important then you could use that to adjust the returned order. For example if the user has previously bought items from the catalogue, then the tags linked to their purchases could have an increased score in the order algorithm.

Traversing 'JSON Array of arrays' in python 'List of lists' format

For reasons I don't understand, the JSON format data I need to parse that comes out of a webservice isn't in nme value pairs. 'For simplicity and to reduce overhead' the returned JSON seems to be in a format that works with python eval but as far as I can see not with javascript (caveat, my javascript is very poor so I could be wrong - php etc, fine. js, not so much!)
So the data is returned as:
[[0, 'OK'],
[['ITEM10314', ['ITEM10397']],
['ITEM10315', ['cornflower']],
['ITEM10397', ['ITEM10315']],
['ITEM10514', ['ITEM10397']],
['ITEM003', []],
['ITEM004', []],
['servertest', ['ITEM004', 'ITEM003']],
['serverroot', []]]]
(in case you are interested, it is a reply from a MKLiveStatus for Nagios LQL host query)
The first array is the status, then subsequent arrays consist of the host monitored in nagios and that host's parents (in an inner array).
Nice, isn't it. But I need to get it into decent key/value pairs and there must be a better way than writing my own parser for this (not least because this is one data output but there are several more in similar formats).
I'm trying to keep this all in native js but if there is a jQuery easy way then I'm easily led to lazyness. No need to worry about old browsers, I don't care, this project ends up using d3.js which won't work on old browsers anyway.
Any suggestions? The depth in this case won't go below what it is here, so that at least is a known. However, I can't just flatten it, I need to know which parents a host has after this.
I have seen a few python-js links here but not arbitrary unknown sized lists in lists.

Something like this should do it
var data = [
[0, "OK"],
[
["ITEM10314", ["ITEM10397"]],
["ITEM10315", ["cornflower"]],
["ITEM10397", ["ITEM10315"]],
["ITEM10514", ["ITEM10397"]],
["ITEM003", []],
["ITEM004", []],
["servertest", ["ITEM004", "ITEM003"]],
["serverroot", []]
]
];
function parse(array) {
var object = {
ok: 1
};
if (!Array.isArray(array) && array[0][0] !== 0 && array[0][1] !== "OK") {
return object;
}
object.ok = 0;
array[1].forEach(function (element) {
object[element[0]] = element[1];
});
return object;
}
console.log(parse(data));
On jsfiddle