Develop Reference

Javascript how to format json values for pie chart DataTable

I have an API that returns 100 rows of lots of different json data including HTTP Status:
{"data": [{"status": 409},{"status": 200},{"status": 404},{"status": 200},{"status": 200},{"status": 200}]}
And I want to make a pie chart which requires a DataTable like this:
['status', count],
['200', 4],
['404', 1],
['409', 1]
What is the most graceful way to accomplish this? I am somewhat familiar with vanilla Javascript but will accept Jquery or other solutions.
I would have thought there was a simple library (?) that would let me choose "status" from my json and it would automagically format it for a pie chart (any pie chart - although I'm trying Google Charts to begin) — but I have not found such a library.
What I have tried...
I made a list of things to learn how to accomplish ... assuming there is no "library" or simple way to select, count (and maybe also order from lowest to highest) the json data for the Google or other pie chatrt, Does this look like a decent plan, or is there a better way?
create a new array with each "status" as an element
from the new array, determine each unique status from the json (I Googled several ways to find unique values in an array, have not figured which one will work for me yet) as yet another new array
using a combination of the first and second new arrays, count the number of times each status appears in the json (I think I can use each or foreach and count up +1 for each occurence of each unique status element)
format each unique status with its count, adding brackets and commas (I think I can use join between name value pairs, so the last one does not have an extra comma, and + concatenate with commas and brackets using single quotes, maybe)
sort the statuses low to high (this will be for color coding from green to red where 200's will be green and 500's will be red)
add the title to each column
Thank you for any advice.

I believe an array.reduce loop will do the trick.
let data = {
"data": [{
"status": 409
}, {
"status": 200
}, {
"status": 404
}, {
"status": 200
}, {
"status": 200
}, {
"status": 200
}]
}
let newdata = data.data.reduce((accum, a) => {
// console.log(accum, a);
let found = 0;
accum.forEach((el, index) => {
if (el[0] == a.status) {
accum[index][1]++;
found = 1
}
})
if (!found) accum.push([a.status.toString(), 1])
return accum
}, [
['status', 'count']
]);
console.log(newdata)

Best way to group by category a list of products | Javascript

I'm trying to find the best way to group by category and iterate products in O (n) to get some insights from the categories.
I have the sample data:
[
{
"code": 25754,
"description": "ADAPTADOR BLUETOOH USB RECEPTOR DE AUDIO P2",
"price": 5.0,
"stock": 10,
"category": {
"id": 1,
"name": "Adapters"
}
},
{
"code": 20212,
"description": "ADAPTADOR CONECTOR HDMI FEMEA L / FEMEA",
"price": 2.8,
"stock": 20,
"category": {
"id": 2,
"name": "Eletronics"
}
},
]
I need to invert the relationship, having a list of categories with corresponding products, and for that i wrote this solution
function group_by_categories(products) {
const categories = {}
for (const product of products) {
const { category, ...cleanedProduct } = product
categories[category.id] = categories[category.id] || category
categories[category.id].products = categories[category.id].products || []
categories[category.id].products.push(cleanedProduct)
}
return Object.values(categories)
}
// returns
[
{ id: 1, name: 'Adapters', products: [ [Object] ] },
{ id: 2, name: 'Eletronics', products: [ [Object] ] }
]
But I am struggling in two things.
Is it the best way to reverse the relationship? How can I replicate this in another language like C, where I have no objects to use as unique keys?
Once you have this type of data, the only way to iterate categories and products (see how many items a category has, for example) is in O (n²)?
I appreciate all the help, even if you can only answer one question. Also, sorry for my bad English, I'm trying to be as clear as possible here.

So you have 3 issues. 1) Use the code/id as keys instead, 2) Use sets instead of arrays, and 3) use an appropriate data structure to avoid duplicating work that you've already done.
You really just want to map the connections not all the data. I believe code is probably unique to the product, so that is your key. Your category id is also likely unique, so you only need to consider that. The mapping structures should only concern themselves with the minimal amount of unique data. This may increase performance a fair bit as well as the amount of data that get's copied is probably 1/10 to 1/100 of yours in terms of # of characters (of course translating that exactly to time saved is difficult). Of course, that's a minor point compared to the O(N^2) performance, but just that would likely speed things up by a bit by that alone.
You should be using sets as well as the hash (object). Idea here is O(1) lookup, O(1) size check, O(1) inclusion test (IE: Does the category X have code Y in it?), and O(1) mapping back to the original data (code Y is a product with this information).
The other key thing is you really just want to map the connections not all the data. I believe code is probably unique to the product, so that is your key. Your category mapping structure should only concern itself with the minimal amount of unique data (this will increase performance by a lot as well).
Keep in mind, if you have access to an actual database that is probably the more ideal solution, 95% or so of the time once you start wanting to do more complex queries. I would say you almost certainly should be using one, there's probably a database that will suit your needs.
That being said, what you need is this if you don't need to go too far with your queries. It looks like you need to answer these three questions:
Given the code, what is the record (sample data)? This is just a simple code:product object. IE: {25754: {code: ..., price: ..., stock: ..., ...}, {20212: {code: ..., price: ..., stock: ..., ...}}
Given a category, what are the codes in that category? In this case you have a category_id:set(codes) lookup. Very important that you add codes to a set and not a list/array. Sets have O(1) to add/delete/inclusion, while lists/arrays have O(1) add, O(N) delete and O(N) inclusion check.
Given a category, how many are in that category? This is just a data[category].size check (length instead of size in some languages).
Main thing is to use dictionaries and sets for performance.
Time to build the lookups is likely O(P) where P is the total number products. Performance for the queries should be O(1) for each one you need to do.
To avoid O(N^2) performance the lookup should only be calculated once. Should you need to add/remove products from categories, you should adjust the lookup itself and not rebuild it every time. This may mean storing it in a database, building when you first run the app and keeping it in memory, or if the # of products isn't too much, building it at each request (by only using the minimal amount of data to build it, it may be more practical). Generally speaking, even with 1000 products, it should take probably milliseconds to build the category lookup and iterate over them.
Basically your category code should look like his after you've written out the methods.
...
category_lookup = build_category_lookup() # O(P)
product_lookup = build_product_lookup() # O(P)
...
products_for_category = product_lookup[category_id] # O(1)
products_count = products_for_category.length # O(1) | C: sizeof(products_for_category)
...
(Mostly code in ruby these days, so snake_case)

Pentaho JavaScript JSON - Concatenate json objects into existing JSON document based on keys

Basically what i'm trying to do is to build up a Json document, with it's pieces all spread into smaller Json pieces, all rows containing the Keys needed to insert the smaller Json bits inside the correct json structure.
My problem starts in that i know next to nothing about JavaScript. So i would have to learn JavaScript from scratch(already doing), So i'll describe as best i can the issue and what i want to achieve.
Issues: In every Pentaho / Json post i see a hardcoded(Like This answer, which is helpful but again, hardcoded number of fields) JavaScript to build a pre-formed Json structure, what i need is a way to to insert / create, dynamically built info into a Json Structure. And it also needs Nested Structures.
So, in the start i would bulk build all the Flat Json structure, which can be easily done with the Json output Step, since it outputs built Json objects, the objects themselves can be built dynamically, and then just concatenated(Again i don't know if this is possible).
After the flat part is done, comes the part of concatenating the other objects inside the correct structure, some are just more plain objects, others are Arrays of Objects(Nested part).
This is where i need confirmation from someone who understands this part that it is possible to achieve inside Pentaho JavaScript, or the only way using Pentaho is to build hardcoded structures. Below i'll show in Json examples what i want.
This would be an example of the Flat JSON:
{
"Apolices": [{
"Apolice": {
"APO_numero_apolice": "1321635113",
"APO_ramo": "312",
"APO_data_proposta": "22-05-2018",
"APO_valor_desconto": 0.0,
"APO_valor_liquido": 1550.39,
"APO_cod_parceiro_negocio": "1000",
"APO_inicio_vigencia": "22-05-2018",
"APO_status_apolice": "EMITIDA"
}
},
{
"Apolice": {
"APO_proposta": "3212121",
"APO_data_registro": "08-08-2018",
"APO_numero_apolice": "7891321498",
"APO_ramo": "515",
"APO_data_proposta": "22-03-2018",
"APO_valor_desconto": 0.0,
"APO_valor_liquido": 2150.72,
"APO_cod_parceiro_negocio": "7548151100",
"APO_inicio_vigencia": "22-07-2018",
"APO_status_apolice": "EMITIDA",
"APO_produto": null,
"APO_codigo_corretor": "812182",
"APO_fim_vigencia": "22-05-2019",
"APO_valor_bruto": 2320.8,
"APO_percentual_comissao": 19,
"APO_iof": null
}
}
]
}
Then in the course of said implementation, the desired outcome would be something like :
{
"Apolices": [{
"Apolice": {
"APO_numero_apolice": "1321635113",
"APO_ramo": "312",
"APO_data_proposta": "22-05-2018",
"APO_valor_desconto": 0.0,
"APO_valor_liquido": 1550.39,
"APO_cod_parceiro_negocio": "1000",
"APO_inicio_vigencia": "22-05-2018",
"APO_status_apolice": "EMITIDA"
},
"Item": {
"ITE_ano_fabricacao": "2001",
"ITE_modelo": "FOCUS SEDAN GHIA 2.0 MPI 16V 4P",
"ITE_ano_modelo": "2001",
"Cobertura": [{
"COB_cod_cobertura": "21",
"COB_valor_importancia": "50000",
"COB_valor_premio": "415,71",
"COB_cobertura": "RCF-V - Danos Materiais"
},
{
"COB_cod_cobertura": "17",
"COB_valor_importancia": "16712",
"COB_valor_premio": "1165,96",
"COB_cobertura": "Colisão,Incêndio e Roubo\/Furto"
}
]
}
},
{
"Apolice": {
"APO_proposta": "3212121",
"APO_data_registro": "08-08-2018",
"APO_numero_apolice": "7891321498",
"APO_ramo": "515",
"APO_data_proposta": "22-03-2018",
"APO_valor_desconto": 0.0,
"APO_valor_liquido": 2150.72,
"APO_cod_parceiro_negocio": "7548151100",
"APO_inicio_vigencia": "22-07-2018",
"APO_status_apolice": "EMITIDA",
"APO_produto": null,
"APO_codigo_corretor": "812182",
"APO_fim_vigencia": "22-05-2019",
"APO_valor_bruto": 2320.8,
"APO_percentual_comissao": 19,
"APO_iof": null
}
}
]
}
The Json Objects of "Item" and "Cobertura", and others, would come as already built Json objects, with the necessary keys to insert the entire object in the correct position of the structure.
I have done next to no coding myself as of yet since i'm still learning JavaScript, but i already know that Pentaho's JavaScript is somewhat limited in resources, that's why i'm asking ahead if this is even possible inside Pentaho/Kettle

Don't be afraid of the resource limitations for Javascript in PDI. It uses the official Rhino OpenSource project which is 100% ES6 compatible.
Of course, the dynamic nature of JS makes it a little bit slower, but I do not think the difference will be significant unless you have a continuous input flow of millions per minutes.

Remove object key prefixes nodejs

This is my first post. I don't like ask help in forums, but I have no choice. I spent a lot of days trying resolve this but I haven't found any solution.
All my problem start with an database query that returns this.
{
"prod_format": "400 ml",
"prod_brand": "Marca",
"prod_image": "192.168.1.45/mini/1046332bb531ab3.jpg",
"prod_name": "Gel de baño o ducha",
"prod_pvp": 2.20,
"prod_shops": [
{
"prod_price": 2.29,
"prod_shop": {
"prod_shopID": 2,
"prod_shop_name": "Tienda",
"prod_shop_image": "192.168.1.45/shops/d4d4sw2.jpg"
}
},
{
"prod_price": 2.19,
"prod_shop": {
"prod_shopID": 5,
"prod_shop_name": "Tienda",
"prod_shop_image": "192.168.1.45/shops/g4hbe72.jpg"
}
}
]
}
I want remove prefixes of keys, prod_ and shop_ in this case.
I can't do this by hand because this is only one of multiple queries that i use, and this data can change, i need an dynamic function that remove these prefixes, to use with more prefixes and more queries.
So far I have been trying:
Recursive function that calls itself to remap inside objects and
arrays, but it doesn't work because NodeJS is asynchronous and make an
loop that never ends.
Use an list function that gets all keys/values to make a new object
with renamed keys, but i cant found anything that works with complex
objects.
Use chained promises, i can't get that it works, but i think that
it is the best way to do this.
In conclusion I want an function that if I put the upper object, I get this code.
{
"format": "400 ml",
"brand": "Marca",
"image": "http://192.168.1.45/mini/1046332bb531ab3.jpg",
"name": "Gel de baño o ducha",
"pvp": 2.20,
"shops": [
{
"price": 2.29,
"shop": {
"shopID": 2,
"name": "Tienda",
"image": "http://192.168.1.45/shops/d4d4sw2.jpg"
}
},
{
"price": 2.19,
"shop": {
"shopID": 5,
"name": "Tienda",
"image": "http://192.168.1.45/shops/g4hbe72.jpg"
}
}
]
}
PS:
I can't edit the original object, because it comes of elasticsearch and i can't edit object keys.
Thanks in advance, I hope that anyone can help me.

Why don't you simply treat your Json data as a string and then replace '"_prod' with '"'? That would remove all occurances of this prefix.
myJsonString.replace('"_prod', '"');
Update: If your values may contain your prefixes, you better use regular expressions instead, just to make sure you only change the keys.

Correct layout for Json data when searching different attributes

I have created a data file using JSON. The data is a selection of variables about tube stations. See below for a section of the file.
This setup works very well for finding stations when you have the station id.
example: when you click on a station on my webpage the elements id is the station code and this allows a popup box to be filled with the stations name and zone.
However, how would it be best to find the location of the station when for example given the full text name?
a) I would have to have a code iterating through every station object and using an if command check the name in the object against the name given.
b) created a set of key:value pairs in a second javascript object which is first used to lookup the station id then searches in the first object for the rest of the station data.
Also how would I find data on all the stations on a given line.
a) iterate through and check for each that the tubelines array contains the line string
b) Yet another object containing all station codes under a line heading.
{
"BST": {
"name": "Baker Street",
"Location": [
51.523129639184,
-0.15688978273689
],
"zone": 1,
"tubelines": [
"Bakerloo",
"Circle",
"Hammersmith & City",
"District"
],
"Bakerloo": {
"stepfreetrain": false
},
"Circle": {
"stepfreetrain": false
},
"Hammersmith & City": {
"stepfreetrain": false
},
"District": {
"stepfreetrain": false
}
},
"CHX": {
"name": "Charing Cross",
"Location": [
51.508358644804,
-0.12478853109718
],
"zone": 1,
"tubelines": [
"Bakerloo",
"Northern"
],
"Bakerloo": {
"stepfreetrain": true
},
"Northern": {
"stepfreetrain": true
}
}
}
So the key question is, is this the best way to lay out the data above and if so which is the quickest method to pull out information?

To answer your question (was too long to post it as comment):
Create these secondary objects within your "initialization" phase. After generating the objects you can make them accessible via a simple "API". Made a small example (using jQuery and assuming your spots are stored in a spots.json file):
var myApp = {
init: function() {
// show loading screen while fetching data and creating the "lookup" objects
showLoader();
var loadSpots = $.ajax({
url: "/spots.json"
});
loadSpots.done(function(spots) {
myApp.spots = spots;
$.each(spots, function(code, details) {
nameLookup[details.name] = spots[code];
});
// All done, hide the loading screen
hideLoader();
});
},
spots: {},
nameLookup: {},
findByCode: function(code) {
return myApp.spots[code];
},
findByName: function(name) {
return myApp.nameLookup[name];
}
};
myApp.init();
// you can then use these API methods to find all information for Baker Street either by name or by code (or by whatever)
myApp.findByCode('BST');
myApp.findByName('Baker Street');

Iterating through one only object: slower
Creating multiple objects with different layouts: redundancy (thus possibly error-prone)
If speed of reading the data is more important than high(er) memory consumption write a function to generate several objects designed for your needs at runtime automatically, ONCE. Don't do that manually since it's error-prone and a monkey task. And don't forget to observe changes to your main object (if it can be modified at runtime) so you can regenerate your optimized objects as soon as an entry gets modified. That costs some performance once when starting your application but then you can read from the generated objects fast and easily.
If low memory consumption is more important than speed of finding entries (for example because your application is running on older devices with low memory) just iterate through the entries. I'm not 100% sure if one iteration per lookup uses less memory than creating several different objects and keeping them in memory - it probably depends on the size of your data object.
Looking up keys on the first level of an object is of course always faster than iterating and searching for values deep within an object but it uses more memory since the additional objects must be stored somewhere.

I would suggest to put the data of all the separate tube lines inside the "tubelines" property.
For example:
{
"tubelines": {
"Bakerloo": {
"stepfreetrain": true
},
"Northern": {
"stepfreetrain": true
}
}
}
That makes looking up data easier as you can traverse the JSON "tree" for each tube line and find it's data.
Also, I find it easier to name the coordinates "lat" and "long" as separate properties of "Location" (Location.lat, Location.long) vs using an array (Location[0], Location[1]) for readability and eliminating possibility of making mistakes when entering data.

I would suspect that given the number of tube stations (270?) that you're not going to notice any particular data arrangement being superior. A simple iteration would be barely noticeable on virtually any platform.
I would rather (as ever) concentrate on making the code legible and easy to manage/maintain, and worry about specific performance issues as they arise.