Highcharts csv data loading and parse into separate array and json - javascript

First post here, and I am quite new to javascript so hopefully explaining my question correct. I am trying to process data from a csv file into JSON and create a Highcharts chart with this data, but don’t know where to start. I looked into a lot of answers but they do not fully seem to address my question. My goal is to produce a Highcharts stacked area chart from a csv file located on the same server.
The Highcart code I now have uses a single array for the x-axis named categories (years in this example):
var categories = [ '1850', '1900', '1950', '2000'];
and a JSON for the dataseries: var data = [{
name: 'Line1',
data: [116, 127, 131, 143]
}, {
name: 'Line2',
data: [206, 217, 221, 233]
}, {
name: 'Line3',
data: [303, 313, 326, 338]
}]
Most datasets and charts used in previous answers use a csv file with vertical representation of time series where the second column is used for the data. For instance the daily stockprice of last year.
But the data.csv file I am planning to use looks like this:
Lines, ‘1850’, ‘1900’, ‘1950’, ‘2000’
Line1, 116, 127, 131, 143
Line2, 206, 217, 221, 233
Line3, 303, 313, 326, 338
I was thinking about using a $.get function like this:$.get(‘data.csv’, function (csvfile) { //existing code for creating the chart… } and create a function for parsing the csv data into the variable Data and the variable Categories. But I have no clue how to do this..
The basic code (with hardcoded data) in this: JSfiddle
Does someone have an idea which javascript code I need to parse CSV data into the json/data of the Highcharts graph ?
EDIT:
I use this code to read the csv data through an ajax call and an process the received data:
$(document).ready(function() {
$.ajax({
type: "GET",
url: "data.csv",
dataType: "text",
success: function(data) {processData(data);}
});
});
function processData(data) {
var table = data.split("\n").map(line => line.split(","));
var categories = table[0].slice(1);
var data = table.slice(1).map(a => ({"name": a[0], "data": a.slice(1)}));
console.log(data);
console.log(categories);};
The data is received correct but the array consists of strings instead of numbers. Which part do i need to modify in the processData function to get number datatype?

Assuming that your csv data arrives in the form of "Lines,1850,1900,1950,2000\nLine1,116,127,131,143\nLine2,206,217,221,233\nLine3,303,313,326,338" then in order to construct your categories and data arrays you might do as follows;
var csvData = "Lines,1850,1900,1950,2000\nLine1,116,127,131,143\nLine2,206,217,221,233\nLine3,303,313,326,338",
table = csvData.split("\n").map(line => line.split(",")),
categories = table[0].slice(1),
data = table.slice(1).map(a => ({"name": a[0], "data": a.slice(1)}));
console.log(categories);
console.log(data);
And if you would like to receive the array contents as number type instead of string you may do as follows;
var csvData = "Lines,1850,1900,1950,2000\nLine1,116,127,131,143\nLine2,206,217,221,233\nLine3,303,313,326,338" ,
table = csvData.split("\n").map(line => line.split(",")),
categories = table[0].slice(1).map(e=>e*1),
data = table.slice(1).map(a => ({"name": a[0], "data": a.slice(1).slice(1).map(e=>e*1)}));
console.log(categories);
console.log(data);

Related

Unable to access JSON data from Javascript

I am passing the following from my Django back-end to my front-end in order to dynamically build a form :
{
"access_key": "93ec6137de00eacee6f8",
"profile_id": "7851E15D64",
"transaction_uuid": "c878c7e6db5657526",
}
Within the browser console, if I pass :
MyJSON = {
"access_key": "93ec6137de00eacee6f8",
"profile_id": "7851E15D64",
"transaction_uuid": "c878c7e6db5657526",
}
Then I can access each value properly, for example, MyJSON.access_key returns 93ec6137de00eacee6f8 perfectly within the console.
However, from my Javascript, I am unable to access any of those values as i get an "undefined".
var obj = JSON.parse(MyJSON)
console.log(obj) // returns the whole JSON String
console.log(typeof(obj)) // returns 'string'
alert(obj[0]) // returns "{", the very first character of the 'string'
alert(obj.access_key) // returns "undefined".
- How can access each Key and Value from MyJSON from my javascript? (not JQuery)
Note that I have reviewed many similar article but I must be missing the obvious ...
Hope you can assist !
Thanks in advance.
EDIT :
I have a list of Fields and a list of Values which i then merge into the below (pass the JSON Validator on https://jsonlint.com/):
{'access_key': '93ec6137d70aada23400eacee6f8', 'profile_id': '7851E53E-96BB-4D4-BD5-0FE61CC15D64', 'transaction_uuid': '00939a90-db7b-41cb-af45-669ec5cc69e8', 'signed_field_names': 'bill_to_forename,bill_to_surname,bill_to_email,bill_to_phone,bill_to_address_line1,bill_to_address_city,bill_to_address_postal_code,bill_to_address_country,transaction_type,reference_number,payment_method,amount,currency,locale,card_type,card_number,card_expiry_date', 'unsigned_field_names': 'card_type,card_number,card_expiry_date', 'signed_date_time': '2021-05-23T16:20:17Z', 'bill_to_forename': 'John', 'bill_to_surname': 'Doe', 'bill_to_email': 'null#cfgfg.com', 'bill_to_phone': '07922889582', 'bill_to_address_line1': '123 Random Street', 'bill_to_address_city': 'London', 'bill_to_address_postal_code': 'RG1T3X', 'bill_to_address_country': 'GB', 'transaction_type': 'sale', 'reference_number': 'o6yejf', 'payment_method': 'card', 'amount': '100', 'currency': 'USD', 'locale': 'en', 'card_type': '001', 'card_number': '4456530000001096', 'card_expiry_date': '12-2026', 'signature': 'Un5gNYB5qZaRazzCDWqrdZuNkTRARIcfAt9aF2a1wbY='}
Back-end Code
FieldList = ['access_key', 'profile_id', 'transaction_uuid', 'signed_field_names', 'unsigned_field_names', 'signed_date_time', 'bill_to_forename', 'bill_to_surname', 'bill_to_email', 'bill_to_phone', 'bill_to_address_line1', 'bill_to_address_city', 'bill_to_address_postal_code', 'bill_to_address_country', 'transaction_type', 'reference_number', 'payment_method', 'amount', 'currency', 'locale', 'card_type', 'card_number', 'card_expiry_date', 'signature']
ValueList = ['93ec6137d0aada23400eacee6f8', '7851E53E-96BB-4DF4-BD55-0FE61CC15D64', 'c4fe96b0-063f-4b94-a6a5-2137bb796bd9', 'bill_to_forename,bill_to_surname,bill_to_email,bill_to_phone,bill_to_address_line1,bill_to_address_city,bill_to_address_postal_code,bill_to_address_country,transaction_type,reference_number,payment_method,amount,currency,locale,card_type,card_number,card_expiry_date', 'card_type,card_number,card_expiry_date', '2021-05-23T16:27:24Z', 'John', 'Doe', 'null#cyrce.com', '07922889582', '123 Random Street', 'London', 'RG1T3X', 'GB', 'sale', 'xl42fn', 'card', '100', 'USD', 'en', '001', '4456530000001096', '12-2026', 'vvb73h0GUpzUrvoG9VDaMc3vQRV5GsL4QTATc7IrrPA=']
NewFormat = dict(zip(FieldList, ValueList))
MyJSON = json.dumps(NewFormat, indent=4)
return JsonResponse(MyJSON, safe=False)
Apologies for the large amount of data.
I am somehow forced to use "safe=False" in my Python back-end otherwise I end up with :
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/home/pi/.local/lib/python3.7/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/pi/Documents/Droplet/Droplet/Harness/sasop2.py", line 543, in signsasop
return JsonResponse(FinalJSONObject)
File "/home/pi/.local/lib/python3.7/site-packages/django/http/response.py", line 561, in __init__
'In order to allow non-dict objects to be serialized set the '
TypeError: In order to allow non-dict objects to be serialized set the safe parameter to False.
Since I am passing safe=False, is that why my front-end does not get the MyJSON as... JSON?
Would this be the root cause of the issue?
Front-End Sample :
xhr.onreadystatechange = function() {
if (xhr.readyState === 4 && xhr.status === 200) {
// Print received data from server
console.log('%c Processed Data \n',
'background: #000000; color: #FFFFFF; font-size: 30px'
,xhr.response);
// Dynamically create the ReconstructedForm
RawProcessedData = xhr.response
console.log(RawProcessedData)
// Convert to JSON
var obj = JSON.parse(RawProcessedData)
console.log(obj)
console.log(typeof(obj))
alert(obj[0])
alert(obj.access_key)
Thanks a lot for your quick input !
As per deceze's answer, I was essentially double parsing both in the back and front end for no reasons.
Removing the json.dumps from the backend allows the JSON object to be passed and managed in the front end without issue.
I have the same issue when i started developing on Django. If you need to pass dictionarys from django to javascripts, the best thing to do is just using django rest framework. It serialize ( in other words, it transform any data into a dictionary/json ) any given data from a model.
But if you want to make this without Django Rest, you should use fetch on javascript. This fetch ( also called as a "Promise") communicate with the backend ( in this case, Django ) and it GET or POST data from the frontend. I will give you an example.
Supose you have this on views.py:
from django.http.response import JsonResponse
def getJSON(request):
MyJSON = {
"access_key": "93ec6137de00eacee6f8",
"profile_id": "7851E15D64",
"transaction_uuid": "c878c7e6db5657526",
}
return JsonResponse(MyJSON)
And the you can link that to the urls.py like this:
urlpatterns = [
path('get-json', views.getJSON, name="get-json")
]
Then you can GET that JSON by doing this on your javascript:
function getDataJson(){
let url = 'get-json'
fetch(url,{
headers: {
"Content-type": "application/json",
},
})
.then(response => response.json())
.then(data=> console.log(data))
}
This will console log your data.

How to dynamically link json data to chart.js

I am trying to create a chart using json data. Currently I have the data within my script, but I would like to link it to a separate .json file within the project folder.
This is what I have currently:
var jsonfile = {
"jsonarray": [{
"name": "Joe",
"age": 12
}, {
"name": "Tom",
"age": 14
}]
};
var labels = jsonfile.jsonarray.map(function(e) {
return e.name;
});
var data = jsonfile.jsonarray.map(function(e) {
return e.age;
});
var ctx = document.getElementById("chart");
var chart = new Chart(ctx, {
type: 'line',
data: {
labels: labels,
datasets: [{
label: 'Graph Line',
data: data,
backgroundColor: 'rgba(0, 119, 204, 0.3)'
}]
}
});
Ideally, I would like the json data to be in a separate file so that it can be dynamically saved.
Thanks :)
You can use the fs module to read and write to a file
var fs = require('fs');
var jsonfile = fs.readFileSync('the/path');
//if you want to write to the file I gave you a link that shows you the way
For more information about the fs module you can check File System | Node.js v12.6.0 Documentation.
You may need to pass by JSON.stringify and JSON.parse to save your object as a string in the file and to parse the string back to an object while reading the file.
Another manner that you may consider is lightweight javascript databases such as IndexedDB which is an embedded database in the browser side, or other databases such as neDB if you want to persist data especially with an electron app.
Have many way to use with your wish
var data = {"a" : "b", "c" : "d"};
then
<script src="data.js" ></script>
or
var json = require('./data.json');
or
$.getJSON("data.json", function(json) {
console.log(json);
});
Good luck!

Remove unwanted columns from CSV file using Papaparse

I have a situation where a user can upload a csv file. This CSV file contains a lot of data, but I am only interested in 2 columns (ID and Date). At the moment, I am parsing the CSV using Papaparse
Papa.parse(ev.data, {
delimiter: "",
newline: "",
quoteChar: '"',
header: true,
error: function(err, file, inputElem, reason) { },
complete: function (results) {
this.parsed_csv = results.data;
}
});
When this is run this.parsed_csv represents objects of data keyed by the field name. So if I JSON.stringify the output is something like this
[
{
"ID": 123456,
"Date": "2012-01-01",
"Irrelevant_Column_1": 123,
"Irrelevant_Column_2": 234,
"Irrelevant_Column_3": 345,
"Irrelevant_Column_4": 456
},
...
]
So my main question is how can I get rid of the columns I dont need, and just produce a new csv containing the columns ID and Date?
Thanks
One thing I realised, is there a way to add dynamic variables. For instance I am letting users select the columns I want to map. Now I need to do something like this
let ID = this.selectedIdCol;
this.parsed_csv = results.data.map(element => ({ID: element.ID, Date: element.Date}));
It is saying that ID is unused however. Thanks
let data = [
{
"ID": 123456,
"Date": "2012-01-01",
"Irrelevant_Column_1": 123,
"Irrelevant_Column_2": 234,
"Irrelevant_Column_3": 345,
"Irrelevant_Column_4": 456
},
...
]
just produce results by using the following code:
data = data.map(element => ({ID: element.ID, Date: element.Date}))
Now you have desired column, please generate a new CSV on these columns
As Serrurier pointed out above, You should use the step/chunk function to alter the data rather than after parse map as in memory data is already available.
PapaParse.parse(file, { skipEmptyLines: true, header: true, step: (results, parser) => {
results.data = _.pick(results.data , [ 'column1' 'column2']);
return results;
}});
Note that if you are loading a huge file, you will have the whole file in memory right after the parsing. Moreover it may freeze the browser due to the heavy workload. You can avoid that by reading and discarding columns :
row by row
chunk by chunk.
You should read Papaparse's FAQ before implementing that. To sum up, you will store required columns by extracting them from the step or chunk callbacks.

How do I pass a PHP variable from one file to another multiple times?

I have a highstock graph in a file called charts.php that reloads the graph data from a secondary file called data.php. Seeing as I have 5 series in my highstock graph I have to call the data.php file 5 times, each time using a different variable, but when I set the variable in my charts.php file it only uses the last set value of this variable. Is is possible to solve this, or is this just the way PHP works? Do I have to create 5 separate data.php files in order for this to work?
Here's basically what my code looks like (showing 2 series to keep it short):
var chart = new Highcharts.StockChart({
chart: {
renderTo: 'chart'
},
title : { text : 'MyChart' },
series : [
{
name : 'Unit 0',
data : [],
tooltip: { valueDecimals: 1, valueSuffix: ' kW' },
},
{
name : 'Unit 1',
data : [],
tooltip: { valueDecimals: 1, valueSuffix: ' kW' },
}
]
});
<?php $_SESSION["unit"] = 0; ?>
$.getJSON( "data.php", function (data) {
chart.series[0].setData( data );
});
<?php $_SESSION["unit"] = 1; ?>
$.getJSON( "data.php", function (data) {
chart.series[1].setData( data );
});
So the problem is that both series[0] and series[1] use the last value of my variable, which in this case is $_SESSION["unit"] = 1. Is the a way to make this work? Any better alternatives?
Maybe if I could create data for all of the series in one file and somehow split that data and pass it to series[0] and series[1] separately, but I have no idea how to do that..
Any help is appreciated.
Maybe you could modify your data.php file, so instead of using $_SESSION["unit"] to determine which var to use, use $_GET["unit"] instead. Then, in your code above, it would be:
$.getJSON( "data.php?unit=0", function (data) {
chart.series[0].setData( data );
});
$.getJSON( "data.php?unit=1", function (data) {
chart.series[1].setData( data );
});
It depends on your needs, if the $unit values are always the same I would just call data.php once and update all the graphs at once using the JS response.
If $unit has a dynamic variable I would send it along with the $.getJSON function.
var data = {
unit: $_GET['unit']
};
$.getJSON( "data.php", data, function (response) {
updateGraph(graph, response);
});
For more on getJSON on the structure of the function, visit http://api.jquery.com/jquery.getjson/

how to get tow series data in highchart guage chart

I am ajax calling a php script that outputs two data points
[epoch data, cpu]
I would like to be able to create highchart guage chart and display cpu utilization and as a data lable, I like to display the time that I get from the php call.
function request_cpu_Data() {
$.ajax({
url: 'get_cpu.php',
success: function(data) {
var myObj = JSON.parse(data);
var point = cpu_chart.series[0].points[0];
var point1 = cpu_chart.series[1].points[0];
var newVal=myObj[1];
var newVal1=myObj[0];
point.update(newVal);
point1.update(newVal1);
setTimeout(request_cpu_Data, 10000000);
},
cache: false
});
}
my series options are like this:
series: [{
name: 'CPU',
data: [0]
},{
name: 'Date',
data: []
}
]
I am getting this error:
Uncaught TypeError: Cannot call method 'update' of undefined
Is this how you set up the second series?
var point1 = cpu_chart.series[1].points[0];
Your second series contains no data, so your cpu_chart.series[1].points is an empty array.
You must use setData if data.lenght === 0, like this:
var secondSeries = chart.series[1];
if (secondSeries.data.length === 0) {
secondSeries.setData([newValue]);
} else {
secondSeries.points[0].update(newValue);
}
I hope that will help you.

Categories

Resources