Visualising amount of similar elements in database in d3.js - javascript

I created a webcrawler which crawls a website and stores the data into mysql. Next, I would like to visualise the amount of the single response codes of the crawled websites using a pie chart. I decided to use d3.js but I don't know how to visualise the total values of the response codes. For example if I have the following data stored in a json file:
{"crawl":[
{"url":"http:/www.example.com", "responseCode":"200"},
{"url":"http:/www.example.com/page1", "responseCode":"200"},
{"url":"http:/www.example.com/page2", "responseCode":"301"}
{"url":"http:/www.example.com/page3", "responseCode":"200"},
{"url":"http:/www.example.com/page4", "responseCode":"404"},
]}
is it possible to visualise how many websites returned a 200, how many returned 301 etc.? So in this case I would like to create a pie chart with the values
200 = 3
301 = 1
404 = 1
Is this possible without having the absolute values?

Related

Create histogram using different data

Im trying to create histograms using javascript, but the main problem that I ran into was data categorizing. I put here small snippets of data as an example
data set 1:
{
"Cu":[0.7,0.8,1.2,1,1.3,1.9,2.7,0.9]
}
data set 1:
{
"Cu":[4.4,5.8,0.8,1,0.6,4,1.1,2.1,1.5]
}
The problem is that each file has different data and I somehow need to create the graph as below
Example:

d3plus not load data from csv

my code:
function d3_chart() {
// sample data array
// instantiate d3plus
var visualization = d3plus.viz()
.container("#viz") // container DIV to hold the visualization
.data("./extra/acc.csv", {"filetype": "csv"}) // data to use with the visualization
.type("line") // visualization type
.y("x") // key to use for y-axis
.x("timestamp") // key to use for x-axis
.draw() // finally, draw the visualization!
}
my csv:
timestamp,x,y,z
0,2019-02-28 12:20:19.631,1.072,-0.153,10.113
1,2019-02-28 12:20:19.731,1.072,-0.153,10.419
2,2019-02-28 12:20:19.831,1.072,-0.153,9.96
3,2019-02-28 12:20:19.931,1.072,-0.153,10.113
4,2019-02-28 12:20:20.031,1.072,-0.153,10.113
5,2019-02-28 12:20:20.132,1.225,-0.153,9.96
6,2019-02-28 12:20:20.231,1.225,-0.153,9.96
7,2019-02-28 12:20:20.331,1.225,-0.153,9.96
8,2019-02-28 12:20:20.431,0.919,-0.306,9.5
9,2019-02-28 12:20:20.531,0.919,0.459,9.807
10,2019-02-28 12:20:20.631,1.225,0.153,10.113
11,2019-02-28 12:20:20.731,1.379,-1.992,10.113
12,2019-02-28 12:20:20.831,1.838,-0.306,9.653
13,2019-02-28 12:20:20.931,0.153,0.766,10.113
14,2019-02-28 12:20:21.032,0.459,1.532,10.266
15,2019-02-28 12:20:21.133,1.072,0.0,9.96
I just got getting message:
No Data Available
What is wrong? I don't find any example in internet with csv loading via this library
Or something know how graph chart from csv via general D3 with simple example?
d3plus seems to be using v3.5.15 of d3.js. Regardless of that, you will need to tell d3plus of how to load the data. Reading the API documentation it seems you will have to load the data using
d3plus.dataLoad(path, [formatter], [key], [callback]) as explained here.
Alternatively, you can use d3.js to parse your csv file and pass it as the data. To do this you can use the
d3.csv.parse(string[, accessor]) as provided in the d3.js CSV API.
Keep in mind in both cases you will need to format your timestamps in the correct time format (For d3.js Time Format API doc), for you to be able to use the time scales. Also at least for d3.js when the data is parsed from CSV all values are string values and hence you will need to change the type of the values using an type conversion function. You can read more about this in a great guide on how to read data by Learnjsdata (d3.js v3, or d3.js v5)
There are several examples out there for d3.js v3 on importing the data for processing which may be a better option overall. Also consider d3plus has not got a github commit in over a year so the library may not be well supported.
I hope this helps and at least gives you a start. If you need more help please leave a comment below.

Python Flask data feed from Pandas Dataframe, dynamically define with unique endpoint

Hi I am building a web app with Flask Python. I got a problem here:
#app.route('/analytics/signals/<ticker_url>')
def analytics_signals_com_page(ticker_url):
all_ticker = full_list
ticker_name = com_name
ticker = ticker_url.upper()
pricerec = sp500[ticker_url.upper()].tolist()
timerec = sp500[ticker_url.upper()].index.tolist()
return render_template('company.html', all_ticker=all_ticker, ticker_name=ticker_name, ticker=ticker, pricerec=pricerec, timerec=timerec)
Here I am defining company pages based on the a page will contain different content. The problem is that everything is fine upto ticker = ticker_url.upper(). It works perfectly fine. But for pricerec and timerec, they make problems.
sp500 is a pandas DataFrame columns being companies like "AAPL", "GOOG","MSFT", and so forth 505 companies and the index are timestamps, and values are the prices at each time.
So what I am doing for the pricerec, I am taking the ticker_url and use it to take the specific company's price and make it as a list. And timerec is to take the index (timestamps) and make it as a list. And I am passing these two variables into the company.html page.
But it makes internal server error. I do not know why it happens.
My expectation was that when a user click a button that href to "~/analytics/signals/aapl" then the company.html page will contain the pricerec and timerec for me to draw a graph. But it didn't work like that. It makes internal server error. I defined those two variables in the javascript also like I did for the other variables(all_ticker, ticker_name, and ticker)
Can anyone help me with this issue?
Thanks!

Dygraph not plotting graphs using csv link

I am not able to plot graph using dygraph with csv link.
If I use http://jsfiddle.net/eM2Mg/ this link it works. When I replace data with link, it just shows empty graph. I tested in debugger tool and I do get proper response from file. If I just try to plot the graph using same data from file but adding the data in javascript as static content like the example provided in jsfiddle it works.
Things I tried -
1. I tried .txt, .csv and file without extension and nothing worked
2. I tried on different data and when I insert data in static way in javascript it works. So data is in url is definitely not incorrect.
3. When checked response for url in debugger tool I get correct response
image of response
html code -
<div id="graph"></div>
Javascript -
g = new Dygraph(document.getElementById("graph"),
// For possible data formats, see http://dygraphs.com/data.html
"https://files.fm/down.php?i=8v88usam&n=testing_file_2.txt",
{
});
Your dates are in the wrong format, see Dygraphs Data Format
Here are some valid date formats for CSV:
2009-07-12
2009/07/12
2009/07/12 12
2009/07/12 12:34
2009/07/12 12:34:56

Passing MySQL data from flask to JavaScript for Google Charts

I've already checked out some similar questions here: How can I pass data from Flask to JavaScript in a template?
I'm trying to create a google chart using this example , where my data is coming from a database query, which is then parsed and formatted to be passed through Google's datasource python library, then converted into a json string. From there it is supposed to be passed through a JavaScript function in my HTML file in order to populate the Google Charts table. This last part seems to be causing me trouble. Here is the meat of my function in flask:
description = {"count": ("number", "Count"),"type": ("string", "Type")}
data=[]
rows = cur.fetchall()
# formats MySQL data for JSON string
for row in rows:
js = {'type': row[0], 'count': row[1]}
data.append(js)
# Loading it into gviz_api.DataTable (from Google Charts template)
data_table = gviz_api.DataTable(description)
data_table.LoadData(data)
# Create a JSON string. (from google charts template)
json= data_table.ToJSon(columns_order=("type", "count"), order_by="count")
render_template('myhtml.html' , json=json)
at the end of my function in flask, which is then passed through the following JavaScript function in my html file here:
<script>
...
// taken from the Google Charts example
google.load('visualization', '1', {packages:['table']});
google.setOnLoadCallback(drawTable);
var json = '{{ json }}'
function drawTable(json) {
var json_table = new google.visualization.Table(document.getElementById('table_div_json'));
var json_data = new google.visualization.DataTable((json), 0.6);
json_table.draw(json_data, {showRowNumber: true});
}
The chart is not drawn and I am left with only the header for the chart. I don't believe this is an issue with the format of my data or the HTML portion (which I've left out), for it is taken straight from the Google Charts template, so I've been trying to find problems in my JavaScript (which I am fairly new to). Can this string not be passed to the JavaScript function in this fashion? What alternative ways can it be done? Any pointers and suggestions would be great.
Probably have to escape it, something like:
{{json | safe}}

Categories

Resources