I've built a web-app that where users can upload contacts using CSV files.
Once the CSV file is uploaded, I'd like to map the header fields to the app's fields.
The issue I'm facing is figuring out an efficient and simple way to return a list of CSV headers from a CSV link.
Ideally, I'd like the list to return via an HTTP request; or alternatively, using JavaScript, so I can pass it back to the app.
For example, here is a CSV file containing hurricane counts:
https://people.sc.fsu.edu/~jburkardt/data/csv/hurricanes.csv
The headers are:
Month, "Average", "2005", "2006", "2007", "2008", "2009", "2010", "2011", "2012", "2013", "2014", "2015"
My idea is to run a Python script in AWS Lambda that gets the CSV headers, then sends them to the app using an HTTP request.
Is there another way to get this done client-side or without having to setup a backend infrustructure like AWS Lambda?
There's an option to stream from requests and you can process CSV per line. See: Body Content Workflow
Example:
import requests
url = "https://raw.githubusercontent.com/codeforamerica/ohana-api/master/data/sample-csv/addresses.csv"
r = requests.get(url, stream=True)
for line in r.iter_lines():
header = line.decode("utf-8").split(",")
print(header)
break
This will give you only the header. I've used an example of raw CSV file from github: sample-csv
The "simple" answer is obviously to download the whole file and work with that. But you may be able to save some time. If the file accepts partial downloads, you are in luck. To check that:
import requests
requests.head("https://people.sc.fsu.edu/%7Ejburkardt/data/csv/hurricanes.csv").headers
Yields
{'Date': 'Mon, 27 Dec 2021 14:00:21 GMT',
'Server': 'Apache/2.4.46 (Fedora) OpenSSL/1.1.1g',
'Last-Modified': 'Mon, 27 Jun 2016 12:37:53 GMT',
'ETag': '"1f6-53641c9fb0a40"',
'Accept-Ranges': 'bytes',
'Content-Length': '502',
'X-Frame-Options': 'SAMEORIGIN',
'Keep-Alive': 'timeout=5, max=100',
'Connection': 'Keep-Alive',
'Content-Type': 'text/csv'}
Note the Accept-Ranges: 'bytes'. You can ask for specific portions of the file. Jackpot!
Then it's a question of working with requests and the headers:
resume_headers = {'Range':'bytes=0-2048'}
r = requests.get(url, stream=True, headers=resume_headers)
for chunk in r.iter_content(chunk_size=1024):
print(chunk)
Output:
b'"Month", "Average", "2005", "2006", "2007", "2008", "2009", "2010", "2011", "2012", "2013", "2014", "2015"\n"May", 0.1, 0, 0, 1, 1, 0, 0, 0, 2, 0, 0, 0 \n"Jun", 0.5, 2, 1, 1, 0, 0, 1, 1, 2, 2, 0, 1\n"Jul", 0.7, 5, 1, 1, 2, 0, 1, 3, 0, 2, 2, 1\n"Aug", 2.3, 6, 3, 2, 4, 4, 4, 7, 8, 2, 2, 3\n"Sep", 3.5, 6, 4, 7, 4, 2, 8, 5, 2, 5, 2, 5\n"Oct", 2.0, 8, 0, 1, 3, 2, 5, 1, 5, 2, 3, 0\n"Nov", 0.5, 3, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1\n"Dec", 0.0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1\n\n'
So we have our header, as well as a couple more lines.
Note that I print, but you can save into a variable, a large string, write to a local CSV file,... whatever suits your need. Also, I selected a range of 2048 bytes, and chunks of 1024 bytes; adapt to the situation!
Related
Edit to add: Looking for the "plotly.js" way to do this. This "small multiple" visualization should have some "plotly.js" solution out there but haven't found it yet.
I am using an array (example element below) to populate traces for plotly.js multiple subplots per their multiple-subplots example
[{
"key": "Ontario|Toronto",
"values": [{
"key": "2020-01-25",
"value": 1
}, {
"key": "2020-01-27",
"value": 1
}, {
"key": "2020-05-12",
"value": 218
}, {
"key": "2020-05-13",
"value": 169
}]
}, {
etc
}]
The array has 94 elements which contain info needed to create each trace. This would result in 94 subplots, one per trace. This plotly.js visualization could also be called "small multiples".
I am creating the traces dynamically and populating subplot definitions in a loop using code below:
// create chart data
var traces = [];
var rowCount = (caseRegionByDate.length / 2).toFixed()
for (var i=0; i<caseRegionByDate.length; i++) {
//console.log(caseRegionByDate[i]['key']);
var trace = {};
var x = [];
var y = [];
for (var j=0; j<caseRegionByDate[i]['values'].length; j++) {
//console.log(caseRegionByDate[i]['values'][j]['key']);
x.push(caseRegionByDate[i]['values'][j]['key']);
y.push(caseRegionByDate[i]['values'][j]['value']);
}
// create trace i
trace = {
"x":x,
"y":y,
"xaxis":"x"+i,
"yaxis":"y"+i,
"type":"scatter"
}
// push trace to traces
traces.push(trace);
}
console.log(JSON.stringify(traces));
var layout = {
grid: {rows: rowCount, columns: 2, pattern: 'independent'},
};
Plotly.newPlot('multiple_charts', traces, layout);
This creates the traces variable populated by each trace that looks like example below. It looks correct:
[{
"x": ["2020-03-16", "2020-03-23", "2020-03-24", "2020-03-25", "2020-03-31", "2020-04-01", "2020-04-02", "2020-04-03", "2020-04-06", "2020-04-07", "2020-04-08", "2020-04-09", "2020-04-10", "2020-04-11", "2020-04-13", "2020-04-14", "2020-04-15", "2020-04-16", "2020-04-17", "2020-04-18", "2020-04-21", "2020-04-22", "2020-04-23", "2020-04-24", "2020-04-25", "2020-04-26", "2020-04-27", "2020-04-28", "2020-04-29", "2020-04-30", "2020-05-01", "2020-05-02", "2020-05-03", "2020-05-04", "2020-05-05", "2020-05-06", "2020-05-07", "2020-05-08", "2020-05-09", "2020-05-10", "2020-05-11", "2020-05-12", "2020-05-13"],
"y": [1, 1, 1, 1, 9, 35, 3, 16, 33, 13, 9, 5, 5, 1, 22, 3, 4, 7, 19, 4, 7, 2, 18, 11, 9, 9, 9, 13, 1, 3, 7, 18, 5, 4, 4, 5, 3, 1, 2, 2, 3, 1, 2],
"xaxis": "x0",
"yaxis": "y0",
"type": "scatter"
}, {
"x": ["2020-03-14", "2020-03-26", "2020-03-27", "2020-04-02", "2020-04-06", "2020-04-09", "2020-04-14", "2020-04-17", "2020-04-18", "2020-04-20", "2020-04-22"],
"y": [1, 1, 1, 2, 2, 3, 1, 1, 1, 2, 1],
"xaxis": "x1",
"yaxis": "y1",
"type": "scatter"
},
etc
]
However, the result appears to be one row with two columns that have all of the traces (there are 94 traces) squashed into them. Here is screenshot.
Any ideas what is happening? I expect to have 48 rows with 2 columns, one subplot per trace.
The only difference from the multiple subplots example is that my xaxis have date strings instead of numbers. Everything else is same.
The subplots are actually being created in 3 x rowCount grid. However they are all squashed vertically as in screenshot.
It appears that a chart's default height and width dimensions, where they are not explicitly defined using layout.height, are what is shown in my screenshot eg too small for 94 subplots.
The quick fix is to simply increase the chart's layout.height size. Then all subplots are visible. Dynamically calculating layout.height, in spirit of Juan's suggestion, relative to number of rows works well.
Apparently it is also possible to set each subplot's x and y domain attributes to resize subplots which will also give desired results.
I know ton of library can prettyprint json just by indenting/newline stuff but here is a line of my heavy json:
"Shape6":{"bounds_start":[0,-6,0],"bounds_end":[3,1,3],"origin":[2,15,-1],"mirror":true,"rotation":[0,0,0.837758],"uv":[15,30]}
All the libraries i found output something like this:
"Shape6": {
"bounds_start": [
0,
-6,
0
],
"bounds_end": [
3,
1,
3
],
"origin": [
2,
15,
-1
],
"mirror": true,
"rotation": [
0,
0,
0.837758
],
"uv": [
15,
30
]
}
But i'm looking for a more human-readable way which not add new lines for small arrays that can fit in a line like:
"Shape6": {
"bounds_start": [0, -6, 0],
"bounds_end": [3, 1, 3],
"origin": [2, 15, -1],
"mirror": true,
"rotation": [0, 0, 0.837758],
"uv": [15, 30]
}
i do want this because my json file is like 6k+ lines on the first example
if you know a js or a php library (for ajax purposes)
i thank you in advance (and sorry for my poor english :))
you can simply do that with using JSON.stringify() function (see the reference https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify)
the first argument is the object you want to beautify, the second one is the function where we can specify the logic of line indention and new line breakings, the 3rd argument of this function (now 4) is the number for indention for your beautified JSON data
JSON.stringify(obj, function(k,v)
{
if (v instanceof Array)
return JSON.stringify(v);
return v;
}, 4);
I need to call a RESTful web service using SSIS.
The web service accepts a JSON object as a parameter - I can call this service by constructing the JSON object using JavaScript:
data = {
CustomerKey: "Demo",
ApiKey: " 9999AA99-999A-9A9A-99A9-99999999A999",
RegistrationNo: "REG N02",
InsuranceRef: "Ins Ref",
VehicleType: "Lorry",
CarMake: "Car Make",
CarModel: "Car Model",
ChassisNo: "ChassisNo",
GrossPlatedWeight: "GP weight",
ManufactureYear: 1999,
VehicleStatus: "Hired",
DeliveryDate: null,
TransferDate: new Date(2004, 4, 1, 0, 0, 0).toMSDate(),
LastMotDate: new Date(2013, 12, 1, 0, 0, 0).toMSDate(),
LastInspectionDate: new Date(2013, 1, 11, 0, 0, 0).toMSDate(),
LastServiceDate: new Date(2014, 2, 2, 0, 0, 0).toMSDate(),
InformationLog: "information log and more information",
VehicleNotes: "Vehicle Notes",
EntityReference: "reference",
UseRootOrgUnit: "true",
AddDictionaryItemIfMissing: "true"};
and posting the object to the web service. The web service also returns a JSON object.
However, javascript is not available to me in SSIS - is it possible to call such a web service using SSIS? If so, can anyone help me with a link to show the steps necessary? (I have no experience with SSIS).
Thanks.
I am working on one game application. Intially I made two loaclstorage, one is userCard and another CompCard and I added 3 cricket players cards details to both the local storage.
My 2 local storage and its deatils :
var user_Card = [["Nixon", "McLean", "West Indies",
45, 314, 0, 1, "12.07", "37.58",
46, 3, 21, 8,
"img/cards/7RBKWQPJAG_NixonMcLean.jpg", 1],
["Brian", "McMillan", "South Africa", 78,
841, 1, 0, "23.36", "36.98", 70, 4, 32, 43,
"img/cards/Y9U5UKA60O_BrianMcMillan.jpg", 2],
["Craig", "McMillan", "New Zealand", 197, 4707, 3,
28, "28.18", "35.04", 49, 3, 20, 44,
"img/cards/WE0NUNG80C_CraigMcMillan.jpg", 3]
];
var comp_Card = [["Geoff", "Miller", "England", 25, 136, 0, 0,
"8.5", "32.52", 25, 3, 27, 4,
"img/cards/7ZPIQXC19H_GeoffMiller.jpg", 4],
["Kyle", "Mills", "New Zealand", 165, 1016,
0, 2, "15.87", "26.74", 235, 5, 25, 40,
"img/cards/P43DTA4ZCX_KyleMills.jpg", 5],
["Minhazul", "Adedin", "Bangladesh", 27, 453, 0, 2,
"18.87", "39.3", 13, 2, 39, 2,
"img/cards/CC8ENY3E09_MinhazulAdedin.jpg", 6]
];
userCard =JSON.parse(localStorage['user_Card']);
compCard =JSON.parse(localStorage['comp_Card']);
When I retrieving all cards details are coming, usercard[0] and compcard [0] is the first card, usercard[1] is the second card, usercard[2] is the third card,
My question is when I play the game, the if usercard wins, I want add the loosing compcard to usercard local storgae. And if compcard wins add the loosing usercard to compcard local storage.
Assume userCard[0] is the first card
compcard[0] is the first card
When I playing the game,if userCard[0] wins, I want to add loosing compcard[0] to localstorage userCard and if compCard[0] wins, I want to add loosing usercard[0] to localstorage compCard
Please provide me the the solution for this.
How you can read from Offline Technologies
Web Storage simply provides a key-value mapping, e.g.
localStorage["name"] = username;. Unfortunately, present
implementations only support string-to-string mappings, so you need to
serialise and de-serialise other data structures. You can do so using
JSON.stringify() and JSON.parse().
so you should try something like this:
var userCard =JSON.parse(localStorage.['user_Card']);
userCard.push(compCard[0]);
localStorage.['user_Card'] = JSON.stringify(userCard));
I need to create a stacked bar chart showing the engine status in a day. Here is the example of what I would like to have:
It looks like a gantt chart, but probably much simpler than a normal gantt chart. I am badly looking for a JavaScript/jQuery charting library which supports this kind of chart. I know lots of gantt chart library available, but wondering which library have the option/setting for the chart I want.
My data would be in this format:
[
{
"day": "2009-07-13",
"work": ["11:16:35-12:03:12", "12:32:48-13:26:28", "13:39:09-13:39:12", "13:41:03-13:41:05", "14:18:09-24:00:00"]
}, {
"day": "2009-07-14",
"work": ["00:00:00-07:22:25", "07:22:25-07:22:28", "10:10:04-10:10:31", "10:10:32-10:15:33", "10:18:07-10:21:19", "11:04:49-11:06:15", "11:12:50-11:19:05", "11:19:11-11:19:19", "11:45:50-11:51:42", "11:51:43-11:53:55", "14:03:13-14:13:04", "14:23:55-14:31:28", "14:31:28-14:38:00", "14:38:00-14:49:04", "16:34:56-16:44:33", "16:46:37-16:48:10", "16:48:11-24:00:00"]
}, {
"day": "2009-07-15",
"work": ["00:00:00-08:16:23", "09:57:57-10:15:05"]
}, {
"day": "2009-07-16",
"work": ["10:02:40-10:05:56", "10:07:16-10:09:26", "10:09:27-10:09:28", "13:18:31-24:00:00"]
}, {
"day": "2009-07-17",
"work": ["00:00:00-08:56:41", "16:07:58-16:08:23"]
}, {
"day": "2009-07-20",
"work": ["14:44:47-14:48:35", "15:09:14-16:47:06", "16:47:05-16:47:10", "16:47:13-16:47:15", "16:47:16-16:47:20"]
}, {
"day": "2009-07-21",
"work": ["10:52:51-16:37:07"]
}, {
"day": "2009-07-24",
"work": ["14:54:38-16:03:07", "16:16:23-16:35:14", "16:35:17-16:41:22", "16:43:37-23:56:37"]
}, {
"day": "2009-07-25",
"work": ["20:36:34-21:24:28", "21:24:43-23:45:53"]
}, {
"day": "2009-07-26",
"work": ["13:46:59-18:09:09"]
}, {
"day": "2009-07-28",
"work": ["13:48:30-13:51:10", "13:51:18-13:51:27", "13:52:17-14:57:31"]
}, {
"day": "2009-07-29",
"work": ["14:50:15-14:50:16", "15:36:17-15:43:51", "15:53:31-16:29:30", "16:57:50-23:07:28"]
}, {
"day": "2009-07-30",
"work": ["11:25:29-11:41:32", "16:06:37-16:33:09", "21:14:04-21:20:18", "21:53:57-22:18:59"]
}
]
The work attribute time slot is when the engine is working, the slots between work time slots is when the engine is off.
Have been looking for this for long. Any suggestion would be greatly appreaciated!
You could use JavaScript InfoVis Toolkit or make your own custom renderer.
Maybe you could modify BarChart example, so that it would display time in y-axis.
If you decide to write your own control then library like Raphael will help you a lot.
Anyways, it seems to be quite simple control, so there isn't any need for external dependencies to Flash, Silverlight etc.
You can try Flot which has a good looking Gantt chart plugin.
Example data:
var d1 = [
[Date.UTC(2010, 0, 1, 11, 23), 5, Date.UTC(2010, 0, 1, 11, 25), "Put Water into Pot"],
[Date.UTC(2010, 0, 1, 11, 35), 5, Date.UTC(2010, 0, 1, 11, 47), "Clean Cooker"],
[Date.UTC(2010, 0, 1, 11, 25), 4, Date.UTC(2010, 0, 1, 11, 27), "Put Pot on Cooker"]
]
From plugin specification:
var data = [
[Date.UTC(2010,0,25,12,45),1,Date.UTC(2010,0,25,13,15],"Name of Step"]
First Parameter is Start of Step.
Second is number of resource.
Third is End of step.
Fourth describes Name for step (used for tooltip).
I'm working on a timeline feature for jqplot. http://www.jqplot.com/
I've a JsFiddle with an example here: http://jsfiddle.net/NVbjv/8/
It's still work in progress and I need to figure out a few things. Take a look at some of my questions here at stackoverflow if you like to learn more. I hope to be able to evolve it in somekind of a plug-in.
( display pointlabel in highlighter jqplot , jqplot text labels on y-axis )