I have data saved in a Microsoft excel file. I need to turn that data into something that a Lambda function can parse.
I think the best way to do this is to convert the excel file into a JSON file (and then my Lambda function can read and parse it).
What's the best way to do this?
To convert the excel data file into a JSON file, I have found some handy online converter tools, like this one. It seems to work.
However, that converter and others add in \r wherever there are line breaks in the data, and \ wherever there are quotes in the data. (the line breaks and especially quotes need to be in the data)
So to properly read the data in the JSON file, I have to then get rid of these changes to the raw data.
Is there another way to do this? Such as a converter that does not change the raw data in this way? Or some method other than a converter?
Once the raw data has been changed (by adding in stuff like \r and \ like I mention above), it becomes cumbersome to remove it. I can do a find/replace to get rid of the changes, but that adds steps that can become costly time wise. And using regex could add performance hits.
**EDIT: Note that I probably need a method that creates an actual document (so a program that produces the data in a client browser would not work). I am looking to create an actual document that my Lambda can then analyze. **
To create a json from excel sheet, I usually prepare the json in excel sheet using excel CONCAT and then copy it. It may not be a perfect way for headers , however it works well for keys and values which generally make up a bigger portion of json.
=CONCAT(CHAR(34),A2,CHAR(34),":",CHAR(34),B2,CHAR(34))
note that CHAR(34) is stands for "
and you can drag it to down from corner for all the rows.
Related
I have this JSON file which contains the data in an interactive map displaying UK food banks run by The Trussell Trust.
food bank data
I am trying to use json.loads() and I keep getting a syntax error. I think Python may be confusing ' with " because every time there's a word like there's it interprets the ``' as the end of a string.
Basically I need to convert the data in that link into any python object.
Edit: fixed the link. Is it possible to get that data and work with it in python or do I need to learn JS?
This should work:
import urllib.request, json
with urllib.request.urlopen("https://www.trusselltrust.org/get-help/find-a-foodbank/foodbank-search/?foodbank_s=all&callback=?") as url:
data = json.loads(url.read().decode()[2:-2])
print(data[0]['foodbank_information']) # example print
The format seems to be JSONP (JSON with padding). json from python does not seem to support this format. If your link changes, especially the callback=? argument, you might want to take a look at this.
I don't know if this is the best way but I would like to have a .js file with an object that I will update once a day. I would not like to make a database for this because my code already works for the object. Via API I will get the data for the day and I would like to update the .js file. I would like to keep the historic data in this file and use it to feed the website, the API would only be used at the end of the day. It is a website with data from covid-19, I am doing it just for learning, so I am open to new approaches. I try to keep this file in github, but for edit this i need to put my user and pass in code, i dont know how turn around this issue.
Storing JavaScript objects to files is almost always done in JSON format.
Converting an object to a JSON string is done with the JSON.stringify() function.
When you read the JSON string back from the file, you covert it back to a JavaScript object with the JSON.parse() function.
I need to store some date stamped data in a JSON file. It is a sensor output. Each day the same JSON file is updated with the additional data. Now, is it possible to put some write protection on already available data to ensure that only new lines could be added to the document and no manual tampering should occur with it?
I suspect that creating checksums after every update may help, but I am not sure how do I implement it? I mean if some part of JSON file is editable then probably checksum is also editable.
Any other way for history protection?
Write protection normally only exists for complete files. So you could revoke write permissions for the file, but then also appending isn't possible anymore.
For ensuring that no tampering has taken place, the standard way would be to cryptographically sign the data. You can do this like this, in principle:
Take the contents of the file.
Add a secret key (any arbitrary string or random characters will do, the longer the better) to this string.
Create a cryptographical checksum (SHA256 hash or similar).
Append this hash to the file. (Newlines before and after.)
You can do this again every time you append something to the file. Because nobody except you knows your secret key, nobody except you will be able to produce the correct hash codes of the part of the file above the hash code.
This will not prevent tampering but it will be detectable.
This is relatively easily done using shell utilities like sha256sum for mere text files. But you have a JSON structure in a file. This is a complex case because the position in the file does not correlate with the age of the data anymore (unlike in a text file which is only being appended to).
To still achieve what you want you need to have an age information on the data. Do you have this? If you provide the JSON structure as #Rohit asked for we might be able to give more detailed advice.
Basically, I want to upload ONLY a CSV file via Javascript or jQuery.
I want to try and do this without any PHP involved.
I need to use a HTML upload form, and then save only it's contents to a multidimensional array or a string.
I do not need to save the uploaded file to the server, I just need to save it's contents to a string as stated.
I have looked far and wide online, yet everything involves PHP.
Is this possible with just Javascript or jQuery?
Thanks in advance
This uses a library I wrote and released under the GPLv3 License: html5csv
The example below uploads a CSV file into the browser, where it is available as an array of arrays.
The library supports various block operations, such as make a table, edit, plot, fit, call a function, save in browser session storage or local storage.
JSFIDDLE
html
Choose a CSV file to load into the application:
<input id='foo' type='file'>
<hr />
js (requires jQuery and html5csv.js)
CSV.begin('#foo').
table('output', {header:1, caption:'Uploaded CSV Data'}).
go();
Here, go() can take a function callback
(e,D), where e will contain an error string or null, and D is an object that may contain D.rows[0][0],...,D.rows[n-1][m-1] for a n x m matrix of data. Row 0 may be a header row.
Asynchronicity is used, in fact enforced in places. So beware that like AJAX, this code will return immediately to the subsequent line, and is best read as setting up a workflow of what to do when the previous step becomes ready.
Saving/Restoring
You can save data into the user's browser localStorage object with .save('local/someKey'). somewhere in the workflow, and data existing in the array at that point will be stored in HTML5 local storage (perhaps even compressed if you include the LZString library as documented), until the browser user deletes it.
Then in the same page or another page on the same web site you can get the data back out with CSV.begin('local/someKey')...
Using the data
You should put any code you want to use the data into a function that can fit either the callbacks expected by html5csv's call or go as documented on the html5csv site.
The jQuery CSV plugin can use client-side file handling (no need for server-side script like PHP):
https://code.google.com/p/jquery-csv/#Client-Side_File_Handling
You can use plugin which allow you to parse CSV into Array.
http://code.google.com/p/jquery-csv/
Features
Convert a CSV String to an array
Convert a multi-line CSV string to a 2D array
Convert a multi-line CSV string to an array of objects (ie header:value pairs)
Convert an array of values to CSV (under development)
Convert an array of objects to CSV (under development)
Hooks/Callbacks to extend the default parsing process
Customizable delimiter (default: ") and separator (default: ,) characters
Node.js support (ie CommonJS importing and async callback support)
To do the upload, you need to be able to read the file off the disc. You can do this with the HTMl5 File API. I'm sure there are jQuery libraries to simplify this, but that's the underlying tech.
Someone else posted a question (and solution) on how to do that with jQuery: html5's file api example with jquery?
Once you've got access to the file in the browser, use a CSV library to work with it.
I am newbie working on a 100% js prototype. It consist of 3 docs: an html page full of xml tags, a small dictionary in a text file format, and a js file with jquery.
The js needs to parse the xml tags (no problem here) and look into the mini-dictionary list for available translations.
Which is the best way to implement the mini-dictionary list. (No more than 50.000 records). Is there a way to load the list into a memory database and access it from js? Which is the usual path to take in this case? What is the simplest and machine-independent way to do this?
Any directions as to where should I research are greatly appreciated.
I would suggest encoding mini-dictionary with JSON data format, and then using AJAX to get that file and parse it. But then you are risking someone will just copy whole dictionary and steal your work.
That is, if you are not using server side language, like PHP. If you are using it, then just store everything into database and request just specific words with AJAX.