I am trying to use a variable which I created using PyScript on one page in another page. This post initially seems to propose a solution, but when importing the variables, it turns out that they have been stored using (what I assume to be) their __repr__() function. That is obviously problematic for, among others, Pandas objects.
Page 1:
<py-script>
# Code that generates these objects is omitted
js.localStorage.setItem("dtypes", dtypes)
js.localStorage.setItem("df", df)
js.localStorage.setItem("target", target)
</py-script>
Page 2:
<py-script>
import js
dtypes = js.localStorage.getItem("dtypes")
df = js.localStorage.getItem("df")
target = js.localStorage.getItem("target")
js.console.log(dtypes) # Returns the stored dictionary as a string
</py-script>
Is there a way to retain these variables as Python objects?
It turns out that I misunderstood what setItem() and getItem() do. From the documentation, it follows that value must be a string (and otherwise will apparently be converted to a string representation).
Thus, my solution for storing Pandas DataFrames or Series is as follows, using to_csv without specifying the path_or_buf argument:
Page 1:
js.sessionStorage.setItem("df", df.to_csv())
Page 2:
df = js.sessionStorage.getItem("df")
df = io.StringIO(df)
df = pd.read_csv(df)
Related
I have been struggling with a site I am scrapping using scrappy.
This site, returns a series of Javascript variables (array) with the products data.
Example:
datos[0] = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
datos[1] = ["12346","3M GREEN CAT5E CABLE","7.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1,"",\'\'];
...
So on...
Fetching the array into a string with scrapy was easy, since the site response prints the variables.
The problem is I want to transform it into Json so I can process it and store it in a database table.
Normally I would use Javascript's function Json.stringify to convert it to Json and post it in PHP.
However when using Python's json.loads and even StringIO I am unable to load the array into json.
Probably is a format error, but I am unable to identify it, since I am not expert in Json nor Python.
EDIT:
I just realize since scrapy is unable to execute Javascript probably the main issue is that the data is just a string. I should format it into a Json format.
Any help is more than welcome.
Thank you.
If you wanted to take an array and create a json object, you could do something like this.
values = ["12345","3M YELLOW CAT5E CABLE","6.81","1","A","N","N","N","N","N",0,0,0,0,0,"0","0","0","0","0","P","001-0030","12","40K8957","28396","250","Due: 30-12-1899",0.0000,1]
keys = [x for x in range(len(values))]
d = dict(zip(keys, values))
x = json.dumps(d)
There is a section in the scrapy doc to find various ways to parse the JavaScript code. For your case, if you just need to have it in an array, you can use the regex to get the data.
Since the website you are scraping is not present in the question, I am assuming this would be a more straightforward way to get it, but you could use whichever way seems suitable.
Background:
I am using PlayFramework to create a webapp (workflow) system which uses the Twirl template engine (scala based) to create my views.
Problem:
I need to store JavaScript variables inside of my template (which renders HTML), where in a separate JS file(s), I will perform operations on later.
Any existing solutions?
This can be done according to these SO questions I found:
This one creates a map where data is stored in and later retrieved.
This one creates a simple example of storing a variable in an inline script tag to use later
Finally, this one uses a more concrete example of storing data in data structures in the template and accessing the data structures in JS after being parsed.
The example created by an active SO contributor, Biesior, helping many developers with Play! Framework questions
To summarise what should be done:
Pass in some data into template
#(myData: DataObject)
Add data passed in an into a script tag with a JS name
<html>
<body>
<script>
let someJSDataName = #myData
</script>
</body>
</html>
Use in JS / Google Chrome Dev console
someJSDataName.toString
This should display some result!
But what is the problem?
Let me show you.
Rendered Html:
//...
<div class="BoxMediumBlue" style="padding: 20px;">
<script>
let facultyDepartments = {Science=[Computer Science, Physics], Biology=[Zooology]};
</script>
<br>
<div class="container-heading">
//...
When attempting to access this data in my Google Chrome dev console:
facultyDepartments
VM1209:1 Uncaught ReferenceError: facultyDepartments is not defined
at <anonymous>:1:1
Just an FYI, using var makes no difference
Am I doing something wrong?
Your definition of facultyDepartments uses syntax that the JS engine doesn't understand:
let facultyDepartments = {Science=[Computer Science, Physics], Biology=[Zooology]};
Objects should contain key-value pairs, where the keys and values are separated by :s, and where the keys and values, when strings, have delimiters like " or '. You should try to get the template to render JSON instead, for example:
let facultyDepartments = {"Science":["Computer Science", "Physics"], "Biology":["Zooology"]};
console.log(facultyDepartments.Science);
(JSON is technically a way of formatting strings, but inserting a JSON string without delimiters so that it's parsed as an object literal works too)
This question already has answers here:
JavaScript raises SyntaxError with data rendered in Jinja template
(3 answers)
Closed 6 years ago.
I am sitting on a Flask based webapplication. In theory I want to load a JSON file from disk and give it to javascript on the website.
def getData():
check_for_update()
with open(LOCAL_file,"rb") as myfile:
data = json.load(myfile)
udate = data["today"]
return (udate, data)
then I send it to the page with
return render_template('./index2.html', udate = thisdata[0], data = json.dumps(thisdata[1]))
Now on the page I simply try
<script>
var myjson = JSON.parse({{data}})
</script>
which then results in something like this
This can't not be parsed.When I copy and paste it it works fine, and python does not complain either.
data is HTML escaped, because Jinja2 by default escapes everything to be safe to embed in an HTML page.
It's much better to not encode to JSON in the view, do this in the template instead, and use the Flask tojson and safe filters.
So in the view pass in thisdata[1] unencoded:
return render_template(
'./index2.html', udate=thisdata[0], data=thisdata[1])
and in the view:
<script>
var myjson = {{ data|tojson|safe }};
</script>
tojson produces JSON data that is also HTML-safe (albeit with " quotes, so it is not suitable for embedding in a HTML tag attribute), and the safe filter can be used to switch off the HTML-encoding. There is no need to use JSON.parse() here, the resulting JSON produced by tojson is a strict JavaScript subset.
See the JSON Support section in the API documentation:
The htmlsafe_dumps() function of this json module is also available as filter called |tojson in Jinja2. Note that inside script tags no escaping must take place, so make sure to disable escaping with |safe if you intend to use it inside script tags[.]
and the Standard Filters section of the Flask Templates documentation:
tojson()
This function converts the given object into JSON representation. This is for example very helpful if you try to generate JavaScript on the fly.
So I'm trying to use Google Map suggest API to request place name suggestions. Unfortunately I can't find the docs for this bit.
Here is an example URI:
http://maps.google.com/maps/suggest?q=lon&cp=3&ll=55.0,-3.5&spn=11.9,1.2&hl=en&gl=uk&v=2
which returns:
{suggestion:[{query:"London",...
I want to use this in python (2.5). Now in proper JSON there would have been quotations around the keys like so:
{"suggestion":[{"query":"London",...
and I could have used simplejson or something, but as it is I'm a bit stuck.
There are two possible solutions here; either I can get to the API code and find an option to return proper JSON, or I do that in python.
Any ideas please.
Ugh, that's indeed pretty annoying. It's a JavaScript literal but it — pointlessly — isn't JSON.
In theory you are supposed to be able to import json.decoder.JSONDecoder from the Python stdlib (or simplejson pre-2.6, which is the same) and subclass it, then pass that subclass to json.loads to override decoder behaviour. In reality this isn't really feasible as json.decoder is full of global cross-references that resist subclassing, and the bit you need to change is slap bang in the middle of def JSONObject.
So it's probably worth looking at other Python JSON libraries. I found this one which, in ‘non-strict’ mode, will parse unquoted object property names:
>>> import demjson
>>> demjson.decode('{suggestion:[{query:"London",interpretation: ...')
{u'suggestion': [{u'query': u'London', u'operation': 2, u'interpretation': ...
I would try to poke around in order to get JSON, but failing that there's this little monstrosity which someone will inevitably yell at me about:
class Iden(object):
def __getitem__(name, index):
return index
notjson = '{...}'
data = eval(notjson, {}, Iden())
import demjson
demjson.decode(google.js)
I found this when trying to parse Google Finance option "JSON" data, which, as many note, isn't JSON-compliant.
demjson saved me writing an obnoxious regex string; it just works.
How do you fix a names mismatch problem, if the client-side names are keywords or reserved words in the server-side language you are using?
The DOJO JavaScript toolkit has a QueryReadStore class that you can subclass to submit REST patterned queries to the server. I'm using this in conjunction w/ the FilteringSelect Dijit.
I can subclass the QueryReadStore and specify the parameters and arguments getting passed to the server. But somewhere along the way, a "start" and "count" parameter are being passed from the client to the server. I went into the API and discovered that the QueryReadStore.js is sending those parameter names.
I'm using Fiddler to confirm what's actually being sent and brought back. The server response is telling me I have a parameter names mismatch, because of the "start" and "count" parameters. The problem is, I can't use "start" and "count" in PL/SQL.
Workaround or correct implementation advice would be appreciated...thx.
//I tried putting the code snippet in here, but since it's largely HTML, that didn't work so well.
While it feels like the wrong thing to do, because I'm hacking at a well tested, nicely written JavaScript toolkit, this is how I fixed the problem:
I went into the DOJOX QueryReadStore.js and replaced the "start" and "count" references with acceptable (to the server-side language) parameter names.
I would have like to handled the issue via my PL/SQL (but I don't know how to get around reserved words) or client-side code (subclassing did not do the trick)...without getting into the internals of the library. But it works, and I can move on.
As opposed to removing it from the API, as you mentioned, you can actually create a subclass with your own fetch, and remove start/count parameters (theoretically). Have a look at this URL for guidance:
http://www.sitepen.com/blog/2008/06/25/web-service-data-store/
Start and count are actually very useful because they allow you to pass params for the query that you can use to filter massive data sets, and it helps to manage client-side paging. I would try to subclass instead, intercept, and remove.
Is your pl/sql program accessed via a URL and mod_plsql? If so, then you can use "flexible parameter passing" and the variables get assigned to an array of name/value pairs.
Define your package spec like this...
create or replace package pkg_name
TYPE plsqltable
IS
TABLE OF VARCHAR2 (32000)
INDEX BY BINARY_INTEGER;
empty plsqltable;
PROCEDURE api (name_array IN plsqltable DEFAULT empty ,
value_array IN plsqltable DEFAULT empty
);
END pkg_name;
Then the body:
CREATE OR REPLACE PACKAGE BODY pkg_name AS
l_count_value number;
l_start_value number;
PROCEDURE proc_name (name_array IN plsqltable DEFAULT empty,
value_array IN plsqltable DEFAULT empty) is
------------
FUNCTION get_value (p_name IN VARCHAR) RETURN VARCHAR2 IS
BEGIN
FOR i IN 1..name_array.COUNT LOOP
IF UPPER(name_array(i)) = UPPER(p_name) THEN
RETURN value_array(i);
END IF;
END LOOP;
RETURN NULL;
END get_value;
----------------------
begin
l_count_value := get_value('count');
l_start_value := get_value('start');
end api;
end pkg_name;
Then you can call pkg_name.api using
http://server/dad/!pkg_name.api?start=3&count=3