I'm using Flask and want to render html pages and directly focus on a particular dom element using /#:id.
Below is the code for default / rendering
#app.route('/')
def my_form():
return render_template("my-form.html")
Below is the function being called upon POST request.
#app.route('/', methods=['POST'])
def my_form_post():
#...code goes here
return render_template("my-form.html")
I want to render my my-form.html page as my-form.html/#output so that it should directly focus upon the desired element in the dom.
But trying return render_template("my-form.html/#output") tells that there's no such file and trying #app.route('/#output', methods=['POST']) doesn't work either.
UPDATE:
Consider this web-app JSON2HTML - http://json2html.herokuapp.com/
What is happening: Whenever a person clicks send button the textarea input and the styling setting is send over to my python backend flask-app as below.
#app.route('/', methods=['POST'])
def my_form_post():
#...code for converting json 2 html goes here
return render_template("my-form.html",data = processed_data)
What I want: Whenever the send button is clicked and the form data is POSTED and processed and the same page is redirected with a new parameter which contains the processed_data to be displayed. My problem is to render the same page appending the fragment identifier #outputTable so that after the conversion the page directly focuses on the desired output the user wants.
The fragment identifier part of the URL is never sent to the server, so Flask does not have that. This is supposed to be handled in the browser. Within Javascript you can access this element as window.location.hash.
If you need to do your highlighting on the server then you need to look for an alternative way of indicating what to highlight that the server receives so that it can give it to the template. For example:
# focus element in the query string
# http://example.com/route?id=123
#app.route('/route')
def route():
id = request.args.get('id')
# id will be the given id or None if not available
return render_template('my-form.html', id = id)
And here is another way:
# focus element in the URL
#http://example.com/route/123
#app.route('/route')
#app.route('/route/<id>')
def route(id = None):
# id is sent as an argument
return render_template('my-form.html', id = id)
Response to the Edit: as I said above, the fragment identifier is handled by the web browser, Flask never sees it. To begin, you have to add <a name="outputTable">Output</a> to your template in the part you want to jump to. You then have two options: The hacky one is to write the action attribute of your form including the hashtag. The better choice is to add an onload Javascript event handler that jumps after the page has loaded.
Related
I want to make a general scraper which can crawl and scrape all data from any type of website including AJAX websites. I have extensively searched the internet but could not find any proper link which can explain me how Scrapy and Splash together can scrape AJAX websites(which includes pagination,form data and clicking on button before page is displayed). Every link I have referred tells me that Javascript websites can be rendered using Splash but there's no good tutorial/explanation about using Splash to render JS websites. Please don't give me solutions related to using browsers(I want to do everything programmatically,headless browser suggestions are welcome..but I want to use Splash).
class FlipSpider(CrawlSpider):
name = "flip"
allowed_domains = ["www.amazon.com"]
start_urls = ['https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=mobile']
rules = (Rule(LinkExtractor(), callback='lol', follow=True),
def parse_start_url(self,response):
yield scrapy.Request(response.url,
self.lol,
meta={'splash':{'endpoint':'render.html','args':{'wait': 5,'iframes':1,}}})
def lol(self, response):
"""
Some code
"""
The problem with Splash and pagination is following:
I wasn't able to product a Lua script that delivers a new webpage (after click on pagination link) that is in format of response. and not pure HTML.
So, my solution is following - to click the link and extract that new generated url and direct a crawler to this new url.
So, I on the page that has pagination link I execute
yield SplashRequest(url=response.url, callback=self.get_url, endpoint="execute", args={'lua_source': script})
with following Lua script
def parse_categories(self, response):
script = """
function main(splash)
assert(splash:go(splash.args.url))
splash:wait(1)
splash:runjs('document.querySelectorAll(".next-page")[0].click()')
splash:wait(1)
return splash:url()
end
"""
and the get_url function
def get_url(self,response):
yield SplashRequest(url=response.body_as_unicode(), callback=self.parse_categories)
This way I was able to loop my queries.
Same way if you don't expect new URL your Lua script can just produce pure html that you have to work our with regex (that is bad) - but this is the best I was able to do.
You can emulate behaviors, like a ckick, or scroll, by writting a JavaScript function and by telling Splash to execute that script when it renders your page.
A little exemple:
You define a JavaScript function that selects an element in the page and then clicks on it:
(source: splash doc)
# Get button element dimensions with javascript and perform mouse click.
_script = """
function main(splash)
assert(splash:go(splash.args.url))
local get_dimensions = splash:jsfunc([[
function () {
var rect = document.getElementById('button').getClientRects()[0];
return {"x": rect.left, "y": rect.top}
}
]])
splash:set_viewport_full()
splash:wait(0.1)
local dimensions = get_dimensions()
splash:mouse_click(dimensions.x, dimensions.y)
-- Wait split second to allow event to propagate.
splash:wait(0.1)
return splash:html()
end
"""
Then, when you request, you modify the endpoint and set it to "execute", and you add "lua_script": _script to the args.
Exemple :
def parse(self, response):
yield SplashRequest(response.url, self.parse_elem,
endpoint="execute",
args={"lua_source": _script})
You will find all the informations about splash scripting here
I just answered a similar question here: scraping ajax based pagination. My solution is to get the current and last pages and then replace the page variable in the request URL.
Also - the other thing you can do is look on the network tab in the browser dev tools and see if you can identify any API that is called. If you look at the requests under XHR you can see those that return json.
You can then call the API directly and parse the json/ html response. Here is the link from the scrapy docs:The Network-tool
I have a custom template tag that returns suppose name of a student and roll number if passed as an argument id of the student.
#register.inclusion_tag('snippet/student_name.html')
def st_name_tag(profile, disp_roll=True, disp_name=True):
#some calculations
return {'full_name':student.name,
'roll':student.roll_number,
}
The template(included) consists of some Html file which is written in a single line(to avoid unterminated string literal error from js).
I simply want to call the st_name_tag from inside the JS function.
My JS looks like:
{% load profile_tag %}
<script type = "text/javascript">
eventclick : function(st){
var div = ('<div></div>');
var st_id = st.id;
if (st.status == 'pass'){
div.append('<p>Student Name:{% st_name_tag '+st_id+' %}</p>');
}
}
So far I tried the above method along with removing the + and '' signs from st_id varaible. That hasnt helped me at all. Help Please!
You are trying to render a template based on the interaction by user. The first happens on the server (server-side as it is often referred to), and the latter happens on the user's browser.
The order that these happen is first to render the template on server, send and present in browser, then user interacts with js. Because of this fact, as I mentioned in the comment, it is not possible to affect the template rendered within javascript.
I would recommend you to use ajax in order to accomplish this. Whenever an iteraction occurs, you asynchronously make a request to the server to present you with new data.
Rendering HTML from a python String in web2py
I am trying to render an anchor link in an html file generated server side in web2py
#username
and the link generates correctly; however when I call it in my view {{=link}} the page does not render it as HTML. I have tried using
mystring.decode('utf-8')
and various other conversions. Passing it to javascript and back to the page displays the link fine. Is there something specific about python strings that do not communicate well with html?
In the controller the string is generated by the function call:
#code barrowed from luca de alfaro's ucsc cmps183 class examples
def regex_text(s):
def makelink(match):
# The title is the matched praase #username
title = match.group(0).strip()
# The page is the striped title 'username' lowercase
page = match.group(1).lower()
return '%s' % (A(title, _href=URL('default', 'profile', args=[page])))
return re.sub(REGEX,makelink, s)
def linkify(s):
return regex_text(s)
def represent_links(s, v):
return linkify(s)
which replaces #username with a link to their profile and args(0) = username and is sent to the view by a controller call
def profile():
link = linkify(string)
return dict(link=link)
For security, web2py templates will automatically escape any text inserted via {{=...}}. To disable the escaping, you can wrap the text in the XML() helper:
{{=XML(link)}}
I am not sure if I worded my question correctly. I'm not actually sure how to go about this at all.
I have a site load.html. Here I can use a textbox to enter an ID, for example 123, and the page will display some information (retrieved via a Javascript function that calls AJAX from the Flask server).
I also have a site, account.html. Here it displays all the IDs associated with an account.
I want to make it so if you click the ID in account.html, it will go to load.html and show the information required.
Basically, after I press the link, I need to change the URL to load.html, then call the Javascript function to display the information associated with the ID.
My original thoughts were to use variable routes in Flask, like #app.route('/load/<int:id>') instead of simply #app.route('/load')
But all /load does is show load.html, not actually load the information. That is done in the Javascript function I talked about earlier.
I'm not sure how to go about doing this. Any ideas?
If I need to explain more, please let me know. Thanks!
To make this more clear, I can go to load.html and call the Javascript function from the web console and it works fine. I'm just not sure how to do this with variable routes in Flask (is that the right way?) since showing the information depends on some Javascript to parse the data returned by Flask.
Flask code loading load.html
#app.route('/load')
def load():
return render_template('load.html')
Flask code returning information
#app.route('/retrieve')
def retrieve():
return jsonify({
'in':in(),
'sb':sb(),
'td':td()
})
/retrieve just returns a data structure from the database that is then parsed by the Javascript and output into the HTML. Now that I think about it, I suppose the variable route has to be in retrieve? Right now I'm using AJAX to send an ID over, should I change that to /retrieve/<int:id>? But how exactly would I retrieve the information, from, example, /retrieve/5? In AJAX I can just have data under the success method, but not for a simple web address.
Suppose if you are passing the data into retrieve from the browser url as
www.example.com/retrieve?Data=5
you can get the data value like
dataValue = request.args.get('Data')
You can specify param in url like /retrieve/<page>
It can use several ways in flask.
One way is
#app.route('/retrieve/', defaults={'page': 0})
#app.route('/retrieve/<page>')
def retrieve():
if page:
#Do page stuff here
return jsonify({
'in':in(),
'sb':sb(),
'td':td()})
Another way is
#app.route('/retrieve/<page>')
def retrieve(page=0):
if page:
#Do your page stuff hear
return jsonify({
'in':in(),
'sb':sb(),
'td':td()
})
Note: You can specify converter also like <int:page>
I need to check that the user is still logged in so that I can prevent them from even opening the form if their session expired (logged out but page hasn't been refreshed).
Here's the pseudo code of what I was trying to do, doesn't work obviously.
// some view.html
$('#someform').click(function() {
ajax(URL('login_status'), [], '');
});
// some controller.py
def login_status():
if not auth.user:
redirect('index')
If you want the full page to redirect as the result of an Ajax request, you'll have to do it via Javascript on the client rather than via a server-side redirect (which will only redirect the Ajax request itself). To make that happen, you can specify ":eval" as the target in the ajax() function, which will result in the returned response being executed as Javascript code (see here). So, something like:
$('#someform').click(function() {
ajax("{{=URL('default', 'login_status')}}", [], ':eval');
});
// some controller.py
def login_status():
if auth.user:
return 'this.show()' # or appropriate Javascript code to show the form
else:
return 'window.location = "%s"' % URL('default', 'index')
So, the Ajax call returns Javascript that either shows the form or redirects the page, depending on whether the user is logged in.
Also, note that the URL() function is a server-side Python function, so you can't just call it within the ajax() function, which is a client-side Javascript function. Instead, you have to put it inside web2py's template delimiters ({{ }}) so the URL will be generated in the template on the server.
UPDATE: Instead of:
return 'window.location = "%s"' % URL('default', 'index')
you can now do:
redirect(URL('default', 'index'), client_side=True)
which will do the same thing behind the scenes.
Also, this might be of help:
{{if auth.is_logged_in():}}