Develop Reference

How to download data behind buttons which call javascript functions?

I was trying to download this data from the website.
https://www.nseindia.com/market-data/oi-spurts
How can scrape this using python?

The JavaScript function downloadCSV is part of gijgo.min.js. It invokes getCSV, which goes through the fields in the table and generates a CSV file on the fly.
Fortunately, you don't have to deal with CSVs or scraping anything from the page. To get the data you want, all you have to do is make an HTTP GET request to the same RESTful API that your browser makes a request to when visiting the page:
def main():
import requests
url = "https://www.nseindia.com/api/live-analysis-oi-spurts-underlyings"
headers = {
"user-agent": "Mozilla/5.0"
}
response = requests.get(url, headers=headers)
response.raise_for_status()
data = response.json()["data"]
print(f"There are {len(data)} items in total.")
print(f"The first item is:\n{data[0]}")
return 0
if __name__ == "__main__":
import sys
sys.exit(main())
Output:
There are 143 items in total.
The first item is:
{'symbol': 'MOTHERSUMI', 'latestOI': 7182, 'prevOI': 4674, 'changeInOI': 2508, 'avgInOI': 53.66, 'volume': 12519, 'futValue': 53892.6066, 'optValue': 3788085280, 'total': 55585.0344, 'premValue': 1692.4278, 'underlyingValue': 104}
>>>

One way to find the final download link to a certain file is to open the debugger of your web browser and to click on the download link while looking into the Networking tab of the debugger.
Normally you will see the request the javascript of the page called as same as the url, the content of the request and so on...
From here you just need to replicate what request was sent by the javascript.

Render template in Django view after AJAX post request

I have managed to send & receive my JSON object in my views.py with a POST request (AJAX), but am unable to return render(request, "pizza/confirmation.html"). I don't want to stay on the same page but rather have my server, do some backend logic on the database and then render a different template confirming that, but I don't see any other way other than AJAX to send across a (large) JSON object. Here is my view:
#login_required
def basket(request):
if request.method == "POST":
selection = json.dumps(request.POST)
print(f"Selection is", selection) # selection comes out OK
context = {"test": "TO DO"}
return render(request, "pizza/confirmation.html", context) # not working
I have tried checking for request.is_ajax() and also tried render_to_string of my html page, but it all looks like my mistake is elsewhere. Also I see in my terminal, that after my POST request, a GET request to my /basket url is called - don't understand why.
Here is my JavaScript snippet:
var jsonStr = JSON.stringify(obj); //obj contains my data
const r = new XMLHttpRequest();
r.open('POST', '/basket');
const data = new FormData();
data.append('selection', jsonStr);
r.onload = () => {
// don't really want any callback and it seems I can only use GET here anyway
};
r.setRequestHeader("X-CSRFToken", getCookie('csrftoken'));
r.send(data);
return false;

In your basket view function, you always render the template as response. You can pass the parameters to your js snippet via HttpResponse. After request completed and you have response in your js function, you can use window.location.href to redirect the page you want. You can also look this answer to get more information.

Django - What is the most pythonic way to update my page via AJAX?

I want someone with "manager" access to be able to view all user activity, and the page should continuously update after every couple of minutes via AJAX.
Should I have both, users and AJAX requests, get sent to the same URL? Or should I have two separate URLs (one that renders a template and another one that only returns JSON)?
Here is how I imagine this should be structured:
def index(request):
context = get_user_activity()
if request.is_ajax():
# Process the data and only return updates they don't already have on their screen
return JsonResponse(updated_data, status = 200)
return render(request, 'activity.html', context)
domain.com/activity/ <-- Users and AJAX requests are sent to this url
OR
domain.com/activity/ <-- Users get sent to this page
domain.com/api/update-activity <-- AJAX requests are made to this URL

dynamically create more php objects on scroll

I want to do the following and do not know how to best approach it:
I have a Facebook like Wall in a Social Website I created.
Every wall Post is a PHP Object with several "comment" objects attached to it.
All Relevant Data is currently fetched from the Database at the time the page loads and works as I wanted it to.
My Wall is getting quite long so I would like to implement infinite scrolling via JQuery and Ajax.
I wanted to send a request for more post objects to a php page which would instantiate them and send the finished objects back. Now I have read that this is bad practice sending PHP Objects via JSON and that it can lead to Problems and potential attack vectors.
How should I approach this? Write the entire wall in Jquery? Have it send Requests for the Data to PHP but display everything with JQuery instead of PHP?
A few pointers would be very appreciated.

You should send AJAX request for needed posts and return ready HTML which you can just append to the container with $('wall_container').append($(new_post_html));

I would say load the initial page and append the data to the wall as you get it. No need to send objects. All the data you need will be in the responseText of the request object
var request = new XMLHttpRequest();
var form = new FormData();
form.append('param_1',val_1);
form.append('param_2',val_2);
request.open("http://example.com/getPostsForWall.php","POST",true);
request.send(form);
request.onreadystatechange = function(){
if(request.status==200 && request.readyState == 4){
//Successful request
response = JSON.parse(request.responseText);
//Want to append a title using jquery to your wall that has an id of wall?
$("#wall").append("<h2 class='wall-post-title'>"+response.post_title+"<h2>");
}
};
This is assuming you have your server side code returning a variable with the key 'post_title' when you send your response

How do I change pages and call a certain Javascript function with Flask?

I am not sure if I worded my question correctly. I'm not actually sure how to go about this at all.
I have a site load.html. Here I can use a textbox to enter an ID, for example 123, and the page will display some information (retrieved via a Javascript function that calls AJAX from the Flask server).
I also have a site, account.html. Here it displays all the IDs associated with an account.
I want to make it so if you click the ID in account.html, it will go to load.html and show the information required.
Basically, after I press the link, I need to change the URL to load.html, then call the Javascript function to display the information associated with the ID.
My original thoughts were to use variable routes in Flask, like #app.route('/load/<int:id>') instead of simply #app.route('/load')
But all /load does is show load.html, not actually load the information. That is done in the Javascript function I talked about earlier.
I'm not sure how to go about doing this. Any ideas?
If I need to explain more, please let me know. Thanks!
To make this more clear, I can go to load.html and call the Javascript function from the web console and it works fine. I'm just not sure how to do this with variable routes in Flask (is that the right way?) since showing the information depends on some Javascript to parse the data returned by Flask.
Flask code loading load.html
#app.route('/load')
def load():
return render_template('load.html')
Flask code returning information
#app.route('/retrieve')
def retrieve():
return jsonify({
'in':in(),
'sb':sb(),
'td':td()
})
/retrieve just returns a data structure from the database that is then parsed by the Javascript and output into the HTML. Now that I think about it, I suppose the variable route has to be in retrieve? Right now I'm using AJAX to send an ID over, should I change that to /retrieve/<int:id>? But how exactly would I retrieve the information, from, example, /retrieve/5? In AJAX I can just have data under the success method, but not for a simple web address.

Suppose if you are passing the data into retrieve from the browser url as
www.example.com/retrieve?Data=5
you can get the data value like
dataValue = request.args.get('Data')

You can specify param in url like /retrieve/<page>
It can use several ways in flask.
One way is
#app.route('/retrieve/', defaults={'page': 0})
#app.route('/retrieve/<page>')
def retrieve():
if page:
#Do page stuff here
return jsonify({
'in':in(),
'sb':sb(),
'td':td()})
Another way is
#app.route('/retrieve/<page>')
def retrieve(page=0):
if page:
#Do your page stuff hear
return jsonify({
'in':in(),
'sb':sb(),
'td':td()
})
Note: You can specify converter also like <int:page>