I need two informations to generate the url to send the POST request to the server: api_token and cid, as you can see in the JS code bellow:
// Run query
fetch(`/ajax/gw-light.php?api_version=1.0&api_token=${encodeURIComponent(config.get('checkForm'))}&input=3&cid=${cid || id}`
I can get the token by parsing the HTML, but the 'cid' is fetched by functions in JavaScript. Is there any way to get this only by Requests?
I know about Selenium, but its very very slow. Requests goes directly to the point.
You cannot do this with Python Requests, you will need to look to use a headless browser as Python Requests does not execute JavaScript. You may be able to see where this code is being executed, for example... if there is an api call being made to get this token, than you could replicate it with Python Requests.
The bottom line is Python Requests and BeautifulSoup do not execute JavaScript it's simply HTTP and not a browser JS engine.
Related
I'm creating an IoT project to send some commands to my Arduino board and receive some details through the internet.
After a lot of search, I recognized that the best solution for me is to convert arduino to a web-server and send requests to it.
I designed the showing page by:
GSMServer server(80);
serer.begin();
GSMClient client = server.available();
client.println("HTML CODE HERE");
but I want to use javascript to parse receiving GET or POSTs and assign them to values in my code.
What should I do?
You will have to build upon GSMServer further so that it can handle different requests and respond different information based on the call.
Have you seen https://www.arduino.cc/en/Tutorial/GSMExamplesWebServer
It shows how different calls can be made as well as how a http request can be responded with a html page.
I need to monitor xhr requests on a website (which I don't control) and get realtime data. For example: tickdata.
My plan: log in with selenium and python after build websocket server. finally, acsess that.
Problem: target xhr data is like crash file. So, after debugging some js files I found out xhr data is crashed for not getting. But, the data I want exists as an array variable in a 3rd js. adding not global object.
Is it possible to get the variable in a running js script?
just like this:
Capture AJAX response with selenium and python
I have been playing around with the requests library in Python 3 for quite some time now, and have decided to create a test program. For this program, I'm using the website https://ytmp3.cc/ as an example. But it turns out that a lot is going on, on the client-side it seems.
Some keys and other stuff are being generated, and I have been using Firefox's built-in network monitor, to figure out in which requests this is being made, but without luck.
As far as I know, the requests-library can't keep a "page" open and modify the DOM and content, by making more requests.
Anyone whom could take a look, and give a qualified guess on how the special keys are generated, and how I could possibly get these for my own requests.
Fx when loading the webpage, the first request made is for the root, and the response contains the webpage HTML. What I noticed is that at the bottom, there's an url containing some key and number.
<script id="cs" src="js/converter-1.0.js?o=7_1a-a&=_1519520467"></script>
id 7_1a-a
number _1519520467`
This is used for making the next request, but then a lot of following requests are being made, and some other keys are made as well. But I can't find where these come from since they are not returned by a request.
I know that when inserting a Youtube link, a request will be made to an url, as seen below.
https://d.ymcdn.cc/check.php?callback=jQuery33107639361236859977_1519520481166&v=eVD9j36Ke94&f=mp3&k=7_1a-a&_=1519520481168
This returns the following:
jQuery33107639361236859977_1519520481166({"sid":"21","hash":"2a6b2475b059101480f7f16f2dde67ac","title":"M\u00d8 - Kamikaze (Official Video)","ce":1,"error":""})
From this I can construct the download url, using the hash from above:
https://yyd.ymcdn.cc/ + 2a6b2475b059101480f7f16f2dde67ac (hash) + /eVD9j36Ke94 (youtube video id)
But how do I get
jQuery33107639361236859977_1519520481166&v=eVD9j36Ke94 and 1519520481168
Which I need to create the request?
You can probably save yourself and the operator of that website a lot of headache by just using youtube-dl, specifically with the --extract-audio --audio-format mp3 options. It's probably what that website itself uses.
youtube-dl is written in Python and can easily be used programatically.
If you insist on sending requests to that website for whatever reason, here's how I'd do it:
callback=jQuery33107639361236859977_1519520481166 specifies the name of the callback for the JSONP request. Any name you provide will be printed back out. For example, passing callback=foo will result in the following response:
foo({...})
You can omit it entirely and the server will serve just a JSON response in this case, which is nice.
_=1519520481168 is just to prevent the response being cached. It's randomly generated, just like the above parameter. The website checks for existence, however, so you have to at least pass something in.
The website, like many, checks for a valid Referer header.
Here's a minimal cURL command line to make a request to that website:
curl 'https://d.ymcdn.cc/check.php?v=eVD9j36Ke94&f=mp3&k=aZa4__&_=1' -H 'Referer: https://ytmp3.cc/'
has someone any ideas, how can i make a request with UI Automation and JS with UIAHost. An example is welcome with POST as method, some data and json as datatyp.
Thank
The easiest way to make an HTTP request via UIAutomation would be to make a cURL request via the shell using performTaskWithPathArgumentsTimeout.
If you are making a POST request with a lot of data, you may want the shell command to read the data from disk (instead of specifying it all on the command line). Our Illuminator project provides a function that lets you write arbitrarily large files to disk from within UIAutomation.
I'm working with a .js client and have and object that I need to write out to a file on the server. A couple of questions - file i/o with JavaScript is new to me... I was planning on using jquery and json. I'm using java serverside. I don't have a problem reading what I get back from my servlet, but the file i/o is killing me! A couple of questions:
I can open a file I generated myself via the .js with an $.ajax call, but it's not handling my json syntax (I tried both an $.getJson and $.ajax - handwritten json, so I might (probably) are doing something wrong with it). I used firebug's console and it looks ok...
How can I write my object to a file on the server?
Then, when I want to read it, what do I need to do to process it? Right now I'm using a jsonFilter function (uses JSON.parse if that's available, otherwise eval) to process data that I'm getting from the servlet.
The object I'm writing isn't simple, but it's not super complex either. There's an array that contains an array, but that shouldn't make a difference if the software is both reading/writing it.
Thanks for any help! I'm at a loss - tried alot of different things.
You can open a file located on the server via ajax by querying the file and loading it into a JSON object. You might want to LINT your JSON
You can not write to an object on the server via the client. This is a severe security breach.
Common practice is to change the JSON data and then send it via ajax to server-side code. The server will then do the file IO.
Yes using JSON.parse otherwise eval is indeed correct. I would recommend json2.js
The data should be fine as long as it passes JSONLint.
Your main issue is that it's impossible to write to the server from the client. Get the client to load the data through ajax change it and then query the server to update the file.
js don't have i/o property;
you should use ajax or http request to send message to server,and tell server to do de i/o action...