I'm scraping the following website Scorebing using requests.
In order to do so, I'm exploring the website to locate the XHR calls and get an url like this
being the code as follows
import requests,json
header={some data from the XHR I got using Postman}
url='https://lv.scorebing.com/ajax/score/data?mt=0&nr=1&corner=1'
response=requests.get(url=url,headers=header,data=json.dumps({}))
response.json()
No problems there. My problem is that if I switch tab, like from Corner to Fixture, no new XHR is called. In fact, only "Live Matches" and "Corners" allows for this direct XHR connection. I see that some js scripts are loaded, but I can't go from there to replicating my previous step.
I know I can scrape this using selenium, and probably using a regular requests to the url of the page and using BSoup, but what I don't understand is why some tabs make XHR calls to load data where other similar ones use js.
I would like to know how can you reverse engineer those js calls in order to get an API similar to the first part.
Firstly,you should know that XHR(XMLHttpRequest) in Chrome will record all the ajax request.
What's Ajax?
Ajax is a set of web development techniques using many web technologies on the client side to create asynchronous web applications.
Ajax could be achieved by JavaScript or jQuery(Well,jQuery is a JavaScript library.It is JavaScipt essentially,but jQuery offer a API about ajax).
In your example page,there are many ajax requests in the source code:
I would like to know how can you reverse engineer those js calls in order to get an API similar to the first part.
If you really want to do it just by the source code,you should:
Send a GET request to the page.
Analysis the source code of the page,then iterate each Javascript.(Also send GET request.)
Find all the ajax requests and also send GET requests,select the data you need from them.
Related
I am working on performance testing using jmeter for my application
I am able to successfully record a test plan. Each http request to server has got __OSVSTATE & viewstate attribute value in request.
While navigating from one page to another I am able to extract this attribute from page using Regular expression extractor and used it in the new subsequent request.
I have some pages in my applications which sends ajax requests multiple times, for each response of request a new __OSVSTATE attribute value is getting generated and sent in the <script> </script> tag json format and it seems this value gets used in the new request.
Can someone give me suggestions to achieve this in JMeter?
Each record inside container has html button using which user clicks accept button.
In JMeter I am recording this whole process, after successful recording when I start script again in the JMeter - result tree shows response only in json format and not in HTML view like other pages.
I am able to execute http request there is one request- '/PerformanceProbe/rest/BeaconInternal/WebScreenClientExecutedEvent' which internally gets execute and fails.
Do you know reason behind this or if you have any other suggestions or solutions please share here.
There are two ways of creating scripts in Jmeter for web applications. Firstly, you can create selenium scripts in jmeter using "JMeter's WebDriver Sampler" which will launch browser, perform different actions. Secondly, you can use the HTTP sampler which will record network requests. i.e. it will not display browser and work on the request/response level. For now, you are using the second method and that is the reason you are unable to see HTML.
The first method is not recommended for high user load because it consumes a lot of memory.
Regarding failure of the network requests, you need to make sure that all the parameters and headers are properly used.
Is there a way with javascript on a page to tamper http requests done by other scripts (on same page)? The other scripts can be from external domains.
Let's say on a page X a script loaded from an external domain performs an http get like GET http://www.example.com?foo=bar is it possible that a previously loaded script in the same page X can capture this request, and tamper with it so it becomes GET http://www.example.com?foo=qux?
In jquery i can achieve this by wrapping the ajax get post methods. But is this possible for plain javascript, no frameworks, working across all page and client's http requests?
It seems to me that for this to be done, the script must be able to override something very deep in the core. If i have to guess i would say it's not possible by design and because of security. What do you think?
p.s. no proxies, no external tools.
No you can't capture the HTTP request going out of your app, once it has gone out of your app.
However, there is one work around possible
if you want to alter some parameters before it goes of your app,
All the other request from your website are AJAX requests
They are invoking your custom method, say customAjax() which can
alter the parameters of the actual request that will go out.
Is there a way to send data packets from an active Python script to a webpage currently running JavaScript?
The specific usage I'm looking for is to give the ability for the webpage, using JavaScript, to tell the Python script information about the current state of the webpage, then for the Python script to interpret that data and then send data back to the webpage, which the JavaScript then uses to decide which function to execute.
This is for a video game bot (legally), so it would need to happen in real time. I'm fairly proficient in Python and web requests, but I'm just getting into JavaScript, so hopefully a solution for this wouldn't be too complex in terms of Javascript.
EDIT: One way I was thinking to accomplish this would be to have Javascript write to a file that the Python script could also read and write to, but a quick google search says that JavaScript is very limited in terms of file I/O. Would there be a way to accomplish this?
For security reasons, javascript in a browser is usually restricted to only communicate with the site it was loaded from.
Given that, that's an AJAX call, a very standard thing to do.
You can make HTTP requests using the XMLHttpRequest API, which Jquery abstracts with $.ajax and $.get. You can also use the lower level Websockets network API:
https://developer.mozilla.org/en-US/docs/WebSockets
Note that the XMLHttpRequest API will only allow requests to the same server, OR requests that return an appropriate Access-Control-Allow-Origin header.
It sounds like the Javascript is only going to send information, not receive any. In that case, you're in luck. I'm guessing you are also running the Javascript and the Python on the same machine.
Run a Python webserver on the machine the browser is running on. Here's a simple example:
http://webpy.org/install
Once visiting http://127.0.0.1:8080/ in your browser gives the message Hello World!, you can start adding more addresses to your website, for example http://127.0.0.1:8080/report_data, http://127.0.0.1:8080/report_event etc.
Your Javascript can then make AJAX requests using jQuery.ajax or XMLHTTPRequest, to the address http://127.0.0.1:8080/report_data, and pass the information as GET parameters.
I want to create a web application that displays data from a public api. I will use d3 (a javascript data-visualization library). I want to retrieve data from the api every ten minutes, and update my page (say it is traffic, or something). I have not built many web applications, how do I get the updates?
Should the js on the client side use a timer to request updates from the server side of my application (perhaps the application is written in Rails or node.js). The server then makes the api call and sends a response asynchronously? Is this called a socket? I have read that HTML5 provides sockets.
Or, perhaps an AJAX request?
Or, does the server side of my application create a timer, make the api call, and then "push" updates to the view. This seems wrong to me, there could be other views in this application, and the server shouldn't have to keep track of which view is active.
Is there a standard pattern for this type of web application? Any examples or tutorials greatly appreciated.
An AJAX request (XMLHttpRequest) is probably the way to go.
I have a very simple example of an XMLHttpRequest (with Java as the backend) here: https://stackoverflow.com/a/18028943/1468130
You could recreate a backend to receive HTTP GET requests in any other server-side language. Just echo back whatever data you retrieved, and xmlhttp.onload() will catch it.
Depending on how complex your data is, you may want to find a JSON library for your server-side language of choice, and serialize your data to JSON before echoing it back to your JS. Then you can use JavaScript's JSON.parse() method to convert your server data to an object that can easily be used by the client script.
If you are using jQuery, it handles AJAX very smoothly, and using $.ajax() would probably be easier than plain-old XMLHttpRequest.
http://api.jquery.com/jQuery.ajax/
(There are examples throughout this page, mostly-concentrated at the bottom.)
It really annoys me how complicated so many of the AJAX tutorials are. At least with jQuery, it's pretty easy.
Basically, you just need to ask a script for something (initiate the request, send url parameters), and wait for the script to give you something back (trigger your onload() or jqxhr.done() functions, supplying those functions with a data parameter).
For your other questions:
Use JavaScript's setTimeout() or setInterval() to initiate an AJAX request every 600000 milliseconds. In the request's onload callback, handle your data and update the page appropriately.
The response will be asynchronous.
This isn't a socket.
"Pushing" probably isn't the way to go in this case.
If I understand correctly and this API is external, then your problem can be divided into two separate sub-problems:
1) Updating data at the server. Server should download data once per N minutes. So, it should not be connected to customers' AJAX calls. If two customers will come to the website at the same time, your server will make two API call, what is not correct.
Actually, you should create a CRON job at the server that will call API and store its' result at the server. In this case your server will always make one call at a time and have quite a fresh information cached.
2) Updating data at clients. If data at customers' browsers should be updated without refreshing the page, then you should use some sort of Ajax. It can make a request to your server once per X minutes to get a fresh data or use so-called long-polling.
I think the most effective way to implement real time Web application is to use Web socket to push changes from the server rather than polling from the client side. This way users can see changes instantaneously once server notify that there is new data available. You can read more on the similar post.
I have tried using nodejs package called socket.io to make a real time virtual classroom. It works quite well for me.
I am relatively new to server side programming, however I am building a framework in order to learn and as end outcome deliver it to the public after it is finished.
I want some parts of the web platform to be able to update only parts of the page on requests, however I do not know anything about AJAX yet.
It would be great if someone could suggest a sort of learning curve to what I am aiming for. Kinda SoundCloud like website, with the ability to update only those parts of the DOM which are requested.
Note: I'm kind of new at this as well, so a grain of salt is advised. I detail a technique that is working for me, but I make no guarantees that it is The Best Way!
AJAX is on the client side and is achieved through either standard javascript (a pain which I wouldn't recommend in most cases) or through a library like jQuery. There are a plethora of tutorials on using jQuery for ajax calls, but the gist is that you are requesting some resource on your server that, when it arrives, calls a callback which does something with the data (this behavior is asynchronous, just as node.js tends to be).
If you have no experience using AJAX for the client, I suggest you start off with a framework like Express, before rolling your own (this also gets into reinventing the wheel). An AJAX call is no different than a standard HTTP request: it can be POST, GET, etc.
You then get into the concept of routing: given a request for some information (AJAX or not!), what should I do and what should I return? A framework does the behind-the-scenes stuff for you, so that all you need to do is specify the resource to be requested, the method in which it is requested, and then the callback which does the server-side processing which returns some data. This data can be a web page, it can be a JSON object, and so on. The key point is that you want to structure it in a way that makes sense for the AJAX call.
Here is a simple example: say I have a web page that will display a bunch of information relevant to the server, such as uptime, load, memory usage, and so forth. First, I write the basic HTML page (say, index.html) that structures this data, and I begin to write a script that makes an AJAX call for this information. I decide that the request (say, /json/stats) will receive a JSON response. On the server-side, I whip up a simple Express script which starts the server and has two routes: the first route will take any request for my / page and serve the index.html. The second route will take any request for /json/stats and make a few calls to figure out the state of the server, construct an object holding this data, and return it as a response. Now, back in the script for my HTML page, I can act on the structure of this object through jQuery in order to build the page.
If you'd like to see some code for this, you can view it here. I suggest looking into the REST architecture (of which this code adheres to) as to gain more conceptual understanding of this topic.