I need to monitor xhr requests on a website (which I don't control) and get realtime data. For example: tickdata.
My plan: log in with selenium and python after build websocket server. finally, acsess that.
Problem: target xhr data is like crash file. So, after debugging some js files I found out xhr data is crashed for not getting. But, the data I want exists as an array variable in a 3rd js. adding not global object.
Is it possible to get the variable in a running js script?
just like this:
Capture AJAX response with selenium and python
Related
I've got an app based on phantomjs that works so:
1. I'm running a php script that gets data from my databse (postgres) as an array,
2. Then via shell_exec I'm running phantomjs script and as argument I'm passing array with data (1),
3. In phantom I'm processing the data - checking domains WHOIS - and collecting for each domain expiration date. As result I'm getting an array that I'm storing in a file,
4. In the end phantom runs php script that gets the data from stored file and saves it in my database.
I'm wondering if there is a better option? Maybe doing everything in the phantomjs script? Maybe there is a js client for postgres?
I'd change workflow starting from step 3 and start saving data right away (PhantomJS is no stranger to crashing so it may not always get to step 4).
You could send data via an AJAX or POST request to an endpoint of your own. It could be another PHP script available via HTTP, even if on localhost. So you'd do another page.open to there and send data.
An even more reliable approach: after processing data execute a local PHP script feeding it data via CLI (or save data to a file like before and feed the script path to it).
I need two informations to generate the url to send the POST request to the server: api_token and cid, as you can see in the JS code bellow:
// Run query
fetch(`/ajax/gw-light.php?api_version=1.0&api_token=${encodeURIComponent(config.get('checkForm'))}&input=3&cid=${cid || id}`
I can get the token by parsing the HTML, but the 'cid' is fetched by functions in JavaScript. Is there any way to get this only by Requests?
I know about Selenium, but its very very slow. Requests goes directly to the point.
You cannot do this with Python Requests, you will need to look to use a headless browser as Python Requests does not execute JavaScript. You may be able to see where this code is being executed, for example... if there is an api call being made to get this token, than you could replicate it with Python Requests.
The bottom line is Python Requests and BeautifulSoup do not execute JavaScript it's simply HTTP and not a browser JS engine.
I am trying to write my first chrome extension. The workflow goes something like this -When the extension is installed and active if a user hovers over a specific product/ID displayed on the page, the extension retrieves related vendor data about the product with the ID.
This is how I thought about this:
Use jQuery attr to access the ID on mouse over.
Post this ID to a retrieve.php file with .post() method
The retrieve.php file retrieves the data from database
Display the data in a tool tip on the web page.
I have some queries for the above process:
I am able to get this working on a local XAMPP server but how will it work online as the chrome extension will not have access to server. What is the way around to retrieve data without using PHP?
I am able to get the logic working but am unable to place these in respective files - Will all my logic reside in background.js ?
Any suggestions on getting this started will be much appreciated.
You could build a very simple API on your server that responds with JSON to any request it receives after processing it. Like this:
{"firstVar":"foo","secondVar":"bar" }
Your chrome extension can then make an xmlhttp request to this server and and process the returned data.(You could also use JSONP and wrap the response in a callback function which will execute as soon as you have the reponse)
The JS extension will be able to deal with the JSON nicely as it can understand that format so you can then choose to display the data in whatever way you want.
Essentially, what you want is a server that can take an ID posted to it and return the corresponding date in a nice and readable format. And a chrome extension that can make an request to a server and then process the response. Build and test them separately (keep positing an ID to the server and see the response and for your JS side at first instead of making requests to your unfinished API just set a static response to begin with which will be the same as an expected response.
I'm a consultant working on a web app that's basically a single page app. All it does is constantly retrieve new json data behind the scenes (like once a minute), and then display it on screen.
Our clients load this app, and leave it running 24/7, for weeks on end. If errors happen when retrieving new json data, the app ignores it and keeps running.
We're rolling out an update, and want the existing clients to either become invalidated, or reload themselves without any user interaction. This feature wasn't "built in" by anyone, and we're trying to do this after the fact.
Is there some way to make the existing clients reload without telling our end users to just reload the page?
The following conditions define the app a bit more:
The app uses jQuery 1.9.0
Runs exclusively in Chrome
Retrieves new json data frequently using jquery
Throws away any errors it finds in json responses and uses old data.
EDIT:
I've had it suggested that we could try the following:
send invalid data through the JSON responses to crash chrome (like 500 megs of data, for example)
send window.location.reload through the JSON response (which supposedly won't work due jquery protecting against this type of thing)
send "script" data in the JSON response and if it gets $.html(....) at some point, then it may run the script as well.
and am open to any suggestions on getting this to reload or kill chrome, so the client is forced to reload the page.
If you're using $.ajax to request your data, and not specifically setting your content type, then you may be able to do the following on the server:
set the content type header to "text/javascript"
respond with javascript, e.g. window.location = "http://www.yoursite.com"
jQuery may eval that, and simply run your javascript.
No it is not possible. As far as I can tell you do not execute code from the JSON response (which is a very good thing). Thus you have no way of altering your current client's behaviour. According to your own statement:
"Throws away any errors it finds in JSON responses and uses old data"
You will not be able to crash the user's browser by sending invalid JSON data as the errors will be suppressed.
You can build in automatic deployment in to future versions by sending an application version number and testing for changes or by using WebSockets (which the application seems better suited to anyway as you can ensure your clients only poll the server when the JSON has actually changed).
If I get it correctly, create a version referance page, and make the client check this page very couple seconds, when you update the file, client will reload itself with this script.
var buildNo = "1.2.0.1";//
var cV = setInterval(checkVersion,(5*1000))//Every 5 sec.
function checkVersion(){
$.ajax({
url:"checkVersion.php?v="+buildNo,
dataType:"JSON",
success:function(d){
if(d.version != buildNo){//if version is different
window.location.reload();
//chrome.runtime.reload(); //for chrome extensions
}
}
})
}
if you cant add extra page, you may just add extra variable to end of your JSON data.
I'm working with a .js client and have and object that I need to write out to a file on the server. A couple of questions - file i/o with JavaScript is new to me... I was planning on using jquery and json. I'm using java serverside. I don't have a problem reading what I get back from my servlet, but the file i/o is killing me! A couple of questions:
I can open a file I generated myself via the .js with an $.ajax call, but it's not handling my json syntax (I tried both an $.getJson and $.ajax - handwritten json, so I might (probably) are doing something wrong with it). I used firebug's console and it looks ok...
How can I write my object to a file on the server?
Then, when I want to read it, what do I need to do to process it? Right now I'm using a jsonFilter function (uses JSON.parse if that's available, otherwise eval) to process data that I'm getting from the servlet.
The object I'm writing isn't simple, but it's not super complex either. There's an array that contains an array, but that shouldn't make a difference if the software is both reading/writing it.
Thanks for any help! I'm at a loss - tried alot of different things.
You can open a file located on the server via ajax by querying the file and loading it into a JSON object. You might want to LINT your JSON
You can not write to an object on the server via the client. This is a severe security breach.
Common practice is to change the JSON data and then send it via ajax to server-side code. The server will then do the file IO.
Yes using JSON.parse otherwise eval is indeed correct. I would recommend json2.js
The data should be fine as long as it passes JSONLint.
Your main issue is that it's impossible to write to the server from the client. Get the client to load the data through ajax change it and then query the server to update the file.
js don't have i/o property;
you should use ajax or http request to send message to server,and tell server to do de i/o action...