I am trying to run some automatic scripts on a webpage with selenium and python, but the problem I'm having is that the webpage is loaded using document.write().
I have to find some elements, but they are not shown because when I view the source, it is shown as document.write(A lot of JS) instead of the html.
How can I do it so that I can view the HTML source code? I know there is the function driver.execute_script(), but I have to specify the script to run, and I don't think it will work.
The page is correctly rendered, only problem is the source cannot be parsed.
As it turns out after some digging into the code, Selenium does the search in the rendered final view, but the problem was not the document.write(), but the fact that the field I was looking for was in an iframe, which selenium could not find on the default frame.
All I had to do was search through the iframes and find the ones that I needed.
Related
I'm experiencing some issues regarding an AJAX script I'm working on.
The page loads perfectly well, and all needed scripts are loaded the same for basic page functionality inside AJAX script, but after hash change, jQuery behaves awkwardly.
Let's take this example.
The custom jQuery script writes an inline CSS propriety for a specific DIV at page loading:
Now, I load the login page for example:
I get back to the main page and inline style disappears as well as the basic loaded functionalities cease to exist after Ajax call:
*
Any experience on this? Does anyone have a clue why this happens? Or even near it... Seems the script unloads on page/hash change, which I don't believe. Or enters in double loop, therefore doubling the classes for HTML. I don't get it.
Already searched a lot and went trough the coding and is fine becasue it works fine alongside with basic HTML. Would appreciate some thoughts on this matter.
Thanks!
I have been trying to find a way to alter the final HTML DOM output (i.e., after JS script adjusts the HTML output) before it gets rendered on the browser.
I found a hook in Wordpress called 'template_redirect' which works well if I want to change the HTML output before JS gets executed but not after. So the before JS script execution, all I see is the JS name only from this hook.
Here's the problem that I'm trying to solve. In my wordpress blog, I'm including a JS script from third-party and this JS is inserting extra tags (such as Schema.org tags regarding that their organization - Schema organization tags). It is about 3000 characters long and it is bloating my webpage. I would like to remove any extra tags that gets attached to my final HTML (it is getting added before </html> tag) before shown in the browser.
Note that I don't see these extra tags when I do view source and they show up only when I do inspect element from Chrome or FF.
Any suggestions on how to remove the HTML DOM after JS execution but before it is shown on the browser via Wordpress hooks or any other way?
Thanks, JK
JS is executed in the users browser not on the sever(in almost all cases). You can either remove the tags after they are rendered using more JS(which won't really help any "bloat"), or take a look at the code for the plugin you are using and alter the plugin
I've been searching for a while now, but I can't figure out how to load an entire page via AJAX and still execute all javascript and css.
Mostly I just end up with the plain text without any CSS.
Is there a way to do this? I tried jQuery.get, jQuery.load and jQuery.ajax, but none really work like that.
I have a different solution. You may try it with an iframe. Use jQuery to append an iframe script including all relevant codes into some part of your page (like some div). This may do it for you including CSS, like;
$('<iframe src="your_page.html"/>').appendTo('#your_div');
Or you may try something like;
$('<iframe src="your_page.html"/>').load(function(){
alert('the iframe is done loading');
}).appendTo('#your_div');
I have solved similar problem as following.
Download the webpage over ajax
Iterate it over and find any <script> and </script> tags
Get content from within these tags as text
Create new <script> element and insert there the code
Append the tag to your webpage
Another thing is you will need to somehow call the script..
I have done it this way:
I set standardized function names like initAddedScript callback which I am calling after appending the script to the page. Same as I have deinitScript called when I do not need the code (and its variables,..) anymore.
I must say this is awful solution, which likely means you have bad application architecture so as I have had:)
With css is it the same, but you do not need any handlers. Just append the style tag to your documents head.
If the page you load doesn't have any style data, then the external stylesheets must have relative paths that are not correct relative to the invoking document. Remember, this isn't an iFrame - you aren't framing an external document in your document, you're combining one document into another.
Another problem is that loading your complete page will also load the doctype, html, head, and body tags - which modern browsers will cope with most of the time, but the results are undefined because it's not valid HTML to jam one document into another wholesale. And this brings me to the third reason why it won't work: CSS links outside of the head section aren't valid, and the misplaced head section caused by your haphazard document-in-document collage.
What I'd do for compliance (and correct rendering) is this, which would be implemented in the Success callback:
Copy all link elements to a new jQuery element.
Copy the contents of all script in the head section
Copy the .html() contents from the loaded document's body tag
Append the link elements (copied out in step 1) to your host document's head
Create a new script tag with your copied script contents and stick it in the head too
Done!
Complicated? Kind of, I guess, but if you really want to load an entire page using AJAX it's your only option. It's also going to cause problems with the page's JavaScript no matter what you do, particularly code that's supposed to run during the initial load. There's nothing you can do about this. If it's a problem, you need to either rewrite the source page to be more load-friendly or you could figure out how to make an iFrame suit your needs.
It's also worth considering whether it'd work to just load your external CSS in the host document in the first place.
I suppose you are looking for something like this:
your page div --> load --> www.some-site.com
After a quik search the closest solution seems to be the one by "And": Load website into DIV
You have to run a web server and create a proxy.php page with this content:
Then your JQuery load() function should be like this:
$("#your_div_id").load("proxy.php?url=http://some-site.com");
NB. I have tested this solution and it should not load all the CSS from the target page, probably you'll have to recreate them. For example the image files stored on the remote server will not loaded, I suppose due to authentication policy.
You will be also able to view only the target page without the possibility to browse the target site.
Anyway I hope this could be a step forward to your solution.
Get your entire webpage as text using ajax
document.open();
document.write(this.responseText);
document.close();
OR
document.documentElement.outerHTML = this.responseText;
But you need to change the path of css and js pages in original webpage if the resulting webpage is in another directory.
According to the jQuery documentation for $.ajax, "included script tags are evaluated when inserted in the DOM.". When I use ajax to grab the content of a dialog box, which in turn contains a script tag for displaying a ReCaptcha box, the ReCaptcha box does not appear when added to the DOM. According to FireBug, the script tag is also now missing from the added content. Navigation directly to the dialog content displays the ReCaptcha just fine.
Does anyone know why this may be occuring and/or know a work around? Any help would be greatly appreciated.
Code in action can be viewed here:
https://dustinhendricks.com/
Then click "Register Now".
Does the script try to do things like document.write()? That won't work when loaded dynamically, only on the initial page load.
In general, scripts contained in HTML that is innerHTML'd is not evaluated consistently across browsers. To fix that, jQuery actually looks for script tags and executes them manually.
But that doesn't really matter. Even if jQuery didn't do that, the script would be running in a different type of context than it normally is when a page is loading for the first time. It's not really an 'inline script' anymore, and a lot of 3rd party scripts were written with it being 'inline' as an assumption. You'll have to figure out what the script is doing and find a way to call it with dynamic content.
Take a look at the Google reCAPTCHA AJAX API. Using the CAPTCHA this way should fix your problem.
I hope it helps!
I have a unique challenge that I'm not sure how to approach:
I need to manufacture a new HTML page from scratch, one that contains a script tag and a paragraph tag with some words in it. Very simple! Once the page is built, I just need to open it in a new tab. As long as a I can call the script tag from within it, a popup is fine too. Basically, I am going to use a library called MathJax which will typeset all the elements on the html page it loads on.
I'm not even sure what this functionality is formally called, or if it's even possible in GWT! Any guidance at all would be appreciated, thanks!
There is no need to use GWT for it. Just write simple servlet that will spit out html with required script tags to load GWT and do whatever else you need and you are done.