I am trying to create a bot with HtmlUnit to get some products from a website automatically, I am getting to list the products, but the site has several branches and for me to change to the branch I want, I need to change the value of a select and click on a button, but after I click the button, I have the same page as before, it does not change anything and it does not return any error or warning, does anyone give me a hand?
final HtmlPage page = webClient.getPage("url");
final HtmlSelect select = (HtmlSelect) page.getElementById("s-ch-select-city");
select.setSelectedAttribute("8", true);
ScriptResult resultFinal = page.executeJavaScript("document.getElementById('s-ch-change-channel').focus()");
webClient.waitForBackgroundJavaScript(10000);
webClient.waitForBackgroundJavaScriptStartingBefore(10000);
synchronized (page) {
page.wait(10000);
}
Page pageResultFinal = resultFinal.getNewPage();
HtmlPage pageCascavel = ((HtmlPage) pageResultFinal);
Without having access to the page i can only guess.
Try to replace
Page pageResultFinal = resultFinal.getNewPage();
with
Page pageResultFinal = page.getEnclosingWindow().getTopWindow().getEnclosedPage();
And check to log for any errors.
Additionally i think doing the js stuff is not required. You can do something like
page.getElementById("s-ch-change-channel").focus();
Related
I know how to implement a function that goes back to the previous page and takes a certain variable when a certain button is clicked.
// The 2 at the end of the url is the variable I want to pass.
<button onclick='location.href="a link that prev page has/2"'> Back to prev page </button>
However, I do not know how to pass a specific variable while returning to the previous page when the back button, which is implemented by default in the Chrome browser, is pressed.
I am trying to get the variable passed from the previous page I returned to and use it. What should I do when I click the browser back button?
If there are different functions to be implemented depending on the browser, it would be thankful if you could tell us about it as well.
+) Each page is implemented as a separate ejs file.
if you want to reload the previous page and know the path i would rather construct the link
const new_location = location.origin + '/page'
const new_url = new URL(new_location)
new_url.searchParams('param1','value')
location = new_url
then node side you can use
const {param1} = req.query
I'm trying to scrape a donation page using htmlunit. I need to fill inputs like nickname, message, amount and then select the method of payment in order to print the url to the payment. I have no problem with filling text inputs, but when I try to select the radio button (payment method), it doesnt work. It just leaves the default button selected.
Page: https://tipo.live/p/bartlomiej-skowron (Polish language)
My code:
WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setPrintContentOnFailingStatusCode(false);
WebRequest request = new WebRequest(new URL("https://tipo.live/p/bartlomiej-skowron"));
HtmlPage page = webClient.getPage(request);
webClient.waitForBackgroundJavaScript(2000);
JavaScriptJobManager manager = page.getEnclosingWindow().getJobManager();
HtmlForm form = page.getForms().get(0);
HtmlButton button = (HtmlButton)form.getElementsByTagName("BUTTON").get(0);
HtmlTextInput username = form.getInputByName("username");
HtmlTextArea message = form.getTextAreaByName("message");
HtmlTextInput amount = form.getInputByName("amount");
HtmlInput payment = form.getInputByValue("7"); //radio button
username.type("Nickname");
message.type("Example Message");
amount.type("25");
payment.setChecked(true); // it doesnt work
webClient.waitForBackgroundJavaScript(500);
button.click();
while(true){
if (manager.getJobCount() <= 0) break;
}
HtmlPage currentPage = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();
System.out.println(currentPage.getUrl());
I also tried to execute javascript, but it doesn't work too:
WebRequest request = new WebRequest(new URL("https://tipo.live/p/bartlomiej-skowron"));
HtmlPage page = webClient.getPage(request);
page.executeJavaScript("document.getElementById(\"7__input\").click();");
page.executeJavaScript("document.getElementsByName(\"amount\")[0].value=\"25\";");
page.executeJavaScript("document.getElementsByName(\"message\")[0].value=\"Example Message\";");
page.executeJavaScript("document.getElementsByName(\"username\")[0].value=\"Nickname\";");
page.executeJavaScript("document.getElementsByClassName(\"octf-btn octf-btn-primary octf-btn-icon\")[0].click();");
Have done some debugging on this. Looks like changing the payment method (the radio button) triggers a server roundtrip to inform about this.
This is not the case if you change the selection using HtmlUnit. The log shows the event is triggered
Firing Event change (Current Target: HTMLElement for HtmlRadioButtonInput[<input x-model="selected" autocomplete="off" id="7__input" class="radio-button" type="radio" name="payment_method" value="7">]);
A a first guess the event handler might not be registered because the loading of one used js library failed (during opening the page)
Loading external JavaScript: https://cdn.jsdelivr.net/gh/alpinejs/alpine#v2.x.x/dist/alpine.min.jsLoad response for GET https://cdn.jsdelivr.net/gh/alpinejs/alpine#v2.x.x/dist/alpine.min.jsCookieSpec
....
Caused by: net.sourceforge.htmlunit.corejs.javascript.EvaluatorException: missing ) after formal parameters (https://cdn.jsdelivr.net/gh/alpinejs/alpine#v2.x.x/dist/alpine.min.js#7)
A deeper analysis requires more time - please open a fix against HtmlUnit on GitHub and i will try to work on this.
I am trying to click a button in HtmlUnit to login to a site (m.pge.com) but it is not working. The browser stays on same page. I had similar issue with another site. I tried searching but didn't get a definitive answer.
Any ideas what I might be doing wrong ?
Here is code..
Web_Client = new WebClient(BrowserVersion.CHROME);
Web_Client.getOptions().setCssEnabled(false);
Web_Client.getOptions().setThrowExceptionOnScriptError(false);
Web_Client.setJavaScriptTimeout(15000);
Web_Client.getOptions().setRedirectEnabled(true);
//Web_Client.setAjaxController(new NicelyResynchronizingAjaxController());
// load home page
HtmlPage page = Web_Client.getPage("https://m.pge.com/#login");
HtmlTextInput userName = loginForm.getFirstByXPath("//*[#id=\"usernameField\"]");
HtmlPasswordInput passWord = loginForm.getFirstByXPath("//*[#id=\"passwordField\"]");
userName.setValueAttribute(user);
passWord.setValueAttribute(password);
HtmlButton button = null;
DomNodeList<DomElement> buttonList = page.getElementsByTagName("button");
for (Iterator buttonIter = buttonList.iterator(); buttonIter.hasNext();) {
DomElement domElement = (DomElement) buttonIter.next();
if(domElement.asText().equals("SIGN IN")) {
button = (HtmlButton)domElement;
}
}
loginpage = button.click();
It is too tedious and unclear to type in comment so i type it as answer and will update with progress.
Part1: login
The first thing is that are you sure you target the button correctly and where did you initialize the HtmlPage loginpage. Normally, when you assign
loginpage = button.click();
It will get you the login page,but sometimes it is weird that it doesn't assign new page but rather change the current page. that is your
page
You may try to print it out before button click and after button click to see if this page changes? that is
System.out.println(page.asText());
button.click(); // no need to assign page, if the page changes itself
System.out.println(page.asText());
Part2: Search
When you have done input textbox and button click, you will have to target the changes element after search again and get its content, for example, if it is a element, you should probably target the element after button click and get the content you want afterwards. If you target it before button click, the content in that element won't changes itself, you need to get that again.
This may be a very rookie error I am making, but my situation is this. I have successfully made a script that when entered into a Google Chrome bookmark, it will bring up a prompt box upon the user clicking on the bookmark. From there, after they hit enter, it brings said user to a list of search results. What I want to do is put another piece of code in this same bookmark so that it automatically clicks on the first result on the search results page after it loads. Any ideas?
javascript: (function MCC() {
var feature = prompt('What is your search query?', '');
if (feature != null) {
window.open('https://www.google.com/#q=' + encodeURIComponent(feature));
window.onload = function MyClient() {
document.getElementById('mHSB').click();
}
}})();
I don't think there is a way to interact with a page inside of another window/frame. Your code snippet would run the function in the current window when the new window loaded.
If you are searching with google you could change the URL, something like
http://www.google.com/search?q=searchTerm&btnI
This should be the same as entering the search term and clicking Im feelin lucky (which should load the first result)
I got that link from
Is there a consistent way to link to Google "I Feel Lucky" result?
If you are searching on your own site/ a site where you can add code maybe a query-string switch would be a good idea.
imagine the following scenario:
i have a jquery-mobile formular, it´s results are linking to its resultpage.
on the resultpage i have this back button:
this works fine to just update the content and keep the submitted form data,
but
what if a user came from a search-engine or similiar extern link, then my back button links back to the searchengine/externLink .
so how do i Differentiate between those who came from my form or anywhere else in a jqm-way ?
i have a "start-search-page" i would love to link to if the user didn´t came from the search and i don´t want to miss the ajax-link from my search to the resultpage, use the same button and idealy i don´t have to set any cookie.
is there any hint or smarter attempt than check the server url from document.referrer ?
thanks in advance
You can check current page url using below code:
var prevUrl = $.mobile.activePage.data('url');
in case u want to perform different actions based on previous URL.
then on save the URL in the global javascript variable and on click of the button check the previous URL and do the your functionality. eg
Before Navigating to page:
var prevUrl = $.mobile.activePage.data('url');
on click of button:
if (prevUrl=="myurl") {
//do something
$.mobile.changePage('#search')
}
else {
$.mobile.changePage('#nothing')
}