Develop Reference

CasperJS Invoke Javascript function on browser console

I am trying to invoke a Javascript method using CasperJS.
The web page simply consist of a link that allows me to change back to my default country.
<a id="defaultCountry" href="javascript:__doPostBack('cty$UK$default','')">Default Country</a>
I want CasperJS to invoke the Javascript method that is called after clicking the link.
I assumed that mimicking a mouse click on the link would invoke the Javascript method but it doesn't.
I've tried the following methods with no success:
casper.then(function() {
casper.click(x('//*[#id="defaultCountry"]'));
casper.evaluate(function() {
__doPostBack('cty$UK$default',''); //this is the javascript function. im not sure if thats how you would call it though
});
or
this.clickLabel('Default Country', 'a');
I know that if I invoke the Javascript function on the browser console, it will work. I simply type:
__doPostBack('cty$UK$default','');
in the console and it magically works.
Any help appreciated!
Edit:
#Rippo This is a snippet of what I ran. It seems like CasperJS bypasses my eval statement. The last few lines are from the console directly. I know my page loads with the selector because of the screenshot. I've even used the casper.waitforselector method to confirm.
casper.thenOpen('http://example.com');
casper.wait(5000, function() {
console.log('page opened');
casper.capture('page.png');
console.log('capture page complete');
});
casper.thenEvaluate(function() {
console.log('invoking javascript');
__doPostBack('cty$UK$default','');
console.log('javascript invoked');
});
This is from the console:
[info] [phantom] wait() finished waiting for 5000ms.
page opened
[debug] [phantom] Capturing page to C:/Users/page.png
[info] [phantom] Capture saved to C:/Users/page.png
capture page complete
[info] [phantom] Step _step 8/8 http://example.com (HTTP 200)
[info] [phantom] Step _step 8/8: done in 16240ms.
[info] [phantom] Done 8 steps in 16259ms
[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "about:blank"
This is the last line from the console before it exits.
Edit 7/17/2016 12:40AM
#Artjom B.
Here's a snippet of the code I ran with your suggestions:
...snipped some prior
function() {
console.log('Page loaded');
casper.capture('page.png');
console.log('Starting 1st postback call');
this.evaluate(function() {
console.log('postback call');
__doPostBack('cty$UK$default','');
});
console.log('passed postback');
casper.capture('post-postback.png');
//At this point, it skips this function and goes straight to console.log then comes back to this function. Not sure why...
casper.then(function() {
console.log('trying to change again');
this.click(x('//*[#id="defaultCountry"]'));
this.evaluate(function() {
__doPostBack('cty$UK$default','');
console.log('javascript invoked');
});
});
//skipped to this console.log
console.log('waiting country to change');
this.waitForSelector('.countryuk',
function() {
console.log('country change completed. Capturing image');
this.capture('uk.png');
},
function() {
console.log('timed out waiting for country to change.');
this.capture('uk-timeout.png');
},5000);
};
Here is the console output:
Page loaded
[debug] [phantom] Capturing page to C:/Users/page.png
[info] [phantom] Capture saved to C:/Users/page.png
Starting 1st postback call
Console: postback call
Error: ReferenceError: Can't find variable: __doPostBack
passed postback
[debug] [phantom] Capturing page to C:/Users/post-postback.png
[info] [phantom] Capture saved to C:/Users/post-postback.png
waiting country to change
[info] [phantom] Step anonymous 10/11 http://example.com/page.aspx?r=2 (HTTP 200)
trying to change again
[debug] [phantom] Mouse event 'mousedown' on selector: xpath selector: //*[#id="defaultCountry"]
[debug] [phantom] Mouse event 'mouseup' on selector: xpath selector: //*[#id="defaultCountry"]
[debug] [phantom] Mouse event 'click' on selector: xpath selector: //*[#id="defaultCountry"]
Error: ReferenceError: Can't find variable: __doPostBack
Error: ReferenceError: Can't find variable: __doPostBack
[info] [phantom] Step anonymous 10/11: done in 22567ms.
[info] [phantom] Step _step 11/11 http://example.com/page.aspx?r=2 (HTTP 200)
[info] [phantom] Step _step 11/11: done in 22573ms.
[warning] [phantom] Casper.waitFor() timeout
timed out waiting for country to change.
[debug] [phantom] Capturing page to C:/Users/uk-timeout.png
[info] [phantom] Capture saved to C:/Users/uk-timeout.png
[info] [phantom] Done 11 steps in 27825ms
[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "about:blank"
Clearly it cant find the postback function. I'm not sure why though. It's not a hidden element. It's exactly as I posted above (a link). It's nested within a bunch of div tags but that's about it.

what about just (remove casper.then and casper.click)
casper.thenEvaluate(function() {
__doPostBack('cty$UK$default','');
});

You can access the href of the link and use the eval function.
var str = document.getElementById("defaultCountry").href;
eval(str.substring(str.indexOf(":") + 1));

casperjs/slimerjs invalid selector in waitForSelector, which works in getElementByXPath?

I have this HTML file, test_xpath.htm:
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
</head>
<body>
<h1>Hello World!</h1>
<div>
Click me!
</div>
</body>
</html>
In the folder of this file, I run PHP CLI > 5.4 as a server like this:
php -S localhost:8080
... so the HTML page is available at http://localhost:8080/test_xpath.htm.
Then I run this SlimerJS code:
// run with:
// SLIMERJSLAUNCHER=/usr/bin/firefox46 /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/casperjs --engine=slimerjs test_xpath.js
var casper = require('casper').create({
verbose: true,
logLevel: 'debug',
userAgent: 'Mozilla/5.0 (X11; Linux i686; rv:43.0) Gecko/20100101 Firefox/43.0',
viewportSize: {width: 1024, height: 768},
pageSettings: {
loadImages: false,//The script is much faster when this field is set to false
loadPlugins: false,
}
});
casper.on("url.changed", function(){
this.then(function(){
this.echo("URL changed " + this.getCurrentUrl()); //getTitle());
});
});
casper.on('remote.message', function(message) {
this.echo('remote message caught: ' + message);
});
casper.start().thenOpen("http://localhost:8080/test_xpath.htm", function() {
console.log("website opened");
});
// set this to true to run - causes "[error] [remote] findAll(): invalid selector provided "//*[text()="Click me!"]":SyntaxError: An invalid or illegal string was specified"
if (false) {
casper.waitForSelector('//*[text()="Click me!"]', function() {
this.echo("I'm sure //*[text()='Click me!'] is available in the DOM");
});
}
casper.then(function(){
this.evaluate(function(){
var aelem = __utils__.getElementByXPath('//*[text()="Click me!"]');
aelem.click();
});
});
casper.then(function(){
casper.capture('TestCapture.png');
});
casper.run();
If I run the JS code as is - that is, the waitForSelector part is disabled by if(false) - then everything runs fine.
However, if I enable that part by changing the if(false) to if(true), I get this:
$ SLIMERJSLAUNCHER=/usr/bin/firefox46 /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/casperjs --engine=slimerjs test_xpath.js
[info] [phantom] Starting...
[info] [phantom] Running suite: 5 steps
[debug] [phantom] opening url: http://localhost:8080/test_xpath.htm, HTTP GET
[debug] [phantom] Navigation requested: url=http://localhost:8080/test_xpath.htm, type=Undefined, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "http://localhost:8080/test_xpath.htm"
[debug] [phantom] Successfully injected Casper client-side utilities
[info] [phantom] Step anonymous 2/6 http://localhost:8080/test_xpath.htm (HTTP 200)
URL changed http://localhost:8080/test_xpath.htm
[info] [phantom] Step anonymous 2/6: done in 235ms.
[info] [phantom] Step anonymous 3/6 http://localhost:8080/test_xpath.htm (HTTP 200)
website opened
[info] [phantom] Step anonymous 3/6: done in 258ms.
[info] [phantom] Step _step 4/6 http://localhost:8080/test_xpath.htm (HTTP 200)
[info] [phantom] Step _step 4/6: done in 276ms.
[error] [remote] findAll(): invalid selector provided "//*[text()="Click me!"]":SyntaxError: An invalid or illegal string was specified
[error] [remote] findAll(): invalid selector provided "//*[text()="Click me!"]":SyntaxError: An invalid or illegal string was specified
[error] [remote] findAll(): invalid selector provided "//*[text()="Click me!"]":SyntaxError: An invalid or illegal string was specified
[error] [remote] findAll(): invalid selector provided "//*[text()="Click me!"]":SyntaxError: An invalid or illegal string was specified
[error] [remote] findAll(): invalid selector provided "//*[text()="Click me!"]":SyntaxError: An invalid or illegal string was specified
....
I truly don't understand this, because http://docs.casperjs.org/en/latest/modules/casper.html#waitforselector says:
Waits until an element matching the provided selector expression exists in remote DOM to process any next step
... where http://docs.casperjs.org/en/latest/selectors.html says:
CasperJS makes a heavy use of selectors in order to work with the DOM, and can transparently use either CSS3 or XPath expressions.
So, XPath should be OK - what's more, I'm using THE EXACT SAME XPATH in:
var aelem = __utils__.getElementByXPath('//*[text()="Click me!"]');
... and there it works, but it fails in:
casper.waitForSelector('//*[text()="Click me!"]', function() { ....
Why does this happen - where am I going wrong? And how can I use this XPath in waitForSelector, if it's possible?

If you pass a string into the waitForSelector() function, it would be interpreted as a CSS selector. To have it work with an XPath expression pass in an object explicitly specifying the selector type:
selectorObject = {
type: 'xpath',
path: '//*[text()="Click me!"]'
}
casper.waitForSelector(selectorObject, function() {
// ...
});
You can solve it with waitForText() as well, sample here.

Loading resource failed on DoPostBack link click

I am trying to download an ZIP file available by clicking on aspx page link with CasperJS. However my programm fail and return "Loading resource failed with status=fail (HTTP 200)".
My code :
var casper = require('casper').create({
pageSettings: {
userName: 'myusername',
password: 'mypwd',
localToRemoteUrlAccessEnabled: 'true',
XSSAuditingEnabled: 'true',
webSecurityEnabled: false
}
});
casper.start('https://mysite/login.aspx');
casper.userAgent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');
// some code for identification...
casper.thenOpen('https://mysite/mypage.aspx');
casper.waitForSelector(
'a[id="myidlink"]',
function() {
this.click('#myidlink');
},
function() {
this.log('TIMEOUT!','error');
this.exit(62);
},
10000
);
casper.waitForSelector('#myidlink');
casper.on('resource.received', function (resource) {
if (resource.contentType == "application/zip" && resource.stage == 'end'){
this.log('resource.id : ' + resource.id, 'info');
this.log('resource.url : ' + resource.url, 'info');
}
});
casper.run();
I execute this code in ubuntu terminal with command :
casperjs --ignore-ssl-errors=true --web-security=false --ssl-protocol=any --verbose --log-level=debug mycasperjs.js
Console log return "loading resource failed" :
[info] [phantom] Starting...
...
[debug] [phantom] Mouse event 'mousedown' on selector: #myidlink
[debug] [phantom] Mouse event 'mouseup' on selector: #myidlink
[debug] [phantom] Mouse event 'click' on selector: #myidlink
[info] [phantom] Step anonymous 9/10: done in 8142ms.
[debug] [phantom] Navigation requested: url=https://mysite/mypage.aspx
[warning] [phantom] Loading resource failed with status=fail (HTTP 200): https://mysite/mypage.aspx
[info] [phantom] Step _step 10/10 mysite/mypage.aspx (HTTP 200)
[info] [phantom] Step _step 10/10: done in 9890ms.
[info] [phantom] resource.id : 49
[info] [phantom] resource.url : mysite/mypage.aspx
[info] [phantom] waitFor() finished in 40ms.
[info] [phantom] Done 10 steps in 9948ms
Versions used :
phantomjs --version
1.9.2
casperjs --version
1.1.0-beta3
Any help is welcome!
Thanks a lot.

CasperJS click fails to show modal window

I am trying to write a really simple CasperJS script to test the login button on a website. After the click event, a modal window is opened on top of the website so you can fill in the login fields.
Using the website, this works pretty well. Using casperjs test my-script-file --engine=slimerjs also works perfectly fine. Without slimerjs, the code fails.
How the button is defined:
<button class="btn btn-strong-submit" data-action="join">Join</button>
My test:
casper.test.begin('testing', 3, function suite(test)
{
casper.start();
casper.thenOpen("http://the-website-to-test-url", function()
{
casper.wait(5000, function(){
casper.thenClick("button.btn.btn-strong-submit");
});
});
casper.then(function () {
casper.wait(3000);
casper.capture("screen.png");
});
casper.then(function(){
casper.assertVisible("div.join_container");
});
casper.run(function()
{
test.done();
})
});
On the website, clicking the button will go to http://the-website-to-test-url.com/#join since the way the website works is it waits for a click and catches is using jQuery and then finds the [data-action], forms #[data-action] and then builds the URL in that way.
But CasperJS does not seem to understand a click which then gets caught from jQuery on the background. Any ideas?
Update on terminal output:
[info] [phantom] Starting...
[info] [phantom] Running suite: 4 steps
[debug] [phantom] opening url: http://www.MYURL.com/, HTTP GET
[debug] [phantom] Navigation requested: url=http://www.MYURL.com/, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "http://www.MYURL.com/"
[debug] [phantom] Successfully injected Casper client-side utilities
[info] [phantom] Step anonymous 2/4 http://www.MYURL.com/ (HTTP 200)
[info] [phantom] Step anonymous 2/4: done in 721ms.
[info] [phantom] Step anonymous 3/4 http://www.MYURL.com/ (HTTP 200)
[info] [phantom] Step anonymous 3/4: done in 731ms.
[info] [phantom] Step _step 4/5 http://www.MYURL.com/ (HTTP 200)
[info] [phantom] Step _step 4/5: done in 752ms.
[info] [phantom] wait() finished waiting for 5000ms.
[debug] [phantom] Mouse event 'mousedown' on selector: .btn.btn-strong-submit
[debug] [phantom] Mouse event 'mouseup' on selector: .btn.btn-strong-submit
[debug] [phantom] Mouse event 'click' on selector: .btn.btn-strong-submit
[info] [phantom] Step anonymous 5/5 http://www.MYURL.com/ (HTTP 200)
//After click URL should have changed to http://www.MYURL.com/#login
[info] [phantom] Step anonymous 5/5: done in 5772ms.
[info] [phantom] Step _step 6/6 http://www.MYURL/ (HTTP 200)
[info] [phantom] Step _step 6/6: done in 5792ms.
[info] [phantom] wait() finished waiting for 3000ms.
[debug] [phantom] Capturing page to /home/TEMP/screen.png
[info] [phantom] Capture saved to /home/TEMP/screen.png
[info] [phantom] Done 6 steps in 9367ms

Expanding on Artjom's comment, I think you should change this:
casper.then(function () {
casper.wait(3000);
casper.capture("screen.png");
});
to be:
casper.wait(3000, function () {
casper.capture("screen.png");
});
This is a bit more descriptive, not just more compact: "Wait 3000ms then do ...".
(Aside: casper.then( doSomething ) and casper.wait(0, doSomething) are the same thing.)
NOTE: the downside of this approach is it adds a 3-second delay to your unit test. You could rewrite it as:
casper.waitUntilVisible('div.join_container', {
casper.capture("screen.png");
});
Then as soon as it is visible it will take the screenshot, then progress to the next step. The downside of this way is that your assert always works; if there is a problem you get a time-out instead of a test failure. (That may or may not matter, it depends on what you are testing and why.)

Get nothing when scraping Facebook using Casperjs

I am trying to use casperjs to scrape the main page of Facebook. I just modified the example code to scrape Google by casperjs: identify the element with class=fbxWelcomeBoxName, and retrieve the link out of it.
Their example works fine for Google.com, but it seems it does not work when scraping on Facebook. I heard about Facebook banning scraping about customer audience. But my scraping has nothing to do with that. Did I miss anything? Thanks in advance!
var links = [];
var casper = require('casper').create();
function getLinks() {
var links = document.querySelectorAll('.fbxWelcomeBoxName');
return Array.prototype.map.call(links, function(e) {
return e.getAttribute("href");
});
}
casper.userAgent('Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.41 Safari/535.1');
casper.start('https://www.facebook.com', function() {
// search for 'casperjs' from facebook form
this.fill('form#login_form', {
email: '***your email***',
pass: '***your password***'
}, true);
});
casper.then(function() {
// aggregate results
links = this.evaluate(getLinks);
});
casper.run(function() {
// echo results in some pretty fashion
this.echo(links.length + ' links found:');
this.echo(' - ' + links.join('\n - ')).exit();
});
Result:
john#john-ubuntu-ssd:~/tests/casperjs$ casperjs --verbose --log-level=debug --ssl-protocol=any test11.js
[info] [phantom] Starting...
[info] [phantom] Running suite: 3 steps
[debug] [phantom] opening url: https://www.facebook.com/, HTTP GET
[debug] [phantom] Navigation requested: url=https://www.facebook.com/, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "https://www.facebook.com/"
[debug] [phantom] Successfully injected Casper client-side utilities
[info] [phantom] Step anonymous 2/3 https://www.facebook.com/ (HTTP 200)
[info] [remote] attempting to fetch form element from selector: 'form#login_form'
[debug] [remote] Set "email" field value to myemail#hotmail.com
[debug] [remote] Set "pass" field value to ********
[info] [remote] submitting form to https://www.facebook.com/login.php?login_attempt=1, HTTP POST
[info] [phantom] Step anonymous 2/3: done in 447ms.
[debug] [phantom] Navigation requested: url=https://www.facebook.com/login.php?login_attempt=1, type=FormSubmitted, willNavigate=true, isMainFrame=true
[debug] [phantom] Navigation requested: url=https://www.facebook.com/, type=FormSubmitted, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "https://www.facebook.com/"
[debug] [phantom] Successfully injected Casper client-side utilities
[info] [phantom] Step anonymous 3/3 https://www.facebook.com/ (HTTP 200)
[info] [phantom] Step anonymous 3/3: done in 2265ms.
[info] [phantom] Done 3 steps in 2284ms
0 links found:
-

PhantomJS has a default viewportSize of 400x300. So you should set it before you start to something desktop-like:
casper.viewportSize = {width: 1920, height: 1080};
If this doesn't help, you should wait for the '.fbxWelcomeBoxName' to appear in the page before scraping it:
casper.waitForSelector('.fbxWelcomeBoxName', function(){
links = this.evaluate(getLinks);
});
By the way, there is no reason to have getLinks. You should use casper.getElementsAttribute:
links = casper.getElementsAttribute('.fbxWelcomeBoxName', 'href');