CasperJS isn't doing the same request as normal browser - javascript

I have made some script in CasperJs to login to page, open specific page and get content of it.
I'm opening desired page with content to scrap with casper.thenOpen and it shows different content on casperjs and different on my browser. I have setted up useragent from my browser and still no effects. Here are request and response headers from casperjs:
http://screenshot.sh/m7QeJGUbqK5Kg
And here from my browser:
http://screenshot.sh/n8j3Gf9QkuXxm
I don't know why the results are different on browser and casperjs. I don't know why there is no cookie in request header in casperjs but i'm sure that it's logged in because previusly in code i can see code allowed only for logged users.
Thanks in advance for any help
Code:
casper.thenOpen('website.com', function() {
this.echo("start");
this.echo(this.fetchText('html'), "INFO");
casper.then(function () {
var json;
var start;
var end;
function checkReload()
{
json = JSON.parse(this.fetchText('html'));
if(json.msg.indexOf('desiredtext') === 0) {
this.echo("good", 'PARAMETER');
return;
}
else
{
this.echo('bad news: ' + json.msg, 'COMMENT');
}
casper.thenOpen('website.com');
this.echo(this.fetchText('html'));
this.wait(1, checkReload);
}
this.then(checkReload);
});
});
But problem isn't in code(I also tried to put request headers from browser to casper but this didn't work)

Related

Check if page is loaded from bfcache, HTTP cache, or newly retrieved

the code below checks whether a url is loaded and then logs to the console. I would like to know if there is simple, clean method to check if a page is loaded from bfcache or http cache? Firefox documentation states that the load event should not be triggered if I go from URL A to B and then hit the back button to URL A, but this is not my experience, both load and PageShow is logged regardless, does anyone know why?
var tabs = require("sdk/tabs");
function onOpen(tab) {
tab.on("pageshow", logPageShow);
tab.on("load", logLoading);
}
function logPageShow(tab) {
console.log(tab.url + " -- loaded (maybe from bfcache?) ");
}
function logLoading(tab) {
console.log(tab.url + " -- loaded (not from bfcache) ");
}
tabs.on('open', onOpen);
I am not sure whether there is any purposeful API for that but a workaround that came to mind is to check value of the performance.timing.responseEnd - performance.timing.requestStart. If it is <= 5 then most likely it is HTTP or back-forward cache. Otherwise, it is a download from the web.
A way to recognize a return to the page through a back button instead of opening up a clean URL is to use history API. For example:
// on page load
var hasCameBack = window.history && window.history.state && window.history.state.customFlag;
if (!hasComeBack) {
// most likely, user has come by following a hyperlink or entering
// a URL into browser's address bar.
// we flag the page's state so that a back/forward navigation
// would reveal that on a comeback-kind of visist.
if (window.history) {
window.history.replaceState({ customFlag: true }, null, null);
}
}
else {
// handle the comeback visit situation
}
See also Manipulating the browser history article at MDN.

Unable to load page resources with PhantomJS

I'm using PhantomJS to get page content for given URL.
The problem is that on some pages PhantomJS can not load some resources (js, css...), and the error I'm getting is:
error code 5, Operation canceled
Web page on which I can reproduce this problem is www.lifehacker.com
The resources I can not get are:
http://x.kinja-static.com/assets/stylesheets/tiger-4ee27d6612a71ee3c68440f8e9c0025c.css
http://c.amazon-adsystem.com/aax2/amzn_ads.js
and some others too...
The command I'm running is:
phantomjs --debug=true --cookies-file=cookies.txt --ignore-ssl-errors=true --ssl-protocol=tlsv1 fetchpage.js http://www.lifehacker.com
and even if I remove options like cookies-file, ignore-ssl-errors, ssl-protocol the result is still the same.
The fetchpage.js script is:
var webPage = require('webpage');
var system = require('system');
var page = webPage.create();
if (system.args.length === 1) {
console.log('Usage: fetchpage.js <some URL>');
phantom.exit(1);
}
var url = system.args[1];
page.open(url, function (status) {
console.log("STATUS: " + status);
if (status !== 'success') {
console.log(
"Error opening url \"" + page.reason_url
+ "\": " + page.reason
+ "\": " + page
);
phantom.exit(1);
} else {
var content = page.content;
console.log(content);
phantom.exit(1);
}
});
If I open that same page in Chrome, page loads just fine. Also if I copy those resource URLs that phantomjs can not load and paste them in Chrome, they load just fine.
I have tried to google for similar problems, but I only found some suggestions about setting timeout which did not work for me.
I have tried the same thing with phantomjs v1.9.0, 1.9.8 and 2.0.1-development.
What's even more interesting, sometimes phantomjs script manages to get full response from all resources, so I'm suspecting on cache, but I couldn't force server to avoid cache. I have tried to send custom headers through phantomjs like this:
...
var page = webPage.create();
page.customHeaders = {
"Cache-Control":"no-cache",
"Pragma":"no-cache"
};
page.open(url, function (status) {
...
but nothing changed.
I am running out of ideas..
For coders who come across this page during their quest to find an solution for resources not completely loading on phantomjs. I had a project where the script would stall/hang on a few resources. It was 50/50 if it would execute or not.
Some digging and I found the following page:
https://github.com/ariya/phantomjs/issues/10652
Where the solution to set an timeout for resources was working out for me:
page.settings.resourceTimeout = 10000;
In regards to the above question I am not sure if this is completely appropiate but at least the information is easier to find now and can be regarded part of an solution to some.

Why Page is not redirected using javascript on using window.location and window.location.href

I am using a login system for my website.
What I am doing is that when the user clicks the submit button an ajax request is being executed to target php script which logs in the user and if user is logged in successfully then echo's back the message "login_successful" to the ajax request. By using if statement I check whether the message is "login_successful" else display the error.
If the message is "login_successful" the script redirect the user to the new page 'user.php' by using window.location = 'user.php';
//I have also tried this.
window.location.href = 'user.php';
But it doesn't work It simply stays on the login page and nothing happens.
but when i check the page source after logging in but no redirection takes place then i got a surprise that the source of the page is for user.php instead of login.php. Somehow window.location hasn't redirected the page but bring the source of user.php to login.php. It's all messed up and I couldn't solve the problem.
Here is the login function which performs the ajax request and then redirect the user-
function login()
{
var e = _("email").value;// Grab the email input by the user.
var p = _("password").value;// Grab the password input by the user.
if(e === "" || p === "")// Check if they are empty.
{
_("status").innerHTML = "Fill out all the form data";//Display error message if fields are empty.
}
else
{
_("loginbtn").style.display = "none";// Hide the login button
_("status").innerHTML = 'please wait ...';//Tell the user to wait while the script works.
// Below id another function which is defined in other js file.
var ajax = ajaxObj("POST", "login.php"); //start the ajax request.
ajax.onreadystatechange = function()// Wait for the request to be complete
{
if(ajaxReturn(ajax) === true) // Ensures if the response is recieved
{
var response = ajax.responseText;// Put the response in a variable
if(response === 'login_successful')// Check if user is logged in succesfully.
{
window.location = 'user.php';//Redirect the user to the new page.(Not working)
}
else
{
_('status').innerHTML = response;// Display the response
_("loginbtn").style.display = "block";// Display the login button
}
}
};
ajax.send("e="+e+"&p="+p);
}
}
I am using the latest version of chrome on a 64 bit windows machine.
Update
OK guys I have verified that the response via php script is 'login_successful' and it doesn't contains any line so that we can trim on it.
check the result by alert in between response condition & use
window.location.href='correctfilepath.php';
correct path of file to be redirected. Check your user.php also as it will not print anything if it will have some error in code
if you use location i prefer to use a absolute path. Maybe this is your problem and a redirect will be working with this. For example:
location.href = '/user.php';
If you want to redirect to a url do it like this:
location.href = '//www.google.com';
(it will automatically use http or https based on the current scheme.
Try setting the full URL for redirection
window.location = window.location.protocol + "//" + window.location.host + "/user.php"
instead of window.location = 'user.php';
Edited:
window.location = "//" + window.location.host + "/user.php"
Javascript redirection works without the protocol (http or https) also.
if(response === 'login_successful')// Check if user is logged in succesfully.
{
window.location = 'user.php';//Redirect the user to the new page.(Not working)
}
else
{
_('status').innerHTML = response;// Display the response
_("loginbtn").style.display = "block";// Display the login button
}
In the else block you had given _('status').innerHTML = response; check that block is executing or not if its is executing then try to change the (=== as == )in the if condition hope it may work.

Javascript window.open not working

Ok. I'm trying to login to twitter. The window is not opening in this code. The response that gets alerted is not null and is a link to a login screen. Any ideas?
var url = "./twitter_login.php";
var con = createPHPRequest();
con.open("POST",url,true);
con.setRequestHeader("Content-type","application/x-www-form-urlencoded");
con.send("");
var response = "";
con.onreadystatechange = function() {
if(con.readyState==4 && con.status==200) {
response = con.responseText;
alert(response);
window.open(response,"twitter","menubar=1,resizable=1,width=350,height=500");
}
}
The standard popup-blocker logic contained in most browsers these days will block any calls to window.open() that are not the direct result of a user action. Code that is triggered by timers or by any asynchronous callback (like your ajax ready function) will be treated as NOT caused directly by user actions and the new popup window will generally be blocked.
You can verify this is what is happening by temporarily changing your browser's popup blocking (turning it off) and see that it then starts working.
Probably what you need to do as a work-around is to create the window upon the user action that started this thread of code and then put the content into the window when you get your ajax response. The browser will probably allow that. I know that's less desirable from a visual perspective, but you can put some temporary content in the window until the ajax response comes in (something like "loading...").
Just had this exact same issue. Just in case you wanted the code that fixed it. I used this:
newWindow = window.open("", "_blank");
request = $.ajax({ ... my request which returns a url to load ... })
request.done((function(_this) {
return function(data, textStatus, jqXHR) {
return newWindow.location = data.Url;
};
})(this));

Response.Redirect AFTER call to JS Alert or Confirm

I am working on a VB.NET web application. When someone successfully changes their password I want to show a popup message that lets them know it was changed successfully. After they click OK I want to redirect them to the main page. Code looks like this:
ClientScript.RegisterStartupScript(Me.GetType(), "confirmScript", "ConfirmNewUser();", True)
Response.Redirect("MainPage.aspx")
Why does the redirect happen and the alert popup never displays?
Try this:
1) Remove Response.Redirect from the code behind.
2) Change the ConfirmNewUser function as given below:
function ConfirmNewUser(){
//Existing Code of ConfirmNewUser
//New Code.
var msg = "Password changed successfully. Press OK to go to Home page Cancel to stay on current page.";
if(confirm(msg)){
window.location.href = "MainPage.aspx";
}
}
You are calling the redirect server side, your script never get a chance to run. use window.location to do the redirect client side, something like this:
function ConfirmNewUser() {
if(confirm("Your password has been changed, click OK to continue")) {
window.location = "MainPage.aspx"; //go to home page
}
}
The reason is because all server-side processing will take place prior to client-side.
One solution would be to pass "MainPage.aspx" to your client script as follows:
ConfirmNewUser('MainPage.aspx');
Your client script would then have to take a URL parameter:
function ConfirmNewUser(url) { ... }
and follow up with a window.location:
...
if(confirm(...))
{
window.location = url;
}
and remove the following from your server code:
Response.Redirect("MainPage.aspx")
Response.Redirect sets the Location http header and a 302-Moved response, the browser will act upon this as soon as it sees it. As headers come before content, your script is never seen or parsed.

Categories

Resources