I have a FuncUnit test case wherein I open a webpage using
F.open("http://www.example.com");
There is a known issue in our page that approximately one in around 20 times the webpage would not load for whatever reason. I want to retry when it does not load. But in FuncUnit there is no way to suppress error if it fails to load the page. Is there a way to suppress error message in Funcunit?
Couldn't something like this work for you?
module("test", {
setup: function() {
let startMeUp = () => F.open('http://www.example.com'); // unfortunately doesn't return a usable value
let checkCondition = () => {
// grab F(F.window), and check for a known element in the page
return elementFound;
};
while(!checkCondition()) {
// element wasn't found, so let's try it again
startMeUp();
// maybe set a wait here
}
}
});
var url = "http://www.example.com"
F.get(url, function(data){
console.log("Made an ajax call to make sure that F.open is always second call");
})
// Wait for few seconds before making the actual F.open
F.wait(5000,function(){
F.open(url);
})
Whenever F.open fails ( i.e during one out of 20 times) I noticed that the second test point that uses F.open passes.However I cannot do F.open again in case the first one does not work as there is no way to suppress F.open error when the page does not load.
So I always make a simple ajax call before making the actual F.open. This is not the right way to do it but it works for us as we don't actually care whether the page is a first/second time load in our tests.
Related
I am loading a page, intercepting its requests, and when a certain element shows up I stop loading and extract the data I need...
Here is the problem that I am faced with.
When simplified the code looks actually something like this:
async function loadPage()
{
var contentLoaded = false;
var content;
//now i say when element shows up, do something
// it is page.waitForSelector but for simplicity, i use a timeout
// because the problem is the same
//i set it to "show up" in 10 seconds here.
//when it shows up, it sets the content to 100 (extracts the content i want)
//and stores it..
setTimeout(()=>{
content = 100;
contentLoaded = true;
},10000)
//Here i have a function that loads the page
//Intercepts request and handles them
//Until content is loaded
page.on('request', req =>{
if(!contentLoaded)
{
// keep loading page
}
})
// this is the piece of code i would like to not run,
// UNTIL i either get the data, or a timeout error
// from page.waitForSelector...
//but javascript will run it if it's not busy with the
//loading function above...
// In 10 seconds the content shows
// and it's stored in DATA, but this piece of code has
// already finished by the time that is done...
// and it returns false...
if(contentLoaded)
{return content}
else
{return false}
}
var x = loadPage();
x.then(console.log); //should log the data or false if error occured
Thank you all for taking the time to read this and help out, I'm a novice so any feedback or even reading material is welcome if you think there is something I'm not fully understanding
Solved
Simple explanation:
Here is what I was trying to accomplish:
Intercept page requests so that I can decide what not to load, and speedup loading
Once an element shows up on the page, i want to extract some data and return it.
I was trying to return it like this: (note, all the browser and error handling will be left out in these since it would just clutter the explanation)
var data = loadPage(url);
async function loadPage(URL)
{
var data;
page.waitForSelector(
var x = //page.evaluate returns data to x...
data = x;
)
return data;
}
Which doesn't work since return runs immediately but waitForSelector runs later, so we always return undefined...
The correct way of doing it, or rather the way it works for me is to return the whole promise, and then extract the data...
var data = loadPage(url);
data.then(//do what needs to be done with the data);
async function loadPage(URL)
{
var data = page.waitForSelector(
var x = //page.evaluate returns data to x...
data = x;
)
return data; // we return data as a promise
}
I hope it's a solid enough explanation, if someone needs to see the whole deal, I could edit the question and place the whole code there...
I have an app I'm working with that is behaving like this... You visit a url /refresh, and it loads the page with a loader/spinner/bar showing for like 5 seconds, then it refreshes the page after it's done. It does this so it can load the latest data that was computed during /refresh.
Right now I am just setting a timeout longer than the loader will most likely stay around, but this is brittle because a bad network connection could put it over the line.
How can I instead "watch" for when the refresh happens? What technique would you recommend. It seems to start to get hairy pretty fast.
Into the nitty gritty, when the loader is showing, when it finishes it is gone for like a half a second before the page reload. So I can't just wait til the loader is gone. It seems like I need to keep some sort of state variable around in the DOM like in localStorage, but can't pinpoint it. Would love some help.
well you could "watch" for the element that display the data using page.$(selector), or if no such element you could also wait for the specific request 's response:
const waitForResponse = (page, url) => {
return new Promise(resolve => {
page.on("response", function callback(response){
if (response.url() === url) {
resolve(response);
page.removeListener("response",callback)
}
})
})
};
const res = await waitForResponse(page,"url of the request you want to wait for");
Wait for Network request before continuing process
I am preparing JavaScript code that shows a random number for user as follows: if the user spend more than two minutes to pass to the next web page or if the actual page has the GET parameter "&source", the random number is replaced by another one. otherwise, the same random number is displayed for all the web pages.
The problem is that the JavaScript code should be executed manually from browser console on each page load: I should prepare a code that can be integrated to any web page from console.
Is there any difference from the normal case (include script with<script></script>)
Thanks for posting! In future posts, please try to provide some code or an example of something you've tried previously.
Anyways, here is a brief example of a script that will check for an existing number, check to see if there is a &source parameter set, begin the timer if there isn't one, and generate a new number if the timer finishes or the parameter is set.
To save the information between pages, you should consider using window.localStorage. This will allow you to check for and save the number to be used on later loads.
Note that this snippet isn't going to work until you bring it into your own page. Also, as #Sorin-Vladu mentioned, you'll have to use a browser extension if you don't have access to modify the pages you're running the script on.
const timeout = 120000
// This can be replaced by your manual execution
window.onload = () => {
start()
}
function start() {
// Attempt to pull the code from storage
let code = localStorage.getItem('code')
console.log(code)
// Get the URL parameters
let urlParams = new URLSearchParams(window.location.search)
// Check to see if the source parameter exists
if (!urlParams.has('source')) {
// If not, begin the timer
setTimeout(() => {
setCode()
}, timeout)
} else {
setCode()
}
}
function setCode() {
const code = Math.floor(Math.random() * 1000000)
localStorage.setItem('code', code)
console.log(code)
}
(Sorry for being a bit descriptive)
I wanted to wait for a page to load completely but after searching on Google it seems that browsers react differently, when we try to use readyState or onLoad.
Also, for application I am working on, it seems that a particular log message ("TNP is ready") appears in console (chrome console or IE developer tool console), when the required page is loaded.
My plan is to execute a small JavaScript code on the browser, using Selenium WebDriver C# and WatiN C# (IE), to get this message from the console log.
Can this be done? Can I get the Last Log generated by console.log ?
Because when I execute the script it might happen that the message is already gone or yet to come. I need to fire it repeatedly.
Any suggestions?
This question, while not exactly a duplicate of In WatiN how to wait until postback is complete, is probably caused by the same thing.
The browser.WaitUntilComplete() method will wait for the page to finish loading, but if you have AJAX going on in the background as the page is loading then WaitUntilComplete won't do the trick.
From my research, there is no way to get log messages from the browser console unless you include some JavaScript to mirror the console and overwrite window.console and provide a mechanism for inspecting console messages.
(function(win) {
var console = win.console;
if (!console)
throw new Error("This browser does not support a debug console");
var mirror = {},
data = {};
var toString = Object.prototype.toString,
mirrorMethod = function(key) {
data[key] = [];
return function() {
data[key].push(arguments);
console[key].apply(console, arguments);
};
};
for (var key in console) {
if (toString.call(console[key]) == "[object Function]") {
mirror[key] = mirrorMethod(key);
}
}
win.console = mirror;
win.consoleMirror = {
getData: function(key) {
return data[key] || [];
}
};
}(this));
JSFiddle: http://jsfiddle.net/fyhLw0to/
And to use:
console.log("foo");
alert(consoleMirror.getData("log")[0][0]);
An alternative is to grab an element using WatiN that you need to interact with, and call WaitUntilExists:
var button = browser.Button("buttonIdThatDoesNotExistYet");
button.WaitUntilExists();
button.Click();
This way you don't have to hack the debug console, or hack XMLHttpRequest to infer when AJAX is done. Your tests will attempt to wait for elements to appear on the page before acting upon them. Test failures will be slower, but your tests should be more robust.
I'm trying to download the HTML of a website that is almost entirely generated by JavaScript. So, I need to simulate browser access and have been playing around with PhantomJS. Problem is, the site uses hashbang URLs and I can't seem to get PhantomJS to process the hashbang -- it just keeps calling up the homepage.
The site is http://www.regulations.gov. The default takes you to #!home. I've tried using the following code (from here) to try and process different hashbangs.
if (phantom.state.length === 0) {
if (phantom.args.length === 0) {
console.log('Usage: loadreg_1.js <some hash>');
phantom.exit();
}
var address = 'http://www.regulations.gov/';
console.log(address);
phantom.state = Date.now().toString();
phantom.open(address);
} else {
var hash = phantom.args[0];
document.location = hash;
console.log(document.location.hash);
var elapsed = Date.now() - new Date().setTime(phantom.state);
if (phantom.loadStatus === 'success') {
if (!first_time) {
var first_time = true;
if (!document.addEventListener) {
console.log('Not SUPPORTED!');
}
phantom.render('result.png');
var markup = document.documentElement.innerHTML;
console.log(markup);
phantom.exit();
}
} else {
console.log('FAIL to load the address');
phantom.exit();
}
}
This code produces the correct hashbang (for instance, I can set the hash to '#!contactus') but it doesn't dynamically generate any different HTML--just the default page. It does, however, correctly output that has when I call document.location.hash.
I've also tried to set the initial address to the hashbang, but then the script just hangs and doesn't do anything. For example, if I set the url to http://www.regulations.gov/#!searchResults;rpp=10;po=0 the script just hangs after printing the address to the terminal and nothing ever happens.
The issue here is that the content of the page loads asynchronously, but you're expecting it to be available as soon as the page is loaded.
In order to scrape a page that loads content asynchronously, you need to wait to scrape until the content you're interested in has been loaded. Depending on the page, there might be different ways of checking, but the easiest is just to check at regular intervals for something you expect to see, until you find it.
The trick here is figuring out what to look for - you need something that won't be present on the page until your desired content has been loaded. In this case, the easiest option I found for top-level pages is to manually input the H1 tags you expect to see on each page, keying them to the hash:
var titleMap = {
'#!contactUs': 'Contact Us',
'#!aboutUs': 'About Us'
// etc for the other pages
};
Then in your success block, you can set a recurring timeout to look for the title you want in an h1 tag. When it shows up, you know you can render the page:
if (phantom.loadStatus === 'success') {
// set a recurring timeout for 300 milliseconds
var timeoutId = window.setInterval(function () {
// check for title element you expect to see
var h1s = document.querySelectorAll('h1');
if (h1s) {
// h1s is a node list, not an array, hence the
// weird syntax here
Array.prototype.forEach.call(h1s, function(h1) {
if (h1.textContent.trim() === titleMap[hash]) {
// we found it!
console.log('Found H1: ' + h1.textContent.trim());
phantom.render('result.png');
console.log("Rendered image.");
// stop the cycle
window.clearInterval(timeoutId);
phantom.exit();
}
});
console.log('Found H1 tags, but not ' + titleMap[hash]);
}
console.log('No H1 tags found.');
}, 300);
}
The above code works for me. But it won't work if you need to scrape search results - you'll need to figure out an identifying element or bit of text that you can look for without having to know the title ahead of time.
Edit: Also, it looks like the newest version of PhantomJS now triggers an onResourceReceived event when it gets new data. I haven't looked into this, but you might be able to bind a listener to this event to achieve the same effect.