I have used puppeteer for one of my projects to open webpages in headless chrome, do some actions and then close the page. These actions, however, are user dependent. I want to attach a lifetime to the page, where it closes automatically after, say 30 minutes, of opening irrespective of whether any action is performed or not.
I have tried setTimeout() functionality of Node JS but it didn't work (or I just couldn't figure how to make it work).
I have tried the following:
const puppeteer = require('puppeteer-core');
const browser = await puppeteer.connect({browserURL: browser_url});
const page = await browser.newPage();
// timer starts ticking here upon creation of new page (maybe in a subroutine and not block the main thread)
/**
..
Do something
..
*/
// timer ends and closePage() is triggered.
const closePage = (page) => {
if (!page.isClosed()) {
page.close();
}
}
But this gives me the following error:
Error: Protocol error: Connection closed. Most likely the page has been closed.
Your provided code should work as excepted. Are you sure the page is still opened after the timeout and it is indeed the same page?
You can try this wrapper for opening pages and closing them correctly.
// since it is async it won't block the eventloop.
// using `await` will allow other functions to execute.
async function openNewPage(browser, timeoutMs) {
const page = await browser.newPage()
setTimeout(async () => {
// you want to use try/catch for omitting unhandled promise rejections.
try {
if(!page.isClosed()) {
await page.close()
}
} catch(err) {
console.error('unexpected error occured when closing page.', err)
}
}, timeoutMs)
}
// use it like so.
const browser = await puppeteer.connect({browserURL: browser_url});
const min30Ms = 30 * 60 * 1000
const page = await openNewPage(browser, min30Ms);
// ...
The above only closes the Tabs in your browser. For closing the puppeteer instance you would have to call browser.close() which could may be what you want?
page.close returns a promise so you need to define closePage as an async function and use await page.close(). I believe #silvan's answer should address the issue, just make sure to replace if condition
if(page.isClosed())
with
if(!page.isClosed())
Related
I'm writing a script to purchase items on Amazon.
const puppeteer = require('puppeteer');
// Insert personal credentials
const email = '';
const password = '';
function press_enter(page) {
return Promise.all([
page.waitForNavigation({waitUntil:'networkidle2'}),
page.keyboard.press(String.fromCharCode(13))
]);
}
function click_wait(page, selector) {
return Promise.all([
page.waitForNavigation({waitUntil:'networkidle2'}),
page.click(selector)
]);
}
(async () => {
const browser = await puppeteer.launch({headless:false, defaultViewport:null, args: ['--start-maximized']});
const page = (await browser.pages())[0];
await page.goto('https://www.amazon.it/');
await click_wait(page, "a[data-nav-role='signin']");
await page.keyboard.type(email);
await press_enter(page);
await page.keyboard.type(password);
await press_enter(page);
// Search for the "signout" button as login proof
if(await page.$('#nav-item-signout') !== null) console.log('Login done!');
else return console.log('Something went wrong during login');
// Navigate to the product page
await page.goto('https://www.amazon.it/dp/B07RL2VWXQ');
// Click "buy now" (choose either Option A or Option B)
// Option A: Here the code get stuck since the page isn't refreshing and page.waitForNavigation() will reach its timeout
// await click_wait(page, "#buy-now-button");
// Option B: Waiting time manually set to 5 seconds (it should be more than enough for popover to fully load)
await Promise.all([page.waitForTimeout(5000), page.click('#buy-now-button')]);
// Conclude the purchase
await click_wait(page, '#turbo-checkout-pyo-button');
})();
So far I can login to Amazon, navigate to a product page and click the Buy Now button.
Then, if delivery address and payment option are all set up, (depending on Amazon domain) it may show up a pop-over box to conclude the purchase.
I wasn't able to replicate the popover response on .com and .co.uk, it seems that these domains will redirect you on a totally new page.
When I explore the page with Chrome Developer Tools I actually see the new chunk of the page being loaded (<!DOCTYPE html>) but I don't know where the representation of this element resides in Puppeteer.
If I use click_wait() to click Buy Now, the script gets stuck (it only returns after the default timeout of page.waitForNavigation()) so it's not considered a refreshing of the page. But even if I just wait a few seconds after clicking Buy Now and then attempt to click input[id='turbo-checkout-pyo-button'] (the orange button "Ordina") Puppeteer throws an error cause it can't find the element, despite it being clearly loaded.
I'm trying to use puppeteer to load a page, submit a form (which takes me to a different URL) and then ideally run something once this new page had loaded. I'm using Node JS, and am generalising my logic into separate files, one of which is search.js as per the below:
const puppeteer = require('puppeteer')
const createSearch = async (param1) => {
puppeteer.launch({
headless: false,
}).then(async browser => {
const page = await browser.newPage(term, location)
await page.goto('https://example.com/')
await page.waitForSelector('body')
await page.evaluate(() => {
const searchForm = document.querySelector('form.searchBar--form')
searchForm.submit() // this takes me to a new page which I need to wait for and then ideally return something.
// I've tried adding code here, but it doesn't run...
}, term, location)
})
}
exports.createSearch = createSearch
I'm then calling my function from my app's entry point...
(async () => {
// current
search.createSearch('test')
// proposed
search.createSearch('test').then(() => {
// trigger puppeteer to look at the new page and start running asserts.
})
})()
Unfortunately, due to the form submitting, I'm unsure how I can wait for the new page to load and run a new function? The new URL will be unknown, and different each time, e.g: https://example.com/page20
After form submit, you need to wait until the page reloads. Please add this following the await page.evaluate() function call.
await page.waitForNavigation();
And then you can perform action you want.
A web page has a button and puppeteer must click that button as soon as possible button becomes visible. This button is not always visible and it is becoming visible for everyone at the same time. So i have to refresh constantly to find that button is became visible. I wrote that script below for to do that:
const browser = await puppeteer.launch({
headless: true,
args: ['--no-sandbox']
});
const page = await browser.newPage()
await page.setViewport({ width: 1920, height: 1080})
//I am calling my pageRefresher method here
async function pageRefresher(page,browser, url) {
try {
await page.goto(url, {waitUntil: 'networkidle2'})
try {
await page.waitForSelector('#ourButton', {timeout: 10});
await page.click('#ourButton')
console.log(`clicked!`)
await browser.close()
} catch (error) {
console.log('catch2 ' + counter + ' ' + error)
counter += 1
await pageRefresher(page, browser, url)
}
}catch (error) {
console.log('catch3' + error)
await browser.close();
}
}
As you can see, my method is recursive. It goes to that page and looking for that button. If there is no button then it calls itself again for redoing the same job until it finds and clicks to that button.
Actually it works well right now. But it is slow. I am running this script meanwhile i am opening the same page on my desktop chrome and i am starting to refresh that page manually. And i am always winning, i am always clicking to that button before the puppeteer.
How can i speed up this process? A script should not lose to a human who has just manual controls like F5 button.
A script should not lose to a human who has just manual controls like F5 button.
It happens because sometimes the rules that puppeteer follows are much stricter than what we consider as a "fully loaded webpage". Even if you as a human can decide whether your desired element is in the DOM already (because you see the element is there) or it is not there (because you don't see it). E.g.: you will see that your button is not there even if the background image is still loading in the background, or the webfonts are still not loaded and you have the fallback fonts, but puppeteer waits for specific events in the background to get the permission either to go to the catch block (timeout) or to grab the desired element (waitForSelector succeeds). It can really depends on the site you are visiting, but you are able to speed up the process of recognition of your desired element.
I give some examples and ideas how you can achieve this.
Ways to speed up recognition of the desired element
1.) If you don't need every network connections for your task you could speed up page loading by replacing waitUntil: 'networkidle2' to waitUntil: 'domcontentloaded' as this event happens usually earlier and will be fired when #ourButton will be already present in the DOM.
The possible options of page.goto/page.reload:
load - consider navigation to be finished when the load event is fired.
domcontentloaded - consider navigation to be finished when the DOMContentLoaded event is fired.
networkidle0 - consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
networkidle2 - consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.
You are winning over the script because of networkidle2 is too strict. You may need this option (e.g. you are visiting a single-page application or later you will need data from the 3rd party network connection e.g. cookies) but in case it is not mandatory you will experience better performance with domcontentloaded.
2.) Instead of constantly navigating to the same url you could use page.reload method in a loop, e.g.:
await page.goto(url, { waitUntil: 'domcontentloaded' })
let selectorExists = await page.$('#ourButton')
while (selectorExists === null) {
await page.reload({ waitUntil: 'domcontentloaded' })
console.log('reload')
selectorExists = await page.$('#ourButton')
}
await page.click('#ourButton')
// code goes on...
Its main benefit is that you are able to shorten and simplify your pageRefresher function. But I experienced also better performance (however I did no benchmarking but I felt it much faster than re-opening a page).
3.) If you don't need every resource type for your task you could also speed up page loading by disabling images or css with the following script:
await page.setRequestInterception(true)
page.on('request', (request) => {
if (request.resourceType() === 'image') request.abort()
else request.continue()
})
[source]
List of resourceType-s.
Try just not awaiting the goto:
page.goto(url) // no await because it doesn't have to resolve fully
await page.waitForSelector('#ourButton') // await this because we need it to be there
Some people like Promise.race for this but this way is simpler
Using the page.$eval() method you can do it as short as this:
await page.goto(url);
page.$eval('button-selector', button => button.click());
By doing so, you combine the actions of searching the desired button and clicking on it into a single line. You will have to await on the page.goto() instruction as you will need the page to be fully loaded before using page.$eval()
1st arg is the selector you need to use to get your HTMLElement in your case a button.
This HTMLElement will be retrieved by running document.querySelector() with the provided selector whitin page context before passing it as argument for the function defined in the following argument.
2nd arg is the function to be executed inside page context wich take the HTMLElement that match the previous selector as argument
The page.$eval() instruction will throw an error if no element is found that match the provided selector.
You can address this in two ways:
prevent the error from triggering at all by testing if your HTMLElement exists before using the page.$eval() method.
await page.goto(url);
if (await page.$('button-selector') != null) // await because page.$() returns a promise
page.$eval('button-selector', button => button.click());
an alternative using only page.$() would be :
await page.goto(url);
if ((button = await page.$('button-selector')) != null)
button.click();
Be sure to encapsulate the left part of the condition inside ( ) otherwise button value will be true or false.
catch the error when it occurs:
you could use this to determine when to reload the page
await page.goto(url);
page.$eval('button-selector', button => button.click())
.catch((err) => {
// log the error here or do some other stuff
});
After some tests it looks like we can't use a try ... catch block to capture the error on the page.$eval() method so the above example is the only way to do so.
For more informations you could check the puppeteer API page for page.$eval()
And if you want to go further in accelerating puppeteer I've found those tutorials really helpfull:
How to speed up Puppeteer scraping with parallelization
Optimizing and Deploying Puppeteer Web Scraper
8 Tips for Faster Puppeteer Screenshots
Edit:
From your code i see you use the page.setViewPort() method to set a viewport size of 1920x1080 px on your page. While it may provides a better viewing when showing the navigator it'll have some impact on performance. It is best practice to use minimal settings when running in headless mode.
When I click the Next button to continue my test the page has a transition, so the password may be inputted, this transition is not allowing me to click on the password input section, so to combat the problem I used the wait method to wait for 1s until the element is located. The error is described after code
const {
Builder,
By,
until,
Capabilities
} = require('selenium-webdriver');
// requiring needed modules
(async function login() {
const pageLoad = new Capabilities().setPageLoadStrategy('normal')
//configuring the way the page loads
let driver = await new Builder().withCapabilities(pageLoad).forBrowser('firefox').build();
try {
await driver.get('https://login.live.com/login.srf?wa=wsignin1.0&rpsnv=13&rver=7.3.6963.0&wp=MBI_SSL&wreply=https://www.microsoft.com/en-us/&lc=1033&id=74335&aadredir=1');
//going to link
var userName = (await driver.findElement(By.css('#i0116'))).sendKeys((USERNAME));
//finding element and typing
(await driver.findElement(By.css('#idSIButton9'))).click();
// clicking element
await driver.wait(until.elementLocated(By.css('#i0116')),1000);
//This is where I think the error is happening
var passWd = (await driver.findElement(By.css('#i0116'))).click().sendKeys(PASSWORD);
(await driver.findElement(By.css('#idSIButton9'))).click();
} catch (error) {
console.log(error)
} finally {
console.log('finished')
}
}())
{ NoSuchElementError: Unable to locate element: #i0116
at Object.throwDecodedError (/home/name/Desktop/projects/Test/Chrome/pecPrea/node_modules/selenium-webdriver/lib/error.js:550:15)
at parseHttpResponse (/home/name/Desktop/projects/Test/Chrome/pecPrea/node_modules/selenium-webdriver/lib/http.js:565:13)
at Executor.execute (/home/name/Desktop/projects/Test/Chrome/pecPrea/node_modules/selenium-webdriver/lib/http.js:491:26)
at process._tickCallback (internal/process/next_tick.js:68:7)
name: 'NoSuchElementError',
remoteStacktrace:
'WebDriverError#chrome://marionette/content/error.js:175:5\nNoSuchElementError#chrome://marionette/content/error.js:387:5\nelement.find/</<#chrome://marionette/content/element.js:330:16\n' }
finished
If you just want to wait for a action to be completed you can try sleep from
java.util.concurrent.TimeUnit:
if u want it in seconds u can use
TimeUnit.SECONDS.sleep(int);
and for minutes you can use
TimeUnit.MINUTES.sleep(int);
The raw way to use sleep would be to use Thread.sleep() but the input here should be in milliseconds of if your program contains multiple sleep statements i would prefer TimeUnit
I chose the incorrect element. Instead of selecting #i0118, the correct element. I STUPIDLY selected #i0116
This question already has answers here:
Can the browser turned headless mid-execution when it was started normally, or vice-versa?
(2 answers)
Closed 5 months ago.
I'd like to load the page with headless off to let me login.
After login I want to hide it, turning on the headless and let it do what it has to do.
How can I turn on/off the headless after launch?
You cannot toggle headless on fly. But you can share the login using cookies and setCookie if you want.
We will create a simple class to keep the code clean (or that's what I believe for these type of work since they usually get big later). You can do this without all these complexity though. Also, Make sure the cookies are serialized. Do not pass array to toe setCookie function.
There will be three main functions.
1. init()
To create a page object. Mostly to make sure the headless and headful version has similar style of browsing, same user agent etc. Note, I did not include the code to set user agents, it's just there to show the concept.
async init(headless) {
const browser = await puppeteer.launch({
headless
});
const page = await browser.newPage();
// do more page stuff before loading, ie: user agent and so on
return {
page,
browser
};
}
2. getLoginCookies()
Example of showing how you can get cookies from the browser.
// will take care of our login using headful
async getLoginCookies() {
const {
page,
browser
} = await this.init(false)
// asume we load page and login here using some method
// and the website sets some cookie
await page.goto('http://httpbin.org/cookies/set/authenticated/true')
// store the cookie somewhere
this.cookies = await page.cookies() // the cookies are collected as array
// close the page and browser, we are done with this
await page.close();
await browser.close();
return true;
}
You won't need such function if you can provide cookies manually. You can use EditThisCookie or any cookie editing tool. You will get an array of all cookies for that site. Here is how you can do this,
3. useHeadless()
Example of showing how you can set cookies to a browser.
// continue with our normal headless stuff
async useHeadless() {
const {
page,
browser
} = await this.init(true)
// we set all cookies we got previously
await page.setCookie(...this.cookies) // three dots represents spread syntax. The cookies are contained in a array.
// verify the cookies are working properly
await page.goto('http://httpbin.org/cookies');
const content = await page.$eval('body', e => e.innerText)
console.log(content)
// do other stuff
// close the page and browser, we are done with this
// deduplicate this however you like
await page.close();
await browser.close();
return true;
}
4. Creating our own awesome puppeteer instance
// let's use this
(async () => {
const loginTester = new myAwesomePuppeteer()
await loginTester.getLoginCookies()
await loginTester.useHeadless()
})()
Full Code
Walk through the code to understand it better. It's all commented.
const puppeteer = require('puppeteer');
class myAwesomePuppeteer {
constructor() {
// keeps the cookies on the class scope
this.cookies;
}
// creates a browser instance and applies all kind of setup
async init(headless) {
const browser = await puppeteer.launch({
headless
});
const page = await browser.newPage();
// do more page stuff before loading, ie: user agent and so on
return {
page,
browser
};
}
// will take care of our login using headful
async getLoginCookies() {
const {
page,
browser
} = await this.init(false)
// asume we load page and login here using some method
// and the website sets some cookie
await page.goto('http://httpbin.org/cookies/set/authenticated/true')
// store the cookie somewhere
this.cookies = await page.cookies()
// close the page and browser, we are done with this
await page.close();
await browser.close();
return true;
}
// continue with our normal headless stuff
async useHeadless() {
const {
page,
browser
} = await this.init(true)
// we set all cookies we got previously
await page.setCookie(...this.cookies)
// verify the cookies are working properly
await page.goto('http://httpbin.org/cookies');
const content = await page.$eval('body', e => e.innerText)
console.log(content)
// do other stuff
// close the page and browser, we are done with this
// deduplicate this however you like
await page.close();
await browser.close();
return true;
}
}
// let's use this
(async () => {
const loginTester = new myAwesomePuppeteer()
await loginTester.getLoginCookies()
await loginTester.useHeadless()
})()
Here is the result,
➜ node app.js
{
"cookies": {
"authenticated": "true"
}
}
So in short,
You can use the cookies function to get cookies.
You can use extensions like Edit This Cookie to get cookies from your normal browser.
You can use setCookie to set any kind of cookie you get from browser.