I am using this code in Try Puppeteer:
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.barchart.com/futures/quotes/ESM19/interactive-chart/fullscreen');
const linkHandlers = await page.$x("//li[contains(text(), '1D')]");
if (linkHandlers.length > 0) {
await linkHandlers[0].click();
} else {
throw new Error("Link not found");
}
await page.$eval('input[name="fieldInput"]', el => el.value = '1');
console.log(await page.content())
// const text = page.evaluate(() => document.querySelector('rect'))
// text.then((r) => {console.log(r[0])})
await page.screenshot({path: 'screenshot.png'});
await browser.close();
The same page loaded in the Chrome browser shows the bars indicating price movements, but in the screenshot obtained in Puppeteer the chart is empty.
Also page.content() gives an html that is completely different from the one I see when inspecting the element in Chrome.
Problem
You are not waiting for the request to resolve when the input is changed. As a change will trigger a request, you should use page.waitForResponse to wait until the data is loaded.
In addition, this is an Angular application, which does not seem to like it if you simply change the value of the field via el.value = '1'. Instead you need to try to behave more like a human (and hit backspace and type the input value).
Solution
First, you get the element handle (input[name="fieldInput") from the document. Then, you focus the element, remove the value inside by pressing backspace. After that you type the desired input value.
The input field now has the correct value, now we need to trigger the blur event by calling blur() on the element. In parallel, we wait for the request to the server to finish. After the request finishes, we should give the page a few milliseconds to render the data.
All together, the resulting code looks like this:
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.barchart.com/futures/quotes/ESM19/interactive-chart/fullscreen');
// wait until the element appears
const linkHandler = await page.waitForXPath("//li[contains(text(), '1D')]");
await linkHandler.click();
// get the input field, focus it, remove what's inside, then type the value
const elementHandle = await page.$('input[name="fieldInput"]');
await elementHandle.focus();
await elementHandle.press('Backspace');
await elementHandle.type('1');
// trigger the blur event and wait for the response from the server
await Promise.all([
page.waitForResponse(response => response.url().includes('https://www.barchart.com/proxies/timeseries/queryminutes.ashx')),
page.evaluate(el => el.blur(), elementHandle)
]);
// give the page a few milliseconds to render the diagram
await page.waitFor(100);
await page.screenshot({path: 'screenshot.png'});
await browser.close();
Code improvement
I also removed the page.$x function and replaced it with the page.waitForXPath function. This makes sure that your scripts waits until the page is loaded and the element you want to click is available before the script continues.
Related
I am testing two websites that are linked between each other. I starts on website one where there is a link (_blank) to the second website. And I want to continue my test on that tab.
test('test', async ({ page }) => {
const browser = await chromium.launch();
const context = await browser.newContext();
await page.goto('https://example.io/');
const [newPage] = await Promise.all([
context.waitForEvent('page'),
page.locator('a.browser-button').first().click() // Opens tab
])
await newPage.waitForLoadState();
console.log(await newPage.title());
await page.screenshot({ path: 'test.png', fullPage: true });
await browser.close();
});
So I click on the button, a new tab opens. And then I want to continue from there.
Instead I get the error:
Timeout of 30000ms exceeded.
context.waitForEvent('page')
I have tried as in documentation as well, dont get it to work either:
https://playwright.dev/docs/pages
Handling new pages is documented here
You can capture the new page from the click event.
const [newPage] = await Promise.all([
context.waitForEvent('page'),
page.locator('a[target="_blank"]').click() // Opens a new tab
])
await newPage.waitForLoadState();
console.log(await newPage.title());
I'd remove the target=_blank attribute on the element and then click it.
await page.$eval("a.browser-button", el => el.removeAttribute("target"))
Then click it and it will open in the same window. Unless you're really determined to test it as is.
First time using Puppeteer and trying to simply click this button
after clicking the deny cookies button. That's my code:
await page.goto('https://myurl.com');
await page.click('a.cc-btn.cc-deny');
// await page.waitForNavigation();
await page.waitForSelector("#detailview_btn_order", {visible: true});
await page.click("#detailview_btn_order");
Clicking the deny cookies button works like a charm. However, it seems the second button can't be identified by Puppeteer. If I don't use waitForSelector it just says it can't find it. If I use it, I get a timeout after 30 seconds even though the website finishes loading after 5 seconds. If I uncomment waitForNavigation (regardless of what options I use) I get a timeout there, even thoug the site loads within seconds. What am I doing wrong? Thanks!
Can you try this:
await page.goto('https://myurl.com');
await Promise.all([
page.click('a.cc-btn.cc-deny'),
page.waitForNavigation(),
]);
const iframeElement = await page.waitForSelector("#my-iframe");
const frame = await iframeElement.contentFrame();
await frame.waitForSelector("#detailview_btn_order", {visible: true});
await frame.click("#detailview_btn_order");
Sometimes there is a race condition between a click and navigation.
I'm trying to use puppeteer to load a page, submit a form (which takes me to a different URL) and then ideally run something once this new page had loaded. I'm using Node JS, and am generalising my logic into separate files, one of which is search.js as per the below:
const puppeteer = require('puppeteer')
const createSearch = async (param1) => {
puppeteer.launch({
headless: false,
}).then(async browser => {
const page = await browser.newPage(term, location)
await page.goto('https://example.com/')
await page.waitForSelector('body')
await page.evaluate(() => {
const searchForm = document.querySelector('form.searchBar--form')
searchForm.submit() // this takes me to a new page which I need to wait for and then ideally return something.
// I've tried adding code here, but it doesn't run...
}, term, location)
})
}
exports.createSearch = createSearch
I'm then calling my function from my app's entry point...
(async () => {
// current
search.createSearch('test')
// proposed
search.createSearch('test').then(() => {
// trigger puppeteer to look at the new page and start running asserts.
})
})()
Unfortunately, due to the form submitting, I'm unsure how I can wait for the new page to load and run a new function? The new URL will be unknown, and different each time, e.g: https://example.com/page20
After form submit, you need to wait until the page reloads. Please add this following the await page.evaluate() function call.
await page.waitForNavigation();
And then you can perform action you want.
On a page I'm scraping with Puppeteer, I have a list with the same id for every li. I am trying to find and click on an element with specific text within this list. I have the following code:
await page.waitFor(5000)
const linkEx = await page.$x("//a[contains(text(), 'Shop')]")
if (linkEx.length > 0) {
await linkEx[0].click()
}
Do you have any idea how I could replace the first line with waiting for the actual text 'Shop'?
I tried await page.waitFor(linkEx), waitForSelector(linkEx) but it's not working.
Also, I would like to replace that a in the second line of code with the actual id (#activities) or something like that but I couldn't find a proper example.
Could you please help me with this issue?
page.waitForXPath what you need here.
Example:
const puppeteer = require('puppeteer')
async function fn() {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://example.com')
// await page.waitForSelector('//a[contains(text(), "More information...")]') // ❌
await page.waitForXPath('//a[contains(text(), "More information...")]') // ✅
const linkEx = await page.$x('//a[contains(text(), "More information...")]')
if (linkEx.length > 0) {
await linkEx[0].click()
}
await browser.close()
}
fn()
Try this for id-based XPath:
"//*[#id='activities' and contains(text(), 'Shop')]"
Did you know? If you right-click on an element in Chrome DevTools "Elements" tab and you select "Copy": there you are able to copy the exact selector or XPath of an element. After that, you can switch to the "Console" tab and with the Chrome API you are able to test the selector's content, so you can prepare it for your puppeteer script. E.g.: $x("//*[#id='activities' and contains(text(), 'Shop')]")[0].href should show the link what you expected to click on, otherwise you need to change on the access, or you need to check if there are more elements with the same selector etc. This may help to find more appropriate selectors.
For puppeteer 19 and newer, waitForXPath() is obsolete. Use the xpath prefix instead
await page.waitForSelector('xpath/' + xpathExpression)
In your case:
const linkEx = await page.waitForSelector('xpath///a[contains(text(), "Shop")]');
await linkEx.click();
Since ESPN does not provide an API, I am trying to use Puppeteer to scrape data about my fantasy football league. However, I am having a hard time trying to login using puppeteer due to the login form being nested with an iframe element.
I have gone to http://www.espn.com/login and selected the iframe. I can't seem to select any of the elements within the iframe except for the main section by doing
frame.$('.main')
This is the code that seems to get the iframe with the login form.
const browser = await puppeteer.launch({headless:false});
const page = await browser.newPage();
await page.goto('http://www.espn.com/login')
await page.waitForSelector("iframe");
const elementHandle = await page.$('div#disneyid-wrapper iframe');
const frame = await elementHandle.contentFrame();
await browser.close()
I want to be able to access the username field, password field, and the login button within the iframe element. Whenever I try to access these fields, I get a return of null.
You can get the iframe using contentFrame as you are doing now, and then call $.
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('http://www.espn.com/login')
const elementHandle = await page.waitForSelector('div#disneyid-wrapper iframe');
const frame = await elementHandle.contentFrame();
await frame.waitForSelector('[ng-model="vm.username"]');
const username = await frame.$('[ng-model="vm.username"]');
await username.type('foo');
await browser.close()
I had an issue with finding stripe elements.
The reason for that is the following:
You can't access an with different origin using JavaScript, it would be a huge security flaw if you could do it. For the same-origin policy browsers block scripts trying to access a frame with a different origin. See more detailed answer here
Therefore when I tried to use puppeteer's methods:Page.frames() and Page.mainFrame(). ElementHandle.contentFrame() I did not return any iframe to me. The problem is that it was happening silently and I couldn't figure out why it couldn't find anything.
Adding these arguments to launch options solved the issue:
'--disable-web-security',
'--disable-features=IsolateOrigins,site-per-process'