I have a problem with getting elements by their selectors.
A page on which I struggle is: http://html5.haxball.com/.
What I have succeded is to log in, but that was kind of a hack, because I used the fact, that the field I need to fill is already selected.
After typing in nick and going into lobby I want to click the button 'Create room'. Its selector:
body > div > div > div > div > div.buttons > button:nth-child(3)
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
args: ['--no-sandbox'], headless: false, slowMo: 10
});
const page = await browser.newPage();
await page.goto('http://html5.haxball.com/index.html');
await page.keyboard.type(name);
await page.keyboard.press('Enter');
//at this point I am logged in.
let buttonSelector = 'body > div > div > div > div > div.buttons > button:nth-child(3)';
await page.waitForSelector('body > div > div');
await page.evaluate(() => {
document.querySelector(buttonSelector).click();
});
browser.close();
})();
after running such code I get error:
UnhandledPromiseRejectionWarning: Error: Evaluation failed: TypeError: Cannot read property 'click' of null
My initial approach was with:
await page.click(buttonSelector);
instead of page.evaluate but it also fails.
What frustrates my the most is the fact that when I run in Chromium console:
document.querySelector(buttonSelector).click();
it works fine.
A few things to note:
The selector you are using to retrieve the button is more complex than it needs to be. Try something simpler like: 'button[data-hook="create"]'.
The game is within an iframe, so you're better off calling document.querySelector using the iframe's document object as opposed to the containing window's document
The function passed to evaluate is executed in a different context than where you are running your node script. For this reason, you have to explicitly pass variables from your node script to the window script otherwise buttonSelector will be undefined:
Making the changes above, your code will input your name and successfully click on "Create Room":
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
args: ['--no-sandbox'], headless: false, slowMo: 10
});
const page = await browser.newPage();
await page.goto('http://html5.haxball.com/index.html');
await page.keyboard.type('Chris');
await page.keyboard.press('Enter');
//at this point I am logged in.
let buttonSelector = 'button[data-hook="create"]';
await page.waitForSelector('body > div > div');
await page.evaluate((buttonSelector) => {
var frame = document.querySelector('iframe');
var frameDocument = frame.contentDocument;
frameDocument.querySelector(buttonSelector).click();
}, buttonSelector);
browser.close();
})();
Related
I am trying to get video src from a movie website. The iframe content is in most websites hidden because it's hosted on another server and the only way to get it as of my knowledge is to use the inspect element of the dev tools to inspect the video and then the iframe shows up with the video src. I have tried using Node JS and Puppeteer but couldn't manage to successfully use the dev tools via puppeteer.
Here is an instance of what I have tried so far.
const puppeteer = require("puppeteer");
async function scrapeMovie() {
const url = 'https://shahed4u.in/%D9%81%D9%8A%D9%84%D9%85-swords-drawn-2022-%D9%85%D8%AA%D8%B1%D8%AC%D9%85-%D8%A7%D9%88%D9%86-%D9%84%D8%A7%D9%8A%D9%86/watch/';
//start pupeteer browser
const browser = await puppeteer.launch({
headless: false,
executablePath: 'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
efaultViewport: null,
// devtools: true,
});
//open a black page
const page = await browser.newPage();
//open the desire page
await page.goto(url,{ waitUntil: 'networkidle2' },);
await page.waitForTimeout(3000);
var i = 1;
while (i <= 10) {
await page.click('li[data-i="0"]')
console.log("here yes")
// await page.select('iframe[frameborder="0"]')'
// await page.click('iframe')
await page.waitForTimeout(3000);
const [el] = await page.$x('//*[#id="vplayer"]/div[2]/div[3]/video');
const test = await page.evaluate(() => {
const videoTag = document.querySelectorAll("jw-video.jw-reset video")
return videoTag.src;
});
if(test !== undefined || el !== undefined) {
console.log("here is the results of test: -> ",test);
console.log("Here is the results of el: ->", el)
break;
}
i++
}
browser.close();
};
**//The result I want is simply video src and then out of it the .mp4 source.**
**//But The result I now get is undefined as expected.**
Another example could be done on this movie site as well;
https://www2.solarmovie.to/solar.html
Note: there are ads on those sites mentioned above but its not an issue right now I exit them manually when the automation of Puppeteer does not, for testing purposes, but later on there will be a way around it for sure.
Any help, note or suggestion would be much appreciated and I am sure there are a lot that are in such situations. Thanks
My test html:
<div id="mainBlock">
<div class="underBlock">
Hello!
</div>
</div>
i try to get content of div with class underBlock like this:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless:false,
});
const page = await browser.newPage();
let response = await page.goto('http://localhost/TestPup/Index.html');
let block = await page.waitForXPath("//div[contains(#class,'underBlock')]")
let frame = await block.contentFrame()
console.log(frame.content())
await browser.close();
})();
but i got error:
TypeError: Cannot read property 'content' of null
As far as I understand, elementHandle.contentFrame() only returns a frame for iframe elements, and you have a regular div that is contained in the main frame, that is, in the page, and inside which there are no frames.
I'm fairly new to Puppeteer and I'm trying to practice keep tracking of a selected item from Amazon. However, I'm facing a problem when I try to retrieve some results from the page.
The way I intended this automation to work is by following these steps:
New tab.
Go to the home page of Amazon.
Enter the given product name in the search element.
Press the enter key.
Return the product title and price.
Check this example below:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless: false,
});
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', (req) => { // don't load any fonts or images on my requests. To Boost the performance
if (req.resourceType() == 'font' /* || req.resourceType() == 'image' || req.resourceType() == 'stylesheet'*/) {
req.abort();
}
else {
req.continue(); {
}
}
});
const baseDomain = 'https://www.amazon.com';
await page.goto(`${baseDomain}/`, { waitUntil: "networkidle0" });
await page.click("#twotabsearchtextbox" ,{delay: 50})
await page.type("#twotabsearchtextbox", "Bose QuietComfort 35 II",{delay: 50});
await page.keyboard.press("Enter");
await page.waitForNavigation({
waitUntil: 'networkidle2',
});
let productTitle = await page.$$(".a-size-medium, .a-color-base, .a-text-normal")[43]; //varible that holds the title of the product
console.log(productTitle );
debugger;
})();
when I execute this code, I get in the console.log a value of undefined for the variable productTitle. I had a lot of trouble with scraping information from a page I navigate to. I used to do page.evaluate() and it only worked when I'm scraping from the page that I have told the browser to go to.
The first problem is on this line:
let productTitle = await page.$$(".a-size-medium, .a-color-base, .a-text-normal")[43];
// is equivalent to:
let productTitle = await (somePromise[43]);
// As you guessed it, a Promise does not have a property `43`,
// so I think you meant to do this instead:
let productTitle = (await page.$$(".a-size-medium, .a-color-base, .a-text-normal"))[43];
Once this is fixed, you don't get the title text, but a handle to the DOM element. So you can do:
let titleElem = (await page.$$(".a-size-medium, .a-color-base, .a-text-normal"))[43];
let productTitle = await titleElem.evaluate(node => node.innerText);
console.log(productTitle); // "Microphone"
However, I'm not sure that simply selecting the 43rd element will always get you the one you want, but if it isn't, that would be a topic for another question.
I'm trying to switch between tabs using playwright tests
but it's not taking control of windows element.
Do we have any method similar to selenium driver.switchto().window() in playwright?
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch({ headless: false, args: ['--start-maximized'] });
const context = await browser.newContext({ viewport: null });
context.on("page", async newPage => {
console.log("***newPage***", await newPage.title())
})
const page = await context.newPage()
const navigationPromise = page.waitForNavigation()
// dummy url
await page.goto('https://www.myapp.com/')
await navigationPromise
// User login
await page.waitForSelector('#username-in')
await page.fill('#username-in', 'username')
await page.fill('#password-in', 'password')
await page.click('//button[contains(text(),"Sign In")]')
await navigationPromise
// User lands in application home page and clicks on link in dashboard
// link will open another application in new tab
await page.click('(//span[text()="launch-app-from-dashboard"])[2]')
await navigationPromise
await page.context()
// Waiting for element to appear in new tab and click on ok button
await page.waitForTimeout(6000)
await page.waitForSelector('//bdi[text()="OK"]')
await page.click('//bdi[text()="OK"]')
})()
Assuming "launch-app-from-dashboard" is creating a new page tag, you can use the following pattern to run the subsequent lines of code on the new page. See multi-page scenarios doc for more examples.
// Get page after a specific action (e.g. clicking a link)
const [newPage] = await Promise.all([
context.waitForEvent('page'),
page.click('a[target="_blank"]') // Opens a new tab
])
await newPage.waitForLoadState();
console.log(await newPage.title());
Since you run headless, it might also be useful to switch the visible tab in the browser with page.bringToFront (docs).
The browserContext?.pages() is an array that contains the tabs opened by your application, from there you can use a temporal page to make a switch, once completed your validations you can switch back.
playwright.pageMain: Page = await playwright.Context.newPage();
playwright.pageTemp: Page;
// Save your current page to Temp
playwright.pageTemp = playwright.pageMain;
// Make the new tab launched your main page
playwright.pageMain = playwright.browserContext?.pages()[1];
expect(await playwright.pageMain.title()).toBe('Tab Title');
Assume you only created one page (via browser context), but for some reason, new pages/tabs open.
You can have a list of all the pages by : context.pages,
Now each element of that list represents a <class 'playwright.async_api._generated.Page'> object.
So, now you can assign each page to any variable and access it. (For eg. page2 = context.pages[1])
it('Open a new tab and check the title', async function () {
await page.click(button, { button: "middle" }); //to open an another tab
await page.waitForTimeout(); // wait for page loading
let pages = await context.pages();
expect(await pages[1].title()).equal('Title'); /to compare the title of the second page
})
I would like to know if I can tell puppeteer to wait until an element is displayed.
const inputValidate = await page.$('input[value=validate]');
await inputValidate.click()
// I want to do something like that
waitElemenentVisble('.btnNext ')
const btnNext = await page.$('.btnNext');
await btnNext.click();
Is there any way I can accomplish this?
I think you can use page.waitForSelector(selector[, options]) function for that purpose.
const puppeteer = require('puppeteer');
puppeteer.launch().then(async browser => {
const browser = await puppeteer.launch({executablePath: "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe", headless: false});
const page = await browser.newPage();
await page.setUserAgent(options.agent);
await page.goto("https://www.url.net", {timeout: 60000, waitUntil: 'domcontentloaded'});
page
.waitForSelector('#myId')
.then(() => console.log('got it'));
browser.close();
});
To check the options avaible, please see the github link.
If you want to ensure the element is actually visible, you have to use
await page.waitForSelector('#myId', {visible: true})
Otherwise you are just looking for the element in the DOM and not checking for visibility.
Note, All the answers submitted until today are incorrect
Because it answer for an element if Exist or Located but NOT Visible or Displayed
The right answer is to check an element size or visibility using page.waitFor() or page.waitForFunction(), see explaination below.
// wait until present on the DOM
// await page.waitForSelector( css_selector );
// wait until "display"-ed
await page.waitForFunction("document.querySelector('.btnNext') && document.querySelector('.btnNext').clientHeight != 0");
// or wait until "visibility" not hidden
await page.waitForFunction("document.querySelector('.btnNext') && document.querySelector('.btnNext').style.visibility != 'hidden'");
const btnNext = await page.$('.btnNext');
await btnNext.click();
Explanation
The element that Exist on the DOM of page not always Visible if has CSS property display:none or visibility:hidden that why using page.waitForSelector(selector) is not good idea, let see the different in the snippet below.
function isExist(selector) {
let el = document.querySelector(selector);
let exist = el.length != 0 ? 'Exist!' : 'Not Exist!';
console.log(selector + ' is ' + exist)
}
function isVisible(selector) {
let el = document.querySelector(selector).clientHeight;
let visible = el != 0 ? 'Visible, ' + el : 'Not Visible, ' + el;
console.log(selector + ' is ' + visible + 'px')
}
isExist('#idA');
isVisible('#idA');
console.log('=============================')
isExist('#idB')
isVisible('#idB')
.bd {border: solid 2px blue;}
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<div class="bd">
<div id="idA" style="display:none">#idA, hidden element</div>
</div>
<br>
<div class="bd">
<div id="idB">#idB, visible element</div>
</div>
on the snippet above the function isExist() is simulate
page.waitForSelector('#myId');
and we can see while running isExist() for both element #idA an #idB is return exist.
But when running isVisible() the #idA is not visible or dislayed.
And here other objects to check if an element is displayed or using CSS property display.
scrollWidth
scrollHeight
offsetTop
offsetWidth
offsetHeight
offsetLeft
clientWidth
clientHeight
for style visibility check with not hidden.
note: I'm not good in Javascript or English, feel free to improve this answer.
You can use page.waitFor(), page.waitForSelector(), or page.waitForXPath() to wait for an element on a page:
// Selectors
const css_selector = '.btnNext';
const xpath_selector = '//*[contains(concat(" ", normalize-space(#class), " "), " btnNext ")]';
// Wait for CSS Selector
await page.waitFor(css_selector);
await page.waitForSelector(css_selector);
// Wait for XPath Selector
await page.waitFor(xpath_selector);
await page.waitForXPath(xpath_selector);
Note: In reference to a frame, you can also use frame.waitFor(), frame.waitForSelector(), or frame.waitForXPath().
Updated answer with some optimizations:
const puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch({headless: true});
const page = await browser.newPage();
await page.goto('https://www.somedomain.com', {waitUntil: 'networkidle2'});
await page.click('input[value=validate]');
await page.waitForSelector('#myId');
await page.click('.btnNext');
console.log('got it');
browser.close();
})();
While I agree with #ewwink answer. Puppeteer's API checks for not hidden by default, so when you do:
await page.waitForSelector('#id', {visible: true})
You get not hidden and visible by CSS.
To ensure rendering you can do as #ewwink's waitForFunction. However to completely answer your question, here's a snippet using puppeteer's API:
async waitElemenentVisble(selector) {
function waitVisible(selector) {
function hasVisibleBoundingBox(element) {
const rect = element.getBoundingClientRect()
return !!(rect.top || rect.bottom || rect.width || rect.height)
}
const elements = [document.querySelectorAll(selector)].filter(hasVisibleBoundingBox)
return elements[0]
}
await page.waitForFunction(waitVisible, {visible: true}, selector)
const jsHandle = await page.evaluateHandle(waitVisible, selector)
return jsHandle.asElement()
}
After writing some methods like this myself, I found expect-puppeteer which does this and more better (see toMatchElement).
async function waitForVisible (selector){
//const selector = '.foo';
return await page.waitForFunction(
(selector) => document.querySelector(selector) && document.querySelector(selector).clientHeight != 0",
{},
selector
);
}
Above function makes it generic, so that you can use it anywhere.
But, if you are using pptr there is another faster and easier solution:
https://pptr.dev/#?product=Puppeteer&version=v10.0.0&show=api-pagewaitforfunctionpagefunction-options-args
page.waitForSelector('#myId', {visible: true})
Just tested this by scraping a fitness website. #ewwink, #0fnt, and #caram have provided the most complete answer.
Just because a DOM element is visible doesn't mean that it's content has been fully populated.
Today, I ran:
await page.waitForSelector("table#some-table", {visible:true})
const data = await page.$eval("table#some-table",(el)=>el.outerHTML)
console.log(data)
And incorrectly received the following, because the table DOM hadn't been populated fully by runtime. You can see that the outerHTML is empty.
user#env:$ <table id="some-table"></table>
Adding a pause of 1 second fixed this, as might be expected:
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
await page.waitForSelector("table#some-table", {visible:true})
await sleep(1000)
const data = await page.$eval("table#some-table",(el)=>el.outerHTML)
console.log(data)
user#env:$ <table id="some-table"><tr><td>Data</td></tr></table>
But so did #ewwink's answer, more elegantly (no artificial timeouts):
await page.waitForSelector("table#some-table", {visible:true})
await page.waitForFunction("document.querySelector('table#sched-records').clientHeight != 0")
const data = await page.$eval("table#some-table",(el)=>el.outerHTML)
console.log(data)
user#env:$ <table id="some-table"><tr><td>Data</td></tr></table>