Develop Reference

How can I click on all links matching a selector with Playwright?

I'm using Playwright to scrape some data. How do I click on all links on the page matching a selector?
const { firefox } = require('playwright');
(async () => {
const browser = await firefox.launch({headless: false, slowMo: 50});
const page = await browser.newPage();
await page.goto('https://www.google.com');
page.pause(); // allow user to manually search for something
const wut = await page.$$eval('a', links => {
links.forEach(async (link) => {
link.click(); // maybe works?
console.log('whoopee'); // doesn't print anything
page.goBack(); // crashes
});
return links;
});
console.log(`wut? ${wut}`); // prints 'wut? undefined'
await browser.close();
})();
Some issues:
console.log inside the $$eval doesn't do anything.
page.goBack() and page.pause() inside the eval cause a crash.
The return value of $$eval is undefined (if I comment out page.goBack() so I get a return value at all). If I return links.length instead of links, it's correct (i.e. it's a positive integer). Huh?
I get similar results with:
const links = await page.locator('a');
await links.evaluateAll(...)
Clearly I don't know what I'm doing. What's the correct code to achieve something like this?
(X-Y problem alert: I don't actually care if I do this with $$eval, Playwright, or frankly even Javascript; all I really want to do is make this work in any language or tool).

const { context } = await launch({ slowMo: 250 });
const page = await context.newPage();
await page.goto('https://stackoverflow.com/questions/70702820/how-can-i-click-on-all-links-matching-a-selector-with-playwright');
const links = page.locator('a:visible');
const linksCount = await links.count();
for (let i = 0; i < linksCount; i++) {
await page.bringToFront();
try {
const [newPage] = await Promise.all([
context.waitForEvent('page', { timeout: 5000 }),
links.nth(i).click({ modifiers: ['Control', 'Shift'] })
]);
await newPage.waitForLoadState();
console.log('Title:', await newPage.title());
console.log('URL: ', page.url());
await newPage.close();
}
catch {
continue;
}
}
There's a number of ways you could do this, but I like this approach the most. Clicking a link, waiting for the page to load, and then going back to the previous page has a lot of problems with it - most importantly is that for many pages the links might change every time the page loads. Ctrl+shift+clicking opens in a new tab, which you can access using the Promise.all pattern and catching the 'page' event.
I only tried this on this page, so I'm sure there's tons of other problems that my arise. But for this page in particular, using 'a:visible' was necessary to prevent getting stuck on hidden links. The whole clicking operation is wrapped in a try/catch because some of the links aren't real links and don't open a new page.
Depending on your use case, it may be easiest just to grab all the hrefs from each link:
const links = page.locator('a:visible');
const linksCount = await links.count();
const hrefs = [];
for (let i = 0; i < linksCount; i++) {
hrefs.push(await links.nth(i).getAttribute('href'));
}
console.log(hrefs);

Try this approach.I will use typescript.
await page.waitForSelector(selector,{timeout:10000});
const links = await page.$$(selector);
for(const link of links)
{
await link.click({timeout:8000});
//your additional code
}
See more on https://youtu.be/54OwsiRa_eE?t=488

Custom Function Not Defined Puppeteer

I made this custom function and put it outside globally which normally would work. I also tried moving it inside the main async puppeteer function but also doesn't work. Its a simple function. In each page evaluate function I call this and pass the selector. But, its saying not defined and promise rejection which is weird because the function isn't a promise....Please help
const grabDomConvertNodlistToArray = (grabDomHtmlPath) => {
// grabbing node list from html selector all
const nList = document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
};
I tried turning the function into an async function adding a new parameter page. I then added async to my evaluate function and then passes the puppeteer page as an argument and still errors and not working.
const grabDomConvertNodlistToArray = async (page, grabDomHtmlPath) => {
try {
// grabbing node list from html selector all
const nList = await page.document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
} catch (error) {
console.log(error);
}
};
So I have your typical puppeteer setup where you awai browser.newPage() then you goto(url). Then i added this;
await page.exposeFunction("grabDomConvertNodlistToArray", grabDomConvertNodlistToArray);
added async to my evaluate callback function aka async() => {}. But still when calling my custom function inside the above evaluate function it doesn't work for some reason.
Found A Solution But, It Doesn't Work For Me. I'm Getting array.forEach is not a method which indicates to me that inside my grabDomConvertNodlistToArray function its not grabbing the nodeList or converting it into an array. If it did then forEach would be a function.
Solution 3
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(someURL);
var functionToInject = function(){
return 1+1;
}
var otherFunctionToInject = function(input){
return 6
}
await page.exposeFunction("functionToInject", functionToInject)
await page.exposeFunction("otherFunctionToInject", otherFunctionToInject)
var data = await page.evaluate(async function(){
console.log('woo I run inside a browser')
return await functionToInject() + await otherFunctionToInject();
});
return data
So erase the two functions above and convert it to use my function below.
const grabDomConvertNodlistToArray = (grabDomHtmlPath) => {
// grabbing node list from html selector all
const nList = document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
};
Running my js file results in an error of array.forEach isn't a function which is weird because if the function worked as intended the const array inside my evaluate function would be an array because its = to the above function which is returning an array. So.....idk whats going on think it has something to do with the document.querySelectorAll() line.
const rlData = async () => {
const browser = await puppeteer.launch(
{
headless: true,
},
{
args: ["--flag-switches-begin", "--disable-features=OutOfBlinkCors", "--flag-switches-end"],
}
);
const pageBodies = await browser.newPage();
await pageBodies.goto("https://test.com/bodies", {
waitUntil: "load",
});
const grabDomConvertNodlistToArray = (grabDomHtmlPath) => {
// grabbing node list from html selector all
const nList = document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
};
await pageBodies.exposeFunction("grabDomConvertNodlistToArray", grabDomConvertNodlistToArray);
const rlBodyNames = await pageBodies.evaluate(async () => {
// grabs all elements in html to make nodelist & converts it to an array
const array = grabDomConvertNodlistToArray(".testbodies > div > h1");
// push the data collected from array into data array and returned
const data = [];
array.forEach((element) => {
data.push(element.textContent);
});
return data;
});
}
rlData();
Guess I'm going to have to move the document.querySelectorAll functionality out of the custom function and back in the evaluate. However, the whole reason of making that custom function was to reduce the same code being used multiple times since my overall crawler is 238 lines long with a lot of repetitiveness. Not being able to call custom functions like mine is horrible for refactoring same code executions.
I gave up trying to get this to work and decided just to do it this way. Yeah it makes your code repetitive if you have more pages to scrape so you will be using the same code many times which is what I was trying to avoid but, puppeteer is the worse for refactoring your code maybe down the line the developers of said package will add the ability to easily use custom functions like how I was trying too.
const testNames = await pageBodies.evaluate(() => {
const nodeList = document.querySelectorAll(".test > div h2");
const array = Array.from(nodeList);
const data = [];
array.forEach((element) => {
data.push(element.textContent);
});
return data;
});

exposeFunction() is not suitable for your case: the exposed function is intended to tranfer data between browser and Node.js contexts so it can be wrapped under the hood in a code that serialize and deserialize arguments and returned data and some unserializable data (as DOM elements) can be lost. Try this instead:
const rlData = async () => {
const browser = await puppeteer.launch(
{
headless: true,
},
{
args: ["--flag-switches-begin", "--disable-features=OutOfBlinkCors", "--flag-switches-end"],
}
);
const pageBodies = await browser.newPage();
await pageBodies.evaluateOnNewDocument(() => {
window.grabDomConvertNodlistToArray = function grabDomConvertNodlistToArray(grabDomHtmlPath) {
// grabbing node list from html selector all
const nList = document.querySelectorAll(grabDomHtmlPath);
// converting nodelist to array to be returned
const array = Array.from(nList);
return array;
}
});
await pageBodies.goto("https://test.com/bodies", {
waitUntil: "load",
});
const rlBodyNames = await pageBodies.evaluate(() => {
// grabs all elements in html to make nodelist & converts it to an array
const array = grabDomConvertNodlistToArray(".testbodies > div > h1");
// push the data collected from array into data array and returned
const data = [];
array.forEach((element) => {
data.push(element.textContent);
});
return data;
});
}
rlData();

Cannot get querySelectorAll to work with puppeteer (returns undefined)

I'm trying to practice some web scraping with prices from a supermarket. It's with node.js and puppeteer. I can navigate throught the website in beginning with accepting cookies and clicking a "load more button". But then when I try to read div's containing the products with querySelectorAll I get stuck. It returns undefined even though I wait for a specific div to be present. What am I missing?
Problem is at the end of the code block.
const { product } = require("puppeteer");
const scraperObjectAll = {
url: 'https://www.bilkatogo.dk/s/?query=',
async scraper(browser) {
let page = await browser.newPage();
console.log(`Navigating to ${this.url}`);
await page.goto(this.url);
// accept cookies
await page.evaluate(_ => {
CookieInformation.submitAllCategories();
});
var productsRead = 0;
var productsTotal = Number.MAX_VALUE;
while (productsRead < 100) {
// Wait for the required DOM to be rendered
await page.waitForSelector('button.btn.btn-dark.border-radius.my-3');
// Click button to read more products
await page.evaluate(_ => {
document.querySelector("button.btn.btn-dark.border-radius.my-3").click()
});
// Wait for it to load the new products
await page.waitForSelector('div.col-10.col-sm-4.col-lg-2.text-center.mt-4.text-secondary');
// Get number of products read and total
const loadProducts = await page.evaluate(_ => {
let p = document.querySelector("div.col-10.col-sm-4.col-lg-2").innerText.replace("INDLÆS FLERE", "").replace("Du har set ","").replace(" ", "").replace(/(\r\n|\n|\r)/gm,"").split("af ");
return p;
});
console.log("Products (read/total): " + loadProducts);
productsRead = loadProducts[0];
productsTotal = loadProducts[1];
// Now waiting for a div element
await page.waitForSelector('div[data-productid]');
const getProducts = await page.evaluate(_ => {
return document.querySelectorAll('div');
});
// PROBLEM HERE!
// Cannot convert undefined or null to object
console.log("LENGTH: " + Array.from(getProducts).length);
}

The callback passed to page.evaluate runs in the emulated page context, not in the standard scope of the Node script. Expressions can't be passed between the page and the Node script without careful considerations: most importantly, if something isn't serializable (converted into plain JSON), it can't be transferred.
querySelectorAll returns a NodeList, and NodeLists only exist on the front-end, not the backend. Similarly, NodeLists contain HTMLElements, which also only exist on the front-end.
Put all the logic that requires using the data that exists only on the front-end inside the .evaluate callback, for example:
const numberOfDivs = await page.evaluate(_ => {
return document.querySelectorAll('div').length;
});
or
const firstDivText = await page.evaluate(_ => {
return document.querySelector('div').textContent;
});

What is the source of my 'Execution Context Destroyed' error?

I am creating a program to scrape forum responses for the online Uni I work for. I managed to successfully navigate to the appropriate pages, but when I tried to include scraping for the list of names of the learners who have responded I receive an 'Execution context was destroyed error'.
So far I tried moving around page.waitFor() methods with varying amounts of timeouts.
const nameLinkList = await page.$$eval(
'.coursename',
(courseLinks => courseLinks.map(link => {
const a = link.querySelector('.coursename > a');
return {
name: a.innerText,
link: a.href
};
}))
);
for (const {
name,
link
} of nameLinkList) {
await Promise.all([
page.waitForNavigation(),
page.goto(link),
page.waitFor(2000),
]);
let [button] = await page.$x("//a[contains(., 'Self')]");
if (button) {
await button.click();
} else {
console.log(name);
console.log('Didnt find link');
}
fs.appendFile('out.csv', name + '\n');
await page.waitFor(1000);
var elementExists = await page.$$('.author .media-body');
if (elementExists) {
await console.log(name);
await page.waitFor(500);
for (let z of elementExists) {
const studentName = await z.$eval('a', a => a.innerText);
await page.waitFor(2000)
await console.log(studentName);
}
}
await page.goto('www.urlwiththelistofcourses.com');
}
I expected it to iterate through each page, logging first the name of the course, followed by the names of any students who posted on the courses particular forum. The thing that confuses me is that unlike previous errors which got stuck on a particular iteration this one is variable, usually in the same area, around the 12-17th iteration, sometimes even earlier.

It seems that a combination of adjusting the waitFor here:
fs.appendFile('out.csv', name + '\n');
await page.waitFor(1000);
var elementExists = await page.$$('.author .media-body');
to 2000, combined with disabling the rendering of css and images solved the problem. The program must have linked away before entering the loop if the page loaded too slowly.

Puppeteer: How to get the contents of each element of a nodelist?

I'm trying to achieve something very trivial: Get a list of elements, and then do something with the innerText of each element.
const tweets = await page.$$('.tweet');
From what I can tell, this returns a nodelist, just like the document.querySelectorAll() method in the browser.
How do I just loop over it and get what I need? I tried various stuff, like:
[...tweets].forEach(tweet => {
console.log(tweet.innerText)
});

page.$$():
You can use a combination of elementHandle.getProperty() and jsHandle.jsonValue() to obtain the innerText from an ElementHandle obtained with page.$$():
const tweets = await page.$$('.tweet');
for (let i = 0; i < tweets.length; i++) {
const tweet = await (await tweets[i].getProperty('innerText')).jsonValue();
console.log(tweet);
}
If you are set on using the forEach() method, you can wrap the loop in a promise:
const tweets = await page.$$('.tweet');
await new Promise((resolve, reject) => {
tweets.forEach(async (tweet, i) => {
tweet = await (await tweet.getProperty('innerText')).jsonValue();
console.log(tweet);
if (i === tweets.length - 1) {
resolve();
}
});
});
page.evaluate():
Alternatively, you can skip using page.$$() entirely, and use page.evaluate():
const tweets = await page.evaluate(() => Array.from(document.getElementsByClassName('tweet'), e => e.innerText));
tweets.forEach(tweet => {
console.log(tweet);
});

According to puppeteer docs here, $$ Does not return a nodelist, instead it returns a Promise of Array of ElementHandle. It's way different then a NodeList.
There are several ways to solve the problem.
1. Using built-in function for loops called page.$$eval
This method runs Array.from(document.querySelectorAll(selector)) within the page and passes it as the first argument to pageFunction.
So to get innerText is like following,
// Find all .tweet, and return innerText for each element, in a array.
const tweets = await page.$$eval('.tweet', element => element.innerText);
2. Pass the elementHandle to the page.evaluate
Whatever you get from await page.$$('.tweet') is an array of elementHandle. If you console, it will say JShandle or ElementHandle depending on the type.
Forget the hard explanation, it's easier to demonstrate.
// let's just call them tweetHandle
const tweetHandles = await page.$$('.tweet');
// loop thru all handles
for(const tweethandle of tweetHandles){
// pass the single handle below
const singleTweet = await page.evaluate(el => el.innerText, tweethandle)
// do whatever you want with the data
console.log(singleTweet)
}
Of course there are multiple ways to solve this problem, Grant Miller also answered few of them in the other answer.