Apologies for any mistake, am completely new to node.js. I am trying to get a button click event from the existing website. For example if I click login button in a www.xyz.com website I need to get that event and print button clicked in my terminal. I tried puppeteer, cheerio and axios, all these has the option to automate a click but not to read a click event. Can somebody help me with it or advice me an alternate possible approach to get it done.
const puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch({headless:false});
const page = await browser.newPage();
await page.goto('https://stackoverflow.com/questions/68126167/how-to-read-a-click-event-from-any-current-existing-website/68126333#68126333', {waitUntil: 'networkidle0'});
const element = window.document.getElementById("upvoteBtn")
element.onclick = function() {
console.log("Upvoted!")
}
})();
Getting this error for the above code
With puppeteer, you have 2 different scopes: Node.js scope and browser scope. Variables from one scope are not defined in other one. Here is a way to communicate between these scopes and to achieve your goal:
import puppeteer from 'puppeteer';
const browser = await puppeteer.launch({ headless: false, defaultViewport: null });
try {
const [page] = await browser.pages();
await page.goto('https://example.org/');
await page.exposeFunction('logClickedElement', (name) => {
console.log(`Clicked element: ${name}.`);
});
await page.evaluate(() => {
document.body.addEventListener('click', (event) => {
window.logClickedElement(event.target.tagName);
});
});
} catch (err) { console.error(err); }
If you know the ID of the button you could get the element then assign a function to it's onclick property. Although this may overwrite the existing functionality.
Say the upvote button had an ID of upvoteBtn (which you can give it in the elements panel).
const element = window.document.getElementById("upvoteBtn")
element.onclick = function() {
console.log("Upvoted!")
}
In the case of SO, the button retains its expected functionality and also prints to the console.
Related
My Code below tries to collect a bunch of hyper links that come under the class name ".jss2". However, I do not think the function within my page.evaluate() is working. When I run the code, the link_list const doesn't get displayed.
I ran the document.querySelectorAll on the Chrome console and that was perfectly fine - really having a hard time with this.
async function testing() {
const browser = await puppeteer.launch({headless:false});
const page = await browser.newPage();
await page.setViewport({width: 1200, height: 800});
await page.goto(url);
const link_list = await this.page.evaluate(() => {
let elements = Array.from(document.querySelectorAll(".jss2"));
let links = elements.map(element => {
return element.href;
});
return (links);
});
console.log(link_list);
}
const link_list = await page.$$eval('.classname', links => links.map(link => link.href));
Found the answer here: PUPPETEER - unable to extract elements on certain websites using page.evaluate(() => document.querySelectorAll())
I'm using Playwright to scrape some data. How do I click on all links on the page matching a selector?
const { firefox } = require('playwright');
(async () => {
const browser = await firefox.launch({headless: false, slowMo: 50});
const page = await browser.newPage();
await page.goto('https://www.google.com');
page.pause(); // allow user to manually search for something
const wut = await page.$$eval('a', links => {
links.forEach(async (link) => {
link.click(); // maybe works?
console.log('whoopee'); // doesn't print anything
page.goBack(); // crashes
});
return links;
});
console.log(`wut? ${wut}`); // prints 'wut? undefined'
await browser.close();
})();
Some issues:
console.log inside the $$eval doesn't do anything.
page.goBack() and page.pause() inside the eval cause a crash.
The return value of $$eval is undefined (if I comment out page.goBack() so I get a return value at all). If I return links.length instead of links, it's correct (i.e. it's a positive integer). Huh?
I get similar results with:
const links = await page.locator('a');
await links.evaluateAll(...)
Clearly I don't know what I'm doing. What's the correct code to achieve something like this?
(X-Y problem alert: I don't actually care if I do this with $$eval, Playwright, or frankly even Javascript; all I really want to do is make this work in any language or tool).
const { context } = await launch({ slowMo: 250 });
const page = await context.newPage();
await page.goto('https://stackoverflow.com/questions/70702820/how-can-i-click-on-all-links-matching-a-selector-with-playwright');
const links = page.locator('a:visible');
const linksCount = await links.count();
for (let i = 0; i < linksCount; i++) {
await page.bringToFront();
try {
const [newPage] = await Promise.all([
context.waitForEvent('page', { timeout: 5000 }),
links.nth(i).click({ modifiers: ['Control', 'Shift'] })
]);
await newPage.waitForLoadState();
console.log('Title:', await newPage.title());
console.log('URL: ', page.url());
await newPage.close();
}
catch {
continue;
}
}
There's a number of ways you could do this, but I like this approach the most. Clicking a link, waiting for the page to load, and then going back to the previous page has a lot of problems with it - most importantly is that for many pages the links might change every time the page loads. Ctrl+shift+clicking opens in a new tab, which you can access using the Promise.all pattern and catching the 'page' event.
I only tried this on this page, so I'm sure there's tons of other problems that my arise. But for this page in particular, using 'a:visible' was necessary to prevent getting stuck on hidden links. The whole clicking operation is wrapped in a try/catch because some of the links aren't real links and don't open a new page.
Depending on your use case, it may be easiest just to grab all the hrefs from each link:
const links = page.locator('a:visible');
const linksCount = await links.count();
const hrefs = [];
for (let i = 0; i < linksCount; i++) {
hrefs.push(await links.nth(i).getAttribute('href'));
}
console.log(hrefs);
Try this approach.I will use typescript.
await page.waitForSelector(selector,{timeout:10000});
const links = await page.$$(selector);
for(const link of links)
{
await link.click({timeout:8000});
//your additional code
}
See more on https://youtu.be/54OwsiRa_eE?t=488
I'm learning to use puppeteer but I'm running into trouble. I'm trying to create a program which takes in a date and finds a famous persons whose birthday is on that date. I have this code:
const puppeteer = require('puppeteer');
try {
(async () => {
console.log('here');
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.famousbirthdays.com/');
console.log('me');
await page.type(document.querySelector('input'), '11-16-1952');
console.log('you clicked');
await page.click(document.querySelector('button'));
console.log('Here');
await page.waitForSelector(
document.querySelector('div[class="list_page"]')
);
let data = await page.evaluate(() => {
let name = document.querySelector('div[class="name"').textContent;
return { name };
});
console.log(data);
browser.close();
})();
} catch (err) {
console.error(err);
}
Im not understand why I'm getting errors at the page.type line? I get an error and cant reach that log of "you clicked". If I read the documentation correctly, .click can take in a selector and text to type into it so I'm pretty sure im using it correctly. I checked on the browser console and document.querySelector('input') does pull up the element I want(the search bar). Any advice is appreciated. Thanks for looking.
I'm trying to figure out how to get the elements in e.g. a JS gallery that loads its images after they have been clicked on.
I'm using a demo of Viewer.js as an example. The element with the classes .viewer-move.viewer-transition isn't in the initial DOM. After clicking on an image the element is available but if I use $eval the string is empty. If I open the console in the Puppeteer browser and execute document.querySelector('.viewer-move.viewer-transition') I get the element but in the code the element isn't available.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://fengyuanchen.github.io/viewerjs/');
await page.click('[data-original="images/tibet-1.jpg"]');
let viewer = await page.$eval('.viewer-move.viewer-transition', el => el.innerHTML);
console.log(viewer);
})();
You get the empty string because the element has no content so inner HTML is empty. outerHTML seems working:
const puppeteer = require('puppeteer');
(async function main() {
try {
const browser = await puppeteer.launch({ headless: false });
const [page] = await browser.pages();
await page.goto('https://fengyuanchen.github.io/viewerjs/');
await page.click('[data-original="images/tibet-1.jpg"]');
await page.waitForSelector('.viewer-move.viewer-transition');
const viewer = await page.$eval('.viewer-move.viewer-transition', el => el.outerHTML);
console.log(viewer);
await browser.close();
} catch (err) {
console.error(err);
}
})();
Since you have to wait until it is available, the most convenient method would be to use await page.waitForSelector(".viewer-move.viewer-transition") which would wait util the element is added to DOM, although this has the caveat that this continues execution the moment that the element is added to DOM, even if it is empty/hidden.
When I open this page with puppeteer, and try to click the element, it throws an Error when it is expected to be possible to click the element.
const puppeteer = require('puppeteer');
const url = "https://www.zapimoveis.com.br/venda/apartamentos/rj+rio-de-janeiro/";
async function run() {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto(url, {waitUntil: 'load'});
const nextPageSelector = "#list > div.pagination > #proximaPagina";
await page.waitForSelector(nextPageSelector, {visible: true})
console.log("Found the button");
await page.click(nextPageSelector)
console.log('clicked');
}
run();
Here's a working version of your code.
const puppeteer = require('puppeteer');
const url = "https://www.zapimoveis.com.br/venda/apartamentos/rj+rio-de-janeiro/";
async function run() {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto(url);
const nextPageSelector = "#list > div.pagination > #proximaPagina";
console.log("Found the button");
await page.evaluate(selector=>{
return document.querySelector(selector).click();
},nextPageSelector)
console.log('clicked');
}
run();
I personally prefer to use page.evaluate, as page.click doesn't work for me neither in some cases and you can execute whatever js on the page directly.
The only thing to know is the syntax :
- first param : function to execute
- second param : the variable(s) to be passed to this function
Found the problem.
When you page.click(selector) or ElementHandle.click(), Puppeteer scrolls to the element, find its center coordinates, and finaly clicks. It uses the function _clickablePoint at node_modules/puppeteer/lib/JSHandle.js to find the coordinates.
The problem with this website (zapimoveis) is that the scroll into the element's viewport is too slow, so Puppeteer can't find its (x,y) center.
One nice way you can click on this element is to use page.evaluate to click it using page javascript. But, there is a gambi's way that I prefer. :) Change the line await page.click(nextPageSelector) by these lines:
try { await page.click(nextPageSelector) } catch (error) {} // sacrifice click :)
await page.waitFor(3000); // time for scrolling
await page.click(nextPageSelector); // this will work