Develop Reference

Pupeteer Clicking a button on a modal but only works but only 2 - 3 out of 10 times

I am attempting to scrape deck lists from aetherhub for personal use. when you get to the page you have to click to make a modal popup and then no matter what I tried I could not make it copy the text in the body of the modal. Second option is to have it copy to the clipboard and then save that to a variable and then work with the string. Bingo! I made it connect and copy and return the deck list. The problem I am having is that I can not get it to work every time. I have tried putting in waits and delays to try and see that would help but i can not seem to get it to work every time. I mostly get this error "Error: Node is either not visible or not an HTMLElement"
const puppeteer = require('puppeteer')
async function getcardlist(url) {
try {
const browser = await puppeteer.launch({headless: false})
const page = await browser.newPage()
const context = await browser.defaultBrowserContext()
await context.overridePermissions(url, ['clipboard-read'])
await page.goto(url, {waitUntil: 'load'})
const exportButton = await page.$('li.nav-item:nth-child(5) > a:nth-child(1)')
await exportButton.click()
await page.waitForSelector('a.mtgaExport')
const mtgaFormatButton = await page.$('a.mtgaExport')
await mtgaFormatButton.click()
await page.waitForSelector('#exportSimpleBtn')
const simplebutton = await page.$('#exportSimpleBtn')
await simplebutton.click()
await page.$('.modal.show', { waitUntil: 'load' })
await page.waitForSelector('.modal-footer > #exportListbtn')
const toClipBoard = await page.$('.modal-footer > #exportListbtn')
await toClipBoard.click()
const copiedText = await page.evaluate(`(async () => await navigator.clipboard.readText())()`)
await browser.close()
return copiedText
} catch (err) {
console.error(err);
}
}
getcardlist('https://aetherhub.com/Deck/rakdos-menacing-menaces')
.then(returnVal => console.log((returnVal)))

When you get the Error
"Error: Node is either not visible or not an HTMLElement"
It's basically saying that's the requested button/element is not found on the page.
so even if you want or do a page.waitForSelector you will get an error (because it does not exist in your DOM). so use headless: false, and inspect element to see if you find your selector

Unable to type after click with Puppeteer

I've tried almost anything but still unable to insert username after clicking input.
I know there is other solutions to do so without clicking I also tried but still not working...
How to fill an input field using Puppeteer?
Unable to login with Puppeteer
Click is working and cursor blinking now all I need is just to 'type' something.
const puppeteer = require('puppeteer');
const creds = {
email: "myemail",
password: "mypassword",
};
(async () => {
const browser = await puppeteer.launch({headless: false, args: ['--start-maximized']});
const page = await browser.newPage();
//set viewport
await page.setViewport({
width: 1920,
height: 1080,
})
await page.goto('https://qa.traffilog.co.il/new_web/index.htm', {waitUntil: 'networkidle0'});
const username = await page.$x('/html/body/div[1]/div/div[1]/div/div[2]/div/div[1]/div/div[2]/div/div[3]/div/div[1]/div/div/div/div/div/div[2]/div/div[1]/div/input[1]');
await username[0].click();
await page.waitForNavigation({ waitUntil: 'networkidle0' });
await page.type(username, creds.email);
await page.waitForNavigation({ waitUntil: 'networkidle0' });
// await browser.close();
})
();

From what I've read in the docs Page.type takes a selector not an ElementHandle. But because you are using an ElementHandle you could try to set the input value by using username[0].value=creds.email

How to click a specific div with a specific class?

I'm new at coding in puppeteer, and I wanted to know how to make it click this: (image)
The code I have rn is this one:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('page link is here');
await page.screenshot({ path: 'game.png' });
const [button] = await page.$x("//button[contains(., 'Accept')]");
if (button) {
await button.click();
}
I want to click it here.
await page.screenshot({ path: 'test.png' });
await browser.close();
})();
Sorry for my bad English 😔👌

If the element highlighted in the screenshot is the one to be clicked, you can simply:
await page.click('.shipyard-item');
I'd like to suggest the excellent puppeteer documentation to consult with for most the use cases.

puppeteer can't get page source after using evaluate()

I'm using puppeteer to interact with a website using the evaluate() function to maniupulate page front (i.e to click on certain items etc...), click through works fine but I can't return the page source after clicking using evaluate.
I have recreated the error in this simplified script below it loads google.com, clicks on 'I feel lucky' and should then return the page source of the loaded page:
const puppeteer = require('puppeteer');
async function main() {
const browser = await puppeteer.launch({
headless: false,
args: ['--no-sandbox']
});
const page = await browser.newPage();
await page.goto('https://www.google.com/', {waitUntil: 'networkidle2'});
response = await page.evaluate(() => {
document.getElementsByClassName('RNmpXc')[1].click()
});
await page.waitForNavigation({waitUntil: 'load'});
console.log(response.text());
}
main();
I get the following error:
TypeError: Cannot read property 'text' of undefined
UPDATE New code following suggestion to use page.content()
const puppeteer = require('puppeteer');
async function main() {
const browser = await puppeteer.launch({
headless: false,
args: ['--no-sandbox']
});
const page = await browser.newPage();
await page.goto('https://www.google.com/', {waitUntil: 'networkidle2'});
await page.evaluate(() => {
document.getElementsByClassName('RNmpXc')[1].click()
});
const source = await page.content()
console.log(source);
}
main();
I am now getting the following error:
Error: Execution context was destroyed, most likely because of a navigation.
My question is: How can I return page source using the .text() method after manipulating the webpage using the evaluate() method?
All suggestions / insight / proposals would be very much appreciated thanks.

Since you're asking for page source after javascript modification, I'd assume you want DOM and not the original HTML content. your evaluate function doesn't return anything which results in undefined response. You can use
const source = await page.evaluate(() => new XMLSerializer().serializeToString(document.doctype) + document.documentElement.outerHTML);
or
const source = await page.content();

Puppeteer Login to Instagram

I'm trying to login into Instagram with Puppeteer, but somehow I'm unable to do it.
Can you help me?
Here is the link I'm using:
https://www.instagram.com/accounts/login/
I tried different stuff. The last code I tried was this:
const puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.instagram.com/accounts/login/');
await page.evaluate();
await afterJS.type('#f29d14ae75303cc', 'username');
await afterJS.type('#f13459e80cdd114', 'password');
await page.pdf({path: 'page.pdf', format: 'A4'});
await browser.close();
})();
Thanks in advance!

OK you're on the right track but just need to change a few things.
Firstly, I have no idea where your afterJS variable comes from? Either way you won't need it.
You're asking for data to be typed into the username and password input fields but aren't asking puppeteer to actually click on the log in button to complete the log in process.
page.evaluate() is used to execute JavaScript code inside of the page context (ie. on the web page loaded in the remote browser). So you don't need to use it here.
I would refactor your code to look like the following:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.instagram.com/accounts/login/');
await page.waitForSelector('input[name="username"]');
await page.type('input[name="username"]', 'username');
await page.type('input[name="password"]', 'password');
await page.click('button[type="submit"]');
// Add a wait for some selector on the home page to load to ensure the next step works correctly
await page.pdf({path: 'page.pdf', format: 'A4'});
await browser.close();
})();
Hopefully this sets you down the right path to getting past the login page!
Update 1:
You've enquired about parsing the text of an element on Instagram... unfortunately I don't have an account on there myself so can't really give you an exact solution but hopefully this still proves of some value.
So you're trying to evaluate an elements text, right? You can do this as follows:
const text = await page.$eval(cssSelector, (element) => {
return element.textContent;
});
All you have to do is replace cssSelector with the selector of the element you wish to retrieve the text from.
Update 2:
OK lastly, you've enquired about scrolling down to an element within a parent element. I'm not going to steal the credit from someone else so here's the answer to that:
How to scroll to an element inside a div?
What you'll have to do is basically follow the instructions in there and get that to work with puppeteer similar to as follows:
await page.evaluate(() => {
const lastLink = document.querySelectorAll('h3 > a')[2];
const topPos = lastLink.offsetTop;
const parentDiv = document.querySelector('div[class*="eo2As"]');
parentDiv.scrollTop = topPos;
});
Bear in mind that I haven't tested that code - I've just directly followed the answer in the URL I've provided. It should work!

You can log in to Instagram using the following example code:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
// Wait until page has loaded
await page.goto('https://www.instagram.com/accounts/login/', {
waitUntil: 'networkidle0',
});
// Wait for log in form
await Promise.all([
page.waitForSelector('[name="username"]'),
page.waitForSelector('[name="password"]'),
page.waitForSelector('[name="submit"]'),
]);
// Enter username and password
await page.type('[name="username"]', 'username');
await page.type('[name="password"]', 'password');
// Submit log in credentials and wait for navigation
await Promise.all([
page.click('[type="submit"]'),
page.waitForNavigation({
waitUntil: 'networkidle0',
}),
]);
// Download PDF
await page.pdf({
path: 'page.pdf',
format: 'A4',
});
await browser.close();
})();