I am trying to create an app that does a google image search of a random word and selects/clicks the first result image.
I am successful until the code is attempting to select the result image and it throws the following error in my terminal:
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection
id: 1): Error: Protocol error (Runtime.callFunctionOn): Cannot find
context with specified id undefined
Here is my code:
const pup = require('puppeteer');
const random = require('random-words');
const url = 'http://images.google.com';
(async() => {
const browser = await pup.launch({headless: false});
const page = await browser.newPage();
await page.goto(url);
const searchBar = await page.$('#lst-ib');
await searchBar.click();
await page.keyboard.type(`${random()}`);
const submit = await page.$('#mKlEF');
await submit.click();
await page.keyboard.type(random());
await page.keyboard.press('Enter');
const pic = await page.evaluate(() => {
return document.querySelectorAll('img');
});
pic.click();
})();
document.querySelectorAll('img') is not serialisable, so it returns undefined (see this issue as reference)
Please use something like: (depends on which element you want to click)
await page.$$eval('img', elements => elements[0].click());
This is a long dead thread but if anyone runs into this issue and the above answer does not apply to you, try adding a simple await page.waitForTimeout(2000). My test was completing but I was getting this error when attempting to await browser.close(); Adding the wait after searching for my final selector seems to have resolved the issue.
Related
When I run the script, the automated web browser does load and so does the website I wanted it to load but it shuts off instantly without doing anything else. I had looked at the docs and used the wait method in case the website I was trying to load does a lot of AJAX loading but it still doesn't seem to be working. Can anyone please explain what is going on and how I am getting this error?
Here is the JavaScript code:
const {Builder, By, until} = require('selenium-webdriver');
(async function helloSelenium() {
let driver = await new Builder().forBrowser('safari').build();
await driver.get('https://www.google.com/?client=safari');
let ele = await driver.wait(until.elementLocated(By.id('gbqfbb'), 10000));
let b = await driver.findElement(By.id('gbqfbb'));
console.log(b.getText());
await driver.quit();
})();
And here is the error:
user#MacBook-Pro webscraper % node webscraper.js
Promise { <pending> }
/Users/user/Documents/Web/webscraper/node_modules/selenium-webdriver/lib/error.js:522
let err = new ctor(data.message)
^
NoSuchSessionError
at Object.throwDecodedError (/Users/user/Documents/Web/webscraper/node_modules/selenium-webdriver/lib/error.js:522:15)
at parseHttpResponse (/Users/user/Documents/Web/webscraper/node_modules/selenium-webdriver/lib/http.js:548:13)
at Executor.execute (/Users/user/Documents/Web/webscraper/node_modules/selenium-webdriver/lib/http.js:474:28)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Driver.execute (/Users/user/Documents/Web/webscraper/node_modules/selenium-webdriver/lib/webdriver.js:735:17) {
remoteStacktrace: ''
}
Try to change the below
From
let ele = await driver.wait(until.elementLocated(By.id('gbqfbb'), 10000));
to
let ele = await driver.wait(until.elementLocated(By.id('gbqfbb')), 10000);
t's also important to notice that if this method throws an exception (timeout exception), so you must try-catch it too.
I'm trying to scrape the link to the next page from this webpage. I know how to scrape that using css selector. However, things go wrong when I attempt to parse the same using xpath. This is what I get instead of the next page link.
const puppeteer = require("puppeteer");
let url = "https://stackoverflow.com/questions/tagged/web-scraping";
(async () => {
const browser = await puppeteer.launch({headless:false});
const [page] = await browser.pages();
await page.goto(url,{waitUntil: 'networkidle2'});
let nextPageLink = await page.$x("//a[#rel='next']", item => item.getAttribute("href"));
// let nextPageLink = await page.$eval("a[rel='next']", elm => elm.href);
console.log("next page:",nextPageLink);
await browser.close();
})();
How can I scrape the link to the next page using xpath?
page.$x(expression) returns an array of element handles. You need either destructuring or index acces to get the first element from the array.
To get a DOM element property from this element handle, you need either evaluating with element handle parameter or element handle API.
const [nextPageLink] = await page.$x("//a[#rel='next']");
const nextPageURL = await nextPageLink.evaluate(link => link.href);
Or:
const [nextPageLink] = await page.$x("//a[#rel='next']");
const nextPageURL = await (await nextPageURL.getProperty('href')).jsonValue();
I'm trying to create a program to google search using selenium
based on this answer,
so far the code looks like this
const { Builder, By, Key, until } = require('selenium-webdriver');
const driver = new Builder().forBrowser("firefox").build();
(async () => {
await driver.get(`https://www.google.com`);
var el = await driver.findElement(By.name('q'));
await driver.wait(until.elementIsVisible(el),1000);
await el.sendKeys('selenium');
var el = await driver.findElement(By.name(`btnK`));
await driver.wait(until.elementIsVisible(el),1000);
await el.click();
console.log('...Task Complete!')
})();
but writing
var el = await driver.findElement(By.something('...'));
await driver.wait(until.elementIsVisible(el),1000);
await el.do_something();
everytime becomes difficult so I tried to make a function like this:
const { Builder, By, Key, until } = require('selenium-webdriver');
const driver = new Builder().forBrowser("firefox").build();
async function whenElement(by_identity,timeout=1000){
var el = await driver.findElement(by_identity);
await driver.wait(until.elementIsVisible(el),timeout);
return el;
}
(async () => {
await driver.get(`https://www.google.com`);
await whenElement(By.name('q')).sendKeys('selenium');
await whenElement(By.name('btnK')).click();
console.log('...Task Complete!')
})();
but it gives this ERROR:
UnhandledPromiseRejectionWarning: TypeError: whenElement(...).sendKeys
is not a function
My aim is to reduce the number of variables and make it as simple as possible
so what exactly am I doing wrong here?
There seems to be an error with the promises. You can only invoke a function on the actual element returned by the Promise and not the promise itself.
You must first wait for the promise that waits for the element whenElement to resolve and then you can use the element and wait for the promise returned by sendKeys to resolve.
const el = await whenElement(By.name('q'));
await el.sendKeys('selenium');
or
await (await whenElement(By.name('q'))).sendKeys('selenium');
or
await whenElement(By.name('q')).then(el => el.sendKeys('selenium'));
I'm learning to use puppeteer but I'm running into trouble. I'm trying to create a program which takes in a date and finds a famous persons whose birthday is on that date. I have this code:
const puppeteer = require('puppeteer');
try {
(async () => {
console.log('here');
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.famousbirthdays.com/');
console.log('me');
await page.type(document.querySelector('input'), '11-16-1952');
console.log('you clicked');
await page.click(document.querySelector('button'));
console.log('Here');
await page.waitForSelector(
document.querySelector('div[class="list_page"]')
);
let data = await page.evaluate(() => {
let name = document.querySelector('div[class="name"').textContent;
return { name };
});
console.log(data);
browser.close();
})();
} catch (err) {
console.error(err);
}
Im not understand why I'm getting errors at the page.type line? I get an error and cant reach that log of "you clicked". If I read the documentation correctly, .click can take in a selector and text to type into it so I'm pretty sure im using it correctly. I checked on the browser console and document.querySelector('input') does pull up the element I want(the search bar). Any advice is appreciated. Thanks for looking.
I'm trying to fill a form with Puppeteer on a webpage, the input has an id and I'm using it as a selector.
The ID is :
#loginPage:SiteTemplate:formulaire:login-field
When I get the selector from chrome it gives me that :
#loginPage\3a SiteTemplate\3a formulaire\3a login-field
And wether I put the first or the second option in Puppeteer it spits me out this error :
Error: Evaluation failed: DOMException: Failed to execute 'querySelector' on 'Document': '#loginPage:SiteTemplate:formulaire:login-field' is not a valid selector.
Here is the code if needed :
(async () => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto('XXX');
await page.click(GOTO_LOGIN_BUTTON_SELECTOR)
await page.waitForNavigation({waitUntil: 'load'});
await page.waitFor(EMAIL_SELECTOR); // here
await page.focus(EMAIL_SELECTOR);
await page.keyboard.type(CREDS.email);
await page.focus(PASSWORD_SELECTOR);
await page.keyboard.type(CREDS.password);
await browser.close();
})();
One option, for an ID like that, is to do as follows:
const EMAIL_SELECTOR = '[id="loginPage:SiteTemplate:formulaire:login-field"]';
Or, if that doesn't work, split it up as follows to work around the use of the ::
const EMAIL_SELECTOR = '[id*="loginPage"][id*="SiteTemplate"][id*="formulaire"][id*="login-field"]';
Hopefully one (or both) of those will help!
I'm not sure but I would drop the line await page.waitForNavigation({waitUntil: 'load'});
And replace await page.waitFor(EMAIL_SELECTOR); // here
with await page.waitForSelector(EMAIL_SELECTOR);
And test to see if just using #login-field or other #loginPage > SiteTemplate > formulaire > login-field
I could be wrong as I'm still working this out too.