Develop Reference

Parse table from HTML with cheerio in node

I'm trying to scrape data from this website (https://www.brvm.org/fr/cours-actions/0) using nodejs in visual studio code.
With the code I can get the tr elements but I want to make a loop which will push every children td of a tr to create a table in html.
Below is my code
import fetch from "node-fetch";
import cheerio from "cheerio";
const scrapedData = [];
async function fetchData(url) {
const response = await fetch(url);
const data = await response.text();
/* console.log(data); */
getData(data);
}
fetchData("https://www.brvm.org/fr/cours-actions/0");
function getData(html) {
const $ = cheerio.load(html);
$("#block-system-main > div > table > tbody", html).each(function () {
console.log($(this).text());
});
}
How can I create the loop?

You can loop over the rows, then use .find on each row to grab its cells:
import cheerio from "cheerio"; // 1.0.0-rc.12
const parseHTML = html => {
const $ = cheerio.load(html);
const sel = "#block-system-main > div > table > tbody tr";
return [...$(sel)].map(e =>
[...$(e).find("td")].map(e => $(e).text())
);
};
(async () => {
const url = "https://www.brvm.org/fr/cours-actions/0";
const response = await fetch(url);
const result = parseHTML(await response.text());
console.table(result);
})();
See also Scraping all rows from table using cheerio.

querySelector on custom elements won't work in external files

So I have this code:-
const extendCSS = (el1, el2) =>{
Array.prototype.slice.call(document.querySelector(el1).attributes).forEach(function(item) {
el2.setAttribute(item.name, item.value);
});
}
const build = (elementType) => tag => {
const query = document.querySelectorAll(tag);
query.forEach(ptag => {
const shadow = ptag.attachShadow({
mode: 'open'
});
const element = document.createElement(elementType);
element.innerHTML = ptag.innerHTML;
extendCSS(tag, element);
element.setAttribute('id', tag);
shadow.host.parentNode.replaceChild(element, shadow.host);
});
};
const h1 = build('h1');
const textarea = build('textarea');
textarea("text-editor");
h1("text-value");
const texteditor = document.querySelector('#text-editor');
const textvalue = document.querySelector('#text-value');
texteditor.addEventListener('keydown', ()=>{
textvalue.innerHTML = texteditor.value;
});
<text-editor></text-editor>
<text-value></text-value>
This build() does one thing:- it selects the custom element, changes its tag type and assign a id of name of the custom tag and replaces it with the desired thing. It is working fine here. but if I code all the thing after the build function in a external .js file and then bind it to the html and run it, but it won't work. How can I resolve this? This is a very important for now.
Help and answers appreciated. Thanks in advance.

How to find document.activeElement in Puppeteer

i want to autofill a form with puppeteer.
I fill out the first input, then click on a button that then creates a new input field that has focus.
How can i select this input? Can i use document.activeElement and how?
let newActivity = 'button.new_activity'
await page.waitForSelector(newActivity)
await page.click(newActivity)
// find active/focused input
await page.type(focusedInput, 'message')

You can use evaluateHandle to get the element handle, and then call the type function on that element.
const el = await page.evaluateHandle(() => document.activeElement);
await el.type('message');

function findFocusedNode(node) {
if (node.focused) {
return node;
}
for (const child of node.children || []) {
const focusedNode = findFocusedNode(child);
if (focusedNode) {
return focusedNode;
}
}
}
const snapshot = await page.accessibility.snapshot();
const focusedNode = findFocusedNode(snapshot);
console.log('focusedNode', focusedNode);
https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#class-accessibility

Iterate over elements in Nodejs

I have a webpage, where I want to hover over all anchor tags and get the styles computed for that tag. This function which I wrote doesn't seem to work as it gives me original style of the anchor and not the hover styles.
Please help.
let data = await page.evaluate(() => {
let elements = document.getElementsByTagName('a');
properties = []
for (var element of elements){
element.focus();
properties.push(JSON.parse(JSON.stringify(window.getComputedStyle(element, null)["backgroundColor"])));
}
return properties;
});

https://developer.mozilla.org/en-US/docs/Web/API/Window/getComputedStyle
try document.getComputedStyle(element, ':hover')

First of all, you should convert results from document.getElementsByTagName to normal array
const elements = [...document.getElementsByTagName('textarea')];
Next to get element property use this syntax:
window.getComputedStyle(element).getPropertyValue("background-color")
Finally, this is a fully working example:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://css-tricks.com/almanac/selectors/f/focus/');
const data = await page.evaluate(() => {
const elements = document.getElementsByTagName('textarea');
return [...elements].map(element => {
element.focus();
return window.getComputedStyle(element).getPropertyValue("background-color");
});
});
console.log(data);
await browser.close();
})();

You can use page.$$() to obtain an ElementHandle array of textarea elements.
Then, you can use the elementHandle.hover() to hover over each element and then page.evaluate() to obtain the computed background color to push to your data array:
const elements = await page.$$( 'textarea' );
const data = [];
for ( let i = 0; i < elements.length; i++ )
{
await elements[i].hover();
data.push( await page.evaluate( element => window.getComputedStyle( element ).backgroundColor, elements[i] ) );
}
console.log( data );

In Puppeteer, how do I get the innerHTML of a selector?

At the moment, I'm trying the following:
const element = await page.$("#myElement");
const html = element.innerHTML;
I'm expecting the HTML to be printed, but instead I'm getting undefined.
What am I doing wrong?

page.evaluate():
You can use page.evaluate() to get the innerHTML of an element:
const inner_html = await page.evaluate(() => document.querySelector('#myElement').innerHTML);
console.log(inner_html);
elementHandle.getProperty() / .jsonValue():
Alternatively, if you must use page.$(), you can access the innerHTML using a combination of elementHandle.getProperty() and elementHandle.jsonValue():
const inner_html = await (await (await page.$('#myElement')).getProperty('innerHTML')).jsonValue();
console.log(inner_html);

You can use page.$eval to access innerHTML pf specified DOM.
Snippet sing page.$eval
const myContent = await page.$eval('#myDiv', (e) => e.innerHTML);
console.log(myContent);
Works great with jest-puppeter.

You have to use an asynchronous function evaluate.
In my case, I was using puppeteer/jest and all await* solutions was trowing
error TS2531: Object is possibly 'null'
innerHTML with if statement was working for me(part of example)
I was searching the member's table for the first unchecked element
while (n) {
let elementI = await page.$('div >' + ':nth-child(' + i + ')' + '> div > div div.v-input__slot > div');
let nameTextI = await page.evaluate(el => el.innerHTML , elementI);
console.log('Mistery' + nameTextI);
{...}
i++;
{...}
}

const html = await page.$eval("#myElement", e => e.innerHTML);