Develop Reference

Convenient way to get input for puppeteer page.click()

Challenge
When using puppeteer page.click('something') one of the first challenges is to make sure that the right 'something' is provided.
I guess this is a very common challenge, yet I did not find any simple way to achieve this.
What I tried so far
In Google Chrome I inspect the element that I want to click. I then get an extensive element description with a class and such. Based on an example I found, my approach is now:
Take the class
Replace all spaces with dots
Try
If it fails, check what is around this and add it as a prefix, for example one or two instances of button.
This does not exactly feel like it is the best way (and sometimes also fails, perhaps due to inaccuracies from my side).
One thing that I notice is that Chrome actually often seems to give a hint hovering over the thing I want to click, I am not sure if that is right but I also did not see a way to copy that (and it can be quite long).
If there is a totally different recommended way (e.g. Looking in the browser for what the name roughly is, and then using puppeteer to list all possible things), that is also fine. I just want to get the right input for page.click()
If you need an example of what I am trying: If you open this question in an incognito tab, you get options like share or follow. Or if you go to a web shop like staples and want to add something to cart.

When using puppeteer page.click('something') one of the first challenges is to make sure that the right 'something' is provided.
Just to be clear, "something" is a CSS selector, so your question seems to reduce to how to write CSS selectors that are accurate. Or, since Puppeteer offers XPath and traditional DOM traversals, we could extend it to include those selection tools as well.
Broader still, if there's a data goal we're interested in, often times there are other routes to get the data that don't involve touching the document at all.
I guess this is a very common challenge, yet I did not find any simple way to achieve this.
That's because there is no simple way to achieve this. It's like asking for the one baseball swing that hits all pitches. Web pages have messy, complex, arbitrary structures that follow thousands of different conventions (or no conventions at all). They can serve up a slightly or completely different page structure on any request. There's no silver-bullet strategy for writing good CSS selectors, and no step-by-step algorithm you can apply to universally "solve" the problem of accurately and robustly selecting elements.
Your goal should be to learn the toolkit and then practice on many different pages to develop an intuition for which tools and tricks work in which contexts and be able to correctly discern the tradeoffs in different approaches. Writing a full guide to this is out of scope, and articles exist elsewhere that cover this in depth, but here are a few high-level rules of thumb:
Look at context: consider the goals of your project, the general structure of the page and patterns on the page. Too many questions on Stack Overflow regarding CSS selectors (but also in general) omit context, which severely constrains the recommendation space, often leading to an XY problem. A few factors that are often relevant:
Whether the scrape is intended to be one-off or a long-running script that should try to anticipate and be resillient to page changes over time
Development time/cost/goal tradeoffs
Whether the data can be obtained by other means than the DOM, like accessing an API, pulling a JSON blob from a <script> tag, accessing a global variable on the window or intercepting a network response.
Considering nesting: is the element in a frame or shadow DOM?
Considering whole-page context: which patterns does the site tend to follow? Are there parent elements that are useful to selecting a child? (often, this is a distant relationship, not visible in a screenshot as provided by OP)
Consider all capabilities provided by your toolkit. For example, OP asked for a selector to close a modal on Stack Overflow; it turns out that none of the elements have particularly great CSS selectors, so using Puppeteer to trigger an Esc key press might be more robust.
Keep it simple: since pages can change at any time, the more constraints you add to the selector, the more likely one of those assumptions will no longer be true, introducing unnecessary points of failure.
Look for unique identifiers first: ids are usually unique on a page (some Google pages seem to scoff at this rule), so those are usually the best bets. For elements without an id, my next steps are typically:
Look for an id in a close parent element and use that, then select the child based on its next-most-unique identifier, usually a class name or combination tag name and attribute (like an input field with a name attribute, for example).
If there are few ids or none nearby, check whether the class name or attribute that is unique. If so, consider using that, likely coupled with a parent container class.
When selecting between class names, pay attention to those that seem temporary or stateful and might be added and removed dynamically. For example, a class of .highlighted-tab might disappear when the element isn't highlighted.
Prefer "bespoke" class names that seem tied to role or logic over generic library class names associated with styling (bootstrap, semantic UI, material UI, tailwind, etc).
Avoid the > operator which can be too rigid, unless you need precision to disambiguate a tree where no other identifiers are available.
Avoid sibling selectors unless unavoidable. Siblings often have more tenuous relationships than parents and children.
Avoid nth-child and nth-of type to the extent possibe. Lists are often reordered or may have fewer or more elements than you expect.
When using anything related to text, generally trim whitespace, ignore case and special characters where appropriate and prefer substrings over exact equality. On the other hand, don't be too loose. Usually, text content and values are weak targets but sometimes necessary.
Avoid pointless steps in a selector, like body > div#container > p > .target which should just be #container .target or #container p .target. body says almost nothing, > is too rigid, div isn't necessary since we have an id (if it changes to a span our new selector will still work), and the p is generic--there are probably no .targets outside of ps anyway.
Avoid browser-generated selectors. These are usually the worst of both worlds: highly vague and rigid at the same time. The goal is to be the opposite: accurate and specific, yet as flexible as possible.
Feel free to break rules as appropriate.

Setting (ARIA) role for HTML custom elements without explicit attribute?

I have a web app that displays and passes around user-editable semantic markup. For a variety of reasons, including security, the markup consists entirely of custom elements (plus the i, b, and u tags). For regular rendering, I simply have styles for all the tags and stick them straight in the DOM. This looks and works great.
Now I'm doing screen-reader testing, and things aren't great. I have a bunch of graphical symbols I want to add labels for. I've got divs that I want to make into landmarks. I've got custom input fields and buttons.
It would be easy enough to just add role= to all the different tag instances. But part of the reason for the custom markup is to eliminate all the redundant information from the strings that get passed around (note: they're also compressed). My <ns-math> tag should always have role="math", and adding 11 bytes to what might be tags around a single character is an actual problem when multiplied out over a whole article. I know this because the app started with a bunch of <span class="... type elements instead of custom.
For the fields and buttons, I've used a shadow DOM approach. This was necessary anyway to get focus/keyboard semantics correct without polluting the semantic markup with a bunch of redundant attributes, so it's easy to also slap the ARIA stuff on the shadow elements. Especially since the inputs are all leaf nodes. But most of my custom tags amount to fancy spans, and are mostly not leaf nodes, so I don't really want to shadow them all when they're working so well in the light DOM.
After a bunch of searching, it seems like the internals.role property from "4.13.7.4 Accessibility semantics" of the HTML standard is maybe what I want. I may well be using it incorrectly (I'm a novice at front-end), but I can't seem to get this to work in recent versions of Firefox or Chrome. I don't get an error, but it seems to have no effect on the accessibility tree. My elements are not form-associated, but my reading is that the ARIAMixin should be functional anyway. This is maybe a working draft? If this is supposed to work in current browsers, does anybody have a code snippet or example?
Is there some other straight-forward way to achieve my goal of accessibility-annotating my custom elements without adding a bunch of explicit attributes to the element instances?

So you want the benefit of adding a role or an aria-attribute without actually adding those attributes? The concept of an "accessibility object model" (AOM) has been bantering around a bit that would let you access and modify the accessibility tree directly but it's still in the works. Here's an article from a couple years ago that talks about it. Nothing official. Just one person's thoughts.

Further research shows that, as of this time, the abstracted accessibility options I'm asking for are not yet implemented.
For the time being: eliminating a number of page-owned enclosing divs from the accessibility hierarchy via role="presentation" significantly improved my overall tree. With those out of the way, the majority of my custom tags seem to be simply semantically ignored. This is mostly fine as the majority of my content is plain text.
Since I already mark up the vast majority of even single-character symbols, I've simply added all my symbols to the markup generator. Since everything is already in custom tags, I then use a shadow DOM span with role="img" and a character-specific aria-label to present the symbolic character.
My solution is still incomplete. I wish that I could convey the full richness of the semantic content I have available.

Creating a React component and states from regex string matches

I'm moving the frontend of a project I'd written primarily with php and jquery over to react as a learning exercise. So far it just seems to be making everything a bit more convoluted but I'm hoping I'll see the benefits soon. Right now it's just good to be familiarising myself with something far more current than what I was using. But I digress...
There's one particular situation I'm translating at the moment that feels simple to do in jquery but feels like I'm having to choose between a bunch of cumbersome or not-recommended solutions with react, such as dangerouslysetinnerhtml
A user can input their own list, as raw text, which the app parses using regex to highlight quantities and items, illustrated above as highlighted by rose and green. New lines are parsed as list elements, highlighted in peach. The parsed quantities can be scaled using a range input, shown above the list. The example UI shows an example after parsing a raw string of text.
Previously - jquery
When I was using jquery I replaced regex matches with <span> elements containing data attributes to create new elements I could target. The quantity scaling function in the app could now reference the original quantity of a list item at any time using a data attribute.
e.g. I need 10 cars is parsed to become <li>I need <span class='highlight quantity' data-count='10'>10</span> <span class='highlight item' data-item='cars'>cars</span></li>
With jquery, whenever the input range is dragged, I just target all elements on the page that have the quantity class, reference their data-count and replace the span value with the current range value multiplied by that data-count. This worked well.
Now - React
I'm getting my head around react components. Referencing the illustration, above, I have a component for dialogs (as there will be other dialogs too), a component for the range input of dialogs, a component for the list, and a component for list items.
As before, when I drag the input range, I want the quantities on the page to change to reflect the new scale set by a user. When picking up react I became familiar with component states and props and thought these would solve my problem beautifully. If I had a scale state I could pass it through the components and anything in the hierarchy could reference it. As I understood it, the hierarchy was important and it was important to hold the state at the top so that lower-level components could be 'fed' it, if that's a good way of putting it. This appeared to be best practice.
But I'm having a tough time getting it to jump through hoops. Maybe I've misunderstood?
Two uncertainties
1
If I've understood the hierarchy of states correctly, I'm setting the state initially inside the App() function, like so:
function App() {
const [scaleValue, setScaleValue] = useState();
const handleScaleChange = (childData) => {
setScaleValue(childData);
}
return (
<React.Fragment>
/~~~/
<UserList rawstring={userText} scale={scaleValue} />
/~~~/
<Dialogs scale={scaleValue} setscale={setScaleValue} parentCallback={handleScaleChange}/>
</React.Fragment>
);
}
But I must use a callback function to change the state as the dialog range I'm using is nested a few levels down in components - passing it back to the state in the parent App() function and then back down to all the nested components using props. It felt much easier to just target elements with a matching class name in jquery and I'm not yet seeing the advantage of having components using states. I thought being able to target list item components with quantity states would have advantages in React that were more apparent.
2
Perhaps the main thing that has me stumped on how to proceed. In jquery, when I was parsing a raw string provided by the user, into html that I could target with specific classes, I just used some regex functions to return html that could be inserted into the dom. Example of the html is per the earlier example:
I need 10 cars is parsed to become <li>I need <span class='highlight quantity' data-count='10'>10</span> <span class='highlight item' data-item='cars'>cars</span></li>
With react, I parse the same string to the same html fine, but I've no idea how to handle it from there, logically? If I return it and insert it, it is treated as raw text and renders the html tags on page. I can use dangerouslysetinnerhtml and it shows fine, but I've loud and clear got the message that this is heavily discouraged. I want to use best practices.
In fact, I likely don't want to be returning quantities wrapped in targetable classes. I gather that I want to be able to assign states to a list item component. I have absolutely no idea what's the best approach (logically) to do this with react. I could return an array from the regex parsing but this feels needlessly complex compared to replacing a matched sub string with a wrapped version of itself.
I feel at this point I may be approaching the translation wrong (or my understanding of using the component system and states) as this feels so much more convoluted than the jquery approach I was using.

not-recommended solutions with react, such as dangerouslysetinnerhtml
That's not what dangerouslySetInnerHTML is about. It's about the fact that an unsanitized string containing HTML/JS could pose a security threat (XSS mostly). In other words, in React you have to explicitly use dangerouslySetInnerHTML if you need to be unsafe, but it's your responsibility to protect yourself, hence the name. Doesn't mean it's not recommended, just tells you to be extra careful.
the hierarchy was important and it was important to hold the state at
the top so that lower-level components could be 'fed' it, if that's a
good way of putting it.
Indeed, at first it might look overly complicated for no good reason. Today.
A few months from now if you'll have to go through that code again, or worse, if someone else will have to, this structure enforcing rules and structure makes it far more easier to make changes. You can simply right click on the "callback" function and "go to definition" in your IDE of choice and see where it's defined, instead of hoping it's not hidden in some JS file or relying on search when the function is named "handleClick" in a project with one thousand click handlers.
It felt much easier to just target elements with a matching class name in jquery
Of course it was, anyone with minimal experience in a large scale project knows this. There always are trade-offs. You need to judge if it makes sense to loose simplicity for the sake of maintainability based on the scale of the project.
With react, I parse the same string to the same html fine, but I've no idea how to handle it from there, logically?
You are mixing parsing with generating HTML. You were parsing the string and generating HTML. With React you have to parse the string and generate JSX. If your code was separating these two concerns then it would have been a lot clearer what the difference is.
Again, if you are the only one working on this and know you'll never touch it again, why bother separating these two parts? On the other hand if you do expect to touch that piece of code later or know there will be other people working on it then the smart option would be to split it so that you can properly unit test it.
I can use dangerouslysetinnerhtml and it shows fine, but I've loud and clear got the message that this is heavily discouraged. I want to use best practices.
No, that's not the point. If you use dangerouslySetInnerHTML then how are you going to interact with those highlighted numbers? document.querySelectorAll? That is indeed against the purpose of React. I think my answers above make it clear what a more suitable approach would be.
feels needlessly complex compared to replacing a matched sub string with a wrapped version of itself.
And it is because you don't understand why someone would use React. You think React is a replacement for jQuery and it is not. It is a way of writing JavaScript applications so that you don't end up with a pile of code no one can follow. Which is a problem that only exists if you eventually end up with a lot of code. Your application has what, 100 lines of JavaScript? Why are you even using jQuery?
My advice is:
Understand that people that make deliberate choices don't choose React because it's fashionable but because it solves some problems.
Figure out what those problems are. I've listed a big one, which is maintainability and that structure and separation of concerns are part of the solution.
Decide if you care about any of the problems React helps solve in any of the projects you work (or will work) on.
If you do care about these problems then put yourself in the appropriate state of mind. React is not meant to replace jQuery, find a project that would benefit from using React.
Understand that React is not the one and only tool you'll ever need.
Some more advice:
definitely have a look at useReducer and then (if you need the extra juicy bits) Redux for state management. All beginner tutorials try to stay away from them but IMO React is an empty shell without proper state management.
React is a clean way of turning state into HTML/CSS. The philosophy is that any action (e.g. a click on a "add a new row" button) should produce a state mutation that translates into manipulating some data, NOT the DOM. Your JSX is one representation of that data! And React does a great (fast) job at manipulating the DOM to reflect the new JSX representation. So start thinking your application in terms of input-output, data structures and actions/reducers that manipulate these data structures.
force yourself to unit test (or, better, TDD) your actions and reducers. Completely decoupled from JSX and DOM! Your entire application logic should be testable in Node CLI. This to me (that you can completely separate logic from presentation) is a major win (and is the base of React Native too).
have a look at https://storybook.js.org/ - this one is another one of those fantastic tools (not only for React) that will completely change your development experience.
be wary of using "component libraries". Most "complicated" things turn into child's play when you really understand how to design them correctly.
be prepared to spend 3x the time you spent before on implementing really simple stuff for a while, until you familiarize yourself with the core concepts and philosophy.

Limitation of Angular and browsers with regards to data loaded

I would like to know what is the maximum data that angular framework can handle. Say, I am displaying a chart using angular and some charting framework like chartjs. I'd like to know up to how many data can the browser display properly, with slowness, or up to when it crashes.

Your question has no simple answer, but I will try to flatten it and give a simple answer, or at least simple things to consider...
Angular (at runtime), like many other frameworks is simply JavaScript,
So let us reduce the question to "Limitation of JavaScript and browsers with regards to data loaded",
JavaScript has no upper limit of memory or storage it can handle,
I've seen JavaScript applications that require more than 15GB of RAM,
and they performed well too.
So assume data size itself is not an issue (unless your application is poorly implemented, leaking memory or just not very efficient, of course).
The main challenge as I see it, is displaying and manipulating the information
without causing unnecessary delay or unresponsiveness.
Displaying the information - let's say you have a list (or a table) containing 1,000,000 possible gifts which you then want to display for the user to select.
Adding the list items to the document one by one is tempting, but will require the browser to repaint after each addition (causing a delay or full unresponsiveness until finished), another way is adding the elements to some DOM element (denoted by N) still being kept in memory, then adding all elements corresponding the list items to the element N (still, just an in memory operation), finally adding N to the document containing the entire list - the will be a much better solution for displaying the large amount of data.
Manipulating the information - displaying is indeed not enough. you would like to move, drag, sort and filter the data being displayed. And as mentioned before, it is a bad idea removing many elements directly from DOM. You should instead remove container from the document's DOM to memory, manipulate the data in it, and then add the container right back to the document. Angular does this kind of magic for you.
(Toggling the 'display:none\block' css attribute of many elements has a similar blocking effect as I recall).
A good practice is implementing an application/page showing only the amount of data that can be processed by a human at a single glance. The rest of it should be considered in the application data-layer, in memory, and should be loaded to display given the appropriate need or request.
To conclude, you can deal with huge amounts of data as long as you provide a mechanism that efficiently filter the displayed information.
I hope it helps...
for further reading:
Slow and fast ways of adding elements to DOM
A question emphasizes the lack of memory limit used by JS
CSS display attribute performance
A good discussion about the reasons for slow DOM
About using HTML5 correctly - old but still true
Once the DOM creation procedure is understood - it much easier to display data without affecting performance / user experience

Using custom HTML attributes for JavaScript purposes? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
One of my colleagues doesn't like to use HTML classes and ids for javascript/jQuery purpose. Therefore I've seen that he had created custom html attribute such as
<div id="myid" class="cssClasses ..." [some-purpose-id]="mycustomId">...</div>
I asked him if it was really a good idea and he replied that he considers that classes and Ids should be reserved for styling.
Personally, I would have used classes. Some classes would have been used for styling, and some other classes would have been used for programming (jQuery selectors). The idea is to keep things appart also. And of course jQuery could set styling classes but if possible not use them for selection. Of course I also use id when appropriate but since an id is unique on a page, I like to do generic stuff using classes.
I would like to know you opinions on the better approach (if there is one). Thank you.

I would recommend against that, but rather use the HTML5 data attribute. Read about it here.

You should use classes and IDs for both javascript and CSS. Your colleague is wrong.
It helps to remember that you want to keep separation between data (html), presentation (css styling), and behavior (javascript). Classes and IDs are there for a reason, and it's not for styling (and not for javascript, either!). They are there to allow you to semantically mark up your document.
When writing proper HTML, your classes and IDs should reflect the actual content and purpose of the data those nodes contain. Then, when you need to style them, you can specify the proper styling for, say the .zip_code class. And when you need special behavior, you can use javascript to apply the required behavior to your .deletable list items.
Of course, sometimes in the real world, there are people who feel very strongly against things, and you end up having to go along with what they want. In this case, you may have to compromise with your colleague and use HTML5's data-* attributes, as someone else suggested. However, while this is a solution that will result in valid HTML, it is not what data-* attributes were designed for, so it isn't technically the correct solution.

No, for the most part, you shouldn't use made-up attributes. These are not valid HTML, and will show up as errors if you try to validate your code.
However in HTML5 you can use custom attributes that begin with data-. You can read more about them on the html5doctor website. data- attributes, however, are more often advocated for storing information, not labels. e.g. for a product you might say data-brand="gucci". Or something.
Using attributes for styling or javascript is misleading, because it's not a dichotomy. Often, you will need to add an id or class to style your markup. If possible, you can reuse these attributes to select elements in javascript without adding attributes. It keeps your code clean, but this is obviously just a preference.
If you need to select an element that doesn't already have a unique id or class that could be used for selection, you could use the data attribute. However using ids and classes is still standard practice.

It may be useful if you want to supply another data inside attributes for multiple purposes, look at the following example:
<script>
function getIt(){
ob = document.getElementById('myid')
alert(ob.getAttributeNode('anotherTitle').value)
}
</script>
<div id="myid" class="cssClasses" anotherTitle="new Title">The div</div>
Test
I used the anotherTitle attribute to save or store another set of data in-which it may be used in another usage.
This is a working example http://jsfiddle.net/vxknh/
The refrence: http://www.w3schools.com/jsref/prop_attr_value.asp

Maybe your colleague was inspired by examples of the declarative style used in the dojo toolkit.
data- is a good way to "stamp" your attributes for a specific purpose (for example all dojo specific attributes are of the form data-dojo-whatever). This reduces the risk of conflict with other scripts or css.
So yes it can be a good idea, but no classes and ids are not reserved for styling.

Classes and ID have their advantages and disadvantages (it is also very dependant on how you use them).
Classes - they were designed to convey styling info, but with advent of jQuery now they have dual purpose, sometimes signifying action (or data) and sometimes styling. This is quite confusing especially on large scale projects. Imagine a designer trying to "re-factor" the CSS and changing these class names on elements that have javascript/jQuery hooked up to it - bad things ensue. You could mitigate against that prefixing your non-styling class names with a "namespace" like ".donttouchthis-jqueryclass", which differentiates the logic from presentation is a very primitive manner. The advantage of using a class is that it can be "stacked" thus it can convey more complex "logic" than an ID for example. From a pure javascript (no jQuery) point of view reliance on classes in your code introduces some backward compatibility issues (getElementsByClassName).
IDs - as the name implies were designed to uniquely identify elements. From a programming point of view IDs offer the fastest way to access elements in the DOM (getElementById). They can also be styled, however the potential to remove an ID when re-factoring is lower (as you can always remove ID styling and give it a class). IDs are unique thus making extensive use of them to store any complex data is very, very limited. But they are great to bind click handlers or uniquely identify chunks of the DOM as long as you only need one of elements identified by them.
Both Classes and IDs have one important rule which limits their usefulness as means for conveying any business logic - they cannot start with a number (and often you just need that database id conveniently assigned to an element)
Custom attributes (not data-) - they can be anything which is kinda cool, however if you care about "validity" of your HTML you will be restricted to certain DTD types and you will need to specify them in your DTD. In reality they do not harm your HTML as browsers are specifically obliged (under the W3C convention) to ignore unknown attributes. The only problem being a clash with potential future developments. If you are using pure javascript they will also be a bit of a pain to use (but this is a subjective option - as suggested in one of the answers above this is a good supplementary option to use them as data storage). The potential for a mixup when re-factoring CSS is small.
Custom data- attributes - these are great if you are using jQuery because you can store really complex data against it (objects etc), when used correctly they really kick ass and give you a nice vocabulary to describe function of your DOM elements. "data-" namespace nicely protects it from any future conflicts with non name-spaced attributes.
Having said all that the best solution is not to overtly rely on any of these, but implement javascript patterns which allow you to pass a parent DOM element and later apply functionality to it based on markup (like a standard jQuery plugin pattern). This way you remove a lot of dependencies on specific attributes/classes/ids that are a part of static HTML source.