Weird button behavior while scrapping bet365 - javascript

I am trying to scrap odd data by Pyppeteer from bet365 follow their UI flow and need for help.
The pages are just like this.
https://www.bet365.com/#/AC/B1/C1/D8/E123540970/F3/
If the match is outdated, just get into one of the matches that not in-played from following.
https://www.bet365.com/#/AS/B1/
For the collapsed button named "Handicap Result" at the bottom of the page, I can normally use my mouse cursor to click on it and make it expand and retrieve its data.
But when I tried to use Pyppeteer's click or javascript to do it, that is querySelector() and click(), it just weirdly failed.
Observing html and xhr after click(), it definitely triggered something and tried to do something, but the content just failed to shown.
I tried to click on different coordinates, key Enter and Space, click with delay, and all failed.
Could someone tell me how could I expand it use javascript?
Was I dissipating wrong events?
Thanks.

This is probably one way they use to avoid scraping.
U tested with pure JS direct from DevTools?
Bet365 its really, really hard to scrape. They use Cloudflare blocking and protection system (including AI to detect autonomous agent like Puppeteer, even if you use stealth mode).

Related

What is the fastest way to find a listener for debugging javascript in a browser?

Background:
I often find myself in the position of debugging a piece of Java script on a web page in an unfamiliar codebase, and often one that has seen many developers and coding approaches. Sometimes I do not even what technologies might be in use, eg. angular etc.
The first time I need to address the Java script is when a specific behaviour is unexpected (ie. it has gone wrong.)
Question:
What tool provides the fastest route to identifying the entry point of the code that is causing the problem?
Example:
I have an html element on a page lets say a button. When that button is clicked I expect to see an http request at the server. There are many ways the element can be associated with its Java script listener. eg JQuery, thrid party plugins such as knockout etc, in house scripts, and so on.
Using developer tools I can start debugging this in the browser but only if I already know the entry point to put a breakpoint on.
Is there a faster method to find the entry point than doing regular expressions searches on the pages code based on intuition and guess work to find what might be attached to that particular element?
For me, the best starting point is in Chrome developer tools. You can:
Choose an element in the elements tab
On the right-hand side of the elements tree, click the "Event Listeners" tab.
Find the event you want to debug (like click)
Click the hyperlink to bring up the code for event listeners, and set breakpoints. Sometimes you have to click the "format code" button (looks like { }) to get the code on multiple lines so that the breakpoint is manageable.
Do the click, and you'll hit your breakpoint, allowing you to step through the code, add watch variables, etc.

How to find which Javascript fires or handles an event?

I am working on a HTML5 application framework, run by a SAP JEE application server, build for companies and their intranet and extranet sites. It is based on the grid framework "Semantic UI" and besides that contains a lot of (also third party) Javascript.
I am currently examining a bug, where clicking a specific icon in a menu, when the page is scrolled down, makes the page somehow scroll back again upwards.
Since there is this great amount of Javascript around, I am currently struggling to find the JS code snippet, which is causing this odd behaviour.
I've read this post here, and got to know event logging in Firebug and inspecting event handlers in Chrome, but that didn't really help me.
I found out that using:
$(<my Elem>).on('click', function(event){
event.preventDefault();
})
I can prevent the scrolling, but I still didn't discover the cause of it.
Has anybody some more advice on how to find the real cause of this?
This behaviour might be caused by several different reasons. One of them that is often overlooked is links like Some JavaScript Handler.
When the JavaScript handler does not properly handle the event (e.g. by calling event.preventDefault(), the HTML link will be followed in addition to the JavaScript handler. Most browsers handle a link to an empty anchor tag # by going to the top of the page. This can easily be avoided when using an empty href attribute like <a href>Some JavaScript Handler</a>.

How do I find what Javascript is running on certain events?

I'll pick Chrome for this example, but I'm open to a solution from any browser.
Use Case:
I have an update button on my website that is used to update item quantities in a shopping cart. I'd like to allow a user to enter a 0 and click update in order to remove the item. Trouble is, there is some listener in some js function that is denying the ability to enter a 0 and click update (after clicking update the old quantity remains).
My question is, what developer tool can I use to find which js function is running during that event? I don't think that Chrome's inspector does this, and I'm not very familiar with Firebug, but I couldn't find the functionality there either.
I feel that I should be able to inspect js firings just like I do css stylings. Is anyone aware of a tool I may use?
I've had to debug some particularly nasty unseen-cause Javascript issues at my job. Knowing the full depth of developer tools like Chrome's is definitely helpful. It undeniably takes some creativity to find places that might be causing the issue, but a few tips:
Tracking down event listeners
Under Chrome's Elements view, try Inspect-ing an element (right-click, Inspect); then, on the right side of the developer view, scroll down to Event Listeners. Here you can view what code files have hooked up an event. Often, this will just point you to a middle-framework from the really devious code you're looking for, but sometimes it will point you in the right direction.
Trapping a DOM modification
Many of the unwanted effects I see are because of something changing some value or attribute on the page that I don't want. Anytime this happens, you can right-click on the element (under the Elements view) and say "Break on..." and the specific scenario you're looking for. When Chrome then hits a breakpoint, you can then look downward in the Stack Trace until you find something recognizable that shouldn't be called.
EDIT after reaching ten votes!
Trapping a JS object modification
If the change you're interested in is code-internal, not in the UI, things get trickier. What's meant by this scenario is that you know somewhere in the code, something incredibly annoying like the following is happening.
company.data.myObject.parameter = undefined;
In this situation, you know myObject is still the same object, but it's being modified, perhaps unintentionally. For that, I often insert the following bit of code, sometimes just through the developer console at some point before said modification happens.
Object.defineProperty(company.data.myObject, 'parameter', {
set: (val) => {
debugger;
}
});
This includes an arrow function - you're only using this for debugging and Chrome supports it, so might as well save keystrokes. What this will do is freeze your debugger as soon as some line of code attempts to modify myObject's "parameter" property. You don't necessarily have to have a global reference to the variable if you can run this line of code from a previous breakpoint that will have the given object in the locals.
Otherwise, if all I'm starting out with is the HTML code, and I want to tie that to Javascript code, I'll often just look for identifying features like "id" elements, and search all JS files in my development directory for it. Normally, I can reach it pretty fast.
Open your page in Firefox with Firebug enabled.
Go to console tab in firebug and click profiling
enter 0 in the textbox and click the button.
Stop profiling.
You will be able to see all the javascript functions which have executed due to your actions. You can view them one by one to figure out which method has caused the mischief.
Go to you code. If you are using jQuery there is going to be a function that will be called with the class or id of that particular update button. Or, if you are using Javascript, there is going to be a function called inside the
<input type="button" name="update" onclick="update()">
These are the two ways to look for the function that is being called; there is no software that I know
Download Firebug for Mozilla Firefox, open it, click on Net and refresh your website. Than, you can see which files are loaded on the page.
If you want to check on errors and what goes wrong with an explanation, than click on console and refresh the page once again. You will see the errors and on which line it goes wrong.
Note: in your console, you can say hold or stop, so that the js file stops loading. And you can edit the script by clicking on script in Firebug. Debugging is simple, as it says on their official page https://getfirebug.com/javascript

Javascript how do I find what function call resulted in a specific UI event?

I want to find out what triggered an event. Namely, the notification bar on this site stackoverflow.com (the bar that tells you when someone has posted an answer to a question you're writing an answer on. It scrolls down slowly from the top and provides a really nice UI for user notifications. I've seen it work on just about ever page.
I imagine it working something (I need to find its name):
special_notification( message );
In the abstract, how do I go about finding out what the call (function name and arguments) looks like that generates that effect when all of the javascript is minified, and I have no idea what include provided it.
Download and install firebug in Firefox.
Go to the URL you're interested in, and open firebug. You might need to reload the page.
Now click on the little arrow icon on the top right hand side of firebug. This will let you highlight any element on the page and provide the corresponding HTML to that element.
Now that you have the id of the element, you should be able to find it in the javascript code. Even if it's minified, the name needs to correspond the DOM name.
To read minified js, you can use a tool like http://jsbeautifier.org.
Regarding your other concern, you want to listen to all the events on a page and know what triggered them and what is the code executed? is that correct?
Update:
There is no way to listen to all the events. If you really need to, you can set up listeners for every event, but you will still miss the custom events, which i guess are what you are after.
I'd suggest you inspect the code using Firebug to learn how the events are used in each case.
You can also listen to all the DOM Events, in jQuery you will do:
$('body').bind('DOMSubtreeModified', function(e){
console.log('DOMSubtreeModified triggered');
console.log(e); //Firebug console.
});
Where e will hold the event information.
Hope that makes sense.

Is there a profitable way to record user actions in textarea?

I need to send bunch of commands to the server on timer - like:
put(0,"hello")
del(4,1)
put(4," is around the corner")
so I need to monitor and record all of the user input and compile/flush it on the timeout (idle), something like macros.
I can record all things happening onKeyUp/onKeyDown/onMouseDown/onMouseUp using textarea cursor position and keys information (and make it cross-browser some time later) but I can't handle things like pasting using mouse right button and selecting 'Paste' or pasting from the menu (I can handle onChange, but I will have no information is it pasted or already recorded as pressed keys and it fires only after focus change). Even pasting from context menu fires some useful info, but the menu from the browser is the only thing, giving nothing for Javascript.
Is there any plugin for jQuery or something like that and do I really have no other ways to implement it without comparing current-document and document-a-second-before?
Upd.: There are events for handling cut/copy/paste: http://www.quirksmode.org/dom/events/cutcopypaste.html , but what about
the undo one?
P.S. I will show a macro-recording code when I'll finish, if someone really needs it. And to finish it properly, I just need the undo handling possibility. Current version is here: http://code.google.com/p/sametimed/source/browse/WebContent/module-editor.js, look for compileCommands method.
There are events for cut/copy/paste you may listen to, depending on browser. So if they are triggered you may use them, otherwise fall back to more tedious work-around.
See: http://www.quirksmode.org/dom/events/cutcopypaste.html

Categories

Resources