I am currently working on a project of finding empty classrooms in our school in real time. For that purpose, I need to extract substitution published on our school page (https://ssnovohradska.edupage.org/substitution/?), since there might be any additional changes.
But when I try to extract the html source code and parse it with bs4, it cannot find the divs(class: "section print-nobreak") that contain the substitution text. When I took a look at the page source code(Ctrl+U) I found that there is only a javascript that prints it all directly.
Is there any way to extract the html after the javascript output has been already rendered?
Thanks for help!
Parsing HTML is unfortunately necessary to solve your problem. But I will explain how to find ways to avoid that in your future projects (not based on this website).
You've correctly noticed that the text is created by JavaScript code running on the page. This could also indicate that the data is either loaded from another resource (XHR/fetch call getting a response from an API) or is stored as a JSON/JS inside of the website's code. (Or is generated from an algorithm, but this is unlikely to be the case in such websites.)
The website actually uses both methods (initial render gets data stored inside of the website's code, but when you switch dates on the calendar it makes AJAX requests). You can see this by searching for ReactDOM.render(React.createElement( in the code. They're providing a HTML string to the createElement call, so I would suggest looking into the AJAX way of doing things.
Now, to check where the resource is located, all you need to do is opening Developer Tools in your favorite browser (usually Control+Shift+I) and navigating to the Network tab. Now that your network tab is open, you need to cause the website to load external data, for example, by pressing a date on the "calendar bar".
Here you will notice many external requests, but we're actually looking only for XHR calls. Click on the XHR button next to the "Filter" text field. That should result in only one request being shown:
Unfortunately for us, the response only contains HTML. Also, API calls are protected - they require a PHP session ID and some sort of a token (__gsh) to not fail. So, going back to step 1 - seems like our only solution is to use regular expressions to find the text between "report_html":"<div class and </div></div></div> from the source code, if you're interested in today's date only. If you want to get contents for tomorrow or any other date - you will need to either fetch the page, save the cookies and find the token to supply to the request and then make that request, or use something like puppeteer or pyppeteer (since you've mentioned BS4) and load the webpage in that. If you aren't doing the data fetching that often, you should be fine overall.
I am redirecting my child window to another url after launching. Actually I am submitting the data to another domain. That domain processing the data and sends to new domain. I need to catch the new domain URL.
I don't think you'll be able to do that. You can't access frames/windows from different origin - that's simply not safe and the browser won't let you.
You could try to work it out differently.
For example, I believe you could have your server side get the other website for you and serve it in your own domain. If you decide to do that, you need to have the reference to your child window first (e.g. var win = window.open(url);) - but still, that needs to be in your domain.
Another way would be to simply post your data using your server side language and then try to read the received page (also using server side). In PHP for example, you could accomplish that with cURL and Simple HTML Dom Parser.
Edit
Just came upon window.postMessage functionality. If the other websites are yours, I think that's the way to go (well, maybe except the limited compatibility in IE 8 & 9: Can I use postMessage).
I faced few problems while using Chrome dev tool. Just want to know whether it's possible and if yes - how. Suggest I have a really massive client side, with hundred of responses per page.
How to find endpoint which handle the response? I mean the first place in js code where the response come in.
How to find the response by it content? For instance, I want to know in which response I've got 45902309509902 value from the table.
How to find endpoint which handle the response?
On the Network tab, you can see where the request was originated, it's the column labelled "Initiator:"
That has a link that will show you the code originating the ajax call (I assume by "response" you're talking about an ajax response). From there, you should be able to find the callback that request is associated with. A lot of times, if you use a library like jQuery, you'll be shown the jQuery code doing the request rather than yours. You can still find what you need, though, by using the un-minified version of the libray, setting a breakpoint on that code (perhaps even a conditional one on, say, the URL being requested), and then when the breakpoint is hit using the call stack to find out where in your code the call actually originates.
How to find the response by it content?
This will be slightly more difficult. Again in the Network tab, you can click each ajax request and see (and search through) the text of there response under the Response sub-tab.
I'm using Jmeter to simulate a flow between two webpages.
The case is complex since the call to the second webpage is done through a form passing a list of input parameters generated by a javascript function called when the user click on the button to pass to the second webpage
If i open firebug console I can see into POST the list of parameters i'm interested in
Due to the fact that i need to parametrize the Jmeter POST with the list of generated parameters is there any solution?
If it's not suitable any solution i'm considering to use regex to extract from the webpage the parameters the javascript function works with. However i really cannot find the parameter value since it is stored into a javascript variable: if i look at the html source code i see something like:
this.cabinNumber[numeroCabina]=$('#cabinnumber'+numeroCabina).val();
but i do not see the real value; (cabinNumberis te variable i'm interested in)
Is there any chance to read directly from HTTP Response Parameters maybe using BeanShell?
Please note that the webpage is developed with Struts1 technology
JMeter is not a browser and it cannot execute JavaScript on client pages, but:
If JavaScript doesn't assume communication with the server (which is unlikely) you should be able to replicate function using any suitable JMeter scripting extension starting with __javaScript function, Beanshell Scripting or even custom implementation of Java Sampler. Or alternatively "interesting" value is already in the response, it just hidden via styles
If it does (which is more common case) you can catch the request via firebug and replicate it with JMeter to retrieve desired cabin number.
You can also try comparing requests captured with firebug to requests captured by JMeter HTTP(S) Test Script Recorder and investigate the differences if any.
I want to implement AJAX like facebook, so my sites can be really fast too. After weeks of research and also knowing about bigPipe (which is not ajax).
so the only thing left was how they are pulling other requests like going to page/profile, I opened up firebug and was just checking things there for what I get if I click on different profiles. But the problem is, firebug doen'tt record any such request and but still page gets loaded with AJAX and changes the HTML also, firebug does show change on html.
So I'm wondering, if they are using iframe to block firebug to see the request or what? Because I want to know how much data they pull on each request. Is it the complete page or only a part of page, because page layout changes as well, depending on the page it is (for example: groups, page, profile, ...).
I would be really grateful if a pro gives some feedback on this, because i cant find it anywhere for weeks.
The reason they use iframe, usually its security. iframes are like new tabs, there is no communication between your page and the iframe facebook page. The iframe has its own cookies and session, so really you need to think about it like another window rather than part of your own page (except for the obvious fact that the output is shown within your page).
That said - the developer mode in chrome does show you the communications to and from the iframe.
When I click on user's profile at facebook, then in Firebug I clearly see how request for data happens, and how div's content changing.
So, what is the question about?
After click on some user profile, Facebook does following GET request:
http://www.facebook.com/ajax/hovercard/user.php?id=100000655044XXX&__a=1
This request's response is a complex JS data, which contain all necessary information to build a new page. There is a array of profile's friends (with names, avatar thumbnails links, etc), array of the profile last entries (again, with thumbnails URLs, annotations, etc.).
There is no magic, no something like code hiding or obfuscation. =)
Looking at face book through google chromes inspector they use ajax to request files the give back javascript which is then used to make any changes to the page.
I don't know why/wether Facebook uses IFRAMEs to asynchroneously load data but I guess there is no special reason behind that. We used IFRAMEs too but now switched to XMLHttpRequest for our projects because it's more flexible. Perhaps the IFRAME method works better on (much) older browsers, but even IE6 supports XMLHttpRequest fine.
Anyway, I'm certain that there is no performance advantage when using IFRAMEs. If you need fast asynchroneous data loading to dynamically update your page, go with XMLHttpRequest since any modern browsers supports it and it's fast as HTTP can be.
If you know about bigPipe then you will be able to understand that,
As you have read about big pipe their response look like this :-
<script type="text/javascript"> bigpipe.onPageArrive({ 'css' : '', '__html' : ' ' }); </script>
So if they ajax then they will not able to use bigpipe, mean if they use ajax and one server they flush buffer, on client there will no effect of that, the ajax oncomplete only will call when complete data received and connection closed, In other words they will not able to use their one of the best page speed technique there,
but what if they use iframe for ajax,, this make point,, they can use their bigpipe in iframe and server will send data like this :-
<script type="text/javascript"> parent.bigpipe.onPageArrive({ 'some' : 'some' });
so server can flush buffer and as soon as buffer will clear, browser will get that, that was not possible in ajax case.
Important :-
They use iframe only when page url change, mean when a new page need to be downloaded that contains the pagelets, for other request like some popup box or notifications etc they simple send ajax request.
All informations are unofficial, Actually i was researching on that, so i found,
( I m not a native english speaker, sorry for spelling and grammer mistakes! )
when you click on different profile, facebook doesn't use ajax for loading the profile
you simple open a new link plain old html... but maybe I misunderstood you