Develop Reference

PowerBi: Query HTML table

What I need
I need to retrieve data from this source . Let's assume I must use only PowerBi for this.
What I did so far
If I use the basic web source option, then the query is just basically an htlm parsing with which I can easily get the data found in the html scope of the page, example:
Source:
The steps I'm following through Web source option:
Query:
(to simplify the example, assume we don't need the dates)
You can download that example .pbix file here.
The problem
The problem is that I need more data, which can't be accessed through the html preview. For example, let's imagine I need to retrieve the data from January 2010 to April 2020. Those king of queries can only be done via this button located in the webpage (which exports the requested data to an Excel workbook):
The idea is to get this process automated, so going to the source and export the excel file all the time is not an option.
Inspecting the element I realized that what it does is execute a javascript function:
The question
As a PowerBi/PowerQuery noob I wonder: Is there any way I can get that data directly with PowerBi (maybe calling the js function somehow)? If there is so, then how?
Thank you in advance.

The solution to my case was to use URL parameters to retrieve de data without parsing the html table.
❌Original URL I was using:
https://gee.bccr.fi.cr/indicadoreseconomicos/Cuadros/frmVerCatCuadro.aspx?idioma=1&CodCuadro=%20400
✔️New URL for the query, adding some parameters:
https://gee.bccr.fi.cr/indicadoreseconomicos/Cuadros/frmVerCatCuadro.aspx?idioma=1&CodCuadro=%20400&Idioma=1&FecInicial=2010/01/01&FecFinal=2040/01/01&Filtro=0&Exportar=True
This procedure only works in this case, because obviously the parameters will not be the same on other web pages.
However, I post this answer to keep the main idea for those who are in a similar situation: first try with the appropriate url parameters to get the data in a different format. Of course you first must know which are the available parameters, which is a limitation.

Extract html sourcecode from a javascript generated output

I am currently working on a project of finding empty classrooms in our school in real time. For that purpose, I need to extract substitution published on our school page (https://ssnovohradska.edupage.org/substitution/?), since there might be any additional changes.
But when I try to extract the html source code and parse it with bs4, it cannot find the divs(class: "section print-nobreak") that contain the substitution text. When I took a look at the page source code(Ctrl+U) I found that there is only a javascript that prints it all directly.
Is there any way to extract the html after the javascript output has been already rendered?
Thanks for help!

Parsing HTML is unfortunately necessary to solve your problem. But I will explain how to find ways to avoid that in your future projects (not based on this website).
You've correctly noticed that the text is created by JavaScript code running on the page. This could also indicate that the data is either loaded from another resource (XHR/fetch call getting a response from an API) or is stored as a JSON/JS inside of the website's code. (Or is generated from an algorithm, but this is unlikely to be the case in such websites.)
The website actually uses both methods (initial render gets data stored inside of the website's code, but when you switch dates on the calendar it makes AJAX requests). You can see this by searching for ReactDOM.render(React.createElement( in the code. They're providing a HTML string to the createElement call, so I would suggest looking into the AJAX way of doing things.
Now, to check where the resource is located, all you need to do is opening Developer Tools in your favorite browser (usually Control+Shift+I) and navigating to the Network tab. Now that your network tab is open, you need to cause the website to load external data, for example, by pressing a date on the "calendar bar".
Here you will notice many external requests, but we're actually looking only for XHR calls. Click on the XHR button next to the "Filter" text field. That should result in only one request being shown:
Unfortunately for us, the response only contains HTML. Also, API calls are protected - they require a PHP session ID and some sort of a token (__gsh) to not fail. So, going back to step 1 - seems like our only solution is to use regular expressions to find the text between "report_html":"<div class and </div></div></div> from the source code, if you're interested in today's date only. If you want to get contents for tomorrow or any other date - you will need to either fetch the page, save the cookies and find the token to supply to the request and then make that request, or use something like puppeteer or pyppeteer (since you've mentioned BS4) and load the webpage in that. If you aren't doing the data fetching that often, you should be fine overall.

Alternative to passing Data to JavaScript from PHP?

I have a fairly large Application and I'm currently trying to find a way around having to pass Data from PHP (User Tokens for 3rd Party API's and such) through the DOM. Currently I use data-* attributes on a single element and parse the Data from that, but it's pretty messy.
I've considered just making the contents of the element encoded JSON with all the config in, which would greatly improve the structure and effectiveness, but at the same time storing sensitive information in the DOM isn't ideal or secure whatsoever.
Getting the data via AJAX is also not so feasible, as the Application requires this information all the time, on any page - so running an AJAX request on every page load before allowing user input or control will be a pain for users and add load to my server.
Something I've considered is having an initial request for information, storing it in the Cache/localStorage along with a checksum of the data, and include the checksum for the up-to-date data in the DOM. So on every page load it'll compare the checksums and if they are different (JavaScript has out-of-date data stored in Cache/localStorage), it'll send another request.
I'd rather not have to go down this route, and I'd like to know if there are any better methods that you can think of. I can't find any alternative methods in other questions/Google, so any help is appreciated.

You could also create a php file and put the header as type javascript. Request this file as a normal javascript file. <script src="config.js.php"></script> (considering the filename is config.js.php) You can structure your javascript code and simply assign values dynamically.
For security, especially if login is required, this file can only be returned once the user is logged in or something. Otherwise you simply return a blank file.

You could also just emit the json you need in your template and assign it to a javascript global.
This would be especially easy if you were using a templating system that supports inheritance like twig. You could then do something like this in the base template for your application:
<script>
MyApp = {};
MyApp.cfg = {{cfg | tojson | safe}};
</script>
where cfg is a php dictionary in the templating context. Those filters aren't twig specific, but there to give you an idea.
It wouldn't be safe if you were storing sensitive information, but it would be easier than storing the info in local storage,

Is fetching remote data server-side and processing it on server faster than passing data to client to handle?

I am developing a web app which functions in a similar way to a search engine (except it's very specific and on a much smaller scale). When the user gives a query, I parse that query, and depending on what it is, proceed to carry out one of the following:
Grab data from an XML file located on another domain (ie: from www.example.com/rss/) which is essentially an RSS feed
Grab the HTML from an external web page, and proceed to parse it to locate text found in a certain div on that page
All the data is plain text, save for a couple of specific queries which will return images. This data will be displayed without requiring a page refresh/redirect.
I understand that there is the same domain policy which prevents me from using Javascript/Ajax to grab this data. An option is to use PHP to do this, but my main concern is the server load.
So my concerns are:
Are there any workarounds to obtain this data client-side instead of server-side?
If there are none, is the optimum solution in my case to: obtain the data via my server, pass it on to the client for parsing (with Javascript/Ajax) and then proceed to display it in the appropriate form?
If the above is my solution, all my server is doing with PHP is obtaining the data from the external domains. In the worst (best?) case scenario, let's say a thousand or so requests are being executed in a minute, is it efficient for my web server to be handling all those requests?
Once I have a clear idea of the flow of events it's much easier to begin.
Thanks.

I just finish a project to do the same request like your req.
My suggestion is:
use to files, [1] for frontend, make ajax call to sen back url; [2] receive ajax call, and get file content from url, then parse xml/html
in that way, it can avoid your php dead in some situation
for php, please look into [DomDocument] class, for parse xml/html, you also need [DOMXPath]
Please read: http://www.php.net/manual/en/class.domdocument.php
No matter what you do, I suggest you always archive the data in you local server.
So, the process become - search your local first, if not exist, then grab from remote also archive for - 24 hrs.
BTW, for your client-side parse idea, I suggest you do so. jQuery can handle both html and xml, for HTML you just need to filter all the js code before parse it.
So the idea become :
ajax call local service
local php grab xm/html (but no parsing)
archive to local
send filter html/xml to frontend, let jQuery to parse it.

HTML is similar to XML. I would suggest grabbing the page as HTML and traversing through it with an XML reader as XML.

Saving dynamically created content to server & load from there

So I have this webpage that I'm making which allows people to create elements on the page on the fly. And I want to be able to save those elements to my server and whenever someone else reloads that page, the webpage will have those saved elements.
I'm not a good web programmer by any means, so take it easy with the web jargon xD
The user created elements are nested 's or lists. Those elements can be deleted at anytime as well.
So I was reading about saving them as JSON but how would I go about doing that as my 's, most of the top level ones will have the same class. Never worked with JSON before, so I'm a real noob at that.
Will the server file keep replacing itself with a brand new copy with each addition/deletion?
And I'd like to get a little help with showing the new elements without updating. On other users page. I read about AJAX real-time updating, like APE, but have no idea how to go about with that. (This is not really needed but would be a nice one to have)
If someone can guide me a little at least, that will be great. Thanks.

The best suitable way to accomplish this is by saving your objects attributes to a database, however other options include XML files etc..
The process of accomplishing it through database is:
If you want to save data to database then you will have to use a server side language like Php or Asp.net, so first step will be to have a database then an active connection to your database on your intermediate file (lets say 'data.php')
Then you need to code your data.php file so that it can take input(usually through GET or POST method) and it can save it to your database
Then you need to pass your data (objects attributes) through AJAX to data.php and save them to your database
On the main file you will have to check whether already some data exists for user, if yes then fetch it from database and display objects accordingly, otherwise set the objects preferences to default