What I need
I need to retrieve data from this source . Let's assume I must use only PowerBi for this.
What I did so far
If I use the basic web source option, then the query is just basically an htlm parsing with which I can easily get the data found in the html scope of the page, example:
Source:
The steps I'm following through Web source option:
Query:
(to simplify the example, assume we don't need the dates)
You can download that example .pbix file here.
The problem
The problem is that I need more data, which can't be accessed through the html preview. For example, let's imagine I need to retrieve the data from January 2010 to April 2020. Those king of queries can only be done via this button located in the webpage (which exports the requested data to an Excel workbook):
The idea is to get this process automated, so going to the source and export the excel file all the time is not an option.
Inspecting the element I realized that what it does is execute a javascript function:
The question
As a PowerBi/PowerQuery noob I wonder: Is there any way I can get that data directly with PowerBi (maybe calling the js function somehow)? If there is so, then how?
Thank you in advance.
The solution to my case was to use URL parameters to retrieve de data without parsing the html table.
❌Original URL I was using:
https://gee.bccr.fi.cr/indicadoreseconomicos/Cuadros/frmVerCatCuadro.aspx?idioma=1&CodCuadro=%20400
✔️New URL for the query, adding some parameters:
https://gee.bccr.fi.cr/indicadoreseconomicos/Cuadros/frmVerCatCuadro.aspx?idioma=1&CodCuadro=%20400&Idioma=1&FecInicial=2010/01/01&FecFinal=2040/01/01&Filtro=0&Exportar=True
This procedure only works in this case, because obviously the parameters will not be the same on other web pages.
However, I post this answer to keep the main idea for those who are in a similar situation: first try with the appropriate url parameters to get the data in a different format. Of course you first must know which are the available parameters, which is a limitation.
Related
I have create a BaseX database and a mytest.xq file containing an XQuery for that database. When I write in my browser the following:
localhost:8984/rest?run=mytest.xq
I get the desired results in an xml form. However I want to perform this using the html language to display those results in a website. Is this possible? If it is, can the results from xml be visualized better for example a table?
I have looked all the documentation regarding baseX http and have not found a way
You can add &method=html to your url like so:
localhost:8984/rest?run=mytest.xq&method=html
As long as you are returning html from your query then it will render with the method argument. You don't need RESTXQ for your simple needs.
The main BaseX page has an example file in the webapp folder called restxq.xqm where you can see how the basic home page is set up. You don't need RESTXQ but you can use the header information from that file in your test query and render your page with that in mind.
Also there is an entire app in the webapp/dba folder that is written entirely in RESTXQ.
I am currently working on a project of finding empty classrooms in our school in real time. For that purpose, I need to extract substitution published on our school page (https://ssnovohradska.edupage.org/substitution/?), since there might be any additional changes.
But when I try to extract the html source code and parse it with bs4, it cannot find the divs(class: "section print-nobreak") that contain the substitution text. When I took a look at the page source code(Ctrl+U) I found that there is only a javascript that prints it all directly.
Is there any way to extract the html after the javascript output has been already rendered?
Thanks for help!
Parsing HTML is unfortunately necessary to solve your problem. But I will explain how to find ways to avoid that in your future projects (not based on this website).
You've correctly noticed that the text is created by JavaScript code running on the page. This could also indicate that the data is either loaded from another resource (XHR/fetch call getting a response from an API) or is stored as a JSON/JS inside of the website's code. (Or is generated from an algorithm, but this is unlikely to be the case in such websites.)
The website actually uses both methods (initial render gets data stored inside of the website's code, but when you switch dates on the calendar it makes AJAX requests). You can see this by searching for ReactDOM.render(React.createElement( in the code. They're providing a HTML string to the createElement call, so I would suggest looking into the AJAX way of doing things.
Now, to check where the resource is located, all you need to do is opening Developer Tools in your favorite browser (usually Control+Shift+I) and navigating to the Network tab. Now that your network tab is open, you need to cause the website to load external data, for example, by pressing a date on the "calendar bar".
Here you will notice many external requests, but we're actually looking only for XHR calls. Click on the XHR button next to the "Filter" text field. That should result in only one request being shown:
Unfortunately for us, the response only contains HTML. Also, API calls are protected - they require a PHP session ID and some sort of a token (__gsh) to not fail. So, going back to step 1 - seems like our only solution is to use regular expressions to find the text between "report_html":"<div class and </div></div></div> from the source code, if you're interested in today's date only. If you want to get contents for tomorrow or any other date - you will need to either fetch the page, save the cookies and find the token to supply to the request and then make that request, or use something like puppeteer or pyppeteer (since you've mentioned BS4) and load the webpage in that. If you aren't doing the data fetching that often, you should be fine overall.
Is it at all possible to post HTML to an HTML file using jquery's post method.
for example:
$.post("database.html", "<div>Content</div>")
once a button is clicked I want to post to the database file and have this current file load in content from the database file using.
$.load("database.html")
Does anyone know how I could implement this method properly?
If I rewrite your question a bit: you want to pass data between two html pages. If you only use static html you cannot do that with javascript only. You have to send the data to a server and then use this data in your second html file.
But I think you quite misunderstood what http can or cannot do, I would recommend reading mozilla explanation
It is possible to pass HTML content in POST from one HTML page to another. But in order to use or process received the data at the second HTML page, you'll need a programming language.
Read the accepted answer to question How to read the post request parameters using javascript.
I have a JSON file which is dynamically and contain match info including an unique id. The JSON is divided into 3 arrays live, upcoming and recent. Since i'm quite new to Javascript i'm wondering what would be the best way to go in order to make this livescore script. I need it to be updating without refreshing browser? What is my options? Maybe someone has a snippet?
The JSON is automatically updates through another script which is connected to a cron job, so the script does not need to do anything regarding the JSON. Only retrieve and show the data.
I'm using dreamhost, which gives me access to shell, so websockets and so on is an option.
You'll need a jquery user to give you a snippit for this one, but in vanilla ecmascript 6, you use an XMLHttpRequest object to get the JSON from your server. This object can request data from the server asynchronously and is triggered by the client/browser so you can update the live match info when and as often as you like. You would just have to write a function to replace the data on the webpage with the new info when it is updated.
Just a simple question, I was wondering why some websites have something like "?lang=EN" in their URL after selecting a language? Is it because their html file or folder containing it is named "?lang=EN", or some other code that does this? I'd like to set the URL like that for my website (has 2 languages). Currently I have folder structure like this:
Language selection: D:/media/index.html
EN site: D:/media/en/index.html
CN site: D:/media/cn/index.html
Files for the website: D:/media/site
Thanks.
First of all, anything after the file extension ( .html ) is a server side function.
The ? is a function for PHP and adds variables to the super global GET array ( in the form: ?variable=value&variable2=value2 ) that is directed to from another page and from that point many things can be done with the data.
Sites that use the ?lang=EN are probably programmed to print out the chunks of text needed on the single page in the places and languages required. Though it is possible using this method to redirect to a language specific directory.
Hope this helps :)
That's because they often have a content management system where the content isn't stored in files necessarily, but in a database. The lang=en is a GET variable from the URL that they retrieve in, for example, PHP, to display the correct content. In your case, however, you can just redirect the user if they click EN or CN to the appropriate locations, in your case, /en/index.html and /cn/index.html.
The url you see at the address bar, whatever comes after "?" is called "QueryString" and with libraries on the server side (based on the developing platform that website is made on) you can access the values. For instance the value of "lang" can be equal to "EN" or "CN" etc.
By the way you can have some http handlers to rewrite the requested url and get your parameters through the url that physically doesn't exists. Like the one you mentioned, "http://yoursite.com/en/default.whatever". I myself prefer this way but as you requested you should use some server side libraries to access the query string values and choose the language of the content you wanna send to client.
Also as one solution that once I used, you can also use some translation service (like translate.google.com) client libraries and call it at client side with jquery or even javascript and translate all the texts on page load. Although it's damn fast in action, it has some issues you will see.
Hope it helps.
PHP uses $_GET to get value from variables from the URL.It gets the value from that LANG variable and then it selects all from a file where are stored all the words in different languages or from the database
You don't need to copy every file and then translate it.
Search for php dynamic pages tutorial in your case. I found THIS.
P.S. PHP is one from many ways to do this.