I'm trying to get web table data from the following below website and extract the first table regarding Policy rates on the right hand-side.
https://www.researchonline.se/macro/our_forecasts
Have use the following code just to see if it spits out the desired data but keep getting error 91. Suspecting something about Javascript that I need to consider in my code? Below is my code.
Dim request As Object
Dim response As String
Dim html As New HTMLDocument
Dim website As String
Dim price As Variant
' Website to go to.
website = "https://www.researchonline.se/macro/our_forecasts"
' Create the object that will make the webpage request.
Set request = CreateObject("MSXML2.XMLHTTP")
' Where to go and how to go there - probably don't need to change this.
request.Open "GET", website, False
' Get fresh data.
request.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
' Send the request for the webpage.
request.send
' Get the webpage response data into a variable.
response = StrConv(request.responseBody, vbUnicode)
' Put the webpage into an html object to make data references easier.
html.body.innerHTML = response
' Get the price variable from the specified element on the page and just check in a message box if that is the correct data.
price = html.getElementsByTagName("table")(0).innerText
' Output the price into a message box.
MsgBox price
Set request = CreateObject("MSXML2.XMLHTTP")
The above cannot execute javascript , Though instead of calling url and fetching it from table. you can directly use their API
for example to get policy rate, the site uses API (There are many apis , i have seen by looking into the site). One example is
https://www.researchonline.se/api/MacroAdmin/GetForecast?name=policy&start=Mon%20Sep%2014%202020&end=Tue%20Sep%2014%202021
which will return an xml kindda response, which you can parse with MSXML2.XMLHTTP
Related
I have a problem sending data with an AJAX query to a php file, basically I use post request to send some data just for testing it out, the request is successful when I checked with developer tools in Chrome, as I can see the data that was sent , but the variable $_POST[] in php is always null, and don't understand why, because my data was sent to the php file.
Tried all kind of possibilities found on here, non of them will let me to go further, all of them will leave my $_POST[] empty. I modified the values inside the data attribute I added or removed content-type but nothing worked.
Here you have my jQuery code.
$.ajax({
method: "POST",
url: "2.php",
data: { name: 'JohnDoe', age: '19' }
}).done(function( msg ) {
alert(msg);
});
This is my PHP code.
<?php
$user=$_POST['name'];
var_dump($user);
?>
Errors
$user=$_POST['name'];
//ERROR - Notice: Undefined index: name in G:\xampp\htdocs\weather\2.php on line 6
//Expected result is 'JohnDoe'.
var_dump($user);
//this is Null
//Expected result is to contain some data
This are my two results I get in php.
The html and php files are in the same folder.
You are making two HTTP requests.
The first one using JavaScript, where you make a POST request and alert the response.
You can see the data from the response there.
You make the second request by typing the address into the address bar, where you make a GET request and have the response rendered as a webpage.
$_POST contains the POSTed data from the current request, not the data from any previous request.
The data you POSTed when you made the first request is not available when the PHP program runs again using the second request as input.
If you want to access that data then you need to explicitly do something to make it persist. This could be linked to the browser (so different users would not see each others data) — such as in a session or a cookie — or it could be independent of the browser — such as in a database — so every visitor to the site could see the data.
I am trying to get a specific dynamic figure from a webpage to excel, I managed to gather all the website get response into a "all" variable which I am supposed to parse to extract my numbers, except for when I check the string variable I can see everything but the required dynamic figure! :) "the attached phot shows the dynamic figure at the very instant was 2.19",
any ideas why I am capturing every thing, would be much appreciated, Thanks in advance
My thoughts:
1.I am guessing is the figures are injected by JavaScript or a server side that might be executing after my XMLHTTP request is processed maybe! if this is the case or else I need your expertise
the website doesn't response unless it sees a specific Html request header, so I might need to mimic the headers of Chrome, I don't know how they look like?
Please see below my code and a screenshot for the figure I would like to capture
'Tools>refrences>microsoft xml v3 must be refrenced
Public Function GetWebSource(ByRef URL As String) As String
Dim xml As IXMLHTTPRequest
On Error Resume Next
Set xml = CreateObject("Microsoft.XMLHTTP")
With xml
.Open "GET", URL, False
.send
GetWebSource = .responseText
End With
Set xml = Nothing
End Function
Sub ADAD()
Dim all As Variant
Dim objHTTP As Object
Dim URL As String
Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")
all = GetWebSource("https://www.tradingview.com/symbols/CRYPTOCAP-ADA.D/")
pos = InStr(all, "tv-symbol-price-quote__value js-symbol-last")
testString = Mid(all, pos, 200)
'I am supposed to see the dynamic figure within the TAG but it is not showing!!
Debug.Print testString
End Sub
HTML for Dynamic Required values
#Tim Williams This is a code using selenium (But it seems doesn't do the trick of getting the value)
PhantomJS Selenium VBA
Sub Test()
Dim bot As Selenium.PhantomJSDriver
Set bot = New Selenium.PhantomJSDriver
With bot
.Get "https://www.tradingview.com/symbols/CRYPTOCAP-ADA.D/"
.Wait 2000
Debug.Print .FindElementByXPath("//div[#class='tv-symbol-price-quote__value js-symbol-last']").Attribute("outerHTML")
End With
End Sub
Chrome VBA Selenium
It seems using PhantomJS doesn't work properly, so here's a Chrome version of selenium in VBA
Private bot As Selenium.ChromeDriver
Sub Test()
Set bot = New Selenium.ChromeDriver
With bot
.Start
.Get "https://www.tradingview.com/symbols/CRYPTOCAP-ADA.D/"
Debug.Print .FindElementByXPath("//div[#class='tv-symbol-price-quote__value js-symbol-last']").Text 'Attribute("outerHTML")
.Quit
End With
End Sub
Python Solution
And this is the working python code that my tutor #QHarr provided in comments
from selenium import webdriver
d = webdriver.Chrome("D:/Webdrivers/chromedriver.exe")
d.get('https://www.tradingview.com/symbols/CRYPTOCAP-ADA.D/')
d.find_element_by_css_selector('.tv-symbol-price-quote__value.js-symbol-last').text
I am implementing fullcalendar on my website.
I have created a div with calendar as id.
I have put data from a SQL query in a php variable and used json encode to get the right format.
This is how I create the calendar :
$html .= "<script src='/wp-content/plugins/biobelt/moment.min.js'></script>
<script src='/wp-content/plugins/biobelt/fullcalendar.min.js'></script>
<link rel= 'stylesheet' href='/wp-content/plugins/biobelt/fullcalendar.css' type='text/css'>
<script>
jQuery(document).ready(function() {
var bevents = '".$buildingevents."'
console.log(bevents)
jQuery('#calendar').fullCalendar({
defaultDate: '" . $_SESSION['statDateFrom'] ."',
editable: true,
events: bevents,
});
});
</script>";
The console log gives me an output of the array that I am passing to events, and it is the correct format :
[{"id":"1","titre":"1","start":"2018-04-09 07:00:01","stop":"2018-04-09 11:00:00"},{"id":"2","titre":"1","start":"2018-04-09 07:00:01","stop":"2018-04-09 11:00:00"},{"id":"3","titre":"2","start":"2018-04-09 16:00:01","stop":"2018-04-09 21:00:00"},{"id":"4","titre":"2","start":"2018-04-09 16:00:01","stop":"2018-04-09 21:00:00"}, etc...
What I get from this is :
GET 403 Forbidden Error
I checked in apache logs, this is because the URL is too long since every field of the array is put into the url.
For some reason I don't want to change the limit request line in apache conf file.
I want to generate a POST instead of GET request.
And I would like to know how it generates a GET request since I didn't put GET anywhere in my files.
EDIT :
according to : https://fullcalendar.io/docs/events-json-feed
Fullcalendar create the get request and the URL. The problem persists since the URL is too long and I want to create a POST request instead. How to do that?
You seem to have misunderstood the documentation somewhat.
You said
The console log gives me an output of the array that I am passing to events, and it is the correct format
And indeed what you've showed does look like a Javascript array. So...it's a static array and not a URL. You do not have any kind of server endpoint to which you can make a separate ajax call to get your events. Therefore the article you linked to (https://fullcalendar.io/docs/events-json-feed) is not relevant. Instead you are providing a static list of events as per the method described at https://fullcalendar.io/docs/events-array).
Except that...you're not. Due to the way you've written your code, you're providing a string instead of an array. That is causing fullCalendar to assume you're providing a URL, and then trying to call that URL, and it's no surprise that it errors.
If you simply remove the single quotes from
var bevents = '".$buildingevents."'
so that it becomes
var bevents = ".$buildingevents.";
then your code should work ok, because this will inject a hard-coded array into the JavaScript instead of a string.
I’m new to PHP and coding in general and I can’t figure this out. I’m trying to get the number of kills from this profile page.
At the moment, the string I am trying to get is:
29362
When I view the page source, this number is nowhere to be seen.
When I use inspect element, however, I find:
<td class="num">29362</td>
How can I get the content shown in inspect element instead of the content shown by viewing the page source?
In using a tool like Firebug for Firefox, or the inspector for Safari and Chrome, you can see that at page load a series of AJAX requests are made for data. Though I didn't dig through all of the data returned by those requests, I do see the data you're looking for in at least one of them:
http://uberstrike.com/profile/all_stats/631163
So at page load JavaScript makes a series of AJAX requests back to the server to get all the data, then it manipulates the DOM to insert it all into the view.
If you wanted, your PHP could directly request the URL I pasted above and json_decode the response. This would produce a data structure for you to use which includes that number in the kills_all_time property.
Quick and dirty example:
<?php
$data_url = 'http://uberstrike.com/profile/all_stats/631163';
$serialized_data = file_get_contents($data_url);
$data = json_decode($serialized_data, true);
var_dump($data['kills_all_time']);
I looked and it looks like there is no API currently, so your best method will be to do an inter-web-server http request. Get the page you want and then it is a lot of string math from there.
I would recommend using string search to find <td class="name">Kills</td> and the kills row will appear right after it. From there its simply extracting the number using string math.
To add to what JAAulde has explained, it seems like there is a method to these AJAX requests. And they are all based on the profile ID that can be found at the end of the URL:
http://uberstrike.com/public_profile/631163
Then in the Safari debugger (which is what I am using) you can see these XHR (XMLHttpRequest) requests which are directly connected to API calls:
Then looking at the data in them shows some really nicely formatted JSON. Great! No scraping! So just go through these URLs to see what you can see:
http://uberstrike.com/profile/items
http://uberstrike.com/profile/user_info/631163
http://uberstrike.com/profile/user_loadout/631163
http://uberstrike.com/profile/all_stats/631163
And looking at the all_stats endpoint shows:
"kills_all_time":29362,
Nice!
So now let’s use some PHP json_decodeing like this:
// Set the URL to the data.
$url = 'http://uberstrike.com/profile/all_stats/631163';
// Get the contenst of the URL via file_get_contents.
$all_stats_json = file_get_contents($url);
// Decode the JSON string with the 'true' optionso we get any array.
$all_stats_json_decoded = json_decode($all_stats_json, true);
// Dump the results for testing.
echo '<pre>';
print_r($all_stats_json_decoded);
echo '</pre>';
Which will dump an array like this:
Array
(
[headshots_record] => 24
[nutshots_record] => 33
[damage_dealt_record] => 6710
[damage_received_record] => 31073
[kills_record] => 50
[smackdowns_record] => 45
[headshots_all_time] => 4299
[nutshots_all_time] => 1925
[kills_all_time] => 29362
[deaths_all_time] => 16491
…
Now to get kills_all_time just do this:
// Return the 'kills_all_time'.
echo $all_stats_json_decoded['kills_all_time'];
Which gives us:
29362
I am trying to fetch some data from an e-commerce fashion website. When i open a category on website I can see 48 items and a load more button in the end. When i click on that button, i see the next 48 items. What happens in backhand is, when i click that load more button an XHR post request is sent and response is returned in JSON format. I want to automate this search and capture the response data in an excel sheet. I am new to programming and not familiar with advance scripting languages so i am working on VBA. My request is getting submitted but not get the response. My category page is http://www.myntra.com/_ and the link where request is sent is http://www.myntra.com/searchws/search/styleids2.
Here is my code :
Sub post()
Dim objHTTP As Object
Dim result As String
Set objIE = CreateObject("InternetExplorer.Application")
objIE.navigate "about:blank"
objIE.Visible = True
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
URL = "http://www.myntra.com/searchws/search/styleids2"
objHTTP.Open "POST", URL, False
objHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
objHTTP.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
objHTTP.send ("query=(full_text_myntra:(_)AND(statusid:1008))&start=0&rows=500&facet=true&facetField=[]")
result = objHTTP.responsetext
objIE.document.Write result
Set objHTTP = Nothing
End Sub
I got quite a few 415 errors when trying to run the query with Postman and it looks like the API was expecting JSON rather than form-urlencoded and the parameters needed to be wrapped in an array so I would check that in your code:
"[{""query"":""(full_text_myntra:(_)AND(statusid:1008))"",""start"":0,""rows"":500,""facet"":true,""facetField"":[]}]"
Additionally, I would recommend using something like Excel-REST (which I made for cases just like this that I was running into) to help with creating the request and handling JSON:
Dim MyntraClient As New RestClient
MyntraClient.BaseUrl = "http://www.myntra.com/"
' With inline JSON
Dim json As String
json = "[{""query"":""(full_text_myntra:(_)AND(statusid:1008))"",""start"":0,""rows"":500,""facet"":true,""facetField"":[]}]"
Dim Response As RestResponse
Set Response = MyntraClient.PostJSON("searchws/search/styleids2", json)
' It's no fun creating json string by hand, instead create it via Dictionary/Collection/Array
Dim SearchParameters As New Dictionary
SearchParameters.Add "query", "(full_text_myntra:(_)AND(statusid:1008))"
SearchParameters.Add "start", 0
SearchParameters.Add "rows", 500
SearchParameters.Add "facet", True
SearchParameters.Add "facetField", Array()
Set Response = MyntraClient.PostJSON("searchws/search/styleids2", Array(SearchParameters))
' Check status, received content, or do something with the data directly
Debug.Print Response.StatusCode
Debug.Print Response.Content
Debug.Print Response.Data("response1")("totalProductsCount")