I am currently working on a VBA code that retrieves the top file from this website (http://infopost.bwpmlp.com/Posting/default.aspx?Mode=Display&Id=27&tspid=100000). I am able to click on the button using Javascript in my code, and I am able to click on open after the download is kicked off. However, I am having trouble saving the file. Because the workbook is being pulled from a website there really isn't a way to set it to the active workbook that I can think of. Currently when I do ActiveWorkbook.SaveAs the code is saving the blank workbook that I am testing the code out of. The file I downloaded seems to not open until the entire code is done running even after I try putting in breaks. Anyone have any ideas? My code is below. Thanks!
Option Explicit
Dim ie As InternetExplorer
Dim h As LongPtr
Private Declare PtrSafe Function FindWindowEx Lib "user32" Alias "FindWindowExA" (ByVal hWnd1 As LongPtr, ByVal hWnd2 As LongPtr, ByVal lpsz1 As String, ByVal lpsz2 As String) As LongPtr
Sub Texas_Gas()
Application.DisplayAlerts = True
Dim ie As Object
Dim IeHandle As Long, FileDownloadHandle As Long, OpenButtonHandle As Long, IePopupBarHandle As Long
Dim AutoMode As Boolean, FileDownloadClassicPopup As Boolean, DownloadComplete As Boolean
Dim Timeout As Date
Dim strSPICE As String, strLink As String
Dim PopupGap As Integer, i As Integer
Set ie = CreateObject("InternetExplorer.Application")
DownloadComplete = False
FileDownloadClassicPopup = False
FileDownloadHandle = 0
IePopupBarHandle = 0
With ie
.Visible = True
.navigate "http://infopost.bwpmlp.com/Posting/default.aspx? Mode=Display&Id=27&tspid=100000"
Do While .Busy: DoEvents: Loop
Do While .readyState <> 4: DoEvents: Loop
End With
ie.document.parentWindow.execScript "javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(""dgITMatrix:0:lnkBtnDownload"", """", true, """", """", false, true))"
Dim o As IUIAutomation
Dim e As IUIAutomationElement
Dim iCnd As IUIAutomationCondition
Set o = New CUIAutomation
h = ie.Hwnd
h = FindWindowEx(h, 0, "Frame Notification Bar", vbNullString)
If h = 0 Then Exit Sub
Set e = o.ElementFromHandle(ByVal h)
Set iCnd = o.CreatePropertyCondition(UIA_NamePropertyId, "Open")
Dim Button As IUIAutomationElement
Set Button = e.FindFirst(TreeScope_Subtree, iCnd)
Dim InvokePattern As IUIAutomationInvokePattern
Set InvokePattern = Button.GetCurrentPattern(UIA_InvokePatternId)
InvokePattern.Invoke
ActiveWorkbook.SaveAs "I:\Cap_Rel\raw_scrapes\Texas_Gas_Transmission\parsed\Texas_Gas_Transmission_CapRel" & Format(Date - 1, "yyyymmdd") & "MACRO", FileFormat:=xlCSV
End Sub
Related
I'm trying to extract values stored inside different buttons on the webpage.
It seems button of each variant has no name, they are just called "variant__box", which are under "variants" div class.
As far as I can tell, values are loaded by javascript on each variant__box.
This is the website to get the data:
https://www.honda.co.uk/motorcycles/range/adventure/crf1100l-africa-twin-adventure-sports/specifications-and-price.html#/
This is the code I've written so far
Dim ie As Object
Dim html As New HTMLDocument
Dim address, str As String
Dim jobDetailsList As Object
Dim jobitem As Object
Set ie = CreateObject("InternetExplorer.Application")
ie.navigate address 'the one mentioned above
ie.Visible = False
While ie.Busy Or ie.readyState < 4
DoEvents
Wend
Set html = ie.document
Set jobDetailsList = html.getElementsByClassName("variants")
For Each jobitem In jobDetailsList
jobitem.Click
str = jobitem.innerText
ActiveSheet.Cells(i, 5).Value = str
i = i + 1
Next jobitem
Set html = Nothing
ie.Quit
Set ie = Nothing
It returns nothing.
If you want to use the IE you can use the following code. But SIM's suggestion is better because IE is then omitted.
Sub ScrapeMotorCycleData()
Dim ie As Object
Dim address, str As String
Dim jobDetailsList As Object
Dim jobitem As Object
Dim i As Long
i = 2
address = "https://www.honda.co.uk/motorcycles/range/adventure/crf1100l-africa-twin-adventure-sports/specifications-and-price.html#/"
Set ie = CreateObject("InternetExplorer.Application")
ie.navigate address 'the one mentioned above
ie.Visible = False
'The following line doesn't do what you want
'While ie.Busy Or ie.readyState < 4: DoEvents: Wend
'You nee a loop here to wait for loading the dynamic content
'Ask for the HTML part you want to scrape
'(No timeout included here, but it can be programmed)
Do
Set jobDetailsList = ie.document.getElementsByClassName("variant__wrapper")
Loop Until jobDetailsList.Length > 0
For Each jobitem In jobDetailsList
ActiveSheet.Cells(i, 5).Value = jobitem.innerText
i = i + 1
Next jobitem
ie.Quit
Set ie = Nothing
End Sub
<div class="r_title">
<h1 data-securitycontent="name">Fidelity® Japan Smaller Companies</h1>
<span class="gry"> FJSCX</span>
<span data-msat="span-securityInformation-star" class="r_star3"></span>
How would I go about extracting r_star3 from this? r_star3 represents 3 stars. So far, I'm able to get the inner text of it, but the stars are symbols so its blank and r_star3 seems to be its own class. I'd like to just extract r_star3 as a string and use if statements to see how many stars it is. Anything helps, thanks.
Edit:
Here's what I have so far using query selector, but the querySelector prints out [object HTMLSpanElement]. I only pasted in the relevant code. This is the site where the stars are located (right by the ticker signs).
.navigate "http://www.morningstar.com/funds/xnas/" & Range("A" & Row.Row).Value & "/quote.html"
Do
DoEvents
Loop Until ie.readyState = READYSTATE_COMPLETE
Dim doc As HTMLDocument
Set doc = ie.document
While ie.readyState <> 4
Wend
Application.Wait (Now + TimeValue("0:00:04"))
Dim tblName As Object
Dim span As Object
On Error Resume Next
'FIND THE STAR (Work in Progress)
Set tblName = doc.getElementsByClassName("reports_nav")(0)
Set span = tblName.getElementsByTagName("span").Item(1)
Dim s As String, rating As Long
s = doc.querySelector("span[class*=""r_star""]")
MsgBox (s)
rating = Replace(Split(Split(s, "class=" & Chr$(34))(1), Chr$(34))(0), "r_star", vbNullString)
Range("C" & Row.Row).Value = rating
MsgBox (rating)
You can obtain target HTML with CSS selector e.g. the following will get the element in question:
span[data-msat="span-securityInformation-star"]
which returns:
Parse result:
You can then parse the OuterHTML from the returned element to get the start rating.
Code:
Option Explicit
Public Sub Get_Information()
Dim IE As New InternetExplorer
With IE
.Visible = True
.navigate "http://www.morningstar.com/funds/xnas/seatx/quote.html"
While .Busy = True Or .readyState < 4: DoEvents: Wend
Dim a As Object, exitTime As Date
exitTime = Now + TimeSerial(0, 0, 5)
Do
DoEvents
On Error Resume Next
Set a = .document.querySelector("span[data-msat=""span-securityInformation-star""]") '<== Loop until time out checking if element has been found and set
On Error GoTo 0
If Now > exitTime Then Exit Do
Loop While a Is Nothing
If a Is Nothing Then Exit Sub
Dim rating As Long
rating = Replace(Split(Split(a.outerHTML, "class=" & Chr$(34))(1), Chr$(34))(0), "r_star", vbNullString)
MsgBox rating
.Quit
End With
End Sub
What I would like to do is to search a website using VBA, putting some words in the left box and getting results on the right.
The problem is that I don't know HTML and I don't know how to refer to this box. I use GetElementByID but I received error in line:
objIE.Document.GetElementByID("text-translation-video-ad").Value = "piłka".
"Object doesn't support this property or method".
Here's my code:
Sub www()
Set objIE = CreateObject("InternetExplorer.Application")
objIE.Top = 0
objIE.Left = 0
objIE.Width = 800
objIE.Height = 600
objIE.AddressBar = 0
objIE.StatusBar = 0
objIE.Toolbar = 0
objIE.Visible = True
objIE.Navigate ("https://pl.pons.com/tłumaczenie-tekstu")
Do
DoEvents
Loop Until objIE.ReadyState = 4
pagesource = objIE.Document.Body.Outerhtml
objIE.Document.GetElementByID("text-translation-video-ad").Value = "piłka"
objIE.Document.GetElementByID("qKeyboardInputInitiator").Click
Do
DoEvents
Loop Until objIE.ReadyState = 4
End Sub
Without changing any language settings, the following translates "Hello"
Code:
Option Explicit
Public Sub GetInfo()
Dim IE As New InternetExplorer, html As HTMLDocument, translation As String
Const TRANSLATION_STRING As String = "Hello"
With IE
.Visible = True
.navigate "https://pl.pons.com/t%C5%82umaczenie-tekstu"
While .Busy Or .readyState < 4: DoEvents: Wend
Set html = .document
With html
.querySelector("textarea.text-translation-source.source").Value = TRANSLATION_STRING
.querySelector("button.btn.btn-primary.submit").Click
Application.Wait Now + TimeSerial(0, 0, 3)
translation = .querySelector("div.translated_text").innerText
End With
Debug.Print translation
'Quit '<== Remember to quit application
End With
End Sub
View:
Print out in immediate window:
Edit:
Late bound version
Option Explicit
Public Sub GetInfo()
Dim IE As Object, html As Object
With CreateObject("InternetExplorer.Application")
.Visible = True
.navigate "https://pl.pons.com/t%C5%82umaczenie-tekstu"
While .Busy Or .readyState < 4: DoEvents: Wend
Set html = CreateObject("htmlfile")
Set html = .document
With html
.getElementsByClassName("text-translation-source source")(0).innerText = "Translate"
.getElementsByClassName("btn btn-primary submit")(0).Click
Application.Wait Now + TimeSerial(0, 0, 2)
Dim i As Long
For i = 0 To .getElementsByClassName("text-translation-target target").Length - 1
Debug.Print .getElementsByClassName("text-translation-target target")(i).innerText
Next i
Stop
End With
.Quit
End With
End Sub
Element with ID "text-translation-video-ad" is a DIV which does not have .Value property. You want to access text area which is descendant of mentioned DIV.
There are 2 elements with tag "textarea" on page, the one which interests you is 1st element, therefore (0) index. Tags in GetElementsByTagName must be capitalized.
objIE.Document.GetElementsByTagName("TEXTAREA")(0).Value = "piłka"
You can also resign from IE automation and take a faster and more reliable approach, without browser automation, which will give you response in JSON format. Setting reference to Microsoft HTML Object Library is required.
Option Explicit
Public Sub Scrape()
Dim WindHttp As Object: Set WindHttp = CreateObject("WinHTTP.WinHTTPRequest.5.1")
Dim htmlDoc As New HTMLDocument
Dim urlName As String, myWord As String, requestString As String
Dim myResults() As String
Dim resultNum As Long
urlName = "https://pl.pons.com/_translate/translate"
myWord = "piłka"
requestString = "source_language=pl&target_language=en&service=deepl&text=" & _
myWord & _
"&lookup=true&requested_by=Web&source_language_confirmed=true"
Set htmlDoc = postDocument(urlName, WindHttp, requestString)
myResults = Split(Replace(Split(Split(htmlDoc.body.innerText, ",")(1), ":")(1), Chr(34), vbNullString), vbCrLf)
For resultNum = LBound(myResults) To UBound(myResults)
Debug.Print myResults(resultNum)
Next resultNum
End Sub
Function postDocument(ByVal urlName As String, myRequest As Object, Optional requestString As String) As HTMLDocument
Set postDocument = New HTMLDocument
With myRequest
.Open "POST", urlName, False
.setRequestHeader "Cache-Control", "no-cache"
.setRequestHeader "Pragma", "no-cache"
.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
If requestString = vbNullString Then
.send
Else
.send requestString
End If
postDocument.body.innerHTML = .responseText
End With
End Function
How do you write VBA code to download a file sitting behind a JavaScript link? There are many resources on how to download a file from a specific link using VBA, however, none show how to download a file behind a JavaScript link.
In example, how do you download the file behind "Export to Spreadsheet" on this website:
https://www.vanguardinvestments.com.au/retail/ret/investments/product.html#/fundDetail/wholesale/portId=8101/assetCode=equity/?prices
Do we still declare and use urlmon?
'Declaration of API function for Office 2010+
Private Declare PtrSafe Function URLDownloadTOFile Lib "urlmon" Alias
"URLDownloadToFileA" ( _
ByVal pCaller As LongPtr, _
ByVal sZURL As String, _
ByVal szFileName As String, _
ByVal dwReserved As LongPtr, _
ByVal lpfnCB As LongPtr _
) As LongPtr
#Else
'Declaration of API function for pre Office 2010 versions
Private Declare Function URLDownloadTOFile Lib "urlmon" Alias
"URLDownloadToFileA" ( _
ByVal pCaller As Long, _
ByVal sZURL As String, _
ByVal szFileName As String, _
ByVal dwReserved As Long, _
ByVal lpfnCB As Long _
) As Long
#End If
Sub DownloadOneFile()
Dim FileURL As String
Dim DestinationFile As String
'How do you modify this to handle a javascript link?
FileURL = "https://www.vanguardinvestments.com.au/retail/ret/investments/product.html#/fundDetail/wholesale/portId=8101/assetCode=equity/?prices"
DestinationFile = "C:\VBA\prices.csv"
URLDownloadToFile 0, FileURL, DestinationFile, 0, 0
End Sub
This will fire the event. Credit to #Greedo for the principle of waiting for page to load by looping until a specified element is visible in the window. Sorry about the dreaded send keys.
Public Sub DownloadFile()
Dim objIE As InternetExplorer, currPage As HTMLDocument, url As String
url = "https://www.vanguardinvestments.com.au/retail/ret/investments/product.html#/fundDetail/wholesale/portId=8101/assetCode=equity/?prices"
Set objIE = New InternetExplorer
objIE.navigate url
Do While objIE.Busy = True Or objIE.readyState <> 4: DoEvents: Loop
Set currPage = objIE.document
objIE.Visible = True
Dim myDiv As HTMLDivElement: Set myDiv = currPage.getElementById("price-distribution")
Dim elemRect As IHTMLRect: Set elemRect = myDiv.getBoundingClientRect
Do Until elemRect.bottom > 0
currPage.parentWindow.scrollBy 0, 10000
Set elemRect = myDiv.getBoundingClientRect
Loop
objIE.document.getElementsByClassName("export_icon hideOnSml ng-binding")(0).FireEvent "onclick"
Application.SendKeys "%{S}"
End Sub
If necessary you might add something like the following before the send keys to ensure window is up but seems to work as is at present.
Dim objShell As Shell
Set objShell = New Shell
Application.Wait Now + TimeSerial(0, 0, 10) 'alter to give enough time for window
For Each objIE In objShell.Windows
If TypeName(objIE.document) = "HTMLDocument" Then
If InStr(objIE.document.title, "vanguard") > 0 Then
objIE.Visible = True
Exit For
End If
End If
Next objIE
So when I visit my page and I set the video mode using this:
$("#pushStreamButton").click(function () {
if (initialized) {
var channelSelected = $("#channelSelect").val();
var gameNameEntered = $("#gameName").val();
var channelNameEntered = $("#channelName").val();
if (gameNameEntered.length < 1) return;
$("#goLiveWindow").dialog("close");
serverHub.server.setVideoMode(channelSelected, gameNameEntered, "", channelNameEntered);
}
});
and this is in the backend
Public Sub setVideoMode(ByVal mode As String, ByVal details As String, ByVal socialMessage As String, ByVal channel As String)
Dim user As FrontPageUser = Connections.matchFirst(Connections.frontPageUsers, Context.ConnectionId)
If user Is Nothing Then Return
If mode = "youtube" Then
Try
My.Computer.FileSystem.DeleteFile(HttpContext.Current.Server.MapPath("/images/streamavatar.gif"))
Catch Ex As Exception
Console.Write(Ex)
End Try
End If
This functions as expected, it does it's job, however I have a button that reverts the channel back to the default youtube
$("#stopStreamButton").click(function () {
serverHub.server.setVideoMode("youtube", "", "", "");
});
After changing the channel using the first button, clicking the second button doesn't do anything, it doesn't even seem to attempt to execute the command, which is bizarre. However if I reload the page the button functions correctly and the channel is reverted.
This issue also isn't present when using SSE/longpolling/foreverframe, only when using websockets as the transport. I'm treading relatively unknown waters with websockets, and I've tried debugging the javascript and the backend code - the javascript doesn't seem to execute correctly, but it doesn't spout any errors, and it never reaches the backend.
If anybody knows why this happens and could explain where I'm going wrong, it'd be appreciated.
Additional: after digging further, it actually appears to stop all functions from the java side and after a while it seems to disconnect and force a refresh, not sure what is causing it, but I know it has to be related to websockets.
edit --
Upon debugging cause, it appears to be linked to another function which sends a push notification to subscribers, when I disable this function it works without hanging and blocking commands, but when this function is initialized it hangs, seemingly after the function has completed
Shared Function sendPushFox(username, pushDetails) As String
Dim avatar = Utils.getAvatarPath(username)
If avatar.Contains("/forum/download/file.php?avatar=") OrElse avatar.Contains("/images/solaire.png") Then
Try
My.Computer.Network.DownloadFile("https://foo.com" & avatar, HttpContext.Current.Server.MapPath("/images/streamavatar.gif"))
Catch ex As Exception
ChatProcessor.postNewMessage(Nothing, Nothing, ChatMessage.MessageType.Channel_Mod, "Problem downloading streamer avatar." & ex.ToString)
End Try
Else
Try
My.Computer.Network.DownloadFile(avatar, HttpContext.Current.Server.MapPath("/images/streamavatar.gif"))
Catch ex As Exception
ChatProcessor.postNewMessage(Nothing, Nothing, ChatMessage.MessageType.Channel_Mod, "Problem downloading streamer avatar." & ex.ToString)
End Try
End If
Dim query As String = "SELECT subscribeid FROM custom_user_data WHERE NOT subscribeid = ' ';"
Dim connection As New MySqlConnection(Utils.connectionString) : connection.Open()
Dim command As MySqlCommand = New MySqlCommand(query, connection)
Dim reader As MySqlDataReader = command.ExecuteReader()
Dim regList As New List(Of String)
Do While reader.Read
regList.Add(reader.GetString(0))
' IO.File.AppendAllText(Utils.serverPath & "errorlog.txt", reg1)
Loop
connection.Close()
Dim query2 As String = "SELECT p256dh FROM custom_user_data WHERE NOT p256dh = ' ';"
Dim connection2 As New MySqlConnection(Utils.connectionString) : connection2.Open()
Dim command2 As MySqlCommand = New MySqlCommand(query2, connection2)
Dim reader2 As MySqlDataReader = command2.ExecuteReader()
Dim regList2 As New List(Of String)
Do While reader2.Read
regList2.Add(reader2.GetString(0))
' IO.File.AppendAllText(Utils.serverPath & "errorlog.txt", reg1)
Loop
connection2.Close()
Dim query3 As String = "SELECT authsecret FROM custom_user_data WHERE NOT authsecret = ' ';"
Dim connection3 As New MySqlConnection(Utils.connectionString) : connection3.Open()
Dim command3 As MySqlCommand = New MySqlCommand(query3, connection3)
Dim reader3 As MySqlDataReader = command3.ExecuteReader()
Dim regList3 As New List(Of String)
Do While reader3.Read
regList3.Add(reader3.GetString(0))
' IO.File.AppendAllText(Utils.serverPath & "errorlog.txt", reg1)
Loop
connection3.Close()
Dim reg1 = regList.ToArray
Dim reg2 = regList2.ToArray
Dim reg3 = regList3.ToArray
Dim payload = "{ ""title"": ""foo bar"", ""body"": """ & username.name.ToString & " playing " & pushDetails.ToString & """, ""icon"" : ""https://foo.com/images/streamavatar.gif"" }"
For i As Integer = 0 To reg1.Length - 1
Dim webPushClient = New WebPushClient()
Dim subject = "https://foo.com"
Dim vapidKeys As VapidDetails = VapidHelper.GenerateVapidKeys()
Dim vapidDetails = New VapidDetails(subject, vapidKeys.PublicKey, vapidKeys.PrivateKey)
Try
Console.WriteLine("Public {0}", vapidKeys.PublicKey)
Console.WriteLine("Private {0}", vapidKeys.PrivateKey)
Dim subscription = New PushSubscription(reg1(i), reg2(i), reg3(i))
webPushClient.SetGCMAPIKey("key here")
webPushClient.SendNotification(subscription, payload, vapidDetails)
Catch Ex As Exception
Console.Write(Ex)
End Try
Next
Return Nothing
End Function
So, it's a clash with websockets, since it functions on longpolling and etc but I'm still not sure why, and it only affects the user who calls the function.
Strangely enough this appears to hang after the "next" which happens here
For i As Integer = 0 To reg1.Length - 1
webPushClient.SetGCMAPIKey("key here")
webPushClient.SendNotification(subscription, payload, vapidDetails)
Next
it functions completely fine, but once it reaches the end of the for, it just never continues, it does literally nothing after Next, no errors, nothing happens, set breaks and results are as expected, I'm just baffled as to why it doesn't continue, and only using websockets trasports, longpolling and etc is fine
I actually figured this out, it turns out that websockets don't like the non asynchronous communication made from the command
webPushClient.SendNotification(subscription, payload, vapidDetails)
Fortunately the webPush lib has an async and changing it to
webPushClient.SendNotificationAsync(subscription, payload, vapidDetails)
fixed everything.