Develop Reference

API in excel to retrieve WIX database JSON file

I am trying to find out how to automatically export my data from WIX with an API call. I know this will be in JSON format and will have to be parsed in excel.
So far my js code on the wix back end looks like this:
import { ok, notFound, serverError } from 'wix-http-functions';
import wixdatra from 'wixdata';
export function my_function() {
let options = {
"headers": {
"content-type": "application/Json"
}
};
return wixData.query("database table name")
.find()
.then(results => {
if (results.items.length > 0) {
options.body ={
"items": results.items
}
return ok(options);
}
}
}
And my VB code looks like this:
Dim hReq As Object, Json As Dictionary
Dim sht As Worksheet
Dim authKey As String
authKey = "xxxkeyxxx"
Set sht = Sheet1
Dim strUrl As String
strUrl = "https:endpointurl"
Set hReq = CreateObject("MSXML2.XMLHTTP")
With hReq
.Open "GET", strUrl, False
.setRequestHeader "Authorization", authKey
.Send
End With
Dim response As String
response = hReq.ResponseText
MsgBox response
End Sub
I imagine both codes are probably wrong for a number of reasons but I am really out of my depth here! Any help would be appreciated.

Option 1 will work, with some modification, for any email provider that exposes an API for sending emails.
I'm not exactly sure what you mean in option 2, but it sounds like you can create an HTTP function on the Velo side that will expose data from a database collection as an API. Then you can call that API from any third party system.

One thing to note, which I am guessing you have discovered is that the JSON returned has the data you want, nested within an "items" JSON key pair, and you will need to parse that out one side or another...

VBA XMLHTTP request doesn't capture dynamic HTML response

I am trying to get a specific dynamic figure from a webpage to excel, I managed to gather all the website get response into a "all" variable which I am supposed to parse to extract my numbers, except for when I check the string variable I can see everything but the required dynamic figure! :) "the attached phot shows the dynamic figure at the very instant was 2.19",
any ideas why I am capturing every thing, would be much appreciated, Thanks in advance
My thoughts:
1.I am guessing is the figures are injected by JavaScript or a server side that might be executing after my XMLHTTP request is processed maybe! if this is the case or else I need your expertise
the website doesn't response unless it sees a specific Html request header, so I might need to mimic the headers of Chrome, I don't know how they look like?
Please see below my code and a screenshot for the figure I would like to capture
'Tools>refrences>microsoft xml v3 must be refrenced
Public Function GetWebSource(ByRef URL As String) As String
Dim xml As IXMLHTTPRequest
On Error Resume Next
Set xml = CreateObject("Microsoft.XMLHTTP")
With xml
.Open "GET", URL, False
.send
GetWebSource = .responseText
End With
Set xml = Nothing
End Function
Sub ADAD()
Dim all As Variant
Dim objHTTP As Object
Dim URL As String
Set objHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")
all = GetWebSource("https://www.tradingview.com/symbols/CRYPTOCAP-ADA.D/")
pos = InStr(all, "tv-symbol-price-quote__value js-symbol-last")
testString = Mid(all, pos, 200)
'I am supposed to see the dynamic figure within the TAG but it is not showing!!
Debug.Print testString
End Sub
HTML for Dynamic Required values

#Tim Williams This is a code using selenium (But it seems doesn't do the trick of getting the value)
PhantomJS Selenium VBA
Sub Test()
Dim bot As Selenium.PhantomJSDriver
Set bot = New Selenium.PhantomJSDriver
With bot
.Get "https://www.tradingview.com/symbols/CRYPTOCAP-ADA.D/"
.Wait 2000
Debug.Print .FindElementByXPath("//div[#class='tv-symbol-price-quote__value js-symbol-last']").Attribute("outerHTML")
End With
End Sub
Chrome VBA Selenium
It seems using PhantomJS doesn't work properly, so here's a Chrome version of selenium in VBA
Private bot As Selenium.ChromeDriver
Sub Test()
Set bot = New Selenium.ChromeDriver
With bot
.Start
.Get "https://www.tradingview.com/symbols/CRYPTOCAP-ADA.D/"
Debug.Print .FindElementByXPath("//div[#class='tv-symbol-price-quote__value js-symbol-last']").Text 'Attribute("outerHTML")
.Quit
End With
End Sub
Python Solution
And this is the working python code that my tutor #QHarr provided in comments
from selenium import webdriver
d = webdriver.Chrome("D:/Webdrivers/chromedriver.exe")
d.get('https://www.tradingview.com/symbols/CRYPTOCAP-ADA.D/')
d.find_element_by_css_selector('.tv-symbol-price-quote__value.js-symbol-last').text

VBA scrape web data which is contained in Javascript

I'm trying to get web table data from the following below website and extract the first table regarding Policy rates on the right hand-side.
https://www.researchonline.se/macro/our_forecasts
Have use the following code just to see if it spits out the desired data but keep getting error 91. Suspecting something about Javascript that I need to consider in my code? Below is my code.
Dim request As Object
Dim response As String
Dim html As New HTMLDocument
Dim website As String
Dim price As Variant
' Website to go to.
website = "https://www.researchonline.se/macro/our_forecasts"
' Create the object that will make the webpage request.
Set request = CreateObject("MSXML2.XMLHTTP")
' Where to go and how to go there - probably don't need to change this.
request.Open "GET", website, False
' Get fresh data.
request.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
' Send the request for the webpage.
request.send
' Get the webpage response data into a variable.
response = StrConv(request.responseBody, vbUnicode)
' Put the webpage into an html object to make data references easier.
html.body.innerHTML = response
' Get the price variable from the specified element on the page and just check in a message box if that is the correct data.
price = html.getElementsByTagName("table")(0).innerText
' Output the price into a message box.
MsgBox price

Set request = CreateObject("MSXML2.XMLHTTP")
The above cannot execute javascript , Though instead of calling url and fetching it from table. you can directly use their API
for example to get policy rate, the site uses API (There are many apis , i have seen by looking into the site). One example is
https://www.researchonline.se/api/MacroAdmin/GetForecast?name=policy&start=Mon%20Sep%2014%202020&end=Tue%20Sep%2014%202021
which will return an xml kindda response, which you can parse with MSXML2.XMLHTTP

URL GET using Basic Auth with Java Servlet

I have a .jsp page which handles some basic html (text field, submit button a table to populate) and javascript. On click it calls my java Servlet class which handles the request and the response from and to the jsp respectfully. I also have a Java class to handle url connection to make a REST call using GET which should return a json string result. This is the scenario: User wishes to make a simple search by entering an ID value to populate the table using REST to connect to a url and get a json response.
I have separately tested the Java class to make a REST call made by URL connection with Basic Authentication to GET the data in json string and it works perfectly fine. (I have coded 4 different methods of doing this and all work fine and return the expected json).
I have separately tested the jsp page making a call to the servlet to populate "dummy" values in the table and the servlet response is fine and the table populates as expected.
THE PROBLEM:
When I tried to populate the table by the values obtained from the GET REST call, it hangs and I get no result. So I tried to investigate why and found out that for some crazy reason, the servlet doesn't like the line of code which sets the header with the basic authentication to get access to the URL. I tried commenting the basic auth 1 line code out (to debug, so now we have no authentication) but passing through some "dummy" hard coded data, and it populates the table.
I actually don't know why the servlet doesn't like it when I set the authentication. My guess is probably it is overwriting the servlet's response path and therefore lost the initial response location to the jsp? (i.e. where to send the json back to?) Can anyone help me here? Anyone know whats happening? is my assumption of the problem correct? if so, how do I overcome this problem?
Javascript call to the servlet:
$("#myForm2").submit(function(e) {
$.ajax({
type: "GET",
url: "servlet1", // the script where the form input is handled.
data: $("#myForm2").serialize(), // serializes the form's elements.
success: function(responseData){
var dataObj = $.parseJSON(responseData)
$('#myTable').append(>>add values to table here<<);
});
e.preventDefault(); // avoid to execute the actual submit of the form.
$("#myForm2")[0].reset(); // clear previous values entered
});
Servlet class, request made by jsp page and the servlet's response:
Rest getCallOnURL = new Rest(); // instance of my rest class
public void doGet(HttpServletRequest request,
HttpServletResponse response)
throws ServletException,IOException {
// create a dummy string "test" to hold the ID value passed in from javascript
String testID = request.getParameter("ID");
// Hard coded - only for testing purposes
if (testID.equals("1")){
//call my java class' method to connect to url and retrieve the json string
getCallOnURL.connectToUrlAndMakeAGetRequest();
valueToPopulateTable = getCallOnURL.testJsonString;
}
response.setContentType(CONTENT_TYPE);
PrintWriter out = response.getWriter();
// using backslash to escape the quotes to make a valid json response
out.print("{\"testRow\": \"valueToPopulateTable\"}");
out.close();
}
My Java Class method to make URL call and Basic Authentication:
String testJsonString = "defaultValue";//testing purposes
//creates a valid encoded authentication string
private String basicAuth(String name, String pass){
String authString = name + ":" + pass;
byte[] authEncBytes = Base64.encodeBase64(authString.getBytes());
String authStringEnc = new String(authEncBytes);
return authStringEnc;
}
private HttpURLConnection prepareGETConnection(final URL verifierURL)
throws IOException {
final HttpURLConnection connection = (HttpURLConnection) verifierURL.openConnection();
connection.setRequestMethod("GET");
//this is the line which authorizes access to the url via Basic Auth
connection.setRequestProperty("Authorization", "Basic " + basicAuth(name, password));
return connection;
}
public void connectToUrlAndMakeAGetRequest(){
try {
String webPage = "URL HERE";// pass it a valid REST url with endpoints
URL url = new URL(webPage);
HttpURLConnection conn = prepareGETConnection(url);
// get the url response json string
InputStream getUrlresponse = conn.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(getUrlresponse));
//get only the first line of the string - testing purposes only!
testJsonString = rd.readLine();
rd.close();
conn.disconnect();
} catch(IOException e){
System.out.println("IO exception: "+e);
}
}

How do I post large data to the server?

I have to send the XML from the client side to the server side.
The method adopted by me was that:
First the xml is converted to string in the javascript and then post as a uri
var url = '/perl/set_zorder_xml.cgi'+'?'+xmlString+'&'+location+'&'+'nocache='+randomnumber;
xml string is the string that contains the xml in string form.
The post function looks like this:
if (window.XMLHttpRequest) {
req_anno = new XMLHttpRequest();
req_anno.open("POST", url, false);
req_anno.send();
}
The problem is that when my xml string is very large then html 414 error occurs i.e url too large.
Is there any way out, Javascript and perl is used

Even though you're doing a POST request, you're still sending the data in the querystring of the URL. Instead you should move the data to be sent as POST data, and remove it from the URL.
req_anno.open("POST", '/perl/set_zorder_xml.cgi', false);
req_anno.send('xml=' + encodeURIComponent(xmlString));
The XHR .send() method accepts the string to be sent as the request body (ie POST data).