Scrape HTML page with Javascript into Excel - javascript

Hoping one of you guys can help me here. I'm trying to import data into excel from this webpage as well as some others that are the same website: https://re.po.st/qZZYQJ - basically I want to take the total clicks figure: 322,627 and have that in a column beside the URL in excel. I tried using importhtml and importxml in google spreadsheets but could not get it to work. I believe it is because this element is rendered in javascript? Does anyone know how to get this element? Is there a VBA code I could use. I'm not the most technical person so my apologies if I'm overseeing some things.
Mike

Function test(byval strLocation as string) as string
Dim driver As SeleniumWrapper.WebDriver
Set driver = New SeleniumWrapper.WebDriver
driver.Start "chrome", "https://re.po.st"
driver.Open strLocation
Dim data1
data1 = driver.FindElementById("sguidtotaltable").FindElementByTagName("span").Text
test = data1
driver.stop 'Stops the browser
End Function

Related

How to use Google Sheets as read-only database for Angular, without making the sheet public

I'm trying to use Google Sheets document as read-only database for my angular application.
I tried some methods to do that, but the problem with all of these methods is that they require the Sheet to be shared publicly (anyone with the link can access the sheet). But what I want is to share it with specific user using Service Account through credentials.
I'm using Angular 14
There is no reference to Angular in Google Sheets for Developers.
If you know any solution or come across an article about this topic, please share it with me.
Thank you.
Here are the steps you'll need to take in order to read from Google Sheets into Angular:
Step 1: Prepare your Google Sheet
1.) Make sure ALL the cells in your sheet are formatted as "Plain text". To do this, click in the upper-left corner of the sheet nexus where the rows and columns intersect to select all cells, then select Format > Number > Plain text from the top menu.
2.) Go to Share, then under "General Access", select "Anyone with the link", then click Done. I believe in your case that this step is optional, since you do not want the sheet to be public.
3.) Go to File > Share > Publish to web in the top menu. Set the scope of what you want to publish, then click Publish. Unfortunately in your case, this step is NOT optional!
Step 2: Fetch the Google Sheet Data
Use the following code example to fetch the raw data from your Google Sheet as plain text:
const docId = '1vLjJqvLGdaS39ccsvoU58kEWXngzV_VXtto07Ki6qVo';
const sheetId = ''; // to get a specific sheet by ID, use '&gid=###'
const url = `https://docs.google.com/spreadsheets/d/${docId}/gviz/tq?tqx=out:json${sheetId}`;
this.http.get(url, {
responseType: 'text',
}).subscribe((response: string): void => {
console.log(response);
});
Step 3: Parse the Raw Text as JSON
Use the following example to parse the raw text to JSON:
const rawJSONText = response.match(/google\.visualization\.Query\.setResponse\(([\s\S\w]+)\)/); // strip the header response
const json = JSON.parse(rawJSONText[1]);
console.log(json);
Hope this helps. Cheers!

Parsing a stringified JSON coming from a Google Sheet Web App

I'm trying to parse a stringified JSON output from a web app created from a google sheets script. I thought it couldn't be that complicated, but I've tried everything I could think of or find out online... so now asking for help if that's OK!
on the web app / Google Sheets side, the code is:
function doGet(e) {
var spreadsheet = SpreadsheetApp.openById('spreadsheetID');
var worksheet = spreadsheet.getSheetByName('Rankings C/U');
var output = JSON.stringify({ data: worksheet.getDataRange().getValues() });
return HtmlService.createHtmlOutput(output);
}
I've published the script, the web app works, I'm OK with that bit.
I've put random values on the spreadsheet: [[1,2],[3,4]] if we speak in matrix format.
on the other end, I've tried a bunch of stuff including .fetch, JSON.parse() to get the data in a usable format within the Google Sites embedded code, but the real issue is that I think I can't get to allocate the payload to a variable?
I'm using Google Sites to fetch the data.
with the basic module "<> embed", with the "by URL" option, with the following code:
https://script.google.com/macros/s/scriptID/exec
I get the following output - that looks what it should be:
{"data":[[1,2],[3,4]]}
but when trying to include this in a script module ("embed code") - no chance!
<form name="get-images">
<input name="test" id="test" value="we'll put the contents of cell A1 here">
</form>
<script>
const form = document.forms['get-images']
var usableVariable = JSON.parse("https://script.google.com/macros/s/scriptID/exec"); // here I'm trying to allocate the stringified JSON to a variable
form.elements['test'].value = usableVariable[1,1]; //allocating the first element of the parsed array
</script>
I'm pretty sure I'm missing something obvious - but now I ran out of ideas!
Thanks for any help :)
I believe your goal as follows.
In your situation, the bottom script is embedded to the Google site.
You want to retrieve the values from doGet and want to put the value of cell "B2" to the input tag.
The settings of Web Apps is Execute the app as: Me and Who has access to the app: Anyone, even Anonymous.
Modification points:
In your case, I think that return ContentService.createTextOutput(output); is suitable instead of return HtmlService.createHtmlOutput(output); in Google Apps Script.
In order to retrieve the values from doGet, in this modification, fetch is used.
You want to retrieve the cell "B2" from usableVariable[1,1];, please modify it to usableVariable[1][1];
When above points are reflected to your script, it becomes as follows.
Modified script:
Google Apps Script side:
function doGet(e) {
var spreadsheet = SpreadsheetApp.openById('spreadsheetID');
var worksheet = spreadsheet.getSheetByName('Rankings C/U');
var output = JSON.stringify({ data: worksheet.getDataRange().getValues() });
return ContentService.createTextOutput(output);
}
HTML & Javascript side:
<form name="get-images">
<input name="test" id="test" value="we'll put the contents of cell A1 here">
</form>
<script>
let url = "https://script.google.com/macros/s/###/exec";
fetch(url)
.then((res) => res.json())
.then((res) => {
const usableVariable = res.data;
const form = document.forms['get-images'];
form.elements['test'].value = usableVariable[1][1]; // usableVariable[1][1] is the cell "B2".
});
</script>
Note:
When you modified the Google Apps Script of Web Apps, please redeploy the Web Apps as new version. By this, the latest script is reflected to the Web Apps. Please be careful this.
In my environment, I could confirm that above HTML & Javascript worked in the Google site by embedding.
References:
Class ContentService
Using Fetch
Web Apps

How to get data from Viadeo with X-Ray and NodeJs

So I am trying to scrape some content with node.js x-ray scraping framework. While I can get the content from a single page but for exemple only for one employee I can't get my head around on how to get for all the employees.
Working Exemple but return me the first employee:
const request =require('request');
const Xray=require('x-ray');
var x = Xray();
x('http://www.viadeo.com/fr/company/unicef',
'.pan',[{
name:'.pan-emp-name',
job:'.pan-emp-pos',
since:'.pan-emp-age'
// job:'#profile #overview-summary-current ol'
}]).write('result.json')
Thank you so much
x('http://www.viadeo.com/fr/company/unicef',
'#pan-emp .pan-employees .pan-empployee',[{
company:'#company-info .company-logo-picture',
nom:'.pan-emp-name',
job:'.pan-emp-pos',
depuis:'.pan-emp-age'
// job:'#profile #overview-summary-current ol'
}]).write('result.json')
Working like a charm,
So now my problem is to get the company info

Python Flask data feed from Pandas Dataframe, dynamically define with unique endpoint

Hi I am building a web app with Flask Python. I got a problem here:
#app.route('/analytics/signals/<ticker_url>')
def analytics_signals_com_page(ticker_url):
all_ticker = full_list
ticker_name = com_name
ticker = ticker_url.upper()
pricerec = sp500[ticker_url.upper()].tolist()
timerec = sp500[ticker_url.upper()].index.tolist()
return render_template('company.html', all_ticker=all_ticker, ticker_name=ticker_name, ticker=ticker, pricerec=pricerec, timerec=timerec)
Here I am defining company pages based on the a page will contain different content. The problem is that everything is fine upto ticker = ticker_url.upper(). It works perfectly fine. But for pricerec and timerec, they make problems.
sp500 is a pandas DataFrame columns being companies like "AAPL", "GOOG","MSFT", and so forth 505 companies and the index are timestamps, and values are the prices at each time.
So what I am doing for the pricerec, I am taking the ticker_url and use it to take the specific company's price and make it as a list. And timerec is to take the index (timestamps) and make it as a list. And I am passing these two variables into the company.html page.
But it makes internal server error. I do not know why it happens.
My expectation was that when a user click a button that href to "~/analytics/signals/aapl" then the company.html page will contain the pricerec and timerec for me to draw a graph. But it didn't work like that. It makes internal server error. I defined those two variables in the javascript also like I did for the other variables(all_ticker, ticker_name, and ticker)
Can anyone help me with this issue?
Thanks!

How can I pull information from 2 different excel cells on the same worksheet using javascript? I

<script language="javascript" >
function GetData(cell,row){
var excel = new ActiveXObject("Excel.Application");
var excel_file = excel.Workbooks.Open("I:Work/database/72550.xls");
var excel_sheet = excel.Worksheets("CA 2012-2013 HMO Plans");
var data = excel_sheet.Cells(cell,row).Value;
document.getElementById('div1').innerText =data;
}
</script>
<input type="button" value="Hearing Services" onClick="GetData(45,2);" />
This is all in a table but I didn't think it was nec to display that, and of course file names are changed. All I need to know is how to pull from two cells at once from the same work sheet to display on a webpage. I got it working put only from one cell. How can I add another cell data to my result text?
is easier to take all the information inside a database, and do a cross join, as you usuing java, i recommend you use oracle, and so a simple cross join, depends on the head text of your excel sheets.
inside your code use odbc connection, you can simple get the connection part if you connect your database inside the sql developer.

Categories

Resources