I'm trying to scrape the first table (FINRA TRACE Bond Market Activity) of this website using Google Apps Script and I'm getting no data.
https://finra-markets.morningstar.com/BondCenter/TRACEMarketAggregateStats.jsp
enter image description here
function myFunction() {
const url = 'https://finra-markets.morningstar.com/BondCenter/TRACEMarketAggregateStats.jsp';
const res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
const $ = Cheerio.load(res);
var data = $('table').first().text();
Logger.log(data);
}
I have also tried from this page and I do not get any result.
https://finra-markets.morningstar.com/transferPage.jsp?path=http%3A%2F%2Fmuni-internal.morningstar.com%2Fpublic%2FMarketBreadth%2FC&_=1655503161665
I can't find a solution on the web and I ask you for help.
Thanks in advance
This page does a lot of things in the background. First, there is a POST request to https://finra-markets.morningstar.com/finralogin.jsp that initiates the session. Then, XHR requests are made to load the data tables. If you grab the cookie by POST'ing to that login page, you can pass it on the desired XHR call. That will return the table. The date you want to fetch the data for can be set with the date URL paramter. Here is an example:
function fetchFinra() {
const LOGIN_URL = "https://finra-markets.morningstar.com/finralogin.jsp";
const DATE = "06/24/2022" //the desired date
let opts = {
method: "POST",
payload: JSON.stringify({redirectPage: "/BondCenter/TRACEMarketAggregateStats.jsp"})
};
let res = UrlFetchApp.fetch(LOGIN_URL, opts);
let cookies = res.getAllHeaders()["Set-Cookie"];
const XHR_URL = `https://finra-markets.morningstar.com/transferPage.jsp?path=http%3A%2F%2Fmuni-internal.morningstar.com%2Fpublic%2FMarketBreadth%2FC&_=${new Date().getTime()}&date=${DATE}`;
res = UrlFetchApp.fetch(XHR_URL, { headers: {'Cookie': cookies.join(";")}} );
const $ = Cheerio.load(res.getContentText());
var data = $('table td').text();
Logger.log(data);
}
Related
I am a beginner with code script. Can you help me with my function please?
I have bot and he send data to google sheets, he send name, phone, date and method of communication. I need that google sheets write in column C date when was get the data from phone. I only now get the date, but in addition i need if - else. "If the column C is not empty send their date since last request", in addition i think I need to add method forEach and method so that the data is updated automatically when phone is received. For this I think need trigger "doGet(e)" from google documentation
(spread sheet image)
Data get from webhook
Here is my code:
function getDate() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var numbers = ss.getActiveSheet().getRange("B2:B1000")
let dateGoogle = new Date();
var rr = ss.getActiveSheet().getRange("C1:C1000").setValue(dateGoogle);
}
Just in case. If you're able to run the function getDate() and all you need is to make it to fill cells in C column only for rows that have filled cells in B column it can be done this way:
function getDate() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var range = ss.getActiveSheet().getData();
var data = range.getValues();
let dateGoogle = new Date();
data.forEach(x => x[2] = (x[2] == '' && x[1] != '') ? dateGoogle : x[2]);
range.setValues(data);
}
If you ask how to run the function getData() via doGet() I have no answer.
Using a doPost()
function doPost(e) {
Logger.log(e.postData.contents);
Logger.log(e.postData.type);
const ss = SpreadsheetApp.getActive();
const sh = ss.getSheetByName("Sheet1");
let data = JSON.parse(e.postData.contents);
let row = [];
['name','phone','date','method'].forEach(k => row.push(data[k]));
Logger.log(JSON.stringify(row))
sh.appendRow(row);
}
Below function Simulate what I imagine the bot can do to send data. This one is sending the data as JSON.
function sendData(obj) {
const url = ScriptApp.getService().getUrl();
const params={"contentType":"application/json","payload":JSON.stringify(obj),"muteHttpExceptions":true,"method":"post","headers": {"Authorization": "Bearer " + ScriptApp.getOAuthToken()}};
UrlFetchApp.fetch(url,params);
}
function saveMyData() {
sendData({name:"name",phone:"phone1",date:"date1",method:"post"});
}
You will have to Deploy the doPost(e) as a webapp.
Question: is there a way to set the request header item
contentType: application/json;odata=nometadata
prior to a call to TableClient.listEntities.
Objective: to receive data payloads uncluttered with odata metadata.
I am using the Azure Data Tables JavaScript API, and would like to specify request header item as follows:-
contentType: application/json;odata=nometadata
I've looked through the documentation (https://learn.microsoft.com/en-us/javascript/api/#azure/data-tables/?view=azure-node-latest) and there are some methods which facilitate changes to the request header, e.g. TableInsertEntityHeaders interface includes a property 'contentType'.
the TableClient.listEntities method includes a parameter (options?: ListTableEntitiesOptions) which does not include header access. So, as far as I can see, there is no obvious functionality supplied by the API to change the Request Header.
thank you
You can specify this in format parameter in the query options. Please see the sample code below:
const { TableClient, AzureNamedKeyCredential } = require("#azure/data-tables");
const account = "account-name";
const accountKey = "account-key";
const tableName = "table-name";
const credential = new AzureNamedKeyCredential(account, accountKey);
const client = new TableClient(`https://${account}.table.core.windows.net`, tableName, credential);
async function main() {
let entitiesIter = client.listEntities({
queryOptions: {
format: "application/json;odata=nometadata"
}
});
let i = 1;
for await (const entity of entitiesIter) {
console.log(`Entity ${i}:`);
console.log(entity);
console.log('==================');
i++;
}
}
main();
I created an Express Backend which sends JSON data as a chunk of text using res.write(JSON.stringify(JSONChunk)).
I want to handle and process each chunck of res.write in react front end and am using the following method:
My Backend pseudo code
for(let i = 0; i < slices; i ++) {
var JSONChunck = await getData(i); // getData operation can take some time
res.write(JSON.stringify(JSONChunck));
}
res.end();
FE pseudocode:
fetch(API, OPTS).then(async (response) => {
const reader = response.body.getReader();
while (true) {
const { done, value } = await reader.read();
if (done) {
break;
}
var text = new TextDecoder("utf-8").decode(value);
var result = JSON.parse(text);
var processedResult = process(result);
}
})
However, when I try the above code in some systems, I receive an error when trying to do JSON.parse(text) and see that the value of 'text' does not fetch full JSON string and only a partial string.
I was wondering if I am doing anything wrong or if there is any better way to do the above.
Any help/suggestions appreciated.
Thank you!
My Goal
I am, in short, attempting to create a script that visits a list of URLs and takes a screenshot of what is on the URL. For context, my goal is to save a snapshot that shows an image is listed on free stock photo sites (like pexels.com and unsplash.com).
My Code So Far
Here is what I have so far:
function stockDatabasePDF(){
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('stock db');
//make the pdf from the sheet
var data = sheet.getRange('A2:A').getValues(); //these are your URLs to check
for ( var i = 0; i < data.length; i++ ) {
if ( data[i][0] !== "" ) { //this means if your data (urls) are NOT blank
var theurl = data[i][0];
var token = ScriptApp.getOAuthToken();
var docurl = UrlFetchApp.fetch(theurl, { headers: { 'Authorization': 'Bearer ' + token } });
var pdf = docurl.getBlob().setName('name').getAs('application/pdf');
}
//save the file to folder on Drive
var fid = '1eHvWjIYyOeB9MQDwovzyXxx8CEIK5aOt';
var folder = DriveApp.getFolderById(fid);
var pdfs = folder.createFile(pdf).getUrl(); //both creates the file and copies the url
}
}
Just FYI... What is in range A2:A is the URLs to the stock photo sites, for example:
https://www.pexels.com/photo/person-riding-brown-horse-3490257/
https://unsplash.com/photos/Cz_SNZZyHgI
The issue
This script seems to ALMOST work. But it runs into one of two problems:
I get a 403 response code (forbidden request)
The screenshot takes, but no images/css/etc. is included.
Here is an example of that
To Close
Any help here would be greatly appreciated. Please let me know if I left anything out or if anyone has any questions. Thank you all!
I am querying the Google Places API with the following set-up:
Backend: Radar Search Request via post-route (using Express and 'request'-module in Node.js)
Frontend: Display results on map, querying Place Detail request after clicking on marker icon (example from API documentation as reference)
All works fine, except that my search request doesn't seem to pass on my restriction of 'only places currently open' via 'opennow=true' within search request. There are surely restaurants open in Amsterdam at the time I send this request, I manually checked it on GoogleMaps. I've been trying all different ways of spelling and searched for similar solutions, but can't find the problem. Any help very much appreciated, thank you.
Code for reference:
app.post('/results', function(req,res){
//PARAMETERS FOR SEARCH QUERY FOR RESTAURANTS, OPEN NOW
const baseurl = 'https://maps.googleapis.com/maps/api/place/radarsearch/json?';
const location = 'location=52.370216,4.895168';
const radius = 'radius=5000';
const type = 'type=food';
const key = 'key='+process.env.googleapikey;
// const opennow = 'openNow=true'; //doesn't narrow down results, doesn't seem to get applied
// const opennow2 = 'open_now=true'; //doesn't narrow down results, doesn't seem to get applied
const opennow3 = 'opennow=true'; //0 search results returned, doesn't seem to work
const queryurl = `${baseurl}${location}&${radius}&${type}&${key}&${opennow3}`;
//SEARCH QUERY TO GOOGLE PLACES API, USING REQUEST MODULE
request({
uri: queryurl,
method: "GET",
timeout: 10000,
followRedirect: true,
maxRedireccts: 10
}, function(err, response, body) {
var allresults = [];
if(err){
console.log(err);
} else {
var responseparsed = JSON.parse(body);
var results = responseparsed.results;
for (var i = 0; i < results.length; i++) {
allresults.push(results[i]);
}
console.log('Maps found >'+results.length+'< results');
}
res.render("results", {allresults: allresults, mapsjsapikey: mapsjsapikey});
}
)
});