Develop Reference

No data scraping a table using Apps Script

I'm trying to scrape the first table (FINRA TRACE Bond Market Activity) of this website using Google Apps Script and I'm getting no data.
https://finra-markets.morningstar.com/BondCenter/TRACEMarketAggregateStats.jsp
enter image description here
function myFunction() {
const url = 'https://finra-markets.morningstar.com/BondCenter/TRACEMarketAggregateStats.jsp';
const res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
const $ = Cheerio.load(res);
var data = $('table').first().text();
Logger.log(data);
}
I have also tried from this page and I do not get any result.
https://finra-markets.morningstar.com/transferPage.jsp?path=http%3A%2F%2Fmuni-internal.morningstar.com%2Fpublic%2FMarketBreadth%2FC&_=1655503161665
I can't find a solution on the web and I ask you for help.
Thanks in advance

This page does a lot of things in the background. First, there is a POST request to https://finra-markets.morningstar.com/finralogin.jsp that initiates the session. Then, XHR requests are made to load the data tables. If you grab the cookie by POST'ing to that login page, you can pass it on the desired XHR call. That will return the table. The date you want to fetch the data for can be set with the date URL paramter. Here is an example:
function fetchFinra() {
const LOGIN_URL = "https://finra-markets.morningstar.com/finralogin.jsp";
const DATE = "06/24/2022" //the desired date
let opts = {
method: "POST",
payload: JSON.stringify({redirectPage: "/BondCenter/TRACEMarketAggregateStats.jsp"})
};
let res = UrlFetchApp.fetch(LOGIN_URL, opts);
let cookies = res.getAllHeaders()["Set-Cookie"];
const XHR_URL = `https://finra-markets.morningstar.com/transferPage.jsp?path=http%3A%2F%2Fmuni-internal.morningstar.com%2Fpublic%2FMarketBreadth%2FC&_=${new Date().getTime()}&date=${DATE}`;
res = UrlFetchApp.fetch(XHR_URL, { headers: {'Cookie': cookies.join(";")}} );
const $ = Cheerio.load(res.getContentText());
var data = $('table td').text();
Logger.log(data);
}

Django/Ajax/Javasdript JSON.parse convert string to JavaScript object

I am using Ajax to to send a get request from the sever, i am querying and getting a particular product from the product model, i want to return result as JavaScript objects but its return this '{' as i try to access the first value from the responseText[0]. How can i convert data return to js object.
here is my code
views.py
def edit_product(request):
product_id = request.GET.get('product_id')
print('THIS IS PRODUCT ID ', product_id)
product = Product.objects.get(pk=product_id)
data = [
{
'name':product.name,
'brand':product.brand,
'price':product.price,
'description':product.description
}
]
return HttpResponse(data)
ajax.js
function getProductEditId(product_id){
//alert(id)
document.getElementById('gridSystemModalLabel').innerHTML='<b> Edit product </b>';
//change modal header from Add product to Edit product
var request = new XMLHttpRequest();
request.open('GET', '/edit_product/?product_id=' + product_id, true);
request.onload = function(){
console.log('hello world', product_id)
//var data = request.responseText
var data = JSON.parse(JSON.stringify(request.responseText));
console.log(data[0])
}
request.send();
}

A HTTP response can not contain a dictionary, you can pass data through a specific format, like for example JSON. Django offers some convenience for that with the JsonResponse [Django-doc]:
from django.http import JsonResponse
def edit_product(request):
product_id = request.GET.get('product_id')
print('THIS IS PRODUCT ID ', product_id)
product = Product.objects.get(pk=product_id)
data = [
{
'name':product.name,
'brand':product.brand,
'price':product.price,
'description':product.description
}
]
return JsonResponse(data, safe=False)
But as the code hints, this is not safe, since an array at the root level was a famous JSON hijacking exploit.
It is typically better to just pass a dictionary (so no list), and then the safe=False parameter can be removed. Although it is not really "safe" to then just assume that all security exploits are gone.
Alternatively, you can use json.dumps [Python-doc] (which is more or less what JsonResponse does internally), but then you thus will probably end with duplicate code.
At the JavaScript side, you then only need to parse the JSON blob to a JavaScript object:
//change modal header from Add product to Edit product
var request = new XMLHttpRequest();
request.open('GET', '/edit_product/?product_id=' + product_id, true);
request.onreadystatechange = function() {
console.log('hello world', product_id)
if(this.status == 200 && this.readyState == 4) {
var data = JSON.parse(this.responseText);
console.log(data[0])
}
}
It is also not very clear to me why you encapsulate the data in a list here: if the response always contains one element, it makes more sense to just pass the dictionary.

How to correctly send and receive deflated data

I'm using protobufs for serializing my data. So far I serialized my data on the server (node restify) send it, receive it (Request is made by XMLHttpRequest) and serialize it on the Client.
Now I want to employ zipping to reduce the transfered file size. I tried using the library pako, which uses zlib.
In a basic script that I used to compare the protobuf and zipping performance to json I used it this way, and there were no problems
var buffer = proto.encode(data,'MyType'); // Own library on top of protobufs
var output = pako.deflate(buffer);
var unpacked = pako.inflate(output);
var decoded = proto.decode(buffer,'MyType');
However if I try to do this in a client-server model I can't get it working.
Server:
server.get('/data', function (req, res) {
const data = getMyData();
const buffer = proto.encode(data, 'MyType');
res.setHeader('content-type', 'application/octet-stream;');
res.setHeader('Content-Encoding', 'gzip;');
return res.send(200,buffer);
});
My own protolibrary serializes the data in protobuf and then deflates it:
...
let buffer = type.encode(message).finish();
buffer = pako.deflate(buffer);
return buffer;
The request looks like this:
public getData(){
return new Promise((resolve,reject) => {
const request = new XMLHttpRequest();
request.open("GET", this.url, true);
request.responseType = "arraybuffer";
request.onload = function(evt) {
const arr = new Uint8Array(request.response);
const payload = proto.decode(request.response ,'MyType')
resolve(payload);
};
request.send();
});
}
The proto.decode method first inflates the buffer buffer = pako.inflate(buffer); and then deserializes it from Protobuf.
If the request is made i get following error: "Uncaught incorrect header check" returned by the inflate method of pako:
function inflate(input, options) {
var inflator = new Inflate(options);
inflator.push(input, true);
// That will never happens, if you don't cheat with options :)
if (inflator.err) { throw inflator.msg || msg[inflator.err]; }
return inflator.result;
}
Also I looked at the request in Postman and found following:
The deflated response looks like this: 120,156,60,221,119,64,21,237,119,39,240,247,246,242,10,49,191,244,178,73,54,157 and has a length of 378564
The same request without deflating (the protobuf) looks like this
�:�:
 (� 0�8#H
 (� 0�8#H
� (�0�8#�Hand has a length of 272613.
I'm assuming, that I'm doing something incorrectly on the server side, since the inflated request is larger than the one not using compression.
Is it the content-type Header? I'm out of ideas.

Sending Username and password to web service using json

Hi i would like to create a login page in html 5, but in order to login i need to send the username and password to a web service, the web service is already created and it will return a value 0 if flase and any other number if true. I can only use javascript and json. Can anyone help me out to understand json as i know almost nothing about it. How would i send the username and password using json and javascript?

var form;
form.onsubmit = function (e) {
// stop the regular form submission
e.preventDefault();
// collect the form data while iterating over the inputs
var data = {};
for (var i = 0, ii = form.length; i < ii; ++i) {
var input = form[i];
if (input.name) {
data[input.name] = input.value;
}
}
// construct an HTTP request
var xhr = new XMLHttpRequest();
xhr.open(form.method, form.action, true);
xhr.setRequestHeader('Content-Type', 'application/json; charset=UTF-8');
// send the collected data as JSON
xhr.send(JSON.stringify(data));
xhr.onloadend = function () {
// done
};
};

Chrome extension: how to pass ArrayBuffer or Blob from content script to the background without losing its type?

I have this content script that downloads some binary data using XHR, which is sent later to the background script:
var self = this;
var xhr = new XMLHttpRequest();
xhr.open('GET', url);
xhr.responseType = 'arraybuffer';
xhr.onload = function(e) {
if (this.status == 200) {
self.data = {
data: xhr.response,
contentType: xhr.getResponseHeader('Content-Type')
};
}
};
xhr.send();
... later ...
sendResponse({data: self.data});
After receiving this data in background script, I'd like to form another XHR request that uploads this binary data to my server, so I do:
var formData = new FormData();
var bb = new WebKitBlobBuilder();
bb.append(data.data);
formData.append("data", bb.getBlob(data.contentType));
var req = new XMLHttpRequest();
req.open("POST", serverUrl);
req.send(formData);
The problem is that the file uploaded to the server contains just this string: "[object Object]". I guess this happens because ArrayBuffer type is lost somehow while transferring it from content process to the background? How can I solve that?

Messages passed between a Content Script and a background page are JSON-serialized.
If you want to transfer an ArrayBuffer object through a JSON-serialized channel, wrap the buffer in a view, before and after transferring.
I show an isolated example, so that the solution is generally applicable, and not just in your case. The example shows how to pass around ArrayBuffers and typed arrays, but the method can also be applied to File and Blob objects, by using the FileReader API.
// In your case: self.data = { data: new Uint8Array(xhr.response), ...
// Generic example:
var example = new ArrayBuffer(10);
var data = {
// Create a view
data: Array.apply(null, new Uint8Array(example)),
contentType: 'x-an-example'
};
// Transport over a JSON-serialized channel. In your case: sendResponse
var transportData = JSON.stringify(data);
//"{"data":[0,0,0,0,0,0,0,0,0,0],"contentType":"x-an-example"}"
// At the receivers end. In your case: chrome.extension.onRequest
var receivedData = JSON.parse(transportData);
// data.data is an Object, NOT an ArrayBuffer or Uint8Array
receivedData.data = new Uint8Array(receivedData.data).buffer;
// Now, receivedData is the expected ArrayBuffer object
This solution has been tested successfully in Chrome 18 and Firefox.
new Uint8Array(xhr.response) is used to create a view of the ArrayBuffer, so that the individual bytes can be read.
Array.apply(null, <Uint8Array>) is used to create a plain array, using the keys from the Uint8Array view. This step reduces the size of the serialized message. WARNING: This method only works for small amounts of data. When the size of the typed array exceeds 125836, a RangeError will be thrown. If you need to handle large pieces of data, use other methods to do the conversion between typed arrays and plain arrays.
At the receivers end, the original buffer can be obtained by creating a new Uint8Array, and reading the buffer attribute.
Implementation in your Google Chrome extension:
// Part of the Content script
self.data = {
data: Array.apply(null, new Uint8Array(xhr.response)),
contentType: xhr.getResponseHeader('Content-Type')
};
...
sendResponse({data: self.data});
// Part of the background page
chrome.runtime.onMessage.addListener(function(data, sender, callback) {
...
data.data = new Uint8Array(data.data).buffer;
Documentation
MDN: Typed Arrays
MDN: ArrayBuffer
MDN: Uint8Array
MDN: <Function> .apply
Google Chrome Extension docs: Messaging > Simple one-time requests
"This lets you send a one-time JSON-serializable message from a content script to extension, or vice versa, respectively"
SO bonus: Upload a File in a Google Chrome Extension - Using a Web worker to request, validate, process and submit binary data.

For Chromium Extensions manifest v3, URL.createObjectURL() approach doesn't work anymore because it is prohibited in the service workers.
The best (easiest) way to pass data from a service worker to a content script (and vice-versa), is to convert the blob into a base64 representation.
const fetchBlob = async url => {
const response = await fetch(url);
const blob = await response.blob();
const base64 = await convertBlobToBase64(blob);
return base64;
};
const convertBlobToBase64 = blob => new Promise(resolve => {
const reader = new FileReader();
reader.readAsDataURL(blob);
reader.onloadend = () => {
const base64data = reader.result;
resolve(base64data);
};
});
Then send the base64 to the content script.
Service worker:
chrome.tabs.sendMessage(sender.tab.id, { type: "LOADED_FILE", base64: base64 });
Content script:
chrome.runtime.onMessage.addListener(async (request, sender) => {
if (request.type == "LOADED_FILE" && sender.id == '<your_extension_id>') {
// do anything you want with the data from the service worker.
// e.g. convert it back to a blob
const response = await fetch(request.base64);
const blob = await response.blob();
}
});