I am working on a chrome extension that was utilizing the google search api, to return an image from an ajax request. Apparently the api is no longer functional, so I was wondering how I could make a similar request with the Custom Search Element Api. I have pasted the portion of my code that was making the request below.
var logo = function (searchTerm, callback, message, name) {
var searchUrl = 'https://ajax.googleapis.com/ajax/services/search/images' +
'?v=1.0&q=' + encodeURIComponent(searchTerm);
var x = new XMLHttpRequest();
x.open('GET', searchUrl);
// The Google image search API responds with JSON, so let Chrome parse it.
x.responseType = 'json';
x.onload = function() {
// Parse and process the response from Google Image Search.
var response = x.response;
if (!response || !response.responseData || !response.responseData.results ||
response.responseData.results.length === 0) {
console.log( "loading error" )
console.log(response)
console.log(response.responseData)
console.log(response.responseData.results)
}
var firstResult = response.responseData.results[0];
// Take the thumbnail instead of the full image to get an approximately
// consistent image size.
var imageUrl = firstResult.tbUrl;
var width = parseInt(firstResult.tbWidth);
var height = parseInt(firstResult.tbHeight);
console.assert(
typeof imageUrl == 'string' && !isNaN(width) && !isNaN(height),
'Unexpected respose from the Google Image Search API!');
callback(imageUrl, width, height, message, name);
};
x.onerror = function() {
alert("error")
document.writeln("network error");
};
x.send();
}
I have been trying to figure it out by reading the the documentation, but I could definitely use some help.
Here's the link of the docs: Using REST.
You need to create your own Custom Search Engine on google developer console, modify the getImageUrl() method accordingly.
change searchUrl
var searchUrl = 'https://www.googleapis.com/customsearch/v1?' +
'key=' + 'YOUR_KEY' +
'&cx=' + '00000:yourcx' +
'&searchType=image' +
'&q=' + encodeURIComponent(searchTerm);
and modify the response object according to the JSON.
You can refer to this question for how to tweak the console and refine search result.
Related
I've written a JS script that consumes data from an http API (endpoint for the GET request: http://api.open-notify.org/iss-now.json). I intend to use it in a page hosted on github pages (which makes use of the https protocol).
Now that it is online I see in the console that this data can't be used in-browser because of the mixed active content error: Blocked loading mixed active content “http://api.open-notify.org/iss-now.json”.
I can't apply this solution because I'm not making use of a server.js file (my content is served by github). I wanted to try this other solution but it requires opening the adapter page in another tab, which just isn't viable.
I'm trying this workaround but https://cors-anywhere.herokuapp.com/http://api.open-notify.org/iss-now.json returns an error (Missing required request header. Must specify one of: origin,x-requested-with). If anyone know how to add headers to the loadJSON method please tell me, I can't find anything in its documentation. I'm not exactly at ease with the syntax of fetch, so when I try it:
var response = fetch("https://cors-anywhere.herokuapp.com/http://api.open-notify.org/iss-now.json", {
headers: {
Origin: window.location.protocol + '//' + window.location.host
}
});
if (response.ok) { // if HTTP-status is 200-299
// get the response body (the method explained below)
var json = response.json();
return(json);
} else {
alert("HTTP-Error: " + response.status);
}
I get to add the "origin" header, only to find myself with a
Cross-Origin Request Blocked: The Same Origin Policy disallows reading
the remote resource at
https://cors-anywhere.herokuapp.com/http://api.open-notify.org/iss-now.json.
(Reason: CORS header ‘Access-Control-Allow-Origin’ missing)
Which as far as I understand can only be corrected server-side.
The github page of cors-anywhere encourages to implement their solution in your own script, by adding this snippet:
(function() {
var cors_api_host = 'cors-anywhere.herokuapp.com';
var cors_api_url = 'https://' + cors_api_host + '/';
var slice = [].slice;
var origin = window.location.protocol + '//' + window.location.host;
var open = XMLHttpRequest.prototype.open;
XMLHttpRequest.prototype.open = function() {
var args = slice.call(arguments);
var targetOrigin = /^https?:\/\/([^\/]+)/i.exec(args[1]);
if (targetOrigin && targetOrigin[0].toLowerCase() !== origin &&
targetOrigin[1] !== cors_api_host) {
args[1] = cors_api_url + args[1];
}
return open.apply(this, args);
};
})();
but I wouldn't know how to implement it, I haven't succeeded integrating it in my code for now.
My code is a bit of a mess right now but I can show you this much:
// global functions for the Tracker sample
function getData() {
// var promise = fetch("http://api.open-notify.org/iss-now.json");
loadJSON("http://api.open-notify.org/iss-now.json", gotData, 'jsonp');
}
function gotData(data) {
background(img)
displaySample()
// this will allow you to see the raw data live in your browser console
//console.log(data.iss_position.latitude);
//console.log(data.iss_position.longitude);
posX = (parseFloat(data.iss_position.latitude * latConst) + translateX)
posY = (parseFloat(data.iss_position.longitude * lonConst)* -1 + translateY)
console.log(posX);
console.log(posY);
fill(250, 50, 50, 90);
ellipse(posX, posY, 10, 10);
}
function draw() {
// case tracker
if (selectedSample === 1) {
translateX = boxSizeWidth / 2;
translateY = boxSizeHeight / 2;
latConst = boxSizeWidth / 360;
lonConst = boxSizeHeight / 180;
if (t === 0) {
getData()
}
}
I also tried finding an https API giving the same data (latitude and longitude of the ISS in real time) but I can't seem to find any for now, and finding a workaround would be interesting anyway.
You could use fetch like this:
fetch("https://cors-anywhere.herokuapp.com/http://api.open-notify.org/iss-now.json", {
headers: { Origin: window.location.host }
})
.then(res => res.json())
.then(res => {
console.log(res);
// gotData(res);
})
.catch(err => {
console.log(err);
});
I am using Facebook Graph API to create a Facebook ads campaign with Google Apps Script.
I need to upload an image to my Facebook ad account. I have already tried to use the image bytes as a Base64 UTF-8 string, but when I call the API I get:
Exception: Limit Exceeded: URLFetch URL Length.
Basically, the string is too long.
I am using the following code:
function uploadTest2() {
var image_id = 'blabla';
var image_blob = DriveApp.getFileById(image_id).getBlob();
var input = image_blob.getBytes();
var docImg = Utilities.base64Encode(input);
var account_id = '1111111111111';
var facebookUrl =
'https://graph.facebook.com/v7.0' +
'/act_' + account_id +
'/adimages?bytes=' + docImg +
'&access_token=' + TOKEN;
Logger.log(facebookUrl);
//var encodedFacebookUrl = encodeURI(facebookUrl);
var options = {
'method' : 'post'
};
var response = UrlFetchApp.fetch(facebookUrl, options);
var results = JSON.parse(response);
Logger.log(response);
}
The image does not exceed 5MB and I have already check the bytes string with an online decoder to verify it.
Do you have any idea on how to use the image URL directly in the post request?
The second version of the code:
function uploadTest2() {
var image_id = 'blabla';
var image_blob = DriveApp.getFileById(image_id).getBlob();
var input = image_blob.getBytes();
var docImg = Utilities.base64Encode(input);
var account_id = '1111111111111';
var facebookUrl =
'https://graph.facebook.com/v7.0' +
'/act_' + account_id +
// '/adimages?bytes=' + encodedImage +
// '&access_token=' + TOKEN;
'/adimages?access_token=' + TOKEN;
Logger.log(facebookUrl);
//var encodedFacebookUrl = encodeURI(facebookUrl);
var options = {
'method' : 'post',
'payload' : image_blob
};
var response = UrlFetchApp.fetch(facebookUrl, options);
var results = JSON.parse(response);
Logger.log(response);
}
Solution
In order to make a post request of an image with UrlFetchApp.fetch() you must provide the method, payload (i.e the body you want to POST) and sometimes the content type (if what we are passing is not a JavaScript object).
If you want to pass a base64Encode object obtained from a blob you should stringify this JSON object.
What the original poster was missing was to pass the payload and after my contribution and his work he finally solved the issue by editing the options variable such as:
var options = {
'method' : 'post',
'contentType': 'application/json',
'payload': JSON.stringify({"bytes": docImg,"name" : 'Test'})};
}
Documentation reference : Class UrlFetchApp
I am building a simple open source Chromium extension that retrieve some data from several urls and then update the DOM. I could find another way to do this than by adding the line to update the DOM inside the callback http1.onreadystatechange
My XMLHttpRequest requests were often stuck on http1.readyState = 3 so I have added a 3rd parameter to http1.open("GET"); to make the request synchronous like this:
http1.open("GET", url, false);
But I am still getting these errors:
results[1].join is not a function at XMLHttpRequest.http.onreadystatechange
annot read property 'join' of undefined at XMLHttpRequest.http.onreadystatechange
Even thought they don't prevent the script from running, I think this isn't the right way to do what I want. So here is my question: how to update the DOM with the responses from several XMLHttpRequest request? Let's say I need to retrieve and compare all the data before updating the DOM. Then is there a way to process all the data at once after we have retrieve all of them (cf my comment on the last line)?
Here is the relevant part of my script, the full script is available here:
var urls = [
["https://www.cnrtl.fr/morphologie/" + keyword, "vtoolbar", "morf_sound"], //best for plural
["https://www.cnrtl.fr/synonymie/" + keyword, "syno_format"],
]
// for test set keyword to any of this word : hibou, tribal, aller, lancer
var resultdiv = document.getElementById("result")
resultdiv.innerText = "requete en cours";
var results = [];
var errors = [];
urls.forEach((item, index) => {
var http = new XMLHttpRequest();
http.onreadystatechange = function () {
if (http.readyState == 4 && http.status == 200) {
parser = new DOMParser();
var ulr1response = parser.parseFromString(http.responseText, "text/html");
if (index == 0) {
//retrieve the data needed, save then in a list and push this list to the main list result
} else if (index == 1) {
//retrieve the data needed, save then in a list and push this list to the main list result
}
// update the DOM
if (results[1] == "") {
resultdiv.innerHTML = results[0].join(", ") + "</br></br>Pas de synonymes trouvés"
} else {
resultdiv.innerHTML = "<b>" + results[0].join(", ") + "</br></br>Synonymes:</b></br>● " + results[1].join('</br>● ')
}
} else {
errors.push(index);
resultdiv.innerText = "Erreur: " + index + " " + http.readyState + " " + http.status;
}
}
http.open("GET", item[0], false);
http.send(null); // null = no parameters
});
// it would be simplier if I could update the DOM here and not in http.onreadystatechange
If you want to execute some code once all requests have succeeded, you can try using Promise.all together with Fetch.
let keyword = "beaucoup";
let parser = new DOMParser();
let urls = [
["https://www.cnrtl.fr/morphologie/" + keyword, "vtoolbar", "morf_sound"], //best for plural
["https://www.cnrtl.fr/synonymie/" + keyword, "syno_format"]
];
let fetchPromises = urls.map(
item => fetch(item[0]).then(
response => parser.parseFromString(response.text(), "text/html")
)
);
Promise.all(fetchPromises).then(
results => {
// code in here executes once all fetchPromises have succeeded
// "results" will be an array of parsed response data
console.log(results);
}
).catch(console.error);
I am building a chrome extension to pull data out of a page to build a url from the data and I want to have that url shortened as the final product. In my content scripts file I make a call out to a url shortener to compress a link. I keep getting returned a promise with no value which crashes react. In devtools I see that the callout is made successfully and the url is returned.
I have tried async await, a full async function, tried forcing the response.toString()
Here is the relevant section of code.
var listingInfo = new Map();
listingInfo.set('Address', 'some standard address');
var tinyLink = '(http://tinyurl.com)/api-create.php?url='; //() because I can't share shortener urls on this site.
/*-----------------------------------GET LINKS--------------------------*/
if(listingInfo.has('Address')){ var mapsLink = \`https://www.google.com/maps/place/${listingInfo.get('Address').replace(new RegExp(" ", "g"), '+')}\`;
tinyLink = \`${tinyLink}${mapsLink} `;
var dirLink = fetch(tinyLink, {
method: "GET",
mode: "no-cors",
headers: {"Content-Type": "text/html"} }).then((response)=>{
return response; });
listingInfo.set('dirLink', dirLink); }
I expected to receive a plain text string because in the network tab of devtools it shows a simple string url and not any JSON, but I keep receiving a resolved promise with value="".
// made this function to use XMLHttpRequest()
const setLink = (propName, url) => {
var xhr = new XMLHttpRequest();
xhr.open("GET", url, true);
xhr.onreadystatechange = function() {
if (xhr.readyState == 4) {
var link = xhr.responseText;
console.log(`link: '${link}'`);
listingInfo.set(propName, link);
}
}
xhr.send();
}
// Then called setLink()
if(listingInfo.has('Address')){
var mapsLink =
`https://www.google.com/maps/place/${listingInfo.get('Address').replace(new
RegExp(" ", "g"), '+')}`;
dirLink = `${tinyLink}${mapsLink}`;
console.log(dirLink);
setLink('dirLink', dirLink);
console.log(dirLink);
}
I would like to scrape for each link on this page the page details page behind.
I can get all informations on this page: PAGE
However, I would like to get all info's on the details page, but the href link looks like that, for example:
href="javascript:subOpen('9ca8ed0fae15d43dc1257e7300345b99')"
Here is my sample spreadsheet using the ImportHTML function to get the general overview.
Google Spreadsheet
Any suggestions how to get the details pages?
UPDATE
I implemented the method the following:
function doGet(e){
var base = 'http://www.ediktsdatei.justiz.gv.at/edikte/ex/exedi3.nsf/'
var feed = UrlFetchApp.fetch(base + 'suche?OpenForm&subf=e&query=%28%5BVKat%5D%3DEH%20%7C%20%5BVKat%5D%3DZH%20%7C%20%5BVKat%5D%3DMH%20%7C%20%5BVKat%5D%3DMW%20%7C%20%5BVKat%5D%3DMSH%20%7C%20%5BVKat%5D%3DGGH%20%7C%20%5BVKat%5D%3DRH%20%7C%20%5BVKat%5D%3DHAN%20%7C%20%5BVKat%5D%3DWE%20%7C%20%5BVKat%5D%3DEW%20%7C%20%5BVKat%5D%3DMAI%20%7C%20%5BVKat%5D%3DDTW%20%7C%20%5BVKat%5D%3DDGW%20%7C%20%5BVKat%5D%3DGA%20%7C%20%5BVKat%5D%3DGW%20%7C%20%5BVKat%5D%3DUL%20%7C%20%5BVKat%5D%3DBBL%20%7C%20%5BVKat%5D%3DLF%20%7C%20%5BVKat%5D%3DGL%20%7C%20%5BVKat%5D%3DSE%20%7C%20%5BVKat%5D%3DSO%29%20AND%20%5BBL%5D%3D0').getContentText();
var d = document.createElement('div'); //assuming you can do this
d.innerHTML = feed;//make the text a dom structure
var arr = d.getElementsByTagName('a') //iterate over the page links
var response = "";
for(var i = 0;i<arr.length;i++){
var atr = arr[i].getAttribute('onclick');
if(atr) atr = atr.match(/subOpen\((.*?)\)/) //if onclick calls subOpen
if(atr && atr.length > 1){ //get the id
var detail = UrlFetchApp.fetch(base + '0/'+atr[1]).getContentText();
response += detail//process the relevant part of the content and append to the reposnse text
}
}
return ContentService.createTextOutput(response);
}
However, I get an error when running the method:
ReferenceError: "document" is not defined. (line 6, file "")
What is the document an object of?
I have update the Google Spreadsheet with a webapp.
You can use Firebug in order to inspect the page contents and javascript. For instance you can find that subOpen is actually an alias to subOpenXML declared in xmlhttp01.js.
function subOpenXML(unid) {/*open found doc from search view*/
if (waiting) return alert(bittewar);
var wState = dynDoc.getElementById('windowState');
wState.value = 'H';/*httpreq pending*/
var last = '';
if (unid==docLinks[0]) {last += '&f=1'; thisdocnum = 1;}
if (unid==docLinks[docLinks.length-1]) {
last += '&l=1';
thisdocnum = docLinks.length;
} else {
for (var i=1;i<docLinks.length-1;i++)
if (unid==docLinks[i]) {thisdocnum = i+1; break;}
}
var url = unid + html_delim + 'OpenDocument'+last + '&bm=2';
httpreq.open('GET', // &rand=' + Math.random();
/*'/edikte/test/ex/exedi31.nsf/0/'+*/ '0/'+url, true);
httpreq.onreadystatechange=onreadystatechange;
// httpreq.setRequestHeader('Accept','text/xml');
httpreq.send(null);
waiting = true;
title2src = firstTextChild(dynDoc.getElementById('title2')).nodeValue;
}
So, after copying the function source and modifying it in firebug's Console tab to add a console.log(url) before the http call, like this:
var url = unid + html_delim + 'OpenDocument'+last + '&bm=2';
console.log(url)
httpreq.open('GET', // &rand=' + Math.random();
/*'/edikte/test/ex/exedi31.nsf/0/'+*/ '0/'+url, true);
You can execute the function declaration in firebug's Console tab and overwrite subOpen with the modified source.
Clickin in the link then will show that the invoked url is composed of the id passed as parameter to subOpen prefixed by '0/', so in the example you posted it would be a GET to:
http://www.ediktsdatei.justiz.gv.at/edikte/ex/exedi3.nsf/0/1fd2313c2e0095bfc1257e49004170ca?OpenDocument&f=1&bm=2
You could also verify this by opening the Network tab in firebug and clicking the link.
Therefore, in order to scrape the details page you'd need to
Parse the id passed to subOpen
Make a GET call to '0/'
Parse the request response
Looking the request response in firebug's Network Tab shows that probably you'll need to do similar parsing to actually get the showed contents, but I haven't looked deep into it.
UPDATE
The importHTML function is not suitable for the kind of scraping you want. Google's HTML or Content Services are better suited for this. You'll need to create a web app and implement the doGet function:
function doGet(e){
var base = 'http://www.ediktsdatei.justiz.gv.at/edikte/ex/exedi3.nsf/'
var feed = UrlFetchApp.fetch(base + 'suche?OpenForm&subf=e&query=%28%5BVKat%5D%3DEH%20%7C%20%5BVKat%5D%3DZH%20%7C%20%5BVKat%5D%3DMH%20%7C%20%5BVKat%5D%3DMW%20%7C%20%5BVKat%5D%3DMSH%20%7C%20%5BVKat%5D%3DGGH%20%7C%20%5BVKat%5D%3DRH%20%7C%20%5BVKat%5D%3DHAN%20%7C%20%5BVKat%5D%3DWE%20%7C%20%5BVKat%5D%3DEW%20%7C%20%5BVKat%5D%3DMAI%20%7C%20%5BVKat%5D%3DDTW%20%7C%20%5BVKat%5D%3DDGW%20%7C%20%5BVKat%5D%3DGA%20%7C%20%5BVKat%5D%3DGW%20%7C%20%5BVKat%5D%3DUL%20%7C%20%5BVKat%5D%3DBBL%20%7C%20%5BVKat%5D%3DLF%20%7C%20%5BVKat%5D%3DGL%20%7C%20%5BVKat%5D%3DSE%20%7C%20%5BVKat%5D%3DSO%29%20AND%20%5BBL%5D%3D0').getContentText();
var response = "";
var match = feed.match(/subOpen\('.*?'\)/g)
if(match){
for(var i = 0; i < match.length;i++){
var m = match[i].match(/\('(.*)'\)/);
if(m && m.length > 1){
var detailText = UrlFetchApp.fetch(base + '0/'+m[1]);
response += //dosomething with detail text
//and concatenate in the response
}
}
}
return ContentService.createTextOutput(response);
}