Develop Reference

How to download an image with Python 3/Selenium if the URL begins with "blob:"?

When using web.whatsapp.de one can see that the link to a recieved image may look like this:
blob:https://web.whatsapp.com/3565e574-b363-4aca-85cd-2d84aa715c39
If the link is copied in to an address window it will open up the image, however - if "blob" is left out - it will simply open a new web whatsapp window.
I am trying to download the image displayed by this link.
But using common techniques such as using request, or urllib.request or even BeautifulSoup always struggle at one point: The "blob" at the beginning of the url will throw an error.
These answers Download file from Blob URL with Python will trhow either the Error
URLError: <urlopen error unknown url type: blob>
or the Error
InvalidSchema: No connection adapters were found for 'blob:https://web.whatsapp.com/f50eac63-6a7f-48a4-a2b8-8558a9ffe015'
(using BeatufilSoup)
Using a native approach like:
import requests
url = 'https://web.whatsapp.com/f50eac63-6a7f-48a4-a2b8-8558a9ffe015'
fileName = 'test.png'
req = requests.get(url)
file = open(fileName, 'wb')
for chunk in req.iter_content(100000):
file.write(chunk)
file.close()
Will simply result in the same error as using BeautifulSoup.
I am controlling Chrome using Selenium in Python, however I was unable to download the image correctly using the provided link.

A blob is a filelike object of raw data stored by the browser.
You can see them at chrome://blob-internals/
It's possible to get the content of a blob with Selenium with a script injection. However, you'll have to comply to the cross origin policy by running the script on the page/domain that created the blob:
def get_file_content_chrome(driver, uri):
result = driver.execute_async_script("""
var uri = arguments[0];
var callback = arguments[1];
var toBase64 = function(buffer){for(var r,n=new Uint8Array(buffer),t=n.length,a=new Uint8Array(4*Math.ceil(t/3)),i=new Uint8Array(64),o=0,c=0;64>c;++c)i[c]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".charCodeAt(c);for(c=0;t-t%3>c;c+=3,o+=4)r=n[c]<<16|n[c+1]<<8|n[c+2],a[o]=i[r>>18],a[o+1]=i[r>>12&63],a[o+2]=i[r>>6&63],a[o+3]=i[63&r];return t%3===1?(r=n[t-1],a[o]=i[r>>2],a[o+1]=i[r<<4&63],a[o+2]=61,a[o+3]=61):t%3===2&&(r=(n[t-2]<<8)+n[t-1],a[o]=i[r>>10],a[o+1]=i[r>>4&63],a[o+2]=i[r<<2&63],a[o+3]=61),new TextDecoder("ascii").decode(a)};
var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer';
xhr.onload = function(){ callback(toBase64(xhr.response)) };
xhr.onerror = function(){ callback(xhr.status) };
xhr.open('GET', uri);
xhr.send();
""", uri)
if type(result) == int :
raise Exception("Request failed with status %s" % result)
return base64.b64decode(result)
bytes = get_file_content_chrome(driver, "blob:https://developer.mozilla.org/7f9557f4-d8c8-4353-9752-5a49e85058f5")

Blobs are not actual files to be remotely retrieved by a URI. Instead, they are programatically generated psuedo-URLs which are mapped to binary data in order to give the browser something to reference. I.e. there is no attribute of <img> to provide raw data so you instead create a blob address to map that data to the standard src attribute.
From the MDN page linked above:
The only way to read content from a Blob is to use a FileReader. The following code reads the content of a Blob as a typed array.
var reader = new FileReader();
reader.addEventListener("loadend", function() {
// reader.result contains the contents of blob as a typed array
});
reader.readAsArrayBuffer(blob);

For people who are trying to do the same in node and selenium, please refer below.
var script = function (blobUrl) {
console.log(arguments);
var uri = arguments[0];
var callback = arguments[arguments.length - 1];
var toBase64 = function(buffer) {
for(var r,n=new Uint8Array(buffer),t=n.length,a=new Uint8Array(4*Math.ceil(t/3)),i=new Uint8Array(64),o=0,c=0;64>c;++c)
i[c]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".charCodeAt(c);for(c=0;t-t%3>c;c+=3,o+=4)r=n[c]<<16|n[c+1]<<8|n[c+2],a[o]=i[r>>18],a[o+1]=i[r>>12&63],a[o+2]=i[r>>6&63],a[o+3]=i[63&r];return t%3===1?(r=n[t-1],a[o]=i[r>>2],a[o+1]=i[r<<4&63],a[o+2]=61,a[o+3]=61):t%3===2&&(r=(n[t-2]<<8)+n[t-1],a[o]=i[r>>10],a[o+1]=i[r>>4&63],a[o+2]=i[r<<2&63],a[o+3]=61),new TextDecoder("ascii").decode(a)
};
var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer';
xhr.onload = function(){ callback(toBase64(xhr.response)) };
xhr.onerror = function(){ callback(xhr.status) };
xhr.open('GET', uri);
xhr.send();
}
driver.executeAsyncScript(script, imgEleSrc).then((result) => {
console.log(result);
})
For detailed explanation, please refer below link
https://medium.com/#anoop.goudar/how-to-get-data-from-blob-url-to-node-js-server-using-selenium-88b1ad57e36d

Google CSE (Chrome Extension) - Parsing JSON Data with JS

I am attempting to query my Google custom search engine in my Chrome extension, and parse the returned JSON.
My goal is to utilize the current tab URL in my request to the CSE, so I tried to adapt the sample code from the Image Search example that Google provides, since it is somewhat similar.
The code uses an XMLHTTPRequest to communicate with the custom search server:
var x = new XMLHttpRequest();
x.open('GET', searchUrl);
// The Google image search API responds with JSON, so let Chrome parse it.
x.responseType = 'json';
x.onload = function() {
// Parse and process the response from Google Image Search.
var response = x.response;
alert(response);
if (!response || !response.responseData || !response.responseData.results ||
response.responseData.results.length === 0) {
errorCallback('No response from Google Image search!');
return;
}
Whenever I try to view the response var, it is undefined. I tried changing the
var response = x.responseText;, but was unsuccessful.
My question is, what is the correct way to use javascript to parse and process the JSON data, while abiding by the Chrome Extension guidelines?
Any help or guidance would be much appreciated.
Thanks,
Blaine

I suppose you didn't add the permission request to the manifest. Try setting it like here. When it comes to parsing, use JSON.parse(x.response). Also make sure you call x.send() (I can't see it in your code) after the end of onload callback.
Hope it helps

See the error inside your request...

var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function() {
if(xhr.readyState === XMLHttpRequest.DONE) {
if(xhr.status === 200) {
var responseText = JSON.parse(xhr.responseText);
// Do what you want with responseText
}
}
};
x.open('GET', searchUrl);

Getting started with calling APIs via XMLHttpRequest

I've just started to learn how to use APIs, but I'm having a little bit of trouble understanding exactly how to utilize them.
I was able to write the following code thanks to some online documentation, but I have some questions about how to add to it:
var xhr = new XMLHttpRequest();
xhr.open("GET", "http://api.openweathermap.org/data/2.5/weather?q=London&mode=xml", false); xhr.send();
console.log(xhr);
Now, when I run this code in my browser and I open the console, I get a dropdown with a whole bunch of stuff under it. First of all, how can I get the code to display JUST the response? I want the console to display just the XML to show when I run my code. Second of all, how do I parse the XML? Is there any way to get a value from an XML element and assign it to a JavaScript variable?
Thanks!

What you are printing is the XMLHttpRequest object itself, while what you actually want is the response from the request you've made. To get the response, you use
xhr.responseXML;
Which is a object of the type document. (See the MDN docs).
To exhibit the code, you can just
console.log(xhr.responseXML);
But to actually work with the response, you'll probably want to do as Josh suggested, and request from the API a JSON-formatted response (instead of a XML-formatted one).
You will probably want to do that asyncronously, too (see this). The page has a more robust example. Let's adapt it to show London's temperature:
var xhr = new XMLHttpRequest();
xhr.open("GET", "http://api.openweathermap.org/data/2.5/weather?q=London&mode=json", true);
xhr.onload = function (e) {
if (xhr.readyState === 4) {
if (xhr.status === 200) {
console.log(xhr.responseText);
var response = JSON.parse(xhr.responseText);
console.log("Temperature(K): " + response.main.temp);
} else {
console.error(xhr.statusText);
}
}
};
xhr.onerror = function (e) {
console.error(xhr.statusText);
};
xhr.send(null);
Being async means that the xhr.onLoad function will be executed as soon as the response is received. Thus all of your code will be executed sequentially, and only when the response is received the onLoad will be executed.

You can get the response as a Document object.
Using your code above, you can reference xhr.responseXML and use the Document documentation to learn how to grab what data you need.

That dropdown is probably your browser formatting the response object in an interactive way for you.
xmlDoc=xhr.responseXML; will give you the actual resulting text
Alternatively: you could request a JSON object to make working with the data much easier.
var xhr = new XMLHttpRequest();
xhr.open("GET", "http://api.openweathermap.org/data/2.5/weather?q=London&mode=json", false); xhr.send();
var data= JSON.parse(xhr.responseText); //data is now a javascript object full of the API data
Note how the url now reads mode=json now.

Using raw image data from ajax request for data URI

I'm trying to use a combination of Ajax and data URIs to load a JPEG image and extract its EXIF data with a single HTTP request. I am modifying a library (https://github.com/kennydude/photosphere) to do this; currently this library uses two HTTP requests to set the source of the image and to get the EXIF data.
Getting the EXIF works, no problem. However I am having difficulty using the raw data from the ajax request as source for the image.
Source code for a small test of the technique:
<!DOCTYPE html>
<html>
<head>
<script type='text/javascript'>
function init()
{
// own ajax library - using it to request a test jpg image
new Ajax().sendRequest
(
"/images/photos/badger.jpg",
{ method : "GET",
callback: function(xmlHTTP)
{
var encoded = btoa (unescape(encodeURIComponent(xmlHTTP.responseText)));
var dataURL="data:image/jpeg;base64,"+encoded;
document.getElementById("image").src = dataURL;
}
}
);
}
</script>
<script type="text/javascript" src="http://www.free-map.org.uk/0.6/js/lib/Ajax.js"></script>
</head>
<body onload='init()'>
<img id="image" alt="data url loaded image" />
</body>
</html>
I get what looks like sensible jpeg data sent back, and the length (in bytes) of the raw data and the base64-encoded-then-unencoded-again raw data is the same. However the attempt to set the image src fails on both Firefox (25) and Chrome (31) (current versions) - chrome displays "broken image" icon suggesting the src is an invalid format.
I used this mozilla page for info on base64 encoding/decoding:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Base64_encoding_and_decoding
Any idea what might be wrong? Looking around I can create the base64 encoded image server side but can it be done client side like this? For one thing, base64 encoding server side obviously increases the data size and the whole purpose of this exercise is to cut down the amount of data being transferred from the server, as well as the number of requests.
Thanks,
Nick

Thanks for that. I've done a bit more digging on this and it turns out there is a solution at least on current versions of Firefox and Chrome (EDIT: IE10 works too). You can use XMLHttpRequest2 and use a typed array (Uint8Array). The following code works:
<!DOCTYPE html>
<html>
<head>
<script type='text/javascript'>
function init()
{
var xmlHTTP = new XMLHttpRequest();
xmlHTTP.open('GET','/images/photos/badger.jpg',true);
// Must include this line - specifies the response type we want
xmlHTTP.responseType = 'arraybuffer';
xmlHTTP.onload = function(e)
{
var arr = new Uint8Array(this.response);
// Convert the int array to a binary string
// We have to use apply() as we are converting an *array*
// and String.fromCharCode() takes one or more single values, not
// an array.
var raw = String.fromCharCode.apply(null,arr);
// This works!!!
var b64=btoa(raw);
var dataURL="data:image/jpeg;base64,"+b64;
document.getElementById("image").src = dataURL;
};
xmlHTTP.send();
}
</script>
</head>
<body onload='init()'>
<img id="image" alt="data url loaded image" />
</body>
</html>
Basically you ask for a binary response, then create an 8-bit unsigned int view of the data before converting it back into a (binary-friendly) string String.fromCharCode(). The apply is necessary as String.fromCharCode() does not accept an array argument. You then use btoa(), create your data url and it then works.
The following resources were useful for this:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays?redirectlocale=en-US&redirectslug=JavaScript%2FTyped_arrays
and
http://www.html5rocks.com/en/tutorials/file/xhr2/
Nick

Nick's answer works very well. But when I did this with a fairly large file, I got a stack overflow on
var raw = String.fromCharCode.apply(null,arr);
Generating the raw string in chunks worked well for me.
var raw = '';
var i,j,subArray,chunk = 5000;
for (i=0,j=arr.length; i<j; i+=chunk) {
subArray = arr.subarray(i,i+chunk);
raw += String.fromCharCode.apply(null, subArray);
}

I had trouble with the ArrayBuffer -> String -> Base64 method described above, but ran across another method using Blob that worked great. It's not a way to convert raw data to Base 64 (as in the title), but it is a way to display raw image data (as in the actual question):
var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer';
xhr.onload = function() {
var blb = new Blob([xhr.response], {type: 'image/png'});
var url = (window.URL || window.webkitURL).createObjectURL(blb);
image.src = url;
}
xhr.open('GET', 'http://whatever.com/wherever');
xhr.send();
All credit goes to Jan Miksovsky, author of this fiddle. I just stumbled across it and thought it'd make a useful addition to this discussion.

Modern ES6 powered solution for image downloading: (without specifying image type)
async function downloadImageFromUrl(url) { // returns dataURL
const xmlHTTP = new XMLHttpRequest();
xmlHTTP.open('GET', url, true);
xmlHTTP.responseType = 'blob';
const imageBlob = await new Promise((resolve, reject) => {
xmlHTTP.onload = e => xmlHTTP.status >= 200 && xmlHTTP.status < 300 && xmlHTTP.response.type.startsWith('image/') ? resolve(xmlHTTP.response) : reject(Error(`wrong status or type: ${xmlHTTP.status}/${xmlHTTP.response.type}`));
xmlHTTP.onerror = reject;
xmlHTTP.send();
});
return blobToDataUrl(imageBlob);
}
function blobToDataUrl(blob) { return new Promise(resolve => {
const reader = new FileReader(); // https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications
reader.onload = e => resolve(e.target.result);
reader.readAsDataURL(blob);
})}
Usage:
downloadImageFromUrl('https://a.b/img.png').then(console.log, console.error)

I've been working for two days on this issue since I needed a solution to render the User's Outlook Profile Picture from the raw data received from Microsoft Graft. I have implemented all the solutions above, with no success. Then I found this git:
get base64 raw data of image from responseBody using jquery ajax
In my case, I just replaced "data:image/png;base64," with "data:image/jpg;base64,"
It works like a charm.

You will have to do base64 encoding on the server side as the responseText is treated as a String, and the response data that the server is sending is binary.

Parse json from an external site with JavaScript

I am trying to parse some json on an external site but I am having trouble. It must be with JavaScript or JQuery, as it is for a chrome extension. To get to the point:
I need to get the number from an external URL with the json {"_visitor_alertsUnread":"0"} and set the number returned to a variable. How do I go about doing this?
I have tried several things such as JSON.parse but it isn't working:(
In short: How do I get the number from this json, which is on an external site, and set it to a variable?

You cannot get data from an external URL (in a different domain) in Javascript unless the site supports JSONP or Cross-Origin Resource Sharing. If it does, then use XMLHttpRequest to get the data and JSON.parse() to read it.
Script:
var xhr = new XMLHttpRequest();
xhr.open( 'GET', 'example.com/json', true );
xhr.onload = function () {
var unread = window.JSON.parse( xhr.responseText )._visitor_alertsUnread;
};
xhr.onerror = function () {
//process error
};
xhr.send();

Try this with http://api.jquery.com/jQuery.getJSON/
$.getJSON('your_url', function (jsonobj) {
var unread;
unread = jsonobj._visitor_alertsUnread;
});