Develop Reference

How to download an image with Python 3/Selenium if the URL begins with "blob:"?

When using web.whatsapp.de one can see that the link to a recieved image may look like this:
blob:https://web.whatsapp.com/3565e574-b363-4aca-85cd-2d84aa715c39
If the link is copied in to an address window it will open up the image, however - if "blob" is left out - it will simply open a new web whatsapp window.
I am trying to download the image displayed by this link.
But using common techniques such as using request, or urllib.request or even BeautifulSoup always struggle at one point: The "blob" at the beginning of the url will throw an error.
These answers Download file from Blob URL with Python will trhow either the Error
URLError: <urlopen error unknown url type: blob>
or the Error
InvalidSchema: No connection adapters were found for 'blob:https://web.whatsapp.com/f50eac63-6a7f-48a4-a2b8-8558a9ffe015'
(using BeatufilSoup)
Using a native approach like:
import requests
url = 'https://web.whatsapp.com/f50eac63-6a7f-48a4-a2b8-8558a9ffe015'
fileName = 'test.png'
req = requests.get(url)
file = open(fileName, 'wb')
for chunk in req.iter_content(100000):
file.write(chunk)
file.close()
Will simply result in the same error as using BeautifulSoup.
I am controlling Chrome using Selenium in Python, however I was unable to download the image correctly using the provided link.

A blob is a filelike object of raw data stored by the browser.
You can see them at chrome://blob-internals/
It's possible to get the content of a blob with Selenium with a script injection. However, you'll have to comply to the cross origin policy by running the script on the page/domain that created the blob:
def get_file_content_chrome(driver, uri):
result = driver.execute_async_script("""
var uri = arguments[0];
var callback = arguments[1];
var toBase64 = function(buffer){for(var r,n=new Uint8Array(buffer),t=n.length,a=new Uint8Array(4*Math.ceil(t/3)),i=new Uint8Array(64),o=0,c=0;64>c;++c)i[c]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".charCodeAt(c);for(c=0;t-t%3>c;c+=3,o+=4)r=n[c]<<16|n[c+1]<<8|n[c+2],a[o]=i[r>>18],a[o+1]=i[r>>12&63],a[o+2]=i[r>>6&63],a[o+3]=i[63&r];return t%3===1?(r=n[t-1],a[o]=i[r>>2],a[o+1]=i[r<<4&63],a[o+2]=61,a[o+3]=61):t%3===2&&(r=(n[t-2]<<8)+n[t-1],a[o]=i[r>>10],a[o+1]=i[r>>4&63],a[o+2]=i[r<<2&63],a[o+3]=61),new TextDecoder("ascii").decode(a)};
var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer';
xhr.onload = function(){ callback(toBase64(xhr.response)) };
xhr.onerror = function(){ callback(xhr.status) };
xhr.open('GET', uri);
xhr.send();
""", uri)
if type(result) == int :
raise Exception("Request failed with status %s" % result)
return base64.b64decode(result)
bytes = get_file_content_chrome(driver, "blob:https://developer.mozilla.org/7f9557f4-d8c8-4353-9752-5a49e85058f5")

Blobs are not actual files to be remotely retrieved by a URI. Instead, they are programatically generated psuedo-URLs which are mapped to binary data in order to give the browser something to reference. I.e. there is no attribute of <img> to provide raw data so you instead create a blob address to map that data to the standard src attribute.
From the MDN page linked above:
The only way to read content from a Blob is to use a FileReader. The following code reads the content of a Blob as a typed array.
var reader = new FileReader();
reader.addEventListener("loadend", function() {
// reader.result contains the contents of blob as a typed array
});
reader.readAsArrayBuffer(blob);

For people who are trying to do the same in node and selenium, please refer below.
var script = function (blobUrl) {
console.log(arguments);
var uri = arguments[0];
var callback = arguments[arguments.length - 1];
var toBase64 = function(buffer) {
for(var r,n=new Uint8Array(buffer),t=n.length,a=new Uint8Array(4*Math.ceil(t/3)),i=new Uint8Array(64),o=0,c=0;64>c;++c)
i[c]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".charCodeAt(c);for(c=0;t-t%3>c;c+=3,o+=4)r=n[c]<<16|n[c+1]<<8|n[c+2],a[o]=i[r>>18],a[o+1]=i[r>>12&63],a[o+2]=i[r>>6&63],a[o+3]=i[63&r];return t%3===1?(r=n[t-1],a[o]=i[r>>2],a[o+1]=i[r<<4&63],a[o+2]=61,a[o+3]=61):t%3===2&&(r=(n[t-2]<<8)+n[t-1],a[o]=i[r>>10],a[o+1]=i[r>>4&63],a[o+2]=i[r<<2&63],a[o+3]=61),new TextDecoder("ascii").decode(a)
};
var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer';
xhr.onload = function(){ callback(toBase64(xhr.response)) };
xhr.onerror = function(){ callback(xhr.status) };
xhr.open('GET', uri);
xhr.send();
}
driver.executeAsyncScript(script, imgEleSrc).then((result) => {
console.log(result);
})
For detailed explanation, please refer below link
https://medium.com/#anoop.goudar/how-to-get-data-from-blob-url-to-node-js-server-using-selenium-88b1ad57e36d

Get Binary data using XMLHttpRequest without redirecting page

I'm trying to load a binary array using XMLHttpRequest. It downloads fine (The console.log is printed), but then Chrome immediately tries to redirect the page to the file itself, which is not very helpful...
let data;
var location = '/static/data/dinar.r4';
var oReq = new XMLHttpRequest();
oReq.open("GET", location, true);
oReq.responseType = 'arraybuffer';
oReq.onload = function (oEvent) {
var arrayBuffer = oReq.response; // Note: not oReq.responseText
if (arrayBuffer) {
console.log(`Successfully loaded data ${location}`);
// data = new Float32Array(arrayBuffer);
}else{
throw new Error(`Could not load ${location}`);
}
};
oReq.send(null);
Am I missing something obvious?

I just came to the conclusion that my issue has nothing to do with the XMLHttpRequest, and rather is because of the presence of the line
var location = ...
This sets the window location, rather than creating a new variable called location...

Upload an image to server using chrome extensions

I am doing a chrome extension capable of getting from a webpage an image, and after I got it, I'm trying to upload it to an intranet server automatically without user iteration.
I am doing this right now.
This is on Content_script.js
...
x = $(frame1).contents().find("#image");
chrome.extension.sendRequest(x[0].src);
...
This is on background.js
chrome.extension.onRequest.addListener(function(links) {
chrome.downloads.download( { url: links ,
conflictAction: "overwrite",
filename: "get_image.jpg" },
function(DescargaId) {
var formData = new FormData();
formData.append("doc", Blob, "~/Downloads/get_image.jpg");
var request = new XMLHttpRequest();
request.open("POST", "http://192.168.0.30/app_get_pictures/upload_img.php");
request.setRequestHeader("Content-Type", "multipart/form-data");
request.send(formData);
} );
This on upload_img.php
...
$uploaddir = $_SERVER['DOCUMENT_ROOT'].'/app_get_pictures/images/';
$uploadfile = $uploaddir . basename($_FILES['doc']['name']);
move_uploaded_file($_FILES['doc']['tmp_name'], $uploadfile);
...
With this, I already download the image successfully to the local machine, but can't upload the image to the server.
It is possible to do this, or even if I can upload the image to the server directly without download it first to the local machine.
Note: I don't have any tag form on a popup page in the extension solution, and I don't have a popup page neither, because as I already said, I don't need any iteration from the user.
Thanks for your help!

Thanks to https://stackoverflow.com/users/934239/xan I resolved this problem using his advise, here is the resulting working code.
...
// With this I can download or get content image into var blob
var xhr = new XMLHttpRequest();
var kima = $(frame1).contents().find("#image");
xhr.open('GET',kima[0].src,true);
xhr.responseType = 'blob';
xhr.onload = function(e) {
if (this.status == 200) {
var blob = new Blob([this.response], {type: 'image/png'});
send_image(blob);
}
};
xhr.send();
....
// After the image is loaded into var blob, it can be send
// to the server side
function send_image(x){
var formData = new FormData();
formData.append("doc", x);
var request = new XMLHttpRequest();
request.open("POST", "http://192.168.0.30/app_get_image/upload_img.php");
request.send(formData);
}
All this code into the content_script of the chrome extension. Also the code of the background using API download isn't needed anymore.
Hope this could works for anybody else.
Thanks again.

Besides the fact that the callback of downloads.download does NOT indicate that the file is already downloaded (only that the download is queued)..
formData.append("doc", Blob, "~/Downloads/get_image.jpg");
What do you think this code does? Documentation, for reference.
The second parameter is supposed to hold the data of the file; the third parameter is just the file name for the purposes of naming anonymous data (e.g. in a Blob)
Instead, you pass the Blob object itself; not an instance of Blob with the data.
In fact, with this architecture, you won't be able to upload the file, since at no point does chrome.downloads API give you access to the file's contents, and you can't just access a file on a disk by filename (which is what I think you thought this code would do).
To actually access the data, you need to request it yourself with XHR (or Fetch API if you want to be "modern"). Then, you get the response object which you can request to be a Blob. Then, you can both upload the blob and invoke chrome.downloads together with createObjectURL to "download" it from your extension's memory.

Download file using Javascript from a bytearray

I have a button in my asp net app that when pressed I am calling a rest service that returns a bytearray.
This bytearray is actually a file that I need to start downloading in the browser when the button is pressed. How can I achieve this?
I am thinking along the lines of writing the bytearray to the response and setting the headers.
Is this thinking correct and does someone have some code samples?
---------Update on 3/25----------------
Thanks Justin but not yet what I need. Please look at this link it will return a file for download. What I need to do is have an event client side that will get this file for download without redirecting to this page. It has to be downloaded from my page and not this link.
http://ops.epo.org/3.0/rest-services/published-data/images/US/5000001/PA/firstpage.pdf?Range=1
If you check it out with Fiddler, you will see how the pdf is received as binary.

You can set the responseType as arraybuffer and access it that way. There is even a way to stream the data using onprogress events. JavaScript has come a long way.
var oReq = new XMLHttpRequest();
oReq.open("GET", "/myfile.png", true);
oReq.responseType = "arraybuffer";
oReq.onload = function (oEvent) {
var arrayBuffer = oReq.response; // Note: not oReq.responseText
if (arrayBuffer) {
var byteArray = new Uint8Array(arrayBuffer);
for (var i = 0; i < byteArray.byteLength; i++) {
// do something with each byte in the array
}
}
};
oReq.send(null);
https://developer.mozilla.org/en-US/docs/DOM/XMLHttpRequest/Sending_and_Receiving_Binary_Data
If you mean that the webservice isn't returning binary data, and instead is JSON data like: [0,1,3,4] then that is a different issue.

As far as i know, the closest thing you can do with javascript is an ajax request that has the server parse the data as text

Ajax Binary Response

Hi I'm wondering if there's anyway to stream a binary response in AJAX? This would be an ultimate solution otherwise I would need to realize the binary image to a file then stream that file to the user with a different URL.
new Ajax.Request('/viewImage?id=123', {
// request returns a binary image inputstream
onSuccess: function(transport) {
// text example
// alert(transport.responseText)
// QUESTION: is there a streaming binary response?
$('imgElem').src = transport.responseBinary;
},
onFailure: function(transport) {
// handle failure
}
});

It might not be possible to stream binary data, but you can use Ajax to retrieve binary data.
This is possible using one of two methods: Javascript Typed Arrays or an XMLHttpResponse overrideMimeType hack. Have a read of a good article on MDN – these examples are taken from there: Sending and Receiving Binary Data
The Typed Array method looks like this:
var oReq = new XMLHttpRequest();
oReq.open("GET", "/myfile.png", true);
oReq.responseType = "arraybuffer";
oReq.onload = function (oEvent) {
var arrayBuffer = oReq.response; // Note: not oReq.responseText
if (arrayBuffer) {
var byteArray = new Uint8Array(arrayBuffer);
for (var i = 0; i < byteArray.byteLength; i++) {
// do something with each byte in the array
}
}
};
oReq.send(null);
Typed Arrays are not supported in IE < 10, Firefox < 4, Chrome < 7, Safari < 5.1 and Opera < 11.6, and mobile support is shaky but improving.
The second method uses an XMLHttpRequest method called overrideMimeType to allow the binary data to be passed through unmodified.
var req = new XMLHttpRequest();
req.open('GET', '/myfile.png', false);
// XHR binary charset opt by Marcus Granado 2006 [http://mgran.blogspot.com]
req.overrideMimeType('text\/plain; charset=x-user-defined');
req.send(null);
if (req.status != 200) return '';
// do stuff with req.responseText;
You get an unparsed binary string, upon which you can use var byte = filestream.charCodeAt(x) & 0xff; to retrieve a specific byte.

This is an extension to Tom Ashworth's response (which helped to put me on the right track with the issue I was facing). This allows you to just get the filestream (FileStreamResult if you are using asp.net mvc) and set it to the img src, which is cool.
var oReq = new XMLHttpRequest();
oReq.open("post", '/somelocation/getmypic', true );
oReq.responseType = "blob";
oReq.onload = function ( oEvent )
{
var blob = oReq.response;
var imgSrc = URL.createObjectURL( blob );
var $img = $( '<img/>', {
"alt": "test image",
"src": imgSrc
} ).appendTo( $( '#bb_theImageContainer' ) );
window.URL.revokeObjectURL( imgSrc );
};
oReq.send( null );
The basic idea is that the data is returned untampered with, it is placed in a blob and then a url is created to that object in memory. See here and here. Note supported browsers.

What you can do, if you are trying to generate an image on the fly, is to just do:
<img src="http://myurl/myfile.php?id=3" />
then you can send the data with the appropriate mimetype.
If you really want to send an image, then you may want to look at the HTML5 canvas tag, but I am not certain how excanvas would work with this, for being cross-platform.
You could write to the canvas, but it would be more efficient to just use the img tag.

When you call your service, you should ask for a dataType: 'binary' response. Then, you can use saveAs(FileSaver.js) to trigger the download or createObjectURL to open in new window.
But, $.ajax doesn't let you download binary content out of the box, it will try to decode your binary from UTF-8 and corrupt it. Either use a jQuery plugin to solve this problem jquery.binarytransport.js
exemplo:
$.ajax({
type: "POST",
url: $("form#data").attr("action"),
data: formData,
dataType: 'binary', //--> using jquery.binarytransport.js
success: function (response) {
// Default response type is blob
saveAs(response, "test.pdf"); //--> using FileSaver.js
let fileURL = URL.createObjectURL(blob);
window.open(fileURL); // open file in new window
}
});
Good look! :)

You can send any response you want, being it plain text, HTML, an image... whatever! It's up to you how to handle it when you receive it.
But... you cannot assign a binary image to the <IMG> src attribute. You'd better just return the URL to the image and assign that instead - well, to be honest, there are some encodings to embed images in the SRC, but they are not cross-browser so you'll want to stay away from them.

I managed to get download of binary file without corrupted data working using jQuery ajax by adding:
xhrFields: {responseType: 'blob'}