Develop Reference

How to download an image with Python 3/Selenium if the URL begins with "blob:"?

When using web.whatsapp.de one can see that the link to a recieved image may look like this:
blob:https://web.whatsapp.com/3565e574-b363-4aca-85cd-2d84aa715c39
If the link is copied in to an address window it will open up the image, however - if "blob" is left out - it will simply open a new web whatsapp window.
I am trying to download the image displayed by this link.
But using common techniques such as using request, or urllib.request or even BeautifulSoup always struggle at one point: The "blob" at the beginning of the url will throw an error.
These answers Download file from Blob URL with Python will trhow either the Error
URLError: <urlopen error unknown url type: blob>
or the Error
InvalidSchema: No connection adapters were found for 'blob:https://web.whatsapp.com/f50eac63-6a7f-48a4-a2b8-8558a9ffe015'
(using BeatufilSoup)
Using a native approach like:
import requests
url = 'https://web.whatsapp.com/f50eac63-6a7f-48a4-a2b8-8558a9ffe015'
fileName = 'test.png'
req = requests.get(url)
file = open(fileName, 'wb')
for chunk in req.iter_content(100000):
file.write(chunk)
file.close()
Will simply result in the same error as using BeautifulSoup.
I am controlling Chrome using Selenium in Python, however I was unable to download the image correctly using the provided link.

A blob is a filelike object of raw data stored by the browser.
You can see them at chrome://blob-internals/
It's possible to get the content of a blob with Selenium with a script injection. However, you'll have to comply to the cross origin policy by running the script on the page/domain that created the blob:
def get_file_content_chrome(driver, uri):
result = driver.execute_async_script("""
var uri = arguments[0];
var callback = arguments[1];
var toBase64 = function(buffer){for(var r,n=new Uint8Array(buffer),t=n.length,a=new Uint8Array(4*Math.ceil(t/3)),i=new Uint8Array(64),o=0,c=0;64>c;++c)i[c]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".charCodeAt(c);for(c=0;t-t%3>c;c+=3,o+=4)r=n[c]<<16|n[c+1]<<8|n[c+2],a[o]=i[r>>18],a[o+1]=i[r>>12&63],a[o+2]=i[r>>6&63],a[o+3]=i[63&r];return t%3===1?(r=n[t-1],a[o]=i[r>>2],a[o+1]=i[r<<4&63],a[o+2]=61,a[o+3]=61):t%3===2&&(r=(n[t-2]<<8)+n[t-1],a[o]=i[r>>10],a[o+1]=i[r>>4&63],a[o+2]=i[r<<2&63],a[o+3]=61),new TextDecoder("ascii").decode(a)};
var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer';
xhr.onload = function(){ callback(toBase64(xhr.response)) };
xhr.onerror = function(){ callback(xhr.status) };
xhr.open('GET', uri);
xhr.send();
""", uri)
if type(result) == int :
raise Exception("Request failed with status %s" % result)
return base64.b64decode(result)
bytes = get_file_content_chrome(driver, "blob:https://developer.mozilla.org/7f9557f4-d8c8-4353-9752-5a49e85058f5")

Blobs are not actual files to be remotely retrieved by a URI. Instead, they are programatically generated psuedo-URLs which are mapped to binary data in order to give the browser something to reference. I.e. there is no attribute of <img> to provide raw data so you instead create a blob address to map that data to the standard src attribute.
From the MDN page linked above:
The only way to read content from a Blob is to use a FileReader. The following code reads the content of a Blob as a typed array.
var reader = new FileReader();
reader.addEventListener("loadend", function() {
// reader.result contains the contents of blob as a typed array
});
reader.readAsArrayBuffer(blob);

For people who are trying to do the same in node and selenium, please refer below.
var script = function (blobUrl) {
console.log(arguments);
var uri = arguments[0];
var callback = arguments[arguments.length - 1];
var toBase64 = function(buffer) {
for(var r,n=new Uint8Array(buffer),t=n.length,a=new Uint8Array(4*Math.ceil(t/3)),i=new Uint8Array(64),o=0,c=0;64>c;++c)
i[c]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".charCodeAt(c);for(c=0;t-t%3>c;c+=3,o+=4)r=n[c]<<16|n[c+1]<<8|n[c+2],a[o]=i[r>>18],a[o+1]=i[r>>12&63],a[o+2]=i[r>>6&63],a[o+3]=i[63&r];return t%3===1?(r=n[t-1],a[o]=i[r>>2],a[o+1]=i[r<<4&63],a[o+2]=61,a[o+3]=61):t%3===2&&(r=(n[t-2]<<8)+n[t-1],a[o]=i[r>>10],a[o+1]=i[r>>4&63],a[o+2]=i[r<<2&63],a[o+3]=61),new TextDecoder("ascii").decode(a)
};
var xhr = new XMLHttpRequest();
xhr.responseType = 'arraybuffer';
xhr.onload = function(){ callback(toBase64(xhr.response)) };
xhr.onerror = function(){ callback(xhr.status) };
xhr.open('GET', uri);
xhr.send();
}
driver.executeAsyncScript(script, imgEleSrc).then((result) => {
console.log(result);
})
For detailed explanation, please refer below link
https://medium.com/#anoop.goudar/how-to-get-data-from-blob-url-to-node-js-server-using-selenium-88b1ad57e36d

Upload an image to server using chrome extensions

I am doing a chrome extension capable of getting from a webpage an image, and after I got it, I'm trying to upload it to an intranet server automatically without user iteration.
I am doing this right now.
This is on Content_script.js
...
x = $(frame1).contents().find("#image");
chrome.extension.sendRequest(x[0].src);
...
This is on background.js
chrome.extension.onRequest.addListener(function(links) {
chrome.downloads.download( { url: links ,
conflictAction: "overwrite",
filename: "get_image.jpg" },
function(DescargaId) {
var formData = new FormData();
formData.append("doc", Blob, "~/Downloads/get_image.jpg");
var request = new XMLHttpRequest();
request.open("POST", "http://192.168.0.30/app_get_pictures/upload_img.php");
request.setRequestHeader("Content-Type", "multipart/form-data");
request.send(formData);
} );
This on upload_img.php
...
$uploaddir = $_SERVER['DOCUMENT_ROOT'].'/app_get_pictures/images/';
$uploadfile = $uploaddir . basename($_FILES['doc']['name']);
move_uploaded_file($_FILES['doc']['tmp_name'], $uploadfile);
...
With this, I already download the image successfully to the local machine, but can't upload the image to the server.
It is possible to do this, or even if I can upload the image to the server directly without download it first to the local machine.
Note: I don't have any tag form on a popup page in the extension solution, and I don't have a popup page neither, because as I already said, I don't need any iteration from the user.
Thanks for your help!

Thanks to https://stackoverflow.com/users/934239/xan I resolved this problem using his advise, here is the resulting working code.
...
// With this I can download or get content image into var blob
var xhr = new XMLHttpRequest();
var kima = $(frame1).contents().find("#image");
xhr.open('GET',kima[0].src,true);
xhr.responseType = 'blob';
xhr.onload = function(e) {
if (this.status == 200) {
var blob = new Blob([this.response], {type: 'image/png'});
send_image(blob);
}
};
xhr.send();
....
// After the image is loaded into var blob, it can be send
// to the server side
function send_image(x){
var formData = new FormData();
formData.append("doc", x);
var request = new XMLHttpRequest();
request.open("POST", "http://192.168.0.30/app_get_image/upload_img.php");
request.send(formData);
}
All this code into the content_script of the chrome extension. Also the code of the background using API download isn't needed anymore.
Hope this could works for anybody else.
Thanks again.

Besides the fact that the callback of downloads.download does NOT indicate that the file is already downloaded (only that the download is queued)..
formData.append("doc", Blob, "~/Downloads/get_image.jpg");
What do you think this code does? Documentation, for reference.
The second parameter is supposed to hold the data of the file; the third parameter is just the file name for the purposes of naming anonymous data (e.g. in a Blob)
Instead, you pass the Blob object itself; not an instance of Blob with the data.
In fact, with this architecture, you won't be able to upload the file, since at no point does chrome.downloads API give you access to the file's contents, and you can't just access a file on a disk by filename (which is what I think you thought this code would do).
To actually access the data, you need to request it yourself with XHR (or Fetch API if you want to be "modern"). Then, you get the response object which you can request to be a Blob. Then, you can both upload the blob and invoke chrome.downloads together with createObjectURL to "download" it from your extension's memory.

Large blob file in Javascript

I have an XHR object that downloads 1GB file.
function getFile(callback)
{
var xhr = new XMLHttpRequest();
xhr.onload = function () {
if (xhr.status == 200) {
callback.apply(xhr);
}else{
console.log("Request error: " + xhr.statusText);
}
};
xhr.open('GET', 'download', true);
xhr.onprogress = updateProgress;
xhr.responseType = "arraybuffer";
xhr.send();
}
But the File API can't load all that into memory even from a worker
it throws out of memory...
btn.addEventListener('click', function() {
getFile(function() {
var worker = new Worker("js/saving.worker.js");
worker.onmessage = function(e) {
saveAs(e.data); // FileSaver.js it creates URL from blob... but its too large
};
worker.postMessage(this.response);
});
});
Web Worker
onmessage = function (e) {
var view = new DataView(e.data, 0);
var file = new File([view], 'file.zip', {type: "application/zip"});
postMessage('file');
};
I'm not trying to compress the file, this file is already compressed from server.
I thought storing it first on indexedDB but i i'll have to load blob or file anyway, even if i do request by range bytes, soon or late i will have to build this giant blob..
I want to create blob: url and send it to user after been downloaded by browser
I'll use FileSystem API for Google Chrome, but i want make something for firefox, i looked into File Handle Api but nothing...
Do i have to build an extension for firefox, in order to do the same thing as FileSystem does for google chrome?
Ubuntu 32 bits

Loading 1gb+ with ajax isn't convenient just for monitoring download progress and filling up the memory.
Instead I would just send the file with a Content-Disposition header to save the file.
There are however ways to go around it to monitor the progress. Option one is to have a second websocket that signals how much you have downloaded while you are downloading normally with a get request. the other option will be described later in the bottom
I know you talked about using Blinks sandboxed filesystem in the conversation. but it has some drawbacks. It may need permission if using persistent storage. It only allows 20% of the available disk that are left. And if chrome needs to free some space then it will throw away any others domains temporary storage that was last used for the most recent file. Beside it doesn't work in private mode.
Not to mention that it has been dropping support for it and may never end up in other browsers - but they will most likely not remove it since many sites still depend on it
The only way to process this large file is with streams. That is why I have created a StreamSaver. This is only going to work in Blink (chrome & opera) ATM but it will eventually be supported by other browsers with the whatwg spec to back it up as a standard.
fetch(url).then(res => {
// One idea is to get the filename from Content-Disposition header...
const size = ~~res.headers.get('Content-Length')
const fileStream = streamSaver.createWriteStream('filename.zip', size)
const writeStream = fileStream.getWriter()
// Later you will be able to just simply do
// res.body.pipeTo(fileStream)
// instead of pumping
const reader = res.body.getReader()
const pump = () => reader.read()
.then(({ value, done }) => {
// here you know how large the value (chunk) is and you can
// figure out the download speed/progress when comparing it to the size
return done
? writeStream.close()
: writeStream.write(value).then(pump)
)
// Start the reader
pump().then(() =>
console.log('Closed the stream, Done writing')
)
})
This will not take up any memory

I have a theory that is if you split the file into chunks and store them in the indexedDB and then later merge them together it will work
A blob isn't made of data... it's more like pointers to where a file can be read from
Meaning if you store them in indexedDB and then do something like this (using FileSaver or alternative)
finalBlob = new Blob([blob_A_fromDB, blob_B_fromDB])
saveAs(finalBlob, 'filename.zip')
But i can't confirm this since i haven't tested it, would be good if someone else could

Blob is cool until you want to download a large file, there is a 600MB limit(chrome) for blob since it stores everything in memory.

Access JSON property value before loading the contents of the file

I have an angularjs project which retrieves JSON files from a server and uses the contents to display the data in the screen.
I'm using a service to load the data, and this service calls the server for a new JSON file every 2 seconds (I removed that from the code below for simplicity).
var data = $resource(:file.json', {}, {
query: {method: 'GET', params: {file: '#file'}}
});
this.load = function(file, myFunction) {
data.query({file:file}, function(data) {
myFunction(data);
}
}
Now, these files can be really big and sometimes there's no need to process the file because there are no changes from the previous one received. I have a property in the JSON file with the version number, and I should not process the file unless that version number is higher than the one in the previous file.
I can do that by calling the query service, which loads the file contents into a js object and then check the version, if the file is really big it might take a while to load it. Is there a way to access that property value (version) ONLY and then, depending on it, load the file into a js object?
EDIT: The thing that I'm guessing is that loading a 1MB JSON file to check a version number inside it might take a while (or maybe no and that $resource action is really fast, anyone knows?), but I'm not really sure that it can be done any other way, as I'm checking a specific property inside the file.
Many thanks in advance.

HTML5 and Javascript now provides a File API which can be used to read the file line by line. You can find information regarding this feature here:
http://www.html5rocks.com/en/tutorials/file/dndfiles/
This will slice the full file into string and take just the first line(asuming the version is in there)
data.substr(0, data.indexOf("\n"));
--
Bonus:
Also in this answer you will find out how to read the first line of a file:
https://stackoverflow.com/a/12227851/2552259
var XHR = new XMLHttpRequest();
XHR.open("GET", "http://hunpony.hu/today/changelog-en.txt", true);
XHR.send();
XHR.onload = function (){
console.log( XHR.responseText.slice(0, XHR.responseText.indexOf("\n")) );
};
Another question with the same topic:
https://stackoverflow.com/a/6861246/2552259
var txtFile = new XMLHttpRequest();
txtFile.open("GET", "http://website.com/file.txt", true);
txtFile.onreadystatechange = function()
{
if (txtFile.readyState === 4) { // document is ready to parse.
if (txtFile.status === 200) { // file is found
allText = txtFile.responseText;
lines = txtFile.responseText.split("\n");
}
}
}
txtFile.send(null);

Do you have access to the json files?
I'm not sure how you generate your json files but you could try adding the version number in the filename and check if a newer filename exists. I have not tested this but maybe it's worth a try.

How to load a PDF into a blob so it can be uploaded?

I'm working on a testing framework that needs to pass files to the drop listener of a PLUpload instance. I need to create blob objects to pass inside a Data Transfer Object of the sort generated on a Drag / Drop event. I have it working fine for text files and image files. I would like to add support for PDF's, but it seems that I can't get the encoding right after retrieving the response. The response is coming back as text because I'm using Sahi to retrieve it in order to avoid Cross-Domain issues.
In short: the string I'm receiving is UTF-8 encoded and therefore the content looks like you opened a PDF with a text editor. I am wondering how to convert this back into the necessary format to create a blob, so that after the document gets uploaded everything looks okay.
What steps do I need to go through to convert the UTF-8 string into the proper blob object? (Yes, I am aware I could submit an XHR request and change the responseType property and (maybe) get closer, however due to complications with the way Sahi operates I'm not going to explain here why I would prefer not to go this route).
Also, I'm not familiar enough but I have a hunch maybe I lose data by retrieving it as a string? If that's the case I'll find another approach.
The existing code and the most recent approach I have tried is here:
var data = '%PDF-1.7%����115 0 obj<</Linearized 1/L ...'
var arr = [];
var utf8 = unescape(encodeURIComponent(data));
for (var i = 0; i < utf8.length; i++) {
arr.push(utf8.charCodeAt(i));
}
var file = new Blob(arr, {type: 'application/pdf'});

It looks like you were close. I just did this for a site which needed to read a PDF from another website and drop it into a fileuploader plugin. Here is what worked for me:
var url = "http://some-websites.com/Pdf/";
//You may not need this part if you have the PDF data locally already
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function () {
if (this.readyState == 4 && this.status == 200) {
//console.log(this.response, typeof this.response);
//now convert your Blob from the response into a File and give it a name
var fileOfBlob = new File([this.response], 'your_file.pdf');
// Now do something with the File
// for filuploader (blueimp), just use the add method
$('#fileupload').fileupload('add', {
files: [ fileOfBlob ],
fileInput: $(this)
});
}
}
xhr.open('GET', url);
xhr.responseType = 'blob';
xhr.send();
I found help on the XHR as blob here. Then this SO answer helped me with naming the File. You might be able to use the Blob by itself, but you won't be able to give it a name unless its passed into a File.