Develop Reference

Using selenium to download a file via window.open

I'm trying to scrape a webpage where clicking a link results in a new window popping open that immediately downloads a csv. I haven't been able to figure out the format of the url since it's fairly dense javascript (and one function is called via the onClick property while another is called as part of the href property. I have not worked with Selenium before, so I was hoping to confirm before getting started that what I want to do is possible. I had read somewhere that downloading files via new popup windows is not necessarily something I can do with Selenium.
Any advice would be greatly appreciated. A this is possible would be very helpful as would as here's how you'd do it even sketched in broad detail. Thanks much!
To be clear, my difficulties primarily stem from the fact that I can't figure out how the URL to download the file is generated. Even looking at the Google chrome network calls, I am not seeing where it is, and it would probably take me many hours to track this down, so I am looking for a solution that relies on clicking specific text in the browser rather than disentangling the cumbersome machinery behind the scenes.

Here's how I download files using Firefox webdriver. It's essentially creating a browser profile so that the default download location for certain file types are set. You can then verify if the file exists at that location.
import os
from selenium import webdriver
browser_profile = webdriver.FirefoxProfile()
# add the file_formats to download
file_formats = ','.join(["text/plain",
"application/pdf",
"application/x-pdf",
"application/force-download"])
preferences = {
"browser.download.folderList": 2,
"browser.download.manager.showWhenStarting": False,
"browser.download.dir": os.getcwd(), # will download to current directory
"browser.download.alertOnEXEOpen": False,
"browser.helperApps.neverAsk.saveToDisk": file_formats,
"browser.download.manager.focusWhenStarting": False,
"browser.helperApps.alwaysAsk.force": False,
"browser.download.manager.showAlertOnComplete": False,
"browser.download.manager.useWindow": False,
"services.sync.prefs.sync.browser.download.manager.showWhenStarting": False,
"pdfjs.disabled": True
}
for pref, val in preferences.items():
browser_profile.set_preference(pref, val)
browser_binary = webdriver.firefox.firefox_binary.FirefoxBinary()
browser = webdriver.Firefox(firefox_binary=browser_binary,
firefox_profile=browser_profile)
# set the file name that will be saved as when you download is complete
file_name = 'ABC.txt'
# goto the link to download the file from it will be automatically
# downloaded to the current directory
file_url = 'http://yourfiledownloadurl.com'
browser.get(file_url)
# verify if the expected file name exists in the current directory
path = os.path.join(os.getcwd(), file_name)
assert os.path.isfile(path)

How to write file in Google Chrome App without prompting?

I am fumbling around with the free Chrome Dev Editor on my Chromebook. I am trying to use the fileSystem to read and write .txt files. It is all very wrapped up, not at all like in C. I can no more tell if I am even allowed to do something, let alone where the proper place is to find out how.
I think the files I can see using the Files thingy are in the sandbox that I am allowed to play in (meaning, folders that are accessible by the app?) The root is called Downloads. Sure enough, if I use all the dot calls and callback arguments for the read, as in the examples at developer.chrome.com/apps/filesystem, it works. But I have to have a prompt
every time for both reads and writes.
A little more Googling came up with this trick: (I think it was here in stackoverflow, in fact) a chrome.runtime call, getPackagedDirectoryEntry, that seems to give me a handle to the folder of my app. Great! That's all I need to not have to go through the prompting. For the readfile, anyway.
But then trying to apply the same trick to the writefile did not work. In fact, it did nothing discernible. No errors, no complaints. Nothing. Even though the write file with prompting works fine (so presumably I have the permissions and Blob construction right.) What to do?
Here is my code:
function test(){
// Samsung 303C Chromebook - Chrome Dev Editor - /Downloads/Daily/main.js
// prompted write
chrome.fileSystem.chooseEntry({type:'saveFile'},function(a){
a.createWriter(function(b){
b.write(new Blob(["Programming fun"],{type:'text/plain'}));
},function(e){trace.innerText = 'error is ' + e;});
});
// unprompted read
chrome.runtime.getPackageDirectoryEntry(function(a){
a.getFile('text.txt',{},function(b){
b.file(function(c){
var d = new FileReader();
d.onloadend = function(){trace.innerText = this.result;};
d.readAsText(c);
});
});
});
// unprompted write - why not?
chrome.runtime.getPackageDirectoryEntry(function(a){
a.getFile('new.txt',{create:true},function(b){
b.createWriter(function(c){
c.write(new Blob(["Miss Manners fan"],{type:'text/plain'}));
},function(e){trace.innerText = 'error is ' + e;});
});
});
}

To be fair, Filesystem API is a big mess of callbacks and it's not unreasonable to get drowned in it.
It's not currently documented, but chrome.runtime.getPackageDirectoryEntry returns a read-only DirectoryEntry, and there is no way to make it writable (it's specifically blacklisted).
You probably don't see an error, because it fails at the getFile stage, for which you don't have an error handler.
Unfortunately, for a Chrome App the only option to write out to a real filesystem is to prompt the user. However, you can retain the entry and ask only once.
If you don't need to write out to the real filesystem but need only internal storage, HTML Filesystem API can help you (yes, it's marked as abandoned, but Chrome maintains it since chrome.fileSystem is built on it).
Extensions additionally have access to chrome.downloads API that enables writing to (but not reading) the Downloads folder.
P.S. What you see in Files app is your "real" local filesystem in ChromeOS + mounted cloud filesystems (e.g. Google Drive)

You can use the basic web Filesystem API. First, add the "unlimitedStorage" permission. Then, copy the packaged files to the sandboxed filesystem, like this:
chrome.runtime.getPackageDirectoryEntry(function(package) {
package.getMetadata(function(metadata) {
webkitRequestFileSystem(PERSISTENT, metadata.size, function(filesystem) {
package.copyTo(filesystem.root)
})
})
})

Implement cross extension message passing in chrome extension and app

I am trying to do cross extension message passing between chrome extension and chrome app according to this article. But I am not sure that how to do it correctly. I used background js to receive and send messages. But no clue whether it is working or not. Actually I want to save file from chrome extension, since it cannot be done I thought this could work. So any idea or suggestion or example is highly welcome.
I have go through many alternatives as also appears in this question. Then one of answer points this example. I found that this example is works fine. I hope that I could use this mechanism to save file using Chrome App's fileSystem API.

The Chrome messaging APIs can only transfer JSON-serializable values. If the files are small, then you could just read the file content using FileReader in the extension, send the message over the external messaging channel to the Chrome App, then save the data using the FileWriter API.
When the files are big, read the file in chunks using file.slice(start, end) then follow the same method as for small files.
Extension:
var app_id = '.... ID of app (32 lowercase a-p characters) ....';
var file = ...; // File or Blob object, e.g. from an <input type=file>
var fr = new FileReader();
fr.onload = function() {
var message = {
blob: fr.result,
filename: file.name,
filetype: file.type
};
chrome.runtime.sendMessage(app_id, message, function(result) {
if (chrome.runtime.lastError) {
// Handle error, e.g. app not installed
console.warn('Error: ' + chrome.runtime.lastError.message);
} else {
// Handle success
console.log('Reply from app: ', result);
}
});
};
fr.onerror = function() { /* handle error */ };
// file or sliced file.
fr.readAsText(file);
App:
chrome.runtime.onMessageExternal.addListener(
function(message, sender, sendResponse) {
// TODO: Validate that sender.id is allowed to invoke the app!
// Do something, e.g. convert back to Blob and do whatever you want.
var blob = new Blob([message.blob], {type: message.filetype});
console.log('TODO: Do something with ' + message.filename + ':', blob);
// Do something, e.g. reply to message
sendResponse('Processed file');
// if you want to send a reply asynchronously, uncomment the next line.
// return true;
});
EDIT: Although the following method using sounded nice in theory, it does not work in practice because a separate SharedWorker process is created for the app / extension.
If you want to send huge files (e.g. Files), then you could implement the following:
Extension: Create proxy.html (content = <script src=proxy.js></script>). (feel free to pick any other name).
Extension: Put proxy.html in web_accessible_resources.
App: Bind a window.onmessage event listener. This event listener will receive messages from the frame you're going to load in the next step.
App: Load chrome-extension://[EXTENSIONID]/proxy.html in a frame within your app. This extension ID can either be hard-coded (see Obtaining Chrome Extension ID for development) or exchanged via the external extension message passing API (make sure that you validate the source - hardcoding the ID would be the best way).
Extension: When proxy.html is loaded, check whether location.ancestorOrigins[0] == 'chrome-extension://[APPID]' to avoid a security leak. Terminate all steps if this condition fails.
Extension: When you want to pass a File or Blob to the app, use parent.postMessage(blob, 'chrome-extension://[APPID]');
App: When it receives the blob from the extension frame, save it to the FileSystem that you obtained through the chrome.fileSystem API.
The last task to solve is getting a file from the extension to the extension frame (proxy.html) that is embedded in the app. This can be done via a SharedWorker, see this answer for an example (you can skip the part that creates a new frame because the extension frame is already created in one of the previous steps).
Note that at the moment (Chrome 35), Files can only be sent with a work-around due to a bug.

How do I turn off the console logging in App Engine Channel API?

I've implemented the Channel API w/ persistence. When I make a channel and connect the socket (this is on the real app, not the local dev_appserver), Firebug goes nuts with log messages. I want to turn these off so I can see my OWN logs but cant find any documentation on how to disable the Channel API console logging.
one thing I'm probably doing differently than most is that I'm connecting cross-domain... which the Channel API supports (note the first message in the stream... if you can view that pic)
Does anyone know?
UPDATE
I finally realized that my code was creating two channels and trying to open/connect them both at the same time... and that was why I was getting a flood of messages. I didn't mean to do this (I know the rules: https://developers.google.com/appengine/docs/python/channel/overview#Caveats )... it was a bug... and once I fixed it, the messages went back to manageable level.
yay

There doesn't appear to be a way to shutoff the Firebug timeStamp log. One way to solve this problem is to edit the code and remove this functionality yourself:
Unpack the extension to a directory in your Mozilla Firefox Profile:
Change directory to your Firefox profile extensions directory. On Ubuntu, this would be something like this:
cd ~/.mozilla/firefox/{random-string}/extensions/
The Firebug extension is identified by firebug#software.joehewitt.com.xpi. Create a new directory of the same name, but without the .xpi, and move the XPI into that directory:
mkdir firebug#software.joehewitt.com
mv firebug#software.joehewitt.com.xpi firebug#software.joehewitt.com
Next, change directories to your newly created Firebug directory, and unpack the extension:
cd firebug#software.joehewitt.com
unzip firebug#software.joehewitt.com.xpi
All of the files should be unpacked so that the extension's directories are in the current directory. Your file structure will look something like this:
$: ~/.mozilla/firefox/{random-string}/extensions/firebug#software.joehewitt.com$ l
chrome.manifest defaults/ firebug#software.joehewitt.com.xpi install.rdf locale/ skin/
content/ docs/ icons/ license.txt modules/
$: ~/.mozilla/firefox/ghlfe0bb.ff5.0/extensions/firebug#software.joehewitt.com$
Open consoleExposed.js in your text editor:
Next, change to the content/firebug/console directory:
cd content/firebug/console
Edit the consoleExposed.js file using your favorite editor:
vim consoleExposed.js
Disable console.timeStamp:
On or near line 215, you'll see the following function:
console.timeStamp = function(label)
{
label = label || "";
if (FBTrace.DBG_CONSOLE)
FBTrace.sysout("consoleExposed.timeStamp; " + label);
var now = new Date();
Firebug.NetMonitor.addTimeStamp(context, now.getTime(), label);
var formattedTime = now.getHours() + ":" + now.getMinutes() + ":" +
now.getSeconds() + "." + now.getMilliseconds();
return logFormatted([formattedTime, label], "timeStamp");
};
Right after the first curly-brace, force the function to return nothing:
console.timeStamp = function(label)
{ return ; // disable timestamp by returning
label = label || "";
if (FBTrace.DBG_CONSOLE)
Restart Firefox and enjoy a world without timeStamp:
After the edits, restart Firebug. You should no longer see the log messages for timeStamp in your console.

On the Development server, when using the ChannelAPI, it essentially degrades into a polling implementation instead of using Comet/long-polling. Thus, in your debugger, you see an endless stream of HTTP requests made to the server to continuously and methodically check for updates.
In essence, these are just AJAX requests, or as Firebug would like to think of them, XMLHttpRequests.
Since your browser is responsible for making these requests, the only way to disable them is to click the small arrow on "Console" in Firebug and uncheck the option for logging XMLHttpRequests.
Of course, this also disables logging for all of your other XMLHttpRequests. But it's a small price to pay for the clarity and serenity of a quiet, well-behaved JavaScript console.
For more helpful information on how to make the most of Firebug, see Firebug Tips and Tricks.
NOTE: This works for both users of the Python SDK as well as the Java SDK. (or Go SDK, assuming it has an equivalent ChannelAPI). This is not limited to only Python Appengine.
UPDATE:
From getFirebug:
Creates a time stamp, which can be used together with HTTP traffic timing to measure when a certain piece of code was executed.
The console.timeStamp method was released in Firebug 1.8.0. The same technique described above can also override this Firebug logging method.
console.timeStamp("This is the type of console logging statement that Google is using!");
The above logging statement would produce the olive text. This method can be disabled using the same techniques which were described in the previous section.
However, Google loads the console object inside of a closure, which means that, once Google's code is initialized, the ChannelAPI object has it's own copy of the console object.
In order to disable console.timeStamp, one would need to disable it as the very first action before anything else loads or runs. in other words, we would need to ensure that Google only gets its hands on the disabled console.timeStamp method.
For best results, load this code above the /_ah/channel/jsapi script tag to ensure the console.timeStamp method is disabled before jsapi loads:
if(window.console) console.timeStamp = function(t) { };
NOTE: Because Google invokes Firebug logging in this manner, the only solution may very well in fact require a bug report or feature request that would allow for programmatically disabling this level of logging. Alternatively, the Firebug team could provide a new version of Firebug that includes the ability to explicitly disable timeStamp log statements, similar to how they've done so with Errors, Warnings, XMLHttpRequests, and other log levels.

How to bypass document.domain limitations when opening local files?

I have a set of HTML files using JavaScript to generate navigation tools, indexing, TOC, etc. These files are only meant to be opened locally (e.g., file://) and not served on a web server. Since Firefox 3.x, we run into the following error when clicking a nav button that would generate a new frame for the TOC:
Error: Permission denied for <file://> to get property Location.href from <file://>.
I understand that this is due to security measures within FF 3.x that were not in 2.x, in that the document.domain does not match, so it's assuming this is cross-site scripting and is denying access.
Is there a way to get around this issue? Perhaps just a switch to turn off/on within Firefox? A bit of JavaScript code to get around it?

In firefox:
In address bar, type about:config,
then type network.automatic-ntlm-auth.trusted-uris in search bar
Enter comma separated list of
servers (i.e.,
intranet,home,company)
Another way is editing the users.js.
In users.js, write:
user_pref("capability.policy.policynames", "localfilelinks");
user_pref("capability.policy.localfilelinks.sites", "http://site1.com http://site2.com");
user_pref("capability.policy.localfilelinks.checkloaduri.enabled", "allAccess");
But if you want to stop all verification, just Write the following line into users.js file:
user_pref("capability.policy.default.checkloaduri.enabled", "allAccess");

You may use this in firefox to read the file.
function readFile(arq) {
netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
var file = Components.classes["#mozilla.org/file/local;1"].createInstance(Components.interfaces.nsILocalFile);
file.initWithPath(arq);
// open an input stream from file
var istream = Components.classes["#mozilla.org/network/file-input-stream;1"].createInstance(Components.interfaces.nsIFileInputStream);
istream.init(file, 0x01, 0444, 0);
istream.QueryInterface(Components.interfaces.nsILineInputStream);
var line = {}, lines = [], hasmore;
do {
hasmore = istream.readLine(line);
lines.push(line.value);
} while(hasmore);
istream.close();
return lines;
}

Cleiton's method will work for yourself, or for any users who you expect will go through this manual process (not likely unless this is a tool for you and your coworkers or something).
I'd hope that this type of thing would not be possible, because if it is, that means that any site out there could start opening up documents on my machine and reading their contents.

You can have all files that you want to access in subfolders relative to the page that is doing the request.
You can also use JSONP to load files from anywhere.

Add "file://" to network.automatic-ntlm-auth.trusted-uris in about:config