Develop Reference

Firefox optimization bug?

Recently when i did some adventofcode, i came across a bug:
The bug happens in firefox except for if you run the code as multiple chunks, but doesnt happen in chrome.
When i run this code in the firefox console, in a tab with url https://www.google.com/robots.txt :
// Fetch input
var input = await fetch("https://www.google.com/robots.txt").then(r => r.text());
var lines = input.split("\n");
// Iterate
while(lines.length > 0) console.log(lines.shift());
It prints nothing, but when i run the code as two pieces (separated by // Iterate) it works. Note that you can change the fetch url and url of the tab, i chose robots.txt and made them the same to avoid CORS errors.
Does anyone know why this happens or how to fix this?
Edit: The code works if you wrap the code in an async function and call it, but it should work regardless.

Test a self deleting/dynamic Webelement in Selenium

What I'm doing
I've been experimenting with Selenium and making a simple program to make my Selenium testing life easier. Part of this is testing webelements and figuring out what methods (clicking submitting ect) make them reload the page, remain static, or become stale without reloading the page. In this question I'm particularly interested in the third case as the first two are already implemented by me.
The problem I'm having
The problem I have is finding a Webelement that goes stale and doesn't cause a page reload. I can't think of a good way to search for one, I don't have the HTML and javascript skills to make one (yet anyways) and I can't verify my code works unless I actually test it.
What I've done/tried
The first thing I thought to look for was a popup but those aren't actually part of the webpage and they're also quite unreliable. I want something thats going to behave consistently because otherwise the test won't work. I think dynamic Webelements, those that change their locators when acted upon will suit my needs but I have no good way of finding them. Any google results for "Self deleting webelement exmaple" or "Webelement goes stale doesn't cause page reload example" or similar, will only give me questions on stackoverflow like this one rather than what I want - concrete examples. The code I'm running simply waits for a staleReferenceException and for an onload event in javascript. If the staleReferenceException occurs but the onload event does not, then I know I've found a self-deleting / dynamic webelement (at least thats what I think is the proper way to detect this). Here is the code I'm running:
try {
//wait until the element goes stale
wait.until(ExpectedConditions.stalenessOf(webElement));
//init the async javascript callback script
String script = "var callback = arguments[arguments.length - 1];" +
"var classToCall = 'SeleniumTest.isPageReloaded';" +
"window.addEventListener('onload'," + "callback(classToCall));";
//execute the script and wait till it returns (unless timeout exceeded)
JavascriptExecutor js = (JavascriptExecutor) driver;
//execute the script and return the java classname to call
//if/when the callback function returns normally
String classToCall = (String) js.executeAsyncScript(script);
clazz = Class.forName(classToCall);
callbackMethod = clazz.getMethod("JavascriptWorking");
callbackMethod.invoke(null,null);
//page reloaded
threadcase = 1;
}
//waiting until webElement becomes stale exceeded timeoutSeconds
catch (TimeoutException e) {
//page was static
threadcase = 2;
}
//waiting until webElement Reloaded the page exceeded timeoutSeconds
catch (ScriptTimeoutException e) {
//the webElement became stale BUT didn't cause a page reload.
threadcase = 3;
As you can notice above there is an int variable named threadcase in this code. The three 'cases' starting from 1 (0 was the starting value which represented a program flow error) represent the three (non-error) possible results of this test:
the page reloads
the page remains static, webelement doesn't change
the page remains static, webelement changes
And I need a good example with which to test the third case.
Solutions I've considered
I've done some basic research into removing webelements in javascript but I A: don't even know if I can act on the page in Selenium like that and B: I'd rather get a test case that just uses the Webpage as is since introducing my edits makes the validity of my testcase reliant on more of my code (which is bad!). So what I need is a good way of finding a webelement that matches my criteria without having to scour the internet with the f12 window open hoping to find that one button that does what I need.
Edit 1
I just tried doing this test more manually, it was suggested in an answer that I manually delete a webelement at the right time and then test my program that way. What I tested was the Google homepage. I tried using the google apps button because when clicked it doesn't cause the whole page to reload. So my thinking was, I'll click it, halt program execution, manually delete it, run the rest of my code, and since no onload events will occur, my program will pass the test. To my suprise thats not what happened.
The exact code I ran is the below. I had my debug stop on the first line:
1 Method callbackMethod = null;
2 try {
3 //wait until the element goes stale
4 wait.until(ExpectedConditions.stalenessOf(webElement));
5 //init the async javascript callback script
6 String script = "var callback = arguments[arguments.length - 1];" +
7 "var classToCall = 'SeleniumTest.isPageReloaded';" +
8 "window.addEventListener('onload', callback(classToCall));";
9 //execute the script and wait till it returns (unless timeout
10 //exceeded)
11 JavascriptExecutor js = (JavascriptExecutor) driver;
12 //execute the script and return the java classname to call if/when
13 //the callback function returns normally
14 String classToCall = (String) js.executeAsyncScript(script);
15 clazz = Class.forName(classToCall);
16 callbackMethod = clazz.getMethod("JavascriptWorking");
17 callbackMethod.invoke(null,null);
18 //page reloaded
19 threadcase = 1;
20 }
21 //waiting until webElement becomes stale exceeded timeoutSeconds
22 catch (TimeoutException e) {
23 //page was static
24 threadcase = 2;
25 }
26 //waiting until webElement Reloaded the page exceeded
27 //timeoutSeconds
28 catch (ScriptTimeoutException e) {
29 //the webElement became stale BUT didn't cause a page reload.
30 threadcase = 3;
31 //trying to get the class from javascript callback failed.
32 }
whats supposed to happen is that a Stale webelement causes the program to stop waiting on line 4, the program progresses, initializes the Javascript callback in lines 6-11 and then on line 14 the call to executeAsyncScript is SUPPOSED to wait untill an 'onload' event which should only occur if the page reloads. Right now its not doing that or I'm blind. I must be confusing the program flow because I'm 99% certain that there are no page reloads happening when I manipulate the DOM to delete the webelement I'm clicking on.
This is the URL I'm trying:
https://www.google.com/webhp?gws_rd=ssl
Simple google homepage, the button I'm deleting is the google apps button (the black 9-grid in the top right)
some info on that element:
class="gb_8 gb_9c gb_R gb_g"
id="gbwa"
Its the general container element for the button itself and the dropdown it creates. I'm deleting this when my program hits the STOP on line 1. Then I go through my program in the debugger. Note (you may have to click inspect element on the button more than once to focus in on it). I'm going to try deleting lower level elements rather than the whole container and see if that changes anything but still this behavior baffles me. The goal here is to get the program flow to threadcase 3 because thats the one we are testing for. There should be no page reloads BUT the webelement should become stale after I manually delete it. I don't have any clue why the javascript callback is running when I can't see a page reload. Let me know if you need more info on what exactly I'm deleting on the google homepage and I'll try sending a picture (with optional freehand circles of course).

I would think that you could debug through a test, place a breakpoint at a suitable point, then use the browsers dev tools to manually update the HTML.
Obviously, if you want this to be a repeatable process it is not an option, but if you are just investigating, then a manual intervention could be suitable

IPython Notebook Javascript: retrieve content from JavaScript variables

Is there a way for a function (called by an IPython Notebook cell) to retrieve the content of a JavaScript variable (for example IPython.notebook.notebook_path which contains the path of the current notebook)?
The following works well when written directly within a cell (for example, based on this question and its comments):
from IPython.display import display,Javascript
Javascript('IPython.notebook.kernel.execute("mypath = " + "\'"+IPython.notebook.notebook_path+"\'");')
But that falls apart if I try to put it in a function:
# this doesn't work
from IPython.display import display,Javascript
def getname():
my_js = """
IPython.notebook.kernel.execute("mypath = " + "\'"+IPython.notebook.notebook_path+"\'");
"""
Javascript(my_js)
return mypath
(And yes, I've tried to make global the mypath variable, both from within the my_js script and from within the function. Also note: don't be fooled by possible leftover values in variables from previous commands; to make sure, use mypath = None; del mypath to reset the variable before calling the function, or restart the kernel.)
Another way to formulate the question is: "what's the scope (time and place) of a variable set by IPython.notebook.kernel.execute()"?
I think it isn't an innocuous question, and is probably related to the mechanism that IPython uses to control its kernels and their variables and that I don't know much about. The following experiment illustrate some aspect of that mechanism. The following works when done in two separate cells, but doesn't work if the two cells are merged:
Cell [1]:
my_out = None
del my_out
my_js = """
IPython.notebook.kernel.execute("my_out = 'hello world'");
"""
Javascript(my_js)
Cell [2]:
print(my_out)
This works and produces the expected hello world. But if you merge the two cells, it doesn't work (NameError: name 'my_out' is not defined).

I think the problem is related with Javascript being asynchronus while python is not. Normally you would think that the Javascript(""" python cmd """) command is executed, and then your print statment should work properly as expected. However, the Javascript command is fired but not executed. Most pobably it is executed after the cell 1 execution is fully completed.
I tried your example with sleep function. Did not help.
The asnyc problem can esaily be seen by adding an alert statement within my_js, but before kernel.execute line. The alert should be fired even before trying a python command execution.
But at the presence of print (my_out) statement within cell 1, you will again get the same error without any alerts. If you take the print line out, you will see the alert poping out within cell 1. But the varibale my_out is set afterwards.
my_out = None
del my_out
my_js = """
**alert ("about to execute python comand");**
IPython.notebook.kernel.execute("my_out = 'hello world'");
"""
Javascript(my_js)
There are other javascript utilities within notebook like IPython.display.display_xxx which varies from displaying video to text object, but even the text object option does not work.
Funny enough, I tested this with my webgl canvas application which displays objects on the HTML5 canvas; display.display_javascript(javascript object) works fine ( which is a looong html5 document) while the two pieces of words of output does not show up?! Maybe I should embed the output into canvas application somewhere, so it s displayed on the canvas :)

I wrote a related question (Cannot get Jupyter notebook to access javascript variables) and came up with a hack that does the job. It uses the fact that the input(prompt) command in Python does block the execution loop and waits for user input. So I looked how this is processed on the Javascript side and inserted interception code there.
The interception code is:
import json
from IPython.display import display, Javascript
display(Javascript("""
const CodeCell = window.IPython.CodeCell;
CodeCell.prototype.native_handle_input_request = CodeCell.prototype.native_handle_input_request || CodeCell.prototype._handle_input_request
CodeCell.prototype._handle_input_request = function(msg) {
try {
// only apply the hack if the command is valid JSON
console.log(msg.content.prompt)
const command = JSON.parse(msg.content.prompt);
const kernel = IPython.notebook.kernel;
// return some value in the Javascript domain, depending on the 'command'.
// for now: specify a 5 second delay and return 'RESPONSE'
kernel.send_input_reply(eval(command["eval"]))
} catch(err) {
console.log('Not a command',msg,err);
this.native_handle_input_request(msg);
}
}
"""))
The interception code looks whether the input prompt is valid JSON, and in that case it executes an action depending on the command argument. In this case, it runs the commend["eval"] javascript expression and returns the result.
After running this cell, you can use:
notebook_path = input(json.dumps({"eval":"IPython.notebook.notebook_path"}))
Quite a hack, I must admit.

Okay, I found a way around the problem: call a Python function from Javascript and have it do all of what I need, rather than returning the name to "above" and work with that name there.
For context: my colleagues and I have many experimental notebooks; we experiment for a while and try various things (in a machine learning context). At the end of each variation/run, I want to save the notebook, copy it under a name that reflects the time, upload it to S3, strip it from its output and push it to git, log the filename, comments, and result scores into a DB, etc. In short, I want to automatically keep track of all of our experiments.
This is what I have so far. At the bottom of my notebooks, I put:
In [127]: import mymodule.utils.lognote as lognote
lognote.snap()
In [128]: # not to be run in the same shot as above
lognote.last
Out[128]: {'file': '/data/notebook-snapshots/2015/06/18/20150618-004408-save-note-exp.ipynb',
'time': datetime.datetime(2015, 6, 18, 0, 44, 8, 419907)}
And in a separate file, e.g. mymodule/utils/lognote.py:
# (...)
from datetime import datetime
from subprocess import call
from os.path import basename, join
from IPython.display import display, Javascript
# TODO: find out where the topdir really is instead of hardcoding it
_notebook_dir = '/data/notebook'
_snapshot_dir = '/data/notebook-snapshots'
def jss():
return """
IPython.notebook.save_notebook();
IPython.notebook.kernel.execute("import mymodule.utils.lognote as lognote");
IPython.notebook.kernel.execute("lognote._snap('" + IPython.notebook.notebook_path + "')");
"""
def js():
return Javascript(jss())
def _snap(x):
global last
snaptime = datetime.now()
src = join(_notebook_dir, x)
dstdir = join(_snapshot_dir, '{}'.format(snaptime.strftime("%Y/%m/%d")))
dstfile = join(dstdir, '{}-{}'.format(snaptime.strftime("%Y%m%d-%H%M%S"), basename(x)))
call(["mkdir", "-p", dstdir])
call(["cp", src, dstfile])
last = {
'time': snaptime,
'file': dstfile
}
def snap():
display(js())

To add to the other great answers, there is a nuance of the browsers attempting to run the jupyter nb javascript magic on nb load.
To demonstrate: create and run the following cell:
%%javascript
IPython.notebook.kernel.execute('1')
Now save the notebook, close it and then re-open it. When you do that, under that cell suddenly you will see an error in red:
Javascript error adding output!
TypeError: Cannot read property 'execute' of null
See your browser Javascript console for more details.
That means the browser has parsed some js code and it tried to run it. This is the error in chrome, it will probably different in a different browser.
I have no idea why this jupyter javascript magic cell is being run on load and why jupyter notebook is not properly escaping things, but the browser sees some js code and so it runs it and it fails, because the notebook kernel doesn't yet exist!
So you must add a check that the object exists:
%%javascript
if (IPython.notebook.kernel) {
IPython.notebook.kernel.execute('1')
}
and now there is no problem on load.
In my case, I needed to save the notebook and run an external script on it, so I ended up using this code:
from IPython.display import display, Javascript
def nb_auto_export():
display(Javascript("if (IPython.notebook) { IPython.notebook.save_notebook() }; if (IPython.notebook.kernel) { IPython.notebook.kernel.execute('!./notebook2script.py ' + IPython.notebook.notebook_name )}"))
and in the last cell of the notebook:
nb_auto_export()

Script not working on page load, but working from console

I wrote a script that's running from ebay listing iframe. It's working fine, it runs on $(document).ready(), sends an AJAX request to a remote server, gets some info, manipulate the DOM on 'success' callback, everything working perfect...
However, I added a piece of code, which should get the document.referrer, and extract some keywords from it, if they exist. Specifically, if a user searches ebay for a product, and clicks on my product from the results, the function extracts the keywords he entered.
Now, the problem is, that function is not running on page load at all. It seems like it blocks the script when it comes to that part. This is the function:
function getKeywords(){
var index = window.parent.document.referrer.indexOf('_nkw');
if (index >= 0){
var currentIndex = index + 5;
var keywords = '';
while (window.parent.document.referrer[currentIndex] !== '&'){
keywords += window.parent.document.referrer[currentIndex++];
}
keywords = keywords.split('+');
return keywords;
}
}
And I tried calling two logs right after it:
console.log('referrer: ' + window.parent.document.referrer);
console.log(getKeywords());
None of them is working. It's like when it comes to that 'window.parent.document.referrer' part, it stops completely.
But, when I put this all in a console, and run it, it works perfectly. It logs the right referrer, and the right keywords.
Does anybody know what might be the issue here?

The reason it is working on the console is because your window object is the outer window reference and not your iframe. Besides that, on the console:
window.parent === window
// ==> true
So, on in fact you are running window.document.referrer and not your frame's window.parent.document.referrer.
If you want to access your frame's window object you should something like
var myFrame = document.getElementsByClassName('my-frame-class')[0];
myFrame.contentWindow === window
// ==> false
myFrame.contentWindow.parent.window === window
// ==> true
This might help you debug your problem, but I guess the browser is just preventing an inner iframe from accessing the parent's window object.

Chrome JavaScript location object

I am trying to start 3 applications from a browser by use of custom protocol names associated with these applications. This might look familiar to other threads started on stackoverflow, I believe that they do not help in resolving this issue so please dont close this thread just yet, it needs a different approach than those suggested in other threads.
example:
ts3server://a.b.c?property1=value1&property2=value2
...
...
to start these applications I would do
location.href = ts3server://a.b.c?property1=value1&property2=value2
location.href = ...
location.href = ...
which would work in FF but not in Chrome
I figured that it might by optimizing the number of writes when there will be effectively only the last change present.
So i did this:
function a ()
{
var apps = ['ts3server://...', 'anotherapp://...', '...'];
b(apps);
}
function b (apps)
{
if (apps.length == 0) return;
location.href = apps[0]; alert(apps[0]);
setTimeout(function (rest) {return function () {b(rest);};} (apps.slice(1)), 1);
}
But it didn't solve my problem (actually only the first location.href assignment is taken into account and even though the other calls happen long enough after the first one (thanks to changing the timeout delay to lets say 10000) the applications do not get started (the alerts are displayed).
If I try accessing each of the URIs separately the apps get started (first I call location.href = uri1 by clicking on one button, then I call location.href = uri2 by clicking again on another button).
Replacing:
location.href = ...
with:
var form = document.createElement('form');
form.action = ...
document.body.appendChild(form);
form.submit();
does not help either, nor does:
var frame = document.createElement('iframe');
frame.src = ...
document.body.appendChild(frame);
Is it possible to do what I am trying to do? How would it be done?
EDIT:
a reworded summary
i want to start MULTIPLE applications after one click on a link or a button like element. I want to achieve that with starting applications associated to custom protocols ... i would hold a list of links (in each link there is one protocol used) and i would try to do "location.src = link" for all items of the list. Which when used with 'for' does optimize to assigning only once (the last value) so i make the function something like recursive function with delay (which eliminates the optimization and really forces 3 distinct calls of location.src = list[head] when the list gets sliced before each call so that all the links are taken into account and they are assigned to the location.src. This all works just fine in Mozilla Firefox, but in google, after the first assignment the rest of the assignments lose effect (they are probably performed but dont trigger the associated application launch))

Are you having trouble looping through the elements? if so try the for..in statement here
Or are you having trouble navigating? if so try window.location.assign(new_location);
[edit]
You can also use window.location = "...";
[edit]
Ok so I did some work, and here is what I got. in the example I open a random ace of spades link. which is a custom protocol. click here and then click on the "click me". The comments show where the JSFiddle debugger found errors.