I have set up jasmine tests for my app. In my app I have javascript that shouldn't work in IE8 :
var foo = Object.create(Array.prototype);
When I run mvn jasmine:bdd and I open the test page in Internet Explorer (with browserMode set to Internet Explorer 8), it fails as expected:
TypeError: Object doesn't support property or method 'create'
However, when I run:
mvn jasmine:test -DbrowserVersion=INTERNET_EXPLORER_8
All of my tests are successful. The logs specify that the browserVersion is set to INTERNET_EXPLORER_8.
I expected both to give me the same result. This is causing an issue with our ci testing since it has let out js errors that I wanted to catch.
Should this be working as I'm expecting and if not, what should I change?
Also, this is the best way I know how to test multiple browsers. Is there a better way that I'm missing?
EDIT
A coworker has tried to dash my hopes that the browserVersion wouldn't even catch such an error and suggested that it was only meant to change the header so that the tests could cover browser specific javascript as well (blocks of code that only run when the browser is a specific version). Is this accurate?
It seems very accurate that the jasmine tests browserVersion property does not change the internals of how the js is run, meaning that the quirks or each browser will not be tested. I have successfully transitioned to using js-test-driver for my tests which can run jasmine tests on actual browsers. This looks to be the right way to do this. Although if you're starting new down this path, I'd look at Karma instead (previously named testacular, and built as an improvement on js-test-driver).
Related
I'm trying of the console to access the "browser" environment e.x. browser.cookies.getAll but this not defined anywhere except extension environment.
If make simple firefox addon (extension) with one .js file where browser API request:
browser.cookies.getAll({}).then(console.log);
get an array with an interactive preview.
Execute from extension
If execute this command in console
How to access "browser" namespace from console?
This is not possible, browser.* or chrome.* are not available on developer console because they need an extension's context to run and developer console runs commands on the context of current page.
The following approach requires learning/knowledge of unit testing and integration testing in JavaScript and node.js, the example provided is over-simplified, this is by no means production ready code.
A better approach for testing your extensions and debugging it is to write tests for it.
Choose a testing framework (Jest, Mocha + chai, etc) and set it up according to your needs
Install sinon-chrome package which provides you with stubs for browser.* methods/apis by running npm install --save-dev sinon-chrome
(Optional) Install webextensions-api-fake which provides you with mocks for browser.* methods/apis by running npm install --save-dev webextensions-api-fake
(Optional) Install webextensions-jsdom which helps you to write tests for your browser_action default_popup, sidebar_action default_panel or background page/scripts
Start writing tests by following the example below
In order to debug your extension, set a breakpoint in your IDE/Editor of choice and run the tests, the execution will stop at the breakpoint and you will have access the states of Objects and Variables at that time in execution. This will help you know what and how exactly things are executing and what's happening to the data you pass around in functions. There is no need for writing console.log statements everywhere for checking your output or variables, debuggers can help with that.
(Optional) webextensions-toolbox is another great tool for writing cross-browser extensions (Your extension will work on chrome, firefox, opera, edge) with the same code base. This also comes with hot-reloading of your extension page, so you don't have to hit refresh every time you make any changes.
By following this approach, it will improve your development workflow and will reduce the number of times you have to hit refresh on your browser.
Example usage of sinon-chrome stubs using jest testing framework.
Lets say you have written your code in yourModule.js then to test/verify that it works in
yourModule.test.js you write:
import browser from 'sinon-chrome';
import yourModule from './lib/yourModule';
describe('moduleName', () => {
beforeAll(() => {
// To make sure yourModule uses the stubbed version
global.browser = browser;
});
it('does something', async () => {
await yourModule();
// Lets assume your module creates two tabs
expect(browser.tabs.create.calledTwice).toBe(true);
// If you want to test how those browser methods where called
expect(browser.tabs.create.firstCall.calledWithExactly({
url: 'https://google.com',
})).toBe(true);
// Notice the usage of `.firstCall` here, this makes sure only the first time
// `browser.tabs.create` was called with the given args.
});
});
When you run this test using jest, yourModule will expect there to exist a global variable browser with the apis it uses which is only possible in a real browser, but we faked it using the sinon-chrome package, your module will execute in node.js environment as expected.
You don't need to run it in the browser to see changes. You just write tests, write code to pass those tests and when all tests pass. Check your extension by running it in the browser, at this point in time your extension will run as you'd expect it to. If you add another feature to yourModule and your tests fail you know exactly what went wrong.
However the above example only makes sure how browser.* methods/apis were called, for you to test the behavior of yourModule you'd need to mock those methods/apis, this is were the webextensions-api-fake package comes in. You can find example in its repo on github.
Examples for testing your browser_action default_popup, sidebar_action default_panel or background page/scripts are also provided in the webextensions-jsdom repo on github.
Update 2:
After forgetting about this for a week (and being sick), I am still out of my depth here. The only news is that I reran the tests in Safari and Firefox, and now Safari always fails on these tests, and Firefox always times out. I assume I've changed something somewhere, but I have no idea where.
I'm also more and more certain there's a timing issue somewhere. Possibly simply code going async where it shouldn't, but more likely it's something being interrupted.
Update:
I'm less interested in finding the actual bug, and way more interested in why it's intermittent. If I can find out why that is, I can probably find the bug, or at least rewrite the code so it's avoided.
TL;DR:
I'm using Karma (with Webpack and Babel) to run tests in Chrome, and most of them are fine, but for some reason 7 tests get intermittent failures.
Details:
So! To work!
The six first tests MOSTLY succeed when I run it in the debug tab, and MIGHT fail. The failure percentage seems higher when running it normally, though. These six tests are related, as they all fail after running a specific method which functions as a safe delete() for some Backbone models. Basically it's meant to check and clear() all linked models in the model to be deleted, and return false if it's not able to do that.
And had the failures been 100%, I am sure I would find the error and wink it out, but the only thing I know is that it has to do with trying to access or change a model that has already been deleted, which seems like it's a timing thing...? Something being run asynchronously but shouldn't perhaps...? I have no idea how to fix it...
The seventh test is a little easier. It's using Jasmine-Jquery to check if a dom element (which starts out empty) gets another div inside after I change something. It's meant to test if Bootstrap's Alert-system is implemented correctly, but has been simplified heavily in order to try to find out why it fails. This test always fails if I run it as a gulp task, but always succeeds if I open the debug tab and rerun the test manually. So my hypothesis is that Chrome doesn't render the DOM correctly the first time, but fixes it if I rerun it in the debug tab...?
TMI:
When I say I open the debug tab and rerun the test manually, I am still inside the same 'gulp test' task, of course. I also use a 'gulp testonce', but the only change there is that it has singleRun enabled and the HTML reporter enabled. It shows the exact same pattern, though I can't check the debug page there, since the browser exits after the tests.
Output from one of the first 6 tests using the html reporter.
Chrome 47.0.2526 (Mac OS X 10.11.2) model library: sentences: no longer has any elements after deleting the sentence and both elements FAILED
Error: No such element
at Controller._delete (/Users/tom/dev/Designer/test/model.spec.js:1344:16 <- webpack:///src/lib/controller.js:107:12)
at Object.<anonymous> (/Users/tom/dev/Designer/test/model.spec.js:143:32 <- webpack:///test/model.spec.js:89:31)
Output from test 7 using the html reporter.
Website tests ยป Messaging system
Expected ({ 0: HTMLNode, length: 1, context: HTMLNode, selector: '#messagefield' }) not to be empty.
at Object.<anonymous> (/Users/tom/dev/Designer/test/website.spec.js:163:39 <- webpack:///test/website.spec.js:109:37)
Now, the first thing you should know is that I have of course tried other browsers, but Safari has the exact same pattern as Chrome, and Firefox gives me the same errors, but the error messages end up taking 80MB of diskspace in my html reporter and SO MUCH TIME to finish, if it even finishes. Most of the time it just disconnects - which ends up being faster.
So I ended up just using Chrome to try to find this specific bug, which has haunted my dreams now for a week.
Source
Tests:
https://dl.dropboxusercontent.com/u/117580/model.spec.js.html
https://dl.dropboxusercontent.com/u/117580/website.spec.js.html
Test output (Since the errors are intermittent, this is really just an example): https://dl.dropboxusercontent.com/u/117580/output.html
OK, all tests now succeed. I THINK this was the answer:
Some tests called controller, and some tests called window.controller. This included some reset() and remove() commands.
After doing a rewrite, I still had failures, so I did another rewrite. As part of that rewrite, I decided to make all calls through window.*, and after that rewrite all tests succeeded.
I've set up a simple testbed for WatiN (ver 2.1) which reads:
var browser = new IE();
browser.GoTo("http://www.google.co.il"); // webpage doesn't matter really
browser.RunScript("alert(123)");
This works only if KB3025390 is not installed. Installing it breaks the above test with an UnAuthorizedAccessException which has HRESULT set to E_ACCESSDENIED. What gives? Is there any workaround?
Update: Using IWebBrowser2.Navigate2 along with "javascript:console.log(123)" type of scripts works however
it makes me feel uneasy using such a backchannel
the scripts run through this back-channel of .Navigate2() may only have a max length of about 2070 chars (give or take) otherwise they get forcibly truncated to this length leading to javascript errors upon attempting to run them
using .Navigate2(), even with the most trivial script, will clog the ready state of Internet Explorer for good in the sense that it will be set to READYSTATE_LOADING without any hope of getting rid of it. In simple terms this means that once you use this hack, you either have to perform every single subsequent operation in WatiN in a "dont-wait-for-webpage-to-load" fashion (GoToNoWait, ClickNoWait etc) lest your code freezes upon waiting for the browser to turn back to READYSTATE_COMPLETE (which will never come about ofcourse as already mentioned).
there appears to be a much broader issue here in the sense that I can't even access the properties of an IHtmlWindow2 object p.e. window.document throws an unauthorized exception again making it virtually impossible to transfer over to the C# world the return-values of the scripts I'm running (using Expando etc) for documents other than window.top.document (for the window.top.document window there is IWebBrowser2.Document which does the trick)
Update#2: The folks over at the selenium project have also noticed this issue:
https://code.google.com/p/selenium/issues/detail?id=8302
A bug report has been created as well:
https://connect.microsoft.com/IE/feedback/details/1062093/installation-of-kb3025390-breaks-out-of-process-javascript-execution-in-ie11
Update#3: IHTMLWindow2.setInterval and IHTMLWindow2.setTimeout also throw UnauthorizedAccess exceptions. These methods are not marked as deprecated in:
http://msdn.microsoft.com/ko-kr/library/windows/desktop/aa741505%28v=vs.85%29.aspx
yet they have wounded up suffering from the same cutbacks all the same.
Update#4: I gave the approach recommended in this post a shot:
https://stackoverflow.com/a/18546866/863651
In order to dynamically invoke the "eval" method of the IHTMLWindow2 object (or any other method really). Got the same "System.UnauthorizedAccessException" as above. So no joy here either.
Microsoft recommends using "eval" over "execscript" however after the above experiment I suspect that they are refering to accessing "eval" only from within the browser.
As far as I can tell thus far, when it comes to the full-fledged IE11+ using "eval" out-of-process (via COM) appears to have been completely prohibited along with any other function-invocation of the window object, the only exception being the back-channel of the .Navigate2() mentioned above.
It turns out Microsoft eventually backpedaled on its decision to kill off .execScript at COM-level. Just install the latest updates for Windows including kb3025390: One of the updates for IE that came after kb3025390 brings back .execScript functionality at COM-level
Note, however, that .execScript is not accessible through IE's javascript anymore. In that context it's gone for good.
fyi: this one is also not working
ieInstance.Document.Script.<methodNameString>(<commaSeperatedParameterString>)
try this worked for me at some places but not all places
ieObject.Navigate "javascript:<methodNameString>(<commaSeperatedParameterString>)", Null, "_parent"
or
ieObject.Navigate2 "javascript:"<methodNameString>(<commaSeperatedParameterString>)", Null, "_parent"
now trying to find out solution using eval
I have found a way around the problem of an update installing automatically. You can just create a simple batch file with following content.
{code}
#echo off
wusa /uninstall /kb:3025390/quiet /norestart
END
{code}
Then go to task scheduler, create a new task for this batch file to run every one hour or day as per your requirements. Add it as a system task so it runs in the background and does not affect the running automations.
I'm using Jasmine to test a backbone app, which works fine in the browser, but when I run it using grunt-contrib-jasmine via phantomjs it chokes on:
var myVar = myCollection.get('myModel').get('myAttribute');
which throws an error:
Testing jasmine specs via phantom
>> TypeError: 'undefined' is not an object (evaluating 'myCollection.get('myModel').get')
but the strange thing is this works just fine if I break it up like this:
var mdl = myCollection.get('myModel');
var myVar = mdl.get('myAttribute');
Is this a bug in phantom or am I missing something in the way I'm writing the code?
Seems like since it works in the browser it's a phantom problem.
The problem turned out to be much bigger than I realize. It's rather complex to explain and frankly doesn't relate at all to this question. Consider this question dead.
My QUnit test suite (source here) is failing in Firefox 23 and later with the error TypeError: document.setUserData is not a function. See this screenshot for more information.
Mysteriously, a different test seems to fail every time, but only one test fails per page load. I see that Firefox removed setUserData, but I am not using that function in my code, nor do I see that function being used in QUnit code. What could be happening here?
It looks like this is a known issue with FF and Selenium:
Apparently, Node.setUserData is no longer supported. The alternative
is to use a WeakMap
https://developer.mozilla.org/en-US/docs/Web/API/Node.setUserData more
specifically, executeScript() fails, because the firefox driver is
injecting a script into the target page and then using
document.setUserData to communicate with that script.
https://code.google.com/p/selenium/issues/detail?id=5710