Update 2:
After forgetting about this for a week (and being sick), I am still out of my depth here. The only news is that I reran the tests in Safari and Firefox, and now Safari always fails on these tests, and Firefox always times out. I assume I've changed something somewhere, but I have no idea where.
I'm also more and more certain there's a timing issue somewhere. Possibly simply code going async where it shouldn't, but more likely it's something being interrupted.
Update:
I'm less interested in finding the actual bug, and way more interested in why it's intermittent. If I can find out why that is, I can probably find the bug, or at least rewrite the code so it's avoided.
TL;DR:
I'm using Karma (with Webpack and Babel) to run tests in Chrome, and most of them are fine, but for some reason 7 tests get intermittent failures.
Details:
So! To work!
The six first tests MOSTLY succeed when I run it in the debug tab, and MIGHT fail. The failure percentage seems higher when running it normally, though. These six tests are related, as they all fail after running a specific method which functions as a safe delete() for some Backbone models. Basically it's meant to check and clear() all linked models in the model to be deleted, and return false if it's not able to do that.
And had the failures been 100%, I am sure I would find the error and wink it out, but the only thing I know is that it has to do with trying to access or change a model that has already been deleted, which seems like it's a timing thing...? Something being run asynchronously but shouldn't perhaps...? I have no idea how to fix it...
The seventh test is a little easier. It's using Jasmine-Jquery to check if a dom element (which starts out empty) gets another div inside after I change something. It's meant to test if Bootstrap's Alert-system is implemented correctly, but has been simplified heavily in order to try to find out why it fails. This test always fails if I run it as a gulp task, but always succeeds if I open the debug tab and rerun the test manually. So my hypothesis is that Chrome doesn't render the DOM correctly the first time, but fixes it if I rerun it in the debug tab...?
TMI:
When I say I open the debug tab and rerun the test manually, I am still inside the same 'gulp test' task, of course. I also use a 'gulp testonce', but the only change there is that it has singleRun enabled and the HTML reporter enabled. It shows the exact same pattern, though I can't check the debug page there, since the browser exits after the tests.
Output from one of the first 6 tests using the html reporter.
Chrome 47.0.2526 (Mac OS X 10.11.2) model library: sentences: no longer has any elements after deleting the sentence and both elements FAILED
Error: No such element
at Controller._delete (/Users/tom/dev/Designer/test/model.spec.js:1344:16 <- webpack:///src/lib/controller.js:107:12)
at Object.<anonymous> (/Users/tom/dev/Designer/test/model.spec.js:143:32 <- webpack:///test/model.spec.js:89:31)
Output from test 7 using the html reporter.
Website tests ยป Messaging system
Expected ({ 0: HTMLNode, length: 1, context: HTMLNode, selector: '#messagefield' }) not to be empty.
at Object.<anonymous> (/Users/tom/dev/Designer/test/website.spec.js:163:39 <- webpack:///test/website.spec.js:109:37)
Now, the first thing you should know is that I have of course tried other browsers, but Safari has the exact same pattern as Chrome, and Firefox gives me the same errors, but the error messages end up taking 80MB of diskspace in my html reporter and SO MUCH TIME to finish, if it even finishes. Most of the time it just disconnects - which ends up being faster.
So I ended up just using Chrome to try to find this specific bug, which has haunted my dreams now for a week.
Source
Tests:
https://dl.dropboxusercontent.com/u/117580/model.spec.js.html
https://dl.dropboxusercontent.com/u/117580/website.spec.js.html
Test output (Since the errors are intermittent, this is really just an example): https://dl.dropboxusercontent.com/u/117580/output.html
OK, all tests now succeed. I THINK this was the answer:
Some tests called controller, and some tests called window.controller. This included some reset() and remove() commands.
After doing a rewrite, I still had failures, so I did another rewrite. As part of that rewrite, I decided to make all calls through window.*, and after that rewrite all tests succeeded.
Related
When my test suite completes, I need to output some stats, i. e. meta info about tests collected during test execution.
I'm trying this:
QUnit.done(() => console.log("some meta info here"))
This works when I run tests in the browser.
But when I run tests in the terminal, the console.log output is not displayed.
There's probably some debug flag, but it will enable all console.log messages and pollute the output greatly.
Instead, I need to output one specific message to the terminal, so that it's logged to CI.
PS console.log messages sent during test execution seem to make it into the terminal successfully.
PPS Using QUnit in an Ember CLI app, against Chrome headless.
This was a tricky one, as I've never had a need to interact with QUnit like, this, but here are my findings each step of the way:
Attempt 1:
That's a weird error, I thought I was passing a callback function :-\
Attempt 2:
After looking up the documentation for Qunit.log, I could see I was using it wrong. Switching to console.log shows the beginning message -- but not the ending message.
Attempt 3:
moduleDone will print something at the end -- but it also prints for every time you use the word module (after everything inside finishes running). So, I guess as a hack if QUnit.done never ends up working, you could keep track of the number of modules started, and modules done, make sure every started modules completes, and if that number is 0 at the end, your test suite is done?
Attempt 4
Turns out, this is only actually helpful for if you want to know the outermost module is done, cause it seems like multiple tests don't run in parallel (which is probably better anyway for test stability).
Attempt 5
https://github.com/qunitjs/qunit/issues/1308
It looks like an issue with the testem adapter :(
Edit: Note that I found the root of my problem after help from #ernst-zwingli, so if you have this same error one of his noted fixes might help you out. My problem is a known issue with Protractor itself, if you think this may be you, I've expanded on my steps to pinpoint the root of the problem after my original question.
I'm trying to use Protractor in an Angular2 (just Angular) application built using angular-cli.
My problem: Elements on an Angular app page are not being found when browser.waitForAngularEnabledis at it's default setting of true (as in 'I believe I am on an angular page and would like for Protractor to do it's magic'). They are being found just fine if I set browser.waitForAngularEnabledto false (as in 'I am not on an angular page and would like to handle this myself, take a seat Protractor'). How do I track down what's causing this on my definitely Angular pages?
I have a product with a non-Angular Auth0 login page that gates access to the rest of the product that is written in Angular (Angular 4.3.2 to be exact). I have successfully traversed logging in on the non-Angular login page. I flipped the waitForAngularEnabled switched to false to facilitate the non-Angular login. I turned it back to true at the point where I expected my initial landing page (Angular) to be loaded, after clicking the submit button. Code is as follows:
browser.waitForAngularEnabled(false);
browser.driver.get('https://dashboard.net/projects');
browser.driver.sleep(10000);
browser.driver.findElement(By.css("[type='email']"));
browser.driver.findElement(By.css("[type='email']")).sendKeys("email#example.com");
browser.driver.findElement(By.css(".auth0-label-submit")).click();
browser.driver.findElement(By.id("passwordInput")).sendKeys("password");
browser.driver.findElement(By.id("submitButton")).click();
browser.driver.sleep(5000); // needed if not waiting for Angular
//browser.waitForAngularEnabled(true); // Back to Protractor land we go
let elementToFind = element(by.className("header-text"));
elementToFind.isDisplayed().then(function() {grabTheDarnLocalStorage()});
expect(elementToFind.isDisplayed()).toBeTruthy();
If I uncomment the browser.waitForAngularEnabled(true); line to state that I'm back in Angular code I get the error trace as follows:
Failed: Timed out waiting for asynchronous Angular tasks to finish after 30 seconds. This may be because the current page is not an Angular application. Please see the FAQ for more details: https://github.com/angular/protractor/blob/master/docs/timeouts.md#waiting-for-angular
While waiting for element with locator - Locator: By(css selector, .header-text)
ScriptTimeoutError: asynchronous script timeout: result was not received in 30 seconds
(Session info: chrome=61.0.3163.100)
(Driver info: chromedriver=2.32.498550 (9dec58e66c31bcc53a9ce3c7226f0c1c5810906a),platform=Windows NT 10.0.14393 x86_64)
at WebDriverError (C:\Users\c-shouston\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\error.js:27:5)
at ScriptTimeoutError (C:\Users\c-shouston\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\error.js:203:5)
at Object.checkLegacyResponse (C:\Users\c-shouston\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\error.js:505:15)
at parseHttpResponse (C:\Users\c-shouston\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\http.js:509:13)
at doSend.then.response (C:\Users\c-shouston\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\http.js:440:13)
at process._tickCallback (internal/process/next_tick.js:109:7)
From: Task: Protractor.waitForAngular() - Locator: By(css selector, .header-text)
I've referenced the FAQ: https://github.com/angular/protractor/blob/master/docs/timeouts.md#waiting-for-angular
I have my devs stating that they don't use $timeout (they use (Edit: NOT $interval) Observable Interval thank you very much) and they're not sure about $http.
I found this solution about the canonical way to debug protractor Angular sync issue issue: Canonical way to debug Protractor-to-Angular sync issues
but I'm not sure the solution works without access to modifying the dev code to run the programmatic tracker. (Edit: I never did figure out how to get this to work)
I also found this about a long timeout you add before each test, but I feel this is unnecessary overhead that makes your overall test execution take longer than it should without understanding the root cause of the problem: https://stackoverflow.com/a/37217167/2718402 (Edit: yeah, this is a bad idea and adds unnecessary time to your tests, please don't do this)
The frustrating bit is that this seems to be a common occurrence and there doesn't seem to be a streamlined documentation on how to deal with it. Logging in with a non-Angular page only to transition to an Angular page. Angular pages not being picked up properly by Protractor. All of the examples I find online are bits of code that I don't have a reference for where they should be at in my overall test framework. I would kill for a full example of someone testing a non-Angular login that transitions to a fully Angular website, with a setup config and real world test cases. (Edit: This is still true, but I can't make one myself as my application is in a bad grey area, note my RCA below for more details.)
I just want the ability to do my login and then successfully transition over to my Angular pages and be able to rely on Protractor to work with my Angular pages. I need to know what to look for that may be a long running asynchronous process (What specifically can I check for in the Chrome dev tools?). I would love to understand what Protractor needs as defaults in order to successfully work with the Angular parts of my app/website (Is there something beyond the presence of <app-root _nghost-c0="" ng-version="4.3.2"> in the HTML?). Before this job I worked in Java, so all of this asynchronicity and Angular is new to me, so I know I'm missing the known things that a seasoned Javascript dev is aware of.
My Solution/Root Cause Analysis
Starting down the list suggested by #ernst-zwingli:
for Angular(2) Check if the object window.getAllAngularRootElements returns at least one value.
It returned at least one value, so I moved on.
useAllAngular2AppRoots: true,
I tried this and still ran into the async timeout.
And if $interval or other long lasting asynchronous tasks are used, there can be issues, because of the zones
Previously #ernst-zwingli also mentioned looking at the testability method, except it was the old way. Through research and testing I found the window object also has a getAllAngularTestabilities method. This led down an interesting rabbit hole. An example output from the Chrome console (put window.getAllAngularTestabilities() in the Chrome console window, look at the resulting list) is as follows:
t:
_callbacks:...,
_didWork:true,
_isZoneStable: true (this looks promising, but why isn't Protractor working then?!?)
_ngZone:
hasPendingMacrotasks: true,
hasPendingMicrotasks: false,
isStable: true
I would think isZoneStable would be enough, but apparently not so for Protractor. Then looking at Macrotasks being true, I had to look up what the heck a Macrotask was: What does hasPendingMacrotasks and hasPendingMicrotasks check for?.
A macrotask can be:
i.e. setTimeout, setInterval, setImmediate
Thus #ernst-zwingli's note about interval's causing problems in the zones was remembered and something finally clicked.
First github issue, about zone instability
Another github issue complaining about the necessity of using browser.driver to get things done along with browser.waitForAngularEnabled. Apparently this is expected behavior, it led me to issue #3349
Issue #3349 - The actual root cause of my issue. My developers do not actively jump in and out of zones around observables. Even though these observables only have one subscriber. Since they live in the angular zone at this time, they are a long running "Macrotask" that Protractor waits infinitely on.
I can't rewrite the code with these wrappers as I am not currently versed enough in Angular to do it safely and we are currently hurtling toward a November deadline. I think I'll have to deal with using browser.driver for the time being and hope I can't get it fixed later. Hopefully my RCA was helpful for you.
In the following I list a set of potential causes and possibilities to fix/resolve them.
How does AngularJS and Angular(2) Work / What can I check in the Browser Dev Mode
I can't explain it as well as Andrey Agibalov in his Blog here, so check it out (also for developers).
Basically, the objects required by Protractor you can check in your Chrome Dev.
for AngularJS
Check if the object window.angular is properly defined, i.e. lookup window.angular.version and also try window.angular.getTestability of your Root element
for Angular(2)
Check if the object window.getAllAngularRootElements returns at least one value.
Root Element (AngularJS)
Potentially your Angular App is somewhere wrapped within the Body as something like <div ng-app="my-app">.
In that case, you must adjust your rootElement: body inside config.ts. Check this answer for details.
Angular(2)
If you're using Angular (aka Angular2), then there are ngZone's introduced. In this case your config.js should additionally contain this:
exports.config = {
framework: 'jasmine',
seleniumAddress: 'http://localhost:4444/wd/hub',
specs: ['spec.js'],
useAllAngular2AppRoots: true,
// rootElement: 'root-element'
};
check in your browser for window.getAllAngularRootElements as the additional line in conf.js is about this.
If you can, maybe use the advantage of multiple zones possible. Create a 2nd zone, configure rootElement: 'root-element'
to only focus on one zone and then move some asynchronous tasks into the other zone until you found, which task(s) lead to timeout. Keep those tasks (if possible) in the separate zone, so Protractor ignores those tasks.
And if $interval or other long lasting asynchronous tasks are used, there can be issues, because of the zones. Repeatedly or long lasting tasks should be started outside the zone and then be moved into the zone as else Protractor could run into timeouts. There is a workaround for developers to apply, in order to avoid these problems for Protractor.
read all about it here
browser.driver. - side remark
browser.driver.get() works as if ignoreSynchronization = true, since you directly assign the Browser Driver and you kind of bypass the synchronization logic of Protractor.
Read more about it in this answer here.
Hope I could give you some more input and you can solve your issue. Please let me know the results.
Could you please set
browser.ignoreSynchronization = true;
and try
I've set up a simple testbed for WatiN (ver 2.1) which reads:
var browser = new IE();
browser.GoTo("http://www.google.co.il"); // webpage doesn't matter really
browser.RunScript("alert(123)");
This works only if KB3025390 is not installed. Installing it breaks the above test with an UnAuthorizedAccessException which has HRESULT set to E_ACCESSDENIED. What gives? Is there any workaround?
Update: Using IWebBrowser2.Navigate2 along with "javascript:console.log(123)" type of scripts works however
it makes me feel uneasy using such a backchannel
the scripts run through this back-channel of .Navigate2() may only have a max length of about 2070 chars (give or take) otherwise they get forcibly truncated to this length leading to javascript errors upon attempting to run them
using .Navigate2(), even with the most trivial script, will clog the ready state of Internet Explorer for good in the sense that it will be set to READYSTATE_LOADING without any hope of getting rid of it. In simple terms this means that once you use this hack, you either have to perform every single subsequent operation in WatiN in a "dont-wait-for-webpage-to-load" fashion (GoToNoWait, ClickNoWait etc) lest your code freezes upon waiting for the browser to turn back to READYSTATE_COMPLETE (which will never come about ofcourse as already mentioned).
there appears to be a much broader issue here in the sense that I can't even access the properties of an IHtmlWindow2 object p.e. window.document throws an unauthorized exception again making it virtually impossible to transfer over to the C# world the return-values of the scripts I'm running (using Expando etc) for documents other than window.top.document (for the window.top.document window there is IWebBrowser2.Document which does the trick)
Update#2: The folks over at the selenium project have also noticed this issue:
https://code.google.com/p/selenium/issues/detail?id=8302
A bug report has been created as well:
https://connect.microsoft.com/IE/feedback/details/1062093/installation-of-kb3025390-breaks-out-of-process-javascript-execution-in-ie11
Update#3: IHTMLWindow2.setInterval and IHTMLWindow2.setTimeout also throw UnauthorizedAccess exceptions. These methods are not marked as deprecated in:
http://msdn.microsoft.com/ko-kr/library/windows/desktop/aa741505%28v=vs.85%29.aspx
yet they have wounded up suffering from the same cutbacks all the same.
Update#4: I gave the approach recommended in this post a shot:
https://stackoverflow.com/a/18546866/863651
In order to dynamically invoke the "eval" method of the IHTMLWindow2 object (or any other method really). Got the same "System.UnauthorizedAccessException" as above. So no joy here either.
Microsoft recommends using "eval" over "execscript" however after the above experiment I suspect that they are refering to accessing "eval" only from within the browser.
As far as I can tell thus far, when it comes to the full-fledged IE11+ using "eval" out-of-process (via COM) appears to have been completely prohibited along with any other function-invocation of the window object, the only exception being the back-channel of the .Navigate2() mentioned above.
It turns out Microsoft eventually backpedaled on its decision to kill off .execScript at COM-level. Just install the latest updates for Windows including kb3025390: One of the updates for IE that came after kb3025390 brings back .execScript functionality at COM-level
Note, however, that .execScript is not accessible through IE's javascript anymore. In that context it's gone for good.
fyi: this one is also not working
ieInstance.Document.Script.<methodNameString>(<commaSeperatedParameterString>)
try this worked for me at some places but not all places
ieObject.Navigate "javascript:<methodNameString>(<commaSeperatedParameterString>)", Null, "_parent"
or
ieObject.Navigate2 "javascript:"<methodNameString>(<commaSeperatedParameterString>)", Null, "_parent"
now trying to find out solution using eval
I have found a way around the problem of an update installing automatically. You can just create a simple batch file with following content.
{code}
#echo off
wusa /uninstall /kb:3025390/quiet /norestart
END
{code}
Then go to task scheduler, create a new task for this batch file to run every one hour or day as per your requirements. Add it as a system task so it runs in the background and does not affect the running automations.
I have a set of QUnit tests that run and pass on their own without any problems. However, when I recently added Blanket.js to measure javascript code coverage to help find the gaps in my testing, and all of a sudden I had several tests failing. Sprinkling in some alerts to help me locate what was happening. I found that all of my failing tests were ones using click events on functions that toggle between states. These tests were being toggled twice. I've looked into what would could cause double activation of the trigger, but in my code the functions aren't bound twice and I'm only including my javascript file once in the header.
I did notice in Firebug Blanket.js makes a GET request to the script I am testing against. Could the GET request be binding functions a second time or are there any other causes to this double activation?
There's a lot going on under the hood of blanket. From what I can tell, it is essentially evaluating the code you tell it to. So anything that would just "run" will be evaluated once by loading in the DOM, and again when blanket evaluates it.
Because Blanket will load this page scripts matched pattern in data-cover-only again to analayse script coverage, so that all your scripts executed a second time, which might bind your click event twice.
You can see it in the network pannel in Chrome Dev tools, and you will see requests like:
I have set up jasmine tests for my app. In my app I have javascript that shouldn't work in IE8 :
var foo = Object.create(Array.prototype);
When I run mvn jasmine:bdd and I open the test page in Internet Explorer (with browserMode set to Internet Explorer 8), it fails as expected:
TypeError: Object doesn't support property or method 'create'
However, when I run:
mvn jasmine:test -DbrowserVersion=INTERNET_EXPLORER_8
All of my tests are successful. The logs specify that the browserVersion is set to INTERNET_EXPLORER_8.
I expected both to give me the same result. This is causing an issue with our ci testing since it has let out js errors that I wanted to catch.
Should this be working as I'm expecting and if not, what should I change?
Also, this is the best way I know how to test multiple browsers. Is there a better way that I'm missing?
EDIT
A coworker has tried to dash my hopes that the browserVersion wouldn't even catch such an error and suggested that it was only meant to change the header so that the tests could cover browser specific javascript as well (blocks of code that only run when the browser is a specific version). Is this accurate?
It seems very accurate that the jasmine tests browserVersion property does not change the internals of how the js is run, meaning that the quirks or each browser will not be tested. I have successfully transitioned to using js-test-driver for my tests which can run jasmine tests on actual browsers. This looks to be the right way to do this. Although if you're starting new down this path, I'd look at Karma instead (previously named testacular, and built as an improvement on js-test-driver).