I have a very basic karma/jasmine setup with one test suite containing two tests. I expect the first test to fail and the second test to pass.
describe("The system", function() {
it("should fail", function() {
expect(true).toBe(false);
});
it("should succeed", function(done) {
setTimeout(function() {
expect(1).toBe(1);
done();
}, 10);
});
});
However, when I run these tests in the browser and click the Debug button to open the Karma DEBUG RUNNER, I see both tests failing, where the second test fails with the error message of the first test. The regular test run (i.e. not in the Karma DEBUG RUNNER) works as expected.
The error message for the second test is:
Uncaught Expected true to be false.
at UserContext.<anonymous> (http://localhost:9876/base/workingspec.js:4:22) thrown
When I disable or remove the first test, the second test passes.
Why do both tests fail in this case? Why does the second test fail with the error message of the first test?
My test setup contains the following packages/versions:
+-- jasmine-core#2.7.0
+-- karma#1.7.0
+-- karma-chrome-launcher#2.2.0
+-- karma-jasmine#1.1.0
`-- karma-jasmine-html-reporter#0.2.2
The problem is indeed in the Debug.js file of the Karma Debug Runner, as #user907860 already hinted at. It is not particular to Jasmine. I have reported the issue and created a fix which has just been merged into the master branch of Karma, so the next release should fix this issue.
At first I thought it was a bug, but after some research with the brilliant Chrome devtools, it seems that this is an expected behavior, at least by Jasmine. It may be, though, a bug in the Karma framework.
In short, the node_modules/karma/static/debug.js file (which is the js-file for the debug page) has these lines (I have Karma v1.7.0):
for (var i = 0; i < result.log.length; i++) {
// Throwing error without losing stack trace
(function (err) {
setTimeout(function () {
throw err
})
})(result.log[i])
}
If you comment the throw line, and restart the karma server, you'll see only console log messages, which should be expected: first FAIL, then PASS, and then the summary.
Verbosely, the bug in Karma may be in it's behavior to report after each spec.
That is what's happening here step by step (I have version "jasmine-core": "^2.6.4", at least this is in my package.json file):
Jasmine runs the first spec and it fails;
Karma reports on this in the log and adds a function, which throws an error to the stack (I assume the reader knows the asynchronous model in JavaScript, if not, then he must read about this in something like
"Effective JavaScript: 68 Specific Ways to Harness the Power of JavaScript" by David Herman, a true gemstone book, or elsewhere). Also, this is important, though I'm not sure, since I haven't look into the code so deep, it registers sort of "globalError", since on the next step
Jasmine runs the second spec, and calls the getJasmineRequireObj().GlobalErrors function (jasmine.js:2204). The "globalError" is detected and the spec instantly becomes a failure. The asynchronous expect is added to the stack after Karma's function, the one, which throws the error
Then the first function (which throws the error) starts execution, add it throws. This function will always be called before the Jasmine asynchronous spec, since in the debug.js there is no time passed to the setTimeout call:
//notice the absent time argument
setTimeout(function () {
throw err
})
Jasmine runs the assync expect from the second spec and it passes
Karma reports on the second spec as a failure and adds the error throwing (if not commented out, then no error occurs and this spec passes) to the stack
Karma's second error is thrown
Below are screen shots with numbers, put by me to illustrate the steps from the list:
with throw not commented in the debug.js file:
and with the commented throw:
Related
I am testing a web app and the test runs reliably in headed mode (cypress open) but has errors in headless mode (cypress run), so it's likely a race condition that I cannot resolve. The error message is:
[36819:0223/163815.745047:ERROR:system_services.cc(34)] SetApplicationIsDaemon: Error Domain=NSOSStatusErrorDomain Code=-50 "paramErr: error in user parameter list" (-50)
This error is mentioned again when Cypress creates a video of the incident:
- Started processing: Compressing to 32 CRF
2022-02-23 17:00:19.700 Cypress Helper[37571:416134] In -[NSApplication(NSQuietSafeQuit) _updateCanQuitQuietlyAndSafely], _LSSetApplicationInformationItem(NSCanQuitQuietlyAndSafely) returned error -50
- Finished processing: /Users/malte.wirz/Documents/iframes-cypress-issue/cypress/videos/iframe-github.js.mp4 (3 seconds)
I created a demo repository here. To reproduce, clone it, run yarn to install, and yarn cypress:run. The test does pass, but with the error mentioned above.
I assume that the error stems from accessing the nested iframes and I tested 5 different approaches so far, but to no avail. I especially made sure that the function getIframeBody waits until each iframe and the requested element is ready. The error also creates a video, but you can only see the successful run, the error message is not visible there.
Any help on how to debug this further is much appreciated.
describe('Testing Iframe within Iframes', () => {
it('Visits the iframe website and accesses the iframe within the iframe', () => {
const getIframeBody = (iframeSelector, elementSelectorInIframe) => {
return cy
.get(iframeSelector)
.its('0.contentDocument.body', {timeout: 30000})
.should((body) => {
expect(Cypress.$(body).has(elementSelectorInIframe).length).gt(0)
})
.then(cy.wrap)
}
// Visiting the page index.html and getting iframe A
cy.visit('index.html').contains('XHR in iframe')
getIframeBody('iframe[data-cy="bankid"]', 'iframe[src="https://tools.bankid.no/bankid-test/auth"]').as('iframeA')
cy.get('#iframeA').within(() => {
getIframeBody('iframe[src="https://tools.bankid.no/bankid-test/auth"]', 'iframe[src^="https://csfe.bankid.no/CentralServerFEJS"]').as('iframeB')
cy.get('#iframeB').within(() => {
getIframeBody('iframe[src^="https://csfe.bankid.no/CentralServerFEJS"]', 'input[type="tel"]').as('iframeC')
// Now we are in the right place and it finds the correct input element.
// However, normal cypress command .type() fails and we have to use library cypress-real-events,
// which provides an event firing system that works literally like in puppeteer
cy.get('#iframeC').find('input[type="tel"]').should('be.visible').realType('12345678912')
// But for the button below, this library now doesn't help anymore:
// "Failed to execute 'getComputedStyle' on 'Window': parameter 1 is not of type 'Element'."
// This was solved by using {scrollBehavior:false}.
cy.get('#iframeC').find('button[type="submit"]').should('be.visible').first().realClick({scrollBehavior:false})
})
})
})
})
I got some feedback that the above "ERROR:system_services.cc(34)" is not critical and does not cause flaky or unsuccessful tests, therefore there are no action points.
I have a Karma test which looks like this:
it('testing if test will fail on exception', function () {
expect(true).toBe(true);
throw "an exception";
expect(false).toBe(true);
});
The problem is this test returns valid, because the exception is thrown before the second test condition was checked. I need a solution that will work on all tests, so implementing try/catch is not the solution I am searching for. Is there a way to set Karma/Jasmine configs so that all tests fail on exception thrown? Any other good ideas?
I was originally using the mocha command line tool to run my tests and they were working fine. I switched to using the wdio command to run my tests. My tests now throw an error with this line of code:
browser.waitForEnabled('#div_id');
With this error:
Promise was rejected with the following reason: Error: selector needs to be typeof `string`
running chrome
Error: Promise was rejected with the following reason: Error: selector needs to be typeof `string`
at elements() - isEnabled.js:18:17
at isEnabled() - waitForEnabled.js:37:22
This was working fine until I started using wdio (specifically I run wdio --spec path/to/file.js). I've run the typeof function on the selector in question and verified that it is, in fact, a string.
The div in question looks like this:
<div class="highlight" id="div_id">
A fair bit of content goes here.
</div>
Why am I seeing this error? How do I fix it?
waitForEnabled() documentation => http://webdriver.io/api/utility/waitForEnabled.html
wdio documentation => http://webdriver.io/guide/testrunner/gettingstarted.html
Update:
I've tried adding a timeout to the waitForEnabled() function. Since I've done so, it sometimes fails, and sometimes does not. More often it fails though.
I'm not marking this as the answer because I have no idea why it works. But passing in all of the optional params to waitForEnabled() makes it work just fine.
As in:
waitForEnabled('#div_id'); Fails. Wheras:
waitForEnabled('#div_id', 99999, false); works without errors.
The fact that you're passing milliseconds & the false option should not make the test pass.
I'd suggest using some of the other waitFor commands, like waitForVisible() or waitForExist(), because I'm assuming this is what you really want from the code example you gave above. The waitForEnabled() command waits for a field that had a disabled html attribute, to not have it anymore. You can read more on that here: https://www.w3schools.com/tags/att_disabled.asp.
I am new to using Jasmine and BDD testing and I am trying to test if a simple page exists.
I have the following code:
describe("when server is running",function () {
it("should serve an index page", function(){
expect(server.app).toBeDefined();
request.get('http://localhost:3000/', function(error, response, body){
expect(response.statusCode).toBeEqual(200);
done();
});
expect(server.app).toBeDefined();
});
});
When the tests are run I get:
Finished in 0.027 seconds
1 test, 2 assertions, 0 failures, 0 skipped
There are obviously 3 assertions and it's the middle one, within the function() call that is not performed (I added the third repetitive one just for checking and commented out each one individually).
I came across the guide at https://semaphoreci.com/community/tutorials/getting-started-with-node-js-and-jasmine and made not of the addition of the done() callback but still no joy.
Is this something wrong with the test, the request or the jasmine-node setup I have.
I'm running this on Ubuntu 14.04.
Thanks.
I noticed that qUnit doesn't give any notice when an exception happens in a later part of the test. For example, running this in a test():
stop();
function myfun(ed) {
console.log('resumed');
start(); //Resume qunit
ok(1,'entered qunit again');
ok(ed.getContent()== 'expected content') // < causes exception, no getContent() yet.
}
R.tinymce.onAddEditor.add(myfun)
in an inner iframe on the page will cause an exception (TypeError: ed.getContent is not a function),
but nothing in Qunit status area tells this. I see 0 failures.
(R being the inner iframe, using technique here: http://www.mattevanoff.com/2011/01/unit-testing-jquery-w-qunit/) Would I be correct in assuming this isn't the best way to go for testing sequences of UI interaction that cause certain results? Is it always better to use something like selenium, even for some mostly-javascript oriented frontend web-app tests?
As a side note, the Firefox console shows the console.log below the exception here, even though it happened first... why?
If you look into qUnit source code, there are two mechanisms handling exceptions. One is controlled by config.notrycatch setting and will wrap test setup, execution and teardown in try..catch blocks. This approach won't help much with exceptions thrown by asynchronous tests however, qUnit isn't the caller there. This is why there is an additional window.onerror handler controlled by Test.ignoreGlobalErrors setting. Both settings are false by default so that both kinds of exceptions are caught. In fact, the following code (essentially same as yours but without TinyMCE-specific parts) produces the expected results for me:
test("foo", function()
{
stop();
function myfun(ed)
{
start();
ok(1, 'entered qunit again');
throw "bar";
}
setTimeout(myfun, 1000);
});
I first see a passed tests with the message "entered qunit again" and then a failed one with the message: "uncaught exception: bar." As to why this doesn't work for you, I can see the following options:
Your qUnit copy is more than two years old, before qUnit issue 134 was fixed and a global exception handler added.
Your code is changing Test.ignoreGlobalErrors setting (unlikely).
There is an existing window.onerror handler that returns true and thus tells qUnit that the error has been handled. I checked whether TinyMCE adds one by default but it doesn't look like it does.
TinyMCE catches errors in event handlers when calling them. This is the logical thing to do when dealing with multiple callbacks, the usual approach is something like this:
for (var i = 0; i < callbacks.length; i++)
{
try
{
callbacks[i]();
}
catch (e)
{
console.error(e);
}
}
By redirecting all exceptions to console.error this makes sure that exceptions are still reported while all callbacks will be called even if one of them throws an exception. However, since the exception is handled jQuery can no longer catch it. Again, I checked whether TinyMCE implements this pattern - it doesn't look like it.
Update: Turns out there is a fifth option that I didn't think of: the exception is fired inside a frame and qUnit didn't set up its global error handler there (already because tracking frame creation is non-trivial, a new frame can be created any time). This should be easily fixed by adding the following code to the frame:
window.onerror = function()
{
if (parent.onerror)
{
// Forward the call to the parent frame
return parent.onerror.apply(parent, arguments);
}
else
return false;
}
Concerning your side-note: the console object doesn't guarantee you any specific order in which messages appear. In fact, the code console.log("foo");throw "bar"; also shows the exception first, followed by the log message. This indicates that log messages are queued and handled delayed, probably for performance reasons. But you would need to look into the implementation of the console object in Firefox to be certain - this is an implementation detail.