I have a Karma test which looks like this:
it('testing if test will fail on exception', function () {
expect(true).toBe(true);
throw "an exception";
expect(false).toBe(true);
});
The problem is this test returns valid, because the exception is thrown before the second test condition was checked. I need a solution that will work on all tests, so implementing try/catch is not the solution I am searching for. Is there a way to set Karma/Jasmine configs so that all tests fail on exception thrown? Any other good ideas?
Related
I know how to check if a test failed in the afterEach() method of mocha: That's explained here: detecting test failures from within afterEach hooks in Mocha
But what about the people using suite and test (tdd) instead of describe and it??
How can I check if the current test failed here? The same code won't work because state would be undefined:
teardown(async () => {
// check if failed:
if (this.currentTest.state === 'failed') {
console.log("fail");
}
});
It seems that it works a little bit different with tdd (using suite and test).
Accessing this.ctx.currentTest instead of this.currentTest worked for me.
Example:
if (this.ctx.currentTest.state === 'failed') {
console.log(`'${this.ctx.currentTest.title}' failed`);
}
I have a very basic karma/jasmine setup with one test suite containing two tests. I expect the first test to fail and the second test to pass.
describe("The system", function() {
it("should fail", function() {
expect(true).toBe(false);
});
it("should succeed", function(done) {
setTimeout(function() {
expect(1).toBe(1);
done();
}, 10);
});
});
However, when I run these tests in the browser and click the Debug button to open the Karma DEBUG RUNNER, I see both tests failing, where the second test fails with the error message of the first test. The regular test run (i.e. not in the Karma DEBUG RUNNER) works as expected.
The error message for the second test is:
Uncaught Expected true to be false.
at UserContext.<anonymous> (http://localhost:9876/base/workingspec.js:4:22) thrown
When I disable or remove the first test, the second test passes.
Why do both tests fail in this case? Why does the second test fail with the error message of the first test?
My test setup contains the following packages/versions:
+-- jasmine-core#2.7.0
+-- karma#1.7.0
+-- karma-chrome-launcher#2.2.0
+-- karma-jasmine#1.1.0
`-- karma-jasmine-html-reporter#0.2.2
The problem is indeed in the Debug.js file of the Karma Debug Runner, as #user907860 already hinted at. It is not particular to Jasmine. I have reported the issue and created a fix which has just been merged into the master branch of Karma, so the next release should fix this issue.
At first I thought it was a bug, but after some research with the brilliant Chrome devtools, it seems that this is an expected behavior, at least by Jasmine. It may be, though, a bug in the Karma framework.
In short, the node_modules/karma/static/debug.js file (which is the js-file for the debug page) has these lines (I have Karma v1.7.0):
for (var i = 0; i < result.log.length; i++) {
// Throwing error without losing stack trace
(function (err) {
setTimeout(function () {
throw err
})
})(result.log[i])
}
If you comment the throw line, and restart the karma server, you'll see only console log messages, which should be expected: first FAIL, then PASS, and then the summary.
Verbosely, the bug in Karma may be in it's behavior to report after each spec.
That is what's happening here step by step (I have version "jasmine-core": "^2.6.4", at least this is in my package.json file):
Jasmine runs the first spec and it fails;
Karma reports on this in the log and adds a function, which throws an error to the stack (I assume the reader knows the asynchronous model in JavaScript, if not, then he must read about this in something like
"Effective JavaScript: 68 Specific Ways to Harness the Power of JavaScript" by David Herman, a true gemstone book, or elsewhere). Also, this is important, though I'm not sure, since I haven't look into the code so deep, it registers sort of "globalError", since on the next step
Jasmine runs the second spec, and calls the getJasmineRequireObj().GlobalErrors function (jasmine.js:2204). The "globalError" is detected and the spec instantly becomes a failure. The asynchronous expect is added to the stack after Karma's function, the one, which throws the error
Then the first function (which throws the error) starts execution, add it throws. This function will always be called before the Jasmine asynchronous spec, since in the debug.js there is no time passed to the setTimeout call:
//notice the absent time argument
setTimeout(function () {
throw err
})
Jasmine runs the assync expect from the second spec and it passes
Karma reports on the second spec as a failure and adds the error throwing (if not commented out, then no error occurs and this spec passes) to the stack
Karma's second error is thrown
Below are screen shots with numbers, put by me to illustrate the steps from the list:
with throw not commented in the debug.js file:
and with the commented throw:
Help me please to find way to validate test result in Protractor not from "expect" function.
I have such code:
describe("The 'toEqual' matcher", function() {
it("works for simple literals and variables", function() {
expect(12).toEqual(12);
if (this.results_.failedCount === 0) {
console.log("This test passed")
}
else{
console.log("This test failed")
}
});});
but when I execute this test I have such message:
Failed: Cannot read property 'failedCount' of undefined.
May be you know such another solution to fix my problem?
Thanks for support.
You should not be counting passes and failures yourself directly in the test. Let the protractor and jasmine runner worry about it. If you need to have a control over it, look into making a jasmine reporter or using one of the many existing, like the jasmine-spec-reporter, for instance.
I am creating HTTP tests with frisby.js which works on top of jasmine.js.
I also have to create some mongoDB objects to test against.
The problem is when I want to clean up these DB objects. When one of the expects fail I want to intercept that and call my own cleanup function. This means that after each failed test, I won't be able to remove the test objects from the DB.
The afterEach function in jasmine does not work properly and jasmine does not have any support for afterAll or beforeAll yet.
That is why I have made the tests as they are today.
it("testing userform get with correct userID and expect correct return", function() {
var innerUserId = userID;
frisby.create('Should retrieve correct userform and return 200 when using a valid userID')
.get(url.urlify('/api/userform', {id: innerUserId}))
.expectStatus(200)
.afterJSON(function(userform){
// If any of these fail, the after function wont run.
// I want to intercept the error so that I can make sure that the cleanUp function is called
// afterEach does not work. I have tried with done()
var useridJSON = userform.UserId.valueOf();
var firstnameJSON = userform.firstname.valueOf();
var surnameJSON = userform.surname.valueOf();
expect(firstnameJSON).toMatch(testUser.firstName);
expect(surnameJSON).toMatch(testUser.surname);
expect(useridJSON).toMatch(innerUserId);
})
.after(function(){
cleanUp(innerUserId);
})
.toss();
});
I am wondering if there is a way to intercept the error for "expect" in frisby or jasmine so that I can make a call to my own cleanup function before exiting.
Full example here
The quickest solution to this problem is to wrap the error code in a try-catch.
This is because if a javascript error occurs, jasmine will NOT keep running assertions. This is different from an assertion error. If an assertion error occurs, jasmine and frisby will keep on testing all the other assertions and then do the "after"-function.
.afterJSON(function(userform){
try {
var useridJSON = userform.UserId.valueOf();
var firstnameJSON = userform.firstname.valueOf();
var surnameJSON = userform.surname.valueOf();
catch(e) {
cleanUp(innerUserId);
// Can do a throw(e.message); here aswell
}
expect(firstnameJSON).toMatch(testUser.firstName);
expect(surnameJSON).toMatch(testUser.surname);
expect(useridJSON).toMatch(innerUserId);
})
This is not the pretty way, but works.
I ended up adding the throw(e) and placed the expects in a finally scope. This way I got jasmine to present all the errors that occured in the test.
As for "before exiting", how about this:
process.on('uncaughtException', function(err) {
console.error(' Caught exception: ' + err);
});
I am testing javascript code that throws an exception (temporarily, early TDD state) but jasmine passes the test.
Is there any way to set up jasmine such that it fails with an unexpected exception?
Are there other javascript unit test frameworks that do not pass such tests?
Make sure you are using the latest version 1.2.0.
The code bellows fails:
describe("must fail on error", function(){
it("a + 1 should produce an error", function(){
expect(a + 1).toEqual(2);
})
});
I stumbled on this post searching for something else related to Jasmine testing. Before seeing this question, I saw this a blog post on testing exceptions with Jasmine. In that post, the author uses the bind command like this:
it('should allow us pass in parameters', function () {
expect(myOtherProcedure.bind(null, 10, 'you generated: ')).not.toThrow();
});