Develop Reference

How to make Cypress skip tests that are stuck in CI? Let's say make it skip after 15 min?

We use Cypress for test automation. And sometimes tests get stuck in Jenkins due to some issues, and the whole execution gets stuck. How to make Cypress skip a test if its execution takes really long time. But the rest suit execution should continue
Couldn't find anything for this issue. Seen some ideas about cancelling the whole suite if one test fails, but it's not what I need

There is a timeout option on the Mocha context, which you can use in your Cypress tests because Cypress is built on top of Mocha.
Before the problem tests, add this command
beforeEach(function() {
this.timeout(60_000) // timeout when stuck
})
Or for every test, add it in the /cypress/support/e2e.js file.
Reference: How to set timeout for test case in cypress
Also Mocha timeouts, can be used at suite level, test level and hook level.
describe('a suite of tests', function () {
this.timeout(500);
it('should take less than 500ms', function (done) {
setTimeout(done, 300);
});
it('should take less than 500ms as well', function (done) {
setTimeout(done, 250);
});
})
Alternatively, you can try the done() method which signals the end of a test has been reached.
it('gets stuck and needs a kick in the ...', (done) => {
// my sticky test code
cy.then(() => done()) // fails the test if never called
})

You could use the standard setTimeout() function in the beginning of each test. All it does is execute a function after a period of time. If the function it executes throws an exception cypress will catch that, fail the test, and move on.
So you can setTimeout() before the test, then clearTimeout() after the test, then you effectively have a test timeout feature.
const testTimeout = () => { throw new Error('Test timed out') };
describe('template spec', () => {
let timeout;
beforeEach(() => {
// Every test gets 1 second to run
timeout = setTimeout(testTimeout, 1000);
});
afterEach(() => {
// Clear the timeout so it can be reset in the next test
clearTimeout(timeout)
});
it('passes', () => {
cy.wait(500);
});
it('fails', () => {
cy.wait(1500);
});
});
If you wanted to handle this at an individual test level:
const testTimeout = () => { throw new Error('Test timed out') };
it('fails', () => {
// Start timeout at beginning of the test
const timeout = setTimeout(testTimeout, 1000);
// Chain clearTimeout() off the last cypress command in the test
cy.wait(1500).then(() => clearTimeout(timeout));
// Do not put clearTimeout() outside of a cypress .then() callback
// clearTimeout(timeout)
});
You need to call clearTimeout() in a chained .then() off the last cypress command. Otherwise the timeout will actually be cleared immediately because cypress commands are asynchronous.
With all that said, I'll also leave you with Mocha's docs on the this.timeout() feature. In all honestly, I couldn't get it to work the way I expected, so I came up with the setTimeout() method https://mochajs.org/#test-level. Hopefully one of these helps.

Cypress has timeout configuration option. If a command takes longer, its will automatically fail
it('test', () => {
cy.wait(1000);
// ...
})
If you want to skip test which failed with timeout, you can use try/catch
try {
cy.wait(1000);
//...
} catch (e) {
....
}

Wait for an own function (which returns a promise) before tests are executed

I'm new to cypress and am trying to figure out how things work.
I have my own function (which calls a test controller server to reset the database). It returns a promise which completes when the DB have been successfully resetted.
function resetDatabase(){
// returns a promise for my REST api call.
}
My goal is to be able to execute it before all tests.
describe('Account test suite', function () {
// how can I call resetDb here and wait for the result
// before the tests below are invoked?
it('can log in', function () {
cy.visit(Cypress.config().testServerUrl + '/Account/Login/')
cy.get('[name="UserName"]').type("admin");
cy.get('[name="Password"]').type("123456");
cy.get('#login-button').click();
});
// .. and more test
})
How can I do that in cypress?
Update
I've tried
before(() => {
return resetDb(Cypress.config().apiServerUrl);
});
But then I get an warning saying:
Cypress detected that you returned a promise in a test, but also invoked one or more cy commands inside of that promise
I'm not invoking cy in resetDb().

Cypress have promises (Cypress.Promise), but they are not real promises, more like duck typing. In fact, Cypress isn't 100% compatible with real promises, they might, or might not, work.
Think of Cypress.Promise as a Task or an Action. They are executed sequentially with all other cypress commands.
To get your function into the Cypress pipeline you can use custom commands. The documentation doesn't state it, but you can return a Cypress.Promise from them.
Cypress.Commands.add('resetDb', function () {
var apiServerUrl = Cypress.config().apiServerUrl;
return new Cypress.Promise((resolve, reject) => {
httpRequest('PUT', apiServerUrl + "/api/test/reset/")
.then(function (data) {
resolve();
})
.catch(function (err) {
reject(err);
});
});
});
That command can then be executed from the test itself, or as in my case from before().
describe('Account', function () {
before(() => {
cy.resetDb();
});
it('can login', function () {
// test code
});
})

You can use cy.wrap( promise ), although there might still be a bug where it never times out (haven't tested).
Otherwise, you can use cy.then() (which is undocumented, can break in the future, and I'm def not doing any favors by promoting internal APIs):
cy.then(() => {
return myAsyncFunction();
});
You can use both of these commands at the top-level of spec like you'd use any command and it'll be enqueued into cypress command queue and executed in order.
But unlike cy.wrap (IIRC), cy.then() supports passing a callback, which means you can execute your async function at the time of the cy command being executed, not at the start of the spec (because expressions passed to cy commands evaluate immediately) --- that's what I'm doing in the example above.

promise & mocha: done() in before or not?

I am reading some tutorials on promise tests in mocha. There is a piece of codes:
before(function(done) {
return Promise.resolve(save(article)).then(function() {
done();
});
});
Why done() called in the then() in the before()? What is the difference between the above codes and the following codes:
before(function(done) {
return Promise.resolve(save(article));
});
Thanks
UPDATE
My question is to compare with the following codes:
before(function() {
return Promise.resolve(save(article));
});
Sorry for the typo.

The first code snippet with the before hook returns a promise and calls done. In Mocha 3.x and over, it will result in this error:
Error: Resolution method is overspecified. Specify a callback *or* return a Promise; not both.
It used to be that it did not particularly matter if you used done and returned a promise, but eventually the Mocha devs figured that specifying both done and returning a promise just meant the test designer made a mistake and it was better to have Mocha pitch a fit rather than silently allow it.
In your 2nd snippet, you have the done argument and return a promise but Mocha will still wait for done to be called and will timeout. (It really should detect the argument and raise an error like in the 1st case, but it doesn't...)
Generally, if you are testing an asynchronous operation that produces a promise, it is simpler to return the promise than use done. Here's an example illustrating the problem:
const assert = require("assert");
// This will result in a test timeout rather than give a nice error
// message.
it("something not tested properly", (done) => {
Promise.resolve(1).then((x) => {
assert.equal(x, 2);
done();
});
});
// Same test as before, but fixed to give you a good error message
// about expecting a value of 2. But look at the code you have to
// write to get Mocha to give you a nice error message.
it("something tested properly", (done) => {
Promise.resolve(1).then((x) => {
assert.equal(x, 2);
done();
}).catch(done);
});
// If you just return the promise, you can avoid having to pepper your
// code with catch closes and calls to done.
it("something tested properly but much simpler", () => {
return Promise.resolve(1).then((x) => {
assert.equal(x, 2);
});
});
With regards to the completion of asynchronous operations, it works the same whether you are using it, before, beforeEach, after or afterEach so even though the example I gave is with it, the same applies to all the hooks.

I am not sure if I understood 100% the question, but the tests will not start until done is called.
beforeEach(function(done) {
setTimeout(function() {
value = 0;
done();
}, 1);
});
This test will not start until the done function is called in the call to beforeEach above. And this spec will not complete until its done is called.
it("should support async execution of test preparation and expectations", function(done) {
value++;
expect(value).toBeGreaterThan(0);
done();
});
You don't have to pass done in your example, just:
before(function() {
return Promise.resolve(save(article));
});
If you do pass done the test runner will expect to be called before continue, otherwise it will probably throw a timeout error.

In this particular case there is no functional difference. The callback, often called done, was introduced to handle asynchronous tests when using callbacks. Returning a Promise is sufficient, but note that you cannot define the done callback, because the test suite will wait until it's called. Use done when you can't easily return a Promise. In your case the second test will be infinite, because you define done, which you never actually call.

mocha test timeout fail insted of assertion fail [duplicate]

One of the things that I find frustrating about Mocha is that when tests fail, they don't give the actual error message of the failing line, Instead, they just end with Error: timeout of 2000ms exceeded. Ensure the done() callback is being called in this test.
Take this test for example:
describe("myTest", function() {
it("should return valid JSON.", function(done) {
api.myCall("valid value").then(function(result) {
console.log(result);
var resultObj = JSON.parse(result);
assert.isFalse(resultObj.hasOwnProperty("error"), "result has an error");
done();
});
});
});
The output is:
myTest
{"error":null,"status":403}
1) should return valid JSON.
0 passing (2s)
1 failing
1) myTest should return valid JSON.:
Error: timeout of 2000ms exceeded. Ensure the done() callback is being called in this test.
The assert.isFalse is failing, but the message that should be displayed ("result has an error") isn't displayed. In fact, processing seems to stop right there because done() is never called. Take that line out and the test passes because done() is called.
So, what am I missing? Why do Mocha tests behave this way? The actual test library I'm using is:
var assert = require("chai").assert;
Does anyone know what I'm doing wrong or why this behaves this way?

It looks like your API is using promises. Before trying anything else, I would suggest checking what the documentation of the API says about promises and how to deal with unhandled exceptions because this may be what is happening here. Some promise implementations require that you call .done() at the end of your call chain to ensure that uncaught exceptions are going to be processed. Some require that some global promise setting be properly configured. The Bluebird documentation gives a good discussion of the issues.
Mocha is capable of handling uncaught exceptions in run-of-the-mill code:
var chai = require("chai");
var assert = chai.assert;
chai.config.includeStack = true;
describe("foo", function() {
it("let the exception be caught by Mocha", function(done) {
setTimeout(function () {
assert.isFalse(true, "foo");
done();
}, 1000);
});
});
This will result in the output:
foo
1) let the exception be caught by Mocha
0 passing (1s)
1 failing
1) foo let the exception be caught by Mocha:
Uncaught AssertionError: foo: expected true to be false
at Assertion.<anonymous> (/tmp/t7/node_modules/chai/lib/chai/core/assertions.js:286:10)
at Assertion.Object.defineProperty.get (/tmp/t7/node_modules/chai/lib/chai/utils/addProperty.js:35:29)
at Function.assert.isFalse (/tmp/t7/node_modules/chai/lib/chai/interface/assert.js:297:31)
at null._onTimeout (/tmp/t7/test.js:8:20)
at Timer.listOnTimeout (timers.js:119:15)

I've encountered the same in my code, using Q for promises.
What happened was:
The assertion inside the then block failed.
The rest of the then block, including the done() statement, was not executed.
Q went looking for a catch block, which wasn't there.
This led to a 'hanging' promise, and thus to a Mocha 2000 ms timeout.
I worked around it by doing something like this:
describe("myTest", function() {
it("should return valid JSON.", function(done) {
api.myCall("valid value").then(function(result) {
console.log(result);
var resultObj = JSON.parse(result);
assert.isFalse(resultObj.hasOwnProperty("error"), "result has an error");
done();
})
.catch(function(err) {
console.error(err);
done(err);
});
});
});

Force-failing a Jasmine test

If I have code in a test that should never be reached (for example the fail clause of a promise sequence), how can I force-fail the test?
I use something like expect(true).toBe(false); but this is not pretty.
The alternative is waiting for the test to timeout, which I want to avoid (because it is slow).

Jasmine provides a global method fail(), which can be used inside spec blocks it() and also allows to use custom error message:
it('should finish successfully', function (done) {
MyService.getNumber()
.success(function (number) {
expect(number).toBe(2);
done();
})
.fail(function (err) {
fail('Unwanted code branch');
});
});
This is built-in Jasmine functionality and it works fine everywhere in comparison with the 'error' method I've mentioned below.
Before update:
You can throw an error from that code branch, it will fail a spec immediately and you'll be able to provide custom error message:
it('should finish successfully', function (done) {
MyService.getNumber()
.success(function (number) {
expect(number).toBe(2);
done();
})
.fail(function (err) {
throw new Error('Unwanted code branch');
});
});
But you should be careful, if you want to throw an error from Promise success handler then(), because the error will be swallowed in there and will never come up. Also you should be aware of the possible error handlers in your app, which might catch this error inside your app, so as a result it won't be able to fail a test.

Thanks TrueWill for bringing my attention to this solution. If you are testing functions that return promises, then you should use the done in the it(). And instead of calling fail() you should call done.fail(). See Jasmine documentation.
Here is an example
describe('initialize', () => {
// Initialize my component using a MOCK axios
let axios = jasmine.createSpyObj<any>('axios', ['get', 'post', 'put', 'delete']);
let mycomponent = new MyComponent(axios);
it('should load the data', done => {
axios.get.and.returnValues(Promise.resolve({ data: dummyList }));
mycomponent.initialize().then(() => {
expect(mycomponent.dataList.length).toEqual(4);
done();
}, done.fail); // <=== NOTICE
});
});