I have some troubles with unit testing code that uses a module that uses child_process exec. Mocha does not work properly when I create an unit test that uses asserts inside or after an exec call. When an assert is triggered, Mocha just will keep running until the given timeout is reached. When the assert is not triggered, everything will work properly.
For example, in this test case Mocha will run until the timeout is reached:
it('someTest', function (done) {
var exec = require('child_process').exec;
exec('ping 8.8.8.8', () => {
assert.deepEqual(1, 2, 'test');
done();
});
});
Then same problem still accrues when the assert is outside the exec:
it('someTest2', function (done) {
var exec = require('child_process').exec;
var promise = new Promise((promiseDone) => {
exec('ping 8.8.8.8', () => {
var success = 1 === 2;
promiseDone(success, '1 has to be 2');
});
});
promise.then((success, message) => {
assert.deepEqual(success, true, message); // Assert stops the test but Mocha does not stop
done();
});
});
A workaround for this problem is to create a before statement where you execute the method that uses exec and save the results, then you create a it(); block that checks those results with asserts.
This does not feel like a proper solution. With this approach, you are doing the unit tests in a before statement and check the results later in an it(); block. That cannot be right.
An example:
var success = true;
before(function (done) {
this.timeout(10000);
var exec = require('child_process').exec;
exec('ping 8.8.8.8', () => {
success = 1 === 2;
done();
});
});
it('someTest1', function (done) {
assert.deepEqual(success, true, 'test');
done();
});
Your ping command takes longer than 2 seconds. I'm not all that familiar with Windows but the attempts that ping makes by default take longer than 2 seconds. So increasing the timeout is one solution.
On a *nix system ping will typically run until you hit Ctrl-C. According to the man on my Debian system, if you want ping to end earlier, you can use -c [count] to set the number of packets to send. Or -w [deadline] to set a deadline by which it must end. For instance, ping -w 1 8.8.8.8 will work just fine with your code. (There's also -W [timeout] which tells ping how long to wait before giving up if there is no response, see the manual for details.)
If you were just using ping as some placeholder command just to test the structure of your Mocha tests, you could use echo foo instead (or whatever the equivalent is on Windows), or something as simple that ends right away.
Related
We use Cypress for test automation. And sometimes tests get stuck in Jenkins due to some issues, and the whole execution gets stuck. How to make Cypress skip a test if its execution takes really long time. But the rest suit execution should continue
Couldn't find anything for this issue. Seen some ideas about cancelling the whole suite if one test fails, but it's not what I need
There is a timeout option on the Mocha context, which you can use in your Cypress tests because Cypress is built on top of Mocha.
Before the problem tests, add this command
beforeEach(function() {
this.timeout(60_000) // timeout when stuck
})
Or for every test, add it in the /cypress/support/e2e.js file.
Reference: How to set timeout for test case in cypress
Also Mocha timeouts, can be used at suite level, test level and hook level.
describe('a suite of tests', function () {
this.timeout(500);
it('should take less than 500ms', function (done) {
setTimeout(done, 300);
});
it('should take less than 500ms as well', function (done) {
setTimeout(done, 250);
});
})
Alternatively, you can try the done() method which signals the end of a test has been reached.
it('gets stuck and needs a kick in the ...', (done) => {
// my sticky test code
cy.then(() => done()) // fails the test if never called
})
You could use the standard setTimeout() function in the beginning of each test. All it does is execute a function after a period of time. If the function it executes throws an exception cypress will catch that, fail the test, and move on.
So you can setTimeout() before the test, then clearTimeout() after the test, then you effectively have a test timeout feature.
const testTimeout = () => { throw new Error('Test timed out') };
describe('template spec', () => {
let timeout;
beforeEach(() => {
// Every test gets 1 second to run
timeout = setTimeout(testTimeout, 1000);
});
afterEach(() => {
// Clear the timeout so it can be reset in the next test
clearTimeout(timeout)
});
it('passes', () => {
cy.wait(500);
});
it('fails', () => {
cy.wait(1500);
});
});
If you wanted to handle this at an individual test level:
const testTimeout = () => { throw new Error('Test timed out') };
it('fails', () => {
// Start timeout at beginning of the test
const timeout = setTimeout(testTimeout, 1000);
// Chain clearTimeout() off the last cypress command in the test
cy.wait(1500).then(() => clearTimeout(timeout));
// Do not put clearTimeout() outside of a cypress .then() callback
// clearTimeout(timeout)
});
You need to call clearTimeout() in a chained .then() off the last cypress command. Otherwise the timeout will actually be cleared immediately because cypress commands are asynchronous.
With all that said, I'll also leave you with Mocha's docs on the this.timeout() feature. In all honestly, I couldn't get it to work the way I expected, so I came up with the setTimeout() method https://mochajs.org/#test-level. Hopefully one of these helps.
Cypress has timeout configuration option. If a command takes longer, its will automatically fail
it('test', () => {
cy.wait(1000);
// ...
})
If you want to skip test which failed with timeout, you can use try/catch
try {
cy.wait(1000);
//...
} catch (e) {
....
}
I have the following code which is designed to create a test db, upgrade it, seed it with data and then run a bunch of mocha tests:-
await createDatabase().then(async () => {
console.log("upgrading database");
await dbUpgrade().then(async () => {
console.log("seeding database");
await seedTestDatabase().then(() => {
After "seedTestDatabase", I call all my test suites. When I run this code, I get the following in the console:-
Opening database ./db/test.sqlite
listening on port 4001
creating database
upgrading database
starting upgrade
0 passing (1ms)
finished upgrade
seeding database
seeding test database
starting tests
For some reason, Mocha decides in the middle of the db upgrade function that it can't find any tests! If I run the functions createDatabase, dbUpgrade then seedDatabase manually one by one, everything works. What's going on with Mocha?
probably, it can work with mochajs
describe('test database is build', function(){
this.timeout(60 * 1000); // mocha terminates tests after 3 seconds with timeout error
it('creates database', function() {
return createDatabase();
});
it('updates database', function() {
return dbUpgrade();
});
it('seeds database', function() {
return seedTestDatabase();
});
});
if we assume each function depicted in question returns promise, we can call all them as test suit items in promise style approach sequently
I'm using Puppeteer and Jest to run some front end tests.
My tests look as follows:
describe("Profile Tab Exists and Clickable: /settings/user", () => {
test(`Assert that you can click the profile tab`, async () => {
await page.waitForSelector(PROFILE.TAB);
await page.click(PROFILE.TAB);
}, 30000);
});
Sometimes, when I run the tests, everything works as expectedly. Other times, I get an error:
Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.
at node_modules/jest-jasmine2/build/queue_runner.js:68:21 <br/>
at Timeout.callback [as _onTimeout] (node_modules/jsdom/lib/jsdom/browser/Window.js:633:19)
This is strange because:
I specified the timeout to be 30000
Whether or not I get this error is seemingly very random
Why is this happening?
The timeout you specify here needs to be shorter than the default timeout.
The default timeout is 5000 and the framework by default is jasmine in case of jest. You can specify the timeout inside the test by adding
jest.setTimeout(30000);
But this would be specific to the test. Or you can set up the configuration file for the framework.
Configuring Jest
// jest.config.js
module.exports = {
// setupTestFrameworkScriptFile has been deprecated in
// favor of setupFilesAfterEnv in jest 24
setupFilesAfterEnv: ['./jest.setup.js']
}
// jest.setup.js
jest.setTimeout(30000)
See also these threads:
setTimeout per test #5055
Make jasmine.DEFAULT_TIMEOUT_INTERVAL configurable #652
P.S.: The misspelling setupFilesAfterEnv (i.e. setupFileAfterEnv) will also throw the same error.
It should call the async/await when it is async from test.
describe("Profile Tab Exists and Clickable: /settings/user", () => {
test(`Assert that you can click the profile tab`, async (done) => {
await page.waitForSelector(PROFILE.TAB);
await page.click(PROFILE.TAB);
done();
}, 30000);
});
The answer to this question has changed as Jest has evolved. Current answer (March 2019):
You can override the timeout of any individual test by adding a third parameter to the it. I.e., it('runs slow', () => {...}, 9999)
You can change the default using jest.setTimeout. To do this:
// Configuration
"setupFilesAfterEnv": [ // NOT setupFiles
"./src/jest/defaultTimeout.js"
],
and
// File: src/jest/defaultTimeout.js
/* Global jest */
jest.setTimeout(1000)
Like others have noted, and not directly related to this, done is not necessary with the async/await approach.
This is a relatively new update, but it is much more straight forward. If you are using Jest 24.9.0 or higher you can just add testTimeout to your config:
// in jest.config.js
module.exports = {
testTimeout: 30000
}
I would like to add (this is a bit long for a comment) that even with a timeout of 3000 my tests would still sometimes (randomly) fail with
Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.
Thanks to Tarun's great answer, I think the shortest way to fix a lot of tests is:
describe('Puppeteer tests', () => {
beforeEach(() => {
jest.setTimeout(10000);
});
test('Best Jest test fest', async () => {
// Blah
});
});
For Jest 24.9+, we just need to add --testTimeout on the command line:
--testTimeout= 10000 // Timeout of 10 seconds
The default timeout value is 5000 (5000 ms - 5 seconds). This will be applicable for all test cases.
Or if you want to give timeout to particular function only then you can use this syntax while declaring the test case.
test(name, fn, timeout)
Example
test('example', async () => {
}, 10000); // Timeout of 10 seconds (default is 5000 ms)
For Jest 24.9+, you can also set the timeout from the command line by adding --testTimeout.
Here's an excerpt from its documentation:
--testTimeout=<number>
Default timeout of a test in milliseconds. Default value: 5000.
Make sure to invoke done(); on callbacks or it simply won't pass the test.
beforeAll((done /* Call it or remove it */ ) => {
done(); // Calling it
});
It applies to all other functions that have a done() callback.
Yet another solution: set the timeout in the Jest configuration file, e.g.:
{ // ... other stuff here
"testTimeout": 90000
}
You can also get timeout errors based on silly typos. For example, this seemingly innocuous mistake:
describe('Something', () => {
it('Should do something', () => {
expect(1).toEqual(1)
})
it('Should do nothing', something_that_does_not_exist => {
expect(1).toEqual(1)
})
})
Produces the following error:
FAIL src/TestNothing.spec.js (5.427s)
● Something › Should do nothing
Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.
at node_modules/jest-jasmine2/build/queue_runner.js:68:21
at Timeout.callback [as _onTimeout] (node_modules/jsdom/lib/jsdom/browser/Window.js:678:19)
While the code sample posted doesn't suffer from this, it might be a cause of failures elsewhere. Also note that I'm not setting a timeout for anything anywhere - either here or in the configuration. The 5000 ms is just the default setting.
I recently ran into this issue for a different reason: I was running some tests synchronously using jest -i, and it would just timeout. For whatever reasoning, running the same tests using jest --runInBand (even though -i is meant to be an alias) doesn't time out.
test accepts a timeout argument. See https://jestjs.io/docs/api#testname-fn-timeout. Here is a sample:
async function wait(millis) {
console.log(`sleeping for ${millis} milliseconds`);
await new Promise(r => setTimeout(r, millis));
console.log("woke up");
}
test('function', async () => {
await wait(5000);
}, 70000);
Mar 14, 2022, Jest 27.5 documentation indicates a new process:
https://jestjs.io/docs/api#beforeallfn-timeout
Pass a second parameter to test with the number of msec before timeout. Works!
test('adds 1 + 2 to equal 3', () => {
expect(3).toBe(3);
},30000);
// In jest.setup.js
jest.setTimeout(30000)
If on Jest <= 23:
// In jest.config.js
module.exports = {
setupTestFrameworkScriptFile: './jest.setup.js'
}
If on Jest > 23:
// In jest.config.js
module.exports = {
setupFilesAfterEnv: ['./jest.setup.js']
}
The timeout problem occurs when either the network is slow or many network calls are made using await. These scenarios exceed the default timeout, i.e., 5000 ms. To avoid the timeout error, simply increase the timeout of globals that support a timeout. A list of globals and their signature can be found here.
For Jest 24.9
Turns out if your expect assertions are wrong, it can sometimes spit out the exceeded timeout error message.
I was able to figure this out by putting console.log() statements in my promise callback and saw the console.log() statements were getting ran in the jest output. Once I fixed my expect assertions, the timeout error went away and tests worked.
I spent way too long to figure this out and hope this helps whoever needs to read this.
In case someone doesn't fix the problem use methods above. I fixed mine by surrounding the async func by an arrow function. As in:
describe("Profile Tab Exists and Clickable: /settings/user", () => {
test(`Assert that you can click the profile tab`, (() => {
async () => {
await page.waitForSelector(PROFILE.TAB)
await page.click(PROFILE.TAB)
}
})(), 30000);
});
For the jest versions greater than 27, you can add useRealTimers on the top of your spec file.
Here is the snippet
import { shortProcess, longProcess } from '../../src/index';
jest.useRealTimers();
describe(`something`, function () {
it('should finish fine', async function () {
await shortProcess();
expect(1).toBe(1);
});
it('should fail with a timeout', async function () {
await longProcess();
expect(1).toBe(1);
});
it('should finish fine again', async function () {
jest.setTimeout(10 * 1000);
await longProcess();
expect(1).toBe(1);
}, 10000);
});
Find the github issue here on jest repository.
This probably won't be terribly helpful to most people visiting this page, but when I was getting this error it had nothing to do with Jest. One of my method calls was getting an empty object and a null exception while running locally. Once I added a null check, the failing tests and console log in question disappeared.
if(response !== null){
this.searchSubj.next(resp);
}
else {
return;
}
For those who are looking for an explanation about
jest --runInBand, you can go to the documentation.
Running Puppeteer in CI environments
GitHub - smooth-code/jest-puppeteer: Run your tests using Jest & Puppeteer
In my case, this error started appearing randomly and wouldn't go away even after setting a timeout of 30000. Simply ending the process in the terminal and rerunning the tests resolved the issue for me. I have also removed the timeout and tests are still passing again.
Dropping my 2 cents here, I had the same issue on dosen of jest unit test (not all of them) and I notice that all started after I added to jestSetup this polyfill for MutuationObservers:
if (!global.MutationObserver) {
global.MutationObserver = function MutationObserverFun(callback) {
this.observe = function(){};
this.disconnect = function(){};
this.trigger = (mockedMutationsList) => {
callback(mockedMutationsList, this);
};
};
}
Once I removed it test start working again correctly. Hope helps someone .
add this in your test, not much to explain
beforeEach(() => {
jest.useFakeTimers()
jest.setTimeout(100000)
})
afterEach(() => {
jest.clearAllTimers()
})
I have this function:
let removePresentation = function(presentationName, callback) {
let rimraf = require('rimraf');
callback();
callback();
callback();
if(!presentationName || !presentationName.trim()) {
callback();
return;
}
presentationName = presentationName.replace('.zip', '');
rimraf('./presentations/' + presentationName, function(err) {
if(err) {
console.log(err);
}
callback();
});
};
exports.removePresentation = removePresentation;
and I am trying to test it with the following:
var chai = require('chai'),
expect = require('chai').expect,
sinonChai = require('sinon-chai'),
sinon = require('sinon'),
mock = require('mock-require');
chai.use(sinonChai);
describe('removePresentation', function() {
var sandbox;
var callback;
var rimrafSpy;
beforeEach(function() {
sandbox = sinon.sandbox.create();
mock('../business/communications_business', {});
rimrafSpy = sinon.spy();
callback = sinon.spy();
mock('rimraf', rimrafSpy);
});
afterEach(function() {
sandbox.restore();
});
it('should call rimraf if presentation name is valid', function(done) {
let RoomStateBusiness = require('../business/roomstate_business');
RoomStateBusiness.removePresentation('name.zip', callback);
expect(rimrafSpy).to.have.been.calledWith('./presentations/name');
expect(callback).to.have.been.called.once;
done();
});
it('should not call rimraf if presentation name is null', function(done) {
let RoomStateBusiness = require('../business/roomstate_business');
RoomStateBusiness.removePresentation(null, callback);
expect(rimrafSpy).not.to.have.been.called;
expect(callback).to.have.been.called.once;
done();
});
it('should not call rimraf if presentation name is whitespace', function(done) {
let RoomStateBusiness = require('../business/roomstate_business');
RoomStateBusiness.removePresentation(' ', callback);
expect(rimrafSpy).not.to.have.been.called;
expect(callback).to.have.been.called.once;
done();
});
it('should not call rimraf if presentation name is empty string', function(done) {
let RoomStateBusiness = require('../business/roomstate_business');
RoomStateBusiness.removePresentation('', callback);
expect(rimrafSpy).not.to.have.been.called;
expect(callback).to.have.been.called.once;
done();
});
});
Even though I am clearly calling callback() multiple times (whilst testing only), expect(callback).to.have.been.called.once; is always asserting to true. I have checked on the Chai api that that expects the call to be exactly once, although it is always passing no matter how many times I call callback(). What am I doing wrong?
There is no such assertion as expect(fn).to.have.been.called.once.
As per sinon-chai docs, there is only:
expect(fn).to.have.been.called
expect(fn).to.have.been.calledOnce
The problem
This is a known problem with chai and why getter-only-assertions are a bad thing. Chai allows you to write a piece of code which looks like a property access (ie. the assertion itself does not end with a function call) to assert...whatever you want to assert. This uses property getters to execute the necessary code.
The problem is that if you make a typo or other mistake, the expression will simply evaluate to undefined (you are accessing a property which does not exist) and no assertion code is ever executed, resulting in a test passing (because tests only fail if an exception is thrown).
In your case, there is an assertion for called, and this most likely returns an object. Unfortunately, that object does not have assertion once, so there is no code executed and the test passes.
The solution
There are 2 options available to you:
Upgrade to Chai 4 and Node.js version with Proxy support (not sure where Proxy support was added, probably Node.js 5 or 6) - chai introduced a safeguard against these issues by proxying all property access through a Proxy object which checks if you are using a valid assertion
Never use getters for assertions and always end your assertions with a function call - this will make sure that if you ever make a mistake the test will fail with the infamous undefined is not a function error
The second option is highly preferred, in my opinion, as there can be no doubt on the correctness of the test case. Chai proxy support can still be turned off even on supported platforms.
Assuming we are talking about sinon-chai it should be calledOnce not called.once. If you do called any number of calls > 0 will pass the test.
I'm using Gulp in my small project in order to run tests and lint my code. When any of those tasks fail, Gulp always exits with return code 0. If I run jshint by hand, it exits with non-zero code as it should.
Here's my very simple gulpfile.
Do I need to somehow explicitly tell Gulp to return a meaningful value? Is this Gulp's fault, or maybe the gulp-jshint and gulp-jasmine plugins are to blame?
You need to 'return gulp.src(...' so that the task waits for the returned stream.
EDIT
Gulp tasks are asynchronous by nature. The actual task is not executed yet at the time of 'gulp.src(...).pipe(...);'. In your example, the gulp tasks mark their results as success before the actual tasks are executed.
There are some ways to make gulp to wait for your actual task. Use a callback or return a stream or promise.
https://github.com/gulpjs/gulp/blob/master/docs/API.md#async-task-support
The easiest way is just returning the stream of 'gulp.src(...).pipe(...)'. If gulp task gets a stream, it will listen to 'end' event and 'error' event. They corresponds to return code 0 and 1. So, a full example for your 'lint' task would be:
gulp.task('lint', function () {
return gulp.src('./*.js')
.pipe(jshint('jshintrc.json'))
.pipe(jshint.reporter('jshint-stylish'));
});
A bonus is that now you can measure the actual time spent on your tasks.
#robrich is right, you have to keep track of exit codes yourself, but there's no need for a forceful approach. The process global is an EventEmitter which you can bind your exit function to.
var exitCode = 0
gulp.task('test', function (done) {
return require('child_process')
.spawn('npm', ['test'], {stdio: 'pipe'})
.on('close', function (code, signal) {
if (code) exitCode = code
done()
})
})
gulp.on('err', function (err) {
process.emit('exit') // or throw err
})
process.on('exit', function () {
process.nextTick(function () {
process.exit(exitCode)
})
})
A fix has been introduced in this commit.
It works something like this:
gulp.src("src/**/*.js")
.pipe(jshint())
.pipe(jshint.reporter("default"))
.pipe(jshint.reporter("fail"));
I have been using it on circleci and it works a treat!
gulp-jshint has been struggling with how to fail the build on jshint fail. On the one hand, we can crash the build inside jshint, but then you never get to the reporter. On the other hand, requiring the reporter to fail the build isn't part of the default reporters. I generally hook up my own reporter that keeps track of fails, and .on('end', function () { will process.exit(1). It's quite brute force, but it works like a charm. See https://github.com/wearefractal/gulp-jshint/issues/10
Similar to Andy Piper answer above, I have found this module stream-combiner2 to be useful when running a sequence of tasks to ensure and exit code is emitted if there is an error somewhere. Can be used something like this
var combiner = require('stream-combiner2');
var tasks = combiner.obj([
gulp.src(files),
task1(options),
task2(options),
gulp.dest('path/to/dest')
]);
tasks.on('error', function () {
process.exit(1)
});
gulp.task('default', function(done) {
return gulp.src( ... ).pipe(jasmine( ... )).on('error', function(err) { done(err); } )
});
Works for me