I'm trying to integrate my test suite with Saucelabs and to determine if the tests have passed or failed I need to do this myself.
Here is the code for the test that I have (notice I'm using Mocha):
describe('Smoke Test', () => {
describe('Login', () => {
it('Should login', (client) => {
pages.login(client).validLogin(client.globals.users.SMOKE.USERNAME, client.globals.users.SMOKE.PASSWORD);
});
});
after((client, done) => {
client.end(() => {
done();
});
});
});
Is it possible in the after block to know if the test have passed or failed?
From some examples that I found, including Saucelabs example I've seen this line:
client.currentTest.results
However currentTest have only name and method attributes.
Well, this might be late reply for you. Hope this is useful for other viewers.
afterEach will have results. Add afterEach to your example.
Apart from this, you can also get results in runner itself. Explore 'reporter' file in mocha-nightwatch. It is in your node_modules.
..\node_modules\mocha-nightwatch\lib\reporters\json.js
There are events like
runner.on(start ..)
and
runner.on(end..)
these will trigger for each tests. Give a try.
Related
Assuming I am working with 2 test files. In one of the step definition files, I implemented some Hooks in them for example:
Test1:
import {
Given,
When,
Then
} from "#badeball/cypress-cucumber-preprocessor";
Given ("I visit Wikipedia", () => {
cy.visit('https://www.wikipedia.org');
});
When ("I search something", () => {
cy.get("input[type='search']").type('Music');
cy.get("button[type='submit']").click();
});
Then ("I check if it is there", () => {
cy.get('.mw-page-title-main').contains('Music')
});
Test2:
import {
Given,
When,
Then,
Before,
After
} from "#badeball/cypress-cucumber-preprocessor";
Before(() => {
cy.log('Execeuted BEFORE Scenario in Test 2');
});
After(() => {
cy.log('Execeuted AFTER Scenario in Test 2');
});
Given ("I visit my uncle", () => {
cy.log("Going to see uncle")
});
When ("I tell him stories", () => {
cy.log("He laughs at my stories")
});
Then ("I eat his food", () => {
cy.log('He is happy and gives me food');
});
When I run Test1, I get the Before and After Hooks being logged in my test and report even though they are not defined in Test1.
The reason why this is an issue for me is because if I am working on a large project with many files with different hooks, these will all be logged in my report which I don't want. I wish to have only the hooks in the test that is being called to be displayed.
Please is there a way of implementing Before and After Hooks without it logging if the test being run does not have them? Thanks.
I am using Cypress v10.11.0, cucumber by Badeball and multiple-cucumber-html-reporter
I tried running what I have in the hooks in each scenario and deleted the hooks, to avoid the Hooks from being logged but it feels reduntant.
I would suggest you to use tags for it. Here are some dummy code snippet for your understanding:
const {After, Before} = require('#cucumber/cucumber') // you can use the cypress cucumber preprocessor here
Before({tags: "#foo and #bar"}, function () {
// This hook will be executed before scenarios tagged with #foo and #bar
});
Before({tags: "#foo or #bar"}, function () {
// This hook will be executed before scenarios tagged with #foo or #bar
});
For reference, please see the documentation:
https://github.com/cucumber/cucumber-js/blob/main/docs/support_files/hooks.md
let's imagine we have a promise that does a large amounts of operations and return helper functions.
A banal example:
const testPromise = testFn => () => {
const helper = Promise.resolve({testHelper: () => 'an helper function'}) // I/O Promise that returns an helper for testing
return helper.then(testFn).finally(() => console.log('tear down'));
}
// This describe would work as expected
describe('Async test approach', () => {
it('A test', testPromise(async ({testHelper}) => {
expect(testHelper()).toBe('an helper function')
}))
})
// This part doesn't work
describe('Async describe approach', testPromise(async ({testHelper}) => {
it('Test 1', () => {
expect(testHelper()).toBe('an helper function')
})
it('Test 2', () => {
expect(testHelper()).not.toBe('A chair')
})
}))
}
What I would like to achieve is something like the second example where I can use async code within describe without re-evaluating testPromise.
describe doesn't handle async so I am not even able to loop and create dynamic tests properly.
I did read many comments around saying that describe should only be a simple way to group tests but... then... how can someone make async generated tests based on I/O result?
Thanks
= ADDITIONAL CONSIDERATION =
Regarding all the comment you guys kindly added, I should have added few additional details...
I am well aware that tests must be defined synchronously :), that is exactly where problems starts. I totally disagree with that and I am trying to find an alternative that avoids before/after and doing it without specifying an external variable. Within Jest issues there was an open one to address that, it seems they did agree on making describe async but they won't do it. The reason is... Jest is using Jasmine implementation of describe and this "fix" should be done in there.
I wanted to avoid beforeAll and afterAll, as much as I could. My purpose was creating an easy (and neat) way to define integration tests tailored on my needs without letting users to worry about initialize and tear down stuff around. I will continue to use the Example 1 above style, that seems the best solution to me, even if it would be clearly a longer process.
Take a look at Defining Tests. The doc says:
Tests must be defined synchronously for Jest to be able to collect your tests.
This is the principle for defining test cases. Which means the it function should be defined synchronously. That's why your second example doesn't work.
Some I/O operations should be done in beforeAll, afterAll, beforeEach, afterEach methods to prepare your test doubles and fixtures. The test should be isolated from the external environment as much as possible.
If you must do this, maybe you can write the dynamically obtained testHelper function to a static js file, and then test it in a synchronous way
As it was noted, describe serves to group tests.
This can be achieved with beforeAll. Since beforeAll should be called any way, it can be moved to testPromise:
const prepareHelpers = (testFn) => {
beforeAll(() => {
...
return helper.then(testFn);
})
}
describe('Async describe approach', () => {
let testHelper;
prepareHelpers(helpers => { testHelper = helpers.testHelper });
...
I am new to jasmine testing and coming from a xUnit .Net background.
Is it possible to label a test or a suite of tests in such a fashion:
[SomeAttribute]
describe("A suite", function() {
it("contains spec with an expectation", function() {
expect(true).toBe(true);
});
});
Does jasmine support any sort of attributes or identifiers? My goal really is to run a describe group of tests twice, with a different setting between test runs. I did not want to duplicate the tests. Is it possible for a test to kick off other tests?
This question is assuming that I am satisfied with duplicating a build step to run the test suit twice, just with a subset of tests for the second run.
Edit: More realistic example of how I would hope to consume it
[Theory]
[TestData(true)]
[TestData(false)]
describe("A suite", function() {
beforeEach(() => {
configureTestBed(/*someHow get input*/);
});
it("contains spec with an expectation", function() {
expect(true).toBe(true);
});
});
What you can do is to define separate functions which will accept parameters from somewhere else in your code.
Something like this would do:
describe('Sample describe', () => {
testFunction(1);
});
function testFunction(param1) {
it('should execute test with params', () => {
console.log(param1);
expect(param1).toBe(1);
});
}
Is there a way to cherry pick the tests and create suits in protractor/jasmine e2e test. I know protractor accepts with wildcard specs (*.spec") in the suites, but I am looking for select few tests in spec files and create a suit to run on protractor. Any help on this is greatly appreciated.
suites can only group test files. Though, I would still think about regrouping the specs inside tests, or splitting them into multiple files so that suites can be used here - it is a great way to organize your tests and group them logically.
If you want to run specific it() blocks/specs from different files as a part of a group - tag them:
describe("test1", function () {
it("should test something (#mytag)", function () {
});
});
describe("test2", function () {
it("should test something else (#mytag)", function () {
});
});
And run with --grep:
protractor conf.js --grep "#mytag"
See also:
Running specs by tag
Alternatively, use focused specs (fdescribe/fit, or ddescribe/iit):
describe("test1", function () {
fit("should test something", function () {
});
});
describe("test2", function () {
fit("should test something else", function () {
});
});
This question relates to the Mocha testing framework for NodeJS.
The default behaviour seems to be to start all the tests, then process the async callbacks as they come in.
When running async tests, I would like to run each test after the async part of the one before has been called.
How can I do this?
The point is not so much that "structured code runs in the order you've structured it" (amaze!) - but rather as #chrisdew suggests, the return orders for async tests cannot be guaranteed. To restate the problem - tests that are further down the (synchronous execution) chain cannot guarantee that required conditions, set by async tests, will be ready they by the time they run.
So if you are requiring certain conditions to be set in the first tests (like a login token or similar), you have to use hooks like before() that test those conditions are set before proceeding.
Wrap the dependent tests in a block and run an async before hook on them (notice the 'done' in the before block):
var someCondition = false
// ... your Async tests setting conditions go up here...
describe('is dependent on someCondition', function(){
// Polls `someCondition` every 1s
var check = function(done) {
if (someCondition) done();
else setTimeout( function(){ check(done) }, 1000 );
}
before(function( done ){
check( done );
});
it('should get here ONLY once someCondition is true', function(){
// Only gets here once `someCondition` is satisfied
});
})
use mocha-steps
it keeps tests sequential regardless if they are async or not (i.e. your done functions still work exactly as they did). It's a direct replacement for it and instead you use step
I'm surprised by what you wrote as I use. I use mocha with bdd style tests (describe/it), and just added some console.logs to my tests to see if your claims hold with my case, but seemingly they don't.
Here is the code fragment that I've used to see the order of "end1" and "start1". They were properly ordered.
describe('Characters start a work', function(){
before(function(){
sinon.stub(statusapp, 'create_message');
});
after(function(){
statusapp.create_message.restore();
});
it('creates the events and sends out a message', function(done){
draftwork.start_job(function(err, work){
statusapp.create_message.callCount.should.equal(1);
draftwork.get('events').length.should.equal(
statusapp.module('jobs').Jobs.get(draftwork.get('job_id')).get('nbr_events')
);
console.log('end1');
done();
});
});
it('triggers work:start event', function(done){
console.log('start2');
statusapp.app.bind('work:start', function(work){
work.id.should.equal(draftwork.id);
statusapp.app.off('work:start');
done();
});
Of course, this could have happened by accident too, but I have plenty of tests, and if they would run in parallel, I would definitely have race conditions, that I don't have.
Please, refer to this issue too from the mocha issue tracker. According to it, tests are run synchronously.
I wanted to solve this same issue with our application, but the accepted answer didn't work well for us. Especially in the someCondition would never be true.
We use promises in our application and these made it very easy to structure the tests accordingly. The key however is still to delay execution through the before hook:
var assert = require( "assert" );
describe( "Application", function() {
var application = require( __dirname + "/../app.js" );
var bootPromise = application.boot();
describe( "#boot()", function() {
it( "should start without errors", function() {
return bootPromise;
} );
} );
describe( "#shutdown()", function() {
before( function() {
return bootPromise;
} );
it( "should be able to shut down cleanly", function() {
return application.shutdown();
} );
} );
} );