Background
I'm working on a single-page application with a JavaScript code base that is well-tested at the unit level, and want to add a functional test suite to verify application behavior closer to the level of what the user sees. At this time, I'm not looking at full end-to-end testing (I'd expect to stub/mock interactions with the back-end) but that may be a goal later on.
Our build is run by npm; gulp/grunt are not currently in use but may be adopted soon as our build complexity grows. We are using Jasmine and Blanket for testing and coverage currently, but for purposes of this question I'm fairly tool-agnostic.
Goal
What I'd like to do is generate code coverage that is comprehensive across both categories of automated tests. That is, if I have a line of code that is exercised during functional testing but isn't exercised in unit tests, my reporting should still mark that as "covered." (If I can get more or deeper information about how code gets exercised, great, but not needed.)
The underlying goal (since XY problems are so common on SO) is to create a forcing function for test completeness ("code coverage must be above X%") without forcing redundancy (as might happen if coverage was measured separately for both) or introducing a false preference for one category of testing (as might happen if coverage was measured only for one category of testing.) There's value to testing the same code in different circumstances, of course, but we are resource-limited and want to focus on the tests that will add the most value.
Question(s)
The short question is "how do I do this?" I haven't found any documentation or examples of this being done. I feel like the answer is "just stick istanbul in front of X which runs both kinds of tests", but don't really know what X looks like.
Some more specific questions I have in mind:
Are there any examples which could be followed of comprehensive coverage being obtained for testing across multiple categories?
What are the constraints one must accept to make this kind of testing feasible?
Am I wrong to think of unit tests and functional tests as separate suites being run in different ways? (The problem becomes trivial if functional and unit tests are not separate suites, but are distinctions one makes on a test-by-test basis. But that's not the sort of test suite I'm used to seeing.)
Is there a reason not to do this? (The fact that I can't find examples of this being done feels conspicuous to me.)
I'm reasonably familiar with the tools that are out there in this category (Karma, Mocha, Jasmine, Blanket, Istanbul, etc...) so I'm not looking for recommendations, per se, but rather some guidance on how to achieve this kind of configuration. However, I'd certainly be interested to hear if some tools are better- or ill-suited for this sort of usage.
Related
I am looking into mutation testing and trying to integrate Stryker into my code base. My application is written in React, Nodejs and currently using Jest for client-side testing and Mocha for server-side testing. I am having a few questions regarding this:
Have anybody tried/looked into mutation testing before? Do you have any thoughts/concerns on this regarding pros and cons?
In terms of Stryker framework, I am curious how Stryker generate mutants? Is there any algorithm used in Stryker to generate mutants?
Any input would be much appreciated. Thank you in advance.
For generic Mutation Testing concepts I totally recomend Pitest (MT implementation for java) docs:
https://pitest.org/quickstart/
https://github.com/hcoles/pitest/blob/master/so_you_want_to_build_mutation_testing_system.md
I also wrote on the topic sometime ago: https://pedrorijo.com/blog/intro-mutation/
For your questions, this should give you a good idea of the existing possibilities. Not sure about striker specific details
Mutation testing is a great tool to improve test "coverage" in a meaningful way. If you want more complete regression coverage in an automated unit test suite, mutation testing can help you find holes in your test coverage, and this can help find and prevent bugs. Be aware though: You still need to think about the types of testing you need. Simply adding tests just to kill off mutants can be damaging. Your tests can end up overly coupled to your implementation this way. However, that doesn't make mutation testing bad, it just means you have to use the tool intelligently, like most tools. Another caveat is that mutation testing can be slow, but as you don't have to run mutation tests as often as the regular test suite, that's not a terrible problem.
Stryker uses specific mutators that look for code that it knows how to mutate. You can find the list of mutators in the handbook. If you are feeling adventurous, you can add your own mutators. But the basic way the algorithm works is:
parse the code into something that can be inspected by the algorithm
look for pieces of code that fit a specific template
modify that code in a way that should still be valid syntax, but changes the behavior in a way that is probably "wrong". (The simplest example is to remove behavior entirely)
The older Firefox "Add-ons" API had a built-in unittest layer sdk/test that allowed testing. This doesn't seem to be available any more.
Additionally the use of "package/require" allowed code to be separated into "js code-only" packages that were testable using node.js. The new, highly structured javascript doesn't share this.
My priorities are (highest to lowest):
Algorithms, "business logic", e.g. parsing input data - no APIs needed - just JavaScript
Internal logic - e.g. background scripts interacting with settings, etc.
UI interactions - I can live without this, but would be nice to test
So how do people test their WebExtensions?
Check out webextension-geckodriver for a worked example of functional testing.
If you want to test interaction with the webextension API you can either do it live (have a test page for your extension and get geckodriver to visit it, for example) or use a fake like sinon-webextension through webextension-jsdom.
To unit test algorithms, just import the functions using jest, mocha, or whatever node unit testing framework you prefer or add them to a test page that you can visit in the browser.
A complete, but old, worked example of webext testing is here: example-webextension.
An example of tests in a real webextension using another fake: vim-vixen
It is still possible to run unit tests in NodeJs.
To illustrate the idea, let us take a look at the Cliqz extension, whose source code is open (Github link: cliqz-oss/browser-core). In comparison other extensions that I have seen so far, the code base is quite large.
In other words, it is not a toy example, but a realistic use case. The drawback, of course, is that due to the complexity, it is harder to understand how the test setup works (how mocking works, the integration into the build system, etc).
To get an idea how tests look like, here is one example:
source: query-sanitizer.es
test: query-sanitizer-test.es
To explain the details of how the mocking works is hard because you would need to dive deep into the build system. From a high level perspective, you will notice that in the test it uses a function called describeModule, which does all the mocking of dependencies.
In the implementation of describeModule, you can see that it uses systemjs to dynamically load ES modules. That trick makes it possible to run the unit tests with NodeJs.
My priorities are (highest to lowest):
Algorithms, "business logic", e.g. parsing input data - no APIs needed - just JavaScript
For these kind of tests, the unit test infrastructure described above is the preferred way. For local development, NodeJs is used to run the tests.
Internal logic - e.g. background scripts interacting with settings, etc.
This is not so different. The idea is still that you can mock dependencies and then do classical unit tests.
It might require some work to allow dependencies to be replaced. As mentioned, as the code base in this example has to run in different environments (e.g., Firefox, Chrome, Edge, ReactNative), platform APIs have to be abstracted (that includes also browser APIs).
UI interactions - I can live without this, but would be nice to test
For testing the UI, there are additional integration tests. I do not want to go into details, but there are examples in the code.
What is important is that the integration tests are not executed with NodeJs, but they require a real browser environment (e.g., Firefox, Chrome). In addition, a local HTTP server is started which can be used to mock API calls.
As a side-note, linters are extremely useful and in comparison easy to setup. Also consider using a typed language (TypeScript), especially when the project becomes bigger and more people are working on it.
You will still need tests, as static analysis will not be able to find logical bugs. However, it helps to eliminate certain types of simple bugs like typos and the overhead (fixing linter errors or adding type annotations) is not very high.
At the moment I try to work with my dev-teammates on some modular system for our javascript solutions. Because we're able to start afresh, we'd like to do it right this time - with tests!
We found Jasmine and Karma for our Unit Tests and Selenium/Nightwatch for our End-to-End tests.
While I was writing unit tests for those components of the system which never touched any DOM through jQuery I was good to go. But one day I came along some components which manipulates the DOM. So far so good. I managed to test them as well thanks to Jasmine-jQuery. Until this point I was sure to still be within the boundaries of Unit-Testing.
Yesterday though, I was sitting in front of a component which will make a navigation bar sticky (or fixed) as soon as the user is about to scroll the webpage down.
I wanted to test this functionality with Jasmine-jQuery again. What I did was - I mocked the users scrolling with "window.scrollTo(0, 2000)" and then checked if the attributes of those navigation bar have been changed.
The Issue:
My teamleader put me on hold because I have stepped over to the domain for End-To-End tests because I needed to mock browser functionality.
My Questions: Is this really the case? In my optinion an End-To-End test should test the orchestration of several functionalities of a system (like ours) within a productive environment. Therefore User-Stories would be the layer I would test with End-To-End tests. Check if the path a user has to go to login and write an article (for example) works the way as intended.
But checking if a javascript component will successfully add/remove attributes to the DOM after some event (like the scroll event) happened - should be a unit test still.
I'm an apprentice developer - I respect the experience of my teamleader - but I still want to make completely sure that things will be done the right way.
So I ask you if you might tell me when Unit-Testing ends and when End-To-End tests begin when writing JavaScript and manipulating the DOM.
Some teammate explained to me that it might be a good way to realize if its a E2E Test specific function if you check how critical a malfunction would affect user experience. But only if you really struggle between unit-testing and E2E. Then you should ask yourself "Would a fail result in a really bad user experience or will only some error be thrown in the console and a little picture wouldn't be loaded properly".
Here you can find a damn good explanation on the difference between tests: What's the difference between unit, functional, acceptance, and integration tests?.
I think Selenium with WebDriver is the best solution for testing E2E. If you're using AngularJS, a very good solution is using Protractor (https://angular.github.io/protractor/#/).
Hope this can help you
Check out this article http://www.itaware.eu/2012/10/19/angularjs-unit-tests-and-end-to-end-tests/ . It deals with Angular specifically but clarifies on the difference between the two. Your team leader is definitely right, its not the fact that your manipulating the DOM that makes it an E2E type tests, its the fact that your simulating user interface.
Can anyone give me examples of large-scale JS apps (including AJAX, different UI widgets, and a sophisticated architecture) with unit tests?
I'm not talking about Selenium tests here, just plain ol' stupid unit tests using mocks, decent result reporting and such.
Not sure why people voted to close, or downvoted the question. Maybe a comment would be nice.
Seriously, I've been trying hard to find unit tested web apps, since I'm having a hard time building mocks and I wonder if it's even possible with reasonable effort. It made me think about the benefits of unit tests on widgets as compared to Selenium tests. People are babbling a lot about unit tests in theory but evidently nobody actually has done it in JS-RIAs. Or have they?
Personally I like Qooxdoo, check it out for your self and see if this is what you want
http://qooxdoo.org/demo#real-life_examples
This is one good tool: http://www.uize.com/
You should look at Jasmine & Sinon.js : http://sinonjs.org/
Here is a good tutorial on testing using Backbone.js, Jasmine & Sinon.js : http://tinnedfruit.com/2011/03/03/testing-backbone-apps-with-jasmine-sinon.html
I also recommend Phantom.js for integration testing... It's a headless browser and much faster than using Selenium... http://www.phantomjs.org/
Btw here is an example of unit-tested app from Pivotal : https://github.com/pivotal/cimonitor. You can find client-side tests there -> cimonitor/public/javascripts/js-common
I'm not sure if this answer will qualify but I'm working on the next iteration of my pet project "Atomic OS" (an OS-metaphor for web developers) which will, eventually, meet your criteria.
I'm working on a related project (which I can't share just yet) that is built on a bare-bones Atomic OS v2 foundation and provides a rich set of UI widgets for mobile web apps.
I built & use JSDog to produce documentation from a subset of JSDoc syntax and unit test runners with QUnit.
For an example of where I'm intending to go with unit tests, please see the Atomic OS documentation. (Click "Docs" in the taskbar and select a class, such as HxJSFS)
Just one perspective:
I work on a web application that is the front end of a video analytics system. (The back end is typically an IP camera, DVR or video router running a very, very lean, embedded web server.) It uses a number of jQueryUI widgets, allows user to configure the device, create video analysis rules, and draw markup over video frames using canvas elements. I think of it as fairly sophisticated.
We use unit tests (originally written for JSUnit, but now using qunit) for a very limited subset of the code. We have unit tests to verify the behavior of business objects, including the ability to serialize/deserialize to/from XML. And we have unit tests to test the basic geometry classes we've written for the canvas markup.
However, we have no unit tests that manipulate the DOM or that verify that the elements on a page are in the correct state. Doing that correctly struck us as too difficult a problem to solve, so we rely on Selinium tests to verify that a given set of inputs will put the DOM into the correct state.
I have a very large website that I need to refactor the front-end of due to maintainability concerns and performance issues:
What's tricky:
Lots of overcomplicated scripts
Over-complicated CSS with a whopping filesize
No selenium tests in place
No js tests in place
Back end developers concerned anything breaking
The site has been live for a while and the client is happy
Tools available:
Multiple servers for testing versions on
Continuos integration setup
Version control
Some thoughts (after having done also bigger refactorings though more on backend side):
create a play-branch where you start to "scribble" refactor. there you can see how the refactoring feels. I often find that especially big refactorings are like starting design/programming. You have to get hands on and see how your refactoring target feels like. This also has great to better estimate the overall effort.
as your client is happy and it is more an investment for future take the incremental approach. Do not do a big-bang refactoring. Try to partition several refactoring areas (e.g. javascript vs. CSS) and also by pages.
start with the easier refactorings steps. They help to get you "fit" with the harder ones.
the scm is one of your biggest friends. It is vital to revert quickly and feel safe when refactoring. Even on your refactoring branch for bigger works you can create further "minor-branches" to go on with bigger refactoring steps.
regression tests are vital. Get tests in place first. 100% test automation is hardly possible, so get one of your QA/Test team-members to help you along the refactoring and regression-test after each bigger refactoring-step. The 'it doesn't work anymore' after 3 months of refactoring would just smash you.
for more difficult refactorings (hard to test) and flaky code structure do 'pair programming'.
The fear backend developers not trusting big refactoring is correct, take them serious and ask for help (especially for the regression testing requirements).
Well, it seems to me that the main problem you're facing is a lack of exiting JavaScript/Selenium tests. Once you've got those in place, you can start the real work of refactoring, taking small steps - confident that mistakes will be picked up.
Never refactor without having tests in place first.
Well I would start out copying the project to a new workspace and creating a place for your version control. This way your back end developers don't need to be scared anymore. Next I would look at the css files because you can change those and see the difference right away. When you like the css you have then you can go on to the script part.
Here you definitely need tests because else there will certainly be some errors or changes which are not wanted. I personaly would migrate the script in very(!) small junks so you are able to find errors very fast and run tests for every little thing you changed. This will lead to the most persistent outcome in my experience.
Another thing you need to think of is the deployment. I mean there are always some requirements or dependencies (I don't know if that's the case in your problem but you should really check that and work with your back end guys so they can give you some hints where the problems could be).
Here are some ideas:
Spend a few hours going over the site, clicking through, getting a sense of functionality
Zero into the core page, or pages you want to focus your main improvements on. Spend an hour on that
Start looking at what's more important:
A) maintainability / quick ability for you or someone else to come in and extend functionality
B) page load performance - how you might reduced it by 100kb plus, and consider if its really worth it based on the type of site it is
C) ui latency. How sticky, intermittent transitions, how slow the site feels once loaded.
Come up with a figure in your head. Is it a two day job for you? 2 weeks? What benefit will it have for all of the above? Now consider how long it might take some new developer you have hired who will now have to go through the same process as you and be at your level.
Once you feel you know all the possible user journeys and variations of a particular component you have singled out, consider how long it would take you to do the following:
recreating the same file hierarchy of a single page onto your own server, outside of the existing slow/unreliable/changing environment/live site or whatever so you can accelerate the speed in which you are about to work on this (approx 3/4 day)
switching headspace between development/research/documenting anything as you go/ interruptions, managing other projects etc (approx 1hr)
create whatever tests you want and develop from scratch and emulating an existing piece of fe architecture, written exactly how you would have wanted it. (1 day per component - html, accessibility, sass, js etc)
consider how you could integrate this single component into the existing architecture/environment and remove the existing piece without breaking anything around it.
integrate it back into the environment. Time is money, but thorough investigation before embarking on a project will protect and prepare you for that important discussion with the business owner.