I am looking into mutation testing and trying to integrate Stryker into my code base. My application is written in React, Nodejs and currently using Jest for client-side testing and Mocha for server-side testing. I am having a few questions regarding this:
Have anybody tried/looked into mutation testing before? Do you have any thoughts/concerns on this regarding pros and cons?
In terms of Stryker framework, I am curious how Stryker generate mutants? Is there any algorithm used in Stryker to generate mutants?
Any input would be much appreciated. Thank you in advance.
For generic Mutation Testing concepts I totally recomend Pitest (MT implementation for java) docs:
https://pitest.org/quickstart/
https://github.com/hcoles/pitest/blob/master/so_you_want_to_build_mutation_testing_system.md
I also wrote on the topic sometime ago: https://pedrorijo.com/blog/intro-mutation/
For your questions, this should give you a good idea of the existing possibilities. Not sure about striker specific details
Mutation testing is a great tool to improve test "coverage" in a meaningful way. If you want more complete regression coverage in an automated unit test suite, mutation testing can help you find holes in your test coverage, and this can help find and prevent bugs. Be aware though: You still need to think about the types of testing you need. Simply adding tests just to kill off mutants can be damaging. Your tests can end up overly coupled to your implementation this way. However, that doesn't make mutation testing bad, it just means you have to use the tool intelligently, like most tools. Another caveat is that mutation testing can be slow, but as you don't have to run mutation tests as often as the regular test suite, that's not a terrible problem.
Stryker uses specific mutators that look for code that it knows how to mutate. You can find the list of mutators in the handbook. If you are feeling adventurous, you can add your own mutators. But the basic way the algorithm works is:
parse the code into something that can be inspected by the algorithm
look for pieces of code that fit a specific template
modify that code in a way that should still be valid syntax, but changes the behavior in a way that is probably "wrong". (The simplest example is to remove behavior entirely)
Related
Background
I'm working on a single-page application with a JavaScript code base that is well-tested at the unit level, and want to add a functional test suite to verify application behavior closer to the level of what the user sees. At this time, I'm not looking at full end-to-end testing (I'd expect to stub/mock interactions with the back-end) but that may be a goal later on.
Our build is run by npm; gulp/grunt are not currently in use but may be adopted soon as our build complexity grows. We are using Jasmine and Blanket for testing and coverage currently, but for purposes of this question I'm fairly tool-agnostic.
Goal
What I'd like to do is generate code coverage that is comprehensive across both categories of automated tests. That is, if I have a line of code that is exercised during functional testing but isn't exercised in unit tests, my reporting should still mark that as "covered." (If I can get more or deeper information about how code gets exercised, great, but not needed.)
The underlying goal (since XY problems are so common on SO) is to create a forcing function for test completeness ("code coverage must be above X%") without forcing redundancy (as might happen if coverage was measured separately for both) or introducing a false preference for one category of testing (as might happen if coverage was measured only for one category of testing.) There's value to testing the same code in different circumstances, of course, but we are resource-limited and want to focus on the tests that will add the most value.
Question(s)
The short question is "how do I do this?" I haven't found any documentation or examples of this being done. I feel like the answer is "just stick istanbul in front of X which runs both kinds of tests", but don't really know what X looks like.
Some more specific questions I have in mind:
Are there any examples which could be followed of comprehensive coverage being obtained for testing across multiple categories?
What are the constraints one must accept to make this kind of testing feasible?
Am I wrong to think of unit tests and functional tests as separate suites being run in different ways? (The problem becomes trivial if functional and unit tests are not separate suites, but are distinctions one makes on a test-by-test basis. But that's not the sort of test suite I'm used to seeing.)
Is there a reason not to do this? (The fact that I can't find examples of this being done feels conspicuous to me.)
I'm reasonably familiar with the tools that are out there in this category (Karma, Mocha, Jasmine, Blanket, Istanbul, etc...) so I'm not looking for recommendations, per se, but rather some guidance on how to achieve this kind of configuration. However, I'd certainly be interested to hear if some tools are better- or ill-suited for this sort of usage.
I have a javascript app within 250 lines and want to add tests to that. Whenever i make a tiny change, i have to run tests for atleast 10 cases manually, which i want to automate.
I could have gone for frameworks as suggested in different posts, but the solution i want is minimum friction and codebase. Something like a single file for unit testing may work.
Is there a way for using JS testing without any frameworks ? I want to write Unit tests / functional tests both. Or if Frameworks is the only option, what frameworks are preferred in terms os ease of plugin to existing code + learning curve.
Unit Testing frameworks are helpful because the provide environments that will run the tests and give you feedback about them. To not use one would be to take implementation of those responsibilities on yourself and would be much more work than it sounds like you are trying to do.
If you are using node, setting up a unit testing framework is very easy. I like to use Karma as a test runner with the mocha testing framework and chai assertion library.
I would go with a framework. While your use case might be simple now, you might find yourself in a project where you need more functionality down the road. You should be familiar with at least one testing framework/library for any language you use seriously. The same goes for build systems and package management.
For javascript, I've only used mocha. It's pretty comprehensive, easy to learn, can be used in browser or with node, and tests actually look really clean and easy to read.
Also, with mocha, you can choose your own assertions library, I use chai. It's built with tdd and bdd in mind, and it makes writing assertions feel more like writing natural sentences than code. Of course, you can still use whatever default assertion facilities your js environment gives you, if any.
its really hard to setup automated unit tests with plain javascript without any framework. if you try to reinvent the wheel of existing javascript unit testing framework yourself, that might be huge effort than your 250 lines of code for which you are writing test.
so if you are going for an framework, jasmine and qunitjs are promising unittesting framework for javascript.
http://jasmine.github.io/
https://qunitjs.com/
I'm building several jQuery plugin and going ahead I thought that will be useful to know some best practice for testing my code in order to be solid.
I usually use Selenium but this just for debugging.
I'd like to know the best way for doing some real test (like TDD, BDD, DDD) for my code.
Any suggestion will be appreciated.
I use Jasmine https://github.com/pivotal/jasmine and would highly recommend it.
I don't think there is a "best" framework for test your code. I just know that you "should" test your code. So with that said, just find a test/spec framework your like and get testing!
I also at times use https://github.com/webadvanced/jMoney for testing small bits of code or for testing prototype code. It displays results in the console and does not require any setup or dependent files.
If you are more of a TDD guy, you can try QUnit http://docs.jquery.com/QUnit
So, you are using a bunch of javascript libraries in a website. Your javascript code calls the several APIs, but every once in a while after an upgrade, one of the API changes, and your code breaks, without you knowing it.
How do you prevent this from happening?
I'm mostly interested in javascript, but any answer regarding dynamically typed languages would be valuable.
I don't think there's much you can do. You always run a risk when updating any piece of software. The best advice is to:
Read and understand documentation about upgrading
Upgrade in your test environment
TEST
Roll out live when you are happy there are no regressions
You should consider building unit tests using tools such as JsUnit and Selenium. As long as your code passes the tests, you're good to go. If some tests fail, you would quickly identify what needs to be fixed.
As an example of a suite of Selenium tests, you can check the Google Maps API Tests, which you can download and run locally in your browser.
Well there are two options:
Don't upgrade
Retest everything after you upgrade.
There is no way to guarantee that an upgrade won't break something. Even if you have something that could check the underlying API and make sure it still all lines up, you can't be certain that the underlying functionality is the same.
I'd like to start a discussion on the best resources for refactoring tips, with an eye towards Front-End JavaScript refactoring.
A friend whose opinion I respect suggests this book, though it uses examples in Java. I understand that the principles of OO refactoring should translate to another language.
Let's talk about refactoring in general and JS refactoring specifically.
Check out Martin's Clean Code for some inspiration. The examples are in Java but the ideas should translate to JavaScript as well.
In order to refactor properly, you'll need to test your code well as having proper tests in place is a prerequisite for it. Try finding testing tools that give you code coverage as this will be extremely valuable.
In case your code hasn't been tested yet you can think testing as a method of learning. You write tests to prove or disprove your assumptions about it. Once you are done and covered it adequately you should be able to refactor the code using various patterns provided.
As Ira mentioned having a tool to detect clones may be valuable. That's definitely one way to look at it.
I think that often having proper perspective is half the solution. If you can state your design in clear terms, you'll end up with better results. Try not to over-engineer it too much by applying every possible pattern you find. :)
Not specifically a refactoring tool, but a tool to detect the number one smell in code: our CloneDR automatically finds duplicated code (copy/paste/edit) and shows you the precise locations and a reasonable abstraction of each clone. You have to resolve the clones by hand, but knowing where they are and having been provided that first cut at an abstraction, is a big help.
EDIT: This is directly a resource for refactoring of Java and JavaScript code. CloneDR works for both of them, using langauge-precise parsing to give precise analysis of clones.
You can see sample clone detection results on Google's Javascript Closure package.