At the moment I try to work with my dev-teammates on some modular system for our javascript solutions. Because we're able to start afresh, we'd like to do it right this time - with tests!
We found Jasmine and Karma for our Unit Tests and Selenium/Nightwatch for our End-to-End tests.
While I was writing unit tests for those components of the system which never touched any DOM through jQuery I was good to go. But one day I came along some components which manipulates the DOM. So far so good. I managed to test them as well thanks to Jasmine-jQuery. Until this point I was sure to still be within the boundaries of Unit-Testing.
Yesterday though, I was sitting in front of a component which will make a navigation bar sticky (or fixed) as soon as the user is about to scroll the webpage down.
I wanted to test this functionality with Jasmine-jQuery again. What I did was - I mocked the users scrolling with "window.scrollTo(0, 2000)" and then checked if the attributes of those navigation bar have been changed.
The Issue:
My teamleader put me on hold because I have stepped over to the domain for End-To-End tests because I needed to mock browser functionality.
My Questions: Is this really the case? In my optinion an End-To-End test should test the orchestration of several functionalities of a system (like ours) within a productive environment. Therefore User-Stories would be the layer I would test with End-To-End tests. Check if the path a user has to go to login and write an article (for example) works the way as intended.
But checking if a javascript component will successfully add/remove attributes to the DOM after some event (like the scroll event) happened - should be a unit test still.
I'm an apprentice developer - I respect the experience of my teamleader - but I still want to make completely sure that things will be done the right way.
So I ask you if you might tell me when Unit-Testing ends and when End-To-End tests begin when writing JavaScript and manipulating the DOM.
Some teammate explained to me that it might be a good way to realize if its a E2E Test specific function if you check how critical a malfunction would affect user experience. But only if you really struggle between unit-testing and E2E. Then you should ask yourself "Would a fail result in a really bad user experience or will only some error be thrown in the console and a little picture wouldn't be loaded properly".
Here you can find a damn good explanation on the difference between tests: What's the difference between unit, functional, acceptance, and integration tests?.
I think Selenium with WebDriver is the best solution for testing E2E. If you're using AngularJS, a very good solution is using Protractor (https://angular.github.io/protractor/#/).
Hope this can help you
Check out this article http://www.itaware.eu/2012/10/19/angularjs-unit-tests-and-end-to-end-tests/ . It deals with Angular specifically but clarifies on the difference between the two. Your team leader is definitely right, its not the fact that your manipulating the DOM that makes it an E2E type tests, its the fact that your simulating user interface.
Related
Background
I'm working on a single-page application with a JavaScript code base that is well-tested at the unit level, and want to add a functional test suite to verify application behavior closer to the level of what the user sees. At this time, I'm not looking at full end-to-end testing (I'd expect to stub/mock interactions with the back-end) but that may be a goal later on.
Our build is run by npm; gulp/grunt are not currently in use but may be adopted soon as our build complexity grows. We are using Jasmine and Blanket for testing and coverage currently, but for purposes of this question I'm fairly tool-agnostic.
Goal
What I'd like to do is generate code coverage that is comprehensive across both categories of automated tests. That is, if I have a line of code that is exercised during functional testing but isn't exercised in unit tests, my reporting should still mark that as "covered." (If I can get more or deeper information about how code gets exercised, great, but not needed.)
The underlying goal (since XY problems are so common on SO) is to create a forcing function for test completeness ("code coverage must be above X%") without forcing redundancy (as might happen if coverage was measured separately for both) or introducing a false preference for one category of testing (as might happen if coverage was measured only for one category of testing.) There's value to testing the same code in different circumstances, of course, but we are resource-limited and want to focus on the tests that will add the most value.
Question(s)
The short question is "how do I do this?" I haven't found any documentation or examples of this being done. I feel like the answer is "just stick istanbul in front of X which runs both kinds of tests", but don't really know what X looks like.
Some more specific questions I have in mind:
Are there any examples which could be followed of comprehensive coverage being obtained for testing across multiple categories?
What are the constraints one must accept to make this kind of testing feasible?
Am I wrong to think of unit tests and functional tests as separate suites being run in different ways? (The problem becomes trivial if functional and unit tests are not separate suites, but are distinctions one makes on a test-by-test basis. But that's not the sort of test suite I'm used to seeing.)
Is there a reason not to do this? (The fact that I can't find examples of this being done feels conspicuous to me.)
I'm reasonably familiar with the tools that are out there in this category (Karma, Mocha, Jasmine, Blanket, Istanbul, etc...) so I'm not looking for recommendations, per se, but rather some guidance on how to achieve this kind of configuration. However, I'd certainly be interested to hear if some tools are better- or ill-suited for this sort of usage.
My TLDR; version of my question is "Is there a way I can integrate with qunit such that Dalek can get the correct context when it needs it, or conversely, can I get Dalek to run setup/teardown asset-pipeline-compiled Ember javascript to build a context for it to run tests on?"
Firstup Dalek look awesome! All my tests are currently written in qunit. I'm having some problems automating tests around a component I'm building in Ember. The component is a kind of WYSIWYG textarea.
(BTW, my qunit tests are being driven from a route within a rails application.)
To automate testing, my qunit scripts have a setup and tearDown that create a pristine textarea each time. Each test creates some content in the textarea, then interacts with it somehow doing some assertions on it.
That's all well and good, except that I require much better browser simulation than qunit can provide me with (and I'm really running out of patience for writing my own range-related browser-response simulation code).
The things I need to do mostly are:
1. Move the caret around using arrow keys, and type characters.
2. Click at specific points in the textarea (not x,y co-ords, but rather a specific points in the text).
It struck me that Dalek could totally help with this, but the way I'm doing this workflow, I think I'd either need Dalek to be remote controllable via my qunit tests, or else somehow rewrite my tests in Dalek, but to do that, I'd need to be able to get Dalek to use jQuery and Ember to create the component and data context for the setup/teardown, which I'm not even sure Dalek supports.
What I really need is part-integration, part-unit testing, and there doesn't seem to be a great answer in the JS/Rails/Ember testing space that will handle this set of conditions.
I fear DalekJS is not the tool you need right now, even if it is "my little tool" I suggest to use Karma (former called Testacular) - which was originally developed to test AngularJS applications: http://karma-runner.github.io/0.12/index.html
You could use it together with Protractor https://github.com/angular/protractor
It depends on selenium, but is fairly easy (compared to some other tools) to set up.
There is also a manual on how to use it in combination with Ember: http://karma-runner.github.io/0.10/plus/emberjs.html
I'm having trouble figuring out out how to do Angular e2e testing with the Play Framework or with Play Integration testing. I feel AngularJS loses a lot of value if I can't do this. I have no idea how I would go about doing e2e with Play. And as far as Play integration testing, the webDriver can see the ng attributes but as with the case for the code I tried it doesn't emulate the proper behavior. Anyone have a similar experience or a solution to this?
Or perhaps is there a good alternative for Angular (such as an MVC framework with a good library) that integrates with Play testing better and can help me reduce my amount of lines for front end code? I'm already using coffeescript, jQuery, and Bootstrap. Thanks!
Please be a little bit more precise about "proper behaviour". What is the bug ? Is the problem about integrating e2e tests with the Play workflow or just having e2e AngularJS green instead of red ?
The difficulty here is to define "End-to-end" since there is no good definition of it, in wikipedia there is no mention about it. You can have have a good idea of what it is just because it is self-speaking, and there is also a excellent thread about end-to-end here :
http://www.geekinterview.com/question_details/40319
So let's say that you want to test all scenarios with Front + Back in a close-to-prod environment.
End-to-end testing AngularJS is supposed to be done with Karma, but, warning, it will be soon changed by ProtractorJS.
Doing e2e test with Karma has serious drawbacks :
It may fail if you test your AngularJS app with your backend, more specifically, anything a little bit too slow in the backend may result in green test that sometimes fails with no good reason.
The killer un-feature to do proper e2e testing with Angular is : you can't double-click, roll the mouse, etc.
Here is my advices :
Use Karma only for Front-End testing, and use good old stubs as back-end. Check proper css element are displayed and that correct HTTP request are made. It will give you great confidence on Front-End dev.
DO NOT use Karma for end-to-end test. I found Sikuli absolutely outstanding to do this job. Use it with fitnesse or testNG for better integration.
May be ProtractorJS will do the job in a very next future.
Sikuli: http://www.sikuli.org/
example of fitnesse + sikuli : https://github.com/xebia/FitnesseSikuli
protractor : https://github.com/angular/protractor
Can anyone give me examples of large-scale JS apps (including AJAX, different UI widgets, and a sophisticated architecture) with unit tests?
I'm not talking about Selenium tests here, just plain ol' stupid unit tests using mocks, decent result reporting and such.
Not sure why people voted to close, or downvoted the question. Maybe a comment would be nice.
Seriously, I've been trying hard to find unit tested web apps, since I'm having a hard time building mocks and I wonder if it's even possible with reasonable effort. It made me think about the benefits of unit tests on widgets as compared to Selenium tests. People are babbling a lot about unit tests in theory but evidently nobody actually has done it in JS-RIAs. Or have they?
Personally I like Qooxdoo, check it out for your self and see if this is what you want
http://qooxdoo.org/demo#real-life_examples
This is one good tool: http://www.uize.com/
You should look at Jasmine & Sinon.js : http://sinonjs.org/
Here is a good tutorial on testing using Backbone.js, Jasmine & Sinon.js : http://tinnedfruit.com/2011/03/03/testing-backbone-apps-with-jasmine-sinon.html
I also recommend Phantom.js for integration testing... It's a headless browser and much faster than using Selenium... http://www.phantomjs.org/
Btw here is an example of unit-tested app from Pivotal : https://github.com/pivotal/cimonitor. You can find client-side tests there -> cimonitor/public/javascripts/js-common
I'm not sure if this answer will qualify but I'm working on the next iteration of my pet project "Atomic OS" (an OS-metaphor for web developers) which will, eventually, meet your criteria.
I'm working on a related project (which I can't share just yet) that is built on a bare-bones Atomic OS v2 foundation and provides a rich set of UI widgets for mobile web apps.
I built & use JSDog to produce documentation from a subset of JSDoc syntax and unit test runners with QUnit.
For an example of where I'm intending to go with unit tests, please see the Atomic OS documentation. (Click "Docs" in the taskbar and select a class, such as HxJSFS)
Just one perspective:
I work on a web application that is the front end of a video analytics system. (The back end is typically an IP camera, DVR or video router running a very, very lean, embedded web server.) It uses a number of jQueryUI widgets, allows user to configure the device, create video analysis rules, and draw markup over video frames using canvas elements. I think of it as fairly sophisticated.
We use unit tests (originally written for JSUnit, but now using qunit) for a very limited subset of the code. We have unit tests to verify the behavior of business objects, including the ability to serialize/deserialize to/from XML. And we have unit tests to test the basic geometry classes we've written for the canvas markup.
However, we have no unit tests that manipulate the DOM or that verify that the elements on a page are in the correct state. Doing that correctly struck us as too difficult a problem to solve, so we rely on Selinium tests to verify that a given set of inputs will put the DOM into the correct state.
I've had a lot of trouble trying to come up with the best way to properly follow TDD principles while developing UI in JavaScript. What's the best way to go about this?
Is it best to separate the visual from the functional? Do you develop the visual elements first, and then write tests and then code for functionality?
I've done some TDD with Javascript in the past, and what I had to do was make the distinction between Unit and Integration tests. Selenium will test your overall site, with the output from the server, its post backs, ajax calls, all of that. But for unit testing, none of that is important.
What you want is just the UI you are going to be interacting with, and your script. The tool you'll use for this is basically JsUnit, which takes an HTML document, with some Javascript functions on the page and executes them in the context of the page. So what you'll be doing is including the Stubbed out HTML on the page with your functions. From there,you can test the interaction of your script with the UI components in the isolated unit of the mocked HTML, your script, and your tests.
That may be a bit confusing so lets see if we can do a little test. Lets to some TDD to assume that after a component is loaded, a list of elements is colored based on the content of the LI.
tests.html
<html>
<head>
<script src="jsunit.js"></script>
<script src="mootools.js"></script>
<script src="yourcontrol.js"></script>
</head>
<body>
<ul id="mockList">
<li>red</li>
<li>green</li>
</ul>
</body>
<script>
function testListColor() {
assertNotEqual( $$("#mockList li")[0].getStyle("background-color", "red") );
var colorInst = new ColorCtrl( "mockList" );
assertEqual( $$("#mockList li")[0].getStyle("background-color", "red") );
}
</script>
</html>
Obviously TDD is a multi-step process, so for our control, we'll need multiple examples.
yourcontrol.js (step1)
function ColorCtrl( id ) {
/* Fail! */
}
yourcontrol.js (step2)
function ColorCtrl( id ) {
$$("#mockList li").forEach(function(item, index) {
item.setStyle("backgrond-color", item.getText());
});
/* Success! */
}
You can probably see the pain point here, you have to keep your mock HTML here on the page in sync with the structure of what your server controls will be. But it does get you a nice system for TDD'ing with JavaScript.
I've never successfully TDDed UI code. The closest we came was indeed to separate UI code as much as possible from the application logic. This is one reason why the model-view-controller pattern is useful - the model and controller can be TDDed without much trouble and without getting too complicated.
In my experience, the view was always left for our user-acceptance tests (we wrote web applications and our UATs used Java's HttpUnit). However, at this level it's really an integration test, without the test-in-isolation property we desire with TDD. Due to this setup, we had to write our controller/model tests/code first, then the UI and corresponding UAT. However, in the Swing GUI code I've been writing lately, I've been writing the GUI code first with stubs to explore my design of the front end, before adding to the controller/model/API. YMMV here though.
So to reiterate, the only advice I can give is what you already seem to suspect - separate your UI code from your logic as much as possible and TDD them.
See also: JavaScript unit test tools for TDD
I've found the MVP architecture to be very suitable for writing testable UIs. Your Presenter and Model classes can simply be 100% unit tested. You only have to worry about the View (which should be a dumb, thin layer only that fires events to the Presenter) for UI testing (with Selenium etc.)
Note that in the I'm talking about using MVP entirely in the UI context, without necessarily crossing to the server-side. Your UI can have its own Presenter and Model that lives entirely on the client-side. The Presenter drives the UI interaction/validation etc. logic while the Model keeps state information and provides a portal to the backend (where you can have a separate Model).
You should also take a look at the Presenter First TDD technique.
This is the primary reason I switched to the Google Web Toolkit ... I develop and test in Java and have a reasonable expectation that the compiled JavaScript will function properly on a variety of browsers. Since TDD is primarily a unit testing function, most of the project can be developed and tested before compilation and deployment.
Integration and Functional test suites verify that the resulting code is functioning as expected after it's deployed to a test server.
I'm just about to start doing Javascript TDD on a new project I am working on. My current plan is to use qunit to do the unit testing. While developing the tests can be run by simply refreshing the test page in a browser.
For continuous integration (and ensuring the tests run in all browsers), I will use Selenium to automatically load the test harness in each browser, and read the result. These tests will be run on every checkin to source control.
I am also going to use JSCoverage to get code coverage analysis of the tests. This will also be automated with Selenium.
I'm currently in the middle of setting this up. I'll update this answer with more exact details once I have the setup hammered out.
Testing tools:
qunit
JSCoverage
Selenium
What I do is to poke the Dom to see if I'm getting what I expect. A great side effect of this is that in making your tests fast, you also make your app fast.
I just released an open source toolkit which will help with JavaScript tdd immensely. It is a composition of many open source tools which gives you a working requirejs backbone app out of the box.
It provides single commands to run: dev web server, jasmine single browser test runner, jasmine js-test-driver multi browser test runner, and concatenization/minification for JavaScript and CSS. It also outputs an unminified version of your app for production debugging, precompiles your handlebar templates, and supports internationalization.
No setup is required. It just works.
http://github.com/davidjnelson/agilejs