The older Firefox "Add-ons" API had a built-in unittest layer sdk/test that allowed testing. This doesn't seem to be available any more.
Additionally the use of "package/require" allowed code to be separated into "js code-only" packages that were testable using node.js. The new, highly structured javascript doesn't share this.
My priorities are (highest to lowest):
Algorithms, "business logic", e.g. parsing input data - no APIs needed - just JavaScript
Internal logic - e.g. background scripts interacting with settings, etc.
UI interactions - I can live without this, but would be nice to test
So how do people test their WebExtensions?
Check out webextension-geckodriver for a worked example of functional testing.
If you want to test interaction with the webextension API you can either do it live (have a test page for your extension and get geckodriver to visit it, for example) or use a fake like sinon-webextension through webextension-jsdom.
To unit test algorithms, just import the functions using jest, mocha, or whatever node unit testing framework you prefer or add them to a test page that you can visit in the browser.
A complete, but old, worked example of webext testing is here: example-webextension.
An example of tests in a real webextension using another fake: vim-vixen
It is still possible to run unit tests in NodeJs.
To illustrate the idea, let us take a look at the Cliqz extension, whose source code is open (Github link: cliqz-oss/browser-core). In comparison other extensions that I have seen so far, the code base is quite large.
In other words, it is not a toy example, but a realistic use case. The drawback, of course, is that due to the complexity, it is harder to understand how the test setup works (how mocking works, the integration into the build system, etc).
To get an idea how tests look like, here is one example:
source: query-sanitizer.es
test: query-sanitizer-test.es
To explain the details of how the mocking works is hard because you would need to dive deep into the build system. From a high level perspective, you will notice that in the test it uses a function called describeModule, which does all the mocking of dependencies.
In the implementation of describeModule, you can see that it uses systemjs to dynamically load ES modules. That trick makes it possible to run the unit tests with NodeJs.
My priorities are (highest to lowest):
Algorithms, "business logic", e.g. parsing input data - no APIs needed - just JavaScript
For these kind of tests, the unit test infrastructure described above is the preferred way. For local development, NodeJs is used to run the tests.
Internal logic - e.g. background scripts interacting with settings, etc.
This is not so different. The idea is still that you can mock dependencies and then do classical unit tests.
It might require some work to allow dependencies to be replaced. As mentioned, as the code base in this example has to run in different environments (e.g., Firefox, Chrome, Edge, ReactNative), platform APIs have to be abstracted (that includes also browser APIs).
UI interactions - I can live without this, but would be nice to test
For testing the UI, there are additional integration tests. I do not want to go into details, but there are examples in the code.
What is important is that the integration tests are not executed with NodeJs, but they require a real browser environment (e.g., Firefox, Chrome). In addition, a local HTTP server is started which can be used to mock API calls.
As a side-note, linters are extremely useful and in comparison easy to setup. Also consider using a typed language (TypeScript), especially when the project becomes bigger and more people are working on it.
You will still need tests, as static analysis will not be able to find logical bugs. However, it helps to eliminate certain types of simple bugs like typos and the overhead (fixing linter errors or adding type annotations) is not very high.
Related
Background
I'm working on a single-page application with a JavaScript code base that is well-tested at the unit level, and want to add a functional test suite to verify application behavior closer to the level of what the user sees. At this time, I'm not looking at full end-to-end testing (I'd expect to stub/mock interactions with the back-end) but that may be a goal later on.
Our build is run by npm; gulp/grunt are not currently in use but may be adopted soon as our build complexity grows. We are using Jasmine and Blanket for testing and coverage currently, but for purposes of this question I'm fairly tool-agnostic.
Goal
What I'd like to do is generate code coverage that is comprehensive across both categories of automated tests. That is, if I have a line of code that is exercised during functional testing but isn't exercised in unit tests, my reporting should still mark that as "covered." (If I can get more or deeper information about how code gets exercised, great, but not needed.)
The underlying goal (since XY problems are so common on SO) is to create a forcing function for test completeness ("code coverage must be above X%") without forcing redundancy (as might happen if coverage was measured separately for both) or introducing a false preference for one category of testing (as might happen if coverage was measured only for one category of testing.) There's value to testing the same code in different circumstances, of course, but we are resource-limited and want to focus on the tests that will add the most value.
Question(s)
The short question is "how do I do this?" I haven't found any documentation or examples of this being done. I feel like the answer is "just stick istanbul in front of X which runs both kinds of tests", but don't really know what X looks like.
Some more specific questions I have in mind:
Are there any examples which could be followed of comprehensive coverage being obtained for testing across multiple categories?
What are the constraints one must accept to make this kind of testing feasible?
Am I wrong to think of unit tests and functional tests as separate suites being run in different ways? (The problem becomes trivial if functional and unit tests are not separate suites, but are distinctions one makes on a test-by-test basis. But that's not the sort of test suite I'm used to seeing.)
Is there a reason not to do this? (The fact that I can't find examples of this being done feels conspicuous to me.)
I'm reasonably familiar with the tools that are out there in this category (Karma, Mocha, Jasmine, Blanket, Istanbul, etc...) so I'm not looking for recommendations, per se, but rather some guidance on how to achieve this kind of configuration. However, I'd certainly be interested to hear if some tools are better- or ill-suited for this sort of usage.
I would like to add integration tests to an (Backbone.js) application I am maintaining, and considering what strategy is preferable and upsides and downsides for each:
Running tests on the client side, with something like jasmine-jquery. Ideally I would like something like ember test helpers which allows writing simple synchronous looking code (while actually running async).
Running tests on the server side using selenium drivers, for example Nightwatch.js.
It is hard to tell which approach has more community and tooling around it, and which projects are more mature. Additionally, I am getting the feeling that running tests on the client side might allow better isolation of tests, while running on the server side, might create tests which run longer and heavier (also to maintain?) but allow to simulate more complex real-user scenarios
Any thoughts would be appreciated
If you care about browser compatibility, then you'll want to run end-to-end tests on different browsers on different platforms, possibly using a cloud-based cross-browser testing service such as SauceLabs, BrowserStack, TestingBot or CrossBrowserTesting.
Since you're using Backbone and are familiar with JavaScript, you might want to pick a JavaScript client for Selenium WebDriver. There are quite a few choices, with Intern leading the pack by far.
I've recently started getting into Angular (with node.js), and many of the tutorials suggest using Protactor, which looks amazing. One thing has me confused though.
I'm used to tests where test data is built before the test, the test is run, and the data is destroyed.
With protractor it seems you start your server, and have your tests run against that server. In the tutorials I've seen, this server is usually the dev environment (populated by seed data I assume). In my experience, the dev database changes as you play around and tweak your app. Furthermore, a protractor test might delete an object, meaning that for the test to be re-run the object would have to be built again.
When using Protractor, what is the standard practice for creating a test environment with before/after hooks for populate. Bonus points if you can point me to some good resources that answer my question.
Depends an how PRO you want to go. Are you interested in testing in dev only? Do you have other environments? How often do you want to test? I test if different environments. One of them has no data because the database is created before running the tests. Other environments have a lot of data.
I gave a talk in the Angular meetup in NY a few months back:
https://github.com/andresdominguez/protractor-meetup
Take a look at slide 35 of the presentation (the link is in the readme file).
I call the rest api directly to generate data for my tests. You can also run a script before you run your tests to make sure that some objects are present.
Can anyone give me examples of large-scale JS apps (including AJAX, different UI widgets, and a sophisticated architecture) with unit tests?
I'm not talking about Selenium tests here, just plain ol' stupid unit tests using mocks, decent result reporting and such.
Not sure why people voted to close, or downvoted the question. Maybe a comment would be nice.
Seriously, I've been trying hard to find unit tested web apps, since I'm having a hard time building mocks and I wonder if it's even possible with reasonable effort. It made me think about the benefits of unit tests on widgets as compared to Selenium tests. People are babbling a lot about unit tests in theory but evidently nobody actually has done it in JS-RIAs. Or have they?
Personally I like Qooxdoo, check it out for your self and see if this is what you want
http://qooxdoo.org/demo#real-life_examples
This is one good tool: http://www.uize.com/
You should look at Jasmine & Sinon.js : http://sinonjs.org/
Here is a good tutorial on testing using Backbone.js, Jasmine & Sinon.js : http://tinnedfruit.com/2011/03/03/testing-backbone-apps-with-jasmine-sinon.html
I also recommend Phantom.js for integration testing... It's a headless browser and much faster than using Selenium... http://www.phantomjs.org/
Btw here is an example of unit-tested app from Pivotal : https://github.com/pivotal/cimonitor. You can find client-side tests there -> cimonitor/public/javascripts/js-common
I'm not sure if this answer will qualify but I'm working on the next iteration of my pet project "Atomic OS" (an OS-metaphor for web developers) which will, eventually, meet your criteria.
I'm working on a related project (which I can't share just yet) that is built on a bare-bones Atomic OS v2 foundation and provides a rich set of UI widgets for mobile web apps.
I built & use JSDog to produce documentation from a subset of JSDoc syntax and unit test runners with QUnit.
For an example of where I'm intending to go with unit tests, please see the Atomic OS documentation. (Click "Docs" in the taskbar and select a class, such as HxJSFS)
Just one perspective:
I work on a web application that is the front end of a video analytics system. (The back end is typically an IP camera, DVR or video router running a very, very lean, embedded web server.) It uses a number of jQueryUI widgets, allows user to configure the device, create video analysis rules, and draw markup over video frames using canvas elements. I think of it as fairly sophisticated.
We use unit tests (originally written for JSUnit, but now using qunit) for a very limited subset of the code. We have unit tests to verify the behavior of business objects, including the ability to serialize/deserialize to/from XML. And we have unit tests to test the basic geometry classes we've written for the canvas markup.
However, we have no unit tests that manipulate the DOM or that verify that the elements on a page are in the correct state. Doing that correctly struck us as too difficult a problem to solve, so we rely on Selinium tests to verify that a given set of inputs will put the DOM into the correct state.
I've had a lot of trouble trying to come up with the best way to properly follow TDD principles while developing UI in JavaScript. What's the best way to go about this?
Is it best to separate the visual from the functional? Do you develop the visual elements first, and then write tests and then code for functionality?
I've done some TDD with Javascript in the past, and what I had to do was make the distinction between Unit and Integration tests. Selenium will test your overall site, with the output from the server, its post backs, ajax calls, all of that. But for unit testing, none of that is important.
What you want is just the UI you are going to be interacting with, and your script. The tool you'll use for this is basically JsUnit, which takes an HTML document, with some Javascript functions on the page and executes them in the context of the page. So what you'll be doing is including the Stubbed out HTML on the page with your functions. From there,you can test the interaction of your script with the UI components in the isolated unit of the mocked HTML, your script, and your tests.
That may be a bit confusing so lets see if we can do a little test. Lets to some TDD to assume that after a component is loaded, a list of elements is colored based on the content of the LI.
tests.html
<html>
<head>
<script src="jsunit.js"></script>
<script src="mootools.js"></script>
<script src="yourcontrol.js"></script>
</head>
<body>
<ul id="mockList">
<li>red</li>
<li>green</li>
</ul>
</body>
<script>
function testListColor() {
assertNotEqual( $$("#mockList li")[0].getStyle("background-color", "red") );
var colorInst = new ColorCtrl( "mockList" );
assertEqual( $$("#mockList li")[0].getStyle("background-color", "red") );
}
</script>
</html>
Obviously TDD is a multi-step process, so for our control, we'll need multiple examples.
yourcontrol.js (step1)
function ColorCtrl( id ) {
/* Fail! */
}
yourcontrol.js (step2)
function ColorCtrl( id ) {
$$("#mockList li").forEach(function(item, index) {
item.setStyle("backgrond-color", item.getText());
});
/* Success! */
}
You can probably see the pain point here, you have to keep your mock HTML here on the page in sync with the structure of what your server controls will be. But it does get you a nice system for TDD'ing with JavaScript.
I've never successfully TDDed UI code. The closest we came was indeed to separate UI code as much as possible from the application logic. This is one reason why the model-view-controller pattern is useful - the model and controller can be TDDed without much trouble and without getting too complicated.
In my experience, the view was always left for our user-acceptance tests (we wrote web applications and our UATs used Java's HttpUnit). However, at this level it's really an integration test, without the test-in-isolation property we desire with TDD. Due to this setup, we had to write our controller/model tests/code first, then the UI and corresponding UAT. However, in the Swing GUI code I've been writing lately, I've been writing the GUI code first with stubs to explore my design of the front end, before adding to the controller/model/API. YMMV here though.
So to reiterate, the only advice I can give is what you already seem to suspect - separate your UI code from your logic as much as possible and TDD them.
See also: JavaScript unit test tools for TDD
I've found the MVP architecture to be very suitable for writing testable UIs. Your Presenter and Model classes can simply be 100% unit tested. You only have to worry about the View (which should be a dumb, thin layer only that fires events to the Presenter) for UI testing (with Selenium etc.)
Note that in the I'm talking about using MVP entirely in the UI context, without necessarily crossing to the server-side. Your UI can have its own Presenter and Model that lives entirely on the client-side. The Presenter drives the UI interaction/validation etc. logic while the Model keeps state information and provides a portal to the backend (where you can have a separate Model).
You should also take a look at the Presenter First TDD technique.
This is the primary reason I switched to the Google Web Toolkit ... I develop and test in Java and have a reasonable expectation that the compiled JavaScript will function properly on a variety of browsers. Since TDD is primarily a unit testing function, most of the project can be developed and tested before compilation and deployment.
Integration and Functional test suites verify that the resulting code is functioning as expected after it's deployed to a test server.
I'm just about to start doing Javascript TDD on a new project I am working on. My current plan is to use qunit to do the unit testing. While developing the tests can be run by simply refreshing the test page in a browser.
For continuous integration (and ensuring the tests run in all browsers), I will use Selenium to automatically load the test harness in each browser, and read the result. These tests will be run on every checkin to source control.
I am also going to use JSCoverage to get code coverage analysis of the tests. This will also be automated with Selenium.
I'm currently in the middle of setting this up. I'll update this answer with more exact details once I have the setup hammered out.
Testing tools:
qunit
JSCoverage
Selenium
What I do is to poke the Dom to see if I'm getting what I expect. A great side effect of this is that in making your tests fast, you also make your app fast.
I just released an open source toolkit which will help with JavaScript tdd immensely. It is a composition of many open source tools which gives you a working requirejs backbone app out of the box.
It provides single commands to run: dev web server, jasmine single browser test runner, jasmine js-test-driver multi browser test runner, and concatenization/minification for JavaScript and CSS. It also outputs an unminified version of your app for production debugging, precompiles your handlebar templates, and supports internationalization.
No setup is required. It just works.
http://github.com/davidjnelson/agilejs