Starting to investigate JavaScript testing frameworks (e.g. QUnit, Jasmine, Mocha etc).
I will require our Developers to be able to follow a TDD/BDD approach, so ideally they will be able to run the tests in Visual Studio (using for example Chutzpah). In addition, the tests will automatically run when the solution is built (Release mode) and on the build server.
However...
Our product is used by many people and we can not control the browser that they use. However, we would like to state that our product will definitely work for various browsers (e.g. Chrome, FireFox, IE) and even for specific versions (e.g. IE 8 => Edge).
That being the case, I'm not clear on how we can take our suite of JavaScript tests and run them for each browser engine. Is this possible? If so, please do let me know how.
Thanks
Griff
Typically, TDD is done using unit tests rather than UI tests. Even if you can prove that these pass in multiple browsers, it doesn't prove that the application works in multiple browsers, just that the JavaScript does what it is supposed to in multiple browsers - it doesn't take into account the HTML or CSS.
Although there is some value in checking that your unit tests pass in multiple browsers, UI tests can go further and show that the tested functionality at least "works" in different browsers and most UI testing frameworks will support running in different browsers.
However, in my opinion, to back up the statement:
we would like to state that our product will definitely work for
various browsers
You really need to manually test in those browsers as in my experience the problems that you see between browsers are typically visual.
Related
I am writing a Polymer 2 application. The default is to transpile ES6 to ES5 so that you can use ES6 syntax and be sure it will just work.
The problem with this is that everybody (even supporting browsers) get to receive transpiled code.
Two questions:
Is it just too crazy to say "no" to legacy browsers, and just stop transpiling?
Is there an easy-ish way to redirect specific browsers to a non-transpiled version of the app?
It really depends on the audience of the app you want to create. As for my own projects, I can see in my Google Analytics that there are still some people who are accessing it via Safari 8, 9 and even IE 11. I cannot just tell them to use a different browser because of several reasons... mostly financial reason (either personal or corporate)
Because of that, it is still a default for me to just transpile back to ES5 (given that I am using Webpack for now while waiting for the script type="module" to stabilize).
As for easiest way, they say if you use Polymer-CLI's serve function, it autotranspiles your code depending on the browser's capabilities.
Or you can have a simple javascript code that tries to check an ES6 method, then if it works, it loads the ES6 version of the bundled code... if not, it loads the ES5 version of the bundled code + the custom-elements-es5-adapter. But this one takes some performance hit because of the wait to parse the initial JS script to check before loading the necessary files instead of loading them right away (I haven't tested this though)
Or you can check in the server what type of browser is calling and then heuristically guess what type of version code you want to send.
As for performance for the overhead of the transpiled code, it's a bit miniscule, given that if you are just using Polymer.Element, you can get at least 12KB of code... then you'll have like 30+KB left to show content, which is more than enough to have a PRPL+50
The simple answer isL use prpl-server-node which does exactly what I was talking about, and more.
Specifically:
Differential Serving
Modern browsers offer great features that improve performance, but most applications need to support older browsers too. prpl-server can serve different versions of your application to different browsers by detecting browser capabilities using the user-agent header.
Builds
prpl-server understands the notion of a build, a variant of your application optimized for a particular set of browser capabilities.
I'm using Node for running my unit-tests.
I have a JavaScript module run in the browser I'd like to test.
My code is "isomorphic", i.e. it avoids language features not available in Node, like exports.
But it uses pure browsers APIs: XMLHttpRequest, FormData and File.
I have found Node's implementations for each of them.
But the one of XMLHttpRequest does not support upload.
So I'm looking for the simplest way to unit-test this code in an environment with these APIs.
The code does not need DOM or other browsers APIs, "only" these three.
I've already used PhantomJS for other needs but:
this will create another test workflow (minor issue),
it supports an older JavaScript version and it would force a complete rewrite of the code to test (major issue),
the code has a lot of NPM dependencies that probably won't be compatible (blocking issue).
As the code is Browserified all these issues may disappear but before going along this way I'd like to be sure.
Is there any chance to get it work with PhantomJS, CasperJS or the like?
Which other alternatives are available?
This is not how you test code that runs in a browser. If it runs in a browser, it needs to be tested in a browser.
You need to look into solutions based on the webdriver spec. The big hairy monster in this ecosystem is Selenium. I'm currently researching this topic because of some issues we've had with using selenium-server. You should also look into Nightwatch and Leadfoot. Webdriver.io is the first recommendation a lot of people recommend, as its a node-based client that wraps (poorly) around Selenium. But the documentation is all over the place and we've run into frequent bugs using it.
I'm going to write bunch of browser extensions (the same functionality for each popular browser). I hope, that some of the code will be shared, but I'm not sure about this yet. For sure some of extensions will use native API. I have not much experience with TDD/BDD, and I thought it's good time to start folowing these ideas from this project.
The problem is, I have no idea how to handle it. Should I write different tests for each browser? How far should I go with these tests? These extensions will be quite simple - some data in a local storage, refreshing a page and listening through web sockets.
And my observation about why is it hard for me - because there is a lot of behaviour, and not so much models, which are also dependent on a platform.
I practise two different ways of testing my browser extensions:
Unit tests
Integration test
Introduction
I will use the cross-browser YouTube Lyrics by Rob W extension as an example throughout this answer. The core of this extension is written in JavaScript and organized with AMD modules. A build script generates the extension files for each browser. With r.js, I streamline the inclusion of browser-specific modules, such as the one for cross-origin HTTP requests and persistent storage (for preferences), and a module with tons of polyfills for IE.
The extension inserts a panel with lyrics for the currently played song on YouTube, Grooveshark and Spotify. I have no control over these third-party sites, so I need an automated way to verify that the extension still works well.
Workflow
During development:
Implement / edit feature, and write a unit test if the feature is not trivial.
Run all unit tests to see if anything broke. If anything is wrong, go back to 1.
Commit to git.
Before release:
Run all unit tests to verify that the individual modules is still working.
Run all integration tests to verify that the extension as whole is still working.
Bump versions, build extensions.
Upload update to the official extension galleries and my website (Safari and IE extensions have to be hosted by yourself) and commit to git.
Unit testing
I use mocha + expect.js to write tests. I don't test every method for each module, just the ones that matter. For instance:
The DOM parsing method. Most DOM parsing methods in the wild (including jQuery) are flawed: Any external resources are loaded and JavaScript is executed.
I verify that the DOM parsing method correctly parses DOM without negative side effects.
The preference module: I verify that data can be saved and returned.
My extension fetches lyrics from external sources. These sources are defined in separate modules. These definitions are recognized and used by the InfoProvider module, which takes a query, (black box), and outputs the search results.
First I test whether the InfoProvider module functions correctly.
Then, for each of the 17 sources, I pass a pre-defined query to the source (with InfoProvider) and verify that the results are expected:
The query succeeds
The returned song title matches (by applying a word similarity algorithm)
The length of the returned lyrics fall inside the expected range.
Whether the UI is not obviously broken, e.g. by clicking on the Close button.
These tests can be run directly from a local server, or within a browser extension. The advantage of the local server is that you can edit the test and refresh the browser to see the results. If all of these tests pass, I run the tests from the browser extension.
By passing an extra parameter debug to my build script, the unit tests are bundled with my extension.
Running the tests within a web page is not sufficient, because the extension's environment may differ from the normal page. For instance, in an Opera 12 extension, there's no global location object.
Remark: I don't include the tests in the release build. Most users don't take the efforts to report and investigate bugs, they will just give a low rating and say something like "Doesn't work". Make sure that your extension functions without obvious bugs before shipping it.
Summary
View modules as black boxes. You don't care what's inside, as long as the output matches is expected or a given input.
Start with testing the critical parts of your extension.
Make sure that the tests can be build and run easily, possibly in a non-extension environment.
Don't forget to run the tests within the extension's execution context, to ensure that there's no constraint or unexpected condition inside the extension's context which break your code.
Integration testing
I use Selenium 2 to test whether my extension still works on YouTube, Grooveshark (3x) and Spotify.
Initially, I just used the Selenium IDE to record tests and see if it worked. That went well, until I needed more flexibility: I wanted to conditionally run a test depending on whether the test account was logged in or not. That's not possible with the default Selenium IDE (it's said to be possible with the FlowControl plugin - I haven't tried).
The Selenium IDE offers an option to export the existing tests in other formats, including JUnit 4 tests (Java). Unfortunately, this result wasn't satisfying. Many commands were not recognized.
So, I abandoned the Selenium IDE, and switched to Selenium.
Note that when you search for "Selenium", you will find information about Selenium RC (Selenium 1) and Selenium WebDriver (Selenium 2). The first is the old and deprecated, the latter (Selenium WebDriver) should be used for new projects.
Once you discovered how the documentation works, it's quite easy to use.
I prefer the documentation at the project page, because it's generally concise (the wiki) and complete (the Java docs).
If you want to get started quickly, read the Getting Started wiki page. If you've got spare time, look through the documentation at SeleniumHQ, in particular the Selenium WebDriver and WebDriver: Advanced Usage.
Selenium Grid is also worth reading. This feature allows you to distribute tests across different (virtual) machines. Great if you want to test your extension in IE8, 9 and 10, simultaneously (to run multiple versions of Internet Explorer, you need virtualization).
Automating tests is nice. What's more nice? Automating installation of extensions!
The ChromeDriver and FirefoxDriver support the installation of extensions, as seen in this example.
For the SafariDriver, I've written two classes to install a custom Safari extension. I've published it and sent in a PR to Selenium, so it might be available to everyone in the future: https://github.com/SeleniumHQ/selenium/pull/87
The OperaDriver does not support installation of custom extensions (technically, it should be possible though).
Note that with the advent of Chromium-powered Opera, the old OperaDriver doesn't work any more.
There's an Internet Explorer Driver, and this one does definitely not allow one to install a custom extension. Internet Explorer doesn't have built-in support for extensions. Extensions are installed through MSI or EXE installers, which are not even integrated in Internet Explorer. So, in order to automatically install your extension in IE, you need to be able to silently run an installer which installs your IE plugin. I haven't tried this yet.
Testing browser extensions posed some difficulty for me as well, but I've settled on implementing tests in a few different areas that I can invoke simultaneously from browsers driven by Selenium.
The steps I use are:
First, I write test code integrated into the extension code that can be activated by simply going to a specific URL. When the extension sees that URL, it begins running the tests.
Then, in the page that activates the testing in the extension I execute server-side tests to be sure the API performs, and record and log issues there. I record the methods invoked, the time they took, and any errors. So I can see the method the extension invoked, the web performance, the business logic performance, and the database performance.
Lastly, I automatically invoke browsers to point at that specific URL and record their performance along with other test information, errors, etc on any given client system using Selenium:
http://docs.seleniumhq.org/
This way I can break down the tests in terms of browser, extension, server, application, and database and link them all together according to specific test sets. It takes a bit of work to put it all together, but once its done you can have a very nice extension testing framework.
Typically for cross-browser extension development in order to maintain a single code-base I use crossrider, but you can do this with any framework or with native extensions as you wish, Selenium won't care, it is just driving the extension to a particular page and allowing you to interact and perform tests.
One nice thing about this approach is you can use it for live users as well. If you are providing support for your extension, have a user go to your test url and immediately you will see the extension and server-side performance. You won't get the Selenium tests of course, but you will capture a lot of issues this way - very useful when you are coding against a variety of browsers and browser versions.
as I am getting more and more into Android PhoneGap app development, I can see more and more nuances and little details between built-in Android browsers throughout the versions. I searched for some official or fan document, which would deal with these browser version differences. But I can't find anything useful.
It's a lot frustrating, because you have to test everything on all versions of Android emulator and if app grows big, it's A LOT of work to test all the features in all versions.
Everyone is excited about HTML5, I was too, but only to the point when I moved to doing the real thing. I realized that there is so many problems when dealing with different versions of Android behaving sometimes a lot differently.
If anyone has some good resource to share, I would be very happy. Thanks
EDIT: Added example of different behaviour betweeen Android browser versions ( but there is many of them):
This works in Android browser in 1.6, 2.2, 2.3 and 2.3.3. But it failes (application crashes or stops JS execution) in Android 2.1:
Object.keys(var).length
You asked a pretty general question. The general purpose answer is that any sort of cross browser development (even cross versions of the same browser) requires that you develop a familiarity with what features are safe across the targeted browsers, which features are not safe across the targeted browsers and which ones must be used only with careful testing or feature detection with fallback.
While one wouldn't exactly expect the type of difference you saw with the one example you referenced, it is clear that that is a fairly new feature in ECMAScript and it's not consistently implemented across normal browsers so I would put it in the category where it is not safe to assume that it works on all versions of Android, even if you've seen it some versions of Android. That means to me that you should only use it if you've explicitly tested that it's reliable in all the browser versions you are targeting or devise a feature test and only use it when you know it's present and reliable or develop a safer work-around.
As I think has been mentioned previously, this thread has a bunch of proposed work-arounds for the specific issue you mentioned.
I am not aware of any detailed written material that would document in advance for you the details of the differences between different Android browser versions. Since it's open source, there are probably developer checkin notes and some level of release notes, but that will probably be like looking for a needle in a haystack and may not even contain what you want. It is rare for any developers to produce such detailed information. We don't generally get that level of detail from any of the existing desktop browsers or iOS browsers and, even if you were on the development team itself, you probably would only see part of this info. I don't think you're going to find official documentation that covers what you want.
You're going to have to learn to treat is as more of an unknown and learn what areas are "safe", what areas require extensive testing before using and what areas are just too risky. Even when doing that, you'll find Android bugs in some version that trip you up. That's just the nature of building on someone else's platform. At least the Android set of browsers are a much simpler target than trying to target all desktop browsers from IE6 to IE9, Firefox 3 to 5, Safari 3 to 5, Opera 9 to 11, Chrome 9 to 12, all Android, all iOS and use HTML5 when available which is what I'm working on.
Once you've been through this wringer a couple times, you will realize that if the newer language/library feature carries any risk, you shouldn't use it at all unless it's absolutely central to what you're trying to accomplish and then you will have to test the hell out of it. If it's something like getting the length of the associative array which is just a programmer's convenience, then it's probably simpler to stick with a work-around that is guaranteed to be safe and just not spend your time dealing with any browser-support risk.
The only official documentation I am aware of is part of the Android developer documentation. If I were a betting man, I would bet that it only covers a subset of what you are seeking.
The general idea behind cross browser Javascript is inline feature testing (at least how I've come to accept it.) I don't know exactly what "features" you are specifically looking for but it's generally wise to test for the existence of a feature set then use it and have a fallback if that doesn't exist. (Even if the fallback is, "This site requires a browser that supports 'foo'")
Since you didn't give any examples, I'll pick on Ajax. It's always best to check for the existence of window.XMLHttpRequest, then act upon it. Of course, this is not performant if you are doing it for every instance of need so you could write a check procedure or a wrapper to accept your list of necessity let your wrapper cache/call the appropriate methods to perform that task.
Without examples of "features" that you are talking about being different from browser to browser though, it's hard to give any concrete advice on direction.
I've searched a bit for this and tried to implement a self-made solution but so far haven't found to be confident with it.
What I need is to write integration tests in Ruby on Rails which interact with JavaScript and get programmatic ways to assert some behaviors. I'm using Test::Unit for the controller/models part but I'm struggling to test some jQuery/JavaScript behaviors used by my app. Mainly it consists in ajax calls and interactions in the UI which updates some sets of information.
I haven't found a solution which makes me confident and which integrates nicely with autotest and the whole red-green process, so for now most parts of my client-side code is untested and that's making me nervous (as it should be :P).
So, does anyone have suggestions for best practices on this issue? Unit testing JS is a bit tricky, as Crockford points out, because it dependes heavily on the current state of the UI and etc and as AFAIK even he hasn't found a good way to implement decent testing...
Shortly: I need to implement tests for some UI behavior which depends on Ajax, integrating with autotest or some other CI tool and haven't found a good and elegant way to do it.
Thanks all for the attention,
Best Regards
AFAIK, outside of a combination of Capybara with Selenium Web-Driver there is very few options for automated testing of JS code.
I use cucumber with capybara and selenium web-driver and because selenium-webdriver actually launches firefox or chrome to go through testing a particular page with ajax call, It does take significantly longer to run through a suite of tests.
There are some alternatives but they dont work all the time or for every situations.
For instance: Capybara with envjs
In April 2011 the thoughtbot guys updated their quest for javascript testing.
Akephalos has fallen out of favor for the following reasons:
Bugs: as previously mentioned, there are bugs in htmlunit,
specifically with jQuery’s live. Although all browser implementations
have bugs, it’s more useful if tests experience the same bugs as
actual browsers.
Compatibility: htmlunit doesn’t fully implement the feature set that
modern browsers do. For example, it doesn’t fully handle DOM ranges or
Ajax file uploads.
Rendering: htmlunit doesn’t actually render the page, so tests that
depend on CSS visibility or positioning won’t work.
Performance: when most of your tests use Javascript, test suites with
htmlunit start to crawl. It takes a while to start up a test with
Akephalos, and a large test suite can easily take 10 or 15 minutes.
So they rolled their own solution which is open source - capybara-webkit. It's still fairly new but it looks like the way to go now.
This article recommends Akephalos.
I have used cucumber and capybara with selenium. This was very frustrating because selenium did not seem to be able to see dynamically generated javascript, despite the fact that capybara was supposed to be waiting for it. That was in January 2011. Things may be different now.
Currently, I am using cucumber and capybara with akephalos. So far, it has been very difficult because 1. it is headless, so you can't see progress. Capybara's "save_and_open" call has helped to a degree. 2. jQuery and akephalos don't seem to play that nicely together. For instance, triggering on a radio button with jquery's .change() works fine in chrome, but doesn't in akephalos. Maybe this is intentional because I later heard somewhere that it doesn't work in IE. I fixed the issue by using .click() instead of .change() for the radio button but since the .change function was set up to run on a bunch of questions, I had to code specifically to get it to work for the test.
The bottom line for me is that automated javascript acceptance testing in a rails env is still immature and possibly more work that it is worth.