Develop Reference

Automated testing of webrtc application?

I am developing a conferencing app, details:
target: chrome browser
server: node.js ( currently windows env)
simplest test scenario would be:
open two browser tabs( open browser if need be)
emulate button click on both.
emulate accept getUserMedia request( hardest part)
more emulation stuff and reading JavaScript variable values and verifying them.
Till now, I have been doing manual testing for all of this, but decided that it would be impractical for the long run.
I have not done much automated testing( just a bit of unit testing). Initially, i thought mocha would do the job, but beginning to think it is not enough.
I need some pointers as to what are tools or alternate test frameworks needed to achieve browser starting, tab opening and giving media sharing permissions.

Chrome has command line flags for skipping get getUserMedia permission as well as using fake devices:
--use-fake-ui-for-media-stream (skips GUM permission prompt)
--use-fake-device-for-media-stream (uses a fake device)
Check http://blog.andyet.com/2014/09/29/testing-webrtc-applications, http://googletesting.blogspot.se/2014/08/chrome-firefox-webrtc-interop-test-pt-1.html for background information
If you want to click buttons, the standard tool for this is selenium. If you prefer Javascript to Java, a binding such as webdriver.io might be what you're looking for.

If you like Ruby, take a look at Capybara. You'll probably also want to use Xvfb so that you can run the tests on a machine that doesn't have a display.
Here is an example snippet that will let you open Chome with the switches that Philipp suggests. I've used this to create a headless video recording test.
Capybara.register_driver :chrome do |app|
switches = %w(disable-popup-blocking disable-translate use-fake-ui-for-media-stream use-fake-device-for-media-stream test-type)
Capybara::Selenium::Driver.new(app, browser: :chrome, switches: switches)
end

If you are looking for webrtc testing, you may like to see loadmultiplier.., it was answered here previously.

Testing browser extensions

I'm going to write bunch of browser extensions (the same functionality for each popular browser). I hope, that some of the code will be shared, but I'm not sure about this yet. For sure some of extensions will use native API. I have not much experience with TDD/BDD, and I thought it's good time to start folowing these ideas from this project.
The problem is, I have no idea how to handle it. Should I write different tests for each browser? How far should I go with these tests? These extensions will be quite simple - some data in a local storage, refreshing a page and listening through web sockets.
And my observation about why is it hard for me - because there is a lot of behaviour, and not so much models, which are also dependent on a platform.

I practise two different ways of testing my browser extensions:
Unit tests
Integration test
Introduction
I will use the cross-browser YouTube Lyrics by Rob W extension as an example throughout this answer. The core of this extension is written in JavaScript and organized with AMD modules. A build script generates the extension files for each browser. With r.js, I streamline the inclusion of browser-specific modules, such as the one for cross-origin HTTP requests and persistent storage (for preferences), and a module with tons of polyfills for IE.
The extension inserts a panel with lyrics for the currently played song on YouTube, Grooveshark and Spotify. I have no control over these third-party sites, so I need an automated way to verify that the extension still works well.
Workflow
During development:
Implement / edit feature, and write a unit test if the feature is not trivial.
Run all unit tests to see if anything broke. If anything is wrong, go back to 1.
Commit to git.
Before release:
Run all unit tests to verify that the individual modules is still working.
Run all integration tests to verify that the extension as whole is still working.
Bump versions, build extensions.
Upload update to the official extension galleries and my website (Safari and IE extensions have to be hosted by yourself) and commit to git.
Unit testing
I use mocha + expect.js to write tests. I don't test every method for each module, just the ones that matter. For instance:
The DOM parsing method. Most DOM parsing methods in the wild (including jQuery) are flawed: Any external resources are loaded and JavaScript is executed.
I verify that the DOM parsing method correctly parses DOM without negative side effects.
The preference module: I verify that data can be saved and returned.
My extension fetches lyrics from external sources. These sources are defined in separate modules. These definitions are recognized and used by the InfoProvider module, which takes a query, (black box), and outputs the search results.
First I test whether the InfoProvider module functions correctly.
Then, for each of the 17 sources, I pass a pre-defined query to the source (with InfoProvider) and verify that the results are expected:
The query succeeds
The returned song title matches (by applying a word similarity algorithm)
The length of the returned lyrics fall inside the expected range.
Whether the UI is not obviously broken, e.g. by clicking on the Close button.
These tests can be run directly from a local server, or within a browser extension. The advantage of the local server is that you can edit the test and refresh the browser to see the results. If all of these tests pass, I run the tests from the browser extension.
By passing an extra parameter debug to my build script, the unit tests are bundled with my extension.
Running the tests within a web page is not sufficient, because the extension's environment may differ from the normal page. For instance, in an Opera 12 extension, there's no global location object.
Remark: I don't include the tests in the release build. Most users don't take the efforts to report and investigate bugs, they will just give a low rating and say something like "Doesn't work". Make sure that your extension functions without obvious bugs before shipping it.
Summary
View modules as black boxes. You don't care what's inside, as long as the output matches is expected or a given input.
Start with testing the critical parts of your extension.
Make sure that the tests can be build and run easily, possibly in a non-extension environment.
Don't forget to run the tests within the extension's execution context, to ensure that there's no constraint or unexpected condition inside the extension's context which break your code.
Integration testing
I use Selenium 2 to test whether my extension still works on YouTube, Grooveshark (3x) and Spotify.
Initially, I just used the Selenium IDE to record tests and see if it worked. That went well, until I needed more flexibility: I wanted to conditionally run a test depending on whether the test account was logged in or not. That's not possible with the default Selenium IDE (it's said to be possible with the FlowControl plugin - I haven't tried).
The Selenium IDE offers an option to export the existing tests in other formats, including JUnit 4 tests (Java). Unfortunately, this result wasn't satisfying. Many commands were not recognized.
So, I abandoned the Selenium IDE, and switched to Selenium.
Note that when you search for "Selenium", you will find information about Selenium RC (Selenium 1) and Selenium WebDriver (Selenium 2). The first is the old and deprecated, the latter (Selenium WebDriver) should be used for new projects.
Once you discovered how the documentation works, it's quite easy to use.
I prefer the documentation at the project page, because it's generally concise (the wiki) and complete (the Java docs).
If you want to get started quickly, read the Getting Started wiki page. If you've got spare time, look through the documentation at SeleniumHQ, in particular the Selenium WebDriver and WebDriver: Advanced Usage.
Selenium Grid is also worth reading. This feature allows you to distribute tests across different (virtual) machines. Great if you want to test your extension in IE8, 9 and 10, simultaneously (to run multiple versions of Internet Explorer, you need virtualization).
Automating tests is nice. What's more nice? Automating installation of extensions!
The ChromeDriver and FirefoxDriver support the installation of extensions, as seen in this example.
For the SafariDriver, I've written two classes to install a custom Safari extension. I've published it and sent in a PR to Selenium, so it might be available to everyone in the future: https://github.com/SeleniumHQ/selenium/pull/87
The OperaDriver does not support installation of custom extensions (technically, it should be possible though).
Note that with the advent of Chromium-powered Opera, the old OperaDriver doesn't work any more.
There's an Internet Explorer Driver, and this one does definitely not allow one to install a custom extension. Internet Explorer doesn't have built-in support for extensions. Extensions are installed through MSI or EXE installers, which are not even integrated in Internet Explorer. So, in order to automatically install your extension in IE, you need to be able to silently run an installer which installs your IE plugin. I haven't tried this yet.

Testing browser extensions posed some difficulty for me as well, but I've settled on implementing tests in a few different areas that I can invoke simultaneously from browsers driven by Selenium.
The steps I use are:
First, I write test code integrated into the extension code that can be activated by simply going to a specific URL. When the extension sees that URL, it begins running the tests.
Then, in the page that activates the testing in the extension I execute server-side tests to be sure the API performs, and record and log issues there. I record the methods invoked, the time they took, and any errors. So I can see the method the extension invoked, the web performance, the business logic performance, and the database performance.
Lastly, I automatically invoke browsers to point at that specific URL and record their performance along with other test information, errors, etc on any given client system using Selenium:
http://docs.seleniumhq.org/
This way I can break down the tests in terms of browser, extension, server, application, and database and link them all together according to specific test sets. It takes a bit of work to put it all together, but once its done you can have a very nice extension testing framework.
Typically for cross-browser extension development in order to maintain a single code-base I use crossrider, but you can do this with any framework or with native extensions as you wish, Selenium won't care, it is just driving the extension to a particular page and allowing you to interact and perform tests.
One nice thing about this approach is you can use it for live users as well. If you are providing support for your extension, have a user go to your test url and immediately you will see the extension and server-side performance. You won't get the Selenium tests of course, but you will capture a lot of issues this way - very useful when you are coding against a variety of browsers and browser versions.

Browser Instrumentation

I need to mimic the behavior of a browser. For Say I need to acess the DOM properties of a webpage (ex. Document.cookie or window.onblur etc) without loading a webpage in an actual browser and without interacting (ex. clicking button, putting mouse over a link etc) in the browser.
Infact, I am trying to do some thing where I have an imaginary browser Object BROWSER. So, I can do :
BROWSER g = BROWSER.load('google.com');
g.document.cookie();
g.window.onblur();
I guess this is known as 'browser instrumentation'. How Can I do it ? Any ideas.. ?

Your best bet is probably Selenium (http://seleniumhq.org/). Although it does use a real browser (which is sort of necessary if you want to test things in a real browser environment), it will allow you to completely automate/control that browser to make it do whatever you want. Using it you can write code like this:
# Pseudo-code to search for Selenium on Google
browser.open('www.google.com')
browser.findElementByCSS('#search').value('Selenium')
browser.findElementByCSS('#submitButton').click()
which sounds like what you are trying to do.

Is this something you're looking for? It's called PhantomJS
From the site:
Full web stack, No browser required
PhantomJS is a headless WebKit with JavaScript API. It has fast and native support for various web standards: DOM handling, CSS selector, JSON, Canvas, and SVG.
Run functional tests with frameworks such as Jasmine, QUnit or CasperJS.

Mechanize and Javascript

I want to use Mechanize to simulate browsing to a web page with active JavaScript, including DOM Events and AJAX, and so far I've found no way to do that.
I looked at some Python client browsers that support JavaScript like Spynner and Zope, and none of them really work for me. Spynner crashes PyQt all the time, and Zope doesn't support JavaScript as it seems.
Is there a way to simulate browsing with Python only (no extra processes) like WATIR or libraries that manipulate Firefox or Internet Explorer while supporting Javascript fully as if actually browsing the page?

I've played with this new alternative to Mechanize (which I love) called Phantom JS.
It is a full web kit browser like Safari or Chrome but is headless and scriptable. You script it with javascript, not python (as far as I know at least).
There are some example scripts to get you started. It's a lot like using Firebug. I've only spent a few min using it but I found I was quite productive right from the start.

From http://wwwsearch.sourceforge.net/mechanize/faq.html#general
If you come across this in a page you want to automate, you have four options. Here they are, roughly in order of simplicity.
Figure out what the JavaScript is doing and emulate it in your Python code: for example, by manually adding cookies to your CookieJar instance, calling methods on HTMLForms, calling urlopen, etc. See above re forms.
Use Java’s HtmlUnit or HttpUnit from Jython, since they know some JavaScript.
Instead of using mechanize, automate a browser instead. For example use MS Internet Explorer via its COM automation interfaces, using the Python for Windows extensions, aka pywin32, aka win32all (e.g. simple function, pamie; pywin32 chapter from the O’Reilly book) or ctypes (example). This kind of thing may also come in useful on Windows for cases where the automation API is lacking. For Firefox, there is PyXPCOM.
Get ambitious and automatically delegate the work to an appropriate interpreter (Mozilla’s JavaScript interpreter, for instance). This is what HtmlUnit and httpunit do. I did a spike along these lines some years ago, but I think it would (still) be quite a lot of work to do well.

Basically if you want something that deals with javascript then you need a real javascript engine, these invariably involve automating a real browser (I'm including headless ones in this).
Java’s HtmlUnit doesn't do a very good job as it doesn't use a javascript engine from an actual browser. Phantom JS sounds ideal (as newz2000 points out) however I find that when manipulating pages with javascript it can be very difficult to debug your script if you can't actually see the page you're dealing with.
This leads to solutions such as Selenium Webdriver which has a full python API to automate various browsers, however you must run a java jar and it actually launches the browser, so not the pure python solution you're after (but I think this is as close as you can get).

You can use Selenium with Python. You can then scrape JavaScript-generated content as well as manipulate the page with additional JavaScript (as well as Python).
# In your virtualenv: pip install selenium
from selenium import webdriver
# Launch Firefox GUI
browser = webdriver.Firefox()
# Alternatively, you can drive PhantomJS without a GUI
# With Node.js installed: `npm install -g phantomjs`
# browser = webdriver.PhantomJS()
# Fetch a webpage
browser.get('http://example.com')
# If you need the whole HTML document
# just like inspecting the rendered page with the console
html = browser.page_source
# Get an element, even if it was created with JS
button = browser.find_element_by_css_selector('div.some-class > \
input.the-submit-button')
# Click on something
button.click()
# Execute some JavaScript (assumes jQuery is loaded on the page)
browser.execute_script("$('html, body').animate({ scrollTop: 500 }, 50);")
You can run the code in a Python REPL and use autocomplete to discover the methods available on browser or whatever element you have selected. Or do something like print(dir(browser)) to see what is available.

An example how to use PyV8, to run JS on a DOM with python can be found here:
https://github.com/buffer/thug
This should be fairly easy to make it run together with mechanize.

Help on testing with Selenium

I'm trying to make some tests on a JavaScript application and someone advised me to use Selenium. I visited its site but I cannot understand what is it and how can I use it for testing. Can someone help me understand?

There are a lot of options and it can be quite daunting to start.
Start with the IDE. It is a Firefox plug-in and would get you writing tests in no time. This is good for semi-automated tests running only on Firefox. And good to get some scripts generated for you to kick-start your tests.
Setup RC. It is a Java program that runs on 'a' box (could be localhost) spawning browsers and running your tests and you can connect to it using variety of languages and program your tests. RC is your friend if you want to automate your testing completely.
As for Grid, it is yet another Java program that manages different RCs on your network which makes it all distributed from browser, load and functionality perspectives. You don't need this initially and when the time comes your work on RC would be reusable 80-100%.

If you're using the Firefox plugin, all you have to do is record a "test". Then generate the testing code in the language you want to run the scripts in. They have an option for Java - and the test can be run standalone (outside of a browser on any platform). The test will attempt to replicate what you did in the browser. If it is able to complete the same steps, your test passes.
Selenium replicates what the browser does when running it's tests and does an admirable job (though not perfect)