I am developing chat application, using node.js socket.io module on a server, and javascript on a client. As it become more complex, I become tired to test different usage scenarios by clicking across browsers. At now I use different browsers in order to separate cookies. So in Firefox there is one chat participant, in Opera - second, etc.
Therefore my question is:
Can I automate this somehow?
I've heard about headless testing engines like Phantom JS, Selenium WebDriver, but I don't know how to write tests between different browsers. In fact, I indeed need to separate cookies, different browsers are here only for this task.
May be I cannot explain my problem to Google :) All answers on such question imply cross-browser development, and testing that aspect.
Any help will be appreciated a lot!
The answers shown at How to test two interacting browsers (e.g. chat app) might be of use if you try to go the Casper/Phantom/SlimerJS route. I'm not marking it as a duplicate as your question is more general, whereas that one is about exactly how to synchronize two CasperJS instances; it is also not specifically about socket.io.
Note: if using Phantom to test Web Sockets you will need to get Phantom 2.x (which is still experimental, and does not have an easily downloadable beta yet), as Phantom 1.x uses an old WebKit. SlimerJs always uses a recent Gecko (i.e. Firefox) engine, so is fine.
Each instance of PhantomJS can have its own cookies. This answer (and the other answer on that question) explains your choices well.
SlimerJS cookies are similar, but persistent cookies are stored in the Mozilla profile, so if you wanted to run multiple instances, you would want to specify an explicit profile for each instance. (Though your chat application may only be using session cookies, in which case this does not matter.)
Anyway, as cookies are used I would definitely run two instances of CasperJS, rather than try to use two tabs, or two frames.
Related
I want to identify a user by unique machine id across all browsers. So came up with an idea of launching a app which determines the machine-id (like electron using nodejs) and passes back the info to the browser that launched it and use for identifying the user.
I have seen something like this been done on canyourunit
Can anyone point me right direction where can i find some info about this kind of approach.
As you would be asking a user to download and execute native code, electron or whatever else, there are several complicated issues to consider.
The obvious first one is that you must consider the user platform (OSX, Windows, Linux...) then, communicating back with the browser directly (which one?) is probably not practical.
Your native application could however send the machine id/fingerprint to your backend so you could identify what web client is what.
This is quite involved, fragile, and even if you only support Windows, will fail in many scenarios (system policies, firewalls, antivirus, proxies, strange network setups...)
All in all, it would seem that you should consider more standard Device fingerprinting techniques if you only need an ID.
Is there any way to consistently detect PhantomJS/CasperJS? I've been dealing with a spat of malicious spambots built with it and have been able to mostly block them based on certain behaviours, but I'm curious if there's a rock-solid way to know if CasperJS is in use, as dealing with constant adaptations gets slightly annoying.
I don't believe in using Captchas. They are a negative user experience and ReCaptcha has never worked to block spam on my MediaWiki installations. As our site has no user registrations (anonymous discussion board), we'd need to have a Captcha entry for every post. We get several thousand legitimate posts a day and a Captcha would see that number divebomb.
I very much share your take on CAPTCHA. I'll list what I have been able to detect so far, for my own detection script, with similar goals. It's only partial, as they are many more headless browsers.
Fairly safe to use exposed window properties to detect/assume those particular headless browser:
window._phantom (or window.callPhantom) //phantomjs
window.__phantomas //PhantomJS-based web perf metrics + monitoring tool
window.Buffer //nodejs
window.emit //couchjs
window.spawn //rhino
The above is gathered from jslint doc and testing with phantom js.
Browser automation drivers (used by BrowserStack or other web capture services for snapshot):
window.webdriver //selenium
window.domAutomation (or window.domAutomationController) //chromium based automation driver
The properties are not always exposed and I am looking into other more robust ways to detect such bots, which I'll probably release as full blown script when done. But that mainly answers your question.
Here is another fairly sound method to detect JS capable headless browsers more broadly:
if (window.outerWidth === 0 && window.outerHeight === 0){ //headless browser }
This should work well because the properties are 0 by default even if a virtual viewport size is set by headless browsers, and by default it can't report a size of a browser window that doesn't exist. In particular, Phantom JS doesn't support outerWith or outerHeight.
ADDENDUM: There is however a Chrome/Blink bug with outer/innerDimensions. Chromium does not report those dimensions when a page loads in a hidden tab, such as when restored from previous session. Safari doesn't seem to have that issue..
Update: Turns out iOS Safari 8+ has a bug with outerWidth & outerHeight at 0, and a Sailfish webview can too. So while it's a signal, it can't be used alone without being mindful of these bugs. Hence, warning: Please don't use this raw snippet unless you really know what you are doing.
PS: If you know of other headless browser properties not listed here, please share in comments.
There is no rock-solid way: PhantomJS, and Selenium, are just software being used to control browser software, instead of a user controlling it.
With PhantomJS 1.x, in particular, I believe there is some JavaScript you can use to crash the browser that exploits a bug in the version of WebKit being used (it is equivalent to Chrome 13, so very few genuine users should be affected). (I remember this being mentioned on the Phantom mailing list a few months back, but I don't know if the exact JS to use was described.) More generally you could use a combination of user-agent matching up with feature detection. E.g. if a browser claims to be "Chrome 23" but does not have a feature that Chrome 23 has (and that Chrome 13 did not have), then get suspicious.
As a user, I hate CAPTCHAs too. But they are quite effective in that they increase the cost for the spammer: he has to write more software or hire humans to read them. (That is why I think easy CAPTCHAs are good enough: the ones that annoy users are those where you have no idea what it says and have to keep pressing reload to get something you recognize.)
One approach (which I believe Google uses) is to show the CAPTCHA conditionally. E.g. users who are logged-in never get shown it. Users who have already done one post this session are not shown it again. Users from IP addresses in a whitelist (which could be built from previous legitimate posts) are not shown them. Or conversely just show them to users from a blacklist of IP ranges.
I know none of those approaches are perfect, sorry.
You could detect phantom on the client-side by checking window.callPhantom property. The minimal script is on the client side is:
var isPhantom = !!window.callPhantom;
Here is a gist with proof of concept that this works.
A spammer could try to delete this property with page.evaluate and then it depends on who is faster. After you tried the detection you do a reload with the post form and a CAPTCHA or not depending on your detection result.
The problem is that you incur a redirect that might annoy your users. This will be necessary with every detection technique on the client. Which can be subverted and changed with onResourceRequested.
Generally, I don't think that this is possible, because you can only detect on the client and send the result to the server. Adding the CAPTCHA combined with the detection step with only one page load does not really add anything as it could be removed just as easily with phantomjs/casperjs. Defense based on user agent also doesn't make sense since it can be easily changed in phantomjs/casperjs.
Our clients use RDP sessions created by clicking a button on our site which uses the old MSRDP.cab file called in a webpage. We also utilize Virtual channels using a custom dll. Both our dll and the MSRDP.ocx are loaded through the object tag and cab files using VBScript.
As you might know, the orginal RDP connection script was written in VBScript. Apparently the ActiveX control will only work if called using VBScript. That has been working for over a decade. However the newer IE browsers and virtually all of the other browsers do not support VBScript.
I realize that I can call the new RDP program mstsc.exe from Javascript for a straight connection. However, we also pass information to the old ocx like the plugin parameters in Advancedsettings which include our dll that uses virtual channels. In addition, we pass the domain and username.
Maybe I'm searching in all the wrong places but I'm turning up no answers in trying to find how to run this ocx in javascript or another solution that would work. I'm thinking there has to be a replacement out there that I'm overlooking.
We are able to force our clients to use IE so we are currently having our clients with IE versions over 10 to use compatibility mode. However I'm sure at some point this may no longer work.
Has anyone out there had a similar problem that you have found a solution for? Any ideas or suggestions would be appreciated. Thanks!
You mentioned it yourself that in the future, your ActiveX may no longer work and supported so I strongly believe that you should invest your time in re-writing your app using a more modern approach instead.
Here's a link to a solution where it is using Jquery and ASP.NET to open an RDP connection:
Open RDP Connection window using jquery - client side
I'm going to write bunch of browser extensions (the same functionality for each popular browser). I hope, that some of the code will be shared, but I'm not sure about this yet. For sure some of extensions will use native API. I have not much experience with TDD/BDD, and I thought it's good time to start folowing these ideas from this project.
The problem is, I have no idea how to handle it. Should I write different tests for each browser? How far should I go with these tests? These extensions will be quite simple - some data in a local storage, refreshing a page and listening through web sockets.
And my observation about why is it hard for me - because there is a lot of behaviour, and not so much models, which are also dependent on a platform.
I practise two different ways of testing my browser extensions:
Unit tests
Integration test
Introduction
I will use the cross-browser YouTube Lyrics by Rob W extension as an example throughout this answer. The core of this extension is written in JavaScript and organized with AMD modules. A build script generates the extension files for each browser. With r.js, I streamline the inclusion of browser-specific modules, such as the one for cross-origin HTTP requests and persistent storage (for preferences), and a module with tons of polyfills for IE.
The extension inserts a panel with lyrics for the currently played song on YouTube, Grooveshark and Spotify. I have no control over these third-party sites, so I need an automated way to verify that the extension still works well.
Workflow
During development:
Implement / edit feature, and write a unit test if the feature is not trivial.
Run all unit tests to see if anything broke. If anything is wrong, go back to 1.
Commit to git.
Before release:
Run all unit tests to verify that the individual modules is still working.
Run all integration tests to verify that the extension as whole is still working.
Bump versions, build extensions.
Upload update to the official extension galleries and my website (Safari and IE extensions have to be hosted by yourself) and commit to git.
Unit testing
I use mocha + expect.js to write tests. I don't test every method for each module, just the ones that matter. For instance:
The DOM parsing method. Most DOM parsing methods in the wild (including jQuery) are flawed: Any external resources are loaded and JavaScript is executed.
I verify that the DOM parsing method correctly parses DOM without negative side effects.
The preference module: I verify that data can be saved and returned.
My extension fetches lyrics from external sources. These sources are defined in separate modules. These definitions are recognized and used by the InfoProvider module, which takes a query, (black box), and outputs the search results.
First I test whether the InfoProvider module functions correctly.
Then, for each of the 17 sources, I pass a pre-defined query to the source (with InfoProvider) and verify that the results are expected:
The query succeeds
The returned song title matches (by applying a word similarity algorithm)
The length of the returned lyrics fall inside the expected range.
Whether the UI is not obviously broken, e.g. by clicking on the Close button.
These tests can be run directly from a local server, or within a browser extension. The advantage of the local server is that you can edit the test and refresh the browser to see the results. If all of these tests pass, I run the tests from the browser extension.
By passing an extra parameter debug to my build script, the unit tests are bundled with my extension.
Running the tests within a web page is not sufficient, because the extension's environment may differ from the normal page. For instance, in an Opera 12 extension, there's no global location object.
Remark: I don't include the tests in the release build. Most users don't take the efforts to report and investigate bugs, they will just give a low rating and say something like "Doesn't work". Make sure that your extension functions without obvious bugs before shipping it.
Summary
View modules as black boxes. You don't care what's inside, as long as the output matches is expected or a given input.
Start with testing the critical parts of your extension.
Make sure that the tests can be build and run easily, possibly in a non-extension environment.
Don't forget to run the tests within the extension's execution context, to ensure that there's no constraint or unexpected condition inside the extension's context which break your code.
Integration testing
I use Selenium 2 to test whether my extension still works on YouTube, Grooveshark (3x) and Spotify.
Initially, I just used the Selenium IDE to record tests and see if it worked. That went well, until I needed more flexibility: I wanted to conditionally run a test depending on whether the test account was logged in or not. That's not possible with the default Selenium IDE (it's said to be possible with the FlowControl plugin - I haven't tried).
The Selenium IDE offers an option to export the existing tests in other formats, including JUnit 4 tests (Java). Unfortunately, this result wasn't satisfying. Many commands were not recognized.
So, I abandoned the Selenium IDE, and switched to Selenium.
Note that when you search for "Selenium", you will find information about Selenium RC (Selenium 1) and Selenium WebDriver (Selenium 2). The first is the old and deprecated, the latter (Selenium WebDriver) should be used for new projects.
Once you discovered how the documentation works, it's quite easy to use.
I prefer the documentation at the project page, because it's generally concise (the wiki) and complete (the Java docs).
If you want to get started quickly, read the Getting Started wiki page. If you've got spare time, look through the documentation at SeleniumHQ, in particular the Selenium WebDriver and WebDriver: Advanced Usage.
Selenium Grid is also worth reading. This feature allows you to distribute tests across different (virtual) machines. Great if you want to test your extension in IE8, 9 and 10, simultaneously (to run multiple versions of Internet Explorer, you need virtualization).
Automating tests is nice. What's more nice? Automating installation of extensions!
The ChromeDriver and FirefoxDriver support the installation of extensions, as seen in this example.
For the SafariDriver, I've written two classes to install a custom Safari extension. I've published it and sent in a PR to Selenium, so it might be available to everyone in the future: https://github.com/SeleniumHQ/selenium/pull/87
The OperaDriver does not support installation of custom extensions (technically, it should be possible though).
Note that with the advent of Chromium-powered Opera, the old OperaDriver doesn't work any more.
There's an Internet Explorer Driver, and this one does definitely not allow one to install a custom extension. Internet Explorer doesn't have built-in support for extensions. Extensions are installed through MSI or EXE installers, which are not even integrated in Internet Explorer. So, in order to automatically install your extension in IE, you need to be able to silently run an installer which installs your IE plugin. I haven't tried this yet.
Testing browser extensions posed some difficulty for me as well, but I've settled on implementing tests in a few different areas that I can invoke simultaneously from browsers driven by Selenium.
The steps I use are:
First, I write test code integrated into the extension code that can be activated by simply going to a specific URL. When the extension sees that URL, it begins running the tests.
Then, in the page that activates the testing in the extension I execute server-side tests to be sure the API performs, and record and log issues there. I record the methods invoked, the time they took, and any errors. So I can see the method the extension invoked, the web performance, the business logic performance, and the database performance.
Lastly, I automatically invoke browsers to point at that specific URL and record their performance along with other test information, errors, etc on any given client system using Selenium:
http://docs.seleniumhq.org/
This way I can break down the tests in terms of browser, extension, server, application, and database and link them all together according to specific test sets. It takes a bit of work to put it all together, but once its done you can have a very nice extension testing framework.
Typically for cross-browser extension development in order to maintain a single code-base I use crossrider, but you can do this with any framework or with native extensions as you wish, Selenium won't care, it is just driving the extension to a particular page and allowing you to interact and perform tests.
One nice thing about this approach is you can use it for live users as well. If you are providing support for your extension, have a user go to your test url and immediately you will see the extension and server-side performance. You won't get the Selenium tests of course, but you will capture a lot of issues this way - very useful when you are coding against a variety of browsers and browser versions.
Please forgive a question that has been addressed in some form or fashion previously. I have numerous test applications that run on various platforms from Windows 95, Windows XP, SUSE, RedHat, and other forms of *NIX. Currently, the mechanism has a native application that queries a database for some information then launches the test application to perform the test. This said program is a "launcher application" which I am trying to convert to a "HTML/Javascript Launcher Application". In addition, this "launcher application" needs to known when the test application is completed, then collect the test result then store the results in a database.
I have read that launching an application (executable) from HTML is not permitted, but this is entirely true in the sense (if I understand correctly). From what I have read here on SO and other sites, I have the following possible solutions:
Registering a unique protocol to an test application (SO Link)
Using Java to launch the application
Adobe AIR or its opensource counter part, Titanium by Appcelerator
Using WSH, though this will only work on Windows platforms
What would be a good (or best ) approach to solution this problem? Ideally, I would like to just use HTML and Javascript, but this may not be possible? Thanks for any advice and any example/sample code would be greatly appreciated.
Mark
Hopefully many people will think, as I do, that this is a 'bad idea'™, simply because in order for it to work you must allow your browser to launch software, which would be a huge hole in security, allowing access for all sorts of nasties.
Given you are cross platform I'd suggest you stick with a dedicated laucher application, but code it in java. This is the most common solution used in the java applications world which regularly crosses os's.
You could give JNLP (java network launch protocol) a try... Really easy to distribute and deploy full fledged applications in a platform independent manner. More or less through the browser if you like...