Parallel End to End Testing With Selenium Services

This package provides an example Node.js setup for running parallel end to
end Selenium tests using one of these cloud services:

* SauceLabs
* BrowserStack
* TestingBot

These services each provide an SSH tunnel application that allows cloud Selenium
servers to run tests against your local web server, or against a web server in
your internal network. By launching the tunnel you connect an endpoint on your
local machine to the remote Selenium service, allowing it to access the servers
that you can see.

Parallel testing can be highly efficient. Since most of the time taken by a test
thread is spent waiting on responses from the remote server, it is perfectly
feasible to run dozens or even hundreds of parallel end to end test suites
through a single SSH tunnel on a single developer machine.

Given that the Selenium tests for a major project usually require hours to run
serially, the ability to easily run in parallel is very helpful. It can crush
down the time taken to a much more reasonable span.

Installation

Install the example package via NPM:

``
npm install selenium-service-example
`

Setup

You will need to obtain a free account with one of the services noted above.
This won't take long - it is a painless signup. Once registered you can find the
necessary account name, API secret and/or key in your account settings.

Add your newly obtained credentials to the relevant section of
config/index.js and set the service property in that file to either
browserlab, saucelabs, or testingbot, depending on which service you are
using.

A free account grants you access to very little in the way of resources, but
it will be sufficient to try out this example of how to set up parallel end to
end tests.

Running the Example Tests

Fire up the Vagrant Ubuntu 12.04 VM; it will provision itself with the latest
stable Node.js version:

`
vagrant up
vagrant ssh
`

Vagrant is used because the SSH tunnel binaries for SauceLabs and BrowserStack
require glibc 2.15 or later as of January 2014. If your host machine is CentOS
or an older version of another distro then the tunnel won't work locally - but
it will in the VM.

Once logged in to the Vagrant VM:

`
cd /vagrant
node runTests
`

This will:

* Launch a simple Express server.
* Launch the SSH tunnel for the configured Selenium service.
* Launch parallel test processes, each running its own set of end to end tests.
* Shut down the SSH tunnel.
* Shut down the Express server.
* Display the results.

Note that the SSH tunnel for any of these services will sometimes fail to
initialize, or will time out waiting on the remote connection. Cloud services
are only as reliable as the ability to launch and maintain servers, which at
this point is still not all that reliable. Redundancy helps, but not
when you are waiting on one specific connection to one specific server.

Errors of this nature will shut down the process, and will be displayed in the
output. You can also look into the log directory to see log files containing
the output from each test runner process, and from the tunnel process itself.

Specifying the Configuration File

You can specify alternative configuration files when running tests, which should
help with experimentation. Paths must be relative to the project directory.
e.g.:

`
node runTests config/my-configuration-file.js
``

Potential Improvements

This is a very simple, crude setup. It makes no attempt to be smart about
monitoring the tunnel stdout and stderr pipes for known issues that may cause
the tunnel process to hang rather than end, for example.

Similarly, there are better ways to channel and present the output from test
processes than to just dump them to logs.

Other Notes

As of Q2 2014, the native binary SSH tunnels for SauceLabs and BrowserStack are
still fairly new. They are not yet as robust as the old Java SSH tunnels that
were used by these services up until Q1 2014, and nor are they as capable of
handling as great a number of test threads.

In BrowserStack's case, you should not try to run more than about 8-10 parallel
test threads through the same tunnel instance as it will definitely run into
issues. If you want greater concurrency then run several tunnel instances in
parallel and split your threads between them. You will also have to impose a
delay of 10-20 seconds between the launch of each tunnel instance, otherwise
BrowserStack's server infrastructure becomes confused and may reject WebDriver
initialization attempts.