Running Selenium Tests with Maven

Selenium and Maven are some of the quickest-acquired friends I have in the world of software development. They do the hard work of testing web applications while I sip my coffee navigating through the code to add fun new features. I'm sure you'll want them to be your friends too, so here's what I found working best in my projects to have them set-up and running together.

Organizing the Environment

Speaking in the language of Maven, Selenium tests fall into the category of integration tests. As such they will be running in the integration-test phase during Maven's build lifecycle, which comes right after package and just before verify. These tests usually are time consuming and if you use the install or deploy goal to have your application deployed on a server for manual testing, you certainly don't want these tests to run each time.

The path to follow here is putting your Selenium tests in a separate Maven module, under one parent POM with the rest of the application modules, and creating a new profile to run them only when explicitly desired. Creating a new module goes as usual, by calling from the parent POM's directory:

mvn artifact:create -DgroupId=[group] -DartifactId=selenium-test-suite -Dpackaging=pom

The artifactId may of course be changed to whatever you feel like. Notice that the packaging type is pom. The reason for that is, that this very module will only contain test classes and no other sources, hence choosing pom packaging will spare us a couple of unnecessary lifecycle phases. You should delete the src/main directory in the newly created module - it won't be needed.

Having pom as the packaging type disables compiling any kind of sources, including test classes, which we do need. To fix this you'll need to configure the maven-compiler-plugin in the test module's POM to actually execute the testCompile goal:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>

We also won't have any unit tests in this module, so we need to tell Maven to skip the usual test phase and have it run when the integration-test goal is reached:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
<executions>
<execution>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>false</skip>
</configuration>
</execution>
</executions>
</plugin>

Last but not least, we need that separate profile for running Selenium tests, which requires editing the parent POM, deleting the selenium-test-suite entry from the list of regular modules and putting the following in the POM:

<profiles>
<profile>
<id>integration</id>
<modules>
<module>selenium-test-suite</module>
</modules>
</profile>
</profiles>

Now whenever Selenium tests need to be run, the parent POM should be called with:

mvn verify -P integration

Where verify may be any other goal including or coming after integration-test.

The Eclipse Case

If you're like me, developing in Eclipse while not having the IDE's artifacts in the code repository and creating them with mvn eclipse:eclipse after checkout, you'll notice that our newly created test suite will refuse to comply. The reason is that the pom packaging type is usually used for multi-module Maven projects and hence only its modules would be imported into Eclipse's workspace. It has been recognized though, that very often pom packaging is used for other purposes and a feature request has already been filed, slated for inclusion in Maven 2.1, to have Eclipse's artifacts generated when no modules for a pom-packaged project exist.

For now you'll have to use a simple workaround, by changing the packaging type of the test suite temporarily to say jar, then running mvn eclipse:eclipse and editing the packaging back to pom. It's usually only required once, after checking out the project from the repository and later when the test suite's or any parent's POM changes, hence shouldn't be that much of a hassle.

Running the Selenium Server

Selenium needs its own server to run any tests and we'll obviously need it up and running when the integration-test phase kicks in, as well as have it killed after testing finishes. Luckily Maven allows us to use two convenient goals - pre-integration-test and post-integration-test, one intuitively running before the test phase and one after.

Starting and stopping the server is conveniently done with the selenium-maven-plugin. You'll need to put the following into your test-suite's POM:

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>selenium-maven-plugin</artifactId>
<executions>
<execution>
<id>start-server</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start-server</goal>
</goals>
<configuration>
<background>true</background>
</configuration>
</execution>
<execution>
<id>stop-server</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop-server</goal>
</goals>
</execution>
</executions>
</plugin>

The tests will also need a session to be running, defining which browser should be used and what address is to be called to get to the application. It's doable from the POM but troublesome and not very flexible. We'll control the sessions in the test classes then later on when writing test code.

Creating Test Classes

To have our tests compiling and running we'll need to add a dependency to the test suite module on the Selenium Java Client drivers:

<dependency>
<groupId>org.openqa.selenium.client-drivers</groupId>
<artifactId>selenium-java-client-driver</artifactId>
<version>0.9.2</version>
<scope>test</scope>
</dependency>

Since this isn't a dependency available in the default repository, you'll also need to add a new one to whichever POM suits you best - parent one perhaps:

<repositories>
<repository>
<id>openqa</id>
<name>OpenQA Repository</name>
<url>http://maven.openqa.org</url&gt;
<layout>default</layout>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
</repositories>

Our tests will be regular unit tests, with regular assert* statements, with operations on Selenium provided objects controlling test behaviour. I assume you're using unit tests project-wide already, so a dependency on jUnit is present somewhere in the POM structure.

As stated previously, Selenium does require starting up a session before running tests, which includes opening a pre-defined browser. We want to have the browser run before each test and killed after, to have a clean session each time a test runs, which brings us to the @Before and @After constructs of jUnit. And since we don't want to put them in each test class separately, we'll create an abstract class containing those, which will become the base for all other test classes we'll write:

public abstract class AbstractSeleniumTestCase {

protected Selenium selenium;

@Before
public void setUp() {
selenium = new DefaultSelenium("localhost", SeleniumServer
.getDefaultPort(), "*firefox", "http://localhost:8080/");
selenium.start();
}

@After
public void tearDown() {
selenium.stop();
}

}

Now every time you create a new test class, make it extend the above. If the new test class requires some additional @Before and @After procedures, make sure they call the super-class's methods:

public class MyTest
extends AbstractSeleniumTestCase {

@Before
public void setUp() {
super.setUp();
[additional code here]
}
}

Designing a test is best now with the regular Selenium method - by using Selenium IDE. It comes conveniently prepared as a Firefox plugin, opening a sidebar and registering all actions done on a website. Do the following:

1. Go to the page under testing
2. Start recording your actions
3. Do your clicks, writes, selections etc.
4. Stop recording
5. Change the Selenium IDE display by choosing Options > Format > Java (you can also choose to export the whole test as a Java file, but since it doesn't create a pure jUnit test, I find my way to be more convenient)
6. Copy the test procedure and paste it into your @Test-annotated method
   

Add assert* statements wherever you find necessary, checking conditions is usually done with different selenium.is* methods (my personal favourite: isSomethingSelected). Make sure you know and understand how to locate your element and value of choice.

Pages Requiring Authentication

It's fairly common that the system you are testing requires most tests to run in an authenticated environment. In that case you'll need a login procedure in place before any test starts (especially since we're restarting the browser for each test). In that case just create another abstract class, extending the base one we created earlier, performing authentication in a @Before-annotated method:

public abstract class AbstractAuthenticatedAccessTest
extends AbstractSeleniumTestCase {

@Before
public void setUp() {
super.setUp();

[authentication procedures here]
}

}

The authentication procedure would be yet another recorded with Selenium IDE. You'll obviously need to have a user added to the system before anyone can login, which may also be done in a @Before method in any of the abstract classes. If different roles are needed for testing (ie. user, moderator, admin etc.) create separate abstract classes for each and extend them as required.


You're done. Running mvn verify -P integration from the parent POM should get your tests running and checking what you ordered them to.

I left out the bit about automatically deploying the whole application for testing and starting/stopping the server, which I'd like to write about at a different time, possibly in connection with setting up Continuum. If you need that in the mean time, you should get your hands on Cargo. As for the above, as usual I'm expecting comments on successes, failures and things I missed out on.

Building Eclipse Plugins with Maven 2

Update: added one step to add some dependencies to the psteclipse plugin after installing it in the repository.

These boys don't place nice. Both strong, both handsome and both sturdy as it gets when trying to make them work together. Getting Maven to compile Eclipse RCP plugins while retaining the ability to develop and test within the IDE has been a tough week's job.

Three different approaches I tried were, in order of appearance:

1. maven-psteclipse-plugin, developed as a custom solution by PrincetonSoftech and released for public use,
2. maven-pde-plugin, using Eclipse's PDE Ant scripts to build the plugins,
3. maven-bundle-plugin, my final sign of desperation, resorting to default OSGi tools and methods.
   

All of them were fine to generate or manage Eclipse-specific artifacts, each of them I made eventually to build single plugins, but none managed to work properly with dependencies between plugins under development.

Eventually I contacted Peter Hagelund (aka Petersen), co-author of the first solution. We exchanged a couple of e-mails, sorted out matters missing in the original article and finally found a bug in the psteclipse plugin. Here's how the whole procedure gets done.

Preliminary steps

You will need Eclipse obviously. Assuming you have already one installed for day-to-day development, you'll need another dummy one to do the automatic testing of plugins. It's a recommendation of the original article, crucial to ensure your workspace will not sustain any damage from the automatic installing and cleaning.

Download the psteclipse mojos from the link at the bottom of the article. Unpacking the zip will show a folder with the sources and the pom.xml file, trying to build it though will likely fail. The POM requires a parent artifact which is missing from the download. If you're interested, you can checkout the whole structure from Codehaus's SVN. Luckily for us there is a target folder within the download containing an already built .jar with the plugin. That need only be installed, which is done with the following command (the parts in bold need substituting for according values):

mvn install:install-file -Dfile=/[path to folder]/target/maven-psteclipse-plugin-[version].jar -DgroupId=org.apache.maven.plugins -DartifactId=maven-psteclipse-plugin -Dversion=[version] -Dpackaging=jar -DgeneratePom=true

Check your Maven repository whether it contains the file at the specified location, which is:

org/apache/maven/plugins/maven-psteclipse-plugin/[version]

You'll also need to add two dependencies to the file there with the .pom extension. Open it and add:

<dependencies>
<dependency>
<groupid>org.apache.maven</groupid>
<artifactid>maven-project</artifactid>
<version>2.0</version>
</dependency>
<dependency>
<groupid>org.apache.maven</groupid>
<artifactid>maven-archiver</artifactid>
<version>2.2</version>
</dependency>
</dependencies>

You can't really call the psteclipse plugin until after setting up a POM for your project, which first requires deciding on the classpath for your plugins. Eclipse requires plugin names to be the same as their classpaths, in which case if I were to develop plugins foo and bar, I'd call them:

com.buggybrain.foo
com.buggybrain.bar

These will also be the artifactId's Maven uses and in turn the folder names for Maven modules under the super POM. My common classpath for all plugins would in this case be:

com.buggybrain

and that's the groupId for all my plugins as well as for Eclipse plugins required for building the whole project. The latter one is a decision made by the psteclipse team, so unless you feel like messing with their mojo code, you'll have to follow it.

Once the decision is made, you can set up the parent POM (aka the super POM). For this you can easily take the template provided in the article. Just make sure to exchange com.princetonsoftech wherever you can find it to the classpath you just decided to use (eg. com.buggybrain).

A note about versions here - it's easiet to keep the version stated only in the super POM and remove it from separate plugin poms - this way it gets inherited down the hierarchy and updating it requires changing only one value. Make sure though your version name includes the -SNAPSHOT suffix. Psteclipse expects that and requires whenever it looks for dependencies between plugins.

With the super POM in place, you should be able to scrape the Eclipse directory for artifacts needed to build your plugins. This is done with the mvn psteclipse:deploy command, executed in the location of the super POM. You'll need to provide two parameters:

• eclipseHome - the directory of the Eclipse installation to scrape
• groupId - the classpath (Maven's group) the plugins will be put in

Psteclipse will also ask you to provide to other parameters - repositoryId and url, but unless you'll be deploying into a remote repository, you need not care about those and provide just any values. The whole command can look like this:

mvn psteclipse:deploy -DeclipseHome=/home/sonic/tmp/eclipse -DgroupId=com.buggybrain -DrepositoryId=a -Durl=a

It will take a moment to fetch all the plugins, copy the jars to the current directory and create two files - install.txt and deploy.txt containing a list of Maven commands to install all the artifacts into a local repository or deploy them to a remote one. Now all you have to do is run commands in whichever file suits you, ie.

bash < install.txt

You'll have to clean up after the deployment and delete all the Eclipse artifacts from the parent POM's directory.

Preparing projects

Decide on which plugins will be binary (no source code, only jars available to other plugins) and which will be source (no jars, only source code). In practice it is possible to have a source plugin containing jars, but psteclipse will make your life hard managing such a combination later, so I recommend you do stick to the convention.

Create folders for some of your plugins, the less for a start the better - you'll have to make sure it all works before adding more stuff later. You can easily use Maven's mvn archetype:create command here. Make sure to change the packaging types to binary-plugin and source-plugin accordingly, as stated in the original article.

Source plugins will require the MANIFEST.MF file in the META-INF directory, which you can easily create empty by hand right now and edit in Eclipse later.

Binary plugins need their pom.xml files edited to include all the dependencies they're suppose to make available to other plugins.

The stuff to commit to your code repository for binary plugins is:

• pom.xml

while your source plugins will require these elements commited:

• pom.xml
• src/ folder
• META-INF/ folder with the MANIFEST.MF inside
  

Now you may execute mvn eclipse:eclipse in the parent folder, then mvn psteclipse:eclipse-plugin for binary plugins separately and import all projects into Eclipse according to the procedures described in the original article. Edit the MANIFEST.MF files for all source plugins, make sure to put names, versions, depencencies with versions (those need to be checked with actual values from Eclipse) and exported packages for dependencies between plugins.

At the end of this process Eclipse should report no build errors.

Compiling with Maven

Back in the Maven world the usual thing you'd do now is go to the parent folder and execute mvn compile to make sure everything is OK. You'll fail. The psteclipse plugin contains a bug here (acknowledged by Peter and to be corrected), which prevents it from seeing dependencies on plugins under development, only looking at the repository.

To make it work you'll have to call mvn install. This will compile each plugin and install it into the local repository, then move to the next one. Whenever one plugin depends on another, the previous one should have its jar installed in Maven's repository and the procedure should go smoothly.

If at this point Maven still reports any missing dependencies between plugins you're working on, check the order of module declarations in the super POM and make sure all plugins that are depended on are placed before those that depend on them.


The above should have covered the whole process. I might have missed something on the way, so I'll be more than happy to receive comments with success as well as failure stories.