Sonar + maven configuration + Jenkins

SonarQube requires quite some configuration to be fully useful!

This article explains how to configure your maven projects to produce reports + how to configure Jenkins with SonarQube, bringing both Unit and Integration tests coverage.

Principle

What are Unit / Integration tests ?

TODO: nice picture

Process overview

To work well SonarQube requires a bit of Maven configuration + Jenkins build adjustment:

• Maven specific properties
• Maven build plug-ins
  • Maven-jacoco (code coverage tool)
  • Maven-surefire (unit tests reports)
  • Maven-failsafe (integration tests reports)
• Maven jacoco dependency

Key points:

• SONAR is configured for JAVA language and it will use Jacoco as coverage tool.
• Each maven module will have its own Unit Tests results (Surefire reports + .exec file) inside its own target directory
• The Integration Tests results are common. Meaning, all modules will write in the same directory. That directory is relative to the maven execution.

Maven configuration

Sonar properties

First of all you need to configure SONAR by setting up some properties and paths.

Technical details:

• Surefire unit tests plugin
  • Outputs in ${project.build.directory}
  • 1 XML file per test class in: ${sonar.junit.reportsPath}
  • SONAR report file (jacoco-ut.exec) will be available at ${sonar.out.unitTestsReport}
• Failsafe integration tests plugin
  • All outputs in the same directory, relative to the local execution: ${session.executionRootDirectory}

POM.xml extract:

    <properties>
        ...

        <!-- Maven plugins -->
        <maven-surefire-plugin.version>2.18.1</maven-surefire-plugin.version>
        <maven-failsafe-plugin.version>2.18.1</maven-failsafe-plugin.version>
 
        <!-- Test frameworks -->
        <junit.version>4.12</junit.version>
 
        <!-- ==== SONARQUBE quality metrics ==== -->
        <maven.jacoco.version>0.7.4.201502262128</maven.jacoco.version>
        <!-- Global Sonar settings. Do not change them! -->
        <sonar.language>java</sonar.language>
        <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
        <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
        <jacoco.lib.path>
            ${settings.localRepository}/org/jacoco/org.jacoco.agent/${maven.jacoco.version}/org.jacoco.agent-${maven.jacoco.version}-runtime.jar
        </jacoco.lib.path>
        <javaagent>${jacoco.lib.path}</javaagent>
        <!-- Don't let Sonar execute tests. We will let Maven do it during build phase -->
        <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
        <!-- Report -->
        <!-- IMPORTANT: integration test path must be ABSOLUTE, especially for muli-modules projects -->
        <!--            ${session.executionRootDirectory} = directory from where the "mvn" command is run -->
        <!-- a) Where sonar will find the standard test reports -->
        <sonar.surefire.reportsPath>${project.build.directory}/surefire-reports</sonar.surefire.reportsPath>
        <sonar.failsafe.reportsPath>${session.executionRootDirectory}/target/failsafe-reports</sonar.failsafe.reportsPath>
        <!-- b) Sonar specific reports -->
        <sonar.jacoco.reportPath>${project.build.directory}/jacoco-unit.exec</sonar.jacoco.reportPath>
        <sonar.jacoco.itReportPath>${session.executionRootDirectory}/target/reports/jacoco-it.exec</sonar.jacoco.itReportPath>
    </properties>

Jacoco plugin (coverage agent)

This is how you should configure this plugin:

    <build>
        <plugins>
            <!-- == Jacoco agent configuration == -->
            <!-- This will auto-generate the right jacoco command for Unit | Integration tests -->
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>${maven.jacoco.version}</version>
                <executions> 
                    <!-- Unit tests configuration -->
                    <execution>
                        <id>prepare-unit-test-agent</id>
                        <phase>process-test-classes</phase>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                        <configuration>
                            <destFile>${sonar.surefire.reportsPath}</destFile>
                            <propertyName>jacoco.agent.ut.arg</propertyName>
                            <append>true</append>
                        </configuration>
                    </execution>
                    <!-- Integration tests configuration -->
                    <execution>
                        <id>prepare-integration-test-agent</id>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                        <configuration>
                            <destFile>${sonar.jacoco.itReportPath}</destFile> 
                            <propertyName>jacoco.agent.it.arg</propertyName>
                            <append>true</append>                 
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin> Surefire (UT) </plugin>
            <plugin> Failsafe (IT) </plugin>

        </plugins>
    </build>

This will defined 2 properties, depending on the maven phase:

	Phase	ArgLine	Value
Unit Tests	process-test-classes	jacoco.agent.ut.arg	-javaagent:${jacoco.lib.path}=destFile=${sonar.jacoco.reportPath},append=true
Integration Tests	pre-integration-test	jacoco.agent.it.arg	-javaagent:${jacoco.lib.path}=destFile=${sonar.jacoco.itReportPath},append=true

These properties will be re-use in the corresponding maven plugin.

Surefire (UT tests)

Surefire will process the Unit Tests.

!! Do not use Surefire for your integration tests because Surefire produce different outputs per maven module !! Integration tests requires to use 1 output for all !

    <build>
        <plugins>

            <plugin> Maven-jacoco </plugin>

            <!-- == process UNIT tests == -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven-surefire-plugin.version}</version>
                <configuration>
                    <!-- Do not run if no tests -->
                    <skipTests>${skipTests}</skipTests>
                    <!-- a) Surefire must create junit reports where sonar can find it later during analysis --> 
                    <reportsDirectory>${sonar.jacoco.reportPath}</reportsDirectory>
                    <!-- b) Jacoco specific settings (command auto-generate by Sonar plugin) -->
                    <argLine>${jacoco.agent.ut.arg}</argLine>
                    <!-- Exclude integration tests -->
                    <excludes>
                        <exclude>**/*IT.java</exclude>
                        <exclude>**/*IntegrationTest.java</exclude>
                        <exclude>**/*Gwt.java</exclude>
                    </excludes>
                </configuration>
            </plugin>

            <plugin> Failsafe (IT) </plugin>

        </plugins>
    </build>

Notes:

• No tests will be run if mvn clean install -DskipTests
• Each module has its own surefire-reports + Sonar report
• The exclusion list is something that depends on your own situation

Failsafe configuration (IT tests)

Failsafe will process the Integration Tests.

!! Do not use Surefire for your integration tests because Surefire produce different outputs per maven module !! Integration tests requires to use 1 output for all !

    <build>
        <plugins>

            <plugin> Maven-jacoco </plugin>
            <plugin> Surefire (UT) </plugin>

            <!-- == process INTEGRATION tests == -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>${maven-failsafe-plugin.version}</version>
                <configuration>
                    <!-- Do not run if no tests -->
                    <skipTests>${skipIntegrationTests}</skipTests>
                    <!-- a) Failsafe must create junit reports where sonar can find it later during analysis -->
                    <reportsDirectory>${sonar.failsafe.reportsPath}</reportsDirectory>
                    <!-- JVM settings -->
                    <argLine>-Xmx1024m -XX:maxPermSize:256m</argLine>
                    <forkCount>1</forkCount>
                    <reuseForks>true</reuseForks>
                    <!-- <runOrder>alphabetical</runOrder> -->
                    <!-- Sonar specific settings (command auto-generate by Sonar plugin) -->                  
                    <argLine>${jacoco.agent.it.arg}</argLine>
                    <includes>
                        <include>**/*IntegrationTest.java</include>
                        <include>**/*IT.java</include>
                    </includes>
                    <excludes>
                        <exclude>**/*Gwt.java</exclude>
                    </excludes>
                </configuration>
                <!-- Only run failsafe when required -->
                <executions>
                    <execution>
                        <id>integration-test</id>
                        <phase>integration-test</phase>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>

Notes:

• No tests will be run if mvn clean install -DskipIntegrationTests
• Only 1 report for all modules, that depends on the ${session.executionRootDirectory}
• The exclusion | inclusion list is something that depends on your own situation
• The JVM settings can be arranged

Maven dependencies

At last, you need to add the jacoco dependency:

    <dependencies>
        ...
        <!-- SonarQube dependencies -->
        <dependency>
            <groupId>org.jacoco</groupId>
            <artifactId>org.jacoco.agent</artifactId>
            <version>${maven.jacoco.version}</version>
            <classifier>runtime</classifier>
            <scope>test</scope>
        </dependency>
    </dependencies>

Now your Maven configuration is complete at last.

Jenkins configuration

Now that you can generate these reports, especially the integration results, you need to configure Jenkins to use them!

To set up the Sonar Jenkins plugin to process already generated JaCoCo data files, the following properties must be specified in the "Additional properties" text box, under the Sonar installation being configured:

• -Dsonar.dynamicAnalysis=reuseReports ==> Tells SonarQube to reuse existing reports for unit tests execution and coverage reports
• -Dsonar.jacoco.itReportPath={ABSOLUTE_PATH_TO_JENKINS}/workspace/target/reports/jacoco-it.exec => integration tests report to process

Real life example

You can check out one of my GitHub project:

• https://github.com/guihome-diaz/Languages > pom.xml

Even better, you can check out this excellent tutorial:

• https://github.com/dgageot/coverage

Appendices

Run tests in specific order

If for some reasons you need to run your tests in a particular order you can say do so in Maven and/or Java

Reminder : if you have to run your tests in a particular order then you should refactor your tests! These tricks should not let you escape from that!

Maven

The runOrder is a nice trick.

    <build>
        <plugins>
            <plugin>
                SUREFIRE or FAILSAFE

                <configuration>
                    <!-- JVM settings -->
                    <argLine>-Xmx1024m -XX:maxPermSize:256m</argLine>
                    <forkCount>1</forkCount>
                    <reuseForks>true</reuseForks>
                    <runOrder>alphabetical</runOrder>
                    <!-- Jacoco execution -->                 
                    <argLine>${jacoco.agent.it.arg}</argLine>
                    ...
                </configuration>
            </plugin>

        </plugins>
    </build>

Java

Since jUnit 4.11 you can set the order of your tests.

• Set order to the Class level: https://github.com/junit-team/junit/wiki/Test-execution-order
• Set custom order: http://memorynotfound.com/run-junit-tests-method-order/

References

This article is based on my daily work in VEHCO.

To update to SonarQube I used the following articles and code examples:

• http://www.aheritier.net/maven-failsafe-sonar-and-jacoco-are-in-a-boat/
• VERY good GitHub example, provided by the same author: https://github.com/dgageot/coverage
• http://stackoverflow.com/questions/26253277/jacoco-agent-for-integration-tests-on-remote-machine

Jenkins configuration:

• http://blog.ccbill.com/2014/07/code-coverage-with-surefire-and-jacoco.html