This article explains how to configure your maven projects to produce reports + how to configure Jenkins with SonarQube, bringing both Unit and Integration tests coverage.

Requirements

Maven must run on Java 8 or +
You must use Sonarqube 7.x or +

Principle

TODO

You just have to define the configuration once, in the parent POM.

Jenkins
- Requirements: plugins / setup
- Job configuration
Maven plugins list
- Reports generation : unit / integration tests
- Jacoco binaries : many unit tests "jacoco.exec" / 1 single aggregation for "jacoco-it.exec"
- SonarQube plugin
Jenkins usage

Maven plugins

Surefire UNIT Tests

Surefire is for UNIT tests = tests that should only use: plain java / Mockito / PowerMock / etc.

By default, the Surefire Plugin will automatically include all test classes with the following wildcard patterns:

**/Test*.java - includes all of its subdirectories and all Java filenames that start with "Test".
**/*Test.java - includes all of its subdirectories and all Java filenames that end with "Test".
**/*Tests.java - includes all of its subdirectories and all Java filenames that end with "Tests".
**/*TestCase.java - includes all of its subdirectories and all Java filenames that end with "TestCase".

If the test classes do not follow any of these naming conventions, then configure Surefire Plugin and specify the tests you want to include.

See Maven Surefire plugin documentation

(i) It is safer to exclude the integration tests patterns as well, depending on your own development setup:

**/*TestAPI.java
**/*IntegrationTest.java

Results are saved in each Maven module, in XML and TXT formats: $module/target/surefire-reports/*

    <properties>
        <!-- Set the reporting settings -->
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>


    <build>
        <plugins>
            <!-- To run UNIT tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <excludes>
                        <exclude>**/*TestAPI</exclude>
                        <exclude>**/*IntegrationTest</exclude>
                    </excludes>
                </configuration>
            </plugin>

            ...
        </plugins>
    </build>

Failsafe INTEGRATION tests

Failsafe will process the Integration Tests = tests that should use Spring test / black box tests / etc.

By default, the Failsafe Plugin will automatically include all test classes with the following wildcard patterns:

**/IT*.java - includes all of its subdirectories and all Java filenames that start with "IT".
**/*IT.java - includes all of its subdirectories and all Java filenames that end with "IT".
**/*ITCase.java - includes all of its subdirectories and all Java filenames that end with "ITCase".

If the test classes do not follow any of these naming conventions, then configure Failsafe Plugin and specify the tests you want to include.

Results are saved in each Maven module:

Results are saved in XML and TXT formats in $module/target/failsafe-reports/*
Results are saved in BINARY (jacoco format) in $module/target/jacoco-it.exec

Notes:

No tests will be run if mvn clean install -DskipIntegrationTests
The exclusion | inclusion list is something that depends on your own situation

    <properties>
        <!-- Set the reporting settings -->
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>


    <build>
        <plugins>

            <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <includes>
                        <include>**/*TestAPI</include>
                        <include>**/*IntegrationTest</include>
                    </includes>
                </configuration>
                <executions>
                    <execution>
                        <id>default-integration-test</id>
                        <goals>
                            <goal>integration-test</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            ...
        </plugins>
    </build>

Jacoco code coverage

Jacoco is a plugin that will:

Parse SUREFIRE and FAILSAFE results
Extract and generate code coverage reports (XML format)
Convert the XML into binaries (*.exec) for SonarQube

Jacoco will generate the following binaries reports:

1 unit test report per project, for every module: $parent/$module/target/jacoco.exec
1 centralized integration test report: $parent/target/jacoco-it.exec

    <properties>
        <!-- Project configuration -->
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <!-- Sonar -->
        <!-- Tell sonar where to look for the binaries coverage files. Property inherited by submodules -->
        <sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
    </properties>


    <build>
        <plugins>
            ... 

            <!-- JACOCO test coverage plugin.
                 Use it to compile SUREFIRE (unit tests) and FAILSAFE (integration tests) reports for SonarQube
                 (i) attach that plugin to Maven TEST phase
                 Reports are generated in "${project.build.directory}/site/jacoco/*" by default
                 Good documentations:
                    https://wiki.onap.org/display/DW/Implementing+Code+Coverage
                    https://www.devcon5.ch/en/blog/2015/05/29/multi-module-integration-test-coverage-sonar-jacoco/
                    https://www.eclemma.org/jacoco/trunk/doc/maven.html
                 -->
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.3</version>
                <configuration>
                    <append>true</append>
                </configuration>
                <executions>
                    <!-- ## UNIT TESTS ## -->
                    <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven SUREFIRE plugin is executed. -->
                    <execution>
                        <id>pre-unit-tests</id>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                    </execution>
                    <!-- Create reports -->
                    <execution>
                        <id>report-unit-tests</id>
                        <goals>
                            <goal>report</goal>
                        </goals>
                    </execution>
                    <!-- ## INTEGRATION TESTS ## -->
                    <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven FAILSAFE plugin is executed. -->
                    <execution>
                        <id>pre-integration-tests</id>
                        <goals>
                            <goal>prepare-agent-integration</goal>
                        </goals>
                        <configuration>
                            <!-- Only 1 destination file to aggregate ALL integration tests reports -->
                            <!-- the "session.executionRootDirectory" = parent folder that is being build by Jenkins -->
                            <destFile>${session.executionRootDirectory}/target/jacoco-it.exec</destFile>
                            <append>true</append>
                        </configuration>
                    </execution>
                    <!-- Create reports -->
                    <execution>
                        <id>report-integration-tests</id>
                        <goals>
                            <goal>report-integration</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>

SonarQube plugin

SonarQube released a maven plugin to work with SONAR. This plugin already includes most of the configuration and default behavior is correct.

Sonar will read Jacoco "exec" binaries files
Sonar will process the Surefire & Failsafe reports

    <properties>
        <!-- Sonar -->
        <!-- Tell sonar where to look for the binaries coverage files. Property inherited by submodules -->
        <sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
    </properties>


    <build>
        <plugins>
            ..

            <!-- SonarQube engine -->
            <plugin>
                <groupId>org.sonarsource.scanner.maven</groupId>
                <artifactId>sonar-maven-plugin</artifactId>
                <version>3.6.0.1398</version>
            </plugin>
        </plugins>
    </build>

Maven complete POM example

This is how the parent POM should look like to use SonarQube with correct coverage:

    <properties>
        <!-- Project configuration -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>

        <!-- Sonar -->
        <!-- Tell sonar where to look for the binaries coverage files. Property inherited by submodules -->
        <sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
    </properties>


    <build>
        <plugins>
            <!-- Build settings -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.3</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>

            <!-- Always generate the source, it is better for debugging -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>3.0.1</version>
                <executions>
                    <execution>
                        <id>attach-sources</id>
                        <goals>
                            <goal>jar-no-fork</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>attach-test-sources</id>
                        <goals>
                            <goal>test-jar-no-fork</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- To run UNIT tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <excludes>
                        <exclude>**/*TestAPI</exclude>
                        <exclude>**/*IntegrationTest</exclude>
                    </excludes>
                </configuration>
            </plugin>

            <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <includes>
                        <include>**/*TestAPI</include>
                        <include>**/*IntegrationTest</include>
                    </includes>
                </configuration>
                <executions>
                    <execution>
                        <id>default-integration-test</id>
                        <goals>
                            <goal>integration-test</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- JACOCO test coverage plugin.
                 Use it to compile SUREFIRE (unit tests) and FAILSAFE (integration tests) reports for SonarQube
                 (i) attach that plugin to Maven TEST phase
                 Reports are generated in "${project.build.directory}/site/jacoco/*" by default
                 Good documentations:
                    https://wiki.onap.org/display/DW/Implementing+Code+Coverage
                    https://www.devcon5.ch/en/blog/2015/05/29/multi-module-integration-test-coverage-sonar-jacoco/
                    https://www.eclemma.org/jacoco/trunk/doc/maven.html
                 -->
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.3</version>
                <configuration>
                    <append>true</append>
                </configuration>
                <executions>
                    <!-- ## UNIT TESTS ## -->
                    <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven SUREFIRE plugin is executed. -->
                    <execution>
                        <id>pre-unit-tests</id>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                    </execution>
                    <!-- Create reports -->
                    <execution>
                        <id>report-unit-tests</id>
                        <goals>
                            <goal>report</goal>
                        </goals>
                    </execution>
                    <!-- ## INTEGRATION TESTS ## -->
                    <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven FAILSAFE plugin is executed. -->
                    <execution>
                        <id>pre-integration-tests</id>
                        <goals>
                            <goal>prepare-agent-integration</goal>
                        </goals>
                        <configuration>
                            <!-- Only 1 destination file to aggregate ALL integration tests reports -->
                            <!-- the "session.executionRootDirectory" = parent folder that is being build by Jenkins -->
                            <destFile>${session.executionRootDirectory}/target/jacoco-it.exec</destFile>
                            <append>true</append>
                        </configuration>
                    </execution>
                    <!-- Create reports -->
                    <execution>
                        <id>report-integration-tests</id>
                        <goals>
                            <goal>report-integration</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- SonarQube engine -->
            <plugin>
                <groupId>org.sonarsource.scanner.maven</groupId>
                <artifactId>sonar-maven-plugin</artifactId>
                <version>3.6.0.1398</version>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        ...
  
        <!-- jUnit -->
        <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>
             <version>${junit.version}</version>
             <scope>test</scope>
        </dependency>      
    </dependencies>

Jenkins

Jenkins plugins

To work with SonarQube, Jenkins requires the following plugins to be installed and enabled:

Maven Integration plugin
SonarQube scanner for Jenkins. See SonarQube documentation

Actions:

Log into Jenkins as an administrator
Go to Manage Jenkin > Manage plugins

Global configuration

Log into Jenkins as an administrator
Go to Manage Jenkins > Configure System
Scroll to the SonarQube servers section
Check Enable injection of SonarQube server configuration as build environment variables

Job configuration

Now that you can generate these reports, especially the integration tests results, you need to configure Jenkins to use them!

Maven build (legacy)

This explain how to configure a Maven build (legacy) with SonarQube Runner.

In the Build Environment section > check Prepare SonarQube Scanner environment

Then, set the Build command line:

Adjust the SonarQube server URL + path to integration tests reports!

Maven build: clean install or clean deploy
Use SonarQube
- Enable plugin: $SONAR_MAVEN_GOAL
- Set SonarQube server URL: -Dsonar.host.url=http://localhost:9000/sonarqube
- Give path of Integration tests: -Dsonar.jacoco.itReportPath=/var/lib/jenkins/workspace/Flow.e.r.s/target/jacoco-it.exec

The configuration above will result in the following command line during execution:

Pipelines

TODO

See SonarQube documentation

SonarQube

By default there will be 1 sonarqube' entry for every Jenkins artifactId that is build (parent name).

How to see the code coverage by module?

To have a quick glance of the project's modules:

SonarQube > project > Code

SonarQube > project > Measures' > Expand coverage > Conditions coverage

How to see measures by module?

By default SonarQube display measures for all the project's modules. This is sometimes not convenient... The old "code" dashboard has been removed, but you can still see the measure by sub-module with a little trick.

Select a module to analyze from the "code" menu

Notice the SonarQube URL and the selection parameter

Copy that &selected=.. parameter

Parse the value on the measures page

Other

jUnit tests ordering

Since jUnit 4.11 you can set the order of your tests. If you have to use that it means something is probably wrong with your tests design!

Set order to the Class level: https://github.com/junit-team/junit/wiki/Test-execution-order
Set custom order: http://memorynotfound.com/run-junit-tests-method-order/

References

This article is based on my daily work in VEHCO + European Parliament + LuxTrust.

This article is based on a lot of research and code browsing. Following articles are very good:

Other article that are relevant for older versions of Sonar (versions < 6.x with Java 7)

Anonymous

Search

Navigation

Navigation

Categories

Wiki tools

Wiki tools

Sonar + maven configuration + Jenkins

Namespaces

Page actions

Contents

Requirements

Principle

Maven plugins

Surefire UNIT Tests

Failsafe INTEGRATION tests

Jacoco code coverage

SonarQube plugin

Maven complete POM example

Jenkins

Jenkins plugins

Global configuration

Job configuration

Maven build (legacy)

Pipelines

SonarQube

How to see the code coverage by module?

How to see measures by module?

Other

jUnit tests ordering

References

Anonymous

Search

Navigation

Wiki tools

Page tools

Categories

Sonar + maven configuration + Jenkins

Contents

Requirements

Principle

Maven plugins

Surefire UNIT Tests

Failsafe INTEGRATION tests

Jacoco code coverage

SonarQube plugin

Maven complete POM example

Jenkins

Jenkins plugins

Global configuration

Job configuration

Maven build (legacy)

Pipelines

SonarQube

How to see the code coverage by module?

How to see measures by module?

Other

jUnit tests ordering

References