Difference between revisions of "Sonar + maven configuration + Jenkins"
Line 89: | Line 89: | ||
− | == | + | ==Surefire Unit Tests== |
Results are saved '''in each Maven module''': | Results are saved '''in each Maven module''': | ||
Line 97: | Line 97: | ||
− | == | + | ==Failsafe Integration tests== |
Only XML and HTML results are saved in each Maven module. '''Consolidate results are saved in the execution path ./target''' | Only XML and HTML results are saved in each Maven module. '''Consolidate results are saved in the execution path ./target''' | ||
Line 113: | Line 113: | ||
<properties> | <properties> | ||
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> | <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> | ||
− | + | <junit.version>4.12</junit.version> | |
+ | |||
<!-- ================================================================= --> | <!-- ================================================================= --> | ||
<!-- ==== FAILSAFE (IT tests) + SUREFIRE (Unit tests) + SONARQUBE ==== --> | <!-- ==== FAILSAFE (IT tests) + SUREFIRE (Unit tests) + SONARQUBE ==== --> | ||
Line 119: | Line 120: | ||
<maven-surefire-plugin.version>2.18.1</maven-surefire-plugin.version> | <maven-surefire-plugin.version>2.18.1</maven-surefire-plugin.version> | ||
<maven-failsafe-plugin.version>2.18.1</maven-failsafe-plugin.version> | <maven-failsafe-plugin.version>2.18.1</maven-failsafe-plugin.version> | ||
− | <maven.jacoco.version>0.7. | + | <maven.jacoco.version>0.7.7.201606060606</maven.jacoco.version> |
<!-- ========================================== --> | <!-- ========================================== --> | ||
Line 153: | Line 154: | ||
− | =Jacoco plugin (coverage agent)= | + | ='''Jacoco plugin''' (coverage agent)= |
This is how you should configure this plugin: | This is how you should configure this plugin: | ||
Line 242: | Line 243: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
+ | This configuration force the generation of both XML and Binary reports. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | ='''Surefire (Unit tests)'''= | |
− | |||
− | |||
Surefire will process the Unit Tests. | Surefire will process the Unit Tests. | ||
!! Do not use Surefire for your integration tests because Surefire produce different outputs per maven module !! | !! Do not use Surefire for your integration tests because Surefire produce different outputs per maven module !! | ||
− | + | ||
<syntaxhighlight lang="xml"> | <syntaxhighlight lang="xml"> | ||
Line 271: | Line 260: | ||
<plugin> Maven-jacoco </plugin> | <plugin> Maven-jacoco </plugin> | ||
− | + | <!-- == process UNIT tests == --> | |
− | + | <plugin> | |
− | + | <groupId>org.apache.maven.plugins</groupId> | |
− | + | <artifactId>maven-surefire-plugin</artifactId> | |
− | + | <version>${maven-surefire-plugin.version}</version> | |
− | + | <configuration> | |
− | + | <!-- Sets the VM argument line used when unit tests are run. --> | |
− | + | <argLine>-Xms256m -Xmx1G -XX:PermSize=256m -XX:MaxPermSize=512m ${surefireArgLine}</argLine> | |
− | + | <!-- Skip unit tests if asked --> | |
− | + | <skipTests>${skipTests}</skipTests> | |
− | + | <!-- Excludes integration tests when unit tests are run. --> | |
− | + | <excludes> | |
− | + | <exclude>**/*IT.java</exclude> | |
− | + | <exclude>**/*IntegrationTest.java</exclude> | |
− | + | <exclude>**/*GWT*.java</exclude> | |
− | + | </excludes> | |
− | + | <!-- Save reports in particular directory --> | |
− | + | <reportsDirectory>${sonar.surefire.reportsPath}</reportsDirectory> | |
− | + | </configuration> | |
− | + | </plugin> | |
<plugin> Failsafe (IT) </plugin> | <plugin> Failsafe (IT) </plugin> | ||
Line 301: | Line 290: | ||
Notes: | Notes: | ||
* No tests will be run if <code>mvn clean install -DskipTests</code> | * No tests will be run if <code>mvn clean install -DskipTests</code> | ||
− | |||
* The exclusion list is something that depends on your own situation | * The exclusion list is something that depends on your own situation | ||
− | = | + | ='''Failsafe (Integration tests)'''= |
Failsafe will process the Integration Tests. | Failsafe will process the Integration Tests. | ||
− | |||
− | |||
<syntaxhighlight lang="xml"> | <syntaxhighlight lang="xml"> | ||
Line 320: | Line 306: | ||
<plugin> Surefire (UT) </plugin> | <plugin> Surefire (UT) </plugin> | ||
− | + | <!-- == process INTEGRATION tests == --> | |
− | + | <plugin> | |
− | + | <groupId>org.apache.maven.plugins</groupId> | |
− | + | <artifactId>maven-failsafe-plugin</artifactId> | |
− | + | <version>${maven-failsafe-plugin.version}</version> | |
− | + | <configuration> | |
− | + | <!-- Sets the VM argument line used when integration tests are run. --> | |
− | + | <argLine>-Xms256m -Xmx1G -XX:PermSize=256m -XX:MaxPermSize=512m ${failsafeArgLine}</argLine> | |
− | + | <!-- Skip integration tests if asked --> | |
− | + | <skipTests>${skipIntegrationTests}</skipTests> | |
− | + | <includes> | |
− | + | <include>**/*IntegrationTest.java</include> | |
− | + | <include>**/*IT.java</include> | |
− | + | </includes> | |
− | + | <!-- Excludes GWT crap --> | |
− | + | <excludes> | |
− | + | <exclude>**/*GWT*.java</exclude> | |
− | + | </excludes> | |
− | + | <!-- Save reports in particular directory --> | |
− | + | <reportsDirectory>${sonar.failsafe.reportsPath}</reportsDirectory> | |
− | + | </configuration> | |
− | + | <!-- Only run failsafe when required --> | |
− | + | <executions> | |
− | + | <execution> | |
− | + | <id>integration-tests</id> | |
− | + | <goals> | |
− | + | <goal>integration-test</goal> | |
− | + | <goal>verify</goal> | |
− | + | </goals> | |
− | + | </execution> | |
− | + | </executions> | |
− | + | </plugin> | |
− | |||
− | |||
− | |||
− | |||
− | |||
</plugins> | </plugins> | ||
Line 365: | Line 346: | ||
Notes: | Notes: | ||
* No tests will be run if <code>mvn clean install -DskipIntegrationTests</code> | * No tests will be run if <code>mvn clean install -DskipIntegrationTests</code> | ||
− | |||
* The exclusion | inclusion list is something that depends on your own situation | * The exclusion | inclusion list is something that depends on your own situation | ||
− | |||
− | + | =Maven '''dependencies'''= | |
At last, you need to add the jacoco dependency: | At last, you need to add the jacoco dependency: | ||
+ | (i) 2016-07-19 # this step is NOT required anymore on latest Jenkins & Sonarqube | ||
<syntaxhighlight lang="xml"> | <syntaxhighlight lang="xml"> | ||
<dependencies> | <dependencies> | ||
− | . | + | <!-- jUnit --> |
+ | <dependency> | ||
+ | <groupId>junit</groupId> | ||
+ | <artifactId>junit</artifactId> | ||
+ | <version>${junit.version}</version> | ||
+ | <scope>test</scope> | ||
+ | </dependency> | ||
+ | |||
<!-- SonarQube dependencies --> | <!-- SonarQube dependencies --> | ||
<dependency> | <dependency> | ||
Line 390: | Line 377: | ||
− | + | =Maven '''profile'''= | |
+ | |||
+ | Put all the following elements inside a Maven profile: | ||
+ | |||
+ | <syntaxhighlight lang="xml"> | ||
+ | <profile> | ||
+ | <id>sonar</id> | ||
+ | <activation> | ||
+ | <activeByDefault>false</activeByDefault> | ||
+ | </activation> | ||
+ | |||
+ | <properties> ... </properties> | ||
+ | <build> <plugins> ... </plugins> </build> | ||
+ | <dependencies> ... </dependencies> | ||
+ | </profile> | ||
+ | </profiles> | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | To use it: | ||
+ | * Standard use = <code>mvn clean install -P sonar</code> | ||
+ | * Skip Unit tests = <code>mvn clean install -DskipTests -P sonar</code> | ||
+ | * Skip Integration tests = <code>mvn clean install -DskipIntegrationTests -P sonar</code> | ||
− | =Jenkins configuration= | + | ='''Jenkins''' configuration= |
Now that you can generate these reports, especially the integration results, you need to configure Jenkins to use them! | Now that you can generate these reports, especially the integration results, you need to configure Jenkins to use them! | ||
Line 403: | Line 411: | ||
* <code>-Dsonar.dynamicAnalysis=reuseReports</code> ==> Tells SonarQube to reuse existing reports for unit tests execution and coverage reports | * <code>-Dsonar.dynamicAnalysis=reuseReports</code> ==> Tells SonarQube to reuse existing reports for unit tests execution and coverage reports | ||
− | * <code>-Dsonar.jacoco.itReportPath={ABSOLUTE_PATH_TO_JENKINS}/workspace/target/reports/jacoco-it.exec</code> => integration tests report to process | + | * <code>-Dsonar.jacoco.itReportPath=${ABSOLUTE_PATH_TO_JENKINS}/workspace/${JENKINS_JOB}/target/reports/jacoco-it.exec</code> => integration tests report to process |
− | |||
Revision as of 20:22, 19 July 2016
SonarQube requires quite some configuration to be fully useful!
This article explains how to configure your maven projects to produce reports + how to configure Jenkins with SonarQube, bringing both Unit and Integration tests coverage.
Contents
Principle
What are Unit / Integration tests ?
By default "code coverage" only consider the tests from the same projects.
Let's take an example to make things easier...
Consider a project with 4 modules such as:
Then consider the following inter-dependencies:
Definition:
- Unit Tests coverage (UT) = code coverage for each module only. == How much tests of 'module A' test 'module A' ??
- Integration Tests coverage (IT) = overall code coverage. == Once global compilation and all the tests have been run, how much of 'module A' was used and covered ?
... In our example:
- For "Module A" only the jUnit in "Module A" are computed in unit test coverage
- The coverage provided by the Module C "tests" is NOT taken into account by default! You need to enable and configure the integration test coverage.
... Why ?!?
This limitation comes from the default SONAR behaviour. By default it creates 1 coverage report per component (maven project). To get the overall coverage you need a common report that each project can improve and use.
Process overview
To work well SonarQube requires a bit of Maven configuration + Jenkins build adjustment:
- Maven specific properties
- Maven build plug-ins
- Maven-jacoco (code coverage tool)
- Maven-surefire (unit tests reports)
- Maven-failsafe (integration tests reports)
- Maven jacoco dependency
Key points:
- SONAR is configured for JAVA language and it will use Jacoco as coverage tool.
- Each Maven module will have its own Unit Tests results (Surefire reports + .exec file) inside its own
target
directory - The Integration Tests results are common. Meaning, all modules will write in the same directory. That directory is relative to the maven execution.
Requirements
- Maven must run on Java 7 or +
- You must use Sonarqube 5.3 or +
Maven / Sonar properties and key points
Maven key properties
Here are some properties you must know and use to configure Sonar:
${project.reporting.outputDirectory}
== module/target/site/${project.build.directory}
== module/target/${session.executionRootDirectory}
== root folder wheremvn clean install
is launched<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
Sonar properties
First of all you need to configure SONAR by setting up some properties and paths, see Surefire and Failsafe below.
To optimize SONAR execution we will set the following behaviour:
- Language == Java
- Coverage tool == JaCoCo
- Behaviour == re-use Surefire and Failsafe reports. Do NOT re-compile for Sonar.
Surefire Unit Tests
Results are saved in each Maven module:
- Results are saved in
XML
format in$module/target/site/surefire-reports
- Results are saved in
HTML
format in$module/target/site/jacoco-ut
. You can openindex.html
to browse the results. - Results are saved in
Binary (jacoco format)
in$module/target/jacoco.exec
Failsafe Integration tests
Only XML and HTML results are saved in each Maven module. Consolidate results are saved in the execution path ./target
- Results are saved in
XML
format in$module/target/site/failsafe-reports
- Results are saved in
HTML
format in$module/target/site/jacoco-it
. You can openindex.html
to browse the results. - Results are saved in
Binary (jacoco format)
in$maven_execution_place/target/jacoco-it.exec
Set properties
POM.xml extract:
<properties>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<junit.version>4.12</junit.version>
<!-- ================================================================= -->
<!-- ==== FAILSAFE (IT tests) + SUREFIRE (Unit tests) + SONARQUBE ==== -->
<!-- ================================================================= -->
<maven-surefire-plugin.version>2.18.1</maven-surefire-plugin.version>
<maven-failsafe-plugin.version>2.18.1</maven-failsafe-plugin.version>
<maven.jacoco.version>0.7.7.201606060606</maven.jacoco.version>
<!-- ========================================== -->
<!-- Global Sonar settings. Do not change them! -->
<!-- ========================================== -->
<sonar.language>java</sonar.language>
<sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
<sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
<jacoco.lib.path>
${settings.localRepository}/org/jacoco/org.jacoco.agent/${maven.jacoco.version}/org.jacoco.agent-${maven.jacoco.version}-runtime.jar
</jacoco.lib.path>
<javaagent>${jacoco.lib.path}</javaagent>
<!-- Don't let Sonar execute tests! Maven will run them and produce some reports for Sonar -->
<sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
<!-- =============== -->
<!-- Quality reports -->
<!-- =============== -->
<!-- XML reports (go to $/target/site/*-reports) -->
<sonar.surefire.reportsPath>${project.reporting.outputDirectory}/surefire-reports</sonar.surefire.reportsPath>
<sonar.failsafe.reportsPath>${project.reporting.outputDirectory}/failsafe-reports</sonar.failsafe.reportsPath>
<!-- Web reports (= go to $/target/site/jacoco-*/index.html to browse results) -->
<jacoco.ut.summary>${project.reporting.outputDirectory}/jacoco-ut</jacoco.ut.summary>
<jacoco.it.summary>${project.reporting.outputDirectory}/jacoco-it</jacoco.it.summary>
<!-- Binary format [= Sonar format] -->
<!-- 1) Unit tests reports must be in each module $/target/ project -->
<jacoco.ut.execution.data.file>${project.build.directory}/jacoco.exec</jacoco.ut.execution.data.file>
<!-- 2) Integration tests reports must be consolidated together. If not then you will probably lost multi-project coverage -->
<jacoco.it.execution.data.file>${session.executionRootDirectory}/target/jacoco-it.exec</jacoco.it.execution.data.file>
</properties>
Jacoco plugin (coverage agent)
This is how you should configure this plugin:
<build>
<plugins>
<!-- #################### -->
<!-- SONAR # CODE METRICS -->
<!-- #################### -->
<!-- This will auto-generate the right jacoco command for Unit | Integration tests -->
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${maven.jacoco.version}</version>
<executions>
<!-- #- - - - - - - - - - - - # -->
<!-- Unit tests configuration -->
<!-- #- - - - - - - - - - - - # -->
<!-- JaCoCo runtime agent which is passed as VM argument when Maven Surefire plugin is executed. -->
<execution>
<id>pre-unit-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<!-- Surefire (Unit tests) reports will be saved as XML in... -->
<destFile>${jacoco.ut.execution.data.file}</destFile>
<propertyName>surefireArgLine</propertyName>
<append>true</append>
</configuration>
</execution>
<!-- Ensures code coverage report for unit tests is created after unit tests have been run. -->
<execution>
<id>post-unit-test</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<!-- Surefire (Unit tests) reports will be saved as Binary in... -->
<dataFile>${jacoco.ut.execution.data.file}</dataFile>
<outputDirectory>${jacoco.ut.summary}</outputDirectory>
<append>true</append>
</configuration>
</execution>
<!-- #- - - - - - - - - - - - # -->
<!-- Integration tests configuration -->
<!-- #- - - - - - - - - - - - # -->
<!-- JaCoCo runtime agent which is passed as VM argument when Maven Failsafe plugin is executed. -->
<execution>
<id>pre-integration-test</id>
<phase>pre-integration-test</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<!-- Failsafe (Integration tests) reports will be saved as XML in... -->
<destFile>${jacoco.it.execution.data.file}</destFile>
<propertyName>failsafeArgLine</propertyName>
<append>true</append>
</configuration>
</execution>
<!-- Ensures code coverage report for integration tests is created after integration tests have been run. -->
<execution>
<id>post-integration-test</id>
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<!-- Failsafe (Integration tests) reports will be saved as Binary in... -->
<dataFile>${jacoco.it.execution.data.file}</dataFile>
<outputDirectory>${jacoco.it.summary}</outputDirectory>
<append>true</append>
</configuration>
</execution>
</executions>
</plugin>
<plugin> Surefire (UT) </plugin>
<plugin> Failsafe (IT) </plugin>
</plugins>
</build>
This configuration force the generation of both XML and Binary reports.
Surefire (Unit tests)
Surefire will process the Unit Tests.
!! Do not use Surefire for your integration tests because Surefire produce different outputs per maven module !!
<build>
<plugins>
<plugin> Maven-jacoco </plugin>
<!-- == process UNIT tests == -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<!-- Sets the VM argument line used when unit tests are run. -->
<argLine>-Xms256m -Xmx1G -XX:PermSize=256m -XX:MaxPermSize=512m ${surefireArgLine}</argLine>
<!-- Skip unit tests if asked -->
<skipTests>${skipTests}</skipTests>
<!-- Excludes integration tests when unit tests are run. -->
<excludes>
<exclude>**/*IT.java</exclude>
<exclude>**/*IntegrationTest.java</exclude>
<exclude>**/*GWT*.java</exclude>
</excludes>
<!-- Save reports in particular directory -->
<reportsDirectory>${sonar.surefire.reportsPath}</reportsDirectory>
</configuration>
</plugin>
<plugin> Failsafe (IT) </plugin>
</plugins>
</build>
Notes:
- No tests will be run if
mvn clean install -DskipTests
- The exclusion list is something that depends on your own situation
Failsafe (Integration tests)
Failsafe will process the Integration Tests.
<build>
<plugins>
<plugin> Maven-jacoco </plugin>
<plugin> Surefire (UT) </plugin>
<!-- == process INTEGRATION tests == -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-failsafe-plugin.version}</version>
<configuration>
<!-- Sets the VM argument line used when integration tests are run. -->
<argLine>-Xms256m -Xmx1G -XX:PermSize=256m -XX:MaxPermSize=512m ${failsafeArgLine}</argLine>
<!-- Skip integration tests if asked -->
<skipTests>${skipIntegrationTests}</skipTests>
<includes>
<include>**/*IntegrationTest.java</include>
<include>**/*IT.java</include>
</includes>
<!-- Excludes GWT crap -->
<excludes>
<exclude>**/*GWT*.java</exclude>
</excludes>
<!-- Save reports in particular directory -->
<reportsDirectory>${sonar.failsafe.reportsPath}</reportsDirectory>
</configuration>
<!-- Only run failsafe when required -->
<executions>
<execution>
<id>integration-tests</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
Notes:
- No tests will be run if
mvn clean install -DskipIntegrationTests
- The exclusion | inclusion list is something that depends on your own situation
Maven dependencies
At last, you need to add the jacoco dependency:
(i) 2016-07-19 # this step is NOT required anymore on latest Jenkins & Sonarqube
<dependencies>
<!-- jUnit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<!-- SonarQube dependencies -->
<dependency>
<groupId>org.jacoco</groupId>
<artifactId>org.jacoco.agent</artifactId>
<version>${maven.jacoco.version}</version>
<classifier>runtime</classifier>
<scope>test</scope>
</dependency>
</dependencies>
Maven profile
Put all the following elements inside a Maven profile:
<profile>
<id>sonar</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties> ... </properties>
<build> <plugins> ... </plugins> </build>
<dependencies> ... </dependencies>
</profile>
</profiles>
To use it:
- Standard use =
mvn clean install -P sonar
- Skip Unit tests =
mvn clean install -DskipTests -P sonar
- Skip Integration tests =
mvn clean install -DskipIntegrationTests -P sonar
Jenkins configuration
Now that you can generate these reports, especially the integration results, you need to configure Jenkins to use them!
To set up the Sonar Jenkins plugin to process already generated JaCoCo data files, the following properties must be specified in the "Additional properties" text box, under the Sonar installation being configured:
-Dsonar.dynamicAnalysis=reuseReports
==> Tells SonarQube to reuse existing reports for unit tests execution and coverage reports-Dsonar.jacoco.itReportPath=${ABSOLUTE_PATH_TO_JENKINS}/workspace/${JENKINS_JOB}/target/reports/jacoco-it.exec
=> integration tests report to process
Real life example
You can check out one of my GitHub project:
Even better, you can check out this excellent tutorial:
Appendices
Run tests in specific order
If for some reasons you need to run your tests in a particular order you can say do so in Maven and/or Java
Reminder : if you have to run your tests in a particular order then you should refactor your tests! These tricks should not let you escape from that!
Maven
The runOrder
is a nice trick.
<build>
<plugins>
<plugin>
SUREFIRE or FAILSAFE
<configuration>
<!-- JVM settings -->
<argLine>-Xmx1024m -XX:maxPermSize:256m</argLine>
<forkCount>1</forkCount>
<reuseForks>true</reuseForks>
<runOrder>alphabetical</runOrder>
<!-- Jacoco execution -->
<argLine>${jacoco.agent.it.arg}</argLine>
...
</configuration>
</plugin>
</plugins>
</build>
Java
Since jUnit 4.11 you can set the order of your tests.
- Set order to the Class level: https://github.com/junit-team/junit/wiki/Test-execution-order
- Set custom order: http://memorynotfound.com/run-junit-tests-method-order/
References
This article is based on my daily work in VEHCO.
To update to SonarQube I used the following articles and code examples:
- http://www.aheritier.net/maven-failsafe-sonar-and-jacoco-are-in-a-boat/
- VERY good GitHub example, provided by the same author: https://github.com/dgageot/coverage
- http://stackoverflow.com/questions/26253277/jacoco-agent-for-integration-tests-on-remote-machine
Jenkins configuration: