Integrate Java Code Coverage with Integration Tests

The importance of software quality is growing steadily and significantly as the role of software increases. Therefore, code coverage is used as an important indicator of software quality. It helps in evaluating the effectiveness of testing by providing data on different coverage items. This metric shows which lines of the source code have been tested and which have not. This insight allows the improvement of testing by defining new test cases for the untested code, thereafter, improving our code quality, which ultimately increases code coverage.

This blog post describes how to generate code coverage reports for integration tests using the JaCoCo library. In this blog post, I am going to explain the steps to generate code coverage reports for integration tests that run against the WSO2 product which is deployed on a remote server.

What is JaCoCo?

Java Code Coverage Library is also known as JaCoCo is an open-source library for measuring code coverage in a codebase. It measures the coverage of instructions, lines, and branches based on the code covered by running test cases and provides a visual report with the highlighted lines of code and the total percentage of code executed in each method. It can easily integrate with the most important frameworks like Gradle, Maven, IntelliJ IDEA, Eclipse, and so on.

How to Generate Code Coverage Reports using JaCoCo?

Now let’s discuss the steps involved in generating code coverage reports from integration tests. Here I have used the JaCoCo Code Coverage Library and JaCoCo Ant Task to collect code coverage stats and generate reports.

In this scenario, the application under test and the test scripts are located on separate servers/locations. Therefore, in order to generate code coverage reports from integration tests we have to follow the below steps.

1. Attach JaCoCo Agent to JVM and record the code coverage.

To attach the JaCoCo agent to the JVM, first, we need to download and place the “jacocoagent.jar” in the lib folder of the product packs which is configured in the remote server. You can download the jacocoagent.jar which corresponds to the version of your Jacoco plugin in Maven from the link.

Then attach the JaCoCo agent as a JVM option and do the byte code instrumentation on the server-side when server startup. “jacoco.exec” will be generated automatically along with the instrumented class dumps. For byte code instrumentation we defined the packages which need to be included and which need to be excluded separately. Whenever a class is loaded, the JaCoCo agent can instrument the class so it can identify which class and the code lines are called. Then builds up the coverage statistics. Code coverage information will be written to the jacoco.exec on JVM exit. To attach the java agent; you can add the following jacocoagent arguments to the start script of your application server. In my case, we have to add this to the wso2server.sh file.

-javaagent:<<path_to_lib_folder>>/lib/jacocoagent.jar=destfile=<<path_to_output_folder>>/jacoco/jacoco.exec,append=true,classdumpdir=<<path_to_output_folder>>/jacoco/classes,includes=<<list of class names that should be included in execution analysis>> \

• destfile: Path to the output file (jacoco.exec) for execution data. This should be any location you wish to store the jacoco.exec file.
• append: If the option is set to true and the execution data file already exists; coverage data is appended to the existing file. If the option is set to false, an existing execution data file will be replaced.
• includes: A list of class names that should be included in execution analysis. The list entries are separated by a colon (:) and may use wildcard characters (* and ?).

As an example:

includes=org.wso2.carbon.user*:org.wso2.carbon.identity.application.authenticator.basicauth*

• classdumpdir: Path to the output directory where all class files seen by the agent are dumped to. This path should be defined relative to the working directory.

There are many more options that we can pass with the JaCoCo agent. You can refer to the link to understand those JVM options.

2. Copy the Coverage Outputs to the Test Server.

After attaching the JaCoCo Agent to JVM Options you can start the server and execute the tests. As mentioned above; coverage information will be written to jacoco.exec on JVM termination. Since our application server is in a remote location, we need to copy the jacoco.exec and the class-dump directory to the location where the tests exist. Therefore during the second step, we have to stop the application server and download the coverage dumps to the test server.

The following command can be used to copy the code coverage outputs from the remote application server.

scp -i "${KEY_PATH}" 
-o UserKnownHostsFile=/dev/null 
-o StrictHostKeyChecking=no "ubuntu@${SERVER_IP}:"${PATH_TO_OUTPUT_DIR}/jacoco/jacoco.zip ${PATH_TO_TEST_DIR}

In order to reduce the manual involvement in the test execution process, we can automate these steps using shell scripts. We can easily run the shell script using the exec-maven-plugin as a post-integration test step.

Assume we have a shell script called “copy-coverage-data.sh” that resides in ${project.basedir}/resources directory.

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${exec.maven.plugin}</version>
<executions>
<execution>
<id>my-exec</id>
<phase>post-integration-test</phase>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>${project.basedir}/resources/copy-coverage-data.sh</executable>
</configuration>
</plugin>

3. Add Maven Plugin to the POM file.

Add maven-antrun-plugin to the pom.xml and download the jacocoant.jar. JaCoCo comes with Ant tasks to launch Java programs with execution recording and for creating coverage reports from the recorded data. All tasks are defined in jacocoant.jar. “jacoco:report” task can be used to create reports in different formats like “.html”, “.csv” and “.xml”. The “jacoco.exec” file and the instrumented class files are taken as inputs to the report task when generating code coverage reports. Here I have created a separate maven module for code coverage and added the following maven plugins for pom.xml.

<profiles>
<profile>
<id>with-tests</id>
<activation>
<property>
<name>!maven.test.skip</name>
</property>
</activation>
<build>
<plugins>
<!-- Download jacocoant.jar -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-jacoco-dependencies</id>
<phase>compile</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.basedir}/target</outputDirectory>
<includeTypes>jar</includeTypes>
<includeArtifactIds>org.jacoco.ant</includeArtifactIds>
<stripVersion>true</stripVersion>
</configuration>
</execution>
</executions>
</plugin>

<!-- Execute Shell Script - Stop the application server and copy the coverage data -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${exec.maven.plugin}</version>
<executions>
<execution>
<id>my-exec</id>
<phase>post-integration-test</phase>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>${project.basedir}/resources/copy-coverage-data.sh</executable>
</configuration>
</plugin>
<!-- Ant plugin - Generate Jacoco Report -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>${antrun-plugin.version}</version>
<executions>
<execution>
<id>code-coverage-reports</id>
<phase>post-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target xmlns:jacoco="antlib:org.jacoco.ant">
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml">
<classpath path="${project.basedir}/target"/>
</taskdef>
<jacoco:report>
<executiondata>
<fileset dir="${project.basedir}">
<include name="jacoco-stats/jacoco.exec"/>
</fileset>
</executiondata>
<structure name="Integration Coverage Report">
<classfiles>
<fileset dir="${project.basedir}">
<include name="jacoco-stats/classes/**"/>
</fileset>
</classfiles>
</structure>
<html destdir="${project.basedir}/target/code-coverage-reports/jacoco"/>
</jacoco:report>
</target>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.jacoco</groupId>
<artifactId>org.jacoco.ant</artifactId>
<version>${jacoco.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</profile>
</profiles>

Now you can run the integration tests against the application deployed in a remote server.

mvn clean install -fae

The code coverage reports will be generated at “/target/code-coverage-reports/jacoco”.

Overall Code Coverage

Class wise Code Coverage

I hope you will find this blog post useful. Thank you for reading :)