under engineered

Getting Code Coverage for e2e tests run on a Java codebase

I recently was asked to help on a project where the team was trying to get the test coverage for e2e tests also known as component tests. I initially tried to say that component tests run on the actual application process and not in the context of a test/source file. So the app is a black box to the running tests and can't be instrumented for getting coverage.

I was wrong!

Running a java app: basics #

The way a Java app runs is quite similar to a NodeJS app. A bundle in Java world is called a jar. You can run an app by just running the jar file which contains built classes and the app dependencies all together.

java -jar <application-code>.jar

That's it, the app server would start and you can run your component/e2e tests using any framework of your liking.

For simplicity I'd choose running manual curls and seeing if the api responds correctly or not. Aim here is to not solve specifically for a test framework but seeing if the app can be instrumented at runtime which would then enable us to get coverage for any framework.

JaCoCo #

This is the library which is used by all Java programmers to get coverage for their unit tests. It stands for Java Code Coverage. (I could've never guessed it in my dreams had I not googled it). Usually people use it to generate the coverage for Unit tests but it also supports runtime coverage.

Runtime code coverage #

This tells you what code is running once the app starts, which code paths are getting executed and how many times. This is done using a Java Instrumentation API which allows you to write a program which can add some instrumentation changes to the generated bytecode. It is commonly used to:

All these methods require you to use a Java Agent which is attached to the code that you want to run. Below is an example on how you'd attach an agent to profile/monitor your code

java -javaagent:<your-monitoring-code>.jar -jar <application-code>.jar

Good news is JaCoCo already ships a java agent jar which can instrument the code at runtime! Wow, framework agnostic test coverage is here :)

I've created a sample repo to test this out, it's present here on https://github.com/ankeetmaini/jacoco-e2e

step 1 - build the jar #

Build your project and get the jar file ready. To build the sample project linked above, clone and run the following

./gradlew build

The jar file would be created at build/libs/rest-service-0.0.1-SNAPSHOT.jar path.

step 2 - run the jar with java agent #

The trick here is to attach the java agent before starting the application code. The java agent jar would add some bytecode to the app jar for instrumenting the code.

java -javaagent:jars/org.jacoco.agent-0.8.6-runtime.jar -jar build/libs/rest-service-0.0.1-SNAPSHOT.jar

I used the jars/ prefix as I stored it inside a jars folder in my codebase. Any valid relative path should work.

step 3 - run your tests #

Now is the time to go ballistic and run all your e2e tests in the framework of your choice, or run some curls, or make some requests via Postman.

For simplicity's sake I'm going to run some curls to test the coverage.

curl localhost:8080/greeting
{"id":1,"content":"Hello, World!"}

Once you're done, shut down the process and a jacoco.exec file should get created at your current working directory.

jacoco.exec file contains the codepath and number of times all the classes and methods were run. This is the file that contains raw coverage, but there's just one problem that it's not human readable.

step 4 - generate pretty HTML reports #

JaCoCo gives another jar which takes in the .exec file and converts it to a report.

java -jar jars/org.jacoco.cli-0.8.6-nodeps.jar report jacoco.exec --classfiles=build/classes --html coverage

jacoco coverage

Make sure you point the folder which contains the class files which were generated while building the file.

Closing #

The above post takes into account where the raw coverage data is dumped in one go when the JVM exits, but there are other options also where you can:

Please see this and the command line interface documentation for more information.

discuss on twitter, because why not?