Mostrando entradas con la etiqueta code coverage. Mostrar todas las entradas
Mostrando entradas con la etiqueta code coverage. Mostrar todas las entradas

lunes, agosto 06, 2012

JaCoCo Jenkins Plugin


Hier soir deux inconnus, Et ce matin, sur l'avenue - Deux amoureux, tout etourdis , Par la longue nuit (Aux Champs Elysées - Joe Dassin)
In my post about JaCoCo I wrote about the problems of using JaCoCo Maven plugin in multimodule Maven project because of having one report for each module separately instead of one report for all modules, and how can be fixed it using JaCoCo Ant Task.

In current post we are going to see how to use Jacoco Jenkins plugin to achieve the same goal of Ant Task and have an overall code coverage statistic of all modules.

First step is installing JaCoCo Jenkins plugin.

Go to Go to Jenkins -> Manage Jenkins -> Plugin Manager -> Available and find for JaCoCo Plugin

Next step, if it is not done before, is configuring your JaCoCo Maven plugin into parent pom:


And finally a post-action must be configured to the job responsible of packaging the application. Note that in previous pom file reports are generated just before package goal is executed.

Go to Configure -> Post-build Actions -> Add post-build action -> Record JaCoCo coverage report.

Then we have to set folders or files containing JaCoCo XML reports, which using previous pom is **/target/site/jacoco/jacoco*.xml, and also set when we consider that a build is healthy in terms of coverage.


Then we can save the job configuration and run build project.

After project is build, a new report will appear just under test result trend graph, called code coverage trend, where we can see the code coverage of all project modules.


From left menu, you can enter to Coverage Report and watch code coverage of each module separately.


Furthermore visiting Jenkins main page a nice quick overview of a job when mouse is over the weather icon is shown:


Keep in mind that this approach for merging code coverage files will only work if you are using Jenkins as a CI system meanwhile Ant Task is more generic solution and can also be used with JaCoCo Jenkins plugin.

We Keep Learning,
Alex.

Music: http://www.youtube.com/watch?v=OAMuNfs89yE



martes, julio 17, 2012

JaCoCo in Maven Multi-Module Projects



Can you blow my whistle baby, whistle baby, Let me know Girl I'm gonna show you how to do it And we start real slow You just put your lips together. (Whistle - Flo Rida)
Code coverage is an important measure used during our development that describes the degree to which source code is tested.

In this post I am going to explain how to run code coverage using Maven and JaCoCo plugin in multi-module projects.

JaCoCo is a code coverage library for Java, which has been created by the EclEmma team. It has a plugin for Eclipse, and can be run with Ant and Maven too.

Now we will focus only in Maven approach.

In a project with only one module is as easy as registering a build plugin:


And now running mvn package, in site/jacoco directory, a coverage report will be present in different formats.



But with multimodule projects a new problem arises. How to merge metrics of all subprojects into only one file, so we can have a quick overview of all subprojects? For now Maven JaCoCo Plugin does not support it.

There are many alternatives and I am going to cite the most common:

  • Sonar. It has the disadvantage that you need to install Sonar (maybe you are already using, but maybe not).
  • Jenkins. Plugin for JaCoCo is still under development. Moreover you need to run a build job to inspect your coverage. This is good in terms of continuous integration but could be a problem if you are trying to "catch" some piece of code that has not covered with already implemented tests.
  • Arquillian JaCoCo Extension. Arquillian is a container test framework that has an extension which during test execution can capture the coverage. Also a good option if you are using Arquillian. The disadvantage is that maybe your project does not require a container.
  • Ant. You can use Ant task with Maven. JaCoCo Ant task can merge results from multiple JaCoCo files result. Note that is the most generic solution, and this is the chosen approach that we are going to use.
First thing to do is add JaCoCo plugin to parent pom so all projects could generate coverage report. Of course if there are modules which does not require coverage, plugin definition should be changed from parent pom to specific projects.


Next step is creating a specific submodule for appending all results of JaCoCo plugin by using Ant task. I suggest  using something like project-name-coverage.

Then let's open generated pom.xml and we are going to insert required plugins to join all coverage information. To append them, as we have already written we are going to use a JaCoCo Ant task which has the ability to open all JaCoCo output files and append all their content into one. So first thing to do is download the jar which contains the JaCoCo Ant task. To automatize download process, we are going to use maven dependency plugin:

During process-test-resources phase Jacoco Ant artifact will be downloaded and copied to target directory, so can be registered into pom without worrying about jar location.

We also need a way to handle Ant tasks from Maven. And this is as simple as using maven antrun plugin, which you can specify any ant command in its configuration section. See next simple example:


Notice that into target tag we can specify any Ant task. And now we are ready to start configuring JaCoCo Ant task. JaCoCo report plugin requires you set the location of build directory, class directory, source directory or generated-source directory. For this purpose we are going set them as properties.

And now the Ant task part which will go into target tag of antrun plugin.

First we need to define report task.

See that org.jacoco.ant.jar file is downloaded by dependency plugin, you don't need to worry about copying it manually.

Then we are going to call report task as defined in taskdef section.


Within executiondata element, we specify locations where JaCoCo execution data files are stored. By default is target directory, and for each project we need to add one entry for each submodule.

Next element is structure. This element defines the report structure, and can be defined with hierarchy of group elements. Each group  should contain class files and source files of all projects that belongs to that group. In our example only one group is used.

And finally we are setting output format using html, xml and csv tags.

Complete Code:


And now simply run mvn clean verify and in my-project-coverage/target/coverage-report, a report with code coverage of all projects will be presented.

Hope you find this post useful.

We Keep Learning,
Alex.

Screencast
Download Code
Music: http://www.youtube.com/watch?v=cSnkWzZ7ZAA

martes, marzo 06, 2012

Keep 'em laughing as you go, Just remember that the last laugh is on you, And always look on the bright side of life..., Always look on the right side of life... (Always Look on the Bright Side of Life - Mony Python)




Integration tests are kind of tests which individual modules are combined and tested as a whole. Moreover integration tests might use system dependent values, accessing external systems like file system, database, web services, ..., and testing multiple aspects of one test case. We can say it is a high-level test.

This differs from unit test where only a single component is tested. Unit tests runs in isolation, mocking-out external components or using in-memory database in case of DAO layers. A unit test might be:
  • Repeatable.
  • Consistent.
  • In Memory.
  • Fast.
  • Self-validating.
  • Testing single concept

The problem when we are writing tests, is how to test rare (or untypical) conditions like "No disk space" in case of accessing file system, or "Connection lost" when executing a database query.

In unit testing this is not a problem you can mock up that component (database connection or filesystem access), generating required output like throwing IOException.

The problem becomes "harder" with integration tests. It would be strange to mock a component, when what you really want to do is validate the real system. So arrived at this point I see two possibilities:
  • Creating a partial mock.
  • Using fault injection.
In this post I am going to show you how to use fault injection approach to test unusual erroneous situations. 

Fault injection is a technique which involves changing application code under test at specific locations. This modifications will introduce faults on error handling code paths which otherwise would rarely be followed.

I am going to talk about how to use fault injection using Byteman in a JUnit test, and run it with Maven.

Let's start coding. Imagine you need to write a backup module, which shall save a string into a local file, but if hard disk is full (IOException is thrown), content shall be sent to remote server.

First we are going to code a class that writes content into file.



Next class, would be the one that sends data through socket but will not be shown, because it is not necessary for this example.

And finally the backup service responsible of managing described behavior.

And now testing time. First of all a brief introduction to Byteman.

Byteman is a tool which allows you to insert/modify code into an application at runtime. These modifications can be used to inject code on your compiled application causing unusual or unexpected operations (aka Fault Injection).

Byteman uses a clear, simple scripting language, based on a formalism called Event Condition Action (ECA) rules to specify where, when and how the original Java code should be transformed.

An example of ECA script is:

But Byteman also supports annotations. And in my opinion, annotations are a better approach than script file, because only watching your test case you can understand what you are exactly testing. If not you should switch context from unit class to script file to understand what are you testing.

So let's create an integration test that that validates that when IOException is thrown while writing content into disk, data is sent to a server.


See that BMUnitRunner (a special jUnit runner that comes with Byteman) is required.

First test called aFileWithContentShouldBeCreated is a standard test that writes Hello world into backup file.

But the second one dataShouldBeSentToServerInCaseOfIOException, has BMRule annotation which will contain when, where and what code should be injected. First parameter is the name of the rule, in this case a description of what we are going to do (throwing an IOException). Next attributes, targetClass and targetMethod configure when injected code should be added. In this case when FileUtils.createFileWithContent method is called. Next attribute targetLocation is location where code is inserted, and in our case is where createFileWithContent method calls write method of BufferedWriter. And finally what to do that obviously in this test is throwing an IOException.

So now you can go to your IDE and run them, and all tests should pass, but if you run through Maven using Surefire plugin, test will not work. To use Byteman with Maven, Surefire plugin should be configured in a specific way.


First important thing is adding tools jar as dependency. This jar provides classes needed in order to dynamically install the Byteman agent.

In Surefire plugin configuration is important to set useManifestOnlyJar to false to ensure that the Byteman jar appears in the classpath of the test JVM.  Also see that we are defining empty environment variables (BYTEMAN_HOME and org.jboss.byteman.home). This is because when it loads the agent the BMUnit package will use environment variable BYTEMAN_HOME or System property org.jboss.byteman.home to locate byteman.jar but only if it is a non-empty string. Otherwise it scans the classpath to locate the jar. Because we want to ensure that jar added on dependency section is used, we are overriding any other configuration present on system.

And now you can run mvn clean test and two tests are successful too.

See that Byteman opens a new world into how we are writing our integration tests, now we can test in an easy way unusual exceptions like Communications Error, Input/Output Exceptions or Out Of Memory Error. Moreover because we are not mocking FileUtils, we are executing real code; for example in our second test, we are running a few lines of FileUtils object until write method is reached. If we had mocked-up FileUtils class, these lines would not be executed. Thanks of using fault injection our code coverage is improved.

Byteman is more than what I have shown you, it also has built-ins designed for testing in multithreaded environments, parameter binding, and an amount of location specifiers, to cite a few things.

I wish you have found this post useful and help you testing rare conditions of your classes.

Download Code
Music: http://www.youtube.com/watch?v=WlBiLNN1NhQ