NoSQLUnit 0.4.0 Released

All across the nation such a strange vibration, People in motion, There's a whole generation with a new explanation, People in motion people in motion (San Francisco - Scott McKenzie)

NoSQLUnit is a JUnit extension to make writing unit and integration tests of systems that use NoSQL backend easier. Visit official page for more information.

In 0.4.0 release, one new NoSQL system is supported and is Cassandra.

Cassandra is a BigTable data model running on an Amazon Dynamo-like infrastructure.

As all databases supported by NoSQLUnit, two set of rules are provided to write Cassandra tests:

First set of JUnit Rules are those responsible of managing database lifecycle; basically starting and stopping Cassandra instance.

• Embedded: com.lordofthejars.nosqlunit.cassandra.EmbeddedCassandra
• Managed: com.lordofthejars.nosqlunit.cassandra.ManagedCassandra

Depending on kind of tests you are implementing (unit test, integration test, deployment tests, …) you will require an embedded, managed or remote approach. Note that for now I haven to implemented an In-Memory approach because there is no in-memory Cassandra  instance, but embedded strategy for unit tests will be the better one. 

Second set of rules are those responsible of maintaining database into known state;

• NoSQLUnit Management: com.lordofthejars.nosqlunit.cassandra.CassandraRule

And finally default dataset file format in Cassandra is json. To make compatible NoSQLUnit with Cassandra-Unit file format, DataLoader of Cassandra-Unit project is used, so same json format file is used.

We will use a very simple example used in Cassandra tutorial as an example of how to write unit tests for systems that uses Cassandra database as backend.

First of all, dataset used to maintain Cassandra into known state:

and finally the test case:

Next release 0.4.1 will contain some internal changes, but no support for new engine. And after these changes, new engines will be supported. Moreover I have decided to open a poll to vote which engine would you like to see in 0.4.2 release:

Visit Poll Here

Stay in touch with the project and of course I am opened to any ideas that you think that could make NoSQLUnit better.

Music: http://www.youtube.com/watch?v=bch1_Ep5M1s

NoSQLUnit 0.3.2 Released

Yo no te pido la luna, tan solo quiero amarte. Quiero ser esa locura que vibra muy dentro de tí (Yo No Te Pido La Luna - Sergio Dalma)

Update: 0.3.3 version has been released providing In-Memory Neo4j lifecycle support.

NoSQLUnit is a JUnit extension to make writing unit and integration tests of systems that use NoSQL backend easier. Visit official page for more information.

In 0.3.2 release, one new NoSQL system is supported and is Neo4j.

Neo4j is a high-performance, NoSQL graph database with all the features of a mature and robust database.

As all databases supported by NoSQLUnit, two set of rules are provided to write Neo4j tests:

First set of JUnit Rules are those responsible of managing database lifecycle; basically starting and stopping Neo4j.

• Embedded: com.lordofthejars.nosqlunit.neo4j.EmbeddedNeo4j
• Managed Wrapping: com.lordofthejars.nosqlunit.neo4j.ManagedWrappingNeoServer
• Managed: com.lordofthejars.nosqlunit.neo4j.ManagedNeoServer

Depending on kind of test you are implementing (unit test, integration test, deployment test, ...) you will require an embedded approach, managed approach or  remote approach. Note that for now I have not implementated of a Neo4j in-memory database at this time (0.3.3 will do), but embedded strategy for unit tests will be the better one. As Neo4j developers will know, you can start remote Neo4j server by calling start/stop scripts (Managed way) or by using an embedded database wrapper by a server (Managed Wrapping way), both of them are supported by NoSQLUnit.

Second set of rules are those responsible of maintaining database into known state;

• NoSQLUnit Management: com.lordofthejars.nosqlunit.neo4j.Neo4jRule

And finally default dataset file format in Neo4j is GraphML. GraphML is a comprehensive and easy-to-use file format for graphs.

We will use the example of finding Neo's friends as an example of how to write unit tests for systems that uses Neo4j databases as backend.

First of all dataset used to maintain Neo4j into known state:

and finally the test case:

Next release will support Cassandra. Stay in touch with the project and of course I am opened to any ideas that you think that could make NoSQLUnit better.

We Keep Learning,
Alex.

Source Code
Music: http://www.youtube.com/watch?v=NHGde2dy3E8

JaCoCo Jenkins Plugin

Hier soir deux inconnus, Et ce matin, sur l'avenue - Deux amoureux, tout etourdis , Par la longue nuit (Aux Champs Elysées - Joe Dassin)

In my post about JaCoCo I wrote about the problems of using JaCoCo Maven plugin in multimodule Maven project because of having one report for each module separately instead of one report for all modules, and how can be fixed it using JaCoCo Ant Task.

In current post we are going to see how to use Jacoco Jenkins plugin to achieve the same goal of Ant Task and have an overall code coverage statistic of all modules.

First step is installing JaCoCo Jenkins plugin.

Go to Go to Jenkins -> Manage Jenkins -> Plugin Manager -> Available and find for JaCoCo Plugin

Next step, if it is not done before, is configuring your JaCoCo Maven plugin into parent pom:


And finally a post-action must be configured to the job responsible of packaging the application. Note that in previous pom file reports are generated just before package goal is executed.

Go to Configure -> Post-build Actions -> Add post-build action -> Record JaCoCo coverage report.

Then we have to set folders or files containing JaCoCo XML reports, which using previous pom is **/target/site/jacoco/jacoco*.xml, and also set when we consider that a build is healthy in terms of coverage.

Then we can save the job configuration and run build project.

After project is build, a new report will appear just under test result trend graph, called code coverage trend, where we can see the code coverage of all project modules.

From left menu, you can enter to Coverage Report and watch code coverage of each module separately.

Furthermore visiting Jenkins main page a nice quick overview of a job when mouse is over the weather icon is shown:

Keep in mind that this approach for merging code coverage files will only work if you are using Jenkins as a CI system meanwhile Ant Task is more generic solution and can also be used with JaCoCo Jenkins plugin.

We Keep Learning,
Alex.

Music: http://www.youtube.com/watch?v=OAMuNfs89yE

Build Flow Jenkins Plugin

Este samba, Que é misto de maracatú, É samba de preto velho, Samba de preto tu (Mais que Nada - Sergio Mendes)

With the advent of Continuous Integration and Continuous Delivery, our builds are split into different steps creating the deployment pipeline. Some of these steps can be for example compile and run fast tests, run slow tests, run automated acceptance tests, or releasing the application, to cite a few.

Most of us we are using Jenkins/Hudson to implement Continuous Integration/Delivery, and we manage job orchestration combining some Jenkins plugins like build pipeline, parameterized-build, join or downstream-ext. We require configuring all of them which implies polluting the job configuration through multiple jobs, which takes the system configuration very complex to maintain.

Build Flow enables us to define an upper level flow item to manage job orchestration and link up rules, using a dedicated DSL.

Let's see a very simple example:

First step is installing the plugin.

Go to Jenkins -> Manage Jenkins -> Plugin Manager -> Available and find for CloudBees Build Flow plugin.

Then you can go to Jenkins -> New Job and you will see a new kind of job called Build Flow. In this example we are going to name it build-all-yy.

And now you only have to program using flow DSL how this job should orchestrate the other jobs.

In "Define build flow using flow DSL" input text you can specify the sequence of commands to execute.

In current example I have already created two jobs, one executing clean compile goal (yy-compile job name) and the other one executing javadoc goal (yy-javadoc job name). I know that this deployment pipeline is not real in a true environment but for now it is enough. 

Then we want javadoc job running after project is compiled.

To configure this we don't have to create any upstream or downstream actions, simply add next lines at DSL text area:

build("yy-compile");

build("yy-javadoc");

Save and execute build-all-yy job and both projects will be built in a sequential way.

Now suppose that we add a third job called yy-sonar which runs sonar goal that generates code quality sonar report. In this case it seems obvious that after project is compiled, generation of javadocs and code quality jobs can be run in parallel. So script is changed to:

build("yy-compile")

parallel (

    {build("yy-javadoc")},

    {build("yy-sonar")}

)

This plugin also supports more operations like retry (similar behaviour of retry-failed-job plugin) or guard-rescue, that it works mostly like a try+finally block. Also you can create parameterized builds, accessing to build execution or printing to Jenkins console. Next example will print build number of yy-compile job execution:

b = build("yy-compile")

out.println b.build.number

And finally you can also have a quick graphical overview of the execution in Status section. It is true that could be improved more, but for now it is acceptable, and can be used without any problem.

Build Flow plugin is in its early stages, in fact it is only at version 0.4. But will be a plugin to be considered in future, and I think it is good to know that it exists. Moreover is being developed by CloudBees folks so it is a guarantee of being fully supported by Jenkins.

We Keep Learning.

Alex.

Music: https://www.youtube.com/watch?v=ZSZ15R2DFBg

Warning: In order to run parallel tasks with the plugin Anonymous users must have Read Job access (Jenkins -> Manage Jenkins -> Configure System). There is an issue already inserted into Jira to fix this problem.

Answering with Mockito

Overwhelmed by industry, Searching for a modern day savior from another place, Inclined toward charity (The Answer - Bad Religion)

When you are writing unit tests, you must keep in mind to not have dependencies to external components. To avoid this we use mock frameworks which for me the easiest one to use is Mockito.

In this post we are going to see an "advanced" technique used in Mockito to return same argument instance on a mocked method using Answer interface.

Suppose we are writing unit tests for class which manages Person and Job classes and as operation it uses a DAO class for inserting the relationship class (M:N) between Person and Job called PersonJob.

For example class under test will look something like:

So in this case it seems obvious that you need to mock personJobDao.

Let's create the mock and record the interaction:

Yes as you can see you don't know what to return, because instance is created by class under test and in the test method you don't know which instance is created by createPersonJob method. To solve this problem, you need to use thenAnswer instead of thenReturn method:

Note that Answer interface requires you to implement answer method, which in our case simply returns the first argument (PersonJob instance) of personJobDao.create method.

Now we can write assertions in peace without worrying about returned instance.

Hope you have found this post useful.

We keep learning
Alex.

Download Code
Music: http://www.youtube.com/watch?v=S2a3q0nIsoM

JaCoCo in Maven Multi-Module Projects

Can you blow my whistle baby, whistle baby, Let me know Girl I'm gonna show you how to do it And we start real slow You just put your lips together. (Whistle - Flo Rida)

Code coverage is an important measure used during our development that describes the degree to which source code is tested.

In this post I am going to explain how to run code coverage using Maven and JaCoCo plugin in multi-module projects.

JaCoCo is a code coverage library for Java, which has been created by the EclEmma team. It has a plugin for Eclipse, and can be run with Ant and Maven too.

Now we will focus only in Maven approach.

In a project with only one module is as easy as registering a build plugin:

And now running mvn package, in site/jacoco directory, a coverage report will be present in different formats.

But with multimodule projects a new problem arises. How to merge metrics of all subprojects into only one file, so we can have a quick overview of all subprojects? For now Maven JaCoCo Plugin does not support it.

There are many alternatives and I am going to cite the most common:

• Sonar. It has the disadvantage that you need to install Sonar (maybe you are already using, but maybe not).
• Jenkins. Plugin for JaCoCo is still under development. Moreover you need to run a build job to inspect your coverage. This is good in terms of continuous integration but could be a problem if you are trying to "catch" some piece of code that has not covered with already implemented tests.
• Arquillian JaCoCo Extension. Arquillian is a container test framework that has an extension which during test execution can capture the coverage. Also a good option if you are using Arquillian. The disadvantage is that maybe your project does not require a container.
• Ant. You can use Ant task with Maven. JaCoCo Ant task can merge results from multiple JaCoCo files result. Note that is the most generic solution, and this is the chosen approach that we are going to use.

First thing to do is add JaCoCo plugin to parent pom so all projects could generate coverage report. Of course if there are modules which does not require coverage, plugin definition should be changed from parent pom to specific projects.

Next step is creating a specific submodule for appending all results of JaCoCo plugin by using Ant task. I suggest  using something like project-name-coverage.

Then let's open generated pom.xml and we are going to insert required plugins to join all coverage information. To append them, as we have already written we are going to use a JaCoCo Ant task which has the ability to open all JaCoCo output files and append all their content into one. So first thing to do is download the jar which contains the JaCoCo Ant task. To automatize download process, we are going to use maven dependency plugin:

During process-test-resources phase Jacoco Ant artifact will be downloaded and copied to target directory, so can be registered into pom without worrying about jar location.

We also need a way to handle Ant tasks from Maven. And this is as simple as using maven antrun plugin, which you can specify any ant command in its configuration section. See next simple example:

Notice that into target tag we can specify any Ant task. And now we are ready to start configuring JaCoCo Ant task. JaCoCo report plugin requires you set the location of build directory, class directory, source directory or generated-source directory. For this purpose we are going set them as properties.

And now the Ant task part which will go into target tag of antrun plugin.

First we need to define report task.

See that org.jacoco.ant.jar file is downloaded by dependency plugin, you don't need to worry about copying it manually.

Then we are going to call report task as defined in taskdef section.

Within executiondata element, we specify locations where JaCoCo execution data files are stored. By default is target directory, and for each project we need to add one entry for each submodule.

Next element is structure. This element defines the report structure, and can be defined with hierarchy of group elements. Each group  should contain class files and source files of all projects that belongs to that group. In our example only one group is used.

And finally we are setting output format using html, xml and csv tags.

Complete Code:

And now simply run mvn clean verify and in my-project-coverage/target/coverage-report, a report with code coverage of all projects will be presented.

Hope you find this post useful.

We Keep Learning,
Alex.

Screencast
Download Code
Music: http://www.youtube.com/watch?v=cSnkWzZ7ZAA

NoSQLUnit 0.3.1 Released

And the last known survivor, Stalks his prey in the night, And he's watching us all with the, Eye of the tiger (Eye Of The Tiger - Survivor)

NoSQLUnit is a JUnit extension to make writing unit and integration tests of systems that use NoSQL backend easier. Visit official page for more information.

In 0.3.1 release, two new features among some bug fixes has been implemented. These two new features are the ability of create simultaneous connections to different backends at same test, and partial support for jsr-330 specification.

Simultaneous engines

Sometimes applications will contain more than one NoSQL engine, for example some parts of your model will be expressed better as a graph ( Neo4J for example), but other parts will be more natural in a column way (for example using Cassandra ). NoSQLUnit supports this kind of scenarios by providing in integration tests a way to not load all datasets into one system, but choosing which datasets are stored in each backend.

For declaring more than one engine, you must give a name to each database Rule using connectionIdentifier() method in configuration instance.

And also you need to provide an identified dataset for each engine, by using withSelectiveLocations attribute of @UsingDataSet annotation. You must set up the pair "named connection" / datasets.

In example we are refreshing database declared on previous example with data located at test3 file.

Also works in expectations annotation:

When you use more than one engine at a time you should take under consideration next rules:

• If location attribute is set, it will use it and will ignore withSelectiveMatcher attribute data. Location data is populated through all registered systems.
• If location is not set, then system tries to insert data defined in withSelectiveMatcher attribute to each backend.
• If withSelectiveMatcher attribute is not set, then default strategy is taken. Note that default strategy will replicate all datasets to defined engines.

You can also use the same approach for inserting data into same engine but in different databases. If you have one MongoDb instance with two databases, you can also write tests for both databases at one time. For example:



Support for JSR-330

NoSQLUnit supports two annotations of JSR-330 aka Dependency Injection for Java. Concretely @Inject and @Named annotations.

During test execution you may need to access underlying class used to load and assert data to execute extra operations to backend. NoSQLUnit will inspect @Inject annotations of test fields, and try to set own driver to attribute. For example in case of MongoDb, com.mongodb.Mongo instance will be injected.

Note that in example we are setting this as second parameter to the Rule.

But if you are using more than one engine at same time, you need a way to distinguish each connection. For fixing this problem, you must use @Named annotation by putting the identifier given in configuration instance. For example:

Next release will support Neo4J and Cassandra. Stay in touch with the project and of course I am opened to any ideas that you think that could make NoSQLUnit better.

Music: http://www.youtube.com/watch?v=btPJPFnesV4

Bye, Bye, 5 * 60 * 1000 //Five Minutes, Bye, Bye

Llueve, llueve, y mientras nos mojamos como tontos. LLueve, llueve, y en un simple charco a veces nos ahogamos. (Llueve - Melendi)

In this post I am going to talk about one class that was first introduced in version 1.5, that I have used too much but talking with some people they said that they didn't know it exists. This class is TimeUnit.

TimeUnit class represents time durations at a given unit of granularity and also provides utility methods to convert to different units, and methods to perform timing delays.

TimeUnit is an enum with seven levels of granularity: DAYS, HOURS, MICROSECONDS, MILLISECONDS, MINUTES, NANOSECONDS and SECONDS.

The first feature that I find useful is the convert method. With this method you can say good bye to typical:

     private static final int FIVE_SECONDS_IN_MILLIS = 1000 * 5;

to something like:

     long duration = TimeUnit.MILLISECONDS.convert(5, TimeUnit.SECONDS);

But also equivalent operations in a better readable method exist. For example the same conversion could be expressed as:

     long duration = TimeUnit.SECONDS.toMillis(5);

The second really useful sets of operations are those related with stopping current thread.

For example you can sleep current thread with method:

     TimeUnit.MINUTES.sleep(5);

instead of:

     Thread.sleep(5*60*1000);

But you can also use it with join and wait with timeout.

     Thread t = new Thread();

     TimeUnit.SECONDS.timedJoin(t, 5);

So as we can see TimeUnit class is though in terms of expressiveness, you can do the same as you do previously but in a more fashionable way. Notice that you can also use static import and code will be even more readable.

Keep Learning,

Alex.

Music: http://www.youtube.com/watch?v=b79HiXuNt7U

NoSQLUnit 0.3.0 Released

Se você me olhar vou querer te pegar, E depois namorar curtição, Que hoje vai rolar... (Balada Boa - Gustavo Lima)

Introduction

Unit testing is a method by which the smallest testable part of an application is validated. Unit tests must follow the FIRST Rules; these are Fast, Isolated, Repeatable, Self-Validated and Timely.

It is strange to think about a JEE application without persistence layer (typical Relational databases or new NoSQL databases) so should be interesting to write unit tests of persistence layer too. When we are writing unit tests of persistence layer we should focus on to not break two main concepts of FIRST rules, the fast and the isolated ones.

Our tests will be fast if they don't access network nor filesystem, and in case of persistence systems network and filesystem are the most used resources. In case of RDBMS ( SQL ), many Java in-memory databases exist like Apache Derby , H2 or HSQLDB . These databases, as their name suggests are embedded into your program and data are stored in memory, so your tests are still fast. The problem is with NoSQL systems, because of their heterogeneity. Some systems work using Document approach (like MongoDb ), other ones Column (like Hbase ), or Graph (like Neo4J ). For this reason the in-memory mode should be provided by the vendor, there is no a generic solution.

Our tests must be isolated from themselves. It is not acceptable that one test method modifies the result of another test method. In case of persistence tests this scenario occurs when previous test method insert an entry to database and next test method execution finds the change. So before execution of each test, database should be found in a known state. Note that if your test found database in a known state, test will be repeatable, if test assertion depends on previous test execution, each execution will be unique. For homogeneous systems like RDBMS , DBUnit exists to maintain database in a known state before each execution. But there is no like DBUnit framework for heterogeneous NoSQL systems.

NoSQLUnit resolves this problem by providing a JUnit extension which helps us to manage lifecycle of NoSQL systems and also take care of maintaining databases into known state.

NoSQLUnit

NoSQLUnit is a JUnit extension to make writing unit and integration tests of systems that use NoSQL backend easier and is composed by two sets of Rules and a group of annotations.

First set of Rules are those responsible of managing database lifecycle; there are two for each supported backend.

• The first one (in case it is possible) it is the in-memory mode. This mode takes care of starting and stopping database system in "in-memory" mode. This mode will be typically used during unit testing execution.

• The second one is the managed mode. This mode is in charge of starting NoSQL server but as remote process (in local machine) and stopping it. This will typically used during integration testing execution.

Second set of Rules are those responsible of maintaining database into known state. Each supported backend will have its own, and can be understood as a connection to defined database which will be used to execute the required operations for maintaining the stability of the system.

Note that because NoSQL databases are heterogeneous, each system will require its own implementation.

And finally two annotations are provided, @UsingDataSet and @ShouldMatchDataSet , (thank you so much Arquillian people for the name) to specify locations of datasets and expected datasets.

MongoDb Example

Now I am going to explain a very simple example of how to use NoSQLUnit, for full explanation of all features provided, please read documentation in link or download in pdf format.

To use NoSQLUnit with MongoDb you only need to add next dependency:

First step is defining which lifecycle management strategy is required for your tests. Depending on kind of test you are implementing (unit test, integration test, deployment test, ...) you will require an in-memory approach, managed approach or remote approach.

For this example we are going to use managed approach using ManagedMongoDb Rule) but note that in-memory MongoDb management is also supported (see documentation how).

Next step is configuring Mongodb rule in charge of maintaining MongoDb database into known state by inserting and deleting defined datasets. You must register MongoDbRule JUnit rule class, which requires a configuration parameter with information like host, port or database name.

To make developer's life easier and code more readable, a fluent interface can be used to create these configuration objects.

Let's see the code:

First thing is a simple POJO class that will be used as model class:

Next business class is the responsible of managing access to MongoDb server:

And now it is time for testing. In next test we are going to validate that a book is inserted correctly into database.

See that first of all we are creating using ClassRule annotation a managed connection to MongoDb server. In this case we are configuring MongoDb path programmatically, but also can be set from MONGO_HOME environment variable. See here full description of all available parameters.

This Rule will be executed when test is loaded and will start a MongoDb instance. Also will shutdown the server when all tests have been executed.

Next Rule is executed before any test method, and is responsible of maintaining database into known state. Note that we are only configuring working database, in this case the test one.

And finally we annotate method test with @UsingDataSet indicating where to find data to be inserted before execution of each test, and @ShouldMatchDataSet locating expected dataset.

We are setting an initial dataset in file initialData.json located at classpath com/lordofthejars/nosqlunit/demo/mongodb/initialData.json and expected dataset called expectedData.json.


Final Notes

Although NoSQLUnit is at early stages, the part of MongoDb is almost finished, in next releases new features and of course new databases will be supported. Next NoSQL supported engines will be Neo4J, Cassandra, HBase and CouchDb.

Also read the documentation where you will find an full explanation of each feature explained here.

And finally any suggestion you have, any recommendation, or any advice will be welcomed.


Stay In Touch


Email: asotobu at gmail.com

Blog: Lord Of The Jars
Twitter: @alexsotob
Github: NoSQLUnit Github

Keep Learning,
Alex

Full Code
Music: http://www.youtube.com/watch?v=8y5CbeHY7X0

Testing Abstract Classes (and Template Method Pattern in Particular)

Sick at heart and lonely, deep in dark despair. Thinking one thought only, where is she tell me where. (Heart Full of Soul - The Yardbirds).

From wikipedia "A template method defines the program skeleton of an algorithm. One or more of the algorithm steps can be overridden by subclasses to allow differing behaviors while ensuring that the overarching algorithm is still followed".

Typically this pattern is composed by two or more classes, one that is an abstract class providing template methods (non-abstract) that have calls to abstract methods implemented by one or more concrete subclasses.

Often template abstract class and concrete implementations reside in the same project, but depending on the scope of the project, these concrete objects will be implemented into another project.

In this post we are going to see how to test template method pattern when concrete classes are implemented on external project, or more general how to test abstract classes.

Let's see a simple example of template method pattern. Consider a class which is responsible of receiving a vector of integers and calculate the Euclidean norm. These integers could be received from multiple sources, and is left to each project to provide a way to obtain them.

The template class looks like:

Now another project could extend previous class and make an implementation of  abstract calculator by providing an implementation of read() method .

Developer that has written a concrete implementation will test only read() method, he can "trust" that developer of abstract class has tested non-abstract methods.

But how are we going to write unit tests over calculate method if class is abstract and an implementation of read() method is required?

The first approach could be creating a fake implementation:

This is not a bad approach, but has some disadvantages:

• Test will be less readable, readers should know the existence of these fake classes and must know exactly what are they doing. 
• As a test writer you will spend time in implementing fake classes, in this case it is simple, but your project could have more than one abstract class without implementation, or even with more than one abstract method.
• Behaviour of fake classes are "hard-coded".

A better way is using Mockito to mock only abstract method meanwhile implementation of non-abstract methods are called.

Mockito simplifies the testing of abstract classes by calling real methods, and only stubbing abstract methods. See that in this case because we are calling real methods by default, instead of using the typical when() then() structure, doReturn schema must be used.

Of course this approach can be only used if your project does not contain a concrete implementation of algorithm or your project will be a part of a 3rd party library on another project. In the other cases the best way of attacking the problem is by testing the implemented class.

Download sourcecode

Music: http://www.youtube.com/watch?v=9O6eGOu27DA

I'm guided by this birthmark on my skin, I'm guided by the beauty of our weapons, First we take Manhattan, then we take Berlin (First We Take Manhattan - Leonard Cohen)

On May 23, I was at Berlin as speaker in LinuxTag. I talked about how to test modern Enterprise Java Applications using open source tools.

Presentation abstract was:

Ten years ago to present, Enterprise Java Applications have suffered many changes. We have moved from Enterprise Applications built with JSP+Servlet and EJB, to much more complex applications. Nowadays with the advent of HTML5 or JavaScript libraries like JQuery, client side development has changed significantly. With the emergence of web frameworks like Spring MVC or JSF,  server side code has quite changed compared to the one used when each web-form was mapped to a Servlet. And also persistence layer has changed with Java Persistence standard or with new database approaches like Data-Grid, Key-Values stores or Document stores.

Moreover, architectural changes have occurred too, REST-web applications have grown in popularity or AJAX is used to create asynchronous web applications. Due to development of Enterprise Java Applications have changed during these years, so testing frameworks have changed accordantly. The main topic of this speech will be how to test Enterprise Java Applications using these new frameworks.

In the first part of this presentation we are going to explore how to test JavaScript written on client side, how to write unit tests of server side code, and how to validate persistence layer. Next part of presentation will be focused on how to write integration tests on server side and acceptance tests on full Enterprise Java Applications (joining client and server side) and an introduction about testing REST-web applications. Finally we will show how to integrate all kind of test on your continuous integration system and run acceptance tests on test environment.

Session will combine theory with interactive practice using only open-source projects.

I have uploaded slides to slideshare so you can  take a look (sorry for red and blue colours):

How to Test Enterprise Java Applications

View more presentations from Alex Soto

Also if you want you can download the code that it was used in demo sections.

Javascript Unit Testing with JS Test Driver
NoSQL Unit Testing with NoSQLUnit
Integration Tests with Arquillian
Acceptance Tests with Thucydides

Please let me warn you that NoSQLUnit is an open source project that I am developing, and it is on early stages, in next month, project will have a better look by supporting more NoSQL systems like Neo4j, Cassandra or CouchDb and having an official (not snapshot) release. If you want you can follow me on Twitter or subscribing to NoSQLUnit github repository and receive the last news of this JUnit extension.

For any question do not hesitate to write them in comments section or sending me an email.

I would like to say thank you to linuxtag folks for treating me so well and all people who came to presentation, for all of them a big thank you.

Music: http://www.youtube.com/watch?v=JTTC_fD598A&ob=av2e

Nasha nasha krovatka delala shik - shik, Ya tvo pianino , a ty moi nastroishchik, My tak letali chto ne zametili tvoyu matʹ, Ahaa..I ona skazala chto ya prosto blaz (Mama Lyuba - Serebro))

When we develop any application, after we finish it or when we end up any of its module, we start optimization process. Most applications contain database access, and if you are using an ORM, probably you will use hibernate. Optimizing hibernate persistence layer, requires to be prepared to read, understand and evaluate sql.

If we take an overview on hibernate configuration, two properties, hibernate.format_sql and hibernate.use_sql_comments, should be enabled to print performed sql code through console.

This is a good start but it seems that we need more information to make an accurate diagnosis of performance like connection events, returned data on queries, or parameters binding (hibernate shows parameters values with question mark ?) . Hence, we need another way to inspect generated sql. Log4jdbc is a jdbc driver that can log sql/jdbc calls. In fact log4jdbc is an implementation of proxy pattern which will automatically load popular jdbc drivers (Oracle, Derby, MySql, PostgreSql, H2, Hsqldb, ...), intercept calls, log information, and then send data to "spied" driver.

In log4jdbc, there are 5 loggers that can be used depending on data to monitor:

• jdbc.sqlonly: logs executed sql with binding arguments replaced with bound data.
• jdbc.sqltiming: logs how long a sql took to execute.
• jdbc.audit: logs all jdbc calls except for ResultSets.
• jdbc.resultset: same as jdbc.audit plus ResultsSets.
• jdbc.connection: logs open and close connection events.

In this post we are going to see how to configure log4jdbc-remix, a fork of log4jdbc, which apart from inheriting log4jdbc capabilities, also let us:

• jdbc.resultsettable: log results set in table format.
• configure it as datasource.
• available in maven repository (log4jdbc is not present on maven repositories).

For this example we are going to use the project created by JPA Spring Template which contains two entities Order and Item associated with one-to-many relationship, and one test that executes some database operations.

First thing to do is add log4jdb-remix and slf4j-log4j12 dependencies to project:

Next thing to do is configure active loggers.  Depending on the data we are interested to monitor, we activate the required loggers. As an example let's configure log4j.xml so result set is printed in table format and also time taken to execute each query is shown.

After configuring loggers, run test and inspect the output.


Output is printed in a fashion format, queries contains bind parameters (not a question mark (?)), and process time is also informed.

Notice that logging more or less information is simply a matter of configuring a log. Moreover depending on log level, more or less information will be provided in each case. If logger is configured in DEBUG class name and line number (if available) at which the sql was executed will be included. In INFO will include only sql, and finally ERROR which show stacktraces if any SQLException occurs.

Optimizing hibernate applications can imply touching many parts of an application (JVM configuration, database engine, network, …) but one very important aspect to take care is the number of queries that are sent to RDBMS (for example N+1 problem), and the amount of data that is retrieved from database (Projection problem), and log4jdbc-remix perfectly fits to help in this purpose.

As final note, log4jdbc(-remix) is a jdbc logger, so it is not necessary to use only in hibernate applications, can be used with any framework that uses a datasource.

I wish this library would help you.

Keep Learning,
Alex

Download Code
Music: http://www.youtube.com/watch?v=h9HRHOXfRBI

Qui dit crise te dis monde dit famine dit tiers- monde, Qui dit fatigue dit réveille encore sourd de la veille, Alors on sort pour oublier tous les problèmes, Alors on danse... (Alors on Danse - Stromae)

Let's introduce another hibernate performance tip. Do you remember the model of previous hibernate post? We had a starship and officer related with a one to many association.

Now we have next requirement:

We shall get all officers assigned to a starship by alphabetical order.

To solve this requirement we can:

1. implementing an HQL query with order by clause.
2. using sort approach.
3. using order approach.

The first solution is good in terms of performance, but implies more work as a developers because we should write a query finding all officers of given starship ordered by name and then create a finder method in DAO layer (in case you are using DAO pattern).

Let's explore the second solution,  we could use SortedSet class as association, and make Officer implements Comparable, so Officer has natural order. This solution implies less work than the first one, but requires using @Sort hibernate annotation on association definition. So let's going to modify previous model to meet our new requirement. Note that there is no equivalent annotation in JPA specification.

First we are going to implement Comparable interface in Officer class.

We are ordering officer by name by simply comparing name field. Next step is annotating association with @Sort.

Notice that now officers association is implemented using SortedSet instead of a List.   Furthermore we are adding @Sort annotation to relationship, stating that officers should be natural ordered. Before finishing this post we will insist more in @Sort topic, but for now it is sufficient.

And finally a method that gets all officers of given starship ordered by name, printing them in log file.

All officers are sorted by their names, but let's examine which queries are sent to RDBMS.

First query is resulting of calling find method on EntityManager instance finding starship.

Because one to many relationships are lazy by default when we call getOfficers method and we access first time to SortedSet, second query is executed to retrieve all officers. See that no order by clause is present on query, but looking carefully on output, officers are retrieved in alphabetical order.

So who is sorting officer entities? The explanation is on @Sort annotation. In hibernate a sorted collection is sorted in memory being Java the responsible of sorting data using compareTo method.

Obviously this method is not the best performance-way to sort a collection of elements. It is likely that we'll need a hybrid solution between using SQL clause and using annotation instead of writing a query.

And this leads us to explain the third possibility, using ordering approach.

@OrderBy annotation, available as hibernate annotation and JPA annotation, let us specifies how to order a collection by adding “order by" clause to generated SQL.

Keep in mind that using javax.persistence.OrderBy allows us to specify the order of the collection via object properties, meanwhile org.hibernate.annotations.OrderBy order a collection appending directly the fragment of SQL (not HQL) to order by clause.

Now Officer class should not be touched, we don't need to implement compareTo method nor a java.util.Comparator. We only need to annotate officers field with @OrderBy annotation. Since in this case we are ordering by simple attribute, JPA annotation is used to maintain fully compatibility to other “JPA ready” ORM engines. By default ascendent order is assumed.

And if we rerun get all officers method, next queries are sent:

Both queries are still executed but note that now select query contains order by clause too.

With this solution you are saving process time allowing RDBMS sorting data in a fast-way, rather than ordering data in Java once received.

Furthermore OrderBy annotation does not force you to use SortedSet or SortedMap collection. You can use any collection like HashMap, HashSet, or even a Bag, because hibernate will use internally a LinkedHashMap, LinkedHashSet or ArrayList respectively.

In this example we have seen the importance of choosing correctly an order strategy. Whenever possible you should try to take advantage of capabilities of RDBMS, so your first option should be using OrderBy annotaion (hibernate or JPA), instead of Sort. But sometimes OrderBy clause will not be enough. In this case, I recommend you using Sort annotation with custom type (using java.util.Comparator class), instead of relaying on natural order to avoid touching model classes.

I wish this post helped you to understand differences between "sort" and "order" in hibernate.

Keep learning.

Music: http://www.youtube.com/watch?v=VHoT4N43jK8&ob=av3n

Why does the rain fall from above? Why do fools fall in love? Why do they fall in love? (Why Do Fools Fall In Love - Frankie Lymon)

More often than not our applications need to send emails to users notifying for example that its account has been created, they have purchased an item, or simply password remaining. When you are writing unit tests there is no problem because probably you will be mocking up interface responsible of sending an email. But what's happen with integration tests?

Maybe the logical path to resolve this problem is installing an email server and execute these tests against it. It is not  a bad idea, but note that you will need to configure your environment before executing your tests.  Your tests will depend on external resources, and this is a bad idea for integration tests. Furthermore these integration tests would not be portable against multiple machines if an email server is not installed previously.

To avoid this problem Dumbster comes to save us. Dumbster is a fake smtp server designed for testing applications that send email messages. It is written in Java so you can start and stop it directly from your tests.

Let's see an example, suppose we are developing an electronic shop, and when an order is placed and email to customer should be sent.

In this case we are going to use Spring Framework 3.1 to create our service layer and will also help us in testing.

Because of teaching purpose, I am not using mail templates, or rich mime types.

First class I am going to show you is Order, which as you can imagine represents an order:

Most important method here is toEmail() that returns email body message.

Next class is service responsible of place an order to delivery system:

This service class uses Spring classes to send an email to customer. See that two methods are present, one that sends a simple message, and the other one called placeOrderWithInvoice that sends an email with an attachment, concretely an invoice in jpg format.

And finally Spring context file:

Note that mail configuration is surrounded by a profile. This means that Spring will only create these beans when application is started up in production mode, and in this case production smtp location is set.

And now let's start with testing:

First of all we must create a Spring context file to configure smtp server location.

See that we are importing application-context.xml file but now we are defining  a new beans profile called integration, where we are redefining smtp connection (changing hostname and port) pointing to fake server.

And finally the test itself.

It is important to explain next parts:

• @ActiveProfiles is an annotation to tell Spring context which environment should be loaded.
• SimpleSmtpServer is the main class of Dumbster.
• @Rule is responsible of starting and stopping smtp server for each method execution.

We have created two tests one that sends a plain message (an_email_should_be_sent_to_customer_confirming_purchase()) and the other one that sends a message with an attachment (an_email_with_invoice_should_be_sent_to_special_customer_confirming_purchase()).

The private methods are simply helper classes to create required classes.

Note that Hamcrest matcher bodyEqualTo comes from BodySmtpMessage class developed specifically for this example.

I wish you have found this post useful, and can give you an alternative when you want to write integration tests involving smtp email service.

Keep Learning,
Alex.

Download code
Music: http://www.youtube.com/watch?v=2sAHiR0rkJg

Hey! Teachers! Leave them kids alone! All in all it's just another brick in the wall. All in all you're just another brick in the wall. (Another Brick In The Wall - Pink Floyd)

In current post I am going to show you how to configure your application to use slf4j and logback as logger solution.

The Simple Logging Facade For Java (slf4j) is a simple facade for various logging frameworks, like JDK logging (java.util.logging), log4j, or logback. Even it contains a binding tat will delegate all logger operations to another well known logging facade called jakarta commons logging (JCL).

Logback is the successor of log4j logger API, in fact both projects have the same father, but logback offers some advantages over log4j, like better performance and less memory consumption, automatic reloading of configuration files, or filter capabilities, to cite a few features.

Native implementation of slf4j is logback, thus using both as logger framework implies zero memory and computational overhead.

First we are going to add slf4j and logback into pom as dependencies. 

Note that three files are required, one for slf4j, and two for logback. The last two dependencies will change depending on you logging framework, if for example you want to still use log4j, instead of having logback dependencies we would have log4j dependency itself and slf4j-log4j12.

Next step is creating the configuration file. Logback supports two formats of configurations files, the traditional way, using XML or using a Groovy DSL style. Let's start with traditional way, and we are going to create a file called logback.xml into classpath. File name is mandatory, but logback-test.xml is also valid. In case that both files are found in classpath the one ended with -test, will be used.

In general file is quite intuitive, we are defining the appender (the output of log messages), in this case to console, a pattern, and finally root level logger (DEBUG) and a different level logger (INFO) for classes present in foo package. 

Obviously this format is much readable than typical log4j.properties. Recall on additivity attribute, the appender named STDOUT is attached to two loggers, to root and to com.lordofthejars.foo. because the root logger is the ancestor of all loggers, logging request made by com.lordofthejars.foo logger will be output twice. To avoid this behavior you can set additivity attribute to false, and message will be printed only once.

Now let's create to classes which will use slf4j. First class called BarComponent is created on com.lordofthejars.bar:

Note two big differences from log4j. The first one is that is no longer required the typical if construction above each log call.  The other one is a pair of '{}'. Only after evaluating whether to log or not, logback will format the message replacing '{}' with the given string value.

The other one called FooComponent is created on com.lordofthejars.foo:

And now calling foo and bar method, with previous configuration, the output produced will be:

Notice that debug lines in foo method are not shown. This is ok, because we have set to be in this way. 

Next step we are going to take is configuring logback, but instead of using xml approach we are going to use groovy DSL approach. Logback will give preference to groovy configuration over xml configuration, so keep in mind it if you are mixing configuration approaches.

So first thing to do is add groovy as dependency.

And then we are going to create the same configuration created previously but in groovy format.

You can identify the same parameters of xml approach but as groovy functions.

I wish you have found this post useful, and in next project, if you can, use slf4j in conjunction with logback, your application will run faster than logging with log4j.

Keep Learning,

Alex.

Download Code

Music: http://www.youtube.com/watch?v=8vWBMnx92nc&ob=av3e