Nasha nasha krovatka delala shik - shik, Ya tvo pianino , a ty moi nastroishchik, My tak letali chto ne zametili tvoyu matʹ, Ahaa..I ona skazala chto ya prosto blaz (Mama Lyuba - Serebro))

When we develop any application, after we finish it or when we end up any of its module, we start optimization process. Most applications contain database access, and if you are using an ORM, probably you will use hibernate. Optimizing hibernate persistence layer, requires to be prepared to read, understand and evaluate sql.

If we take an overview on hibernate configuration, two properties, hibernate.format_sql and hibernate.use_sql_comments, should be enabled to print performed sql code through console.

This is a good start but it seems that we need more information to make an accurate diagnosis of performance like connection events, returned data on queries, or parameters binding (hibernate shows parameters values with question mark ?) . Hence, we need another way to inspect generated sql. Log4jdbc is a jdbc driver that can log sql/jdbc calls. In fact log4jdbc is an implementation of proxy pattern which will automatically load popular jdbc drivers (Oracle, Derby, MySql, PostgreSql, H2, Hsqldb, ...), intercept calls, log information, and then send data to "spied" driver.

In log4jdbc, there are 5 loggers that can be used depending on data to monitor:

• jdbc.sqlonly: logs executed sql with binding arguments replaced with bound data.
• jdbc.sqltiming: logs how long a sql took to execute.
• jdbc.audit: logs all jdbc calls except for ResultSets.
• jdbc.resultset: same as jdbc.audit plus ResultsSets.
• jdbc.connection: logs open and close connection events.

In this post we are going to see how to configure log4jdbc-remix, a fork of log4jdbc, which apart from inheriting log4jdbc capabilities, also let us:

• jdbc.resultsettable: log results set in table format.
• configure it as datasource.
• available in maven repository (log4jdbc is not present on maven repositories).

For this example we are going to use the project created by JPA Spring Template which contains two entities Order and Item associated with one-to-many relationship, and one test that executes some database operations.

First thing to do is add log4jdb-remix and slf4j-log4j12 dependencies to project:

Next thing to do is configure active loggers.  Depending on the data we are interested to monitor, we activate the required loggers. As an example let's configure log4j.xml so result set is printed in table format and also time taken to execute each query is shown.

After configuring loggers, run test and inspect the output.


Output is printed in a fashion format, queries contains bind parameters (not a question mark (?)), and process time is also informed.

Notice that logging more or less information is simply a matter of configuring a log. Moreover depending on log level, more or less information will be provided in each case. If logger is configured in DEBUG class name and line number (if available) at which the sql was executed will be included. In INFO will include only sql, and finally ERROR which show stacktraces if any SQLException occurs.

Optimizing hibernate applications can imply touching many parts of an application (JVM configuration, database engine, network, …) but one very important aspect to take care is the number of queries that are sent to RDBMS (for example N+1 problem), and the amount of data that is retrieved from database (Projection problem), and log4jdbc-remix perfectly fits to help in this purpose.

As final note, log4jdbc(-remix) is a jdbc logger, so it is not necessary to use only in hibernate applications, can be used with any framework that uses a datasource.

I wish this library would help you.

Keep Learning,
Alex

Download Code
Music: http://www.youtube.com/watch?v=h9HRHOXfRBI

Qui dit crise te dis monde dit famine dit tiers- monde, Qui dit fatigue dit réveille encore sourd de la veille, Alors on sort pour oublier tous les problèmes, Alors on danse... (Alors on Danse - Stromae)

Let's introduce another hibernate performance tip. Do you remember the model of previous hibernate post? We had a starship and officer related with a one to many association.

Now we have next requirement:

We shall get all officers assigned to a starship by alphabetical order.

To solve this requirement we can:

1. implementing an HQL query with order by clause.
2. using sort approach.
3. using order approach.

The first solution is good in terms of performance, but implies more work as a developers because we should write a query finding all officers of given starship ordered by name and then create a finder method in DAO layer (in case you are using DAO pattern).

Let's explore the second solution,  we could use SortedSet class as association, and make Officer implements Comparable, so Officer has natural order. This solution implies less work than the first one, but requires using @Sort hibernate annotation on association definition. So let's going to modify previous model to meet our new requirement. Note that there is no equivalent annotation in JPA specification.

First we are going to implement Comparable interface in Officer class.

We are ordering officer by name by simply comparing name field. Next step is annotating association with @Sort.

Notice that now officers association is implemented using SortedSet instead of a List.   Furthermore we are adding @Sort annotation to relationship, stating that officers should be natural ordered. Before finishing this post we will insist more in @Sort topic, but for now it is sufficient.

And finally a method that gets all officers of given starship ordered by name, printing them in log file.

All officers are sorted by their names, but let's examine which queries are sent to RDBMS.

First query is resulting of calling find method on EntityManager instance finding starship.

Because one to many relationships are lazy by default when we call getOfficers method and we access first time to SortedSet, second query is executed to retrieve all officers. See that no order by clause is present on query, but looking carefully on output, officers are retrieved in alphabetical order.

So who is sorting officer entities? The explanation is on @Sort annotation. In hibernate a sorted collection is sorted in memory being Java the responsible of sorting data using compareTo method.

Obviously this method is not the best performance-way to sort a collection of elements. It is likely that we'll need a hybrid solution between using SQL clause and using annotation instead of writing a query.

And this leads us to explain the third possibility, using ordering approach.

@OrderBy annotation, available as hibernate annotation and JPA annotation, let us specifies how to order a collection by adding “order by" clause to generated SQL.

Keep in mind that using javax.persistence.OrderBy allows us to specify the order of the collection via object properties, meanwhile org.hibernate.annotations.OrderBy order a collection appending directly the fragment of SQL (not HQL) to order by clause.

Now Officer class should not be touched, we don't need to implement compareTo method nor a java.util.Comparator. We only need to annotate officers field with @OrderBy annotation. Since in this case we are ordering by simple attribute, JPA annotation is used to maintain fully compatibility to other “JPA ready” ORM engines. By default ascendent order is assumed.

And if we rerun get all officers method, next queries are sent:

Both queries are still executed but note that now select query contains order by clause too.

With this solution you are saving process time allowing RDBMS sorting data in a fast-way, rather than ordering data in Java once received.

Furthermore OrderBy annotation does not force you to use SortedSet or SortedMap collection. You can use any collection like HashMap, HashSet, or even a Bag, because hibernate will use internally a LinkedHashMap, LinkedHashSet or ArrayList respectively.

In this example we have seen the importance of choosing correctly an order strategy. Whenever possible you should try to take advantage of capabilities of RDBMS, so your first option should be using OrderBy annotaion (hibernate or JPA), instead of Sort. But sometimes OrderBy clause will not be enough. In this case, I recommend you using Sort annotation with custom type (using java.util.Comparator class), instead of relaying on natural order to avoid touching model classes.

I wish this post helped you to understand differences between "sort" and "order" in hibernate.

Keep learning.

Music: http://www.youtube.com/watch?v=VHoT4N43jK8&ob=av3n

Moi je pense à l'enfant, Entouré de soldats, Moi je pense à l'enfant, Qui demande pourquoi (Non Non Rien N'a Changé - Les Poppys)

After 8 years developing server and embedded applications using Hibernate as ORM, squeezing my brain seeking solutions to improve Hibernate performance, reading blogs and attending conferences, I decided to share this knowledge acquired during these years with you.

This is the first post of many more posts to come:

Last year I went to Devoxx as speaker but also I attended Patrycja Wegrzynowicz conference about Hibernate Anti-Patterns. In that presentation Patrycja shows us an anti-pattern that shocks me because it proved to expect the unexpected.

We are going to see the effect it has when Hibernate detects a dirty collection and should re-create it.

Let's start with the model we are going to use, only two classes related with one-to-many association:

In previous classes, we should pay attention in three important points:

• we are annotating at property level instead of field level.
• @OneToMany and @ManyToOne uses default options (apart from cascade definition)
• officers getter on Starship class returns an immutable list. 

To test model configuration, we are going to create a test which creates and persists one Starship and seven Officers, and in different Transaction and EntityManager finds created Starship.

Now that we have created this test, we can run it and we are going to observe Hibernate console output.

See the number of queries executed during first commit (persisting objects) and during commit of second transaction (finding a Starship). In total and ignoring sequence generator, we can count 22 inserts, 2 selects and 1 delete, not bad when we are only creating 8 objects and 1 find by primary key.

At this point let's examine why these SQL queries are executed:

First eight inserts are unavoidable; they are required by inserting data into database.

Next seven inserts are required because we have annotated getOfficers property without mappedBy attribute. If we look closely at Hibernate documentation, it points us that “Without describing any physical mapping, a unidirectional one to many with join table is used.”

Next group of queries are even stranger, the first select statement is to find Starship by id, but what are these deletes and inserts of data that we have already created?

During commit Hibernate validates whether collection properties are dirty by comparing object references. When a collection is marked as dirty, Hibernate needs to re-create whole collection, even containing the same objects. In our case when we are getting officers we are returning a different collection instance, concretely an unmodifiable list, so Hibernate considers officers collection as dirty.

Because a join table is used, Starship_Officer table should be re-created, deleting previous inserted tuples and inserting the new ones (although they have the same values).

Let's try to fix this problem. We start by mapping a bidirectional one-to-many association, with many-to-one side as owning side.

And now we rerun the same test again and we inspect the output again.

Although we have reduced the number of SQL statements, from 25 to 10, we still have an unnecessary query, the ones just in commit section of second transaction. Why if officers are lazy by default (JPA specification), and we are not getting officers in transaction, Hibernate executes a select on Officers table?  By the same reason as previously configuration, returned collection has different Java identifier, so Hibernate marks it as newly instantiated collection, but now obviously join table operations are no longer required. We have reduced the number of queries but we still have a performance problem. It is likely that we'll need some other solution, and the solution is not the most obvious one, we are not going to return collection objects returned by Hibernate, we might expand on this later, but we are going to change annotations location.

What we are going to do is to change mapping location from property approach to use field mapping. Simply we are going to move all annotations to class attributes rather than on getters.

And finally we are going to run the test again, and see what's happen:

Why using property mapping Hibernate runs queries during commit and using field mapping are not executed? When a Transaction is committed, Hibernate execute a flush to  synchronize the underlying persistent store with persistable state held in memory. When property mapping is used, Hibernate calls getter/setter methods to synchronize data, and in case of getOfficers method, it returns a dirty collection (because of unmodifiableList call). On the other side when we are using field mapping, Hibernate gets directly the field, so collection is not considered dirty and no re-creation is required.

But we have not finished yet, I suppose you are wondering why we have not removed Collections.unmodifiableList from getter, returning Hibernate collection? Yes I agree with you that we finished quickly, and change would look like @OneToMany(cascade={CascadeType.ALL}) public List<Officer> getOfficers() {officers;} but returning original collection ends up with an encapsulation problem, in fact we are broken encapsulation!. We could add to mutable list anything we like; we could apply uncontrolled changes to the internal state of an object.

Using an  unmodifiableList is an approach to use to avoid breaking encapsulation, but of course we could have used different accessors for public access and hibernate access, and not calling  Collections.unmodifiableList method.

Considering what we have seen today, I suggest you to use always field annotations instead of property mapping, we are going to save from a plenty of surprises.

Hope you have found this post useful.

Screencast of example shown here:



Download code
Music: http://www.youtube.com/watch?v=H14VIsnr6aA

Giuro per sempre a te, Di viver, morire per te, Se tu sarai con me lo so, Dea Roma, vincerò!

One of the common problems of people that start using Hibernate is performance, if you don't have much experience in Hibernate you will find how quickly your application becomes slow. If you enable sql traces, you would see how many queries are sent to database that can be avoided with little Hibernate knowledge. In current post I am going to explain how to use Hibernate Query Cache to avoid amount of traffic between your application and database.

Hibernate offers two caching levels:

• The first level cache is the session cache. Objects are cached within the current session and they are only alive until the session is closed.
• The second level cache exists as long as the session factory is alive. Keep in mind that in case of Hibernate, second level cache is not a tree of objects; object instances are not cached, instead it stores attribute values.

After this brief introduction (so brief I know) about Hibernate cache, let's see what is Query Cache and how is interrelated with second level cache.

Query Cache is responsible for caching the combination of query and values provided as parameters as key, and list of identifiers of objects returned by query execution as values. Note that using Query Cache requires a second level cache too because when query result is get from cache (that is a list of identifiers), Hibernate will load objects using cached identifiers from second level.

To sum up, and as a conceptual schema, given next query: "from Country where population > :number", after first execution, Hibernate caches would contain next fictional values (note that number parameter is set to 1000):

L2 Cache
[
id:1, {name='Spain', population=1000, ....}
id:2, {name='Germany', population=2000,...}
....
]
QueryCache
[{from Country where population > :number, 1000}, {id:2}]

So before start using Query Cache, we need to configure cache of second level.
First of all you must decide what cache provider you are going to use. For this example Ehcache is chosen, but refer to Hibernate documentation for complete list of all supported providers.

To configure second level cache, set next Hibernate properties:

hibernate.cache.provider_class = org.hibernate.cache.EhCacheProvider
hibernate.cache.use_structured_entries = true
hibernate.cache.use_second_level_cache = true

And if you are using annotation approach, annotate cachable entities with:

@Cacheable
@Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)

See that in this case cache concurrency strategy is NONSTRICT_READ_WRITE, but depending on cache provider, other strategies can be followed like TRANSACTIONAL, READ_ONLY, ... take a look at cache section of Hibernate documentation to chose the one that fits better with your requirements.

And finally add Ehcache dependencies:

<dependency>
<groupId>net.sf.ehcache</groupId>
<artifactId>ehcache-core</artifactId>
<version>2.5.0</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-ehcache</artifactId>
<version>3.6.0.Final</version>
</dependency>

Now second level cache is configured, but not query cache; anyway we are not far from our goal.

Set hibernate.cache.use_query_cache property to true.

And for each cachable query, we must call setCachable method during query creation:

List<Country> list = session.createQuery("from Country where population > 1000").setCacheable(true).list();

To make example more practical I have uploaded a full query cache example with Spring Framework. To see clearly that query cache works I have used one public database hosted in ensembl.org. The Ensembl project produces genome databases for vertebrates and other eukaryotic species, and makes this information freely available online. In this example query to dna table is cached.

First of all Hibernate configuration:

It is a simple Hibernate configuration, using properties previously explained to configure second level cache.

Entity class is an entity that represents a sequence of DNA.

To try query cache, we are going to implement one test where same query is executed multiple times.

We can see that we are returning first fifty dna sequences, and if you execute it, you will see that elapsed time between creation of query and commiting transaction is printed. As you can suppose only first iteration takes about 5 seconds to get all data, but the other ones only milliseconds.

The foreach line just before query iteration will print object identifier through console. If you look carefully none of these identifiers will be repeated during all execution. This fact just goes to show you that Hibernate cache does not save objects but properties values, and the object itself is created each time.

Last note, remember that Hibernate does not cache associations by default.

Now after writing a query, think if it will contain static data and if it will be executed often. If it is the case, query cache is your friend to make Hibernate applications run faster.


Download Code

Music: http://www.youtube.com/watch?v=fw1VJSU92mw