Are Configuration Files Documentation? ... Yes!!! TomEE + Asciidoctor Example

Normally in our projects we generate a considerable amount of documentation. Some documentation is directly a document written as is, for example Software Requirements Specifications (SRS) or Software Design Specifications (SDS), but there are other pieces of our projects that although they are not a document directly, they are consulted by developers/testers/managers, customers ... and treat it as is, for example user stories written by developers and/or QA in Cucumber or JBehave format or  javadoc. And now I ask to myself, are configuration files documentation too? Are people interested in having them in a document and treat them as documentation? And the answer is YES!!!. In this post we are going to see how to transform TomEE's configuration file into a document using an Asciidoctor extension I have developed specially for this occasion.

In TomEE, resources and containers can be configured inside a file called tomee.xml among other places. But what really makes interesting is that a subset of this file can be put inside the project, so each web application is responsible of configuring their own resources, for example DataSources, JMS Queues, ....  And having the ability of putting specific configuration of the server inside the application also opens the doors of creating more than one configuration file per application, and load one or another depending on the environment we are going to deploy the application. For example:

• In developer environment the DataSource could point to an embedded database.
• In QA environment the DataSource could point to a database installed on localhost.
• In Stage environment the same DataSource points to an external IP1.
• In Production environment the same DataSource points to an external IP2.

So as you can see, it would be interesting to have all these information as readable documentation, so everyone involved in the project could review it and for example know exactly which servers are being used in each environment as datasources, or configuration parameters like pool size of stateless manager.

And this is exactly what tomee-asciidoctor-extension does. It allows us to embed Apache TomEE configuration files into an AsciiDoc documentation.

Let's see an example of Apache TomEE configuration file:

This is a typical configuration file where for example you can see some DataSources defined, custom resource or definition of TransactionManager.

Probably your projects will have a document called for example infrastructure (or something similar) where all information related with the software that is being used in the project and  environments used will be described.

A simple example of these kind of documents can be:

Note that besides describing the software that is going to be used for solving our problem, in this case an Apache TomEE, we are using the tomee-asciidoctor-extension to add the Apache TomEE's container and resources configuration file, so when someone wants to inspect the documentation will also see exactly the configuration values that had Apache TomEE at time of generating the document. If you look carefully you only need to add next line to transform a Apache TomEE configuration file into documentation:

include::tomee:/home/alex/git/asciidoctorj-extensions/tomee-asciidoctorj-extensions/src/test/resources/resources.xml[initialLevel=2]

In this case we are including a file called resources.xml. Note that in order to extension detects an Apache TomEE configuration file, we need to use the special keyword tomee: followed by the location of the file. Also note that we are using an attribute called initialLevel. Because we are embedding one document into another one, we need a way to set at which level the document will be rendered, and this is what this attribute is designed for.

Previous document rendered with default style sheet looks like:

See that thanks of Asciidoctor and its extension system, we are able to convert configuration files to documentation.

You can see the code of this extension at: https://github.com/lordofthejars/asciidoctorj-extensions/tree/master/tomee-asciidoctorj-extensions

We keep learning,
Alex.

Siente y baila y goza, Que la vida es una sola, Voy a reír, voy a bailar, Vive, sigue, Siempre pa'lante (Vivir Mi Mida - Marc Anthony)

Music: http://www.youtube.com/watch?v=YXnjy5YlDwk

Basta Imaginar e Ele Está Partindo, Sereno e Lindo E Se a Gente Quiser....... Ele Vai Pousar.

This week Springsource people have uploaded the first milestone of Spring Framework 3.1.

For my diary work, Environment Abstraction is the best feature developed in current milestone.

Environment Abstraction allows us grouping bean definitions for activation in specific environments. For example you can define two data sources, where one will be used in testing (HSQL configuration) and the other one in production (PostgreSQL). I am sure that in your life using Spring, you have had uploaded by mistake the test database configuration into production code. Environment Abstraction tries to avoid this situation.

Now I have showed you a typical example, but you can also create a custom resolution of place-holders, so for example in test mode you use a properties file with log level property configured to TRACE, and in production the properties file configured to INFO.

Another example that I find it interesting is in acceptance tests. One of my tasks is design and develop planners for robots. In Unit Testing we are mocking the hardware interface with the robot. But in acceptance tests, instead of mocking the hardware interface, we inject an emulator interface, so you can see how a fictional software robot is moving in your screen and validate that all movements are correct visually.

In previous Spring versions, what we have is three Spring XML files, one for production, another one for acceptance tests, and one main file that imports one of both files, so depending on environment we import one file or another. As you can imagine this implies a manual process between production environment and testing environment; although is not tedious, can be omitted so having emulator hardware interface in production instead of driver interface to robot.

With Spring 3.1, only one file is required with both configurations, and depending on environment Spring loads the required beans.

An example of what I am explaining is:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.1.xsd">
     <beans profile="test, integration">
          <bean .../>
          <bean .../>
          <context:property-placeholder .../>
     </beans>
     <beans profile="production">
          <bean .../>
          <bean .../>
          <context:property-placeholder .../>
     </beans>
     <beans>
          <bean ..../>
     </beans>
</beans>

First thing you will notice is that beans tag is still the root but also is self-nested. The first two nested beans have an attribute called profile. This attribute inform when the beans it contains are active. First beans are active when configured profile is test or integration, while second ones when production profile is active. The last one definition is always activated.

Continuously we are writing something like "depending on environment", and I suppose you are wondering how you specify that environment.

You have an environment variable called spring.profiles.active where you specify which is the current profile. For setting this variable you can use an export in case of *nix systems, or as parameter execution -Dspring.profiles.active, or in context.xml in case of web applications, ...

For example a valid environment definition should be:

export spring.profiles.active=test

In case you are using Java-based Container Configuration, @Profile("profileName") can be used in beans definition, for specifying profile name.

ApplicationContext has getEnvironment() method for specifying which profiles are activated.

AnnotationConfigApplicationContext ctx = ...
ctx.getEnvironment().setActiveProfiles("dev");
ctx.register(...)
ctx.refresh();

I have developed a simple example that have two classes, where one prints passed message in upper case, and the other one prints the message between sharp character. Depending on profile configuration it changes message format. Moreover the same example is provided but using Java-based Container Configuration.

Download code from my GITHub if you want to see the source code.

I hope you find this new feature as useful as I find, and more important, once you have configured your machines with required environment variable, you won't worry any more about changing configuration files.

Kurikaesu Ayamachi No Sonotabi Hito Wa Tada Aoi Sora No Aosawo Shiru

Maven 3 presents a new way to configure POM files. In Maven 2 POM files have to be configured using a XML file. In Maven 3 can be used the old way, but also as Groovy file. Groovy is a nice script language that offers functions for implementing a DSL (Domain Specific Language).

Let's look an example of Maven 3 POM file:

XML file:

<dependencies>
     <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>4.7</version>
          <scope>test</scope>
    </dependency>
</dependencies>

can be translated to:

Groovy file:

dependencies {
      dependency { groupId 'junit'; artifactId 'junit'; version '4.7'; scope 'test' }
}

As you have noted Groovy file is as understandable as XML file, but much more readable, only two lines instead of 6.

After watch this new form, I got curious about how Maven people have created the parser with Groovy. In fact all is reduced to Closures, and Dynamic Invocation. Both capabilities are implemented by Groovy. So my next step was implementing some XML configuration as Groovy configuration and develop the "parser" in Groovy.

First of all I have chosen Liquibase configuration file, I have trunked a lot (for making the example simple).

Then I have implemented how I wanted to look the Groovy configuration file:

XML configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.9"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd">
     <createTable tableName="department">
          <column name="id" type="int"></column>
          <column name="name" type="string"></column>
     </createTable>
</databaseChangeLog>

If you look original XML configuration you will notice that I have removed some tags. As I have said, this reduction has been done for simplifying developing.

And Groovy configuration file:

databaseChangeLog {
     createTable {
          id {type "int"}
         name {type "string"}
     }
}

Nice reduction from 9 lines to 6 lines in so simple example.

Now it is time for some Groovy theory.

In Groovy a Closure is composed by a definition (name), and arguments and statements to execute. In formal way:

name { [closureArguments->] statements }

if you look close to Groovy configuration file, this can be seen in first statement:

databaseChangeLog {
....
}

you have a closure name, a brace, no arguments, and a list of statements.

In Groovy a function can be called like in Java functionName(param1, param2, ...); but also without parentheses.

An example can be found in line 3:

id {type "int"}

the function call is named type and a string parameter is passed with value int.

We have almost done, we know that databaseChangeLog and createTable are closures name, as is id and name.

And I suppose you are wondering, "you are right I can see closures, I can see functions, but I need to read configured values". Yes you are right and this is a Groovy class that resides in another Script file.

With this information we have almost done, let me show the implementation and comment the last trick.

First of all we need to define a ExpandoMetaClass (that allows you to dynamically add methods, constructors, properties and static methods using a neat closure syntax) for matching the parent method, in our case databaseChangeLog.

script.metaClass = createEMC(script.class, {
     ExpandoMetaClass emc ->
     emc.databaseChangeLog = {
     Closure cl ->
     cl.delegate = new DatabaseChangeLogDelegate();
     cl.resolveStrategy = Closure.DELEGATE_FIRST
     cl()
   }
})

static ExpandoMetaClass createEMC(Class scriptClass, Closure cl) {
     ExpandoMetaClass emc = new ExpandoMetaClass(scriptClass, false)
     cl(emc)
     emc.initialize()
     return emc
}

This class have two important points.

First one, emc.databaseChangeLog.

Here we are telling that exists a method databaseChangeLog.

Second one:

cl.delegate = new DatabaseChangeLogDelegate();

In this line we are telling to Groovy that statements (method calls) defined into databaseChangeLog closure should be delegated to specific delegation class and after this delegation (Closure.DELEGATE_FIRST) the closure (method content) should be executed (cl()).

And delegate class is:

class DatabaseChangeLogDelegate {
     void createTable(Closure cl) {
           cl.delegate = new AttributeNameDelegate();
           cl.resolveStrategy = Closure.DELEGATE_FIRST;
           println "Creating Table";
           cl();
     }
}

DatabaseChangeLogDelegate delagate defines that when createTable method is found should create a new delegate, print that A Table is Created and finally calls the closure.

So recapitulate what Groovy executes after all initialization completes, or more formaly when script.run() (this runs our configuration Groovy script) is executed.

The script main method is executed (databaseChangeLog), this method is found into ExpandoMetaClass and what Groovy do is create the new delegation and calls databaseChangeLog. This method have into its body another method called (createTable). And tries to execute, how? finding them into the delegation. Look DatabaseChangeLogDelegate class, it contains a createTable method. This method creates a new delegation, cl.delegate = new AttributeNameDelegate();, for next methods and it is executed (cl()) which implies finding two methods, id and name.

I know it is a little bit confusing, think about delegation class like an @Around aspect in Spring AOP, that can execute some logic before and after, and instead of having proceed call have a call to closure (cl());

And now the last trick, you can think, yep man, you are redefining a methods that you know their name like databaseChangeLog, createTable or type. But what happens with methods that are defined by user and can be any valid function name like id or name, each table will create their own attribute names, so how to deal with methods that we don't know what are their names in developing time?

Groovy defines a method in delegates called methodMissing. This method is called when in current delegate class, the method definition is not found.

class AttributeNameDelegate {
     def methodMissing(String name, Object args) {
           if (args.length == 1) {
                   if (args[0] instanceof Closure) {
                        args[0].delegate = new AttributeTypeDelegate();
                        args[0].resolveStrategy = Closure.DELEGATE_FIRST;
                        println name
                       args[0]()
                  }
          }else {
                 throw new MissingMethodException(name, this.class, args as Object[])
          }
    }
}

As you can see, the same approach is used, but before calling body method the name of the function is printed and closure is called (first argument).

Final code is:

package liquibase
import groovy.lang.Closure;
import groovy.lang.GroovyShell;
import groovy.lang.Script;

class GroovyLiquibase {
     static void main(String[] args) {
          runLiquidParser();
     }

static void runLiquidParser() {
     runLiquidParser new File("test.groovy");
}

static void runLiquidParser(File file) {
     Script script = new GroovyShell().parse(file);
     script.metaClass = createEMC(script.class, {
           ExpandoMetaClass emc ->
           emc.databaseChangeLog = {
                   Closure cl ->
                   cl.delegate = new DatabaseChangeLogDelegate();
                   cl.resolveStrategy = Closure.DELEGATE_FIRST
                   cl()
          }
     })
     script.run()
}

static ExpandoMetaClass createEMC(Class scriptClass, Closure cl) {
      ExpandoMetaClass emc = new ExpandoMetaClass(scriptClass, false)
      cl(emc)
      emc.initialize()
      return emc
     }
}

class DatabaseChangeLogDelegate {
      void createTable(Closure cl) {
          cl.delegate = new AttributeNameDelegate();
          cl.resolveStrategy = Closure.DELEGATE_FIRST;
          cl();
      }
}

class AttributeNameDelegate {
      def methodMissing(String name, Object args) {
           if (args.length == 1) {
               if (args[0] instanceof Closure) {
                    args[0].delegate = new AttributeTypeDelegate();
                    args[0].resolveStrategy = Closure.DELEGATE_FIRST;
                    println name
                   args[0]()
              }
          }else {
                  throw new MissingMethodException(name, this.class, args as Object[])
         }
      }
}

class AttributeTypeDelegate {
      void type(String type) {
            println type
      }
}

First look seems very complicated code, but you will find that is easy to develop, and repeatable, after you have defined one Delegate are all exactly equals but changing required method names and configuration.

Some of the advantages of DSL approach for configuration files are:

• Less information for expressing the same. Our new configuration file for Liquibase is essentially an XML file but without the noise generated by the XML tags.
• More concise and more readable.
• Enhance extensibility. Because it is based on script language, can be extended with script statements. For example, in case of Maven, if you wanted to execute some Groovy statements during an specific goal, you should import a plugin (GMaven) and configure it. See GMaven website for examples for watching why this feature can be interesting http://docs.codehaus.org/display/GMAVEN/Executing+Groovy+Code. But if your configuration file is already a Groovy file you can put Groovy statements natively, without adding any new plugin or configure it.

With Groovy you can create your own Doman-Specific Language. What I suggest is that you start thinking about creating a DSL for configuration file when you are writing a submodule that will be reused in many other projects, and in each one different configuration will be required.

Of course developing a DSL can be used in different situations rather than defined here. Some of you could think that XML files is the best way for creating configuration files, but I think is always nice to know different valid possibilities, and what offer.

Download Source Code