Deploying JEE Artifacts with Jenkins

Un mondo, Soltanto adesso, io ti guardo, Nel tuo silenzio io mi perdo, E sono niente accanto a te (Il Mondo - Jimmy Fontana)

With the advent of Continuous Integration and Continuous Delivery, our builds are split into different steps creating the deployment pipeline. Some of these steps can be for example compile and run fast tests, run slow tests, run automated acceptance tests, or releasing the application, to cite a few.

The final steps of our deployment pipeline, implies a deployment of our product (in case of JEE project a war or ear) to production-like environment, for UAT or to production system when product is released.

In this post we are going to see how we can configure Jenkins to manage the deployment of a Java Enterprise Application correctly.

First thing to do is creating  the application, in this case a very simple web application in Java (in fact is only one jsp which prints a Hello World!! message) and mavenize it to create a war file (bar.war) when package goal is executed.

Then we need to create a Jenkins job (called bar-web) which is the responsible of compiling, and running unit tests. 

After this job would  come other jobs like running integration tests, running more tests, static code analysis (aka code quality), or uploading artifacts to  artifacts repository but won't be shown here.

And finally, the last steps which imply deploying previous generated code to staging environment (for running User Acceptance Tests for example) and after key users give the ok, deploying to production environment.

So let's see how to create these final steps in Jenkins. Note that binary file created in previous steps  (bar-web in our case) must be used in all these steps. This is because of two reasons, the first one is that your deployment pipeline should be run as fast as possible and obviously compiling in each step the code is not the best way to get it, the second one is that each time you compile your sources, increases the chance of not being compiling sources of previous steps. To achieve this goal we can follow two strategies, the first one is uploading binary files to artifact repository (like Nexus or Artifactory) and get from there in each job. The second one is using copy-artifacts Jenkins plugin to get binary files generated by previous step.

Let's see how to configure Jenkins for the first approach.

Using artifact repository approach, requires that you download the version we want to deploy from repository and then deploy it to external environment; in our case deploying to a web server.  All these steps are done by using maven-cargo-plugin.

Then we only have to create a new Jenkins job, named bar-to-staging, which will run cargo:redeploy Maven goal, and Cargo plugin will be the responsible to deploy bar-web to web server.

This approach has one advantage and one disadvantage.  The main advantage is that you are not bound to Jenkins, you can use Maven alone, or any other CI that supports Maven. The main disadvantage is that relies on artefacts repository, and this plan a new problem, deployment pipeline involves many steps, and between these steps (normally if you are building a snapshot version), a new artefact could be uploaded to artefacts repository with same version, and use it in the middle of pipeline execution. Of course this scenario can be avoided by managing permissions in artefact repository. 

The other approach is use Jenkins plugin, called copy-artifact-plugin. In this case Jenkins acts as an artefact repository, so artifacts created in previous step are used in next step without involving any external repository. Using this approach we cannot use maven-cargo-plugin, but we can use deploy- jenkins-plugin in conjunction with copy-artifacts-plugin. 

So let's see how to implement this approach.

First thing is create a Jenkins build job (bar-web), which creates the war file. Note that two Post-build actions are defined, first one is Archive the artifacts, which is used to store generated files so copy artifacts plugin can copy them to another workspace.  The other one is Build other projects, which in this case, calls a job which is responsible of deploying war file to staging directory (bar deploy-to-staging).

Next thing is create bar deploy-to-staging build job, which main action is deploying war file generated by previous build job to Tomcat server.

For this second build job, you should configure Copy artifacts plugin to copy previous generated files to current workspace, so in Build section, in Copy artifacts from another project section, we set from which build job we want to copy the artifact (in our case bar-web) and which artifacts we want to copy. And finally in Post-build actions section, we must configure which file should be deployed to Tomcat (bar.web), remember that this file is the compiled and packaged by previous build jobs, and finally set Tomcat parameters. And execution pipeline looks something like:

Note that a third build job has been added which deploys war file to production server.

This second approach is the counter part of the first approach, you can be sure that the artefact used in previous step of pipeline will be the one used in all steps, but you are bounded to Jenkins/Hudson.

So if you are going to create a policy in your artefact repository so only pipeline executor can upload artefacts to repository, first approach is better, but if you are not using an external artefact repository (you use Jenkins as is) then second approach is the best one to assure that packaged artefact in previous steps are not modified by parallel steps.

After file is deployed to server, acceptance tests or UAT tests can be executed without any problem.

I wish that now we can address the final steps of our deployment pipeline in a secure and better way.

We keep learning,
Alex.

Music: http://www.youtube.com/watch?v=HFyCfFJC0no

Build Flow Jenkins Plugin

Este samba, Que é misto de maracatú, É samba de preto velho, Samba de preto tu (Mais que Nada - Sergio Mendes)

With the advent of Continuous Integration and Continuous Delivery, our builds are split into different steps creating the deployment pipeline. Some of these steps can be for example compile and run fast tests, run slow tests, run automated acceptance tests, or releasing the application, to cite a few.

Most of us we are using Jenkins/Hudson to implement Continuous Integration/Delivery, and we manage job orchestration combining some Jenkins plugins like build pipeline, parameterized-build, join or downstream-ext. We require configuring all of them which implies polluting the job configuration through multiple jobs, which takes the system configuration very complex to maintain.

Build Flow enables us to define an upper level flow item to manage job orchestration and link up rules, using a dedicated DSL.

Let's see a very simple example:

First step is installing the plugin.

Go to Jenkins -> Manage Jenkins -> Plugin Manager -> Available and find for CloudBees Build Flow plugin.

Then you can go to Jenkins -> New Job and you will see a new kind of job called Build Flow. In this example we are going to name it build-all-yy.

And now you only have to program using flow DSL how this job should orchestrate the other jobs.

In "Define build flow using flow DSL" input text you can specify the sequence of commands to execute.

In current example I have already created two jobs, one executing clean compile goal (yy-compile job name) and the other one executing javadoc goal (yy-javadoc job name). I know that this deployment pipeline is not real in a true environment but for now it is enough. 

Then we want javadoc job running after project is compiled.

To configure this we don't have to create any upstream or downstream actions, simply add next lines at DSL text area:

build("yy-compile");

build("yy-javadoc");

Save and execute build-all-yy job and both projects will be built in a sequential way.

Now suppose that we add a third job called yy-sonar which runs sonar goal that generates code quality sonar report. In this case it seems obvious that after project is compiled, generation of javadocs and code quality jobs can be run in parallel. So script is changed to:

build("yy-compile")

parallel (

    {build("yy-javadoc")},

    {build("yy-sonar")}

)

This plugin also supports more operations like retry (similar behaviour of retry-failed-job plugin) or guard-rescue, that it works mostly like a try+finally block. Also you can create parameterized builds, accessing to build execution or printing to Jenkins console. Next example will print build number of yy-compile job execution:

b = build("yy-compile")

out.println b.build.number

And finally you can also have a quick graphical overview of the execution in Status section. It is true that could be improved more, but for now it is acceptable, and can be used without any problem.

Build Flow plugin is in its early stages, in fact it is only at version 0.4. But will be a plugin to be considered in future, and I think it is good to know that it exists. Moreover is being developed by CloudBees folks so it is a guarantee of being fully supported by Jenkins.

We Keep Learning.

Alex.

Music: https://www.youtube.com/watch?v=ZSZ15R2DFBg

Warning: In order to run parallel tasks with the plugin Anonymous users must have Read Job access (Jenkins -> Manage Jenkins -> Configure System). There is an issue already inserted into Jira to fix this problem.