Adding Chaos on OpenShift cluster

Pumba is a chaos testing tool for Docker and Kubernetes (hence OpenShift too). It allows you to add some chaos on containers instances such as stopping, killing or removing containers randomly. But it also adding some network chaos such as delaying, packet loss or re-ordering.

You can see in next screen recording how to add Pumba on OpenShift cluster and add some chaos on it.

The security calls that you need to run before deploying Pumba are the next ones:

	oc adm policy --as system:admin add-cluster-role-to-user cluster-admin developer
	oc adm policy add-scc-to-user privileged system:serviceaccount:fasttest:default
	oc edit scc restrict

view raw security.sh hosted with ❤ by GitHub

Also the Kubernetes script to deploy Pumba in OpenShift is:

	# If you are running Kubernetes >= 1.1.0. You can take advantage of DaemonSets to automatically deploy the Pumba on all your nodes.
	# On 1.1.x you'll need to explicitly enable the DaemonSets extension, see http://kubernetes.io/v1.1/docs/admin/daemons.html#caveats.
	# You'll then be able to deploy the DaemonSet with the command
	# `kubectl create -f pumba_kube.yaml`
	# If you are not running Kubernetes >= 1.1.0 or do not want to use DaemonSets, you can also run the Pumba as a regular docker container on each node you want to make chaos.
	# `docker run -d -v /var/run/docker.sock:/var/run/docker.sock gaiaadm/pumba pumba --random --interval 3m kill --signal SIGKILL"`
	apiVersion: extensions/v1beta1
	kind: DaemonSet
	metadata:
	name: pumba
	spec:
	template:
	metadata:
	labels:
	app: pumba
	name: pumba
	spec:
	containers:
	- image: gaiaadm/pumba:master
	imagePullPolicy: Always
	name: pumba
	command: ["pumba"]
	args: ["--random", "--debug", "--interval", "30s", "kill", "--signal", "SIGKILL", "re2:.*hello.*"]
	securityContext:
	runAsUser: 0
	volumeMounts:
	- name: dockersocket
	mountPath: /var/run/docker.sock
	volumes:
	- hostPath:
	path: /var/run/docker.sock
	name: dockersocket

view raw pumba_kube.yml hosted with ❤ by GitHub

I have sent a PR to upstream Pumba project with this file but meanwhile is not accepted you can use it this one.

I'd like to say thank you to Slava Semushin and Jorge Morales for helping me on understanding the OpenShift security model.

We keep learning,

Alex.

Ce joli rajolinet, que les oques tonifique, si le fique en une pique, mantindra le pompis net (El baró de Bidet - La Trinca)

Music: https://www.youtube.com/watch?v=4JWIbKGe4gA

Follow me at https://twitter.com/alexsotob

Testing Code that requires a mail server

Almost all applications has one common requirement, they need to send an email notifying something to a registered user. It might be an invoice, a confirmation of an action or a password recovery. How to test this use case might be challenging, using mocks and stubs are ok for unit tests, but having a component test that tests the whole stack. 

In this post I am going to show you how Docker and MailHog, can help you on testing this part of code.

MailHog is  super simple SMTP server for email testing tool for developers:

• Configure your application to use MailHog for SMTP delivery
• View messages in the web UI, or retrieve them with the JSON API
• Optionally release messages to real SMTP servers for delivery
• Docker image with MailHog installed

Notice that since you can retrieve any message sent to mail server using JSON API, it makes really simple to validate if the message has been really delivered and of course assert on any of message fields.

Arquillian Cube is an Arquillian extension that can be used to manager Docker containers in your tests. To use it you need a Docker daemon running on a computer (it can be local or not), but probably it will be at local.

Arquillian Cube offers three different ways to define container(s):

• Defining a docker-compose file.
• Defining a Container Object.
• Using Container Object DSL.

In this example I am going to show you Container Object DSL approach, but any of the others works as well.

To use Container Object DSL you simply need to instantiate the ContainerDslRule (in case you are using JUnit Rules) or use Arquillian runner in case of using JUnit, TestNG or Spock. You can read more about Container Object DSL at http://arquillian.org/arquillian-cube/#_arquillian_cube_and_container_object_dsl

As an example of definition of Redis container:

	@ClassRule
	public static ContainerDslRule redis = new ContainerDslRule("redis:3.2.6")
	.withPortBinding(6379);
	Jedis jedis = new Jedis(redis.getIpAddress(), redis.getBindPort(6379));
	jedis.set("foo", "bar");

view raw RedisDsl.java hosted with ❤ by GitHub

When running this test, Redis Docker image is started, tests are executed and finally the Docker instance is stopped.

So let's see how to do the same but instead of Redis using Mailhog Docker image.

It is important to notice that ContainerDslRule is a generic class that  can be extended to become more specific to a concrete use case. And this is what we are going to do for Mailhog.

	public class MailhogContainer extends ContainerDslRule {
	protected MailhogContainer(String image) {
	super(image);
	}
	public static MailhogContainer createDefaultMailhogContainer() {
	MailhogContainer mailhogContainer = new MailhogContainer("mailhog/mailhog:v1.0.0");
	mailhogContainer.withPortBinding(1025, 8025);
	return mailhogContainer;
	}
	public void should_receive_email_with_subject(String subject) {
	final ArrayList<String> expected = new ArrayList<>();
	expected.add(subject);
	RestAssured.given()
	.when()
	.get("http://" + this.getIpAddress() + ":" + this.getBindPort(8025) + "/api/v1/messages")
	.then()
	.assertThat()
	.body("Content.Headers.Subject", CoreMatchers.hasItems(expected));
	}
	}

view raw MailhogContainer.java hosted with ❤ by GitHub

First of all we need to create a class extending from ContainerDslRule, so everything is still a JUnit rule, but a customized one. Then we create a factory method which creates the MailhogContainer object, setting the image name and the binding ports. Finally an assertion method is used to connect to Rest API of Mailhog server to check if there is any message with given subject.

Then we can write a test using this new rule.

	public class MailServiceTest {
	@ClassRule
	public static MailhogContainer mailhog = MailhogContainer.createDefaultMailhogContainer();
	@Test
	public void should_send_an_email() throws Exception {
	// given
	final MailService mailService = new MailService(
	mailhog.getIpAddress(),
	mailhog.getBindPort(1025));
	// when
	mailService.send("lollypop", "lolly.pop@mymail.com", "Invoice: 1234",
	"Thank you very much for buying this product. Here's your invoice");
	// then
	mailhog.should_receive_email_with_subject("Invoice: 1234");
	}
	}

view raw MailServerTest.java hosted with ❤ by GitHub

This test just configures MailService class with Docker container configuration, it sends an email and finally it delegates to container object to validate if the email has been received or not.

Notice that putting everything into an object makes this object reusable in other tests and even in other projects. You can create an independent project with all your custom developed container objects and just reuse them importing them as test jar in hosted project.

Code: https://github.com/lordofthejars/mailtest

We keep learning,
Alex.

'Cause I'm kind of like Han Solo always stroking my own wookie, I'm the root of all that's evil yeah but you can call me cookie (Fire Water Burn - Bloodhound Gang)

Music: https://www.youtube.com/watch?v=Adgx9wt63NY

Follow me at https://twitter.com/alexsotob