Automated software release in complex environments

Automating software release might seem like an impossible task in a complex environment. I would like to give you, my dear reader handy tips and tricks for achieving it.

Software journey

Either you know it or not, the piece of software that enables you to read this article went a long journey. It was invented, thought through, designed, developed, tested, fixed and then delivered to you. Software continues to evolve. New functionality is invented, designed, developed, tested, fixed and delivered to you again. This cycle happens over and over again. If it doesn’t the software is dead, and I wouldn’t use it if I were you J

Note the last cycle in the software journey: delivery. I like to call this process Software Release or Unleashing the Software. Release depends on many factors. One of them is software purpose. It is enough to make desktop application available (via web site download for example) so the user can get it and install it on a desktop. Web applications are delivered to a web server.

Release process

Software Release process could be simple or could be very complicated. Lets examine three examples:

1. Desktop application release could involve copying (perhaps extracting) number of files into known location and perhaps making some changes to saved files (for example user files with configuration and settings in user directory).
2. Web application release could involve pushing new version of the application to web servers, restarting them, migrating the data stored in database.
3. Multi tiered enterprise applications could involve stopping a number of services, waiting for suitable services state (for example empty EMS service), migrating the data schemas (for example database migration), re-starting multiple applications, restoring application state, etc.

As you can see complexity of the process could be great. Some steps in the process could depend on another. Other steps could take great amount of time. Sometimes it is impossible to stop part of the application or a service for required release.

Whatever the problematic, time consuming and fragile steps of the release process are, you can be sure that sooner or later, someone will make a mistake during the release, causing downtime or even damage to a system.

That is why it is important to have automated release.

Automated release

My favorite form of automated release is One-Click-Deploy. A tool (script most likely) that will perform all the hard work, with a single user interaction.

Some of the benefits of release automation like this are:

• Reduced mistakes when performing manual steps
• No time wasted by a poor soul who have to go through manual process
• Reusability and simplicity of using the same tool to release into different environments
• A tool that verifies the release process itself (if something went wrong, it probably means that process needs updating)

Complex environments

Automating in complex environment is even more important then anywhere else. This is due to the fact that it is much simpler to make a mistake while releasing and cause problems. It also reduced the great amount of time consumed by manual steps.

It might be hard to automate release in complex systems. Consider the system I work on my current project.

Large distributed cache makes it difficult to stop the application, as data will be lost. EMS topics are read by database persistence application and fail-safe environment, so we need to wait for the topics to be read entirely. Once everything is stopped we need to roll the new code base to 20 – 40 physical hosts and apply all necessary configurations changes. Persistence layer needs to be migrated to represent Domain correctly. When we restart applications we need to reload it to the previous, valid state (load 300 GB worth of data into distributed cache). That is only a tip of the Iceberg that represents how difficult it could be to automate the release.

There are things that you can do or adopt on a project that would make it much easier for the automation to happen.

Tips and tricks

Tools and deployment environments

Use tools and deployment environments that have accessible API for management.

The environment or the containers that you deploy to should have open API for the management. If you are deploying into web server, you would like to be able to programmatically stop the server deploy new code, perform configuration changes, start it again and validate.

Most of the cloud services contain this kind of API. Typical Java Web containers are manageable via a public API.

If there is no public API but there is a web console for management it is possible to automate via typical Web Testing tool (eg: Selenium, Geb).

If you are starting the project and still have a chance to select appropriate environment, then you are lucky. You can make your life easier by selecting environment with public management API. If you are deep into the project and it is too late, there are always a ways to hack the system. Give it a try.

Use a tool that will be flexible and won’t enforce specific way of working.

Number of times I was on a project where the tool used for a release was enforcing a specific way of working. Instead of being productive in automating release process I was forced to fight it and hack it.

There are better things to automate your release with than XML.

If the tool doesn’t entirely do what you need, wrap it, enhance it or dump it in favor of other.

Many tools that are typically used for deployment are quite flexible and have build in mechanism for extending it. I found that writing a plugin or simply wrapping the tool in another process that controls it’s startup, makes it simpler to use and produces repeatable pattern.

If you are having more trouble with the tool itself or simply can’t get result required, don’t be afraid and dump it. There are always other tools. In a worse scenario you can write something simple and tailored for your specific needs.

Application design

Build your application in a way that will support state persistence and restoration.

It is not only the process and tools that you are creating or going to use that should support the automation; it should be the Application itself.

In the example of a complex environment I provided, I mentioned the distributed cache. If the cache is full of data, releasing new version of application could potentially cause the data to be lost. You need to think of a way to get it back to the previous state. The release process should accommodate for restoring the state from some king of storage (eg. Disk, database, replicated cluster, etc.).

Process

Keep your configuration with your source code.

Having environment configurations collocated with the source code in the source version control system has number of benefits:

• It could be unit tested.
• Provides a history of configuration changes. It is always possible to revert to a previous configuration that was working.
• Configuration lives close to developers, who usually know the most on subject of required configuration changes.
• Configuration ships with the deployable.
• It contains a name of a person that modified it, so whenever configuration changes are not clear you can always ask the person who made a change.

Automate every step.

Don’t leave a chance of error creeping into your process by allowing manual steps. It might take some extra effort to automate those simple, little steps but the award is saved time and reliably released software.

Make sure to have everything that application needs to live contained within the release.

Don’t leave anything out: extra libraries, additional software installed, new versions, etc. Ship it all as a part of the release. It doesn’t have to be collocated with a source code (however it is quite beneficial sometimes). Make sure it is accessible from every point you are deploying your software into.

Keep logs, summary and history of releases.

Having release log helps to track the progress of release, identify issues and even test the release itself.  It is also handy when release takes long time and nursing it needs to be handed over to another person.

Summary page helps quickly identify the version of the software released into environment.

Famous last words

I know that sometimes automation might seem impossible but I also believe that impossible doesn’t exist. There is only easy and less easy to do. Automate your release and make your and your comrades life more enjoyable.

Many happy automated releases.

Greg