When Build Challenges Lead to Creative Solutions

A good build environment needs the right balance among complexity, speed, cost, and adaptability. With our recent move to CircleCI, OpenNMS has achieved that balance and even created two customized tools that others might find useful.

Complexity of a Large-Scale Project

At over 5 million lines of code, OpenNMS is one of the largest open source Java projects. With three flavors of OpenNMS — Horizon, Meridian, PoweredBy — there is a lot to keep track of.

In brief, our workflow sees bug fixes go into our oldest relevant branch. New features and enhancements go into the Horizon “release-NN.x” branch or “develop”. With automated continuous integration builds, we merge bug and security fixes made to older foundation branches forward:

Build Process Took Too Long

Before moving everything to CircleCI, builds were taking 20-30 minutes on a good machine, but running the test suite took an additional nine hours with the integration and smoke tests. The situation was becoming untenable: waiting for a build, then possibly adding a fix, then waiting again for a new build sometimes meant two days or more to get a green build.

Now, after migration, the entire build time, including testing, takes only 1.5 hours.

How We Did It

We took advantage of CircleCI’s parallel build feature to use multiple machines for the integration tests. This greatly speeds up the processing time, meaning that developers can build, test, fix, and rebuild within a few hours instead of days.

Tests are more self-contained. Previously, with Bamboo, creating new tests could take a week to get things set up. Now we have Dockerized our smoke tests so that they are repeatable without having to use crazy environments.

At this point, we have automated and moved anything significant that was in Bamboo into CircleCI. We will move other parts of the code base that don’t get updated very often on an as-needed basis.

Migration Challenges

The main challenge in migrating to CircleCI was the test part of the CircleCI workflow. For example, to test something that happens an hour into the process, you have to wait an hour to see if it worked. The longer into the process, the longer the wait.

Trying to figure out how to get code signing into the mix was also an issue, because you don't want to leave your private keys anywhere that automated code is accessing. The way to address this in CircleCI is to create an environment variable that contains a text-encoded version of your private key, and then when the build initializes, decode it and write it to disk.

Part of CircleCI's security setup involves making it so that these variables are not sent along to pull-request builds, so someone couldn't create a malicious pull request and extract our private keys.

To solve the code signing problem, we created an orb to sign packages.

Signing Packages with a Custom Orb

An orb is a shareable package of CircleCI configuration for use in builds. We created an orb that automates the process of turning those environment variables into Java jar-signing and GnuPG signing configs, and then also provides commands to perform the signing once you've created unsigned Debian and RPM packages.

This orb is available on the CircleCI website for anyone to use in their own CircleCI environments.

Disadvantages of the New Environment

Cost can be a disadvantage, as CircleCI server management can be expensive, as the cost of chaining multiple machines is a factor, particularly if we need to add more machines to handle the load.

But if you factor in the amount of time Ben Reed, Principal Software Developer and Build Manager at OpenNMS, spent “babysitting” the previous system, we are probably saving money.

“I feel like we are able to do just as much, if not more, with Circle than Bamboo,” Ben says. He adds that CircleCI is pretty easy to work with. “There is not a lot of figuring out how the environment works. It’s simple, but it can be built up to do other things that are more complex.”

What Does This Mean for OpenNMS Users?

Changes to the build environment may affect people like developers who do their own builds, and PoweredBy users who distribute OpenNMS under their own commercial End User’s License Agreement (EULA). PoweredBy users should read the PDF document in the PoweredBy repo that describes the steps needed to work with the new build environment.

Docker Improvements As Well

During the migration process, we also worked to cut down the size of the Horizon and Minion Docker containers. Each Docker command runs and saves snapshots of the file system. These layers build up, until we saw 500MB in Horizon and 200MB in Meridian and Sentinel that weren’t being used. We wrote a separate Docker container that runs a yum server that removes the files, so you never have the temporary files being saved.

Frustrating at Times, But in the End, Worth It

Ben estimates that the work was 50/50 between enjoyment and frustration. “It’s very gratifying when you’re done work and can see it all go through the flow,” he says. “It’s not the most enjoyable project, but finishing it delivered a great deal of satisfaction.”