This was originally posted on the Zenoss Google+ Page. Go ahead, Circle Us!
Part of my job at Zenoss is creating and managing ZenPacks. You can refer to my last blog post on configuration management for a summary of what a ZenPack is if needed. ZenPacks can be thought of as each being isolated software projects, so a lot of standard software development practices can be applied to them. In this article I will walk through some of the specifics of how Zenoss builds and test ZenPacks in hopes that you might pick up some tidbits that you can use in your own projects.
Automatic building, testing, packaging and deploying of software is often referred to as “continuous integration.” There are good tools are the market that can drive this process for you. Popular options include CruiseControl, TeamCity, Travis CI, Jenkins and many others. Zenoss uses Jenkins primarily because it is open source, has an enormous plugin library, has a very active community, is language-agnostic and is mature.
Typically when continuous integration is setup for a project, it is done on a project-by-project basis because new projects aren’t added very often. Adding a project is usually the one manual step required to get all of the benefits that continuous integration provides. Our setup requires one more step of automation, the automatic discovery of new projects. The reason this is necessary is that as of this writing there are 428 ZenPacks (each its own project) that need to be tested. Our community and employees are constantly creating new ZenPacks.
It turns out that a good way to fulfill this automatic project discovery requirement was a Jenkins project that would poll our repository collections for changes. We maintain git repositories here and here. Each of these repositories is merely a collection of submodules that each point to a repository for each ZenPack. Anytime a new submodule as added to one of these repositories the Jenkins job will add an appropriate project for it.
Another need we have is to test each ZenPack in a variety of Zenoss configurations. For example, a ZenPack that claims to be compatible with Zenoss versions >= 3 should be tested against the core Zenoss platform with no other ZenPacks installed and a commercial version of Zenoss with all of the typical ZenPacks installed. These two configurations should be tried for Zenoss version 3.2.1, 4.1.1 and the latest development version of Zenoss. Furthermore these combinations should be tested on a 64bit and 32bit platform.
Fortunately Jenkins provides a “matrix” style project that fulfills this need perfectly. With this you can have a farm of build servers that advertise the capabilities that they offer. For instance, one build server could advertise that it has a “Zenoss 3.2” capability and a “64bit” capability. When your project is built, each of the permutations will be farmed out to an appropriate build server. The following screenshot shows the matrix for our CloudStack ZenPack.
The interesting thing about this matrix is that the grey dots are not valid combinations. Jenkins allows you to specify which axes in the matrix are valid by setting a combination filter. The following combination filter produces the above matrix.
((zenoss_version == "3.2" && zenoss_flavor in ["platform", "core", "enterprise"]) || (zenoss_version == "4.1" && zenoss_flavor in ["resmgr"]) || (zenoss_version == "4.2" && zenoss_flavor in ["platform", "core", "resmgr"])) && zenoss_version in ["3.2", "4.1", "4.2"]
The way it works is that only axes that evaluate to true in the above filter will be built. It might look nasty, but it is the basis for all of our ZenPack projects and it works well. Further dependencies within the ZenPacks themselves might add to the filter.
The real meat of continuous integration comes next. Jenkins lets you define build steps for your project. Simply put these are commands that are run on the build slave. Each ZenPack project has eight build steps configured and they perform the following actions.
setup: This prepares the Zenoss environment in which the ZenPack will be tested. We use LVM snapshots for this to enforce a clean testing environment. Each ZenPack test creates a new snapshot of a known-clean Zenoss installation that meets the project’s requirements.
pyflakes: This is a static analysis step that will test all of the Python code in the ZenPack for known syntax errors and some style problems. This is falls under the “linting” bucket and can used to improve code quality.
pep8: Another static analysis step that looks exclusively at code style. PEP-8 is a published standard and commonly accepted as the Python code style guide to use if you want to use one. This can help in projects that must be maintained by multiple authors who may not otherwise share the same coding style.
build: This builds the ZenPack. Since ZenPacks are extended Python eggs, this simply does a “python setup.py bdist_egg” to turn the ZenPack source into a Python egg.
install: Installs the egg built by the previous step into the running Zenoss configuration.
generictests: Runs a series of other generic tests that are applicable to all ZenPacks. These can be used to identify backwards-compatibility problems when portions of the ZenPack framework change.
unittests: Runs all of the unit tests that are part of the ZenPack. These allow the ZenPack author to do specific testing every time the ZenPack is updated.
remove: Removes the ZenPack from the running Zenoss installation.
If any of these steps fail, the build is marked as failed and any artifacts (ZenPack eggs) created are not published. If all of the steps succeed, the resulting ZenPack is published.
One final option that we chose to use in Jenkins is repository polling for all of the projects. This means that anytime a ZenPack’s source is updated, it will run through the process described above. This makes the entire lifecycle fully automatic.
To date this system has generated 744 unique ZenPack eggs and performs roughly 100 builds each day. Dozens of bugs in ZenPacks and the ZenPack framework were discovered and resolved during the rollout of this system, and it is my hope that it continues to provide higher quality ZenPacks to users with less effort from all.
