I have been doing some patches on Delorean lately so I thought it could be a good idea to describe it in this article as it is not a well known tool.
Delorean is a tool to build rpm packages on each commit of a set of git repositories. The goal of this tool is to detect as soon as possible packaging issues with the upstream branches of all the projects we are taking care of. The target repositories are of course the OpenStack related git repositories that we care about in RDO.
To understand how Delorean works, you must first understand how to build a package from the git repositories.
Building an rpm package from the git repositories
We need 2 git repositories to build a package:
- the upstream git repositories with an RDO specific branch to store patches.
- the dist-git repositories where you store the spec file and the various files needed for packaging like the files describing the services you want to start and the patches you need to apply on the sources.
The patches in the dist-git repository are generated from the RDO branch in the upstream git repository. I’ll not enter into to much detail here as in RDO there are very very few patches in the packages as the goal is to deliver the pristine upstream experience. It carries for example some patches to the OpenStack Horizon package to change the theme and add some RDO logos.
In Delorean, the RDO team does not provide any extra patch as we want to test the packaging of the upstream repositories without any modification like we do in the RDO releases so it is using directly the upstream git repositories.
How Delorean knows which packages to build
Delorean uses the rdopkg python module to get the description of the packages it needs to build. Rdopkg uses a YAML file that describes the name of the packages, where to get the git repositories from and who are the maintainers of the packages. This YAML file called rdo.yml is part of the rdoinfo git repository. Rdopkg stores a clone of this repository in the home directory of the user using Delorean under $HOME/.rdopkg/rdoinfo.
For example here is the information for the OpenStack Nova package:
package-configs: core: name: openstack-%(project)s upstream: git://git.openstack.org/openstack/%(project)s master-distgit: git://github.com/openstack-packages/%(project)s packages: - project: nova conf: core maintainers: - email maintainer1 - email maintainer2
It means for the nova project, we use the package configuration named core so the package name is openstack-nova, the upstream git repo is git://git.openstack.org/openstack/nova and the dist-git repo is git://github.com/openstack-packages/nova. The keys defined in the package-configs section can also be overridden in each project’s section if needed.
All the packages are described like that and Delorean can use these git repositories for its own needs.
How Delorean builds packages
Each time Delorean is launched it does the following in sequence:
- for all the packages described in rdo.yml:
- clone or update the 2 git repositories by package under the <DATADIR>/<package> for the git upstream repository and under <DATADIR>/<package>_distro for the dist-git repository.
- gather the new commits sha1 and timestamps.
- according to the timestamps build the packages for each commit that was not previously built in chronological order (oldest first) using these steps:
- point the upstream repository to the targeted commit.
- create a source tarball using setup.py sdist.
- extract the version number using setup.py --version.
- do some black magic to deduce the version of the package and the release of the package from the extracted version.
- inject the name of tarball, the version and release in the spec file in the dist-git repository.
- create the source package.
- build the binary package.
- install the built package to test that the dependencies are right.
- store the build status in the sqlite database with the sha1 and timestamp of the commit.
- all the logs and generated rpm (binary and source) are stored under <DATADIR>/repos/<2 first letters of the sha1>/<3rd and 4th letters of the sha1>/<sha1>_<partial dist-git sha1>/.
- If the installation is successful, a repository is built using a set of symlinks from all the packages that were built and installed successfully before. This way you can install packages from this repository to test a specific commit.
- If the installation or the build failed, send an email to the maintainers of the package.
- If the installation is successful, a symlink is created to point to the last successful build commit under the name current.
- Generate reports.
Where to access the public repositories and status reports
For RDO, we have two Delorean instances targeting OpenStack master branch for Fedora and CentOS. You can access these repositories at:
There are also 2 reports generated at the end of each build stored in the <DATADIR>/repos directory:
- a chronological report called report.html (Fedora and CentOS ones)
- a global status report call status_report.html (Fedora and CentOS ones).
Validation
The /current repository is used as the input to other validation jobs inside the RDO project or in the OpenStack CI itself. These jobs have the ability to update links to repositories should they pass CI.
http://trunk.rdoproject.org/centos7/current-passed-ci/ is updated by a packstack job, when it passes it “promotes” a current repository so people who want to use packstack with trunk packages can use this and know they worked in at least 1 setup.
http://trunk.rdoproject.org/centos7/current-tripleo/ is updated (manually at the moment) when we run passing tripleo-ci on a trunk repository. This is currently triggered manually at the moment but tripleo hopes to soon automate it.
The Puppet OpenStack CI is using the /current-passed-ci repository to deploy the latest version of OpenStack. While other OpenStack projects are using devstack to deploy OpenStack from source, the Puppet OpenStack modules are deploying realistic scenarios where packaging is extensively used. As the Puppet team needs to push features that are specific to the newest version of OpenStack, it is using the packages from Delorean to test OpenStack trunk in Puppet modules. This CI is essential to develop Puppet modules because some OpenStack installers (like RDO manager) are directly using them downstream.
Source code
If you want to deep dive to understand all the inner details or if you want to use Delorean, you can find the source code and the documentation at the following URL: https://github.com/openstack-packages/delorean.
If you want to start using Delorean for your own needs, or contribute to RDO, you may want to read this document: https://www.rdoproject.org/packaging/rdo-packaging.html#master-pkg-guide.