Build RPM packages from Development Trees.
Use RPMs to deploy and test development versions of your package, during your development cycle. Quickly build RPM packages, using incremental builds from your worktree. Use RPM to install the packages locally and easily rollback by downgrading back to the distro packages.
In order to use git-rpmbuild, simply check out a development tree from the upstream git repository. Create and check out your own development branches, add your own commits and maybe even keep uncommitted changes on your worktree.
Then build RPM packages for your package using:
$ git rpmbuild
This will then detect which package you're building and download the RPM specfile from Fedora, and use that specfile to build RPM packages using the development sources from your worktree.
The build is incremental, so it's typically pretty quick if you're only modifying a handful of files in the project in between each build, since only those files and the targets that depend on them need to be rebuilt each time.
Once the RPMs are built, you can install them using:
$ sudo rpm -Fvh ~/rpmbuild/RPMS/*/mypackage-*.rpm
The RPM "Release:" part of the version is used both to prefer latest versions on installs (making it easy to accumulate builds around) and to keep enough information on each package to track it back to the sources.
The "Release:" field will include a full timestamp (YYYYMMDDHHMMSS
), so that
more recent builds will be considered newer. The rpm -F
command will prefer
those, so when you use it on an RPMS/
directory with many builds, the latest
one will be installed by it.
Furthermore, "Release:" will include the output of git describe
, which
typically includes the closest tag (which is typically used for the upstream
version), plus the name of the local branch checked out (so if you're using
feature branches, you can recognize which of them was used in a specific set of
packages you built.)
If you'd like to rollback an install, back to the packages shipped by your distribution, you can then use the following command:
$ sudo dnf downgrade mypackage
The 14-digit timestamp also makes it likely that the git rpmbuild
packages
will have a higher version than the distribution packages, so normally dnf
or
yum
will not replace them, even if the distribution has released newer
releases of that same package. (It is possible that the distro packages will
become newer if they release a new major version, though.)
There are many advantages of using git rpmbuild
over installing binaries
locally using sudo ninja install
or sudo make install
.
Some of the advantages of this approach are:
-
Same build configuration as the distro: Since the build is using the instructions from the RPM specfile, the
./configure
ormeson
invocation will incorporate all the options used in the build of the package for your distribution, such as the typical directories (prefix, bindir, libdir, etc.) as well as configuration options to enable/disable certain features. -
Integrity of package and binaries on the system: Since a package is being installed (instead of binaries from a local build replacing existing binaries owned by specific packages), the RPM database will show that everything looks correct. In particular, you can use the
rpm -V
command to verify that the files owned by your package have not been replaced or tampered with, which will work correctly since the RPMs installed have the correct checksums of the binaries built from the development tree. -
Properly handling configuration files: Configuration files will be handled in the same way as RPM upgrades from the distro, with
.rpmnew
and.rpmsave
files, rather than blindingly replacing them with defaults from the development tree, which could easily happen if installing directly from the tree to the system root filesystem. -
Easy to track currently running version: By looking at the package name, you can quickly see which branch was checked out when these binaries were built, and using the information from
git describe
also present in the "Release:" field, it is possible to check out the exact commit that was used for this particular build. -
Ability to rollback and to switch builds: Rolling back is easy using a command such as
dnf downgrade
to go back to the packages from the distro. It is also possible to keep multiple development builds and switch between them, by simply usingrpm -U --oldpackage
to install a previous build, with no need to check out a different branch on the source tree and rebuild those binaries, or to keep multiple git worktrees around. -
Less interference from the normal distro package upgrades: If you're installing binaries over an existing package, if an upgrade for that package becomes available and the machine has automatic updates configured, it will most likely clobber the local binaries while installing that upgrade. By having the development binaries installed through an RPM with a version string that is most likely to be much larger than the one from the distro, it's unlikely that newer distro packages will replace the development ones, thus preventing them from being inadvertently replaced back with the release version of the package while you were meaning to test the new version.
-
Ability to install development binaries on other machines: This is often useful in machines that do not have a development environment present (so wouldn't be able to run
make install
orninja install
, etc.) By building packages from the development sources, it's easy to ship the packages to those machines and simply install them usingrpm
ordnf
, the tools that will be already present there.
There are other similar approaches that involve creating RPM packages from
development trees (such as
packit
), but those typically
involve packing the sources into an archive, building an SRPM and rebuilding
that from scratch every time.
This can be pretty time consuming, even when using caching for the compilation
steps (ccache
). That makes these methods great for scheduled builds (nightly,
weekly) and for background batch methods (integration testing, etc.) But they're
not great for development, when you're trying to build repeatedly and shaving
time off your commit/build/deploy/test cycle can improve your productivity
significantly.
To illustrate the difference in both methods, see the videos below. To build systemd RPMs from the SRPM package takes over 5 minutes on a fairly capable laptop:
Using a tool that would build a full-blown SRPM would take even longer, since
the git archive
step must also be performed to generate an SRPM from the
worktree.
On the other hand, to build those RPMs from a local development tree using
git rpmbuild
takes under 1 minute, since the build is incremental:
Note that it only takes 45 seconds, yet there is time to run all the tests and save all the RPM packages on disk. Bringing 5 minutes down to 1 minute make quite a difference when you're actively developing and need a quick turnaround!
It works by leveraging two arguments of rpmbuild
:
-
--build-in-place
: Build from locally checked out sources. Sets_builddir
to point to the current working directory. -
--noprep
: Skip the%prep
step altogether. This means not only the source tarball is not unpacked, but any patches from the RPM specfile will not be applied.
Together, these result in a local build being performed and the normal build tools ensure an incremental build will be used, since the RPM build tree is now the same as our development build tree.
Pre-requisite for the code is having Python 3.7 installed and having a working
rpmbuild
in the system, that's about it.
To work from git, simply clone the repository and make sure the git-rpmbuild
script is on the $PATH
, either by adding the git worktree to $PATH
or by
creating a symlink or copy of that script somewhere that's already in your
$PATH
(such as ~/.local/bin
, assuming you've added it.)
I also maintain a COPR for git-rpmbuild for Fedora 30+ at: https://copr.fedorainfracloud.org/coprs/filbranden/git-rpmbuild/
In order to use it, simply enable it with:
$ sudo dnf copr enable filbranden/git-rpmbuild
And install the package using:
$ sudo dnf install git-rpmbuild
There are, of course, limitations to this approach.
Only packages which perform limited actions in the %prep
step will work with
this approach. Typically that means only unpacking the source and applying
patches for backports that are already present upstream. Since the whole %prep
step is skipped, the development tree must be similar enough to what the
specfile's %build
and %install
otherwise expect.
(Here's an
example
of moving a step from %prep
to %install
in the specfile to solve one such
problem.)
This will also only work for packages that exist in Fedora, since that's where the specfiles come from.
It also expects that the development worktree is close enough to what the latest
specfile is building, so that the instructions will still be applicable.
Sometimes if either development tree or specfile are having heavy changes, it's
not that easy to keep them both working in sync. Examples are files that are
shipped and not declared under %files
, or build instructions that have changed
since the build system has been tweaked or reconfigured.
We also need the RPM build not to mess with the source tree (otherwise those
changes will mess up our development environment.) One example of problematic
package here is selinux-policy, which builds from the same tree 3 times (for
each type of policy, such as "targeted" and "strict") while replacing a handful
of files before each build. That will surely break git-rpmbuild
.
git-rpmbuild
also makes some assumptions about tags in the development git
repository, since it uses them to populate the "Version:" and "Release:" fields
of the built RPMs. For now, it expects tags to match a version number, possibly
including a simple "v" prefix to that number.
These packages are known to work with git-rpmbuild
:
systemd
numactl
oomd
meson
Currently not working:
-
rpm
: Tags in the development git repository are in the formatrpm-${version}
, which currently wants to userpm
as a "Version:". Additionally, as ofrpm-4.15.0-alpha-69-g858d6babd
, there's a build breakage inrpmmodule.c
aboutvariable 'moduledef' has initializer but incomplete type
which needs some investigation. -
vim
: For some reason I geterror: Bad source: ~/.local/share/git-rpmbuild/fedora-src/vim/vim-8.1-2019.tar.bz2: No such file or directory
. It's expected that this file doesn't exist, but it's unexpected that this would break the build, since with--noprep
no one should be trying to use it... I might need a version of RPM with more debugging available to dig into this one, but see above about that 😆. One more quirk of Vim is that the tags include patch number (such as v8.1.2026), but the Vim specfile would prefer to use simply 8.1 as the version and move 2026 to the "Release:" field only. -
asciinema
: Fails withInstalled (but unpackaged) file(s) found
for a handful of Markdown files. Possibly easy to address in the specfile.
One related project is packit
, in
fact I started to look into whether I could include a feature such as this one
into packit
itself rather than having a separate tool, but without much
success.
I would like to see a similar approach for Debian-based systems, since often I would like to test my systems on Ubuntu or Debian and being able to use a similar approach would be very nice. (See above for advantages.)
Obviously, having more packages working on git-rpmbuild
is a big priority, so
feel free to report success/failure if you use it in any of your packages. PRs
addressing issues that fix build of packages you care about are also very
welcome!
I would like to use epochs to make versions of git-rpmbuild
RPMs even newer
than distro packages, but there are some tricky cases when specfiles build
multiple packages and the packages depend on each other and they specify the
dependency with an exact version match that doesn't include any epoch...
(Parsing and generating specfiles is pretty hacky!)
More flexibility in handling the "Release:" tag would be nice.
Per-package overrides, perhaps with a dotfile at the top of the git tree, or perhaps a local ConfigParser file somewhere in the user's home directory, would be good if packages have special requirements that need to be configured for those packages only.
Right now there are no command-line arguments, but it would be nice to add options to enable or disable certain features. Examples are generation of debuginfo and debugsource packages (currently always disabled), or whether to run the package tests during the RPM build step (currently always enabled.) Project name detection currently depends on the directory name, having a command-line argument/flag or per-project config file in the git tree to override would also be a good idea.
While the primary purpose of this package is to build RPMs in place from a development worktree, being able to generate a SRPM as well would be nice, as that could be then used to feed a COPR repository and to automate nightly and weekly builds for projects.