Copr builders powered by bootc

As of today, all Copr builder virtual machines are now being spawned from bootc images, which is no small feat because the builder infrastructure involves multiple architectures (x86_64, aarch64, ppc64le, s390x), multiple clouds (Amazon AWS, IBM Cloud), and on-premise hypervisors. It scales up to 400 builders running simultaneously and peaking at 30k builds a day.

Before bootc

You can find some interesting history and previous numbers in Pavel’s article - Fedora Copr farm of builders - status of July 2021. The part it leaves out is how we used to generate the Copr builder images.

The process is documented in the official Copr documentation. In a nutshell, it involved manually spawning a VM from a fresh Fedora Cloud image, running Ansible playbooks to provision it, and then using custom scripts to upload the image to the right place. Because we need to build the images natively, we had to follow this process for every architecture.

The easiest workflow was for x86_64 builders running on our own hypervisors. It meant connecting to the hypervisor using SSH and running a custom copr-image script from the praiskup/helpers repository. While its usage looks innocent, internally it had to execute many virt-sysprep commands. It also required some guestfish hacks to modify cloud-init configuration inside of the image so that it works outside of an actual cloud. Then, finally, using the upload-qcow2-images script to upload the image into libvirt.

The same exact workflow for ppc64le builders. However, internally it had a special case uploading the image also to OSU OSL OpenStack.

For s390x builders, we don’t have a hypervisor where we could natively build the image. Thus we needed to spawn a new VM in IBM Cloud and run the previously mentioned copr-image script inside of it. Once finished, we needed to upload the image to IBM Cloud. This is supposed to be done using the ibmcloud tool, but the problem is that it is not FOSS, and as such, it cannot be packaged for Fedora. We don’t want to run random binaries from the internet, so we containerized it.

At this point, only x86_64 and aarch64 images for Amazon AWS remain.

While not straightforward to create a new AMI from a local qcow2 image, it’s quite easy to create an AMI from a running EC2 instance. That was our strategy. Spawn a new instance from a fresh Fedora Cloud image, provision it, and then create an AMI from it.

Current situation

I disliked exactly three aspects concerning the previous solution. It required a lot of manual work, the process was different for every cloud and architecture, and the bus factor was less than one.

Even though at this moment generating a fresh set of builder images still requires about the same amount of manual work as before, there is a potential for future automation. By switching to bootc and Image Builder, we were able to offload some dirty work to them while also unifying the process to follow the same steps for all architectures and clouds (with minor caveats).

The current process is temporarily documented here. We spawn a VM for every architecture and use it to build the system image from our Containerfile via the quay.io/centos-bootc/bootc-image-builder. Then we fetch the results and upload them where needed.

For Amazon AWS, we can utilize the image-builder upload feature which is amazing. But for other clouds and hypervisors, we still need our custom upload-qcow2-images and quay.io/praiskup/ibmcloud-cli. If image-builder could implement the missing support and enable uploading to all of them, that would be a major win for us.

Future plans

My goal is simple, I want one-button deployment. Well, almost.

When a change is made to our Containerfile, or when triggered manually, or periodically after a period of inactivity, I want the images to be automatically built for all architectures and uploaded to all the necessary places. Then seeing a list of image names and AMIs that I can either choose to use or ignore.

The bootc-image-builder-action seems like the perfect candidate, but the problem is that it cannot natively build images for ppc64le and s390x.

SNThrailkill recommended GitLab Runners but that would require us to maintain the runner VMs, which is annoying. Moreover, there is a potential chicken-and-egg problem, meaning that if we break our image, we might not be able to spawn a VM to build a new working image. We also wouldn’t be able to use the existing GitHub action and would have to port it for GitLab.

At this moment, our team is leaning towards Konflux and a tekton pipeline for building images. Fedora Konflux instance is limited to x86_64 and aarch64, so we would temporarily have to use an internal Red Hat instance which provides all the architectures needed by us.

Many questions are yet to be answered. Is Konflux ready? Does the pipeline for building images already exist? Does it support everything we need? Is it built on top of image-builder so that we can use its upload feature?

Pitfalls along the way

Hopefully, this can help Image Builder and bootc developers better understand their blind spots in the onboarding process, and also prevent new users from repeating the same mistakes.

Before discovering that bootc exists, our original approach was to use just Image Builder and its blueprints, and automatize the process using Packit. There were several problems. It was easy to build the image locally from our blueprint, but it wasn’t possible to upload the same blueprint to be built in a hosted Image Builder service. Additionally, I had several issues with the Blueprint TOML format. The order of operations is pre-defined (e.g. all users are always created before any packages are installed). There is no escape hatch to run a custom command. And finally, it’s yet another specification to learn. My recommendation? Just go with bootc.

Our main problem with bootc is the immutability of the filesystem. Can somebody please help me understand whether the immutable filesystem is a fundamental building block, a key piece of technology that enables bootable containers, or whether it is an unrelated feature? If it is technologically possible, our team would love to see officially supported mutable bootc base images. Currently, we are going forward with a hack to make the root filesystem transient.

One of the issues that probably stems out of the immutable filesystem is the necessity to change the default location of the RPM database. This hack is baked into the bootc base images and we needed to revert it because it causes Mock to fail under some specific circumstances. This unfortunately cost us many many hours of debugging.

The process of building system images is quite storage intensive in /var/lib/containers and /run. To avoid running out of disk space on our virtual machines, we had to turn our swap partition into a data volume and mount the problematic directories there. Not sure if there is something that image-builder can do to make this a less of problem.

We build the system images natively on VMs of the same architecture that they are targeted for, but then we fetch all of them to an x86_64 machine and upload the images to the respective clouds from there. We discovered a bug in cross-arch upload to AWS, which was promptly confirmed and fixed by the image-builder team. Big customer satisfaction right here.

We also struggled with setting up AWS permissions for the image-builder upload command to work correctly. We tried running it, fixing the insufficient permissions it complained about, running it again, and again, and so on. I don’t recommend this approach. It turns out there is a documentation page with instructions.

I hope this chapter doesn’t come across as too discouraging. In fact, we found workarounds for all of our problems, and we are now happily using this in production. So you can probably too.