RFD5 - Testing Farm support for Fedora, CentOS Stream and RHEL in Image Mode

Status: ⏳️ in progress

This RFD describes the future testing process of Image Mode for Fedora, CentOS Stream and RHEL in Testing Farm.

For more information about Image Mode see:

Introduction

Image mode for RHEL, Fedora and CentOS Stream is an important new feature of our operating systems. It is crucial that these operating systems are validated as a part of the standard development workflows and that the testing entry barrier for these variations of operating systems is as low as possible. That means they need to get first-class support in tmt and Testing Farm, which form the standard tooling and infrastructure for testing Fedora, RHEL and CentOS Stream.

MVP

Please, do not consider the MVP as the final solution for testing Image Mode. It is for early enablement and the next sections of the document will deal with defining how the final and approved solution will look like. Please provide feedback directly to this RFD proposal to shape the future of testing according to your needs.

Testing Farm team has implemented an MVP for testing Image mode on latest nightly composes of RHEL 9.6.0, CentOS Stream 9 and Fedora-40 with the Testing Farm 2024-10.1 release.

Objectives

The MVP has the following objectives:

  • Provide a consistent test development and execution with standard Fedora, CentOS Stream and RHEL.

  • Support x86_64 and aarch64 architectures.

  • Support nested virtualization via Openstack (Red Hat Ranch) and bare metal support on AWS (Public Ranch).

  • Gather feedback from the users and provide a common ground to agree on the final solution for testing.

  • Support multi-host scenarios with current limitations.

  • Execute existing tmt package or composes level tests.

  • Restart Testing Farm request on Image Mode with a single command.

Please note that the MVP is not:

  • Usable in RHEL Gating.

  • Supported for ppc64le and s390x architectures.

  • Supported in Beaker, Azure and IBM Cloud.

  • Supporting artifact installation (e.g. koji builds, copr builds, etc)

Testing Environment

Testing Farm team has decided that the tests will start testing in a booted system in Image Mode. This makes it possible to support existing testing infrastructure without the complexity of building, distributing the pre-created images to all supported infrastructures and booting them. In other terms, you get a deployed system in a pre-defined configuration (see the section below for Containerfile details), similar to a system ready for "day 2" operations as described in the Image mode for Red Hat Enterprise Linux: A quick start guide.

The tests can build, push and rebase on the image your testing needs, testing environment reboot is supported to boot in your specific configuration. See the Examples section for testing details.

Composes

Testing Farm has built images for the following operating systems for consumption via tmt and Testing Farm:

The Qcow2 images can be used with `tmt’s virtual provision plugin. See the Examples section for details.

tmt

Image mode support for tmt was added in teemtee/tmt#3229. Currently, tmt uses rpm-ostree install -A to install all required dependencies for the tests execution into the test environment.

Examples

The examples document the most common use cases for the MVP.

tmt

Use the these examples to try out local development use cases using tmt.

The image sizes are over 1GiB, so make sure to be patient if your network connection is slow. The download will be needed only once, as the download images are cached by tmt.

Please use tmt version tmt-1.38 or newer.

When calling tmt-reboot from a script the command needs to be called via bash -lc tmt-reboot due to a bug being fixed in tmt-1.39.
Provision and login to the system for experimenting (x86_64).
tmt run provision -h virtual --image http://images.osci.redhat.com/RHEL-9.6.0-image-mode-x86_64.qcow2 login finish
Provision and login to the system for experimenting (aarch64).

Running this command on x86_64 will use qemu system emulation and expect it to be slow.

The emulation does not currently work for Fedora-40. The testcloud maintainers are looking into the issue.

 
tmt run provision -h virtual --image http://images.osci.redhat.com/RHEL-9.6.0-image-mode-aarch64.qcow2 --arch aarch64 login finish
Execute bash component functional tests against RHEL-9.6.0-image-mode locally.
rhpkg clone -a rpms/bash
cd bash
tmt run -a provision -h virtual --image http://images.osci.redhat.com/RHEL-9.6.0-image-mode-x86_64.qcow2
Switch to a new container image container which installed nginx

This use case is currently broken, we are looking into it.

 
# provision the machine and login
tmt run -vv provision -h virtual --image /images/RHEL-9.6.0-image-mode-x86_64.qcow2 login

# create a container file which installs and enables nginx
cat > Containerfile << EOF
FROM images.paas.redhat.com/testingfarm/rhel-bootc:9.6
RUN dnf -y install nginx && systemctl enable nginx
EOF

# build the container image
podman build -t localhost/test .

# switch to the newly created container on next boot
bootc switch --transport containers-storage localhost/test

# reboot
reboot

# reconnect back to the tmt machine
tmt run --last login

systemctl status nginx
Cleanup created guests

After the experiments you might some VM leftovers.

tmt clean guests

Testing Farm

Use these examples to try out common use cases using the Testing Farm service.

See Testing Farm onboarding docs if you are not onboarded. See testing-farm CLI command installation instructions. Make sure to use correct token for the ranch you are using. For RHEL use Red Hat ranch token, for others Public ranch.
Restart cockpit gating tests run against RHEL-9.6.0-image-mode.

You can lookup an existing CI run via CI Dashboard.

  • Lookup a component for RHEL-9.6.0 and the test job you would like to restart

  • Click Run Details to get a link to the Testing Farm artifacts

testing-farm restart --compose RHEL-9.6.0-image-mode https://artifacts.osci.redhat.com/testing-farm/69e6b99c-25b2-4a2a-8efb-305eb7bd314f/
Reserve a machine with RHEL-9.6.0-image-mode.
testing-farm reserve --compose RHEL-9.6.0-image-mode
Execute bash component functional tests in Testing Farm on x86_64.
rhpkg clone -a rpms/bash
cd bash
testing-farm request --compose RHEL-9.6.0-image-mode
Execute bash component functional tests in Testing Farm on aarch64 and x86_64.
rhpkg clone -a rpms/bash
cd bash
testing-farm request --compose RHEL-9.6.0-image-mode --arch aarch64,x86_64
Reserve an x86_64 machine in Testing Farm with RHEL-9.6.0.
testing-farm reserve --compose RHEL-9.6.0-image-mode
Reserve a aarch64 machine in Testing Farm with RHEL-9.6.0.
testing-farm reserve --compose RHEL-9.6.0-image-mode --arch aarch64

Run tests against a new brew build

Currently Testing Farm does not support artifact installation. To workaround the problem you have to install the artifacts yourself by building a new container and rebooting into it in the execute phase.

Production Support

Please see this merge request for proposal of the final solution for testing Image Mode in Testing Farm.

Known issues

  • Fedora and CentOS Stream9 images are unavailable, should be fixed during 03/2025.

  • The aarch64 images are unavilable, should be fixed during 03/2025.

  • (Red Hat only) For additional issues see our Jira issues for Image Mode RHEL.