Testing Farm release 2026-09.2

On Red Hat ranch, Testing Farm now disables the rhel-buildroot repository before running the tests. Tests that installed packages from it relied on a broken configuration and can fail now. If your tests need the buildroot repository, enable it with the buildroot prepare feature (TFT-5133):

In the upcoming release 2026-11.1, scheduled for the first half of November, we will tighten the retention policy for Red Hat ranch. For details see TFT-3595. Please provide feedback directly in the issue so we can fine tune the change according to your expectations.

tmt was updated to 1.79 from 1.78.

Testing Farm CLI tool was updated to v0.0.39 from v0.0.38, please update your installation.

Artemis was updated to v0.0.93 from v0.0.92 on Red Hat ranch (TFT-5191).

Red Hat ranch supports nested virtualization on AWS, next to OpenStack, which reaches its end of life at the end of October. The AWS pools now use the new Intel c8i, m8i and r8i instance families to provide this capability. Use the virtualization.is-supported: true hardware requirement to ask for machines with virtualization support (TFT-4506, testing-farm/artemis!1602):

Testing Farm now reports the progress of each stage of a request. The API exposes the pipeline stages (initialization, test-discovery, testing, archiving) and the per-plan stages (provisioning, guest-setup, running, cleanup) in run.stages, each with its state and timestamps (TFT-4876, TFT-4878). The results viewer does not show the stages yet.

Red Hat ranch now offers new Fedora composes built from the Fedora Cloud Base images on AWS, for example Fedora-44-Cloud-Base. They use new compose names, next to the legacy composes shared with 1minutetip. You can now provision Fedora aarch64 guests on AWS, which before were available only from Beaker (TFT-5121).

Public ranch now offers the same GPU guests as Red Hat ranch, AWS g4dn instances with an NVIDIA T4 GPU, which are easier to get. The old pools used the p2 and p3 instance families, which AWS no longer offers, so requests for GK210 (Tesla K80) or GV100 (Tesla V100) cannot be provisioned anymore. See the GPU offerings for the supported devices (TFT-5006, testing-farm/infrastructure!1598).

The testing-farm reserve command sets TMT_REBOOT_TIMEOUT to the reservation duration. You can now reboot a reserved machine into an offline upgrade or a slow bare metal boot without losing it after 10 minutes (testing-farm/cli!244).

The CLI has a new global --debug option, also available as the TESTING_FARM_DEBUG environment variable, which prints full tracebacks of unexpected errors (testing-farm/cli!241).

Testing Farm raised TMT_CONNECT_TIMEOUT from tmt’s default of 120 seconds and TMT_REBOOT_TIMEOUT from 600 seconds, both to 1200 seconds. Bare metal guests take up to 820 seconds to boot, and tmt gave up on them before they came back (testing-farm/infrastructure!1601, testing-farm/infrastructure!1625).

The GET /v0.1/requests endpoint accepts the updated_before and updated_after filters (testing-farm/nucleus!335).

The worker image includes the tmt+prepare-artifact plugin (testing-farm/gluetool-modules!1099).

We added monitoring of the dispatchers, the API database connection pool, CockroachDB and the archiving duration (TFT-4873, TFT-4947, TFT-4949, TFT-4941).

The API now aborts database statements that run longer than 60 seconds and caches token lookups to improve its stability (TFT-4940).

Fixed tmt-reboot waking the wrong test on a reserved machine. When you ran your own tmt against a guest from testing-farm reserve, both tmt instances shared /var/tmp/tmt-test.pid, and your test could hang until its duration expired (TFT-5167).

Fixed reservations extended past 99 hours, which tmt killed with Test exceeded maximum duration of 99h (TFT-5153).

Fixed the reproducer, which failed to download the tmt environment file with 404 Not Found (TFT-5141).

Fixed requests from forks of repositories with .testing-farm.yaml secrets. The secrets stay bound to the original repository URL, and Testing Farm retried the failed decryption for 10 minutes before it killed the request with an infrastructure error. Testing Farm now skips the secrets it cannot decrypt with a warning and runs the tests (TFT-5158).

Fixed copr-build artifact installation failing with 'utf-8' codec can’t decode byte when the Copr builder log contained invalid UTF-8 (TFT-5189).

Fixed Fedora ELN guest setup failing with Could not find the requested service systemd-resolved (TFT-5129).

Fixed the EPEL repository missing on CentOS Stream 10 on Public ranch (testing-farm/infrastructure!1585).

Fixed the updated field of requests, which never changed after the request was created, so the updated_before and updated_after filters returned wrong results (testing-farm/nucleus!345).

Fixed POST /v0.1/requests returning 500 Internal Server Error on database serialization conflicts (testing-farm/nucleus!315).

Fixed provisioning on Public ranch stalling for hours when the Artemis workers deadlocked on debug logging (testing-farm/infrastructure!1594).

Fixed the installation of standard-test-roles after pagure.io shut down. Testing Farm now uses the testing-farm/standard-test-roles mirror (testing-farm/gluetool-modules!1108).

Fixed the token dialog in the web UI, which offered only the Public and Red Hat ranches. It now lists every ranch your account can access (testing-farm/ui!16).

Fixed option names in the CLI usage output, which showed for example --AGE instead of --age (testing-farm/cli!243).

Fixed testing-farm watch writing ANSI color codes into JSON output redirected to a file (testing-farm/cli!236).

Added an FAQ entry on testing with a newly installed kernel. Testing Farm does not reboot the guest after artifact installation, so the entry shows a prepare step which installs the kernel and reboots (testing-farm/docs/root!293).

Added an FAQ entry for the Artifacts unreachable via SSL CLI error in CI containers without access to the Red Hat network, fixed by the --skip-summary option (testing-farm/docs/root!289).

Documented how to reboot a reserved machine and when you still need to set TMT_REBOOT_TIMEOUT yourself (testing-farm/docs/root!291).

Updated the list of GPU offerings (testing-farm/docs/root!290).

We would like to thank the following external contributors for contributing to Testing Farm.

List of important packages bundled in the worker image.

Number of processed requests in the last 30 days was 258818.

The average error rate for Testing Farm was approximately 3.40%.