Testing Farm release 2026-10.1

Red Hat ranch no longer provisions guests in PSI OpenStack, which is being decommissioned. We removed the baseosci-openstack and baseosci-openstack-staging pools. If your requests name one of these pools, remove it and Artemis will pick another pool (TFT-5241, baseos-qe/ansible-baseos-ci!12777).

tmt 1.80 changed the precedence of test and guest environment variables. If a test and a guest define the same variable, check which value your test receives now (tmt documentation).

On Red Hat ranch, the ppc64le IBM Cloud images built after 2026-10-15 no longer carry custom network configuration and use the stock RHEL cloud image setup, including eth0-style interface names. If your tests rename the network interfaces, see the workaround in TFT-5119 (TFT-5176, baseos-qe/ansible-baseos-ci!12702).

tmt was updated to 1.80 from 1.79. The worker ships 1.80 with the fixes from teemtee/tmt#5286 (testing-farm/gluetool-modules!1124).

Artemis was updated to v0.0.95 from v0.0.93 on Red Hat ranch. See also the release notes of v0.0.94.

Testing Farm can now explain failed tests with Log Detective. For each failed test, or for a failed plan without a failed test, the worker sends the relevant logs for analysis and stores the explanation, a suggested solution and the relevant log snippets in results.xml. The results viewer shows them behind a toggle, which appears only when the request has an AI summary. The analysis never changes the test results (testing-farm/gluetool-modules!1091, testing-farm/oculus!104, testing-farm/infrastructure!1679). Each AI summary has a "Feedback" button, where you can rate the summary and tell us what was wrong or helpful (testing-farm/oculus!94).

The results viewer has new controls next to the "Show passed tests" toggle. Two buttons expand or collapse all plans, and a search box filters tests by name as you type. The search shows matching passed tests even with "Show passed tests" off, and the controls work while the request still runs (TFT-5220, testing-farm/oculus!102).

Red Hat ranch now runs tmt web. The results viewer links each plan and test to its tmt metadata, as it already does on Public ranch.

On both Public and Red Hat ranch, the artifacts link of a queued request now opens an HTML page instead of a bare 404. The page follows the request state in the API until a worker picks up the request.

On Red Hat ranch, bare metal requests no longer need to name one of the aws-07-bare-metal to aws-10-bare-metal pools. Artemis sends requests with the virtualization.is-virtualized: false hardware requirement to AWS bare metal first and falls back to Beaker. Please stop naming the *-bare-metal pools in your requests (TFT-5202, baseos-qe/ansible-baseos-ci!12863, testing-farm/docs/root!295).

Fedora-latest and Fedora-latest-image-mode now point to Fedora 45, and Fedora 45 guests on Public ranch use the GA images instead of the prerelease ones (testing-farm/infrastructure!1683).

testing-farm list shows the request ID for requests without artifacts yet, for example queued ones, instead of <unavailable> (testing-farm/cli!242).

Red Hat ranch upgrades beakerlib to the latest upstream build on RHEL 8 and newer, RHEL 10 included. Your tests can use newer features such as rlImport --all (testing-farm/ranch-redhat!12).

Image mode composes on Red Hat ranch get their own symbolic names with the -image-mode suffix, for example RHEL-9.9.0-rc-image-mode. Testing Farm also imports the CTC image mode composes, so you can test them on bare metal in Beaker (TFT-4802, TFT-5242, baseos-qe/ansible-baseos-ci!12764).

The multihost pipeline installs artifacts before the default user prepare phases, matching the regular pipeline (TFT-5281, testing-farm/gluetool-modules!1122).

Testing Farm calls your notification webhook only when the request state changes (testing-farm/gluetool-modules!1118).

Testing Farm no longer uploads the repositories tmt clones for imported plans, the work-*/import/plans/ directories, to the artifacts storage. They hold no results or logs, and the largest RHEL requests stored about 657,000 of their files (testing-farm/infrastructure!1673, testing-farm/ranch-public!24, testing-farm/ranch-redhat!11).

The /container/build test writes the digest of the pushed image to $TMT_PLAN_DATA/target-image.digest, so a later step can sign that exact image. It also retries buildah build on transient failures, 3 times by default (testing-farm/tests!97, testing-farm/tests!92).

The API authenticates API keys faster. The token lookup took more database time than any other query and slowed the API down during the outage on 2026-10-21 (testing-farm/nucleus!349).

Artemis dispatches provisioning tasks without idling between dispatcher ticks while it has work to do, and its routing fetches the pool metrics once per request instead of once per routing policy (testing-farm/artemis!1645, testing-farm/artemis!1651, testing-farm/artemis!1652). It also reads the AWS VPC ID from the pool configuration instead of asking AWS for it on every new instance, which lowers the EC2 API request throttling (TFT-5211, testing-farm/artemis!1642).

Workers fetch the results viewer from the configuration container and the standalone Python builds from S3, instead of downloading them from gitlab.com for each request. gitlab.com rate limited the workers, and requests failed with 429 Too Many Requests (TFT-5265, testing-farm/infrastructure!1648, testing-farm/standalone-python!4).

We added monitoring of the Artemis databases and message brokers on Public ranch and of the Beaker download mirror. The leftover requests page lists requests stuck in the new, queued or running state (TFT-4053, TFT-4952).

Fixed RHEL 6 testing on Red Hat ranch. Ansible fell back to the system Python 2.6, which ansible-core does not support, and the guest repositories still pointed to the decommissioned download.eng.bos.redhat.com host (TFT-5204, testing-farm/infrastructure!1623, testing-farm/infrastructure!1635, testing-farm/infrastructure!1642).

Fixed kickstart snippets on image mode Beaker machines. Artemis lost the kickstart.script, kickstart.pre-install and kickstart.post-install fields of your request, together with any custom partitioning. To partition the disk with kickstart.script, also set no_autopart in kickstart.metadata (TFT-4430, testing-farm/artemis!1653).

Fixed plans with : in their name, for example /plans/php:8.2, on image mode guests. The prepare step failed with Error: validating volumes: faccessat /var/ARTIFACTS/work-php: no such file or directory (TFT-5289, testing-farm/gluetool-modules!1126).

Fixed requests testing an annotated git tag, which failed with trying to write non-commit object (testing-farm/gluetool-modules!1127).

Fixed requests with update_from_artifacts enabled, which failed on every guest with 'dict object' has no attribute 'rc'. The reboot check after the upgrade now also runs on RHEL 6 (testing-farm/infrastructure!1603).

Fixed long reservations killed during archiving for running out of memory. Redacting secrets loaded each log file into memory at twice its size, and the guest console log of a 99 hour reservation grew to several GiB. Testing Farm now redacts the logs in constant memory and skips files it already checked (TFT-5159, testing-farm/gluetool-modules!1102, testing-farm/gluetool-modules!1103, testing-farm/gluetool-modules!1104, testing-farm/ranch-public!27, testing-farm/ranch-redhat!19).

Fixed cloning private repositories from gitlab.cee.redhat.com on Red Hat ranch, which failed with fatal: could not read Username for 'https://gitlab.cee.redhat.com' (TFT-5318, testing-farm/infrastructure!1682).

Fixed image mode guests in multihost requests on Red Hat ranch, which never became ready because root login stayed blocked (TFT-5320, testing-farm/ranch-redhat!18).

Fixed the NTP check timing out on CentOS Stream 9 and Fedora 45 guests on Red Hat ranch with slow DNS resolvers (testing-farm/profiles!113).

Fixed request cancellation after a temporary API outage. The worker checks for cancellation on a timer, and a failed API call stopped that timer for the rest of the request, so Testing Farm ignored your cancellation (TFT-4698, testing-farm/gluetool-modules!1114).

Fixed pipelines which kept running after the request timeout. On Public ranch, one such pipeline held its worker slot for 23 days after Testing Farm reported its request as canceled (testing-farm/infrastructure!1670).

Fixed Public ranch requests failing with allocating lock for new container: allocation failed; exceeded num_locks (2048) (testing-farm/infrastructure!1660).

Fixed stage progress updates, which the API rejected with field required once a request had test results (testing-farm/gluetool-modules!1112).

Fixed GET /v0.1/requests?state=cancel-requested, which returned 500 Internal Server Error (TFT-5243, testing-farm/nucleus!346).

Fixed the API documentation at /docs, which showed a blank page (testing-farm/nucleus!347).

Fixed AWS provisioning on Red Hat ranch for flavors without nested virtualization support. Artemis requested the NestedVirtualization CPU option for every flavor, and now uses it only on the c8i, m8i and r8i families which support it (testing-farm/artemis!1640, testing-farm/artemis!1641).

Fixed random test failures on RHIVOS guests on Red Hat ranch. Artemis picked the c7g.medium AWS flavor, which has too little memory for them. We removed the flavor, and every AWS guest now gets at least 4 GiB of memory (baseos-qe/ansible-baseos-ci!12736).

Fixed the EPEL repositories on CentOS Stream 10 guests, which pointed to EPEL 10.3 instead of EPEL 10.4 (TFT-5196, testing-farm/profiles!105).

Fixed the build tag repository on Fedora ELN guests. ELN reports the version of the RHEL release it targets, so the repository pointed to a nonexistent f<version>-build tag instead of eln-build (ci/tickets#595, testing-farm/profiles!112).

The artifacts documentation describes what install: true and install: false do for each artifact type, and explains the difference between --repository and --repository-file (testing-farm/docs/root!292).

The service-level agreement now includes a queue time SLO: Testing Farm aims to start processing a request within 1 hour of its submission. You can track the queue time in the Testing Farm metrics dashboard (testing-farm/docs/root!297).

Added an FAQ entry on finding the core dumps collected by the tmt coredump check, which the check results do not link yet (testing-farm/docs/root!283).

The bare metal documentation now describes how Red Hat ranch routes bare metal requests to AWS bare metal first and falls back to Beaker (testing-farm/docs/root!295).

We would like to thank the following external contributors for contributing to Testing Farm.

List of important packages bundled in the worker image.

Number of processed requests in the last 30 days was 291446.

The average error rate for Testing Farm was approximately 3.03%.