Testing Process

This page describes the testing process of a single request in Testing Farm.

The section does not properly describe the process for multihost testing.

Overview

Testing Farm request can execute tests in multiple test environments, depending on the request’s environment array.

Moreover, for each tmt plan or STI playbook the given tests are run in a separate test environment.

Testing Farm runs tests in up to 12 testing environments in parallel for each test request.

For example, for a test request with 2 architectures:

testing-farm request --git-url https://gitlab.com/testing-farm/tests \
                     --compose Fedora-Rawhide \
                     --arch x86_64,aarch64 \
                     --plan /testing-farm/system

Testing Farm’s request ID is stored in TESTING_FARM_REQUEST_ID enviroment variable directly on the provisioned system. There are also TESTING_FARM_TEST_TYPE, TESTING_FARM_GIT_URL and TESTING_FARM_GIT_REF set to provide additional info.

The test request body will contain two environments:

Each architecture has to be executed in a distinct test environment.

 
{
    "test": {
        "fmf": {
            "url": "https://gitlab.com/testing-farm/tests",
            "ref": "main",
            "name": "/testing-farm/system"
        }
    },
    "environments": [
        {
            "arch": "x86_64",
            "os": {
                "compose": "Fedora-Rawhide"
            }
        },
        {
            "arch": "aarch64",
            "os": {
                "compose": "Fedora-Rawhide"
            }
        }
    ]
}

And because the plan /testing-farm/system in the https://gitlab.com/testing-farm/tests repository contains multiple plans:

❯ tmt plan ls /testing-farm/system
/testing-farm/system/block-devices
/testing-farm/system/cpuinfo
/testing-farm/system/file-system-usage
/testing-farm/system/meminfo
/testing-farm/system/mount
/testing-farm/system/os-release

Testing Farm will run each tmt plan in all requested test environments.

The test execution schedule will look like this:

Fedora-Rawhide:x86_64:/testing-farm/system/block-devices
Fedora-Rawhide:x86_64:/testing-farm/system/cpuinfo
Fedora-Rawhide:x86_64:/testing-farm/system/file-system-usage
Fedora-Rawhide:x86_64:/testing-farm/system/meminfo
Fedora-Rawhide:x86_64:/testing-farm/system/mount
Fedora-Rawhide:x86_64:/testing-farm/system/os-release

Fedora-Rawhide:aarch64:/testing-farm/system/block-devices
Fedora-Rawhide:aarch64:/testing-farm/system/cpuinfo
Fedora-Rawhide:aarch64:/testing-farm/system/file-system-usage
Fedora-Rawhide:aarch64:/testing-farm/system/meminfo
Fedora-Rawhide:aarch64:/testing-farm/system/mount
Fedora-Rawhide:aarch64:/testing-farm/system/os-release

Test Environment Execution

Let’s dive into a single test environment execution.

Each test environment execution in Testing Farm consists from these phases:

The following table shows details about which component is responsible for the execution of the given phase.

Phase Responsible

provisioning of test environment

Testing Farm Worker (1)

preparation of test environment

Testing Farm Worker (2)

tests execution

tmt

results reporting

Testing Farm Worker
1 When the test environment is a container, tmt is responsible for this phase.
2 Preparation of test environment is skipped for containers, note that you can use prepare step in tmt instead.

Because tmt is used only for executing tests, it is not easy to fully reproduce the testing done by Testing Farm on your localhost. We are actively working on resolving the issue, in the future tmt should take over most of the responsibitlities from Testing Farm.

Test Environment Provisioning

This phase takes care of provisioning the test environment in one of the Testing Farm provisioning pools.

Testing Farm uses Artemis for provisioning bare-metal or virtual machines.

In case of containers, container provisioner plugin from tmt is used, and Testing Farm is not involved in provisioning.

Test Environment Preparation

The test environment preparation consists of these sub-phases:

  • pre-artifact installation

  • artifact installation (optional, depends if artifact installation requested)

  • post-artifact installation

We usually refer to this phase internally as guest-setup, as it takes care of setting up a test guest. You may find it under this name in pipeline logs.

pre-artifact installation

Pre-artifact installation is a set of preparation tasks run before artifacts are installed. They tasks are implemented via Ansible.

For implementation see:

artifact installation

Artifact installation takes care of reliably installing given artifacts specified in the test request.

Currently Testing Farm supports the following artifacts:

Artifact 🌍 Public Ranch 🎩 Red Hat Ranch

Fedora Koji Build (fedora-koji-build)

✅ supported

❌ not supported

Red Hat Brew Build (redhat-brew-build)

❌ not supported

✅ supported

Fedora Copr Build (fedora-copr-build)

✅ supported

❌ not supported

Yum Repository (repository)

✅ supported

✅ supported

Yum Repository File (repository-file)

✅ supported

✅ supported

post-artifact installation

Post-artifact installation is a set of preparation tasks run after artifacts are installed.

Test Execution

Test execution is done via tmt run.

When testing against a bare-metal or virtual machine, the provision step is overwritten with the connect plugin and the connection details of the provisioned machine by Artemis.

Results Reporting

During this phase Testing Farm archives all the test artifacts, creates JUnit and HTML results and switches the request to error or complete state.

The archiving of the artifacts and JUnit results generation is done after the results is switched into the final state. This can cause race conditions, we hope to fix these problems in Q2/2023 and Q3/2023.