Integrations

This page describes how to integrate Testing Farm with other systems and services.

Jenkins

Jenkins is able to call Testing Farm API using its groovy Jenkinsfile. The key functionality is provided via Fedora CI pipeline library, which is available as a global shared library.

Before you start your experiments, please make sure you already have your API key generated (see onboarding). Following steps are valid at least for Jenkins 2.346.1.

Configure Jenkins

To make integration with Testing Farm work properly you need to install two following plugins:

Storing API key as credentials

Save your Testing Farm API key securely using Jenkins functionality for managing credentials and avoid hardcoding it directly into your Jenkinsfile.

To do it you need to access Jenkis configuration via Dashboard > Administration, then go into the Security section and enter Manage security. You need to click on domain (e.g. (global)) of the proper store you intend to use for keeping the credentials (do not click the store name, just the domain).

Here you can create a new secret item via Add credentials. For the intended Testing Farm Key use the type Secret text and specify its ID. It will be used later from groove code.

If you prefer configuration as a code coming from Jenkins plugin, you can use following syntax:

credentials:
  system:
    domainCredentials:
    - credentials:
      - string:
          description: "testing farm apikey"
          id: "testing-farm-api-key"
          scope: GLOBAL
          secret: "TESTING-FARM-API-KEY"

Fedora CI pipeline library

Global shared libraries are added via Dashboard > Administration > Configures system section. Then you need to find Global Pipeline Libraries and click on Add.

Fill name ideally in human-readable as we will import the library using this value later), Default version stands for git branch of Fedora CI pipeline library we intend to use (e.g. rh-stable).

The most important part is Retrieval method (select Modern SCM) and Source Code Management (select git). Then fill the URL of the library as Project Repository path.

Library caching can be configured via Cache fetched versions on controller and this Refresh time in minutes input. As a result, the library is not downloaded every time Jenkins job is executed.

Configuration as a code example follows:

globalLibraries:
- cachingConfiguration:
    refreshTimeMinutes: 1440
    defaultVersion: "rh-stable"
    name: "fedora-pipeline-library"
    retriever:
    modernSCM:
        scm:
        git:
            id: "1a8f6945-f39b-45fd-b7b6-e66e2a3dea81"
            remote: "https://github.com/fedora-ci/jenkins-pipeline-library"
            traits:
            - "gitBranchDiscovery"

Jenkinsfile

The most important step is to prepare Jenkinsfile. Key part of it is the specification of the request we intend to send to the Testing Farm API. Please read the the API. Syntax of request in Jenkinsfile is groovy based, but when it is sent the content is just wrapped into JSON expected by the Testing Farm API.

Jenkinsfile example is based on following Jenkinsfile provided by Fedora CI. Let’s check its key parts.

Library import and environment settings

Imports of global shared libraries are provided via the lib function. Please note, there is also a branch setting to be used after the @ character. This is not necessary if the value is not different from the value specified in the Default version setting provided by Jenkins UI.

libraries {
    lib("fedora-pipeline-library@rh-stable")
}

Access to Testing Farm API key is stored via Jenkins credentials functionality:

FEDORA_CI_TESTING_FARM_API_URL specifies the endpoint for API calls (it is the same no matter whether the public or Red Hat ranch is used).
On the contrary FEDORA_CI_TESTING_FARM_ARTIFACTS_URL sets the artifacts storage that differs based on ranch used.

See the details about artifacts store in documentation services section.

environment {
    TESTING_FARM_API_KEY = credentials('testing-farm-api-key')
    FEDORA_CI_TESTING_FARM_API_URL = "https://api.testing-farm.io"
    FEDORA_CI_TESTING_FARM_ARTIFACTS_URL = "http://artifacts.dev.testing-farm.io"
}

Request preparation and sending

Testing Farm supports two types of tests: TMT (using fmf — flexible metadata format) and STI tests. Example below uses TMT. The most important parts are the git repository containing the tests to be passed in the url parameter together with ref (branch, tag or commit).

def requestPayload = [
    api_key: env.TESTING_FARM_API_KEY,
    test: [
        fmf: [
            url: "https://github.com/teemtee/tmt/",
            ref: "main",
            path: ".",
            name "/tests/prepare/basic"
        ]
    ]

Example above triggers test plan with the name basic used for testing of TMT project itself.

The environments section allows us to determine details regarding the system to be provisioned. We can select its architecture and also compose (operating system) to be deployed here. List of available composes depends on ranch — here you can find the list of available composes in the public ranch.

environments: [
    [
        arch: "aarch64",
        os: [
            compose: "CentOS-Stream-8-aarch64"
        ]
    ]
]

Environment section is quite powerful. For example we can set the number of artifacts (packages from koji-build for example) to be installed on a provisioned system and a lot more.

After a request is sent we quickly obtain the initial response containing the structure of request itself and also the unique identifier — id — of the job which is going to be processed via Testing Farm. This id can be used for example to get information about the details of the request.

def initial_response = submitTestingFarmRequest(payloadMap: requestPayload)
tftRequestId = initial_response['id']

Using webhooks to obtain results

Webhooks are a preferable way to obtain the results of the job. We need to call the registerWebhook function first.

hook = registerWebhook()

And then use its url in notification section of Testing Farm API call.

notification: [
    webhook: [
        url: hook.getURL()
    ]
]

Finally we get the result of the job itself. job_response contains information about the run pointing us to artifacts with details about test results. Information about requests being invalid or failure of the job (for infrastructure reasons etc) this information is also reported here.

def job_response = waitForTestingFarm(requestId: tftRequestId, hook: hook)
echo "${job_response.apiResponse}"

It is strongly recommended to set agent to none so executor is not blocked until job finishes.

Jenkinsfile example

Example integrating Fedora CI pipeline library for call to Testing Farm API.

pipeline {
    libraries {
        lib("fedora-pipeline-library@rh-stable")
    }
    agent {
        label jSlaveLabel
    }
    environment {
        TESTING_FARM_API_KEY = credentials('testing-farm-api-key')
        FEDORA_CI_TESTING_FARM_API_URL = "https://api.testing-farm.io"
        FEDORA_CI_TESTING_FARM_ARTIFACTS_URL = "http://artifacts.dev.testing-farm.io"
    }
    stages {
        stage("Run tests") {
            steps {
                script {
                    hook = registerWebhook()
                    def requestPayload = [
                        api_key: env.TESTING_FARM_API_KEY,
                        test: [
                            fmf: [
                                url: "https://github.com/teemtee/tmt/",
                                ref: "main",
                                path: ".",
                                name: "/tests/prepare/basic"
                            ]
                        ],
                        environments: [
                            [
                                arch: "aarch64",
                                os: [
                                    compose: "CentOS-Stream-8-aarch64"
                                ]
                            ]
                        ],
                        notification: [
                            webhook: [
                                url: hook.getURL()
                            ]
                        ]
                    ]
                    def initial_response = submitTestingFarmRequest(payloadMap: requestPayload)
                    tftRequestId = initial_response['id']
                }
            }
        }
        stage('Wait for Testing Farm results') {
            steps {
                script {
                    def job_response = waitForTestingFarm(requestId: tftRequestId, hook: hook)
                    echo "Testing Farm response obtained!"
                    echo "${job_response.apiResponse}"
                }
            }
        }
    }
}

GitLab CI

You can integrate Testing Farm with GitLab CI to create Testing Farm requests from your GitLab CI jobs. This can be done by using the testing-farm CLI container image. See Testing Farm CLI for more details about the tool and the CLI container image.

Example contents of .gitlab-ci.yaml:

testing-farm-job:
  stage: test
  image: quay.io/testing-farm/cli:latest
  script:
    - testing-farm request --git-ref $CI_MERGE_REQUEST_REF_PATH --git-url $CI_MERGE_REQUEST_PROJECT_URL

It is required to pass your Testing Farm API key to the testing-farm CLI, it can be passed via an environment variable, this variable can be added in the repository settings. In your repository in the GitLab web UI, navigate to Settings → CI/CD → Variables → Add variable. As the Key, use TESTING_FARM_API_TOKEN, as the Value, use your Testing Farm API key.

GitHub Actions

Testing Farm can be integrated with GitHub using testing-farm-as-github-action.