Testing Farm Contribution Guide

Software

Poetry

All our Python projects MUST use Poetry for packaging. It is required to use poetry version 1.4.2.

To install Poetry use the following command:

curl -sSL https://install.python-poetry.org | python3 - --version 1.4.2

Ansible

It is required to use Ansible version 2.14.* both in development environment and Python projects.

To install Ansible use the following command:

sudo dnf install ansible-core-2.14.*

Exceptions

For the infrastructure repository it is required to use Ansible version 2.15.* due to this bug.

Goss

It is required to use Goss version v0.3.21.

To install, download the Goss binary according to your system from GitHub releases.

Style

Readability and consistency are essential to maintaining a healthy codebase. In this section we documented the best practices to follow when contributing to our codebase.

Documentation

When writing documentation, prefer one sentence per line. For more details, see here.

Templates

When working with templates, we recommend using spaces to separate different elements, such as variables, operators, filters, etc. This helps to improve the readability and maintainability of our templates and ensures that our template code is easy to understand and modify.

This applies to all templating languages we use, including:

Here are some examples of how to use spaces correctly in templates:

  • Go template:

    {{ if eq .Status "pending" }}
  {{ $total := add .Quantity .Reserved }}
  {{ if gt $total 10 }}
    The total quantity is greater than 10.
  {{ end }}
{{ end }}

  • Jinja2 template:

    {% if user.is_authenticated %}
  {% for item in cart %}
    {% set price = item.price * item.quantity %}
    <li>{{ item.name }} - {{ price | currency }}</li>
  {% endfor %}
{% endif %}

Here’s what not to do:

{{if eq .Status "pending"}}
  {{for item in cart}}
    {{set price=item.price*item.quantity}}
    <li>{{item.name}}-{{price|currency}}</li>
  {{end}}
{{endif}}

In this example, there are no spaces between the different elements of the template, making it difficult to read and understand.

Reviews

We appreciate all your contributions. Please make sure you follow these review rules we agreed upon in Testing Farm.

Comments Resolving

The contributor is obligated to resolve comments that had been addressed and require no further discussion.

In case more discussion is needed, leave the review comment open and wait for a reply from the reviewer.

If the reviewer is not responding, feel free to mention them in the comment.

If the reviewer is not responding even after being mentioned, please mention the following individuals:

  • mvadkert

  • happz

Required Approvals

We require at least 2 approvals for all our merge requests.

For RFDs, we require all Testing Farm Team members to provide an approval.

Release

The Testing Farm project is released biweekly on Monday. If there are urgent changes which need to be released quickly, a hotfix release may be created to address the important problem sooner.

The release versioning follows RFD3 - Testing Farm Release Process. Note that the release process is currently a work in progress and not yet completely finished.

Checklist

The release checklist is advised to be placed in the merge request description for the release notes.

### Preparation

- [ ] Create a new MR for the new release in the [documentation](https://gitlab.com/testing-farm/docs/root) project
- [ ] Make yourself aware of all the issues in the release by looking at the correct [Jira release](https://issues.redhat.com/projects/TFT?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page) and [GitLab milestone](https://gitlab.com/groups/testing-farm/-/milestones)
- [ ] Go through all of the items and make sure they are in `RELEASE PENDING` or `CLOSED` state in Jira or are `merged` in GitLab
- [ ] Sync with the Artemis release lead on the version of the Artemis to be deployed for the release, verify worker using correct Artemis API version to ensure all API features are available

### Release

> [!note]
> Check the [instructions](https://docs.testing-farm.io/Testing%20Farm/0.1/contributing.html#branching) for this section.

- [ ] Create a release branch in [nucleus](https://gitlab.com/testing-farm/nucleus) project and tag the release
- [ ] Create a release branch in [infrastructure](https://gitlab.com/testing-farm/infrastructure) project and tag the release
- [ ] Create a release branch in [gluetool-modules](https://gitlab.com/testing-farm/gluetool-modules) project and tag the release
- [ ] Create a release branch in [dispatcher](https://gitlab.com/testing-farm/dispatcher) project and tag the release
- [ ] Create a release branch in [ui](https://gitlab.com/testing-farm/ui) project and tag the release
- [ ] Tag a new CLI release (note that this uses a classic semver versioning currently) and release CLI in GitLab

### Deploying

> [!note]
> Check the [instructions](https://docs.testing-farm.io/Testing%20Farm/0.1/contributing.html#deployment) for this section.

- [ ] Announce on Slack that new Testing Farm release is about to be deployed
- [ ] Deploy new API, apply new database migrations, bump version in `/about` endpoint, observe [Sentry](https://red-hat-it.sentry.io) for possible issues
- [ ] Deploy new Dispatcher on Red Hat ranch
- [ ] Deploy new Dispatcher on Public ranch
- [ ] Deploy new UI
- [ ] Deploy new tmt web
- [ ] Deploy new config, jobs and image on a single Public ranch worker, observe [Sentry](https://red-hat-it.sentry.io) for issues
- [ ] Deploy all Public ranch workers, observe Sentry for issues
- [ ] Deploy new config, jobs and image on a single Red Hat ranch worker, observe [Sentry](https://sentry.engineering.redhat.com) for issues
- [ ] Deploy all Red Hat ranch workers, observe Sentry for issues
- [ ] Check for leftover queued requests

### Announcement

- [ ] Merge the release documentation
- [ ] Send out announcement email similar to [this one](https://groups.google.com/a/redhat.com/g/tft/c/hT3qOQfyUX0)
- [ ] Send and pin a short message summarizing the release to [#testing-farm slack channel](https://redhat.enterprise.slack.com/archives/C03BRN71JAF) and [tmt Fedora Matrix channel](https://matrix.to/#/#tmt:fedoraproject.org)

### Post-release
- [ ] Create a new GitLab milestone by running `gitlab create-milestone --name "TF YYYY-MM.Z"` in the infrastructure repository
- [ ] Create a new Jira release "TF YYYY-MM.Z"

Branching and tagging

First of all, check the difference between main and the latest release branch. If there are no changes since the last release, skip the repository.

To create a release branch, create a branch from main with name in release/YYYY-MM format. For release branch and tag naming, see the versioning section of RFD3 - Testing Farm Release Process. In case it is a second or further release this month, the release branch may already exist. In that case, hard reset the branch to main.

After creating the branch, create a release tag in release/YYYY-MM.MICRO format targeting the release branch. Be consistent with the MICRO version across repositories. That means, if you release release/2026-02.2, use .2 as MICRO even if a repository doesn’t have a release/2026-02.1 tag. Make sure the triggered CI automation has finished successfully.

CLI tagging

This process doesn’t apply to the CLI. To create the CLI release, skip branch creation and create a tag in vX.Y.Z format with a version higher than the previous one, targeting the main branch. Finally, create a release from the tag.

Deployment

Services

Several components (API, UI, dispatcher, tmt-web) are deployed as services on the public server. Every component has a source code directory in /opt, an executable in /usr/local/bin/, and a service file in /etc/systemd/service/.

Follow these steps to deploy a component:

  • Log in to the server as the testing-farm user. This is especially important for running poetry install later on.

  • Go to the source code folder of the component.

  • Check the git history for the presence of a local changes commit. Note the commit hash — it will need to be cherry-picked later.

  • Checkout the new tag with git fetch && git checkout <TAG_NAME>.

  • Cherry-pick the local changes commit on top if it exists, but do not push it.

  • Check if the component configuration needs any additional changes; fixup or squash them into the local changes commit.

  • Reinstall the virtual environment with poetry install. Make sure to run this under the testing-farm user, otherwise it might break the environment.

  • Stop the corresponding service.

  • For the API, migrate the database if needed with poetry run alembic upgrade head.

  • Start the service back. Watch Sentry and journalctl for possible issues.

Workers

Follow these steps to deploy image, jobs, and config on workers:

  • Go to your local infrastructure repository.

  • Make sure you have the latest developer environment by executing make clean && direnv allow.

  • In ansible/inventory/group_vars/testing_farm_redhat_workers.yaml and ansible/inventory/group_vars/testing_farm_public_workers.yaml, edit the release variables to point to the latest tag.

  • Choose one public worker candidate for test deployment with ansible-inventory --graph.

  • Deploy image, jobs, and config to the selected worker:

    ansible-playbook -e deployment=production -l <WORKER_NAME> -t update_config -t update_jobs -t update_image ansible/playbooks/testing-farm.yaml

  • Watch Nomad and Sentry for possible issues.

  • Repeat the previous three steps for one of the Red Hat workers.

  • If everything works, deploy changes to all workers with make update/pipeline/image && make update/pipeline/config && make update/pipeline/jobs.

  • Send a merge request with the changed release variable.