RFD3 - Testing Farm Release Process

Status: 💡 planned

This RFD proposes changes to how Testing Farm is deployed as a whole to make the release process unified and automated across all Testing Farm components.

Introduction

Current state contains severel issues. Each Testing Farm component is deployed in a different manner and there is a lack of proper versioning (except for worker). Brief description of how each component is deployed:

  • API — updated manually on the Testing Farm public server.

  • CLI — container is released after each change to main, pypi package is released ad-hoc.

  • oculus — deployed automatically after each change to main branch.

  • worker — manually tagging a built container and manually updating public ranch configuration.

  • worker-redhat — similar as worker.

  • citool-config (for each ranch) — deployed by running an Ansible playbook in gitlab.com/testing-farm/infrastructure repository.

  • dispatcher (for each ranch) — deployed manually.

Future state

Containers

Each Testing Farm component will be bundled in a container and properly versioned using tags. There will be two types of containers — data containers and code containers with the relation that e.g. worker container will mount citool-config container to read configuration from.

Testing Farm as a whole will consist of following set of containers:

  • quay.io/testing-farm/api

  • quay.io/testing-farm/dispatcher

  • quay.io/testing-farm/cli

  • quay.io/testing-farm/docs

  • quay.io/testing-farm/ui

  • quay.io/testing-farm/ranch-public

  • quay.io/testing-farm/ranch-redhat

  • quay.io/testing-farm/worker-public

  • quay.io/testing-farm/worker-redhat

Repositories

Testing Farm as a whole will consist of following repositories:

  • gitlab.com/testing-farm/api — new repo, will be created from the current gitlab.com/testing-farm/nucleus repository.

  • gitlab.com/testing-farm/dispatcher — new repo, will be created from the current gitlab.com/testing-farm/nucleus repository.

  • gitlab.com/testing-farm/cli

  • gitlab.com/testing-farm/docs

  • gitlab.com/testing-farm/ui — rename of gitlab.com/testing-farm/oculus.

  • gitlab.com/testing-farm/ranch-public — new repo, will be created from the current gitlab.com/testing-farm/infrastructure repository.

  • gitlab.com/testing-farm/ranch-redhat — new repo, will be created from the current gitlab.com/testing-farm/infrastructure repository.

  • gitlab.com/testing-farm/gluetool-modules

  • gitlab.com/testing-farm/worker-public — new repo, it will only retag the quay.io/testing-farm/gluetool-modules image.

  • gitlab.cee.redhat.com/testing-farm/worker-redhat — new repo, will be created from the current gitlab.cee.redhat.com/baseos-qe/gluetool-modules-internal repository.

  • gitlab.com/testing-farm/infrastructure

  • gitlab.com/testing-farm/deployments — new repo, described in a later section.

Versioning

Testing Farm versioning follows calver principles. It has YYYY-MM.MICRO format, e.g. 2024-01.1. The calendar YYYY-MM part is used for monthly Testing Farm releases and the MICRO part is for small hotfixes made throughout the month.

Currently, Testing Farm version as a whole corresponds to version of the worker. This will change with the new deployment process. The version of Testing Farm as a whole will be tracked in the new gitlab.com/testing-farm/deployments repository and each component will be also versioned using the same version format and release processes.

When it is time to release a new monthly version of Testing Farm and any component contains changes in main branch that are supposed to go to the new release, a new branch will be created with the name release/YYYY-MM from a commit that’s about to be released and this commit will be tagged as release/YYYY-MM.1. From now on, the development will continue in the main branch and if it’s necessary to create a patch for a released component, it will be cherry-picked from main to the branch release/YYYY-MM. Patches will be released by tagging commits in this branch as release/YYYY-MM.MICRO with the increasing MICRO number. Upon tagging a commit, a container image will be built with corresponding tag.

rfd3 repo branching

Deployment

Deployment will be performed inside a new repository gitlab.com/testing-farm/deployments. This repository will contain a branch for each Testing Farm deployment (production and staging) and each branch will have a file testing-farm.yaml with container names and their tags that reflect the deployment version of each component. Since some releases are with a small significance, there will be a notification field which indicates whether an automatic message with release notes should be sent to various channels.

In production branch, the testing-farm.yaml file will freeze all container tags to a specific version. When a component did not change for a particular monthly Testing Farm release, it is fine to keep an older version:

---
notification: false
containers:
  api: "quay.io/testing-farm/api:2024-01.1"
  ui: "quay.io/testing-farm/ui:2023-12.2"
  docs: "quay.io/testing-farm/docs:2024-01.1"
  cli: "quay.io/testing-farm/cli:2024-01.1"
  ranch-public: "quay.io/testing-farm/ranch-public:2024-01.1"
  ranch-redhat: "quay.io/testing-farm/ranch-redhat:2024-01.1"

In staging branch, by default, we will specify the latest tags to ensure that staging deployment is up-to-date with the latest container versions. This means that no actions will be needed to deploy staging but there will still be a possibility to freeze a component when it is necessary to do a revert when the latest is e.g. broken for any reason:

---
notification: false
containers:
  api: "quay.io/testing-farm/api:latest"
  ui: "quay.io/testing-farm/ui:2023-12.2"
  docs: "quay.io/testing-farm/docs:latest"
  cli: "quay.io/testing-farm/cli:latest"
  ranch-public: "quay.io/testing-farm/ranch-public:latest"
  ranch-redhat: "quay.io/testing-farm/ranch-redhat:latest"

The testing-farm.yaml file does not cover ranch-specific components, such as:

  • quay.io/testing-farm/dispatcher,

  • quay.io/testing-farm/worker-public, and

  • quay.io/testing-farm/worker-redhat.

These will be a part of ranch configuration located in gitlab.com/testing-farm/ranch-public and gitlab.cee.redhat.com/testing-farm/ranch-redhat repositories. Each repo will contain a dispatcher image and one worker image.