Testing Farm Team Gathering 2023


Testing Farm Team is working virtually for the whole year. We are currently distributed between 3 countries - Czech Republic, Austria and Israel. We are quite happy with our current remote working style, and we feel well productive. Nevertheless, we try to meet each year in-person to strengthen our team and discuss planning topics, hack and spend time together.

This year we met a bit later, our annual team gathering took place in Brno in the first week of October. Our meetup was hosted in the great Red Hat Brno office premises in Technology Park.

We spent the whole week together, and we want to share with you some details of our time. We usually spend 4 days working, and reserve one day for leisure and non-working activities.

Day 1

On the first day our team lead Miroslav Vadkerti kicked off the day looking back on the last year for Testing Farm. In total, we believe we had a very successful year.

Testing Farm, our main service, has surpassed projected 700k test requests a year, growing from 500k. We onboarded 84 new users primarily from the field of Linux engineering, but we have grown over our main use case of testing operating systems. Strimzi and Debezium open-source projects are now using Testing Farm to drive their testing and the service helped to overcome some struggles they had with GitHub Actions, Azure and testing against some proprietary Oracle bits.

Significant improvements were made to the service, and we got some great contributions from various Red Hat teams and community. The Cockpit team revamped our Oculus results viewer, significantly improving the user experience. The Testing Farm CLI was substantially improved and our users now get an easy way to reserve testing resources from Testing Farm for debugging their testing issues.

We are now managing more parts of our infrastructure, including the Kubernetes cluster, with infrastructure as code and GitOps. Integration testing was hugely improved, making it possible to spot issues early and stabilize our releases. In total we were able to pull off 6 Testing Farm releases, 3 BaseOS CI releases and unbelivable 22 Artemis releases.

We automated our most complex process of updating composes, and we can now update RHEL images internally every day. Thanks to migrating to Slack we significantly improved our communication and support for our users.

For the end of the year, we plan to make several major improvements to the Testing Farm, including:

  • Multi-host testing will be out with our 2023-10.1 release.

  • A refactored API server based on Fast API is coming in 2023-10.1 release.

  • Secrets support for Packit is planned in the 2023-11.1 release.

  • The Testing Farm UI is expected to be launched before Christmas, scheduled in the 2023-12.1 release.

  • We plan to improve the security posture, by introducing RBAC roles for our users, planned for 2023-12.1 release.

Next year, our deliverables include:

  • Fedora and RHEL SSO support, to make onboarding much easier.

  • Support for IBM Cloud, to gain access to s390x and ppc64le architectures besides Beaker.

  • Support for GCP and Azure being added by the Leapp team.

Day 2 - Adventure and Team Building Beyond Work

Our second day at the team gathering was exclusively reserved for fun and team bonding, away from the usual office buzz.

Morning Trek

We kicked off the day with an invigorating walk, starting from our BRQ office. The destination: the scenic trails of Medlanky Hill. The few kilometers hike was not just a physical activity, but also an opportunity to appreciate the beauty around us. The picturesque views along the way were a refreshing change from our daily screens, as you can see from the attached photos.


A Vietnamese Lunch

Post the walk, we treated ourselves to an authentic Vietnamese lunch. The restaurant we chose is renowned for its traditional flavors and did not disappoint. We indulged in a variety of dishes, the highlight being the Vietnamese coffee, a perfect blend of strong and sweet, a must-try for any coffee aficionado.

vietnamese lunch
vietnamese coffee

Go-karts in Modrice

The afternoon was all about adrenaline and friendly competition. We headed to an outdoor go-kart track in Modrice, boasting a 400-meter circuit. The challenge: four 15-minute races. The thrill of speed, coupled with the spirit of competition, brought out a different side of everyone. It was exhilarating to see team members showcasing their driving skills, revealing hidden talents outside of the work sphere.

go karts 1
go karts 2
go karts 3

Laser Game

Our final activity of the day was a laser game in a renowned Brno arena. Split into two sessions, we started with a team deathmatch, fostering collaboration and strategy within smaller groups. The climax was the single person death match, a true test of individual skill and agility. The arena’s setup and the game’s immersive experience brought out a mix of competitiveness and camaraderie among the team.

laser game


The packed day, filled with diverse activities, was not just about fun, but also about strengthening bonds and building our team outside the confines of work. Events like these are crucial in creating a cohesive team, fostering a sense of belonging and understanding among members. The day’s experiences left us with not just memories but also lessons in teamwork, resilience, and the joy of shared experiences. As we head back to our routines, these moments will undoubtedly enrich our collaboration and work culture.

Day 3 - Overview of Testing Farm, Open Hours, Dinner with Testing Farm

After the fun activities of the previous day, we were looking forward to some serious work and study.

In the morning we reviewed all the aspects of Testing Farm:

  • How the service works

  • How it is deployed

  • The release procedure

  • How Testing Farm’s test systems differ from BaseOS CI’s

  • What big features we are working on

In the afternoon we held an "open hours" session where anyone from other teams could come and talk to us.

  • SST QE leads

  • BaseOS QE

  • Packit team

  • Reviewed priorities for Q4 & Q1, checking they aligned with our users' expectations

At the end of the session, Jan gave a great demo of a PoC multi-host pipeline, during which discussed the general logic and corner cases.

The third day of our team gathering marked a significant moment, as it was not just about our team but also about fostering connections with others. The highlight of the day was a grand lunch at Portoriko restaurant. This meal was special because we were joined by our friends from various other teams, including the Packit team, tmt developers, users from Strimzi and Debezium, and some of the Linux QE folks. Discussions ranged from recent technological advancements to personal hobbies, creating a tapestry of stories and experiences. The casual setting allowed everyone to relax and connect on a personal level, building relationships that go beyond the confines of professional roles.


Day 4 - BaseOS CI Discussion, Auto-scaling, Documentation

Next on the list to discuss was BaseOS CI. It was decided that we limit adding new features or improvements to BaseOS CI, putting it in maintenance mode. It is expected that eventually, all BaseOS CI use cases will be migrated to Testing Farm. Rather than perfect the BaseOS CI IaC, we will take backups and restore if needed.

Things that were discussed for improvement were:

  • Speed up guest-setup

  • Container tft testing

Ondrej demonstrated the auto-scaling functionality that was implemented for BaseOS CI workers and discussions were had around:

  • Max number of workers

  • What metrics to gather

  • What data to log

On this lovely Autumn day, we also discussed secret management and worked on improving documentation.

We got into Antora hacking a bit more and are uniting our documentation tree, which was planned for some time, and now is becoming a reality. We believe this aid in discoverability of information for users, contributors, and new members to Testing Farm, as the documentation will be less fragmented and there will be fewer places to look.

In our "contributability" discussions, we created a vision of an overall goal towards enabling contribution and individual goals for each git repository, as very often git repositories are the gateway developers have to a project. It seems that "reading the code" really is widely practiced and we want to honor this great engineering tradition.

Day 5 - Project Management, The Disagreement Conversation Game

On the last day, we spent part of the morning on the rooftop of one of the Red Hat office buildings with the fresh air and sunshine until we could no longer see our laptop screens.

We discussed how to improve planning with use of issue labels, created a definition of done and an acceptance criteria for issues, and decided to pursue making most of our issues public.

We also discussed what we would do if we all won the lottery, or if one of us won the lottery - AKA: The Bus Factor. All critical services that Testing Farm needs were listed. Which ones Testing Farm Team was responsible for, and what failures would result in outage were discussed. We created a criteria for risk mitigation, which includes a criteria for acceptable disaster recovery process and documentation. Listed were also services that we rely on, but are not responsible for.

Lastly, we disagreed.

This was an experimental exercise in which:

  • A list of short articles/essays/text is available for each participant to choose from.

  • Each participant takes a turn being the presenter, summarizing the conclusion of the article (accurate or not).

  • Each audience member takes a turn disagreeing with the conclusion/idea/concept/etc presented.

The presenter can practice being disagreed with, listening to and considering opposing ideas. The "disgreer" can practice giving objections clearly and politely.

We want to have a healthy flow of ideas and make sure that everyone feels comfortable disagreeing and being disagreed with.

After some friendly chatting and farewells we parted ways, happy to have seen each other and inspired by the progress made towards a stable robust software testing service.