Testing Farm Team Gathering 2023

Testing Farm Team is working virtually for the whole year. We are currently distributed between 3 countries - Czech Republic, Austria and Israel. We are quite happy with our current remote working style, and we feel well productive. Nevertheless, we try to meet each year in-person to strengthen our team and discuss planning topics, hack and spend time together.

This year we met a bit later, our annual team gathering took place in Brno in the first week of October. Our meetup was hosted in the great Red Hat Brno office premises in Technology Park.

We spent the whole week together, and we want to share with you some details of our time. We usually spend 4 days working, and reserve one day for leisure and non-working activities.

On the first day our team lead Miroslav Vadkerti kicked off the day looking back on the last year for Testing Farm. In total, we believe we had a very successful year.

Testing Farm, our main service, has surpassed projected 700k test requests a year, growing from 500k. We onboarded 84 new users primarily from the field of Linux engineering, but we have grown over our main use case of testing operating systems. Strimzi and Debezium open-source projects are now using Testing Farm to drive their testing and the service helped to overcome some struggles they had with GitHub Actions, Azure and testing against some proprietary Oracle bits.

Significant improvements were made to the service, and we got some great contributions from various Red Hat teams and community. The Cockpit team revamped our Oculus results viewer, significantly improving the user experience. The Testing Farm CLI was substantially improved and our users now get an easy way to reserve testing resources from Testing Farm for debugging their testing issues.

We are now managing more parts of our infrastructure, including the Kubernetes cluster, with infrastructure as code and GitOps. Integration testing was hugely improved, making it possible to spot issues early and stabilize our releases. In total we were able to pull off 6 Testing Farm releases, 3 BaseOS CI releases and unbelivable 22 Artemis releases.

We automated our most complex process of updating composes, and we can now update RHEL images internally every day. Thanks to migrating to Slack we significantly improved our communication and support for our users.

For the end of the year, we plan to make several major improvements to the Testing Farm, including:

Next year, our deliverables include:

Our second day at the team gathering was exclusively reserved for fun and team bonding, away from the usual office buzz.

After the fun activities of the previous day, we were looking forward to some serious work and study.

In the morning we reviewed all the aspects of Testing Farm:

In the afternoon we held an "open hours" session where anyone from other teams could come and talk to us.

At the end of the session, Jan gave a great demo of a PoC multi-host pipeline, during which discussed the general logic and corner cases.

The third day of our team gathering marked a significant moment, as it was not just about our team but also about fostering connections with others. The highlight of the day was a grand lunch at Portoriko restaurant. This meal was special because we were joined by our friends from various other teams, including the Packit team, tmt developers, users from Strimzi and Debezium, and some of the Linux QE folks. Discussions ranged from recent technological advancements to personal hobbies, creating a tapestry of stories and experiences. The casual setting allowed everyone to relax and connect on a personal level, building relationships that go beyond the confines of professional roles.

Next on the list to discuss was BaseOS CI. It was decided that we limit adding new features or improvements to BaseOS CI, putting it in maintenance mode. It is expected that eventually, all BaseOS CI use cases will be migrated to Testing Farm. Rather than perfect the BaseOS CI IaC, we will take backups and restore if needed.

Things that were discussed for improvement were:

Ondrej demonstrated the auto-scaling functionality that was implemented for BaseOS CI workers and discussions were had around:

On this lovely Autumn day, we also discussed secret management and worked on improving documentation.

We got into Antora hacking a bit more and are uniting our documentation tree, which was planned for some time, and now is becoming a reality. We believe this aid in discoverability of information for users, contributors, and new members to Testing Farm, as the documentation will be less fragmented and there will be fewer places to look.

In our "contributability" discussions, we created a vision of an overall goal towards enabling contribution and individual goals for each git repository, as very often git repositories are the gateway developers have to a project. It seems that "reading the code" really is widely practiced and we want to honor this great engineering tradition.

On the last day, we spent part of the morning on the rooftop of one of the Red Hat office buildings with the fresh air and sunshine until we could no longer see our laptop screens.

We discussed how to improve planning with use of issue labels, created a definition of done and an acceptance criteria for issues, and decided to pursue making most of our issues public.

We also discussed what we would do if we all won the lottery, or if one of us won the lottery - AKA: The Bus Factor. All critical services that Testing Farm needs were listed. Which ones Testing Farm Team was responsible for, and what failures would result in outage were discussed. We created a criteria for risk mitigation, which includes a criteria for acceptable disaster recovery process and documentation. Listed were also services that we rely on, but are not responsible for.

Lastly, we disagreed.

This was an experimental exercise in which:

The presenter can practice being disagreed with, listening to and considering opposing ideas. The "disgreer" can practice giving objections clearly and politely.

We want to have a healthy flow of ideas and make sure that everyone feels comfortable disagreeing and being disagreed with.

After some friendly chatting and farewells we parted ways, happy to have seen each other and inspired by the progress made towards a stable robust software testing service.