Improvement to PR testing in the konflux-ci/release-service-catalog repo

[Scott Hebert]

Hi everyone,

I'm excited to share some significant progress the Konflux-CI Release team has made in improving the robustness and stability of PR testing for the konflux-ci/release-service-catalog repository.

Our primary goal was to enhance the experience of merging a PR while maintaining or improving test coverage. We focused on ensuring that checks run reliably and PRs can be merged more quickly.

Previously, the end-to-end testing was a monolithic, flaky test suite. It was difficult to track the progress of a running PR test and challenging to debug issues locally outside of our continuous integration environment.

To address these challenges, we made the following key changes:

• Switched to a Bash-based test framework from the previous Go-based one.

• Made tests easy to run locally, simplifying the debugging process.

• Provided direct links to logs for real-time test progress tracking.

• Separated each test suite to be executed by its own IntegrationTestScenario.

• Tests are more self-contained and reduce any reliance on existing resources (including secrets) outside of namespaces.

• Tests are more rigorous when cleaning up after a test has completed with an option to skip cleanup for debugging.
  
  

To measure the effect of our improvements, we looked at two key metrics:

• check time: time from PR creation to all checks successful
• number of retries: number of retries requested to get checks to be successful

The results have been impressive:

• ✅ IMPROVEMENT: Average check time: Reduced by 90% (257.5 hours faster).

• ✅ IMPROVEMENT: Median check time: Reduced by 90% (78.2 hours faster).

• ✅ IMPROVEMENT: Average retries: Reduced by 70% (1.7 fewer retries).

We believe these improvements could benefit other teams facing similar PR testing challenges and encourage you to explore how our approach might be adapted for your own repositories.

Thanks,

Scott and the Konflux-CI Release Team

===================================================

Here is a detailed report (thanks to Cursor!):

📈 ANALYSIS RESULTS
===================
📅 Improvement date: 2025-07-22
📊 PRs before improvement: 49
📊 PRs after improvement: 37

⏱️  CHECK COMPLETION TIME COMPARISON
===================================
📈 Before improvement - Average: 270.7 hours, Median: 80.9 hours (39 successful PRs)
📈 After improvement  - Average: 13.3 hours, Median: 2.7 hours (26 successful PRs)

🔄 RETRY COMPARISON
==================
📈 Before improvement - Average: 2.2 retries per PR (28/49 PRs needed retries, 57.1%)
📈 After improvement  - Average: 0.5 retries per PR (9/37 PRs needed retries, 24.3%)

💬 Before improvement - 26/49 PRs had /retest comments (53.0%)
💬 After improvement  - 7/37 PRs had /retest comments (18.9%)

✅ IMPROVEMENT: Average check time reduced by 257.5 hours (90.0% faster)
✅ IMPROVEMENT: Median check time reduced by 78.2 hours (90.0% faster)
✅ IMPROVEMENT: Average retries reduced by 1.7 (70.0% fewer retries)

[Jeff Burke]

Nice Job! I know this was not easy. Thank you and the Release Team for the investment. 

Thanks,
Jeff

--
You received this message because you are subscribed to the Google Groups "Konflux CI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to konflux+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/konflux/dd0f153f-ac21-4333-9ab2-7784e6725746n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bill Rainford]

I’m very happy to see this come together. It took a lot of work but the benefits are clear. Thank you to Scott and the Release Team for making this happen.

Bill Rainford

SENIOR ENGINEERING MANAGER

RHTAP, GitOps & Konflux (Enterprise Contract, Release, Integration)

Portfolio & Delivery

Red Hat, Lowell, MA, USA

brai...@redhat.com    IM/Slack: brain

To view this discussion visit https://groups.google.com/d/msgid/konflux/CAO88Us0RFDV7iD0-VtAgqh6p7eOYd7JdSOeZcCDaxPbQBANu2A%40mail.gmail.com.