Perma failing and continuously unhealthy jobs (call for action!)

[Davanum Srinivas]

Folks,

We have been struggling with keeping our test grids green and it has a bunch of consequences for everything including releases.

For example, here's a list of jobs that are currently RED for a while now (opened yesterday!) : 

• https://github.com/kubernetes/kubernetes/issues/109521

This is not a new problem and sig-testing has been trying to think about this, see:

• https://github.com/kubernetes/test-infra/issues/18601 ("Kubernetes CI Policy: create and enforce policy of removing continuously unhealthy jobs")
• https://github.com/kubernetes/test-infra/issues/18600 ("Kubernetes CI Policy: remove egregiously perma-failing jobs")
  

We also have the reliability KEP that wants to help push folks to do better:

• https://github.com/kubernetes/enhancements/pull/3139

For now this is a request for action from sig chairs and leads to look for things under their SIG that they want to clean up one way or another, by fixing issues or removing the unnecessary CI jobs. Look through these json files to start:

• http://storage.googleapis.com/k8s-metrics/failures-latest.json
  
• http://storage.googleapis.com/k8s-metrics/flakes-latest.json
  

Please prioritize this work! We should use the 2 weeks (the 1.24 pushed out) to clean up our house. If you need help, please come to sig-testing. 

thanks,

Dims

--

Davanum Srinivas :: https://twitter.com/dims

[Davanum Srinivas]

Folks,

To help you all dig into and prioritize fixing our CI jobs we now have web pages for each of the metrics we capture here:

https://storage.googleapis.com/k8s-metrics/index.html

Please dig into Broken/Failing/Flaking jobs for your sig(s), open issues and clean things up!

This will help us be more healthy with good ci signals, better quality and importantly more sustainable by conserving CI resources (save 💰 💰 💰 money!)

Wish you all a very Happy New Year 2026!

Thanks,

Dims