Perma failing and continuously unhealthy jobs (call for action!)

[Davanum Srinivas]

Folks,

We have been struggling with keeping our test grids green and it has a bunch of consequences for everything including releases.

For example, here's a list of jobs that are currently RED for a while now (opened yesterday!) : 

• https://github.com/kubernetes/kubernetes/issues/109521

This is not a new problem and sig-testing has been trying to think about this, see:

• https://github.com/kubernetes/test-infra/issues/18601 ("Kubernetes CI Policy: create and enforce policy of removing continuously unhealthy jobs")
• https://github.com/kubernetes/test-infra/issues/18600 ("Kubernetes CI Policy: remove egregiously perma-failing jobs")
  

We also have the reliability KEP that wants to help push folks to do better:

• https://github.com/kubernetes/enhancements/pull/3139

For now this is a request for action from sig chairs and leads to look for things under their SIG that they want to clean up one way or another, by fixing issues or removing the unnecessary CI jobs. Look through these json files to start:

• http://storage.googleapis.com/k8s-metrics/failures-latest.json
  
• http://storage.googleapis.com/k8s-metrics/flakes-latest.json
  

Please prioritize this work! We should use the 2 weeks (the 1.24 pushed out) to clean up our house. If you need help, please come to sig-testing. 

thanks,

Dims

--

Davanum Srinivas :: https://twitter.com/dims