COSI API review - Status `Conditions` discussion

[Blaine Gardner]

Hi all!

The sig-storage Container Object Storage Interface project is getting an API review from Joel Speed who does broader k/k API reviews.

https://github.com/kubernetes-sigs/container-object-storage-interface/pull/199

Here is COSI’s v1alpha2 KEP PR reference, for anyone who wants to look for more context: https://github.com/kubernetes/enhancements/pull/4599

One of the things Joel has suggested is for us is to use ‘Conditions’ in our status instead of the ‘ReadyToUse’ boolean and the timestamped error. However, I also understand that Conditions have been avoided in other sig-storage APIs in recent memory, so I’d like to open more discussion around this.

COSI has often used Volume Snapshotter as a reference when developing our API, which is where this boolean and this error originate. Xing reported that during development of the Volume Snapshotter API, there were discussions that included Michelle Au and Tim Hockin (CC’ed here) where Conditions were specifically avoided. If Michelle, Tim, or anyone has a memory or notes from those past discussions, it would be helpful for this conversation. Xing has noted that one aspect of the prior conversation was that the Conditions should avoid being used to implement a state machine.

My goal is to arrive at a consensus in sig-storage about whether or not it is a continued recommendation to avoid status Conditions in storage APIs, along with reasoning to support this since it is opposite of k/k’s recommendation.

Thank you all for you time and inputs on this topic!

Blaine

-----

I’ll separate my own thoughts on this topic here:

Based on my interactions with Joel, I believe it is in the better interest of COSI to take Joel’s feedback and include conditions in our APIs. The boolean would become a “Ready” condition, and the error would become a to-be-named error condition. 

For errors, I believe that Conditions will be helpful for COSI to distinguish between controller errors and sidecar errors when debugging BucketAccess, which is passed from one to the other during its lifetime.

For the “Ready” condition, I believe Conditions will be more helpful for understanding the system state. Something we have seen is that some object storage backends rotate bucket keys automatically with some frequency. A COSI reconcile failure after such a rotation would merit transitioning from ready to non-ready even after initial provisioning is successful. It might be beneficial for COSI to distinguish between initial provisioning success versus follow-up success. We have also observed that self-hosted object stores can change endpoints over their lifetimes. For example, an admin may update an HTTP store to use TLS, which requires updating the user-facing view to a new HTTPS endpoint. If COSI were to encounter an error during the reconcile that provides this update, it would be beneficial for admins to be made aware.

As a note against the “Ready” Condition, we will have to take care to clarify what “Ready” means, and we may want to be more specific than just “Ready”. It is straightforward for the COSI system to determine whether initial resource provisioning was successful or not. It is more complicated to accurately understand whether the resource is *currently* ​ ready when considering the cases I mentioned above where authentication or location info is changing. It seems useful to me to separate the ideas of “this succeeded once” from “this failed recently and may (or may not) affect availability.”

I think that the current API could support these cases, but I think it would be useful to rename the ‘ReadyToUse’ boolean to ‘InitialProvisionSuccessful’ or something similar. This would indicate that the resource exists, even when there are subsequent reconcile issues. The existing error type could provide some hint that a subsequent provision encountered an error that might (or might not) limit accessibility.

Any suggestions on how to adequately separate initial versus follow-up readiness would be very welcome.

Again, many thanks for your attention!

Blaine

[Tim Hockin]

In general boolean fields are a terrible idea.  Conditions are better, though still imperfect in some ways.  I do not recall why we might have avoided Conditions, especially in favor of a naked bool.

"Ready" has been problematic over and over, in particular when we want to change or sharpen what it means, ESPECIALLY when multiple input signals add up to "being ready".  Gateway API invested in a family of conditions which are more precise.  Endpoints had to add new conditions which capture the more specific individual signals.

Tim

[Blaine Gardner]

Thanks for the input, Tim. 

I’m curious if you would recommend that we avoid a ‘Ready’ condition. 

With COSI’s v1alpha2 API, we haven’t designed interfaces for drivers to report on system readiness end-to-end. E.g., when the system is offline or when a key is revoked on the backend. I also think this is out of scope for what we could plan for in v1alpha2, though I would be interested in pursuing this more after critical features are ready.

Given the current scope of COSI v1alpha2, I think the best ‘Ready’ definitions we could achieve would be:

• ‘False’ - resources are initialized with ‘Ready: False’ before provisioning (resource definitely isn’t ready)
• ‘True’ - upon any successful reconcile where the driver reports success on the RPC call (driver reports ready)
• ‘Unknown’ - after any RPC call reports an error (we don’t know if the error is temporary or permanent, but the message could help users/admins/devs determine that for themselves)

IMO, this would be sufficient for us for now, but I’d appreciate thoughts. From a system perspective, ‘Ready’ is would be proxy for whether the reconcile was successful (True) or not (Unknown) rather than a representation of whether it’s ready for consumption end-to-end. 

Our ‘ReadyToUse’ boolean, which I expect we will all agree to replace with a condition, originally meant that the resource was provisioned successfully once. It was designed not to revert to false upon reconciliation errors. To represent that boolean as a condition, I think something like ‘Provisioned’ would be helpful. In the absence of a clear end-to-end readiness metric, ‘Provisioned: True’ with another condition reporting an error would hint that the error is likely temporary.

In the interest of keeping things appropriately simple for our alpha, that might leave us with 2 useful conditions:

• ‘Ready’ (usually True/Unknown) - for reporting driver provision success/failure
• ‘Provisioned’ (True/False) - for reporting whether the backend resource was ever created in the past

I’m also still unsure if controller reconciliation issues (e.g., failed Get() or Update() errors) should be reported as ‘Ready: Unknown’ in this plan, but we can dig into that if/as needed.

Thanks again,

Blaine

From: Tim Hockin <tho...@google.com>
Date: Thursday, December 11, 2025 at 15:52
To: Blaine Gardner <Blaine....@ibm.com>
Cc: sig-s...@kubernetes.io <sig-s...@kubernetes.io>, xingy...@gmail.com <xingy...@gmail.com>, ms...@google.com <ms...@google.com>, Blaine Gardner <brga...@redhat.com>, mateusz.u...@gmail.com <mateusz.u...@gmail.com>, Joel Speed <jsp...@redhat.com>
Subject: [EXTERNAL] Re: COSI API review - Status `Conditions` discussion

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

 

[Tim Hockin]

On Wed, Dec 17, 2025 at 8:18 AM Blaine Gardner <Blaine....@ibm.com> wrote:

Thanks for the input, Tim. 

I’m curious if you would recommend that we avoid a ‘Ready’ condition.

It depends - can you be more precise?  What EXACTLY does "ready" mean?  Are you sure that every consumer is going to use it the same way?  What about likely evolutions?

Let me illustrate by example.  Endpoints has a "ready" field.  Historically that means that the endpoint's pod is ready.  Then we decided that it would be good to serve from endpoints which are terminating if that is all that's available (e.g. a daemonset).  We "fixed" ready to mean "passing readiness, regardless of terminating", but that isn't enough information and it constitutes a breaking change for some consumers.  So we ultimately had to decompose it into constituent signals: "serving" and "terminating".  Now we can tell the difference between "serving && !terminating" vs. "serving && terminating", while consumers of "ready" are unchanged.  If we need to decompose it further we can, but it was somewhat painful.

 

With COSI’s v1alpha2 API, we haven’t designed interfaces for drivers to report on system readiness end-to-end. E.g., when the system is offline or when a key is revoked on the backend. I also think this is out of scope for what we could plan for in v1alpha2, though I would be interested in pursuing this more after critical features are ready.

End-to-end is contextual -- availability may be regional or zonal.  Getting info about keys can also be contextual - not every storage impl is going to use fixed secrets, they might be short-lived tokens assigned to workloads.  So what does "ready" mean?  

Another example: Gateway API decomposes it into 3 more precise signals: https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1/gateway_types.go#L963-L967

[Hemant Kumar]

On Thu, Dec 11, 2025 at 6:14 PM 'Tim Hockin' via sig-storage <sig-s...@kubernetes.io> wrote:
>
> In general boolean fields are a terrible idea.  Conditions are better, though still imperfect in some ways.  I do not recall why we might have avoided Conditions, especially in favor of a naked bool.
>

We have not avoided conditions as such , but IIRC - the case was (coming fresh from resizing and volumemdify reviews), we should not use conditions to drive and record state machine transitions. For example for resizing - a PVC may go between various states (ControllerResizeFailed, NodeResizePending etc), but we decided *not* to use conditions to drive this statemachine. For example, if a PVC expansion failed, users can recover to old PVC size or VAC, but we aren't checking conditions to decide if that is safe to do (we have a separate enum like field for that).

[Tim Hockin]

State-machines in general make for bad APIs which are hard to evolve - extra caution.  Conditions were intended to represent orthogonal "facts" (originally intended to be negative things, but then "Ready" appeared).  Obviously, given more than 2 or 3 orthogonal facts, not all combinations are intended to be valid.  The goal was that consumers of Conditions know which things they want to know and can ask about those things, rather than inferring them from some macro "state", whose definition might change over time (e.g. what does it mean to be "ready").

[Mateusz Urbanek]

Hi!

After reading the discussion, I think I agree with the "status.ready"
being a bad idea. I see the benefit of moving to conditions and
expressing the state this way.

Blaine, what about setting a few minimal/basic conditions as of now?

On Bucket:
- "Provisioned": The Driver has been successfully instructed to create
the bucket. We are not asserting it "works", only that we finished our
job.

On BucketClaim:
- "Provisioned": The Driver has been successfully instructed to create
the bucket. We are not asserting it "works", only that we finished our
job.
(I'm not sure if we want to copy the condition from the Bucket?)
- "Bound": The Claim has been matched to a Bucket and a BucketClass.

On BucketAccess:
- "Accepted": The referenced Claim exists, and the BucketAccessClass
allows this access.
- "Minted": The driver returned credentials. We hold the payload.
- "Published": The Secret object was successfully written to the
user's namespace.

In future we can extend this by adding things like "Deleting" to
Bucket/BucketClaim, "Rotating" to BucketAccess, etc.

Sincerely
Mateusz Urbanek

[Blaine Gardner]

After reviews and feedback from Joel and Tim that press for more thought into what statuses (especially “Ready”) mean, and a quick meeting with Hemant, I have arrived at a final draft for our conditions. I’ll summarize in the email here.

There are many more details in this GH issue for anyone who wants to dig in or get more info about what I’ve written below:

*
Detailed stakeholder needs identified in this comment: https://github.com/kubernetes-sigs/container-object-storage-interface/issues/207#issuecomment-3667263662
*
The 4th draft corresponding to this email (and future discussion/iteration) occurs below the comment linked above

Hemant brought up a good point about having a clear status that represents readiness, even if it isn’t perfectly accurate to the backend state. He stressed that it's important for this status to be purely monotonic. From my understanding, in the PV/PVC world and Volume Snapshot world, once a resource is successfully provisioned once, it is counted as ready for users. A PV can transition from there to a failed state, but that failed state is final; there is no transitioning back to provisioned.

Hemant brought up cloud platform per-resource billing as a case where this monotonicity is critical. An unprovisioned resource is not billable. A provisioned resource is billable. A failed resource is no longer billable. But the resource should never flap between billable and unbillable states because this would make billing calculations substantially more difficult.

Because the “Ready” condition can be problematic, I’ve proposed that we use a “Provisioned” condition for this instead. Provisioned would be “Unknown” initially. It would transition to “True” after a successful, valid RPC provision. It can transition to “False” if COSI is able to confidently determine that provisioning is impossible or if a resource is definitively lost forever. Once “False”, this is final.

I also identified 2 other conditions that are important for helping end users and administrators troubleshoot issues, especially issues that come up after initial provisioning (day 2). These don’t strongly correlate to readiness in a systemic way, and they are not monotonic.


*
“ProvisionFailed” (negative polarity) reports issues with RPC provisioning calls. This could be as minor as a temporary DNS outage or as bad as a total backend system loss. Storing this status is important for debugging.
*
“ResourcesValidated” (positive polarity) reports when the resource and any referenced resources are in a good state where COSI is able to make RPC calls. Referenced resources are protected, but if they are forcefully removed, this will help debug why.

Any minor issues that come up during reconciliation (like kube API throttling or resource update conflicts) are not important to represent on conditions. That is my understanding, at least. They will be emitted as Events (and obviously in logs) to be helpful.

Hemant recommended representing a first-class “Provisioned” status as an enum in the status. After thinking on that more, this sounds similar to “Phase” which is recommended against in the Kube API Conventions. I think it may be best to plan to omit this, but if anyone recommends otherwise, please chime in.

I’ll begin to make KEP changes based on this (final?) draft. If this plan sounds like it has issues, please chime in here or in the linked issue.

Thanks again everyone for the input, and happy holidays, happy New Year, and happy PTO!
Blaine