[kubernetes/kubernetes] [Storage] Add support for capacity isolation of shared partitions (#43607)

[Vish Kannan]

Based on kubernetes/community#306

This issue is meant to track work items and help collaboration between the community on adding support for capacity isolation of shared partitions.

Note: Priority of these features can change based on the number of collaborators.

v1.7

Note: @jingxu97 has already been making progress on some of the items for v1.7.

• Extend EmptyDir API Spec to include Size (Owner: )
• Define new resources in the API to surface node storage capacity (Owner: )
• Define new resources in the API to consume local storage (Owner: )
• Gate local storage as a resource with an alpha feature gate
• Kubelet expose local storage capacity for the primary partitions (Owner: )
• Kubelet supports & enforces storage reservation in Node Allocatable (Owner: )
• Kubelet enforces local storage limits by running a control loop that evicts pods that exceed their storage limits (Owner: )
• Kubelet specifies container logs and overlay limits via CRI to let runtimes apply limits (Owner: )
• Add a scheduler predicate that ensures that local storage is available while binding pods to nodes (Owner: )
• Add node e2es to ensure storage limit enforcement (Owner: )

---

v1.8

• Add cluster e2es to verify scheduler local storage support (Owner: )
• Require storage limits for all pods (or for Gu and Bu pods). Decision TBD (Owner: )
• Introduce global limit ranges (Owner: )
• Add local storage to downwards API (Owner: )
• Extend Quota and Limitranges to support storage (Owner: )
• Extend Metrics API to include storage and expose storage usage via kubectl top (Owner: )
• Support for rotating logs whenever containers' reach their logs storage limits (Owner: )

---

v1.9

• Add default I/O rate limits for logs from containers (Owner: )
• Minimize disk metrics update latency by introducing support for project quota (Owner: )
• Extend Node Problem Detector to expose shared partition failures (Owner: )

cc @jingxu97 @kubernetes/sig-node-feature-requests @kubernetes/sig-storage-feature-requests

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Vish Kannan]

@derekwaynecarr kindly vet this plan and add any missing items.

[David Oppenheimer]

cc/ @kubernetes/sig-scheduling-feature-requests

[Klaus Ma]

I'd like to contribute "scheduler predicate" if not owner now :).

[Timothy St. Clair]

This is pretty impactful, we should enqueue this for the next SIG meeting.

[Vish Kannan]

@k82cn some of the work items have already been been prototyped including the predicates you mention. Give us a couple of days to update owners on the list of work items.

[David Oppenheimer]

@timothysc I wonder if in-person discussions about this should be channeled to the "resource management working group" ... I'm a little worried about spreading discussion between node and scheduling.

[Vish Kannan]

@davidopp +1 on having discussions in the resource management working group.

[Klaus Ma]

@k82cn some of the work items have already been been prototyped including the predicates you mention. Give us a couple of days to update owners on the list of work items.

@vishh, thanks for your feedback, that's great :).

[Haoran Wang]

A naive question what's "resource management working group" ?

[David Oppenheimer]

A naive question what's "resource management working group" ?

We've mentioned it on the scheduling SIG mailing list. Please join the scheduling SIG mailing list
https://groups.google.com/forum/#!forum/kubernetes-sig-scheduling

The resource management working group was mentioned here
https://groups.google.com/d/msg/kubernetes-sig-scheduling/a-nvvjjP70M/ZeiC1BfHFAAJ

[Haoran Wang]

@davidopp Thanks, I am happy to take some task if needed.

[Timothy St. Clair]

@davidopp I'm ok with that, but we should PSA broadcast in the SIGs about topics like this to ensure interested parties are "in the loop".

[Timothy St. Clair]

/cc @kensimon

[Vish Kannan]

@timothysc https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/ytg3VQU7VYk/FxiqKRqoBAAJ

[Mik Vyatskov]

@vishh

Kubelet specifies container logs and overlay limits via CRI to let runtimes apply limits (Owner: @jingxu97)

What do you mean by that? I don't see anything about CRI and runtime applying limits in kubernetes/community#306 How does it interact with log rotation?

[Vish Kannan]

@crassirostris It will be an extension to the existing CRI API. kubernetes/community#306 is a high level user facing proposal. A detailed design doc will be published by @jingxu97 at-least for the v1.7 work items.

A short summary around logging is as follows:
To prevent I/O abuse, kubelet through the CRI needs to have runtimes rate-limit logging from container's stdout and stderr. This is not possible with all runtimes, and so it will be an optional features which some runtimes will support, for example containerd.
Kubelet needs to keep long running services available for weeks or months and not evict them because they used up all their log space. For this reason, kubelet will have to rotate logs on demand to keep the overall log usage within user specified limits.
Now the kubelet needs to interact with logging agents to prevent premature log rotation. This is an API I'm hoping you'd drive. Generate requirements around rotation and we can discuss having the kubelet meet those requirements.
Does this makes sense?
May be this topic requires a dedicated meeting?

[Yu-Ju Hong]

To prevent I/O abuse, kubelet through the CRI needs to have runtimes rate-limit logging from container's stdout and stderr. This is not possible with all runtimes, and so it will be an optional features which some runtimes will support, for example containerd.

@vishh is there an issue for this? We need to define what "rate limit" actually means. For example, journald just dropped log messages, and it could be frustrating to debug without relevant information.

[Vish Kannan]

@yujuhong None exists as of now. We do not plan on tackling logging until v1.8.
If it's urgent we can at-least spec out an API in v1.7 time frame.
I'm thinking of not dropping but instead just blocking. For example in the case of containerd, if a container is limited to 100 write IOPS, then the containerd logger can stall writes if that limit has been exceeded. This will result in application stalling which will hopefully be caught by application level metrics (using a latency metric around glog for example).

[Klaus Ma]

It seems all work items for 1.7 are done, should we move this to 1.8 milestone?

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Incomplete

@jingxu97 @vishh

Action required: This issue requires label changes. If the required changes are not made within 6 days, the issue will be moved out of the v1.8 milestone.

kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup'].
priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm'].

Additional instructions available here

[Jing Xu]

'kind/feature'
'priority/important-soon'

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Incomplete

@jingxu97 @vishh

Action required: This issue requires label changes. If the required changes are not made within 6 days, the issue will be moved out of the v1.8 milestone.

kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup'].
priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm'].

Additional instructions available here

—

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Incomplete

@jingxu97 @vishh

Action required: This issue requires label changes. If the required changes are not made within 5 days, the issue will be moved out of the v1.8 milestone.

kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup'].
priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm'].

Additional instructions available here

—

[NickrenREN]

/kind feature
/priority important-soon

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Complete

@jingxu97 @vishh

Issue label settings:

• sig/node sig/scheduling sig/storage: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/feature: New functionality.

Additional instructions available here

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Complete

@jingxu97 @vishh

Issue label settings:

• sig/node sig/scheduling sig/storage: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/feature: New functionality.

Additional instructions available here The commands available for adding these labels are documented here

[Dawn Chen]

I retarget this to v1.9 since all work items for 1.8 are done.

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Issue Needs Approval

@jingxu97 @vishh @kubernetes/sig-node-bugs @kubernetes/sig-scheduling-bugs @kubernetes/sig-storage-bugs

Action required: This issue must have the status/approved-for-milestone label applied by a SIG maintainer. If the label is not applied within 6 days, the issue will be moved out of the v1.9 milestone.

Issue Labels

• sig/node sig/scheduling sig/storage: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/feature: New functionality.

Help

• Additional instructions
• Commands for setting labels

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Issue Needs Approval

@jingxu97 @vishh @kubernetes/sig-node-bugs @kubernetes/sig-scheduling-bugs @kubernetes/sig-storage-bugs

Action required: This issue must have the status/approved-for-milestone label applied by a SIG maintainer. If the label is not applied within 5 days, the issue will be moved out of the v1.9 milestone.

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Incomplete

@jingxu97 @vishh

Action required: This issue requires label changes. If the required changes are not made within 2 days, the issue will be moved out of the v1.9 milestone.

kind: Must specify exactly one of kind/bug, kind/cleanup or kind/feature.

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Incomplete

@jingxu97 @vishh

Action required: This issue requires label changes. If the required changes are not made within 1 day, the issue will be moved out of the v1.9 milestone.

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Labels Incomplete

@jingxu97 @vishh

Action required: This issue requires label changes. If the required changes are not made within 0 days, the issue will be moved out of the v1.9 milestone.

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Removed

@jingxu97 @vishh

Important: This issue was missing labels required for the v1.9 milestone for more than 3 days:

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[Maxim Ivanov]

/lifecycle frozen

[Jing Xu]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Maxim Ivanov]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.

/lifecycle stale

[Maxim Ivanov]

/remove-lifecycle stale

[Maxim Ivanov]

/lifecycle frozen

[Elana Hashman]

/remove-kind bug

@jingxu97 where are we with this?

—
You are receiving this because you are on a team that was mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

[David Porter]

/cc

—

Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/43607/1147752584@github.com>