Re: [kubernetes/kubernetes] [Feature Request] Support disk quota for docker runtime (#54384)

[Ning Xie]

Before we add support for configuring size command line argument we need to first think about:

• How to specify the value for size.
• Where to specify the value for size.
• This feature is only available for overlay2&&xfs. How do we detect this pre-requirement correctly.

Maybe @kubernetes/sig-storage-feature-requests @jingxu97 @vishh can give some advises.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Ning Xie]

/remove-lifecycle stale

[Bingshen Wang]

@andyxning Kubernetes 1.8 add Local ephemeral storage resource type. Is it possible to pass ephemeral-storage value to storage-opt size when primary partition support xfs+pquota.

[Jacky Wu]

In my opinion, I think ephemeral-storage is not a quota for POD, instead, it should be named with limitation.

As kubelet would like to evict pod if the ephemeral-storage limit exceeded, and adding storage-opt with docker run would set container overlay fs quota, and would throws no space left error if the quota is exceeded, rather than killing the pod. Currently, I think ephemeral-storage is also an option for disk resource limit.

Thanks.

[Yuquan Ren]

Is it possible to pass ephemeral-storage value to storage-opt size when primary partition support xfs+pquota.

I do not think we need to pass ephemeral-storage to storage-opt, since ephemeral-storage will also limit container overlay usage. but as @Colstuwjx said, we will evict the pod if container usage exceeds the limit.

[Bingshen Wang]

@Colstuwjx @NickrenREN Others resource limitations like cpu, memory is the hard limit, container can't use more than resource above limit value on the system/kernel layer. So those limitations not relate to pod eviction.
Since ephemeral-storage not use disk quota to implement limitation, the kubelet must evict pod when disk usage exceed ephemeral-storage limitation. If using disk quota to do limitation, applications in containers will got an error of no space left, And then applications can do something to decrease disk usage. I think this behavior will more consistent with others limitations.

[Yuquan Ren]

@BSWANG I agree kubernetes acts differently for container ephemeral-storage and cpu&memory limit.
But regarding the param passing, i want to say : ephemeral-storage is kind of different from cpu and memory. It only cares about the root partition, if an optional runtime partition is used, kubernetes will not calculate the image layer or writable layers usage of containers.

BTW: referring to the docker doc : https://docs.docker.com/engine/reference/commandline/run/#set-storage-driver-options-per-container, there are still some limits for storage-opt

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

—

[Robert Krawitz]

The problem with ephemeral storage is even worse than the du latency; it's possible for a pod (attached) to create a file and delete it, but keep the file descriptor around. Without quota support, any pod or container within it can use as much ephemeral storage as it wants and never get caught doing so.

Remove the unlink "/tmp/foo" and the pod gets evicted as it should (but not after chewing through an enormous amount of disk space, if it misses the reaper!)

diskhog.yaml.txt

[Matthew Wong]

@RobertKrawitz it looks like jingxu97 was working on "Local Ephemeral Storage Capacity Isolation" this but idk the current status kubernetes/features#361 , you may wish to contact her

[Nikhita Raghunath]

/remove-lifecycle stale

[Shinebayar G.]

Any support on this? I think this feature is definitely useful.

[Robert Krawitz]

In progress (not this issue per se); see #66928

[xuchen]

Hi @RobertKrawitz Thanks for the PR , I notice that PR will implement a quota enforcement right?
'--storage-opt size=xxx' will coming soon?

—
You are receiving this because you are on a team that was mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

[Jacky Wu]

I have been checked the document, and I think, for now, it's focusing on monitoring, not enforcement of quota limit.

From my sight, there is some considerations:

1. it should limit pod storage space, to avoid abusing of storage consuming in multi-tenant case, monitor is the basically thing, limit and enforcement is really needed for our users;

2. we are suffering from perf issue about using du to monitor, I want the LocalStorageCapacityIsolationFSQuotaMonitoring gone beta ASAP, but the beta criterion still need to be determined as @dashpole said ;

3. current implement of LocalStorageCapacityIsolationFSQuotaMonitoring directly use kernel and file system level features, I think it's okay for emptyDir and local storage, but it makes split of CSI and LocalStorage into two lines, I am wondering whether we should add a uniformed DiskQuota field, just like CPU and Memory, and let them implement the details;

Thanks @RobertKrawitz great job!

[Paco Xu]

If I understand correctly, this should be solved by kubernetes/enhancements#1029. Currently, it is an alpha-level feature.

I asked to promote it to beta and after some discussions, we think there are several things to do before promoting it to beta. But for this issue, I think we can close it as resolved. And it should be tracked in kubernetes/enhancements#1029.

See kubernetes/enhancements#1029 (comment) for more details.

Action Items in 1.22～1.23

• add some metrics or make more visibility
• benchmark the feature
• check e2e testings: e2e evolution

Action Items in 1.24-1.25

• promote it to beta

[Aditi Sharma]

/triage accepted

[Kubernetes Triage Robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

—

Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/54384/1421935018@github.com>

[Kubernetes Prow Robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/kubernetes/issues/54384/1421935075@github.com>