Re: [kubernetes/enhancements] Local Ephemeral Storage Capacity Isolation (#361)

[Patrick Lang]

I mentioned this in the meeting today, but I wanted to add it here too. I think that apps must have a way to identify what quotas exist, and adapt to them. That would be a blocker from enabling enforcement later because if an app keeps exceeding quota and is killed repeatedly - that's bad. At the very least can we get this visible in a downward API if existing filesystem mechanisms won't work to discover the quota?

From a Kubernetes API standpoint, we need to be careful not to require an OS or cloud provider specific implementation. On Windows filesystems, quotas are not available so we would probably use a loopback volume to implement this. That preserves the Windows API behavior where if an app queries for the free space, it gets the space within that loopback volume.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Claire Laurence]

@jingxu97 @vishh Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[Kendrick Coleman]

Hello @jingxu97 @vishh , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. This will also require a KEP for inclusion

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[arunbpt7]

We try the feature by this scenario, and found that it can not limit the capacity of container.

The behavior you describe should work regardless of this feature. Make sure you have --root-dir set correctly. Docker reports its root directory to the kubelet, so as long as your images are stored on the same partition that contains /var/lib/docker (or whatever your docker root dir is), this should work correctly.

Hello ,

I would like to check if there any way to restrict Pods usage on ephemeral storage /var/lib/docker) regardless of mounted on node root fs or separate file system of /var/lib/docker .

Because the pods run time writable layers or logs growing up and that will fill out /var/lib/docker filesystem . This behavior is getting fill up the file system and stooping other pods to run.

It would be great if we restrict pods use a limited amount of ephemeral storage on cluster wide . eg: set 20G quota for PODS that mean each pods can use only 20GB on ephemeral storage , if need more space should use the PV. if there any possibility to do that

[David Ashpole]

Yes, you can do this with ephemeral storage. See the documentation. Make sure you have eviction enabled for both the "imagefs" and the "nodefs" (documentation).

[arunbpt7]

Yes, you can do this with ephemeral storage. See the documentation. Make sure you have eviction enabled for both the "imagefs" and the "nodefs" (documentation).

Thanks for the update , have defined ephemeral-storage request and limit in resources (spec.hard.requests.ephemeral-storage , spec.hard.limits.ephemeral-storage) on the deployment and verified that evictionHard: is enabled for "imagefs and "nodefs" on the node . but when when deploying the pod and it is not restricting the pod to use the defined ephemeral storage . when creating large file inside the container it is still able to create files more that the ephemeral-storage request and limit.

evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%

---

containers:
- name: busybox
image:
resources:
requests:
ephemeral-storage: "500Mi"
limits:
ephemeral-storage: "500Mi"

[David Ashpole]

Sounds like a bug. Feel free to open a separate issue and cc me, as this is for feature tracking.

[arunbpt7]

Sounds like a bug. Feel free to open a separate issue and cc me, as this is for feature tracking.

Thank you , have opened a new issue ( local ephemeral Storage limitation for pods in the cluster #1094)

[Kendrick Coleman]

Hi @arunbpt7 @jingxu97 @vishh , I'm the 1.16 Enhancement Lead/Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[Jeremy Rickard]

Hey there @arunbpt7 @jingxu97 @vishh -- 1.17 Enhancements shadow here 👋 . I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, I'll add it to the 1.17 tracking sheet (https://bit.ly/k8s117-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll also need to convert the design proposal into a KEP. To be accepted in the release, all enhancements MUST have a KEP, the KEP MUST be merged, in an implementable state, and have both graduation criteria/test plan.

Thanks!

[Nabarun Pal]

Hey there @arunbpt7 @jingxu97 @vishh -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.18 or having a major change in its current level?

The current release schedule is:

• Monday, January 6th - Release Cycle Begins
• Tuesday, January 28th EOD PST - Enhancements Freeze
• Thursday, March 5th, EOD PST - Code Freeze
• Monday, March 16th - Docs must be completed and reviewed
• Tuesday, March 24th - Kubernetes 1.18.0 Released

To be included in the release,

1. The KEP PR must be merged
2. The KEP must be in an implementable state
3. The KEP must have test plans and graduation criteria.

If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements

Thanks! :)

—
You are receiving this because you are on a team that was mentioned.

Reply to this email directly, view it on GitHub, or unsubscribe.

[Nabarun Pal]

@arunbpt7 @jingxu97 @vishh Just a friendly reminder, we are just 7 days away from the Enhancement Freeze (Tuesday, January 28th).

[Nabarun Pal]

@arunbpt7 @jingxu97 @vishh Just a friendly reminder, we are just 2 days away from the Enhancement Freeze (3 PM Pacific Time, Tuesday, January 28th).

[Nabarun Pal]

Unfortunately, the deadline for the 1.18 Enhancement freeze has passed. For now, this is being removed from the milestone. If there is a need to get this in, please file an enhancement exception.

[Zhihong Yu]

@jingxu97
Can you review this ?

[Jing Xu]

currently we don't have any pending work on this feature. Before we can go to GA, just need to make sure all functions are working as expected for both linux and windows. From e2e tests, I think it works.

@tedyu , is there something particular I need to review?

[Zhihong Yu]

@jingxu97
Nothing in particular.
There is linked issue, such as #78865 but I think they can be addressed on their own.

[Mirek Sedzinski]

Hey there @jingxu97 -- 1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

1. The KEP PR must be merged in an implementable state

1. The KEP must have test plans

1. The KEP must have graduation criteria.

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released
• Thursday, August 20: Week 19 - Release Retrospective

If you do, I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[Mirek Sedzinski]

Hi there @jingxu97 ,

Kind reminder about my question above.

Regards,
Mirek

[Mirek Sedzinski]

[Mirek Sedzinski]

Hi there @vishh ,

[Mirek Sedzinski]

[Mirek Sedzinski]

Hey @vishh , Enhancement shadow for the v1.19 release cycle here. Just following up on my earlier update to inform you of the
upcoming Enhancement Freeze scheduled on Tuesday, May 19.

[Nabarun Pal]

@vishh -- Unfortunately the deadline for the 1.19 Enhancement freeze has passed. For now, this is being removed from the milestone and 1.19 tracking sheet. If there is a need to get this in, please file an enhancement exception.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Anton Bessonov]

/remove-lifecycle stale

[Kirsten]

Hi @jingxu97 @vishh

Enhancements Lead here. Any plans to graduate this in 1.20?

Thanks,
Kirsten

[Kirsten]

Hi @jingxu97 @vishh

Following up: 1.20 Enhancements Freeze is October 6th. Could you let us know if you are planning to GA in 1.20? To be included in the milestone:
The KEP must be merged in an implementable state

The KEP must have test plans
The KEP must have graduation criteria

This enhancement is quite old and not in the current format, see: https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template

Also if you could update the description as requested: #361 (comment) that would be great.

Thanks
Kirsten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

—

[Immánuel!]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[Immánuel!]

/remove-lifecycle stale

[Immánuel!]

/lifecycle frozen

[Immánuel!]

I don't know if it's against the rules to make it frozen but it started to be annoying :D If you think it's better to bump it up constantly, then please set accordingly.

[Anton Bessonov]

The stale bot is one of the best ways to disrespect the community. It's a spit into the face.

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen.
Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[Benjamin Elder]

/lifecycle frozen
kubernetes/test-infra#22488

[Xing Yang]

/milestone v1.25

—

Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1126088946@github.com>

[Priyanka Saggu]

Hello @jingxu97 @vishh 👋, 1.25 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PST on Thursday June 16, 2022.

For note, This enhancement is targeting for stage stable for 1.25 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

Looks like for this one, we will need doing the following:

• Need to convert the design proposal into a KEP following the latest template to address all open checkboxes above and get it merged under the k/enhancements repo by 1.25 enhancements Freeze.

For note, the status of this enhancement is marked as at risk. Please update the issue description as suggested in the comment #361 (comment). Thank you!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1140767827@github.com>

[Priyanka Saggu]

Hello @jingxu97 👋 , just a quick check-in again, as we approach the 1.25 enhancements freeze.

The open KEP PR #3361 addresses all the checkboxes but the PRR review. Could you please also update the open PR to include a PRR approval file, as well? Thank you!

For note, the current status of the enhancement is at at-risk.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1152859125@github.com>

[Jing Xu]

I added the PRR approval file in this PR. is that ok? Thanks! @Priyankasaggu11929

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1154472542@github.com>

[Priyanka Saggu]

Thanks so much for the update, @jingxu97! 🙂

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1154737717@github.com>

[Priyanka Saggu]

Hello @jingxu97 👋, just a quick check-in again, as we approach the 1.25 enhancements freeze.

Please plan to get the open KEP PR #3361 merged before enhancements freeze on Thursday, June 23, 2022 at 18:00 PM PT.

For note, the current status of the enhancement is atat-risk. Thank you!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1161256243@github.com>

[Xing Yang]

Hi @Priyankasaggu11929,

FYI, KEP PR #3361 is replaced with PR #3422.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1163742226@github.com>

[Priyanka Saggu]

With KEP PR #3422 merged now, the enhancement is ready for the upcoming Enhancements Freeze.

For note, the status is now marked as tracked. Thank you so much!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1165024996@github.com>

[Benjamin Elder]

I'm pretty sure kubelet crashes early if it can't detect how much storage is available even if you are not setting a limit only if this featuregate is on.

See: kubernetes-sigs/kind#2559 (this always felt like a broken workaround but too many things to do 🙃)

I'm not sure if there's any users in another scenario vs kind rootless where the kubelet disk check controlled by this feature gate is a problem.

But I ask that we look into this before removing the featuregate, given the lentgthy route to GA I wouldn't be surprised to learn that someone else is disabling this for similar reasons.

A quick skim shows that at least also minikube disables this featuregate when running on certain filesystems

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1172435157@github.com>

[Carol Valencia]

Hello @jingxu97 👋, 1.25 Release Docs shadow here.
This enhancement is marked as ‘Needs Docs’ for 1.25 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.25 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by August 4.
 Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release. 


Thank you!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1176363076@github.com>

[Jing Xu]

@BenTheElder thank you for bringing up the issue. So the problem here is in some system such kind rootless, it is hard (or not possible) to get storage usage information and it will fail to start kubelet.

If storage usage information is not feasible in some systems, I am wondering whether the following logic can help avoid blocking GA while not breaking the existing system behavior:

• if no storage request/limit is not set, fail to get storage usage information persistently will not block kubelet start..

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1179832614@github.com>

[Dawn Chen]

Thanks for coming to SIG Node to review earlier today.

Re: #361 (comment)

I think local ephemeral storage support is very critical feature for a cluster. Without it being enabled, the cluster cannot be treated as production-ready from our earlier experience with K8s and GKE. So far we only know kind cluster and minkube disable the feature, not any production offers. In this case, can we make the feature default on, but introducing a config to disable it for those features?

I have a concern to introduce another kubelet config for such critical features. If really needed, can we automatically detect if the system has this capability?

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1189392168@github.com>

[Priyanka Saggu]

Hello @jingxu97 👋

Checking in once more as we approach 1.25 code freeze at 01:00 UTC on Wednesday, 3rd August 2022.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.

I was unable to find out any k/k PRs promoting KEP 361 to stable. Could you please point me,if any related PRs are currently open or confirm if they're yet to be raised?

The status of the enhancement is currently marked as at-risk.

Please also update the issue description with the relevant links for tracking purpose. Thank you so much!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1191386134@github.com>

[Benjamin Elder]

So far we only know kind cluster and minkube disable the feature, not any production offers.

I'm not aware of any production offerings, but so far all tools I've found that run Kubernetes rootless require opting out of LocalStorageCapacityIsolation feature gate, more can be found with https://grep.app/search?q=LocalStorageCapacityIsolation

https://github.com/podenv/silverkube

https://github.com/saschagrunert/kubernix

https://github.com/rootless-containers/usernetes/

k3d-io/k3d#802

canonical/microk8s#1587 (comment)

In this case, can we make the feature default on, but introducing a config to disable it for those features?

I think this makes sense. It can be a kubelet config field.

I have a concern to introduce another kubelet config for such critical features. If really needed, can we automatically detect if the system has this capability?

I feel like it might be concerning if it turns off automatically and admins are not aware that isolation is not be respected?
With a config field disabling it is explicit. If there's a bug in this check it might be difficult to catch and problematic for production.

Unless there's a status on the pod or something this might be difficult to track down when isolation is being ignored by automatic disablement.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1194457546@github.com>

[Ana Margarita Medina]

Hi @jingxu97 👋🏽

I’m one of the 1.25 Release Comms Shadow, Thanks for submitting this KEP for a Feature Blog. We have the code freeze upcoming and need to have a placeholder PR up already. Once you have it up, leave a comment on here (or slack me, and I will add it to the tracking sheet)

Let me know if you have any questions or if I can assist in any way :)

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1197499042@github.com>

[Jing Xu]

the PR is ready for review #3422

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1198934892@github.com>

[Ana Margarita Medina]

@jingxu97 Awesome, Thank you! Just updated our tracking document

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1199771862@github.com>

[Priyanka Saggu]

With the k/k code PR #3422 now merged, this enhancement is ready for the 1.25 code freeze. The status is now marked as tracked. Thank you!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1200649352@github.com>

[Priyanka Saggu]

Hello @jingxu97 👋

Just a gentle reminder from the enhancement team as we approach 1.25 code freeze at 01:00 UTC on Wednesday, 3rd August 2022. (less than a week to go)

Please plan to have the open k/k PR merged before then:

• kubernetes/kubernetes#111513

The status of this enhancement is currently marked as at risk

Thank you

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1200652516@github.com>

[Jing Xu]

the PR to promote GA kubernetes/kubernetes#111513 got approval and lgtm already

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1204254007@github.com>

[Benjamin Elder]

kubernetes/kubernetes#111513 merged (thanks @jingxu97 !)

I've sent a heads up to each of the projects i've identified that will need to migrate, besides the release note.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1204581666@github.com>

[Jing Xu]

Great thanks for @BenTheElder on helping it!

kubernetes/kubernetes#111513 merged (thanks @jingxu97 !)

I've sent a heads up to each of the projects i've identified that will need to migrate, besides the release note.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1204592668@github.com>

[Cat Chu]

Hey there @jingxu97 👋, 1.25 Release Docs Shadow here!

This enhancement is still marked as ‘Needs Docs’ for 1.25 release.

Tomorrow (August 4th) is the deadline for opening a placeholder PR against dev-1.25 branch in the k/website repo.

Please follow the steps detailed in the documentation to open the PR. This PR can be just a placeholder at this for now, as final docs PRs are due August 9th.

For more info, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1204660417@github.com>

[Jing Xu]

The PR is kubernetes/website#35687

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1204686190@github.com>

[Ana Margarita Medina]

Hi @jingxu97
I just checked on #3422 today and we don't have any blog content on the PR.

The blog PR needs to be made against k/website . An example of a blog post can be seen on kubernetes/website#33979.

Let me know if you have any questions, and I'm happy to help!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1205477921@github.com>

[Ana Margarita Medina]

Hi @jingxu97!

Just wanted to ping you again about the PR for the feature blog post. Let me know if you have any questions/I can assist!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1212207939@github.com>

[Jing Xu]

I posted a new PR for doc change due to some issue with my old one. kubernetes/website#35989

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1215932295@github.com>

[Jing Xu]

@AnaMMedina21 blog PR is created kubernetes/website#36025
Thank you!

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1217122035@github.com>

[KeithTt]

Same problem here, Docker Root Dir: /opt/docker.

May I create a soft link to /var/lib/docker?

kubectl drain node
systemctl stop kubelet
systemctl stop docker

mv /var/lib/docker /var/lib/docker.bak
ln -sv /opt/docker /var/lib/docker

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1275510129@github.com>

[Xing Yang]

Closed #361 as completed.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issue/361/issue_event/10668750859@github.com>

[Xing Yang]

This is GA. Closing this issue.

—
Reply to this email directly, view it on GitHub, or unsubscribe.

You are receiving this because you are on a team that was mentioned.Message ID: <kubernetes/enhancements/issues/361/1764983885@github.com>