Re: [kubernetes/enhancements] Dynamic Maximum volume count (#554)
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Michelle Au
/remove-lifecycle stale
Claire Laurence
@gnufied Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP
Hemant Kumar
This issue may see some internal refactoring changes but we are not sure yet. But feature will remain in beta in 1.14.
Kendrick Coleman
@gnufied any work here planned for 1.15? If not, we will stay in beta. Thanks
Michelle Au
This will remain beta in 1.15
Andrew Mills
Hey all, I was just wondering if there was a good workaround for this issue? Currently working in AKS and not able to consistently startup pods because they are being assigned to nodes with no more room to attach disks.
Hemant Kumar
By AKS you mean Azure container offering? Azure Disks support dynamic limits via this KEP. Which issue you are specifically referring to? May be open a github issue?
Andrew Mills
Sorry for the ambiguity @gnufied. There are several issues already open, not by me, but this is probably the most relevant: Azure/AKS#670
Yes, I'm referring to the Azure Kubernetes Service offering. We have a cluster built of small VM's (4 disks max) for dev purposes. If I have a node with 3 disks attached and submit a new request to have k8s stand up a pod that requires 2 disks, it may assign that work to the pod with only 1 disk available. This causes the pod to crash due to not being able to fulfill the number of disks I need and I am never able to successfully get that pod up.
As far as I understand it, it's all related to the primary bullet for the issue on this thread: Add support for dynamic and generic mechanism of maximum volume per node.
I will gladly open a new issue if that is a better forum to discuss what I'm seeing and potential workarounds, but I'm not sure if that's on this repo or one for AKS since both are in the mix.
Thanks for your help.
Hemant Kumar
@ammills01 which version of k8s? Dynamic volume limits for Azure Disks was fixed awhile back(1.12) - kubernetes/kubernetes#67772 and should correctly take into account number of volumes that can be attached to a node.
For small VMs, I wonder this is a problem for root disk consuming one disk and hence k8s incorrectly assuming that node still has 2 disks. Can you open a github issue? cc @andyzhangx does Azure needs logic to reserve one disk for root disk?
Andy Zhang
k8s v1.12 should already have Dynamic volume limits for Azure Disks(that only account for data disk), while @ammills01 's question is if there is 3 data disks already, and one pod need 2 data disks, will it mount one data disk first, and then when mount the second disk, k8s volume scheduler found it's already hit max disk num? Or volume scheduler will first check whether the node could mount 2 disks in the beginning? I am not sure about the logic here.
Andrew Mills
k8s v1.12 should already have Dynamic volume limits for Azure Disks(that only account for data disk), while @ammills01 's question is if there is 3 data disks already, and one pod need 2 data disks, will it mount one data disk first, and then when mount the second disk, k8s volume scheduler found it's already hit max disk num? Or volume scheduler will first check whether the node could mount 2 disks in the beginning? I am not sure about the logic here.
That is exactly my question @andyzhangx. I'm working on a new Git issue now that explains what I'm using when it comes to VM's for my AKS cluster, AKS version, the files I'm using for k8s deployment, what I see when I query the nodes (kubectl get nodes -o json), the error I see, etc...
According to the Azure portal, this is using AKS 1.13.5, so it should have whatever fix was in for 1.12. Do you know if we need to have --feature-gates=CSINodeInfo enabled or something for this to pick up the right values? When I query a node and spit out the JSON, I see 2 values for 'attachable-volumes-azure-disk'. One value is under 'allocatable', the other under 'capacity'. In my case, both say 4. I'm not sure why, but I expected capacity to show the maximum for the VM, not taking into account what is currently allocated, and allocatable to indicate what is left to be allocated, taking into account the number of disks already in use.
Hemant Kumar
@andyzhangx no the scheduler will look for "viability" of both volumes in a pod at once, not one at a time.
Michelle Au
Even though this feature is remaining as beta, it's going to go under significant redesign. The KEP is here: #942
Saad Ali
/milestone v1.15
simplytunde
Hey @gnufied Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!
simplytunde
@gnufied The placeholder PR is due Thursday May 30th.
Kendrick Coleman
Hi @gnufied @msau42 @saad-ali . Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.
Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.
If you know this will slip, please reply back and let us know. Thanks!
Kendrick Coleman
Hi @gnufied @msau42 @saad-ali, today is code freeze for the 1.15 release cycle. I do not see a reply for any k/k PRs to track for this merge. It's now being marked as At Risk in the 1.15 Enhancement Tracking Sheet. If there is no response, or you respond with PRs to track and they are not merged by EOD PST, this will be dropped from the 1.15 Milestone. After this point, only release-blocking issues and PRs will be allowed in the milestone with an exception.
Kendrick Coleman
/milestone clear
Hemant Kumar
This enhancement is going to miss 1.15 release window. There is a PR open but it wasn't merged during the given timeline - kubernetes/kubernetes#77595
bradley childs
/milestone clear
Hemant Kumar
Further update - we have requested feature exception for this. The feature exception is still being discussed and needs approval from sig-scheduling folks. cc @bsalamat @ravisantoshgudimetla
bradley childs
/milestone v1.15
Kendrick Coleman
/milestone clear
Hemant Kumar
This feature will remain in beta in 1.16
Justin Santa Barbara
We recently discovered that this is even more dynamic than it used to be, because ENIs now count against volume attachments on some AWS instance types: kubernetes/kubernetes#80967
Kristin Martin
Hello @gnufied -- 1.17 Enhancement Shadow here! 🙂
I wanted to reach out to see if this enhancement will be graduating to stable in 1.17?
Please let me know so that this enhancement can be added to 1.17 tracking sheet.
Thank you!
🔔Friendly Reminder
The current release schedule is
- Monday, September 23 - Release Cycle Begins
- Tuesday, October 15, EOD PST - Enhancements Freeze
- Thursday, November 14, EOD PST - Code Freeze
- Tuesday, November 19 - Docs must be completed and reviewed
- Monday, December 9 - Kubernetes 1.17.0 Released
Fabio Bertinatto
/assign
Fabio Bertinatto
Hi @kcmartin; we intend to graduate this feature to GA in v1.17.
Kristin Martin
Thanks @bertinatto !
/milestone v1.17
Kristin Martin
/stage stable
Irvi Aini
Hello @gnufied I'm one of the v1.17 docs shadows.
Does this enhancement for (or the work planned for v1.17) require any new docs (or modifications to existing docs)? If not, can you please update the 1.17 Enhancement Tracker Sheet (or let me know and I'll do so)
If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, November 8th, it can just be a placeholder PR at this time. Let me know if you have any questions!
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
Irvi Aini
Since we're approaching Docs placeholder PR deadline on Nov 8th. Please try to get one in against k/website dev-1.17 branch.
Fabio Bertinatto
@irvifa: created placeholder PR here: kubernetes/website#17432
Kristin Martin
Hi @gnufied
I am one of the Enhancements Shadows for the 1.17 Release Team. We are very near to Code Freeze (Nov 14th) for this release cycle. Just checking in about the progress of this enhancement. I see that https://github.com/kubernetes/kubernetes/pull/77595 was filed in relation to this. Are there any other PRs related to this enhancement? If yes, can you please link them here?
Thank you in advance 😄
Fabio Bertinatto
Hi @kcmartin.
These are the open PRs related to this feature:
Docs: kubernetes/website#17432
Kubernetes: kubernetes/kubernetes#83568
Kristin Martin
Thank you @bertinatto !
Jeremy Rickard
Hey @gnufied @bertinatto , Happy New Year! 1.18 Enhancements lead here 👋 Thanks for getting this across the line in 1.17!!
I'm going though and doing some cleanup for the milestone and checking on things that graduated in the last release. Since this graduated to GA in 1.17, I'd like to close this issue out but the KEP is still marked as implementable. Could you submit a PR to update the KEP to implemented and then we can close this issue out?
Thanks so much!
Fabio Bertinatto
Hey @gnufied @bertinatto , Happy New Year! 1.18 Enhancements lead here wave Thanks for getting this across the line in 1.17!!
I'm going though and doing some cleanup for the milestone and checking on things that graduated in the last release. Since this graduated to GA in 1.17, I'd like to close this issue out but the KEP is still marked as implementable. Could you submit a PR to update the KEP to
implementedand then we can close this issue out?Thanks so much!
Hi @jeremyrickard,
I just created #1433 to update the KEP status.
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
Nabarun Pal
/remove-lifecycle stale
Nabarun Pal
Hi @bertinatto, thank you so much for updating the status. :)
Nabarun Pal
Closing this enhancement issue since the KEP has been implemented.
/close
Kubernetes Prow Robot
@palnabarun: Closing this issue.
In response to this:
Closing this enhancement issue since the KEP has been implemented.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Kubernetes Prow Robot
Closed #554.