Re: [kubernetes/kubernetes] EventRateLimit - Admission Control - Fails to start kube-apiserver (#62861)
k8s-ci-robot
@iponnam: Reiterating the mentions to trigger a notification:
@kubernetes/sig-api-machinery-bugs
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Jordan Liggitt
what version are you running with?
Pavan Surya Prakash Ponnam
Kubernetes Version 1.9.5
The API Server pod not creating itself
Attaching kubelet logs.
kubelet.log
Pavan Surya Prakash Ponnam
Referred the following document:
https://kubernetes.io/docs/admin/admission-controllers/#eventratelimit-alpha
Daniel Smith
@gmarek maybe knows?
Chao Xu
Do you have the apiserver log?
Pavan Surya Prakash Ponnam
@caesarxuchao ApiServer is not all creating. So I have attached kubelet logs
Pavan Surya Prakash Ponnam
@gmarek Hello Marek,
Could you please assist me in this regard.
Thank you.
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
Pavan Surya Prakash Ponnam
/remove-lifecycle stale
liuyusen
Got the same issue on 1.9.6, my cluster is setup using kops
mirwan
Got the same issue on 1.11.2, my cluster is setup using kubespray.
apiserver logs "Error: unknown api groups eventratelimit.admission.k8s.io"
Jacob Crowther
I am running into the same error that @mirwan details.
@deads2k @derekwaynecarr You are the owners listed for the eventratelimit here Would you look into this issue? EventRateLimit appears to be broken.
As a side note, removing --runtime-config=...,eventratelimit.admission.k8s.io/v1alpha1=true allows the apiserver to start, but appears to disable EventRateLimit.
This is a rather important feature for multi-tenant clusters. May we get an update?
Daniel Smith
I imagine there is a feature gate needed to enable this? But I can't find it.
Jordan Liggitt
I don't think there's a feature gate… you have to opt in to this alpha feature by enabling the admission plugin and specifying the alpha admission config
Jacob Crowther
@liggitt Do you know of any step that is missing from the ticket description?
Jordan Liggitt
The config is not a served rest api. I would not expect --runtime-config to be modified in order to use that feature.
David Eads
When you remove it from the --runtime-config does it work? I would expect it to. That flag is for REST resources and this just an admission config type.
Jacob Crowther
@deads2k What would be the preferred way to validate? I have a test cluster up and can look into it.
Jacob Crowther
This may be built on false assumptions, but the following test produces no 429 responses.
Initial test with Curl to validate token and endpoint produces 200 response:
$ curl $APISERVER/api/v1/namespaces/test/pods --header "Authorization: Bearer $TOKEN" --insecure
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/test/pods",
"resourceVersion": "53412"
},
"items": []
}
Same test with Apache Benchmark to produce load:
$ ab -n 1000 -c 100 -H "Authorization: Bearer $TOKEN" $APISERVER/api/v1/namespaces/test/services
(no 429 errors produced -- all 200 OK)
Cluster configuration:
$ cat /etc/kubernetes/admission-control/eventRateLimit.yaml
kind: Configuration
apiVersion: eventratelimit.admission.k8s.io/v1alpha1
limits:
- type: Namespace
qps: 1
burst: 1
- type: User
qps: 1
burst: 1
$ cat /etc/kubernetes/admission-control/control-config.yaml
kind: AdmissionConfiguration
apiVersion: apiserver.k8s.io/v1alpha1
plugins:
- name: EventRateLimit
path: /etc/kubernetes/admission-control/eventRateLimit.yaml
$ cat /etc/kubernetes/manifests/kube-apiserver.yaml
...
command:
- /hyperkube
- apiserver
- --enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,EventRateLimit
- --admission-control-config-file=/etc/kubernetes/admission-control/control-config.yaml
- --runtime-config=api/all,admissionregistration.k8s.io/v1alpha1
...
From what I understand (and this is the potentially false assumption) the purpose of the EventRateLimit admission plugin is to limit the number of REST requests that can be made against the apiserver in any given moment. Limits can be enforced by namespace, user, or source+object (haven't played with that last one yet). But effectively, this rate limit is intended to ensure equal opportunity of making kubernetes API calls for multiple users of a cluster.
Is this assumption right? Documentation is sparse, so I am having trouble determining if an "event" is a REST API call or something else. @deads2k Would you clarify and ensure my test is accurate?
Derek Carr
@jcrowthe the admission plugin is used to optionally limit the number Event resources that can be created. historically, we had issues where control plane components, kubelets could spam the API server with large numbers of event resources. that has largely been mitigated client-side using the event recorder client. The admission controller does not care about general purpose REST requests.
mattmceuen
I hit the same issue trying to use EventRateLimit with the --runtime-config flag; leaving admissionregistration.k8s.io/v1alpha1 off of that seems to get the admission controller working fine.
I think this is a documentation bug, since the admission controller doc is what's telling us to use --runtime-config for this.
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
—
fejta-bot
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
fejta-bot
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Kubernetes Prow Robot
Closed #62861.
Kubernetes Prow Robot
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Munai Das Udasin
I too am facing the same issue on kubernetes 1.13.5 cluster. The issue should be reopened.
Daniel Smith
/reopen
I don't think we got a resolution on this, ideally someone who knows the feature can tell us if the documentation is wrong or if there is some other bug.
Daniel Smith
Reopened #62861.
fejta-bot
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
—
Kubernetes Prow Robot
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—
Kubernetes Prow Robot
Closed #62861.
Pavan Surya Prakash Ponnam
/reopen
Kubernetes Prow Robot
Reopened #62861.
Kubernetes Prow Robot
@iponnam: Reopened this issue.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—
fejta-bot
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
—
Kubernetes Prow Robot
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—
Kubernetes Prow Robot
Closed #62861.
zbialik
Was there a resolution for this? I don't get why this was closed?
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
zbialik
/reopen
Kubernetes Prow Robot
@zbialik: You can't reopen an issue/PR unless you authored it or you are a collaborator.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—
Kubernetes Prow Robot
Reopened #62861.
Jordan Liggitt
/reopen
ideally someone who knows the feature can tell us if the documentation is wrong or if there is some other bug.
it is a documentation bug. --runtime-config is for REST APIs. This is a config API and needs no special enablement. The runtime-config bit should be removed from the docs.
Kubernetes Prow Robot
@liggitt: Reopened this issue.
In response to this:
/reopen
ideally someone who knows the feature can tell us if the documentation is wrong or if there is some other bug.
it is a documentation bug.
--runtime-configis for REST APIs. This is a config API and needs no special enablement. The runtime-config bit should be removed from the docs.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—
cvernooy23
also having this issue on 1.16.2
fejta-bot
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
—
Kubernetes Prow Robot
Closed #62861.
Kubernetes Prow Robot
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—
kubefetch
This piece of EventRateLimits rate limit has very standard documentation for configuration. I installed my friend's deployment above, and still can't boot API-Server, hoping to come up with a best practice. I'm also a little confused about how much configuration I should have in rate production