Endpoints API - watching for changes

Matthew Ceroni

unread,

Jun 21, 2016, 3:32:49 PM6/21/16

to kubernetes-dev

I am using a side car container to monitoring the Endpoints API for changes (/api/v1/endpoints?watch=true) to update an external load balancer. Basically the solution demoed here :

https://www.nginx.com/resources/webinars/bringing-kubernetes-to-the-edge-with-nginx-plus/

However in doing so I get constant updates/modifications to the following endpoints :

- kubernetes

- kube-controller-manager

- calico-policy-election

- kube-scheduler

As the solution polls the Nginx status for every update this causes lots of traffic to hit the Nginx server (7k requests in a 1 hour period).

Is this normal? For these services endpoints to change every second or so?

Thanks

Matthew Ceroni

unread,

Jun 21, 2016, 4:00:24 PM6/21/16

to kubernetes-dev

Also note that if you delete all PODs in a SVC there is no update sent for the final POD removed. Therefore you don't get that notification and can't update the external LB to remove the last entry.

Prashanth B

unread,

Jun 21, 2016, 4:16:46 PM6/21/16

to Matthew Ceroni, kubernetes-dev

The nginx ingress controller does something similar: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx

On Tue, Jun 21, 2016 at 1:00 PM, Matthew Ceroni <matthe...@gmail.com> wrote:

Also note that if you delete all PODs in a SVC there is no update sent for the final POD removed. Therefore you don't get that notification and can't update the external LB to remove the last entry.

This sounds like a bug, please try to trigger it with the nginx controller. Or if you can invoke it with a script that polls kubectl, or any other method that uses a standard interface to the cluster (create a client via NewInCluster in a pod, for example), please file it. I can't reproduce it, or I don't understand the problem you're describing:

$ kubectl create -f ~/rtmp/echo-app.yaml

service "echoheaders" created

replicationcontroller "echoheaders" created

$ kubectl get ep echoheaders --watch-only

$ kubectl delete rc echoheaders

and the watc-only window shows

NAME ENDPOINTS AGE

echoheaders <none> 35s

On Tuesday, June 21, 2016 at 12:32:49 PM UTC-7, Matthew Ceroni wrote:
I am using a side car container to monitoring the Endpoints API for changes (/api/v1/endpoints?watch=true) to update an external load balancer. Basically the solution demoed here :
https://www.nginx.com/resources/webinars/bringing-kubernetes-to-the-edge-with-nginx-plus/

However in doing so I get constant updates/modifications to the following endpoints :

- kubernetes
- kube-controller-manager
- calico-policy-election
- kube-scheduler

As the solution polls the Nginx status for every update this causes lots of traffic to hit the Nginx server (7k requests in a 1 hour period).

Is this normal? For these services endpoints to change every second or so?

Thanks

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/f43a366e-cd6a-4bc1-9c05-5d55eb27ae74%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Matthew Ceroni

unread,

Jun 21, 2016, 6:52:25 PM6/21/16

to kubernetes-dev

Ah, I stand corrected. I actually do see the final event when scaled to 0 or deleted all together. The code in motorboat doesn't handle that situation.

I was missing it as the output is cluttered do to constant MODIFIED events for the services I listed above.

Therefore if I run

➜ nginx git:(master) kubectl --namespace=kube-system get ep kube-controller-manager --watch-only

NAME ENDPOINTS AGE

kube-controller-manager <none> 4h

I just keeps updating every second with the same output. Is this to be expected? Same thing happens with kube-scheduler, and calico-policy-election.