API service complaints "no endpoints available for service" even there are endpoints for the service
2,122 views
Skip to first unread message
lin.for...@gmail.com
Aug 25, 2016, 10:05:09 PM8/25/16
to Kubernetes developer/contributor discussion
Hi,
I have a Kubernetes on Mesos env, and enable "X509 Client Cert", "Service Account Token" and "OpenID Connect Token" for API server. Here are the options that API server has
/km apiserver --secure-port=8001 --bind-address=0.0.0.0 --insecure-port=8888 --insecure-bind-address=0.0.0.0 --etcd-servers=http://127.0.0.1:4001 --service-cluster-ip-range=10.0.0.1/24 --cloud-provider=mesos --cloud-config=/etc/kubernetes/conf/mesos-cloud.conf --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,SecurityContextDeny,ResourceQuota --oidc-issuer-url=https://master.cfc/acs/api/v1/auth --oidc-client-id=cwc-service --oidc-ca-file=/etc/kubernetes/conf/oidc.crt --oidc-groups-claim=projects --tls-cert-file=/etc/kubernetes/conf/server.cert --tls-private-key-file=/etc/kubernetes/conf/server.key --client-ca-file=/etc/kubernetes/conf/ca.crt --service-account-key-file=/etc/kubernetes/conf/server.key --v=1
and control manager
/km controller-manager --host-port-endpoints=false --master=127.0.0.1:8888 --cloud-provider=mesos --cloud-config=/etc/kubernetes/conf/mesos-cloud.conf --root-ca-file=/etc/kubernetes/conf/ca.crt --service-account-private-key-file=/etc/kubernetes/conf/server.key --v=1 --leader-elect=true
But I have problem to access my service now. This is a simple httpd service
# kubectl describe service web1
Name: web1
Namespace: default
Labels: app=web1
Selector: app=web1
Type: ClusterIP
IP: 10.0.0.73
Port: httpd 80/TCP
Endpoints: 10.1.52.5:80,10.1.52.6:80
Session Affinity: None
No events.
# curl http://127.0.0.1:8888/api/v1/proxy/namespaces/default/services/web1
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service \"web1\"",
"reason": "ServiceUnavailable",
"code": 503
}
I can access my service with pod IP,
# curl http://10.1.52.5:80
<html><body><h1>It works!</h1></body></html>
# curl http://10.1.52.6:80
<html><body><h1>It works!</h1></body></html>
Have problem with ClusterIP
# curl -vvv http://10.0.0.73
* Rebuilt URL to: http://10.0.0.73/
* Trying 10.0.0.73...
And API server complaints "no endpoints"
I0826 01:28:42.375042 1 genericapiserver.go:690] Serving securely on 0.0.0.0:8001
I0826 01:28:42.375342 1 genericapiserver.go:734] Serving insecurely on 0.0.0.0:8888
I0826 01:40:47.758855 1 <autogenerated>:1] Error getting ResourceLocation: no endpoints available for service "web1"
I0826 01:40:56.613708 1 <autogenerated>:1] Error getting ResourceLocation: endpoints "heapster" not found
I0826 01:40:56.614342 1 <autogenerated>:1] Error getting ResourceLocation: endpoints "heapster" not found
Can someone help me on this issue?
Thanks
Liqiang Lin
I have a Kubernetes on Mesos env, and enable "X509 Client Cert", "Service Account Token" and "OpenID Connect Token" for API server. Here are the options that API server has
/km apiserver --secure-port=8001 --bind-address=0.0.0.0 --insecure-port=8888 --insecure-bind-address=0.0.0.0 --etcd-servers=http://127.0.0.1:4001 --service-cluster-ip-range=10.0.0.1/24 --cloud-provider=mesos --cloud-config=/etc/kubernetes/conf/mesos-cloud.conf --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,SecurityContextDeny,ResourceQuota --oidc-issuer-url=https://master.cfc/acs/api/v1/auth --oidc-client-id=cwc-service --oidc-ca-file=/etc/kubernetes/conf/oidc.crt --oidc-groups-claim=projects --tls-cert-file=/etc/kubernetes/conf/server.cert --tls-private-key-file=/etc/kubernetes/conf/server.key --client-ca-file=/etc/kubernetes/conf/ca.crt --service-account-key-file=/etc/kubernetes/conf/server.key --v=1
and control manager
/km controller-manager --host-port-endpoints=false --master=127.0.0.1:8888 --cloud-provider=mesos --cloud-config=/etc/kubernetes/conf/mesos-cloud.conf --root-ca-file=/etc/kubernetes/conf/ca.crt --service-account-private-key-file=/etc/kubernetes/conf/server.key --v=1 --leader-elect=true
But I have problem to access my service now. This is a simple httpd service
# kubectl describe service web1
Name: web1
Namespace: default
Labels: app=web1
Selector: app=web1
Type: ClusterIP
IP: 10.0.0.73
Port: httpd 80/TCP
Endpoints: 10.1.52.5:80,10.1.52.6:80
Session Affinity: None
No events.
# curl http://127.0.0.1:8888/api/v1/proxy/namespaces/default/services/web1
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service \"web1\"",
"reason": "ServiceUnavailable",
"code": 503
}
I can access my service with pod IP,
# curl http://10.1.52.5:80
<html><body><h1>It works!</h1></body></html>
# curl http://10.1.52.6:80
<html><body><h1>It works!</h1></body></html>
Have problem with ClusterIP
# curl -vvv http://10.0.0.73
* Rebuilt URL to: http://10.0.0.73/
* Trying 10.0.0.73...
And API server complaints "no endpoints"
I0826 01:28:42.375042 1 genericapiserver.go:690] Serving securely on 0.0.0.0:8001
I0826 01:28:42.375342 1 genericapiserver.go:734] Serving insecurely on 0.0.0.0:8888
I0826 01:40:47.758855 1 <autogenerated>:1] Error getting ResourceLocation: no endpoints available for service "web1"
I0826 01:40:56.613708 1 <autogenerated>:1] Error getting ResourceLocation: endpoints "heapster" not found
I0826 01:40:56.614342 1 <autogenerated>:1] Error getting ResourceLocation: endpoints "heapster" not found
Can someone help me on this issue?
Thanks
Liqiang Lin
lin.for...@gmail.com
Aug 25, 2016, 10:53:19 PM8/25/16
to Kubernetes developer/contributor discussion
It seems I need to specify service port,
# curl -vvv http://127.0.0.1:8888/api/v1/proxy/namespaces/default/services/web1:80
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /api/v1/proxy/namespaces/default/services/web1:80 HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: /api/v1/proxy/namespaces/default/services/web1:80/
< Date: Fri, 26 Aug 2016 02:46:41 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 127.0.0.1 left intact
However I still cannot access my service because clusterIP of my service is not routable.
# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
FLANNEL all -- 10.1.0.0/16 anywhere
MASQUERADE all -- !10.1.0.0/16 10.1.0.0/16
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain FLANNEL (1 references)
target prot opt source destination
ACCEPT all -- anywhere 10.1.0.0/16
MASQUERADE all -- anywhere !base-address.mcast.net/4
I missed a lot of NAT rules for the services, here are the rules I get from a working env
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- anywhere 192.168.3.1 /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- anywhere 192.168.3.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- anywhere 192.168.3.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-XGLOHA7QRQ3V22RZ tcp -- anywhere 192.168.3.224 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:http
KUBE-SVC-BJM46V3U5RZHCFRZ tcp -- anywhere 192.168.3.225 /* kube-system/heapster: cluster IP */ tcp dpt:http
KUBE-SVC-XSYIQLTHGS5R5BTP tcp -- anywhere 192.168.3.249 /* default/web1:tcp-80-80-71ht1 cluster IP */ tcp dpt:http
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Not sure which component of Kubernetes is responsible for creating these iptable rules for Kubernetes services.
# curl -vvv http://127.0.0.1:8888/api/v1/proxy/namespaces/default/services/web1:80
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /api/v1/proxy/namespaces/default/services/web1:80 HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: /api/v1/proxy/namespaces/default/services/web1:80/
< Date: Fri, 26 Aug 2016 02:46:41 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 127.0.0.1 left intact
However I still cannot access my service because clusterIP of my service is not routable.
# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
FLANNEL all -- 10.1.0.0/16 anywhere
MASQUERADE all -- !10.1.0.0/16 10.1.0.0/16
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain FLANNEL (1 references)
target prot opt source destination
ACCEPT all -- anywhere 10.1.0.0/16
MASQUERADE all -- anywhere !base-address.mcast.net/4
I missed a lot of NAT rules for the services, here are the rules I get from a working env
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- anywhere 192.168.3.1 /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- anywhere 192.168.3.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- anywhere 192.168.3.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-XGLOHA7QRQ3V22RZ tcp -- anywhere 192.168.3.224 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:http
KUBE-SVC-BJM46V3U5RZHCFRZ tcp -- anywhere 192.168.3.225 /* kube-system/heapster: cluster IP */ tcp dpt:http
KUBE-SVC-XSYIQLTHGS5R5BTP tcp -- anywhere 192.168.3.249 /* default/web1:tcp-80-80-71ht1 cluster IP */ tcp dpt:http
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Not sure which component of Kubernetes is responsible for creating these iptable rules for Kubernetes services.
lin.for...@gmail.com
Aug 25, 2016, 11:02:11 PM8/25/16
to Kubernetes developer/contributor discussion
I need to specify service port when I access it,
# curl -vvv http://127.0.0.1:8888/api/v1/proxy/namespaces/default/services/web1:80
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /api/v1/proxy/namespaces/default/services/web1:80 HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: /api/v1/proxy/namespaces/default/services/web1:80/
< Date: Fri, 26 Aug 2016 02:46:41 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 127.0.0.1 left intact
But I still cannot access my service with clusterip. It seems I missed a lot of iptable NAT rules for services.
# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
FLANNEL all -- 10.1.0.0/16 anywhere
MASQUERADE all -- !10.1.0.0/16 10.1.0.0/16
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain FLANNEL (1 references)
target prot opt source destination
ACCEPT all -- anywhere 10.1.0.0/16
MASQUERADE all -- anywhere !base-address.mcast.net/4
Not sure which component of kubernetes is responsible for creating iptables rules for services.
Here is NAT rules I got from a working env,
Here is NAT rules I got from a working env,
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- anywhere 192.168.3.1 /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-TCOU7JCQXEZGVUNU udp -- anywhere 192.168.3.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- anywhere 192.168.3.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:domain
KUBE-SVC-XGLOHA7QRQ3V22RZ tcp -- anywhere 192.168.3.224 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:http
KUBE-SVC-BJM46V3U5RZHCFRZ tcp -- anywhere 192.168.3.225 /* kube-system/heapster: cluster IP */ tcp dpt:http
KUBE-SVC-XSYIQLTHGS5R5BTP tcp -- anywhere 192.168.3.249 /* default/web1:tcp-80-80-71ht1 cluster IP */ tcp dpt:http
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
On Friday, August 26, 2016 at 10:05:09 AM UTC+8, lin.for...@gmail.com wrote:
lin.for...@gmail.com
Aug 25, 2016, 11:20:21 PM8/25/16
to Kubernetes developer/contributor discussion
If I try to access service through service proxy in API server, it has nothing to do with NAT ruels in iptable (kube-proxy help create NAT rules for services). So the problem is still that API server complaints "no available endpoint for service". Not sure how API server find service endpoints, and why it complaints even there are endpoints.
Thanks
Liqiang Lin
Thanks
Liqiang Lin
Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages