Failed to connect to external service from pod

Roger Song

unread,

Apr 24, 2017, 9:29:53 AM4/24/17

to Kubernetes user discussion and Q&A

Hi all,

Kindly help me review this issue. Thanks!

[ Description ]

I am newbie to k8s, recently setup k8s cluster on top of CentOS 7.3 with kubeadm 1.6.1.

Master: k8s

Minions: host01, host02, host03

In one of pods(po/tulip-saas-xnode), I tried to make connection from pod to external rds service(172.172.10.16:3306). That's the reason I setup service&endpoint "tulip-saas-db2" manually, as follows:

===========

# kubectl get service tulip-saas-db2 -o yaml

apiVersion: v1

kind: Service

metadata:

creationTimestamp: 2017-04-24T07:46:10Z

name: tulip-saas-db2

namespace: default

resourceVersion: "905529"

selfLink: /api/v1/namespaces/default/services/tulip-saas-db2

uid: 153b3520-28c2-11e7-a272-000c29235036

spec:

clusterIP: 10.111.128.117

ports:

- port: 3306

protocol: TCP

targetPort: 3306

sessionAffinity: None

type: ClusterIP

status:

loadBalancer: {}

# kubectl get endpoints tulip-saas-db2 -o yaml

apiVersion: v1

kind: Endpoints

metadata:

creationTimestamp: 2017-04-24T07:46:10Z

name: tulip-saas-db2

namespace: default

resourceVersion: "905533"

selfLink: /api/v1/namespaces/default/endpoints/tulip-saas-db2

uid: 15552d0d-28c2-11e7-a272-000c29235036

subsets:

- addresses:

- ip: 172.172.10.16

ports:

- port: 3306

protocol: TCP

==========

I'm able to connect to the port in host02 OS via cluster IP.

==========

[root@host02 .kube]# iptables-save | grep tulip-saas-db2

-A KUBE-SEP-TS2EMOGZXA7V27BD -s 172.172.10.16/32 -m comment --comment "default/tulip-saas-db2:" -j KUBE-MARK-MASQ

-A KUBE-SEP-TS2EMOGZXA7V27BD -p tcp -m comment --comment "default/tulip-saas-db2:" -m tcp -j DNAT --to-destination 172.172.10.16:3306

-A KUBE-SERVICES ! -s 172.10.0.0/16 -d 10.111.128.117/32 -p tcp -m comment --comment "default/tulip-saas-db2: cluster IP" -m tcp --dport 3306 -j KUBE-MARK-MASQ

-A KUBE-SERVICES -d 10.111.128.117/32 -p tcp -m comment --comment "default/tulip-saas-db2: cluster IP" -m tcp --dport 3306 -j KUBE-SVC-ASAFJW2B6372ZEVA

-A KUBE-SVC-ASAFJW2B6372ZEVA -m comment --comment "default/tulip-saas-db2:" -j KUBE-SEP-TS2EMOGZXA7V27BD

[root@host02 .kube]# telnet 10.111.128.117 3306

Trying 10.111.128.117...

Connected to 10.111.128.117.

Escape character is '^]'.

=====

The pod is running on host02

====

# kubectl get pod --all-namespaces -o wide | grep tulip

default tulip-saas-xnode-3216045024-ctctp 1/1 Running 1 8h 172.10.2.22 host02.corp.mooit.net

====

Inside the pod, service name can be resolved. However, I'm not able to connect to the port.

===

[root@tulip-saas-xnode-3216045024-ctctp /]# nslookup tulip-saas-db2

Server: 10.96.0.10

Address: 10.96.0.10#53

Name: tulip-saas-db2.default.svc.cluster.local

Address: 10.111.128.117

[root@tulip-saas-xnode-3216045024-ctctp /]# telnet tulip-saas-db2 3306

Trying 10.111.128.117...

^C

===

kube-proxy logs in host02 doesn't give any message. I tried to delete the pod, and let DS create it again, no lucky. No remarkable messages in /var/log/message.

======

[root@k8s manifests]# kubectl logs -f po/kube-proxy-p279k -n kube-system

I0424 12:37:24.220402 1 server.go:225] Using iptables Proxier.

I0424 12:37:24.301205 1 server.go:249] Tearing down userspace rules.

I0424 12:37:24.433983 1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072

I0424 12:37:24.435683 1 conntrack.go:66] Setting conntrack hashsize to 32768

I0424 12:37:24.436164 1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400

I0424 12:37:24.436217 1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600

=======

I tried to follow https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/, but failed to understand in which layer the problem takes place. firewalld is disabled in all nodes.

Had tried some RBAC stuff for kube-proxy account referring to https://github.com/uruddarraju/kubernetes-rbac-policies

[ env ]

# kubeadm version

kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

# kubectl get all --all-namespaces

NAMESPACE NAME READY STATUS RESTARTS AGE

default po/busybox 1/1 Running 1 3h

default po/http-svc-zzj1q 1/1 Running 1 4d

default po/nginx-deployment-4234284026-04wb3 1/1 Running 2 3d

default po/nginx-deployment-4234284026-pdvml 1/1 Running 1 3d

default po/tulip-saas-xnode-3216045024-ctctp 1/1 Running 1 8h

kube-system po/default-http-backend-2198840601-7wdbk 1/1 Running 2 4d

kube-system po/etcd-k8s.corp.mooit.net 1/1 Running 11 7d

kube-system po/kube-apiserver-k8s.corp.mooit.net 1/1 Running 6 6d

kube-system po/kube-controller-manager-k8s.corp.mooit.net 1/1 Running 13 7d

kube-system po/kube-dns-3913472980-mtml5 3/3 Running 96 7d

kube-system po/kube-flannel-ds-57crg 2/2 Running 7 7d

kube-system po/kube-flannel-ds-bn66x 2/2 Running 4 7d

kube-system po/kube-flannel-ds-wxj4d 2/2 Running 3 7d

kube-system po/kube-flannel-ds-xk9wh 2/2 Running 56 7d

kube-system po/kube-proxy-mp6xr 1/1 Running 10 7d

kube-system po/kube-proxy-p279k 1/1 Running 0 7m

kube-system po/kube-proxy-qqdvd 1/1 Running 2 7d

kube-system po/kube-proxy-vjmnw 1/1 Running 1 7d

kube-system po/kube-scheduler-k8s.corp.mooit.net 1/1 Running 13 7d

kube-system po/kubernetes-dashboard-915795657-wf3fp 1/1 Running 2 6d

kube-system po/nginx-ingress-lb-0q6n8 1/1 Running 1 2d

kube-system po/nginx-ingress-lb-20km8 1/1 Running 2 2d

kube-system po/nginx-ingress-lb-fk7nd 1/1 Running 1 2d

kube-system po/nginx-ingress-lb-q0z4c 1/1 Running 1 2d

NAMESPACE NAME DESIRED CURRENT READY AGE

default rc/http-svc 1 1 1 4d

NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

default svc/http-svc 10.109.111.193 <nodes> 80:30301/TCP 4d

default svc/kubernetes 10.96.0.1 <none> 443/TCP 7d

default svc/nginx-svc 10.105.48.156 <nodes> 80:30302/TCP 3d

default svc/tulip-saas-db2 10.111.128.117 <none> 3306/TCP 5h

default svc/tulip-saas-xnode 10.106.241.164 <nodes> 80:30189/TCP 1d

kube-system svc/default-http-backend 10.98.17.92 <none> 80/TCP 4d

kube-system svc/kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 7d

kube-system svc/kubernetes-dashboard 10.106.75.115 <nodes> 80:32416/TCP 7d

NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE

default deploy/nginx-deployment 2 2 2 2 3d

default deploy/tulip-saas-xnode 1 1 1 1 8h

kube-system deploy/default-http-backend 1 1 1 1 4d

kube-system deploy/kube-dns 1 1 1 1 7d

kube-system deploy/kubernetes-dashboard 1 1 1 1 7d

NAMESPACE NAME DESIRED CURRENT READY AGE

default rs/nginx-deployment-4234284026 2 2 2 3d

default rs/tulip-saas-xnode-3216045024 1 1 1 8h

kube-system rs/default-http-backend-2198840601 1 1 1 4d

kube-system rs/kube-dns-3913472980 1 1 1 7d

kube-system rs/kubernetes-dashboard-915795657 1 1 1 6d

[root@k8s manifests]# kubectl get ds --all-namespaces

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE

kube-system kube-flannel-ds 4 4 4 4 4 beta.kubernetes.io/arch=amd64 7d

kube-system kube-proxy 4 4 4 4 4 <none> 7d

kube-system nginx-ingress-lb 4 4 4 4 4 <none> 2d

# kubectl get ing --all-namespaces

NAMESPACE NAME HOSTS ADDRESS PORTS AGE

default ng k8s.corp.mooit.net 172.172.10.23... 80, 443 3d

default test-http k8s.corp.mooit.net 172.172.10.23... 80, 443 3d

default tulip-saas-xnode xnode.svr.mooit.net 172.172.10.23... 80, 443 1d

kube-system dashboard k8s.corp.mooit.net 172.172.10.23... 80, 443 3d

Roger Song

unread,

Apr 24, 2017, 10:33:51 AM4/24/17

to Kubernetes user discussion and Q&A

BTW, 172.172.10.0/24 is host subnet. pod network is 172.10.0.0/16.

Inside Pod:

[root@tulip-saas-xnode-3216045024-ctctp /]# netstat -rn

Kernel IP routing table

Destination Gateway Genmask Flags MSS Window irtt Iface

0.0.0.0 172.10.2.1 0.0.0.0 UG 0 0 0 eth0

10.244.0.0 172.10.2.1 255.255.0.0 UG 0 0 0 eth0

172.10.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

Tim Hockin

unread,

Apr 24, 2017, 11:20:35 AM4/24/17

to kubernet...@googlegroups.com

What network driver are you using? kubenet? CNI + flannel? CNI +
weave? CNI + calico?

What is your Service cluster IP range?

Can you ping from the pod to its own Node IP?

Can you ping from the pod to a different Node IP?

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.

Rijie Song

unread,

Apr 24, 2017, 11:59:58 AM4/24/17

to kubernet...@googlegroups.com

Thanks for response, Tim.

1. What network driver are you using? kubenet? CNI + flannel? CNI +

weave? CNI + calico?

CNI+flannel.

flannel pod output on this particular node:

[root@k8s manifests]# kubectl logs -f po/kube-flannel-ds-bn66x -n kube-system -c kube-flannel

I0424 06:33:46.210053 1 kube.go:109] Waiting 10m0s for node controller to sync

I0424 06:33:46.210339 1 kube.go:289] starting kube subnet manager

I0424 06:33:47.218505 1 kube.go:116] Node controller sync successful

I0424 06:33:47.218556 1 main.go:132] Installing signal handlers

I0424 06:33:47.218656 1 manager.go:136] Determining IP address of default interface

I0424 06:33:47.219642 1 manager.go:149] Using interface with name ens160 and address 172.172.10.32

I0424 06:33:47.219673 1 manager.go:166] Defaulting external address to interface address (172.172.10.32)

I0424 06:33:47.517408 1 ipmasq.go:47] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN

I0424 06:33:47.524679 1 ipmasq.go:47] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE

I0424 06:33:47.530320 1 ipmasq.go:47] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE

I0424 06:33:47.535263 1 manager.go:250] Lease acquired: 172.10.2.0/24

I0424 06:33:47.536092 1 network.go:58] Watching for L3 misses

I0424 06:33:47.536117 1 network.go:66] Watching for new subnet leases

2. What is your Service cluster IP range?

- --service-cluster-ip-range=10.96.0.0/12

3. Can you ping from the pod to its own Node IP?

Yes.

[root@tulip-saas-xnode-2720274701-3fmqd /]# ping 172.172.10.32

PING 172.172.10.32 (172.172.10.32) 56(84) bytes of data.

64 bytes from 172.172.10.32: icmp_seq=1 ttl=64 time=0.108 ms

64 bytes from 172.172.10.32: icmp_seq=2 ttl=64 time=0.076 ms

4. Can you ping from the pod to a different Node IP?

No.

[root@tulip-saas-xnode-2720274701-3fmqd /]# ping 172.172.10.31

PING 172.172.10.31 (172.172.10.31) 56(84) bytes of data.

^C

> email to kubernetes-users+unsubscribe@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/zQB4eS5BaGs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--

Regards
Rijie Song

Tim Hockin

unread,

Apr 24, 2017, 12:56:55 PM4/24/17

to kubernet...@googlegroups.com

On Mon, Apr 24, 2017 at 8:59 AM, Rijie Song <rijie...@gmail.com> wrote:
> Thanks for response, Tim.
>
> 1. What network driver are you using? kubenet? CNI + flannel? CNI +
> weave? CNI + calico?
>
> CNI+flannel.

I don't have first-hand up-to-date flannel notes...

can you ping from a pod to a different pod on the other node?

>> > email to kubernetes-use...@googlegroups.com.
>> > To post to this group, send email to kubernet...@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/kubernetes-users.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/kubernetes-users/zQB4eS5BaGs/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> kubernetes-use...@googlegroups.com.
>> To post to this group, send email to kubernet...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Regards
> Rijie Song
>

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.

Rijie Song

unread,

Apr 24, 2017, 1:02:10 PM4/24/17

to kubernet...@googlegroups.com

Yes, pod network works.

[root@k8s manifests]# kubectl get pod -o wide

NAME READY STATUS RESTARTS AGE IP NODE

busybox 1/1 Running 1 7h 172.10.2.23 host02.corp.mooit.net

http-svc-zzj1q 1/1 Running 1 4d 172.10.1.13 host01.corp.mooit.net

nginx-deployment-4234284026-04wb3 1/1 Running 2 3d 172.10.3.20 host03.corp.mooit.net

nginx-deployment-4234284026-pdvml 1/1 Running 1 3d 172.10.1.12 host01.corp.mooit.net

traceroute-3470717669-0lprk 1/1 Running 16 2h 172.10.3.28 host03.corp.mooit.net

tulip-saas-xnode-2720274701-3fmqd 1/1 Running 0 1h 172.10.2.25 host02.corp.mooit.net

[root@tulip-saas-xnode-2720274701-3fmqd /]# ping 172.10.1.12

PING 172.10.1.12 (172.10.1.12) 56(84) bytes of data.

64 bytes from 172.10.1.12: icmp_seq=1 ttl=62 time=1.49 ms

64 bytes from 172.10.1.12: icmp_seq=2 ttl=62 time=0.419 ms

64 bytes from 172.10.1.12: icmp_seq=3 ttl=62 time=0.400 ms

^C

--- 172.10.1.12 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2519ms

rtt min/avg/max/mdev = 0.400/0.770/1.491/0.509 ms

[root@tulip-saas-xnode-2720274701-3fmqd /]# ping 172.10.3.20

PING 172.10.3.20 (172.10.3.20) 56(84) bytes of data.

64 bytes from 172.10.3.20: icmp_seq=1 ttl=62 time=1.83 ms

64 bytes from 172.10.3.20: icmp_seq=2 ttl=62 time=0.444 ms

>> > email to kubernetes-users+unsubscribe@googlegroups.com.
>> > To post to this group, send email to kubernetes-users@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/kubernetes-users.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/kubernetes-users/zQB4eS5BaGs/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> kubernetes-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to kubernetes-users@googlegroups.com.

>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Regards
> Rijie Song
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-users+unsubscribe@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--

Regards
Rijie Song

Tim Hockin

unread,

Apr 24, 2017, 1:59:29 PM4/24/17

to kubernet...@googlegroups.com

On Mon, Apr 24, 2017 at 8:59 AM, Rijie Song <rijie...@gmail.com> wrote:

/12 is a LARGE service range - it allows 1 million Service IPs in your
cluster. That's unusual, and may cause problems elsewhere.

>> > email to kubernetes-use...@googlegroups.com.
>> > To post to this group, send email to kubernet...@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/kubernetes-users.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/kubernetes-users/zQB4eS5BaGs/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> kubernetes-use...@googlegroups.com.
>> To post to this group, send email to kubernet...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Regards
> Rijie Song
>

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.

Tim Hockin

unread,

Apr 24, 2017, 2:03:15 PM4/24/17

to kubernet...@googlegroups.com

So you can reach one Service (DNS) but not another? I would start
with doing some tcpdump to see what packets are moving around.

Roger Song

unread,

Apr 24, 2017, 10:11:23 PM4/24/17

to Kubernetes user discussion and Q&A

Thanks Tim, I will try to look inside.

In fact, /12 was created by kubeadm.

In CNI+flannel, what's the expected route that the packets going from container to node network?

Tim Hockin

unread,

Apr 25, 2017, 12:11:56 AM4/25/17

to kubernet...@googlegroups.com

They should come out the bridge and to the node directly.

Reply all

Reply to author

Forward