Kubernetes service listen on ipv4

[Dan Maldonado]

============== Nodes run:
Kernel Version:        4.13.5-coreos-r2
 OS Image:            Container Linux by CoreOS 1520.6.0 (Ladybug)
 Operating System:        linux
 Architecture:            amd64
 Container Runtime Version:    docker://1.12.6
 Kubelet Version:        v1.7.2+coreos.0
 Kube-Proxy Version:        v1.7.2+coreos.0


============== Kubernetes service: 

apiVersion: v1
kind: Service
metadata:
  name: mail-service
  namespace: production 
  labels:
     support-app: "true"
spec:
  type: NodePort
  ports:
    - name: smtp
      protocol: TCP
      port: 25
      nodePort: 30025
    - name: imaps
      protocol: TCP
      port: 993
      nodePort: 30993
    - name: mailsend
      protocol: TCP
      port: 587
      nodePort: 30587
  selector:
    app: mail
    environment: production  

=========== on node:
...
tcp6       0      0 :::30587                :::*                    LISTEN    
tcp6       0      0 :::30025                :::*                    LISTEN    
...
tcp6       0      0 :::30993                :::*                    LISTEN    
...

========== from another host:

$ nmap 192.168.1.55

Starting Nmap 6.40 ( http://nmap.org ) at 2017-10-18 17:26 EDT
Nmap scan report for 192.168.1.55
Host is up (0.0040s latency).
Not shown: 998 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
30000/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds
$

==========:

The 30000/tcp open port above is for another service and we can get to that one no problem.  It is
the multi port (mail) service that we are having a problem resolving.

I can't seem to figure out how to make docker/rkt/kubeproxy... listen on IPV4.
We need to support IPV4.

Has anyone else run into this issue?

I may be missing something obvious.

[Euan Kemp]

I think there are a few things being conflated here.

First of all, I think the tcp4/tcp6 thing is a red herring. I suspect
you'll notice that 30000 (the working service port) is also listed as
tcp6 only.

On most modern Linux distros, including Container Linux, listening on
tcp6 will also imply tcp4.

Second, the nmap bit: by default, nmap will only scan a set of well
known ports, not all ports. Try using the `-p` flag if you want to
verify whether that port shows up as listening properly.


I suspect your problem is somewhere else. The next things I'd check
would be whether "kubectl describe svc <service>" shows any endpoints
or not. If it doesn't show any endpoints, then it's expected to not
route traffic anywhere (since no pods matching its selector are
available).

If there are endpoints and things still don't work, checking the
iptables rules kube-proxy added for correctness would be my next step
after that.

Good luck,
Euan

[Dan Maldonado]

I adjusted the scan:

$ nmap -p 30000-30994 192.168.1.55

Starting Nmap 6.40 ( http://nmap.org ) at 2017-10-19 15:08 EDT
Nmap scan report for 10.15.0.174
Host is up (0.0023s latency).
Not shown: 994 closed ports
PORT      STATE SERVICE
30021/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds
$

$ kubectl -n production describe svc/mail-service
Name:            mail-service
Namespace:        production
Labels:            support-app=true
Annotations:        <none>
Selector:        app=mail,environment=production
Type:            NodePort
IP:            10.3.0.46
Port:            smtp    25/TCP
NodePort:        smtp    30025/TCP
Endpoints:        10.2.38.3:25
Port:            imaps    993/TCP
NodePort:        imaps    30993/TCP
Endpoints:        10.2.38.3:993
Port:            mailsend    587/TCP
NodePort:        mailsend    30587/TCP
Endpoints:        10.2.38.3:587
Session Affinity:    None
Events:            <none>

#====================: On the node that is running the pod
# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere           

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        
DOCKER-ISOLATION  all  --  anywhere             anywhere           
DOCKER     all  --  anywhere             anywhere           
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere           
ACCEPT     all  --  anywhere             anywhere           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere           

Chain DOCKER (1 references)
target     prot opt source               destination        

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination        
RETURN     all  --  anywhere             anywhere           

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination        
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination  


I will be asking the netadmin to check iptables but I am suspicious about Container Linux overriding any settings I may change.

[Dan Maldonado]

This was completely my fault.  I failed to realize that the mail server was miss-configured. 
It was working properly and one of the accounts had been placed in the file incorrectly. 
I fixed that and everything is working as expected.