how to triage kubectl port-forward

[jph...@appnexus.com]

I'm trying to connect through kubectl port-forward to the registry addon I started in my cluster. I added the kube-proxy as a DaemonSet.

But It seems like I'm getting shut down either at the port-forwarding process, or in between that and the cluster.

I've tried connecting to the registry itself:

kubectl port-forward --namespace kube-system kube-registry-v0-8hefg 5000:5000 &
Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000

docker push localhost:5000/jphelan/hellonode
Put http://localhost:5000/v1/repositories/jphelan/hellonode/: dial tcp [::1]:5000: getsockopt: connection refused

or connecting to a proxy

kubectl port-forward --namespace kube-system kube-registry-proxy-ds-ivnid 5000:5000 &
[1] 2108
Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000

docker push localhost:5000/jphelan/hellonode
The push refers to a repository [localhost:5000/jphelan/hellonode]
Put http://localhost:5000/v1/repositories/jphelan/hellonode/: dial tcp [::1]:5000: getsockopt: connection refused

Looking at my pods, it looks like both have tcp:5000 : 

kubectl --namespace="kube-system" get pods
NAME                                READY     STATUS    RESTARTS   AGE
kube-dns-v19-6drm0                  3/3       Running   0          1d
kube-registry-proxy-ds-ivnid        1/1       Running   0          48m
kube-registry-proxy-ds-x41lm        1/1       Running   0          48m
kube-registry-proxy-ds-zqbkz        1/1       Running   0          48m
kube-registry-v0-8hefp              1/1       Running   0          1d
kubernetes-dashboard-v1.4.0-y2ppx   1/1       Running   0          1d

kubectl --namespace="kube-system" describe pods/kube-registry-v0-8hefp
Name: kube-registry-v0-8hefp
Namespace: kube-system
Node: 10.1.113.164/10.1.113.164
Start Time: Thu, 29 Sep 2016 12:45:27 -0400
Labels: k8s-app=kube-registry
 kubernetes.io/cluster-service=true
 version=v0
Status: Running
IP: 172.16.54.2
Controllers: ReplicationController/kube-registry-v0
Containers:
  registry:
    Container ID: docker://bb111c7b526e424f28a33f285c1ead3af032153eee3b8a4fdb479c0f1ce6122b
    Image: registry:2
    Image ID: docker://sha256:541a6732eadb598740116d1f5d18806ea3404aab8710789c9b5c86a53d1945f2
    Port: 5000/TCP
    Limits:
      cpu: 100m
      memory: 100Mi
    Requests:
      cpu: 100m
      memory: 100Mi
    State: Running
      Started: Thu, 29 Sep 2016 12:45:31 -0400
    Ready: True
    Restart Count: 0
    Environment Variables:
      REGISTRY_HTTP_ADDR: :5000
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
Conditions:
  Type Status
  Initialized True
  Ready True
  PodScheduled True
Volumes:
  image-store:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-token-x9ss1:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-x9ss1
QoS Tier: Guaranteed
No events.

kubectl --namespace="kube-system" describe pods/kube-registry-proxy-ds-ivnid
Name: kube-registry-proxy-ds-ivnid
Namespace: kube-system
Node: 10.1.113.164/10.1.113.164
Start Time: Fri, 30 Sep 2016 15:54:09 -0400
Labels: name=kube-registry-proxy-pod
Status: Running
IP: 172.16.54.3
Controllers: DaemonSet/kube-registry-proxy-ds
Containers:
  kube-registry-proxy:
    Container ID: docker://d9fec4fec52ac7025021381ca76e3107716a11ac0c461c1dcb3e98a58bf64113
    Image: gcr.io/google_containers/kube-registry-proxy:0.3
    Image ID: docker://sha256:b86ac3f11a0c35b480138b9c3a555c2d8abb5b40023df3dd257ce9d432ee6006
    Port: 5000/TCP
    Limits:
      cpu: 100m
      memory: 50Mi
    Requests:
      cpu: 100m
      memory: 50Mi
    State: Running
      Started: Fri, 30 Sep 2016 15:54:10 -0400
    Ready: True
    Restart Count: 0
    Environment Variables:
      REGISTRY_HOST: kube-registry.kube-system.svc.cluster.local
      REGISTRY_PORT: 5000
      FORWARD_PORT: 5000
Conditions:
  Type Status
  Initialized True
  Ready True
  PodScheduled True
Volumes:
  default-token-x9ss1:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-x9ss1
QoS Tier: Guaranteed
Events:
  FirstSeen LastSeen Count From SubobjectPath Type Reason Message
  --------- -------- ----- ---- ------------- -------- ------ -------
  55m 55m 1 {kubelet 10.1.113.164} spec.containers{kube-registry-proxy} Normal Pulled Container image "gcr.io/google_containers/kube-registry-proxy:0.3" already present on machine
  55m 55m 1 {kubelet 10.1.113.164} spec.containers{kube-registry-proxy} Normal Created Created container with docker id d9fec4fec52a; Security:[seccomp=unconfined]
  55m 55m 1 {kubelet 10.1.113.164} spec.containers{kube-registry-proxy} Normal Started Started container with docker id d9fec4fec52a

Can you help me figure out how to debug this?

[Daniel Smith]

Things to try:

* Does some other process have port 5000 open?
* Try curling the proxy endpoint-- <apiserverip>/api/v1/namespaces/kube-system/pods/kube-registry-proxy-ds-ivnid:5000/proxy; if that doesn't work you might get a more detailed error message

* Check http vs https (although I'd expect a different error message)

* run kubectl with -v=9 (not sure if it'll help on portforward, but worth a try)

* Look in apiserver logs to see if apiserver is noticing your call

* Look it the kubelet's logs to see if it noticed the call

* Look in the registry's logs to see if it's seeing any calls

* dust off your tcpdump-fu

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/aab67676-f951-4d92-9473-d9407dbc7ac4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jph...@appnexus.com]

Hey Dan Smith,

Thanks for the response.

This was very interesting. I did:

kubectl port-forward --v=9 --namespace kube-system kube-registry-proxy-ds-ivnid 5000:5000 &
...

I1003 11:42:04.952325   15200 round_trippers.go:299] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "User-Agent: kubectl/v1.4.0 (darwin/amd64) kubernetes/a16c0a7" http://10.1.113.162:8080/api/v1/namespaces/kube-system/pods/kube-registry-proxy-ds-ivnid/portforward

I1003 11:42:04.986302   15200 round_trippers.go:318] POST http://10.1.113.162:8080/api/v1/namespaces/kube-system/pods/kube-registry-proxy-ds-ivnid/portforward 101 Switching Protocols in 33 milliseconds

I1003 11:42:04.986334   15200 round_trippers.go:324] Response Headers:

I1003 11:42:04.986340   15200 round_trippers.go:327]     Connection: Upgrade

I1003 11:42:04.986346   15200 round_trippers.go:327]     Upgrade: SPDY/3.1

I1003 11:42:04.986350   15200 round_trippers.go:327]     X-Stream-Protocol-Version: portforward.k8s.io

I1003 11:42:04.986355   15200 round_trippers.go:327]     Date: Mon, 03 Oct 2016 15:41:57 GMT

I1003 11:42:04.986361   15200 round_trippers.go:318] POST http://10.1.113.162:8080/api/v1/namespaces/kube-system/pods/kube-registry-proxy-ds-ivnid/portforward 101 Switching Protocols in 34 milliseconds

I1003 11:42:04.986366   15200 round_trippers.go:324] Response Headers:

I1003 11:42:04.986371   15200 round_trippers.go:327]     Connection: Upgrade

I1003 11:42:04.986376   15200 round_trippers.go:327]     Upgrade: SPDY/3.1

I1003 11:42:04.986401   15200 round_trippers.go:327]     X-Stream-Protocol-Version: portforward.k8s.io

I1003 11:42:04.986407   15200 round_trippers.go:327]     Date: Mon, 03 Oct 2016 15:41:57 GMT

and this was very helpful to learn more.

The problem ended up being that I'm running docker for mac - and the portforwarding is happening on my mac, not in the docker for mac special vm. I ended up running docker-machine's boot2docker instead and just running kubectl portforward from inside that vm.

I'm not sure of an easier way to do this without running some proxy in the vm. 

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

[Brendan Burns]

You should be able to set up port-forwarding in virtual box that will forward from your Mac into the VM that Docker is running...

[jph...@appnexus.com]

Hey Brendan,

Thanks for your reply. I spent some time reading the manual and it seems like it's not possible to forward ports this way to localhost without going into the vm and starting a proxy or touching iptables: https://www.virtualbox.org/manual/ch06.html#natforward. Have you done this? I feel like it should be possible, but I can't find it.

-Jack