esp container failed to bind to curious port

[Doug Tangren]

I'm having some trouble using the endpoints-runtime container as a side car container in a kubernetes pod.

I'm running the endpoints runtime is as a side car container in my kubernetes pod deployment. My server listens on port 9090 and I've configured the endpoints-runtime to listen on 9091. 

    spec:

      containers:

      - name: esp

        image: gcr.io/endpoints-release/endpoints-runtime:1

        imagePullPolicy: Always

        args: [

          "-p", "9091",

          "-a", "127.0.0.1:9090",

          "-s", "{{ SERVICE  }}",

          "-v", "{{ VERSION }}",

        ]

I have a CD system set up where I create a canary deployment that I first validate before creating a stable deployment. The canary deployment works as advertised

$ kubectl logs -f canary-pod esp

INFO:Fetching an access token from the metadata service

INFO:Fetching the service configuration from the service management service

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt
... request logs

However my stable deployment results in a set of pods stuck in a CrashLoopBackOff state because the esp container fails to start with the following error.

$ kubectl logs -f stable-pod esp

INFO:Fetching an access token from the metadata service

INFO:Fetching the service configuration from the service management service

nginx: [warn] Using trusted CA certificates file: /etc/nginx/trusted-ca-certificates.crt

2017/03/14 14:41:43 [emerg] 1#1: bind() to 0.0.0.0:8090 failed (98: Address already in use)

2017/03/14 14:41:43 [emerg] 1#1: bind() to 0.0.0.0:8090 failed (98: Address already in use)

2017/03/14 14:41:43 [emerg] 1#1: bind() to 0.0.0.0:8090 failed (98: Address already in use)

2017/03/14 14:41:43 [emerg] 1#1: bind() to 0.0.0.0:8090 failed (98: Address already in use)

2017/03/14 14:41:43 [emerg] 1#1: bind() to 0.0.0.0:8090 failed (98: Address already in use)

2017/03/14 14:41:43 [emerg] 1#1: still could not bind()

It seems nginx trying to bind on port 8090. It's not clear why its trying to bind to that particular port. I wasn't able to find any cloud endpoint documentation on what may cause this error.

 Is there anything I may be doing wrong?

[Doug Tangren]

Problem resolved. I think there was some state left over from a previous deployment. Deleting then recreating deployment fixed the issue.