"Failed to fetch service config" error in starting ESP in container engine

[AL]

I had a service running in container engine with ESP and it was running yesterday. Today when I recreate the service / deployment, I am getting errors indicating that the ESP could not fetch the service config. I tried the esp-echo example and received the same error log. I have verified with gcloud service-management describe SERVICE-NAME that the service is defined as I expected. How can I fix this?

2017-06-27T14:48:08.528873964Z INFO:Fetching an access token from the metadata service

2017-06-27T14:48:08.531273023Z INFO:Fetching the service configuration from the service management service

2017-06-27T14:48:28.545814838Z WARNING:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff517040c50>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /v1/services/echo-api.endpoints.cds-dev-155819.cloud.goog/config?configId=2017-06-27r0

2017-06-27T14:48:48.558417535Z WARNING:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff517040e90>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /v1/services/echo-api.endpoints.cds-dev-155819.cloud.goog/config?configId=2017-06-27r0

2017-06-27T14:49:08.575736373Z WARNING:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff51705e050>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /v1/services/echo-api.endpoints.cds-dev-155819.cloud.goog/config?configId=2017-06-27r0

2017-06-27T14:49:28.597791794Z ERROR:Failed to fetch service config

[Wencheng Lu]

Hi, AL,

Is this still happening? we tried "gcloud service-management configs describe 2017-06-27r0 --service echo-api.endpoints.cds-dev-155819.cloud.goog", and it did fetch successfully. Sounds like some transient connection issue.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/a22947eb-c8a7-4a5c-b6cf-e17d503d2f5e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[AL]

I redeployed this morning and it is still happening. Does the following configuration for esp container look correct?

      containers:

      # [START esp]

      - name: esp

        image: gcr.io/endpoints-release/endpoints-runtime:1

        args: [

          "-p", "8081",

          "-a", "127.0.0.1:8080",

          "-s", "echo-api.endpoints.cds-dev-155819.cloud.goog",

          "-v", "2017-06-27r0",

        ]

      # [END esp]

        ports:

        - containerPort: 8081

[Wencheng Lu]

Hi, AL,

your configuration looks good to me. I also just tried to run this esp-echo sample myself, and it works for me.

Could you manually test if you can fetch service config from your vm instance? 

(1) ssh to your vm instance

(2) curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" -H "Metadata-Flavor: Google"
(3) note down the bear token printed above as $token env variable
(4) curl https://servicemanagement.googleapis.com/v1/services/echo-api.endpoints.cds-dev-155819.cloud.goog/config?configId=2017-06-27r0 -H "Authorization":"Bearer $token"

this should fetch service config successfully if your vm and network is set up correctly.

The next thing to look at is your vm scope. Could you check at the bottom of your "VM instance details" page? look for "Cloud API access scopes" section. You may also click "Equivalent REST" and see listed scopes.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsubscri...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/a22947eb-c8a7-4a5c-b6cf-e17d503d2f5e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endpoints+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/67a3df99-35fe-4c1c-935f-b55f1bfc330b%40googlegroups.com.

[Wayne Zhang]

For this morning re-deployment, could you try to get its logs for us?  

One thing I suspected that somehow your container could not resolve DSN "https://servicemanagement.googleapis.com"

Could you login into your VM or container to see if you can ping that DNS name?

[AL]

I created two more containers this afternoon, one without cloud datastore enabled, and one with cloud datastore enabled. I can run esp-echo sample just fine in both containers. Don’t know what went wrong with the container I created two days ago L

 

Thank you all for your help. Please consider the problem solved.