Client.Timeout exceeded while awaiting headers

[Phani Piduri]

We are running consul in OpenShift cluster. All services have been developed by Spring Boot/Cloud APIs and they have been registered successfully in consul. There is a health point exposed using SpringBoot actuator. The health point itself works just fine when try to hit using curl. But from Consul seeing a below errors frequently which causes issues in discovering the service.

Any suggestions would be great help.. 

2016/08/05 05:57:15 [WARN] agent: http request failed 'http://10.1.0.18:9080/health': Get http://10.1.0.18:9080/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

2016/08/05 05:57:58 [WARN] agent: http request failed 'http://10.1.1.20:9091/health': Get http://10.1.1.20:9091/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[Anand Nadar]

Did you find a solution for this? I have seen very similar errors appearing in my consul cluster.

[Anand Nadar]

Same here. On AWS and this starts happening all of a sudden. The application seems to be working fine if I hit the health endpoint independently through curl.

However consul seems to be freaking out.

Any leads on this?

[Kyle Havlovitz]

Hi Anand, if Consul is getting a timeout while connecting to the health endpoint, curl might just have a higher implicit timeout. You could experiment with this by setting the "timeout" field in the health check's configuration. For more information see the health check docs: https://www.consul.io/docs/agent/checks.html. If that fails you could try changing the check to a script check and using cURL directly just to try and track down any differences.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/6fbe281a-2e6e-42aa-8516-28deea859913%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anand Nadar]

Thanks Kyle for your response to this.

Digged in a little more around this. The time this happened my Spring boot instance; found in the access logs that it had returned 503 as the response code to my health checks. This is the default actuator health check implementation that returns 503 - service unavailable when it finds a registered resource unavailable in the application.

I wonder if consul is interpreting the 503 as a Client.timeout on its side, and why not print the 503 as ERROR instead of printing the failure as warning with an altogether different message? Cant tell at this point whats happening. Its perhaps doing the right thing by marking the service as unavailable but the warning in the log is totally misleading.

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.

[Anand Nadar]

About the timeout comment, I verified that all the 503's returned by the application were well within 25ms. So this is not a timeout issue as the default heath time is set at 10 sec.

[Toni Mas]

I faced the same issue. It was the config-server check  failing:

  "configServer" : {
   "status" : "DOWN",
   "error" : "java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing"
 },

Fixed by canceling that check:
cloud:
   config:
     server:
       health:
         enabled: false