consul status reported as DOWN on Actuator /health

[Joe Pardi]

Consul believes my service is "critical" because the health message is being reported as "DOWN" (See below), with a 503 http status code.  The Serf health status is passing, so that doesn't seem to be the problem.  I have Consul running within a single virtualbox vm in a docker container, as a server agent (see below).

Any ideas where to look?  - Joe

*********  docker run command  **************

docker run -d -v /home/joe/Apps/consul/data:/consul/data -p 8300:8300 -p 8301:8301 -p 8301:8301/udp -p 8302:8302/udp -p 8302:8302 -p 8400:8400 -p 8500:8500 --name=dev-consul --net=host consul agent -server -ui -bind=127.0.0.1 -bootstrap

*********  /health   **************

{

• status: "DOWN",
• diskSpace: 

  {

  • status: "UP",
  • total: 293124263936,
  • free: 56603447296,
  • threshold: 10485760

  },
• db: 

  {

  • status: "UP",
  • database: "PostgreSQL",
  • hello: 1

  },
• refreshScope: 

  {

  • status: "UP"

  },
• hystrix: 

  {

  • status: "UP"

  },
• consul: 

  {

  • status: "DOWN",
  • services: 

    {

    • consul: [ ],
    • event-calendar-service: 

      [

      • "author=Joe Pardi"

      ]

    },
  • error: "java.lang.IllegalArgumentException: Value must not be null"

  }

}

[Nicholas Capo]

What sort of health check do you have configured?
Can you run that check manually?
If it's an HTTP check, then your service needs to return a 2xx status code in order to pass.

This error is probably from you application since consul isn't written in Java:

"java.lang.IllegalArgumentException: Value must not be null"

Nicholas

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/5fd2172e-1d28-4cca-b1e9-70a67a220208%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joe Pardi]

The microservice is written using Spring Boot and the Actuator plugin provides a default HTTP-based health check (/health).  When I run the health check manually from the browser it returns the JSON and http 503 which I copied in my original question.  I think Consul has something to do with it because the status: "DOWN" is reported in the consul section.  When I check the health check in the Consul console, it matches what the manual check provides via the browser.

HTTP GET http://192.168.1.83:8080/health: 503  Output: {"status":"DOWN","diskSpace":{"status":"UP","total":293124263936,"free":56604626944,"threshold":10485760},"db":{"status":"UP","database":"PostgreSQL","hello":1},"refreshScope":{"status":"UP"},"hystrix":{"status":"UP"},"consul":{"status":"DOWN","services":{"consul":[],"event-calendar-service":["author=Joe Pardi"]},"error":"java.lang.IllegalArgumentException: Value must not be null"}} 

I'll put the Java app in debug mode to see if an error of some sort is being logged ...

[Kaushal Rai]

Hi @Joe Pardi,

Were you able to resolve the issue?

I'm facing similar issue and kind of clueless as to what could have gone  wrong here.

Some directions would be helpful.

HTTP GET http://192.168.1.83:8080/health: 503  Output: {"status":"DOWN","diskSpace":{"status":"UP","total":293124263936,"free":56604626944,"threshold":10485760},"db":{"status":"UP","database":"PostgreSQL","hello":1},"refreshScope":{"status":"UP"},"hystrix":{"status":"UP"},"consul":{"status":"DOWN","services":{"consul":[],"event-calendar-service":["author=Joe Pardi"]},"error":"java.lang.IllegalArgumentException: Value must not be null"}} 

-

Kaushal

[Nicholas Capo]

I'm order to pass an HTTP health check your service must return a 2xx status code.
A 503 return will fail the health check, no mater the content of the json body.

Nicholas

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/bc6b3722-3ee5-40d9-811b-d562db881e8e%40googlegroups.com.

[Werner Dijkerman]

We have the same problem on an Consul 1.0.0 with ACL enabled. An ACL enabled environment with Consul 0.9.3 works fine without any issues.

Are you using ACL's in Consul?

We have different microservices with different kind of spring boot versions (From 1.3.x to 1.5.8), all with the same issue.

Op maandag 6 november 2017 00:39:22 UTC+1 schreef Joe Pardi:

[Werner Dijkerman]

Well, that is the problem. The /health page that is been access by Consul to determine if the service is running is returning a 503.

The reason that the /health page is returning a 503 is that the Spring Boot application has an issue with Consul "java.lang.IllegalArgumentException: Value must not be null".

In mij case this is something with ACL's specific to Consul 1.0.0, but I don't know if that is also the case for the Topic Starter.

Kind regards,

Werner

Op donderdag 9 november 2017 04:19:28 UTC+1 schreef Nicholas Capo:

[Kaushal Rai]

I have consul v1.0.0 and have not applied any ACL explicitly.

I just started springboot application with spring cloud and consul discovery modules...

springBootVersion = '1.4.7.RELEASE'

springCloudVersion = 'Dalston.SR4'

dependencies {

compile('org.springframework.boot:spring-boot-starter-actuator')

compile('org.springframework.cloud:spring-cloud-starter-consul-config')

compile('org.springframework.cloud:spring-cloud-starter-consul-discovery')

compile('org.springframework.boot:spring-boot-starter-web')

testCompile('org.springframework.boot:spring-boot-starter-test')

}

A very basic setup for integration tests with Consul. Service gets registered but remains critical.

I intend to use @Value & @EnableConfigurationProperties  separately for fetching values from Consul too.

Regards,

Kaushlendra Rai

[Patrick Ullrich]

Ran into the same issue. Seems to be related to spring-cloud-consul and consul version 1. Downgrade to e.g. 0.9.3 seems to resolve the issue.

read more here: 

https://github.com/spring-cloud/spring-cloud-consul/issues/365

Cheers,

...Patrick

[Werner Dijkerman]

Thank you Patrick!

Op donderdag 9 november 2017 16:47:23 UTC+1 schreef Patrick Ullrich:

[Joe Pardi]

Ok, so I managed to get the health check to work with version 1.  I had to have the management config in my bootstrap.yml file.  If it is placed into the Consul config, the health check fails.  Also, I had to set management.health.consul.enabled to false.  With those two settings, only could I get the health check to work.  This was Spring Boot 1.5.8.RELEASE and Spring Cloud Dalston.SR4.

----- boostrap.xml ----

management:

  security:

    enabled: false

  health:

    consul:

      enabled: false   # must be false to pass /health check

[zzhoo8]

tks, it works after downgrading consul to 0.9.3.

在 2017年11月9日星期四 UTC+8下午11:47:23，Patrick Ullrich写道：

[ankush....@vuclip.com]

Its consul version problem ,use previous version application.