Exception while constructing an ImmutableNode - missing taggedAddresses

[Peter Heitman]

I have multiple clusters of consul servers (in a QA environment). Most are working without issue. On one cluster, one of the systems is logging an Exception every two minutes

2016-07-18 08:38:49,716 ERROR [com.hazelcast.spi.discovery.integration.DiscoveryService] (cached4) [10.210.5.65]:5701 [dev] [3.6.2] discoverNodes() unexpected error: Consul request failed: com.orbitz.consul.ConsulException: Consul request failed

at com.orbitz.consul.util.Http.extractConsulResponse(Http.java:49)

at com.orbitz.consul.HealthClient.getHealthyServiceInstances(HealthClient.java:317)

at com.orbitz.consul.HealthClient.getHealthyServiceInstances(HealthClient.java:300)

at org.bitsofinfo.hazelcast.discovery.consul.ConsulDiscoveryStrategy.discoverNodes(ConsulDiscoveryStrategy.java:183)

at com.jungo.hazelcast.ConsulDiscoveryStrategyWrapper.discoverNodes(ConsulDiscoveryStrategyWrapper.java:61)

at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:74)

at com.hazelcast.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:42)

at com.hazelcast.cluster.impl.TcpIpJoiner.searchForOtherClusters(TcpIpJoiner.java:501)

at com.hazelcast.cluster.impl.SplitBrainHandler.searchForOtherClusters(SplitBrainHandler.java:75)

at com.hazelcast.cluster.impl.SplitBrainHandler.run(SplitBrainHandler.java:42)

at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:212)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_66]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_66]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]

at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)

at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

Caused by: com.orbitz.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class com.orbitz.consul.model.health.ImmutableNode] value failed (java.lang.IllegalStateException): Cannot build Node, some of required attributes are not set [taggedAddresses]

 at [Source: java.io.InputStreamReader@2e79eda5; line: 1, column: 95] (through reference chain: java.util.ArrayList[0]->com.orbitz.consul.model.health.Json["Node"])

at com.orbitz.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:262)

at com.orbitz.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:445)

at com.orbitz.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:464)

Note that the reported problem is the missing required attribute taggedAddresses. Invoking

curl http://localhost:8500/v1/health/service/hazelcast

I can see in fact that the taggedAddresses property is not returned. The strange thing though is that it isn't returned on any system - not on the other systems in this cluster and not on any of the other systems that I have deployed (which are all configured similarly). However, this is the only system logging this exception every two minutes. Note that as I have said that they are all configured the same and they all have the same versions of all of the components/jar files/classes.

I've googled to try and understand the problem, but I am stumped. Does anyone have any idea why I am getting this exception and/or have any recommendations on what to do to resolve it?

Peter

[Evan Gilman]

Hey Peter, sorry for the trouble.

Can you post your consul config as well as version number?

[Peter Heitman]

My consul version:

Consul v0.6.3

Consul Protocol: 3 (Understands back to: 1)

My consul config:

{

    "bootstrap": false,

    "server": true,

    "datacenter": "dc1",

    "node_name": "ph-dev-lion-cl-2",

    "data_dir": "/var/consul",

    "encrypt": "2ZWHI8O46lnl5nKuQ4r9+A==",

    "bootstrap_expect": 3,

    "log_level": "INFO",

    "bind_addr": "10.210.5.65",

    "rejoin_after_leave": true,

    "retry_join": ["ph-dev-lion-cl-1","ph-dev-lion-sl-1"],

    "skip_leave_on_interrupt": true,

    "disable_update_check": true,

    "ui": true,

    "domain": "primehome"

[Evan Gilman]

The structure your exception is referring to was introduced in Consul 0.6.4, however you are running Consul 0.6.3.

Is your discovery library communicating with Consul over RPC? If so, it was likely built against a newer version of Consul. It would be useful to know more, as there might be a compatibility issue lurking in there somewhere.

[Peter Heitman]

Thanks - I'm sure that was it. I was using 0.6.3 as the server in our test environment, but recently started using consul-client for hazelcast service discovery which was likely expecting 0.6.4. I really appreciate the quick help to resolve this.