Consul, Consul-haproxy and DNS resilience

[Raul]

Hi guys, first great job on Consul and Serf! Compared to a lot of frameworks and given its tremendous functionality its a pleasure to use. It just works without a boatload of complex configuration, and I hope we don't lose this.

I just came across consul-haproxy, and got me thinking a bit of  the resilience of DNS for service discovery. A lot of apps resolve at start and not runtime dynamically, and it's not consistent what approach any app would take, for instance nginx and haproxy by most accounts resolve at start.

So without a reload they will not re-resolve any dns names, which then necessitates the use of something like consul-haproxy which reload configs.

So in other scenarios consul may have marked some nodes down, and be offering updated healthy nodes but the app could most likely still be pointing to an instance which has already been marked down by Consul, untill reloaded. How does one work around this, or am I missing something?

Thanks
Raul

[Armon Dadgar]

Hey Raul,

By default, DNS serves with a 0 TTL value. So in many cases those results should not be

cached. For applications that are erroneously caching, you can use consul-haproxy to

setup their configuration and automatically reload them on changes.

Hope that helps!

Best Regards,

Armon Dadgar

--
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raul]

Hi Armon,

I find Consul elegant to use, but it appears from some early research consul-haproxy would be needed more often than not. The TTL would not matter in this case as no querying would happen untill reload.

I came across an interesting post on this in the context of service discovery by Igor from smartstack.

Service discovery in the cloud

It appears from the post a large number of apps cache dns on load, refreshing only on restart or reload. Would you agree with this?

Maxim Dounin, a well known Nginx expert, suggests a workaround here to use a variable for backend, to force Nginx to resolve dynamically.
On this 2 year post the Haproxy author thinks differently, but this is 2 years old, and dynamic host resolution is the most requested features in Haproxy.

Would love to hear other consul users experiences on this .

Thanks
Raul

On Friday, August 15, 2014 11:47:52 PM UTC+5:30, Armon Dadgar wrote:

Hey Raul,

By default, DNS serves with a 0 TTL value. So in many cases those results should not be

cached. For applications that are erroneously caching, you can use consul-haproxy to

setup their configuration and automatically reload them on changes.

Hope that helps!

Best Regards,

Armon Dadgar

[Armon Dadgar]

Raul,

I think this issue depends a lot on the particular runtime of the application. Some runtimes like the JVM are particularly bad about this. Others like Go have no such issues.

It's something that requires playing with for sure.

For cases with DNS caches being problematic, using consul-haproxy is an easy work around.

Armon Dadgar

Sent from my iPhone

[David Pelaez]

You can take a look at srv-router I wrote it for an internal project at Vlipco and solves the same problem that consul-haproxy but using nginx+lua. It doesn't have caching at the moment, which makes haproxy a better choice in the meantime, but it's an extremely simple solution (1 script mostly) that could be worth looking.