consul-template: dedup, env variables and "same template" definition

[Ivan Savčić]

Hello,

I'm considering enabling the dedup flag due to a high number of watches, so I would avoid DDoSing the Consul server.

If a template has a {{ $hostname := env "HOSTNAME" -}} part which is later used in a key lookup (ie. key name == $hostname), where $HOSTNAME is of course different on all nodes using that template and is equal to the hostname of a node, will this template be regarded as unique on all nodes making the dedup moot?

Example:

{{ $hostname := env "HOSTNAME" -}}

{{ $listenIP := key_or_default (printf "ip/%s" $hostname) "127.0.0.1" -}}

Is every node's template being treated as unique in this case?

Thanks,

Ivan

[Armon Dadgar]

Hey Ivan,

I took a look at the code, and I think that because the hash of the template itself is the same,

all the nodes will treat this as a single template. However, the instance that is doing the actual

rendering (lock winner), will use it’s hostname to watch the “ip/hostname” key, so when the other

templates attempt to render, they will be missing the appropriate value for that key.

In this scenario, I’m not sure deduplication would help, since effectively every node is watching

a distinct key in Consul, so there is no real de-duplication of effort that is possible. That mode is meant

to reduce duplicate watches (e.g. 50 nodes watching the 1 set of keys) instead here it’s 50 nodes

watching 50 distinct keys.

One way you could restructure this is to write a single key to Consul, which is a JSON object of {“hostname”: “ip”}.

Then you could refactor to something like:

{{ $hostname := env “HOSTNAME” -}}

{{ $listenMap := key “ip_map” | toJSON }}

{{ $listenIP := $listenMap[$hostname }}

This way all the nodes are watching the same key, so the de-duplicaiton could actually take effect, and then

the client side is able to do the final lookup to determine the appropriate key.

Hope that helps!

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/18198738-08b6-4bd4-8e5b-9f8f8b151d24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.