consul-template: Exec mode with multiple "template" declarations

[Chris Stevens]

I have been running CT with a php-fpm application under "exec mode" for the last week or so.

The app has worked perfectly, but I noticed that the php-fpm process was being reloaded roughly every 6-8 minutes.

After digging in to, I noticed that CT was triggering the re-render and reload.

If I remove all but 1 "template" declaration, the process becomes stable and with no unexpected reloads.

The docs seem to indicate that multiple "template" declarations are allowed with exec, but the examples only show one.

Should exec mode work with more than 1 "template" declaration?

My use case is the php-fpm process run via "exec" and 4 application templates that need to be rendered / managed. See "app.hcl" below.

- Chris

====

consul-template: 0.16.0

consul: 0.7.0

vault: 0.6.2

app.hcl:

consul = "127.0.0.1:8500"

log_level = "DEBUG"

wait = "5s:10s"

vault {

  address = "https://active.vault.service.consul:8200"

  unwrap_token = true

}

syslog {

  enabled = true

}

deduplicate {

  enabled = true

}

exec {
  command = "php-fpm --fpm-config provision/php/php-fpm.conf"
  splay = "5s"
  reload_signal = "SIGUSR2"
  kill_signal = "SIGQUIT"
  kill_timeout = "10s"
}


# Application configuration
template {
  source = "provision/config/config.ctmpl"
  destination = "app/Config/config.php"
  perms = 0640
  backup = false
}

# 3 other "template" declarations removed ....

Select log entries with 1 "template" directive active in app.hcl:

2016/10/07 20:19:25.731458 [DEBUG] (runner) enabling global quiescence for "provision/config/config.ctmpl"
2016/10/07 20:19:25.731673 [INFO] (runner) watching 55 dependencies
2016/10/07 20:19:25.731773 [INFO] (child) spawning "php-fpm" ["--fpm-config" "provision/php/php-fpm.conf"]
[07-Oct-2016 20:19:25] NOTICE: fpm is running, pid 1736
[07-Oct-2016 20:19:25] NOTICE: ready to handle connections
[07-Oct-2016 20:19:25] NOTICE: systemd monitor interval set to 10000ms

Select log entries with 2 "template" directives active in app.hcl:

2016/10/07 20:16:01.397742 [INFO] (runner) watching 75 dependencies

2016/10/07 20:16:58.914282 [INFO] (child) reloading process

2016/10/07 20:16:58.914595 [DEBUG] (child) waiting 0.54s for random splay

2016/10/07 20:16:59.455619 [DEBUG] (runner) enabling global quiescence for "provision/config/cache.ctmpl"

2016/10/07 20:16:59.455691 [INFO] (runner) watching 75 dependencies

[07-Oct-2016 20:16:59] NOTICE: Reloading in progress ...

....

2016/10/07 20:18:14.131450 [INFO] (child) reloading process

2016/10/07 20:18:14.131759 [DEBUG] (child) waiting 3.59s for random splay

2016/10/07 20:18:17.721804 [DEBUG] (runner) enabling global quiescence for "provision/config/config.ctmpl"

2016/10/07 20:18:17.721859 [INFO] (runner) watching 75 dependencies

[07-Oct-2016 20:18:17] NOTICE: Reloading in progress ...

[Chris Stevens]

I just tried these scenarios against CT 0.17.0-dev. Same results. Stable with 1 "template" directive. Frequent reloading with more than 1 "template" directive.

I also noticed that CT 0.17.0-dev was not able to unwrap a wrapped token, but need to try to isolate that issue further.

[Chris Stevens]

Still trying to track this down...

It looks like a few of the runner_test.go tests configure a single template for "exec" mode:

https://github.com/hashicorp/consul-template/blob/master/manager/runner_test.go#L1402

https://github.com/hashicorp/consul-template/blob/master/manager/runner_test.go#L1500 

Would the reloading/re-rendering issues be observable via these tests if we added a second template directive that sets the "baz" key to "contents" for instance?

Are there other "exec mode" tests that might be better to check for this behavior instead?

- Chris

[James Phillips]

Hi Chris,

Would you mind opening an issue over in consul-template for this? We
can track this down over there.

Thanks!

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/consul-tool/446698e2-54aa-4044-9fec-5b6ce3d4cd6d%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Chris Stevens]

Hi James,

Seth and I worked through some of this back in October:

https://github.com/hashicorp/consul-template/issues/750

I was able to observe stable operation by removing the "wait" directive.

This was for a PoC. We should be returning to implement it for real soon, so I can test it again with the latest builds.

- Chris