Consul-Template 0.21.1 High Memory Usage

[Corey Fonseca]

We're running consul-template 0.21.1 within all of our microservice application containers. Our actual template is written to pull 13 different keys from Consul, and two different keys from Vault. Consul-Template will of course read the template, render the output to an appsettings.json file, and finally, start our application. This all works as expected. 

Consul-Template is running using the default behavior of running as a long-lived process/daemon. It does not shut down, and it watches for changes to our config keys/values in Consul. We're noticing that the longer our microservices are running (and thus, consul-template), the more memory consul-template consumes. Memory usage slowly continues to grow and grow over time, until our kubernetes nodes run out of resources and the pods are evicted. This is causing issues for obvious reasons, and we're not sure what is causing Consul-Template to behave this way. 

Here's an example of one of our templates:

{{- $Env := "qa" -}}

{{- $Svc := "ExampleService" -}}

{

    "Dependency1ServerHost": "{{ printf "services/%s/%s/Dependency1ServerHost" $Svc $Env | key }}",

    "Dependency1ServerPort": "{{ printf "services/%s/%s/Dependency1ServerPort" $Svc $Env | key }}",

    "AllowedHosts": "{{ printf "services/%s/%s/AllowedHosts" $Svc $Env | key }}",

    "Something": {

        "SomethingElse": {

            "SomethingElse": {

                "Enabled": "{{ printf "services/%s/%s/Something/SomethingElse/SomethingElse/Enabled" $Svc $Env | key }}",

                "LogLevel": "{{ printf "services/%s/%s/Something/SomethingElse/SomethingElse/LogLevel" $Svc $Env | key }}",

                "Name": "{{ printf "services/%s/%s/Something/SomethingElse/SomethingElse/Name" $Svc $Env | key }}"

            },

        "Enabled": "{{ printf "services/%s/%s/Something/SomethingElse/Enabled" $Svc $Env | key }}"

        }

    },

    "ConnectionString": "{{ with printf "secret/services/%s/%s" $Svc $Env | secret }}{{ .Data.ConnectionString }}{{ end }}",

    "RandomKey": "{{ printf "services/%s/%s/RandomKey" $Svc $Env | key }}",

    "ExampleKey": "{{ printf "services/%s/%s/ExampleKey" $Svc $Env | key }}",

    "RedisConnectionString": "{{ with printf "secret/services/%s/%s" $Svc $Env | secret }}{{ .Data.RedisConnectionString }}{{ end }}",

    "ServiceName": "{{ printf $Svc }}"

}

Here's an example of our consul-template config file:

consul {

    address = "consulserver:8500"

    token = "jgieowjgieodsjgiedsogjs"

    retry {

        enabled = true

        attempts = 5

        backoff = "250ms"

        max_backoff = "10s"

    }

}

vault {

    address = "https://vaultserver:8200"

    vault_agent_token_file = "file.txt"

    ssl {

        server_name = "vaultserver.domain.com"

        verify = false

    }

}

reload_signal = "SIGHUP"

kill_signal = "SIGINT"

max_stale = "5m"

wait {

    min = "5s"

    max = "60s"

}

template {

    command_timeout = "30s"

    error_on_missing_key = true

    source = "appsettings.tpl"

    destination = "appsettings.json"

}

exec {

    splay = "5s"

    kill_timeout = "3s"

}

The template obviously has a lot of separate reads due to the way it's written, but it's written that way only because we're mixing Consul data and Vault data into the same file, and it needs to be in a valid .json format. I'm not sure how else to do this.

Any suggestions on how to make my template more efficient, or otherwise handle the growing memory utilization of our consul-template processes? Any idea why consul-template continues to use more and more memory the longer it runs? This is quite concerning and is causing us major trouble right now.

Thanks in advance for any thoughts/suggestions.

-Corey

[John Eikenberry]

Hey Corey, thanks for reaching out.

This really sounds like a memory leak as consul-template should grow for a bit then level out, not keep growing endlessly. Would you mind filing an issue in github about this?

Thanks.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/a3564d32-72c1-4e2f-9600-c2e1ea33fd12%40googlegroups.com.

--

John Eikenberry <jeike...@hashicorp.com>

________________________________________________________________________
"Perfection is attained, not when no more can be added, but when no more
 can be removed." -- Antoine de Saint-Exupery

[Corey Fonseca]

Hey John,

Thanks for the quick reply! We ended up upgrading to 0.21.3 and that appears to have fixed our memory issue. I'm happy to post a GitHub issue if you'd still like me to, although I'm not sure if it's still important or relevant. Thoughts?

Thanks again,

Corey

To unsubscribe from this group and stop receiving emails from it, send an email to consu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/a3564d32-72c1-4e2f-9600-c2e1ea33fd12%40googlegroups.com.

[Corey Fonseca]

Just to add, here's a Grafana graph showing memory usage statistics from our GKE cluster over the last 36h. The high, spikey memory usage is from using consul-template 0.21.1. At ~16:00, we upgraded to 0.21.3 and redeployed all of our pods. Memory usage dropped substantially and has been flat since.

http://prntscr.com/p47hyb

[John Eikenberry]

That is great news Corey.

I had planned on looking into the possibility it was related to the regression with non-renewable secrets constantly refreshing (fixed in 0.21.3). It wasn't directly obvious from your config/template but it seemed like that might have been the cause. Sorry about that.

Don't worry about filing a ticket. This was the first report of a memory leak and it was most certainly that regression, so there is no need.

Thanks.

On Tue, Sep 10, 2019 at 11:05 AM Corey Fonseca <cfon...@blueport.com> wrote:

Just to add, here's a Grafana graph showing memory usage statistics from our GKE cluster over the last 36h. The high, spikey memory usage is from using consul-template 0.21.1. At ~16:00, we upgraded to 0.21.3 and redeployed all of our pods. Memory usage dropped substantially and has been flat since.

http://prntscr.com/p47hyb

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/afea067b-7801-4db3-a388-e694327fa9eb%40googlegroups.com.