Jira (PUP-9692) hiera explain should redact values converted to Sensitive in --debug mode

23 views
Skip to first unread message

Henrik Lindberg (JIRA)

unread,
May 5, 2019, 4:28:03 AM5/5/19
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
 
Puppet / New Feature PUP-9692
hiera explain should redact values converted to Sensitive in --debug mode
Change By: Henrik Lindberg
Summary: Passwords not redacted hiera explain should redact values converted to Sensitive in -- debug mode
Component/s: Types and Providers
Component/s: Hiera & Lookup
Issue Type: Bug New Feature
*Puppet Version:*
*Puppet Server Version:*
*OS Name/Version:*

Describe your issue in as much detail as possible…
Describe steps to reproduce…

*Desired Behavior:*

*Actual Behavior:*

Please take a moment and attach any relevant log output and/or manifests. This will help us immensely when troubleshooting When using hiera's convert_to Sensitive, the issue.

Examples:
Run puppet agent with explain output that is turned on by -- test --trace -- debug

Relevant sections of {{/var/log/puppetlabs/puppetserver/puppetserver.log}} or any applicable logs from will include the same directory.

For more detailed information turn clear text looked up value as the server logs by upping explainer traces the log level in the server's logback lookup . xml

Relevant sections of configurations files (puppet Later this value is converted to Sensitive . conf, hiera.conf, Server's conf.d, defaults/sysconfig)

For memory issues with server heap dumps are also helpful The desire is that such values would be redacted in the log .
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Sanjay Tripathi (JIRA)

unread,
May 6, 2019, 2:36:04 AM5/6/19
to puppe...@googlegroups.com
Sanjay Tripathi commented on New Feature PUP-9692
 
Re: hiera explain should redact values converted to Sensitive in --debug mode

Hello Henrik,

Instead of converting to Sensitive, is there any other way to prevent logging of passwords in clear-text when  --debug is used? Our passwords are encrypted in the yaml but get logged in clear-text when the hiera interpolation is performed. From checking the documentation, converting to Sensitive seems to be the only way to avoid it. But if there is a better or easier way to not have the passwords logged in debug mode, that would be very helpful.

Thanks.

Henrik Lindberg (JIRA)

unread,
May 6, 2019, 4:04:03 AM5/6/19
to puppe...@googlegroups.com

Sanjay Tripathi The best (most secure) you can do is to use this: https://forge.puppet.com/puppet/vault_lookup as when it is used the secret is never in clear text when compiling, nor is it in the catalog. Instead it is resolved with a call from the agent to the external vault service when the catalog is applied on the agent.

Sanjay Tripathi (JIRA)

unread,
May 6, 2019, 2:30:03 PM5/6/19
to puppe...@googlegroups.com

Thanks, Henrik. We are still on Puppet 5.5, whereas the documentation of vault_lookup states: This is expected to be run using the Deferred type, which requires Puppet 6.0.0 or later.

Can you please suggest a solution which works with Puppet 5.x, as well as older versions?

Henrik Lindberg (JIRA)

unread,
May 6, 2019, 5:36:03 PM5/6/19
to puppe...@googlegroups.com

The support for Deferred in Puppet 6.0 was introduced for the very purpose of making the entire chain secure. So, no I don't know of another solution that would remove the possibility that clear text gets logged when development tools (like --debug) are used.

Sanjay Tripathi (JIRA)

unread,
May 6, 2019, 6:08:02 PM5/6/19
to puppe...@googlegroups.com

Thanks, Henrik.
The vault_lookup doc link states "wrap the result in Puppet's Sensitive type, which prevents the value from being logged."

So it looks like Sensitive is the way to redact the value in the log – Puppet 6 or even older version.

I will try your suggestion about alias but other than using convert_to, if there is anything else I need to change for using Sensitive, please let me know.

Sanjay Tripathi (JIRA)

unread,
May 6, 2019, 9:20:02 PM5/6/19
to puppe...@googlegroups.com

Can this ticket be changed to a problem/bug, rather than a "feature"? Except turning off --debug, there is no other solution/workaround.

Thanks.

Henrik Lindberg (JIRA)

unread,
May 7, 2019, 9:16:02 AM5/7/19
to puppe...@googlegroups.com

It is not a bug in the sense it does what it is supposed/specified to do. We never tried to design it such that --debug output would redact values because you need to be able to debug how it comes up with a Sensitive value... There is no difference in terms of priorities if it is a "new feature" or "bug" in that a new feature can have higher prio than a bug. A status of "new feature" simply indicates that there is some work to do wrt defining what the behaviour should be.

You should not do regular runs with --debug turned out. That slows down the system quite a lot and it produces a lot of output. So don't do that unless actually debugging.

The deferred lookup using vault will as i understand it return Sensitive values automatically. It is fine to also wrap them in Sensitive on the compiling side. It would be visible in the --debug output, but it would only be the key that is in clear text, not the value (since that is looked up much later on the agent side).

Sanjay Tripathi (JIRA)

unread,
May 7, 2019, 8:58:02 PM5/7/19
to puppe...@googlegroups.com

Thanks, Henrik.

I do agree that debug mode generates a lot of clutter and should be avoided in production.

But regardless of the mode in which it is run, the passwords should never be exposed in clear-text.

If there's some undocumented hook for processing the debug output, we would like to use that as a short-term workaround.

Also, we frequently ask our customers for debug output when we are investigating issues, which exposes their passwords.

So an expedited fix for Puppet 5.5 (and lower) would be very helpful!

Henrik Lindberg (JIRA)

unread,
May 8, 2019, 6:58:03 AM5/8/19
to puppe...@googlegroups.com

I suppose it would be possible to use a different logging implementation - they are configurable. Logging gets clear text, so would need to pass some kind of custom filter to identify and redact sensitive text, which more or less only works if you know what will be logged since there is no indication in any way that something is sensitive or not - it is just text.

Jorie Tappa (JIRA)

unread,
May 13, 2019, 1:11:03 PM5/13/19
to puppe...@googlegroups.com
Jorie Tappa updated an issue
 
Change By: Jorie Tappa
Team: Server

Sanjay Tripathi (JIRA)

unread,
Jun 20, 2019, 12:44:03 AM6/20/19
to puppe...@googlegroups.com
Sanjay Tripathi commented on New Feature PUP-9692
 
Re: hiera explain should redact values converted to Sensitive in --debug mode

Hello, Is there any possibility of this issue getting fixed in the near future? We would like to up-take the fix of https://tickets.puppetlabs.com/browse/PUP-9693 (thanks for fixing it!) but it would be really useful to have the data redacted in the log, when the debug flag is passed, for data which has convert_to: "Sensitive". We "heavily rely" on the debug flag for all the environment information Puppet logs in debug mode. Thanks.

Henrik Lindberg (JIRA)

unread,
Jun 20, 2019, 6:28:02 AM6/20/19
to puppe...@googlegroups.com

Yet another difficulty with this is that the conversion to Sensitive occurs much later than all the logging that comes from hiera explain - it is just getting values from data stores, and at that point have no idea that a value later will become a Sensitive value. It may follow complicated chains with aliases and interpolations, merges etc. across three layers of hiera.yaml, call backend functions etc.

Without a major rewrite of the hiera --explain support I think the only thing to do is to enable an opt-out of the --explain output when --debug is on.
That would be something that could be implemented with reasonable effort.

Josh Cooper (Jira)

unread,
Oct 23, 2020, 8:02:03 PM10/23/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Affects Version/s: PUP 5.5.z
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Cas Donoghue (Jira)

unread,
Dec 2, 2020, 2:45:03 PM12/2/20
to puppe...@googlegroups.com
Cas Donoghue commented on New Feature PUP-9692
 
Re: hiera explain should redact values converted to Sensitive in --debug mode

Josh Cooper were there any recent changes in how Sensitive works that would have fixed this issue?

Cas Donoghue (Jira)

unread,
Dec 17, 2020, 7:14:03 PM12/17/20
to puppe...@googlegroups.com
Cas Donoghue commented on New Feature PUP-9692

zendesk.jira (Jira)

unread,
Jan 5, 2021, 11:51:04 AM1/5/21
to puppe...@googlegroups.com
zendesk.jira updated an issue
 
Change By: zendesk.jira
Labels: jira_escalated

zendesk.jira (Jira)

unread,
Jan 5, 2021, 11:51:04 AM1/5/21
to puppe...@googlegroups.com
zendesk.jira updated an issue
Change By: zendesk.jira
Zendesk Ticket Count: 1
Zendesk Ticket IDs: 42587

Josh Cooper (Jira)

unread,
Mar 1, 2021, 2:58:01 PM3/1/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Environment: Problem 1:
When puppet agent is run with --debug option, passwords get logged in plain-text, although the passwords are stored encrypted in the yaml.

Example:
2019-05-03 15:58:36 -0700 Puppet (debug): Lookup of 'my_con_pwd'
Searching for "my_con_pwd"
Global Data Provider (hiera configuration version 3)
Using configuration "/home/tstuser/puppet/hiera.yaml"
Hierarchy entry "eyaml"
Path "/home/tstuser/puppet/production/data/defaults.yaml"
Original path: "defaults"
No such key: "my_con_pwd"
Path "/home/tstuser/puppet/production/data/my_prod_config.yaml"
Original path: "my_prod_config."
Found key: "my_con_pwd" value: "MyT3stP"

The password is not redacted even if lookup_options is added for this data, with convert_to: Sensitive --

The lookup_options is set in common.yaml like:

lookup_options:
"^*my_con_pwd":
convert_to:
- "Sensitive"

Puppet seems to detect that this data is sensitive,  because it displays the following message after the lookup:

Applying convert_to lookup_option with arguments [Sensitive]

But the value is not redacted, as shown in the log snippet.

Problem 2:
If lookup_option is configured as shown above but this data is interpolated for another setting in a list, like:
my_env_list:
  db_settings:
     db_pwd: "%\{hiera('my_con_pwd')}"

then the following error occurs:

2019-05-03 16:11:26 -0700 Puppet (err): Evaluation Error: Error while evaluating a Function Call, Lookup of key failed: The convert_to lookup_option for key 'my_con_pwd' raised error: undefined method `call_function' for #<Hiera::Scope:0x0000000003a1cab8>

Please let me know if there is any other way to redact passwords when --debug is used. Although this occurs only when --debug is passed, it is not acceptable security practice.

Josh Cooper (Jira)

unread,
Mar 1, 2021, 2:58:02 PM3/1/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
When using hiera's convert_to Sensitive, the explain output that is turned on by --debug will include the clear text looked up value as the explainer traces the lookup. Later this value is converted to Sensitive.


The desire is that such values would be redacted in the log.


*Original Text*

Josh Cooper (Jira)

unread,
Nov 30, 2021, 4:18:01 PM11/30/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Epic Link: PUP-11371
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Josh Cooper (Jira)

unread,
Oct 26, 2022, 2:57:04 PM10/26/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Team: Froyo Phoenix
This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages