Jira (PUP-8969) Sensitive parameters are not redacted from reports / agent output when used in templates

19 views
Skip to first unread message

Erik Hansen (JIRA)

unread,
Jun 27, 2018, 11:51:04 AM6/27/18
to puppe...@googlegroups.com
Erik Hansen updated an issue
 
Puppet / Bug PUP-8969
Sensitive parameters are not redacted from reports / agent output when used in templates
Change By: Erik Hansen
Priority: Normal Major
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Erik Hansen (JIRA)

unread,
Jun 27, 2018, 11:51:05 AM6/27/18
to puppe...@googlegroups.com
Erik Hansen created an issue
Issue Type: Bug Bug
Affects Versions: PUP 5.5.2
Assignee: Unassigned
Created: 2018/06/27 8:50 AM
Priority: Normal Normal
Reporter: Erik Hansen

Description: When I use the sensitive data type I expect it to be treated as sensitive when used in templates.  However it is not.
 
Steps to reproduce:  I've created a small module / repro case here: https://github.com/suckatrash/sensitive  Just apply the class "sensitive"
 
Actual Results
 
If I declare the class "sensitive" it will create two test files /test1 and /test2. 
 
If I make any changes in the files and run puppet again, I'll see:

+This string should be redacted: 'this is sensitive'

in the output of the puppet run and in the report.
 
If I don't unwrap the sensitive parameter and use the commented line in one of the templates I'll see this:
 

 +This string should be redacted: 'Sensitive [value redacted]'  

 
Which is great, but that text ends up in the file as well.
 
Expected Results:  I expect Puppet to output the '[value redacted]' string in reports and stdout, but I expect the cleartext string to be in the file on the system.

Josh Cooper (JIRA)

unread,
Jun 27, 2018, 2:43:10 PM6/27/18
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-8969
 
Re: Sensitive parameters are not redacted from reports / agent output when used in templates

If you wrap the template as Sensitive, then it works as expected for me:

file {'/tmp/test1':
  ensure => present,
  content => Sensitive(template('sensitive/test1.erb')),
}
 
file {'/tmp/test2':
  ensure => present,
  content => Sensitive(template('sensitive/test2.epp')),
}

Note redacting sensitive file content was fixed in PUP-6627:

$ touch /tmp/test{1,2}
$ bx puppet apply -e "include sensitive" --modulepath ~/work/modules --show_diff
Notice: Compiled catalog for localhost in environment production in 0.03 seconds
Notice: /Stage[main]/Sensitive/File[/tmp/test1]/content: [diff redacted]
Notice: /Stage[main]/Sensitive/File[/tmp/test1]/content: changed [redacted] to [redacted]
Notice: /Stage[main]/Sensitive/File[/tmp/test2]/content: [diff redacted]
Notice: /Stage[main]/Sensitive/File[/tmp/test2]/content: changed [redacted] to [redacted]
Notice: Applied catalog in 0.05 seconds
$ cat /tmp/test{1,2}
 
 
 
This string should be redacted: 'this is sensitive'
 
 
 
This string should be redacted: 'this is sensitive'
$ grep redacted ~/.puppetlabs/opt/puppet/cache/state/last_run_report.yaml
  message: "[diff redacted]"
  message: changed [redacted] to [redacted]
  message: "[diff redacted]"
  message: changed [redacted] to [redacted]
      previous_value: "[redacted]"
      desired_value: "[redacted]"
      message: changed [redacted] to [redacted]
      redacted: true
      previous_value: "[redacted]"
      desired_value: "[redacted]"
      message: changed [redacted] to [redacted]
      redacted: true
$ grep 'this is sensitive' ~/.puppetlabs/opt/puppet/cache/state/last_run_report.yaml
$

Reid Vandewiele (JIRA)

unread,
Jun 27, 2018, 2:44:03 PM6/27/18
to puppe...@googlegroups.com

I added a PR which I think clarifies the current limitations of Sensitive. It might make sense to break this ticket up into two, if I understand the problem correctly.

1: Provide a kind of Sensitive interpolation, wherein you can create a new Sensitive string by interpolating non-sensitive Strings and Sensitive Strings, where the end result should be a Sensitive string with a context-sensitive redacted print value. E.g. let something like Sensitive("safe text ${sensitive} more safe text") print as safe text [redacted] more safe text.

2: Make the show_diff operation of the File resource respect Sensitive types

Josh Cooper (JIRA)

unread,
Jun 27, 2018, 2:45:06 PM6/27/18
to puppe...@googlegroups.com

Erik Hansen (JIRA)

unread,
Jun 27, 2018, 3:00:10 PM6/27/18
to puppe...@googlegroups.com
Erik Hansen commented on Bug PUP-8969
 
Re: Sensitive parameters are not redacted from reports / agent output when used in templates

Josh Cooper - what I was expecting from the sensitive type is that the whole of the diff output would not be redacted, but only the sensitive string from within the diff.

Maybe this is more of a feature request, but I think there's an assumption that the Sensitive type should be respected everywhere in Puppet (and if it isn't we should document where Sensitive param values would be exposed)

Josh Cooper (JIRA)

unread,
Jun 27, 2018, 6:31:03 PM6/27/18
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-8969

Capturing additional comments. Ethan Brown recommended moving the sensitive data into a different file. The script can be managed as a normal file, and diffs displayed, while the sensitive data file content could be marked as sensitive:

file {'/tmp/test1':
  ensure => file,
  content => template('sensitive/test1.erb'),
}
file {'/tmp/credentials':
  ensure => file,
  owner => '...',
  group => '...',
  mode => '0600',
  content => Sensitive($credentials),
}

Erik Hansen (JIRA)

unread,
Jul 9, 2018, 3:42:05 PM7/9/18
to puppe...@googlegroups.com

Erik Hansen (JIRA)

unread,
Jul 9, 2018, 3:42:05 PM7/9/18
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Jul 9, 2018, 4:18:05 PM7/9/18
to puppe...@googlegroups.com
Henrik Lindberg commented on Improvement PUP-8969
 
Re: Sensitive parameters are not redacted from reports / agent output when used in templates

In order to support something like this, we need a new data type and rich-data turned on. We cannot send the result of the template directly to the catalog since we only have a choice of sensitive or not for the entire value. The new SensitiveSegments data type will essentially contain an Array of String mixed with instances of Sensitive. A to_s on an SensitiveSegments will render each with to_s and get redacted for all sensitive segments. An unwrap on a SensitiveSegments would unwrap and join all elements.

In addition to this, the resource logic needs to be able to handle SensitiveSegments data values and do the right operation unwrap or to_s depending on what it is doing (just like it does today by checking the binary sensitive flag), but it is more difficult as the value cannot just be unwrapped and used, the actual SensitiveSegment instance must be used to keep track of the parts of the string which should be redacted.

Further, there must be new features for EPP to allow it to use Sensitive values in the text. It will then produce an instance of SensitiveSegments instead of an instance of String. Doing the same for ERB would be very difficult. It could probable do this when seeing <%= Sensitive('secret') %> - i.e. that all interpolated expressions that evaluate to Sensitive makes it evaluate to the internal array of a SensitiveSegment instead of a String.

Another approach would be to make the File resource have features to combine a template with late binding values and know how to deal with both sensitive and encrypted information. It would expand the template on the agent side and now that it should use "redacted" or similar for sensitive/encrypted replacements. The work on Deferred / agent-side-function would help with that. Also complex to achieve.

All together this is a substantial amount of work. Ping Eric Sorenson for help with prio.

Rob Braden (JIRA)

unread,
Aug 1, 2018, 5:09:12 PM8/1/18
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Sep 26, 2018, 8:54:16 PM9/26/18
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Nov 19, 2018, 6:28:03 AM11/19/18
to puppe...@googlegroups.com
 
Re: Sensitive parameters are not redacted from reports / agent output when used in templates

Thought about this again. The main problem is that the value produced by the template is a single value from the perspective of a resource and it cannot be both sensitive and not sensitive (in part) at the same time. Further, resources require the value in clear text to operate on, and when it performs reporting it checks if the entire value is sensitive and wraps it (the entire value) in a Sensitive at that point. Since value is in clear text it cannot make it only partially sensitive. This means it needs to get an instance of SensitiveSegments as I commented on earlier. Since 6.0.0 rich-data is default, so there is no problem getting such a value across in the catalog, but then resources will see that data type. This means the value must be assembled two different ways, as clear text for operations, and as partially redacted values when reporting. We probably need a new kind of Parameter/Property that can store the SensitiveSegment, produce the clear text variant when initialized, and the partially redacted when the harness is reporting. That would mean that paths of logic that checks if a parameter is a sensitive parameter also needs to check for a parameter with SensitiveSegments and then obtain the partially redacted result.

This approach would work without major rewrite of the resource harness, but it is not trivial to achieve.

Jorie Tappa (JIRA)

unread,
Jun 10, 2019, 2:50:04 PM6/10/19
to puppe...@googlegroups.com

Jorie Tappa (JIRA)

unread,
Jun 12, 2019, 4:22:04 PM6/12/19
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Jun 13, 2019, 2:34:03 AM6/13/19
to puppe...@googlegroups.com
 
Re: Sensitive parameters are not redacted from reports / agent output when used in templates

A much simpler thing to do would be to make EPP wrap the result in a Sensitive if it interpolates something Sensitive. Doing the same for ERB means providing a different ERB or figuring out if it is possible to monkey patch Ruby's.

Josh Cooper (JIRA)

unread,
Sep 17, 2019, 11:28:05 PM9/17/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Oct 4, 2019, 6:50:03 PM10/4/19
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-8969
 
Re: Sensitive parameters are not redacted from reports / agent output when used in templates

Related I filed PUP-10092 to support interpolating sensitive values, which Reid Vandewiele had suggested earlier in this thread.

Austin Boyd (JIRA)

unread,
Dec 12, 2019, 8:47:04 AM12/12/19
to puppe...@googlegroups.com
Austin Boyd updated an issue
 
Change By: Austin Boyd
Zendesk Ticket IDs: 35573
Zendesk Ticket Count: 1

Corey Hickey (JIRA)

unread,
Dec 17, 2019, 9:28:04 PM12/17/19
to puppe...@googlegroups.com
Corey Hickey commented on Improvement PUP-8969
 
Re: Sensitive parameters are not redacted from reports / agent output when used in templates

I came upon this ticket because Sensitive didn't behave quite like I expected.

1. I expected Sensitive to behave like a flag rather than a distinct data type; thus, a variable could be declared as Sensitive at any point and then passed to classes, templates, defined types, etc. which didn't necessarily need to know/care whether the value is sensitive. Only the low-level printing/logging functions would redact as necessary.

2. I expected that any evaluation via template, string interpolation, etc. would inherit the Sensitive flag of any Sensitive components. I did not necessarily expect that the result could have mixed sensitive/non-sensitive components recognized during printing, though this would be very nice for viewing diffs of files that only contain some sensitive content.

Maybe my expectations were out of line, but that sort of thing does seem like useful behavior.

Thanks,
Corey

Josh Cooper (Jira)

unread,
Jun 6, 2020, 7:08:03 PM6/6/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Epic Link: PUP- 9637 8587
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Aug 27, 2020, 12:34:03 PM8/27/20
to puppe...@googlegroups.com

I expected Sensitive to behave like a flag rather than a distinct data type

Henrik Lindberg described the performance impact of doing that in PUP-10092, so the current behavior is unlikely to change.

I expected that any evaluation via template, string interpolation, etc. would inherit the Sensitive flag of any Sensitive components.

It is possible to do this for EPP templates as Henrik Lindberg described above.

Henrik Lindberg Thoughts on how to implement that? It looks like we could add a sensitive attribute to the EppExpression. Could the EppEvaluator set the sensitive attribute on the expression if it encounters a Sensitive token? Is there a better way for the evaluator to know that a sensitive value has been unwrapped, and automatically wrap the result in Sensitive?

Henrik Lindberg (Jira)

unread,
Aug 27, 2020, 6:56:04 PM8/27/20
to puppe...@googlegroups.com

Josh Cooper EPP results in AST elements specific to EPP for rendering a value as a string. That evaluation could evaluate the expression, check if the result is a Sensitive, and then instead of getting a REDACTED result it would unwrap the sensitive value and turn that into a string. It would then need to communicate the the top level that the entire result should be wrapped in a Sensitive. The passing back of that value is the most difficult as it needs to handle nesting (one EPP could call another for example).

Well - you managed to nerd snipe me

The relevant parts is in the `evaluator_impl.rb`

  def eval_EppExpression(o, scope)
    scope["@epp"] = []
    evaluate(o.body, scope)
    result = scope["@epp"].join
    result
  end
 
  def eval_RenderStringExpression(o, scope)
    scope["@epp"] << o.value.dup
    nil
  end
 
  def eval_RenderExpression(o, scope)
    scope["@epp"] << string(evaluate(o.expr, scope), scope)
    nil
  end

As you can see the EppExpression is where it joins all the fragments into the final string. It would need to be able to tell Sensitive entries apart from the others. The RenderExpression should for example not call string() if the evaluate() results in a Sensitive, and instead just add the Sensitive instance to the scope's @epp array. The RenderStringExpression is for the verbatim template text.

Then in EppExpression it would map the @epp array such that if it encountered a Sensitive values it would unwrap it and set the flag. Then it would join the segments, and finally if the flag was set, wrap the result in a Sensitive.

As a consequence code that evaluates epp (like the epp application for testing etc) needs to be updated to handle the case that it may get a Sensitive. The return type of epp() and inline_epp() may not be declared, if it is it needs to be specified as Variant[String, Sensitive[String]].

Hope that helps.

Josh Cooper (Jira)

unread,
Sep 15, 2020, 1:26:03 PM9/15/20
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages