Jira (PUP-10612) Wrap values in Sensitive in eyaml_lookup_key

33 views
Skip to first unread message

Tomáš Virtus

unread,
Aug 7, 2020, 6:21:03 PM8/7/20
to puppe...@googlegroups.com
Tomáš Virtus created an issue
 
Puppet / New Feature PUP-10612
Wrap values in Sensitive in eyaml_lookup_key
Issue Type: New Feature New Feature
Assignee: Unassigned
Components: Functions, Hiera & Lookup
Created: 2020/08/07 3:20 PM
Priority: Normal Normal
Reporter: Tomáš Virtus

Values looked up with eyaml_lookup_key are returned as strings. This reveals them in console, reports, etc. To prevent this the values should be wrapped in Sensitive type. Currently automatic lookup files when class parameter is of type Sensitive[String] and value looked up is of type String. This leads people to suggest manually converting each sensitive value to Sensitive type via lookup_options:

lookup_options:
  '^profile::.+::sensitive_\w+$':
    convert_to: 'Sensitive' 

While this kind of does the job, it's fragile and repetitious. While I can imagine use case where someone would want encrypted values revealed in diffs, I'd guess that for most use-cases the value looked up by eyaml_lookup_key should be wrapped in Sensitive.

I've managed to hack together the following function in my environment to do it:

Puppet::Functions.create_function(:eyaml_lookup_key_sensitive) do
  def eyaml_lookup_key_sensitive(key, options, context)
    cleartext = call_function('eyaml_lookup_key', key, options, context)
    Puppet::Pops::Types::PSensitiveType::Sensitive.new(cleartext)
  end
end

But this would be better done in eyaml_lookup_key.rb to directly store wrapped values into the cache.

In order to not break backward compatibility the opt-in could be stored in options hash as e.g. convert_to_sensitive boolean option. To make it fancier, the values of this opt-in could be:

  • none: current behaviour
  • encrypted: only wrap encrypted values
  • all: wrap all values

Another option would be to make Puppet automatically convert String to Sensitive[String] to make automatic class parameter lookup work. This will however still reveal values in e.g. puppet lookup.

 

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Tomáš Virtus

unread,
Aug 7, 2020, 6:25:03 PM8/7/20
to puppe...@googlegroups.com
Tomáš Virtus updated an issue
Change By: Tomáš Virtus
Values looked up with {{eyaml_lookup_key}} are returned as strings. This reveals them in console, reports, etc. To prevent this the values should be wrapped in {{Sensitive}} type. Currently automatic lookup files when class parameter is of type {{Sensitive[String]}} and value looked up is of type {{String}}. This leads people to suggest manually converting each sensitive value to {{Sensitive}} type via {{lookup_options}}:
* [https://blog.example42.com/2019/04/04/puppet_sensitive_data/]
* [https://puppet.com/blog/my-journey-securing-sensitive-data-puppet-code/]

{code:yaml}lookup_options:
  '^profile::.+::sensitive_\w+$':
    convert_to: 'Sensitive' {code}
While this kind of does the job, it's fragile and repetitious. While I can imagine use case where someone would want encrypted values revealed in diffs, I'd guess that for most use-cases the value looked up by {{eyaml_lookup_key}} should be wrapped in {{Sensitive}}.


I've managed to hack together the following function in my environment to do it:
{code:ruby}Puppet::Functions.create_function(:eyaml_lookup_key_sensitive) do

  def eyaml_lookup_key_sensitive(key, options, context)
    cleartext = call_function('eyaml_lookup_key', key, options, context)
    Puppet::Pops::Types::PSensitiveType::Sensitive.new(cleartext)
  end
end
{code}

But this would be better done in {{eyaml_lookup_key.rb}} to directly store wrapped values into the cache.

In order to not break backward compatibility the opt-in could be stored in {{options}} hash as e.g. {{convert_to_sensitive}} boolean option. To make it fancier, the values of this opt-in could be:
* {{none}}: current behaviour
* {{encrypted}}: only wrap encrypted values
* {{all}}: wrap all values

Use case for this is when someone sets {{eyaml_lookup_key}} and its options in {{defaults}} and scatters encrypted values throughout layers (i.e. when there's no one dedicated layer for sensitive keys) while accepting performance hit of lookup function vs {{data_hash}}.

Another option would be to make Puppet automatically convert {{String}} to {{Sensitive[String]}} to make automatic class parameter lookup work. This will however still reveal values in e.g. {{puppet lookup}}.

 

Tomáš Virtus

unread,
Aug 8, 2020, 7:59:03 AM8/8/20
to puppe...@googlegroups.com
Tomáš Virtus commented on New Feature PUP-10612
 
Re: Wrap values in Sensitive in eyaml_lookup_key

These are issue about improving the documentation, AFAIK taking what's suggested in the blogs I linked. This issue however about fragility of the proposed solution.

Josh Cooper (Jira)

unread,
Nov 17, 2020, 3:50:03 PM11/17/20
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-10612

The none and encrypted options make sense to me and seem pretty easy to implement:

# diff --git a/lib/puppet/functions/eyaml_lookup_key.rb b/lib/puppet/functions/eyaml_lookup_key.rb
index 5fbae35b3a..d6c6573c80 100644
--- a/lib/puppet/functions/eyaml_lookup_key.rb
+++ b/lib/puppet/functions/eyaml_lookup_key.rb
@@ -88,6 +88,11 @@ Puppet::Functions.create_function(:eyaml_lookup_key) do
       begin
         tokens = Hiera::Backend::Eyaml::Parser::ParserFactory.hiera_backend_parser.parse(data)
         data = tokens.map(&:to_plain_text).join.chomp
+        if options['convert_to_sensitive'] == 'encrypted'
+          data = Puppet::Pops::Types::PSensitiveType::Sensitive.new(data)
+        else
+          data
+        end
       rescue StandardError => ex
         raise Puppet::DataBinding::LookupError,
           _("hiera-eyaml backend error decrypting %{data} when looking up %{key} in %{path}. Error was %{message}") % { data: data, key: key, path: options['path'], message: ex.message }

That code doesn't take into account all, but I don't think that'd be hard either. To test this, I did the following:

# gem install hiera-eyaml
# puppetserver gem install hiera-eyaml
# eyaml createkeys
# mkdir /etc/puppetlabs/puppet/eyaml
# cp ./keys/* /etc/puppetlabs/puppet/eyaml/
# eyaml encrypt -s 'hello there'
....
# cat /etc/puppetlabs/code/environments/production/hiera.yaml
---
version: 5
defaults:
  # The default value for "datadir" is "data" under the same directory as the hiera.yaml
  # file (this file)
  # When specifying a datadir, make sure the directory exists.
  # See https://puppet.com/docs/puppet/latest/environments_about.html for further details on environments.
  # datadir: data
  # data_hash: yaml_data
hierarchy:
  - name: "Secrets"
    lookup_key: eyaml_lookup_key
    path: "common.eyaml"
    options:
      convert_to_sensitive: encrypted
      pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
      pkcs7_public_key:  /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
# cat /etc/puppetlabs/code/environments/production/data/common.eyaml
---
profile::hiera_test::password: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAHbfIep0iZWSU+GDkxTRHvGzdiCjkcYEe8gnV1z3wYIdmU2gqgJQ6Qovh3bzlENVs3GLE4cyJw9JFDyvHu4+UnbjDOuVQb0YrIWyA6QnZLPIs05qejMm78MNG/wFZFiyaD5vuK5/mgK09ilJwib6Xxd3vFQ0kUnL59yoswpYLTUfjl15U4usfCRT9TYSW1hrJ2FQYkpcoObGhkaqxtNXlR1lB0rlPDjqSdAmneiNUhZA7Dm/fp2Ola40L6hUWNrS6jN+A5TcBjmn6ZHvgz17J92f3VH1mWcXSBg2u34m12DQh6V74io+D6xg2eXAnWmt0/ak1DeNtdTnPIpe+JVF9zTA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBAH7hzPI9TwHcDLu6I50NQUgBCeSg/b5b7FXIWM+9BiGdpV]
# puppet lookup profile::hiera_test::password
--- !ruby/object:Puppet::Pops::Types::PSensitiveType::Sensitive
value: hello there

With the convert_to_sensitive option removed, I get:

# puppet lookup profile::hiera_test::password
--- hello there

Josh Cooper (Jira)

unread,
Aug 19, 2021, 1:42:02 AM8/19/21
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Josh Cooper
 
Change By: Josh Cooper
Assignee: Josh Cooper
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Josh Cooper (Jira)

unread,
Aug 20, 2021, 5:56:02 PM8/20/21
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-10612
 
Re: Wrap values in Sensitive in eyaml_lookup_key

An alternate approach would be to allow the user to specify the pcore representation of the sensitive value as Henrik Lindberg suggested in https://tickets.puppetlabs.com/browse/PUP-10654?focusedCommentId=757774&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-757774

So the example above could be written as:

profile::hiera_test::password: {"__ptype"=>"Sensitive", "__pvalue"=>"ENC[PKCS7,MIIBeQYJ...]"}

That way the type is unambiguous and we don't need lookup_options. Also that might fix the "revealing values in puppet lookup issue"?

Josh Cooper (Jira)

unread,
Aug 20, 2021, 6:18:02 PM8/20/21
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-10612

Linking some related tickets where this has been discussed extensively

Josh Cooper (Jira)

unread,
Aug 20, 2021, 6:18:03 PM8/20/21
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Henrik Lindberg
 
Change By: Josh Cooper
Assignee: Josh Cooper Henrik Lindberg

Josh Cooper (Jira)

unread,
Nov 30, 2021, 4:14:03 PM11/30/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Epic Link: PUP-11371
Reply all
Reply to author
Forward
0 new messages