Vault pseudo leaf encryption

[ralo...@gmail.com]

Hi all.

(reposting because of a silly copy/paste error - sorry about that.)

Vault was a great addition to Ansible.

Some complained about the lack of leaf encryption.

I just published a blog post about a few of our practices involving Ansible,

and here is an excerpt about how we easily work around this limitation.

Hope it helps some of you.

(Blog post is here: http://reinteractive.net/posts/167-ansible-real-life-good-practices)

Excerpt:

Vault Pseudo leaf encryption

Very recently, with version 1.5, Ansible introduced Ansible Vault, a way to encrypt data in the playbook, and decrypt it at run time. This feature was highly requested, and gives Ansible its true place among platform management tools.

The thing is: what we like about Ansible is the readability, and encryption has a way of making things, well, less readable…
ansible-vault command will encrypt or decrypt the whole var file, you can not encrypt just the value of a variable. The solution is simple enough: create a second var file, just for the sensitive data. But this raises another issue: your variables are now spread over multiple files, and some of them encrypted. This can get messy. For instance, if you define a dictionary of variables and only one of them is sensitive, you have to encrypt the whole dictionary.

Leaf encryption was (is) a feature request, but in the meantime, there is an elegant way of keeping it both readable and secure: nested variables.

For every sensitive variable, you create a prefixed double that goes in an encrypted file.

# var_file
db_password: {{ vaulted_db_passord }}
# and for a dctionnary
aws: 
  - "access_key_id='abcdefgh'"
  - "secret_access_key='{{ vaulted_aws_secret_access_key }}'"



# vault_file
vaulted_db_passord: a_super_secret
vaulted_aws_secret_access_key: the_aws_secret

That way, you can manipulate all your vars like before, knowing the vaulted version stays encrypted. You can even solve the problem of having someone responsible for the encrypted file and the rest of the team never seeing its content but still being able to manage var files as they need.

[Michael DeHaan]

This article had some good things in it, though I disagree with some points -- including the part where it suggests only using scalar variables.

What this really shows isn't specific to vault -- it's just that you can use variables in other variables.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/2a8a43b6-b358-46a4-9d38-7874efc2fa8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Petros Moisiadis]

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/2a8a43b6-b358-46a4-9d38-7874efc2fa8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Well, it will work, but this is still a workaround. You still have to maintain two files and edit them both for a single addition. Also, it becomes more complicated if you have repeated variable names and values. For example, how much complexity you have to introduce in your separate vault file in order to handle a simple variable file like the following ?

system_users:
  - username: john
    password: secret_password_1
  - username: alice
    password: secret_password_2

db_users:
  - username: john
    password: another_secret_password_1
  - username: alice
    password: another_secret_password_2


Unfortunately, the Ansible team has not (yet) given an answer on whether a command-line option to enable a simple syntax for leaf-node encryption mode would be considered for ansible-vault (keeping the current whole-file encryption mode as the default mode). There was a feature request for this mode and discussion by many people _before_ vault's release and it seems it is still desired by people _after_ vault's release.

[Michael DeHaan]

 I'd definitely resent any inference of evil-doing from "The Ansible Team" here :)

While I'm open to pull requests to enhancements within reason, we've had the discussion a few times already that this could be served better by a "password2" style plugin that used the VaultLib classes.

--Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/532C4F9E.6030303%40yahoo.gr.

[Petros Moisiadis]

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEVJ8QONjy_P7ZeMAJWiZAa-RFR7HSBDRJoVRwmX4t5H%3DKZ86g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

I do not understand how this "password2" style plugin would serve leaf-node encryption better. What would be the user interface? Wouldn't it use a binary to spawn the editor after decrypting the file and encrypt the file after saving it, like ansible-vault conveniently does? Could you please give an example of a possible UI to support a workflow like this: create a new variable file, save it doing leaf-node only encryption, push it to GIT, check it out later, edit it, commit the changed file back and have a meaningful diff between the two versions?

[ralo...@gmail.com]

Well, it will work, but this is still a workaround. You still have to maintain two files and edit them both for a single addition. Also, it becomes more complicated if you have repeated variable names and values.

It is a trick, no doubt

But it's simple, pretty straightforward and self documented.

We use it everyday and don't feel any overhead because of it.

Passwords are not edited so often, and in the end not that many.

 

For example, how much complexity you have to introduce in your separate vault file in order to handle a simple variable file like the following ?

Of course, you end up duplicating your password entries.

But with a consistent convention it's painless.

 

Unfortunately, the Ansible team has not (yet) given an answer on whether a command-line option to enable a simple syntax for leaf-node encryption mode would be considered for ansible-vault (keeping the current whole-file encryption mode as the default mode). There was a feature request for this mode and discussion by many people _before_ vault's release and it seems it is still desired by people _after_ vault's release.

I read the thread at the time, and agreed that leaf encryption was a better way,

but since we use this, I really don't feel it's that necessary, on a day to day usage.

raphael. 

[Petros Moisiadis]

On 03/21/14 17:57, ralo...@gmail.com wrote:

Well, it will work, but this is still a workaround. You still have to maintain two files and edit them both for a single addition. Also, it becomes more complicated if you have repeated variable names and values.

It is a trick, no doubt

But it's simple, pretty straightforward and self documented.

We use it everyday and don't feel any overhead because of it.

Maybe the fact that you try flattening/namespacing your vars as much as possible, as you are suggesting in your article, helps you in keeping the complexity of your vault files in sane levels. This is not going to be the case with people that prefer to have a more hierarchical representation of their data (dicts nesting other dicts and lists, with possible repeating identifiers).

Passwords are not edited so often, and in the end not that many.

 

For example, how much complexity you have to introduce in your separate vault file in order to handle a simple variable file like the following ?

Of course, you end up duplicating your password entries.

But with a consistent convention it's painless.

Besides duplication of data entry and the increased difficulty coming with writing vault-files for data with varying hierarchal representation, there is also another important feature missing from ansible-vault that your workaround cannot handle too: Auditing capabilities. You still cannot tell whether a specific secret has changed and by whom. A structure that allows the provability of the source of a security breach is quite important, especially in environments that need to conform to security standards.

[Michael DeHaan]

quoted:  " Auditing capabilities. You still cannot tell whether a specific secret has changed and by whom. A structure that allows the provability of the source of a security breach is quite important, especially in environments that need to conform to security standards."

Eh, I'm not sold.  While it's possible to make arguments in both directions, it is just as easy to state that letting someone know *WHAT* variables are entered into the file makes it more easy to say "this is the file I want to crack".  The other thing is that the encrypted values for each "leaf" are going to be crazy long and you are also encrypting *less* data in each.    If you do need the data about what changed, you *STILL* have the history of changes on the file, who made them, and can decrypt each step if you really wanted.   The data isn't lost.

As an aside, I will correct myself -- the password lookup plugin generates passwords, so it's not really an answer to this problem to make a password plugin using VaultLib.   

However, it should also be noted that the approach of "leaf node encryption" doesn't work for other reasons -- people will quickly want a file *half* encrypted, see OP's post, which just makes everything get really complicated real quick.

I'm open to an ansible.cfg option pull request that might enable a mode like this, but it needs to be off by default.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/532C8A3D.6030702%40yahoo.gr.

[Petros Moisiadis]

On 03/21/2014 10:05 PM, Michael DeHaan wrote:

quoted:  " Auditing capabilities. You still cannot tell whether a specific secret has changed and by whom. A structure that allows the provability of the source of a security breach is quite important, especially in environments that need to conform to security standards."

Eh, I'm not sold.  While it's possible to make arguments in both directions, it is just as easy to state that letting someone know *WHAT* variables are entered into the file makes it more easy to say "this is the file I want to crack".  The other thing is that the encrypted values for each "leaf" are going to be crazy long and you are also encrypting *less* data in each.    If you do need the data about what changed, you *STILL* have the history of changes on the file, who made them, and can decrypt each step if you really wanted.   The data isn't lost.

You would say "this is the file I want to crack" for a fully encrypted file too (it must have something important to be encrypted in the first place).

Also, with AES and PKCS5 padding the cipher-text length can be calculated as: (original_length / 16 + 1) * 16. So a secret 16 bytes long would be encrypted to 32 bytes of cipher-text. Probably you will also store the IV (16 bytes), so you end up with 48 bytes. You will also base64-encode this to have finally a ~65 bytes long string. So, ok, you will have about 4 times longer strings. Why is this a problem?

Moreover, with leaf-node encryption you are not encrypting *less* data. You are only encrypting the part of data that *needs to be secret*.

As for the auditing part, things are not so simple. If you commit files encrypted as a whole, auditing data may be possible (just possible, not easy) only if passwords to decrypt them are or can be available. But passwords can be forgotten if they are stored in human minds, or can be deleted (deliberately or by accident) if stored in password stores. Normally, you would only have to recreate your secrets in these cases and replace the old ones. With whole file encryption this is not enough. Besides losing your passwords, you also lose your whole data structures, that you will have to rewrite. Also, who is going to be the auditor? If he is an outsider (as security requirements may demand), is he trusted? Can you pass him the passwords to your clear text secrets? Even if all these are not a problem for you, to make auditing possible with whole-file encryption commits, you would still have to create your own tool that would involve checking out revisions, decrypting them, running out-of-rcs-history diffs between them and producing an output in a meaningful format with all needed information (commiter, timestamps, etc.). Not easy. With leaf-node encryption commits, you wouldn't need that. You would just use the history tools provided by your rcs. Your data structures and their history would not depend on passwords and you would be able to take full advantage of the distributed nature of your rcs in case of a central infrastructure attack.

As an aside, I will correct myself -- the password lookup plugin generates passwords, so it's not really an answer to this problem to make a password plugin using VaultLib.   

However, it should also be noted that the approach of "leaf node encryption" doesn't work for other reasons -- people will quickly want a file *half* encrypted, see OP's post, which just makes everything get really complicated real quick.

I cannot see what that *half* encrypted file problem is in the OP's post. As I said, with leaf node encryption you are only encrypting your secrets in leaf nodes, not your whole data structures.

I'm open to an ansible.cfg option pull request that might enable a mode like this, but it needs to be off by default.

This is really nice. :)

[Michael DeHaan]

Arguments can be made for every single choice made in Ansible to be done exactly the opposite way.   It's kind of the nature of open source and everybody on the internet thinking differently.  

Submit a pull request for a non-default option and something can possibly be done here, however we're not going to be removing the default mode it has now.