Secret variables / data: encryption

[Jan-Piet Mens]

I believe this topic has been requested before, and we talked about it
at some length this afternoon in Antwerp.

People wanting to store "confidential" variables with their playbooks,
say, in a repo, cannot really do that for fear of exposing the secret
variables.

Would there be some interest in having support in Ansible for encrypted
variables? Before investing a lot of time and then having the idea
dropped, I'd like a bit of feedback on this.

I see two possible ways we could go about implementing this. Take
host_vars/ for example, we could add what I'd like to call encrypted
"pouches" of YAML which would be decrypted by Ansible when it attempts
to access the pouch.

host_vars/www normal YAML variables
host_vars/mail.pouch encrypted YAML / base64-encoded

Alternatively, a method proposed by Toshaan which I also rather like is
mix encrypted variables into a regular vars file:

---
name: John Doe
password.crypt: aGVsbG8gd29ybGQgb2YgYW5zaWJsZQo=

Ansible would then decrypt `password.crypt', returning the clear-text in
a variable called `password'.

As to encryption itself, there are a number of possibilities. I'm
tending towards using symmetric encryption instead of public key encryption
because of the possibility of easily sharing the key. With public-key we
would need to decide what system to use, and this would entail
additional Python module requirements for the Ansible manager. If we can
use pycrypto (which is already a dependency for Paramiko) that would be
a bonus.

Thoughts?

-JP

[Michael DeHaan]

This is very interesting... I like the 'pouch' idea, though I am not sure I like the name 'pouch' for it :)

The trick is you also want to decrypt it when editing the file, so it needs something for that too, that spawns $EDITOR around editing it, and Ansible needs to recognize if any of these are used to ask for unlock passwords, which sounds like it *MAY* need to be another flag.

Are people going to want to use different passwords for different files?

The idea of handling it with a lookup plugin sounds a little promising as it's not far off from the way the password plugin looks now for generating new passwords.  (Is everybody aware of that?)

Mostly I'm interested in how the user would interact with editing the file, and in what situations and when they are asked for a password.

        -JP

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Jan-Piet Mens]

> This is very interesting... I like the 'pouch' idea, though I am not sure I
> like the name 'pouch' for it :)

Call it container or whatever -- I think a name that clearly identifies
it as something special is good. Considering how much you like cows,
maby 'udder' is what we need ... ? ;-)

> Are people going to want to use different passwords for different files?

I'm not thinking passwords (as in "type that in"), but rather key files
(stored in a well-known path or specified as cfg variable/environment
variable).

We need something that can work unattended (think cron) and something
which can be shared accross a team of people working on a distributed
copy of the playbook repo.

> The idea of handling it with a lookup plugin sounds a little promising as
> it's not far off from the way the password plugin looks now for generating
> new passwords. (Is everybody aware of that?)

I wasn't, no.

> Mostly I'm interested in how the user would interact with editing the file,
> and in what situations and when they are asked for a password.

I think a small utility which decrypts with said key file, invokes
$EDITOR, and encrypts would suffice. On the other hand, as alluded to,
people could write their own stuff using `openssl' cli.

-JP

[Vincent Van der Kussen]

On Sun, Jun 30, 2013 at 9:55 AM, Jan-Piet Mens <jpm...@gmail.com> wrote:
>> This is very interesting... I like the 'pouch' idea, though I am not sure I
>> like the name 'pouch' for it :)
>
> Call it container or whatever -- I think a name that clearly identifies
> it as something special is good. Considering how much you like cows,
> maby 'udder' is what we need ... ? ;-)

Just a suggestion here : vault

>
>> Are people going to want to use different passwords for different files?
>
> I'm not thinking passwords (as in "type that in"), but rather key files
> (stored in a well-known path or specified as cfg variable/environment
> variable).
>
> We need something that can work unattended (think cron) and something
> which can be shared accross a team of people working on a distributed
> copy of the playbook repo.
>
>> The idea of handling it with a lookup plugin sounds a little promising as
>> it's not far off from the way the password plugin looks now for generating
>> new passwords. (Is everybody aware of that?)
>
> I wasn't, no.
>
>> Mostly I'm interested in how the user would interact with editing the file,
>> and in what situations and when they are asked for a password.
>
> I think a small utility which decrypts with said key file, invokes
> $EDITOR, and encrypts would suffice. On the other hand, as alluded to,
> people could write their own stuff using `openssl' cli.

As discussed yesterday, i think what Chef does comes close to what we
want to achieve

http://docs.opscode.com/essentials_data_bags.html

Vincent

[Marco Vito Moscaritolo]

We actually use git+git-crypt to encript / decrypt files (SSL Cert,
GPG Keys, ..) and use hashed variable for users password.

Pros:
- Work from now
- Easy to implement
- Choise encrypt system that we prefer

Cons:
- Only GIT compatible (don't work with SVN/BZR/...)
- File is TOTALLY crypted (no just a single variable)

JM2C

> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
Ciao
Marco
--
web: http://mavimo.org
msn: mav...@tiscali.it
gtalk: mav...@gmail.com
mob: +39 393 9249923
skype: marco.moscaritolo
twitter: http://twitter.com/mavimo

[Dave Cottlehuber]

On 30 June 2013 12:43, Marco Vito Moscaritolo <mav...@gmail.com> wrote:
> We actually use git+git-crypt to encript / decrypt files (SSL Cert,
> GPG Keys, ..) and use hashed variable for users password.
>
> Pros:
> - Work from now
> - Easy to implement
> - Choise encrypt system that we prefer
>
> Cons:
> - Only GIT compatible (don't work with SVN/BZR/...)
> - File is TOTALLY crypted (no just a single variable)
>
> JM2C

Sounds interesting! Can you share how to do this? A blog post would be PURRfect.

A+
Dave

[Michael DeHaan]

" knife data bag from file BAG_NAME ITEM_NAME.json"

This is perhaps the worst CLI design I have ever seen :)

Perhaps someone can try to boil it down to something more simpler in how we think this would look at CLI level.

[Jan-Piet Mens]

> " knife data bag from file BAG_NAME ITEM_NAME.json"
>
> This is perhaps the worst CLI design I have ever seen :)

It *is* the worst I've ever seen. No worries -- we'll make that much
better (which isn't hard to do)

-JP

[GW]

Hi,

Why not use the 'password' lookup plugin (with_password or {{ lookup('password',...) }}) or 'file' lookup plugin? This way all confidential variables can be stored in a directory 'credentials' and either you do not commit them at all (the ones who have it could use it) or use .zip/.rar with password or encrypted loopback image and commit the encrypted archive?

Greetings,
  gw 

[Michael DeHaan]

The password lookup plugin generates passwords when they are not already set, it is not a means of keeping confidental data that you can hand enter.

The "vault" idea is a neat idea.

--

[Petros Moisiadis]

Another approach would be to have a 'secret_vars' playbook section as a list with the names of variables whose values are expected to be in a format like this:
shared_key_configuration_name!base64.b64encode(encrypted_value)

For example, in playbooks:

secret_vars:
    - mysql_root_password
    - some_api_key

and in the inventory or in variable files :

   mysql_root_password: mykey!cXd5OGduMzQ1MjM0I2VyQCMmXiozNGdRVyM=
   some_api_key: anotherkey!M2N2OTIzNDVqdm5zZGZnOTVmcmU5NTR2ei8jIw==

Then, multiple shared key configurations would normally be placed in the ansible configuration, like this:

[shared_key_mykey]
cipher=AES
keyfile=/path/to/mykeyfile

[shared_key_anotherkey]
cipher=DES
keyfile=/path/to/anotherkeyfile

That way it would also be super easy to hide variables that are in secret_vars list from ansible's logs which is also a very important security enhancement measure.

What do you think?

[Michael DeHaan]

I kind this much harder to explain than JP Men's suggestion.

We will be going with that :)

[Michael DeHaan]

Ignore my inability to plac'e apostrophe's correctly btw :)

[Petros Moisiadis]

On second thought, to make things even simpler, 'secret_vars'  is not even needed.
A 'decrypt'  lookup plugin would be just enough to get the decrypted value of a variable in a format like the one I suggested.
That way, the format can be made even simpler by moving away 'shared_key_configuration_name' from it and giving it instead as a parameter to the 'decrypt' lookup plugin call.

Pretty simple I think, without extra playbook syntax or additional, specially-handled files. And perhaps it has the easiest implementation since lookup logic is already available.

[Michael DeHaan]

I generally am not a fan of lookup plugin syntax, especially when it has to return a data structure.  It's fine for some things, but the main intention of lookup plugins were to enable things like "with_items" and then they got used for things like file lookups later, and the syntax is ok... but it's something I find myself using rarely.

I really like the vault idea because it would work transparently with vars_files and other things, and *not* require extra syntax.   

I am tentatively thinking

ansible-vault --unlock <path>

ansible-vault --lock <path>

And then when ansible runs if a vault is used, it would prompt for the unlock password for that vault.

The question is do you need multiple vault passwords at the same time or not.

The other slight thing is you want to make sure none of your files get checked into git while locked down though, which might be a little tricky... I would hate to have to have people to know how to set up git triggers for this, but that might be the only way.

[Petros Moisiadis]

On 07/01/2013 02:28 AM, Michael DeHaan wrote:

I generally am not a fan of lookup plugin syntax, especially when it has to return a data structure.  It's fine for some things, but the main intention of lookup plugins were to enable things like "with_items" and then they got used for things like file lookups later, and the syntax is ok... but it's something I find myself using rarely.

I really like the vault idea because it would work transparently with vars_files and other things, and *not* require extra syntax.   

I am tentatively thinking

ansible-vault --unlock <path>

ansible-vault --lock <path>

And then when ansible runs if a vault is used, it would prompt for the unlock password for that vault.

The question is do you need multiple vault passwords at the same time or not.

In my opinion, a requirement to always prompt for a password does not fit with the purpose of automating things.
Instead, there can be multiple vault configuration sections in the ansible configuration and ansible-vault could be called without a path (getting the paths from the configuration). For example:

[vault_main]
path=/path/to/vaults/main

[vault_custom]
path=/path/to/vaults/custom
password_file=/path/to/vault_passwords/custom
cipher=Twofish

If 'password_file' is set, it will be used to read the password for unlocking the vault. Only if it is not set would ansible utilities prompt to enter a password.
Running an ansible binary would involve invoking ansible-vault to unlock the vaults in temporary paths (perhaps somewhere inside /tmp or ~/.ansible) with local user readonly access. The script would read vars from there, flag them internally as 'sensitive' so as not to leak them in logs, and, in the end, it would invoke again ansible-vault to remove these temporary paths.
Password files could be shared among machines/users that run ansible. For even better security, they could be stored encrypted with the ssh public key of the local user and be decrypted before being used. ssh-agent could be used to avoid repeated password input prompts in the same fashion that it can be used to ssh to remote hosts.
The 'cipher' setting could be used to support different encryption algorithms. If missing, a default one would be used.
Editing vaults could be done with an 'ansible-vault --edit {vault_name | vault_path}' command that would unlock to a temporary path, spawn an editor and, on exit, encrypt the temporary path and move it to the permanent (locked) vault path.

The other slight thing is you want to make sure none of your files get checked into git while locked down though, which might be a little tricky... I would hate to have to have people to know how to set up git triggers for this, but that might be the only way.

What do you mean by 'make sure none of your files get checked into git while locked down'? Isn't it the desirable behaviour to always check the files into GIT in a locked state?

[Jan-Piet Mens]

Thanks for all your comments and ideas. I have submitted a pull-request
[1] which implements this as a filter. I've tried to make it so that
people can use as many keys as they desire. The page with the PR [1]
contains a blurb about how to get started and how to use it.

A utility called ansible-vault is included.

Regards,

-JP


[1] https://github.com/ansible/ansible/pull/3393

[Jan-Piet Mens]

> In my opinion, a requirement to always prompt for a password does not
> fit with the purpose of automating things.

Full ACK

-JP

[Jan-Piet Mens]

> If 'password_file' is set, it will be used to read the password for
> unlocking the vault. Only if it is not set would ansible utilities
> prompt to enter a password.

I'm not understanding the whole lock/unlock thing. IMO (and that is how
I've done it) encrypted values are contained in vars files and are
decrypted on use.

-JP

[Petros Moisiadis]

Correct. Encrypting / Decrypting are better words than Locking /
Unlocking to describe the usage I suggested previously (decrypting vault
files somewhere temporarily while vars are being used), but I used the
latter coming from a discussion with references to that terminology.
Also, although I suggested using temporary files for the decrypted
vaults earlier, on second thought, I think that they should be better
kept temporarily in memory. That way you can be sure that no decrypted
values are left over on unpredicted conditions (e.g. on crashes).

[Petros Moisiadis]

Very nice. Thanks!
Are (or can they be) jinja filters also available inside playbooks?

[Jan-Piet Mens]

> Are (or can they be) jinja filters also available inside playbooks?

To a certain extent, I believe. For example, this works:

- action: shell echo '{{ mypasswd | vault }}' > /tmp/00

Regards,

-JP

[Michael DeHaan]

"encrypted values are contained in vars files and are
decrypted on use."

Where do they get the inputs to decrypt them from?

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Michael DeHaan]

First off, thanks for the initiative.

This is too early for a core pull request IMHO as there's still a pretty good discussion on the list on how this might work.

I'm also finding the interface somewhat confusing because it's still using lookup plugins (which I asked that it not), and it requires using the keyczar CLI.

If we can dumb things down, I'd like to have this as a core feature, but right now, there's a lot of reading/understanding required for it's use, and it seems to require a bit too much thinking.

Not trying to be negative here, I just think it needs to be really smooth to fit in with the rest of Ansible.

--Michael

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jan-Piet Mens]

> I'm also finding the interface somewhat confusing because it's still using
> lookup plugins (which I asked that it not),

I don't understand. It's a Jinja2 filter. Where's the lookup plugin?
(Maybe I'm confusing the terminology.)

> and it requires using the
> keyczar CLI.

For creating a key (or more than one key) initially, yes. I see
absolutely no point in re-inventing the wheel, particularly as crypto
stuff is concerned. People much more clever than I have done that.

After a key is initially created, that's it; no more keyczar CLI. That's
the reason I submitted the ansible-vault utility. (Of course, I could
add key creation to that, which would avoid using the keyczar CLI
entirely.)

> If we can dumb things down, I'd like to have this as a core feature, but
> right now, there's a lot of reading/understanding required for it's use,
> and it seems to require a bit too much thinking.

Pardon me, but I disagree.

1. Create a key.
2. Use it.

I've been verbose in my doc (on the PR page) so as to be as clear as
possible.

-JP

[Petros Moisiadis]

Great.
Now what is needed is a way to make sure that decrypted values are not
being leaked in ansible logs, though I am not sure what is the correct
place to do the checks.
Maybe the easiest thing to do would be to add a global "log" attribute
that could be also overridden by tasks. Then it could be set to "log:
no" so that no logs are produced (quite intrusive), set to "log: debug"
to log everything or set to other levels such as "critical", "warning",
"notice", etc with each module having to decide what to log based on the
active log level.

[Michael DeHaan]

I guess I'm saying "too many steps" here.

I just want it to be as smooth as Possible.

I think it could be done so you didn't even need the Jinja2 filter, you could just use all the vars files unencrypted during Ansible playbook runtime, but they would be locked when you try to edit them, and could have a full vars_files or two full of content that you could edit plaintext when needed.

Sort of like how ssh-agent doesn't require re-entering your SSH key password, right?

        -JP

[Petros Moisiadis]

On 07/01/13 17:56, Michael DeHaan wrote:
> I guess I'm saying "too many steps" here.
>
> I just want it to be as smooth as Possible.
>
> I think it could be done so you didn't even need the Jinja2 filter,
> you could just use all the vars files unencrypted during Ansible
> playbook runtime, but they would be locked when you try to edit them,
> and could have a full vars_files or two full of content that you could
> edit plaintext when needed.
>

The Jinja2 filter is nice. It allows to store mixed encrypted and
unencrypted values and it can be used both in playbooks and templates.
You don't have to encrypt/lock the whole variable file just because you
have some password vars inside it (the usual case). In addition, it
allows to have different encryption keys for different vars without
having to create separate vault files for each key.

keyczar is a good choice too. It has a nice abstraction over handling
key stores and keys, it is tested, and keyczart's code could be
incorporated in the ansible-vault utility in order to have one utility
for all vault tasks, which could be also taught to edit encrypted values
directly inside variable files (e.g. "ansible-vault --key ckey --set
mysecretvar='plaintext' varsfile").

[Jan-Piet Mens]

> I just want it to be as smooth as Possible.

It is. :) [Admittedly, with the exception that ansible-vault utility
needs more love.]

> I think it could be done so you didn't even need the Jinja2 filter, you
> could just use all the vars files unencrypted during Ansible playbook
> runtime, but they would be locked when you try to edit them, and could have
> a full vars_files or two full of content that you could edit plaintext when
> needed.

I'm (very slowly) starting to understand what you want: everything is
encrypted and gets magically decrypted on Ansible run. This is way
overkill, and poses a number of problems respectively has a number of
inconveniences:

1. How to detect whether a particular vars file is encrypted (by file
name extension, by content)?
2. Completely disables things like printing/sending a vars file
3. Makes version control near to meaningless -- having all the
crypto-stuff in the files.
4. Requires some 'magic' (like the SSH agent you mention below) which is
able to feed Ansible with keys. Even if we pass a single key to
Ansible, the next point:
5. Makes using more than a key terribly difficult. We discussed using
some form of PKI in Antwerp, but more or less chucked that due to
'what do we encrypt to?' The current user? The current host? The
target host? How do teams share keys, etc. Most complicated, I think.

The original intention was to encrypt really secret stuff, like
passwords for MySQL (my.cnf) and things like that. Most variables do not
need to be encrypted, and I believe the current PR solves that rather
neatly: encrypt only what absolutely needs to be encrypted.

> Sort of like how ssh-agent doesn't require re-entering your SSH key
> password, right?

I'm hearing you, but I don't think this is the way to go.

-JP

[Jan-Piet Mens]

> In addition, it
> allows to have different encryption keys for different vars without
> having to create separate vault files for each key.

Bingo!

> keyczar is a good choice too. It has a nice abstraction over handling
> key stores and keys, it is tested, and keyczart's code could be
> incorporated in the ansible-vault utility in order to have one utility
> for all vault tasks, which could be also taught to edit encrypted values
> directly inside variable files (e.g. "ansible-vault --key ckey --set
> mysecretvar='plaintext' varsfile").

Indeed. I would like to augment ansible-vault like so:

ansible-vault -N -k mynewkey # create new key w/ all requirements
echo secret | ansible-vault -k mynewkey - # encrypt

Hide the keyczar CLI utility and making of all directories, etc. Add a
`-l' option to list keys, maybe `-r' to remove, and that's about it.

I like your idea of an --edit function. #feature

-JP

[GW]

On Mon, 1 Jul 2013 16:19:45 +0200, Jan-Piet Mens wrote:

I'm also finding the interface somewhat confusing because it's still using
lookup plugins (which I asked that it not),

I don't understand. It's a Jinja2 filter. Where's the lookup plugin?
(Maybe I'm confusing the terminology.)

But it could probably be simply rewritten to be a lookup plugin, right?

To me the most obvious approach that doesn't add new concepts is to use it as a lookup plugin and storing passwords in individual files, exactly how the password lookup plugin does it. So you would put in your inventory things like:

mysql_password: "{{ lookup('decrypt',  'credentials/path/to') }}"

Although this is pretty similar to what can be done with this Jinja2 filter (if I'm not mistaken):
mysql_password: "{{ lookup('file',  'credentials/path/to')|vault }}"

After thinking about this.. Actually all lookup plugins could have 1:1 mapping with Jinja2 filters, like this:

{{ mysql_config_file|file }}

{{ "mysql/mysqld.conf"|file }}

{{ user_password_path|password }}
{{ "credentials/user_password"|password }}
{{ mysql_password_file|file|vault }}



{{ lookup('XXX', YYY) }} --> {{ YYY|XXX }}

This would look much nicer and more intuitive to use, or not? Thoughts?

Greetings,
  gw 

[Michael DeHaan]

Somewhat OT, as I have not had a time to read the details on the vault stuff yet, but I would *LOVE* all lookup plugins to work as Jinja2 filters, because I dislike the whole {{ lookup(...) }} thing for being too dense.

Would love a better way.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dag Wieers]

On Mon, 1 Jul 2013, Michael DeHaan wrote:

> Somewhat OT, as I have not had a time to read the details on the vault
> stuff yet, but I would *LOVE* all lookup plugins to work as Jinja2 filters,
> because I dislike the whole {{ lookup(...) }} thing for being too dense.
>
> Would love a better way.

Problem we discovered for the 'mandatory' filter is that filters only get
the values passed, which basically means errors from filters are vague
since they have no context (no filename, no line number, no variable
name, only filter name and value).

This means that if you have a template/playbook with more than one filter,
the error will not tell you which occurence of the filter failed.

I opened a feature request for jinja2, but that's only going to be useful
in the distant future...

https://github.com/mitsuhiko/jinja2/issues/245

--
-- dag wieers, d...@wieers.com, http://dag.wieers.com/
-- dagit linux solutions, con...@dagit.net, http://dagit.net/

[Any errors in spelling, tact or fact are transmission errors]

[Jan-Piet Mens]

> [1] https://github.com/ansible/ansible/pull/3393

We've had quite a bit of positive feedback on the idea, as well as some
loud doubts, and I want to recap how I think this should be done.

Intention is to protect certain variables in vars files.

---
nameserver: 10.0.0.1
admin: Jane Jolie
dbpassword: "AHchx1Va5fPZdEAY6lg+GEmejsl4tGfCNKxMFP7tM7Y7kl"
webservertype: nginx

Most vars will not be encrypted. This is meant as a protection to vars
which contain sensitive data -- basically clear-text passwords that
nobody wants to have floating around in files that get added to
repositories, etc.

I don't think encrypting vars files completely is useful: makes things
like `grep', and `diff' impossible to use.

It must be kept as simple as possible, and as secure as possible. For
this reason, using AES secret keys has charm. Keyczar implements proven
crypto stuff and is easy to add.

Secret keys mean: the person who has the key can encrypt and decrypt
data. As such, keys will be kept *outside* of the playbook repos (e.g.
/etc/keystore), but they can be distributed, e.g. to team members.
Having keys means Ansible can decrypt data on the fly, without anybody
having to enter passwords. Entering passwords is one thing we don't
want, as it breaks automation.

On the other hand, allowing more than a single key is useful in
multi-tenant situations, but more than a single key is not required.

The interface (ansible-vault) needs to be slimmed down a bit more, which
is not hard to do. I envision two basic steps:

1. Create a new key with all files, dirs, etc. automatically: ansible-vault --createkey keyname
2. Use it to encrypt data to store in a vars file: ansible-vault --key keyname -

The idea (Petros' I think) to add an --edit option to ansible-vault is
not hard, but that means loading the YAML and rewriting it; this would
a) re-order the YAML output and b) loose comments. I'm not sure either
would be satisfactory, but this can be implemented if desired.

Regards,

-JP

[Vincent Van der Kussen]

That sums it up pretty nicely.

+1000 !

VIncent

[Serge van Ginderachter]

Most vars will not be encrypted. This is meant as a protection to vars
which contain sensitive data -- basically clear-text passwords that
nobody wants to have floating around in files that get added to
repositories, etc.

I don't think encrypting vars files completely is useful: makes things
like `grep', and `diff' impossible to use.

+1 

On the other hand, allowing more than a single key is useful in
multi-tenant situations, but more than a single key is not required.

Does this mean mulltiple keys which each can decrypt all passwords? So different users get different keys, and a key can be revoked without impacting other users?

If yes, +1

Thanks,

Serge

[Jan-Piet Mens]

> Does this mean mulltiple keys which each can decrypt all passwords? So
> different users get different keys, and a key can be revoked without
> impacting other users?

Not to start with, no; you have to choose a key to encrypt a particular
value with. If you revoke (i.e. delete) the key, the value can no longer
be decrypted.

The way I envision this is having a key for cust1's vars, and another
for cust2's vars, and I use vault on a per-customer basis.

Keep it simple. :)

-JP

[Petros Moisiadis]

On 07/02/13 12:44, Jan-Piet Mens wrote:

The idea (Petros' I think) to add an --edit option to ansible-vault is not hard, but that means loading the YAML and rewriting it; this would a) re-order the YAML output and b) loose comments. I'm not sure either would be satisfactory, but this can be implemented if desired. Regards, -JP

This could be worked around like this:
$ ansible-vault --edit vars_file

And then:
1) ansible-vault: Create a copy of vars_file to edit to a temporary file
2) ansible-vault: Spawn editor on temporary file
3) human: Edit as you wish. If you want a value to be encrypted, use some 'template magic' like this: "myval: {{ plaintext | vault_encrypt('keyname') }}"
4) human: Save and exit from editor
5) ansible-vault: Parse temporary file line by line, process occurrences of "{{ plaintext | vault_encrypt('keyname') }}" and replace them with the processed output.
6) ansible-vault: Move temporary file in place of the original file.

What do you think?

[Jan-Piet Mens]

> 3) human: Edit as you wish. If you want a value to be encrypted, use
> some 'template magic' like this: "myval: {{ plaintext |
> vault_encrypt('keyname') }}"

Basically yes, but:

For a new variable, human enters, say,

password: vault(verysecret)

that's easy to parse upon saving: grab what's inside vault(), encrypt,
and store as

password: "Axkkwkekke..="

And on the second round, i.e. when human edits that file a second time?
How do we find the encrypted values? ;-) We can't just go guessing that
anything that looks like base64 is a "vault" variable....

I'm thinking along the lines of having the encrypted values stored in
the vars file as

password: "vault:Axkkwkekke..="

in other words, prefixed with "vault:", which would a) give human who
looks at file an indication that it's not just base64-encoded but
belongs to vault and shouldn't be touched, and b) would allow for
parsing when editing, as you suggest. (Obviously the Jinja2 vault()
filter would strip that off before trying to decrypt it.)

Would that be viable?

-JP

[Brian Coca]

I know this is a BIG ask, but for environments with multiple Admins an implementation of Shamir's secret would rock.

[Petros Moisiadis]

I like the "vault:" prefix idea.

Maybe it can also be used when editing files through ansible-vault, so that, before spawning the editor, all 'vault:Axkkwkekke..='  values are translated to a ready-to-edit 'vault(my_plaintext_secret)' expression. For variables encrypted with a key other than the default, it could try decrypting the value with each available key until it succeeds so as to produce a correct 'vault(key=matched_key_keyname, my_plaintext_secret)' expression.

Another option to avoid that loop (but adding some more text in the ecrypted-value syntax) would be to use a vault+keyname combination prefix. For example,

password: "vault#keyname:Axkkwkekke..="

[Jan-Piet Mens]

> I know this is a BIG ask, but for environments with multiple Admins an
> implementation of Shamir's secret would rock.

How would that work? Say you have 3 admins, all must be present (and
feed a third of a key) when running a playbook? Hmm. And how do they
cron' it?

-JP

[Jeroen Hoekx]

On 2 July 2013 15:44, Brian Coca <bria...@gmail.com> wrote:
> I know this is a BIG ask, but for environments with multiple Admins an
> implementation of Shamir's secret would rock.

Interesting. How would you see that in practice?

The scenario I can imagine right now is that you need two keys to
decrypt the secret. One could be stored on the management server and
the other should be provided by the sysadmin? That way password
sharing between people is not needed.

(My masters thesis was about this and I've been looking for a
practical use of it)

Jeroen

[Brian Coca]

Shamir's secret works as n/m, where n = min number of keys to decrypt and m = total number of keys.

No cron in this case. You don't need all admin, you can define n of them, it can be 1. This way you can have a key per admin and have a 'quorum' defined to how many of them need access to decrypt. 

I've used this in the past to store highly sensitive material (hardware root keys, enterprise key that decrypts everyone's email, etc).

        -JP

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Brian Coca
Stultorum infinitus est numerus
0110000101110010011001010110111000100111011101000010000001111001011011110111010100100000011100110110110101100001011100100111010000100001

Pedo mellon a minno

[Lorin Hochstein]

I understand the motivation behind putting the secrets into the playbook and then converting them with an encrypting preprocessor, since it increases usability. But I'd be nervous about somebody accidentally committing the file to a version control system before running the preprocessor.

Lorin

--

Lorin Hochstein

Lead Architect - Cloud Services

Nimbis Services, Inc.

www.nimbisservices.com

[Jan-Piet Mens]

> I understand the motivation behind putting the secrets into the playbook
> and then converting them with an encrypting preprocessor, since it
> increases usability. But I'd be nervous about somebody accidentally
> committing the file to a version control system before running the
> preprocessor.

We're talking of an 'editing' front-end, so this cannot (famous last
words) really happen. A human uses `ansible-vault --edit' and is dropped
into an editor. If said human adds secrets without wrapping them in
a 'magic' token, then yes, it will happen. Otherwise, secrets are
encrypted on the way out of the editor.

Anyway -- this is premature until the basics are settled (see my Recap
message :-)

-JP

[Brian Coca]

I don't really know how to apply this to an interactive app, I've mostly used for offline storage and in an emergency get the quorum needed to give access to the 'highly sensitive secure material'

[Marco Vito Moscaritolo]

This is a snippet of what I use:

https://coderwall.com/p/kucyaw

Let me know if ther's some info that are missing or if someone have
better ideas :)

Marco

On Sun, Jun 30, 2013 at 3:29 PM, Dave Cottlehuber <d...@jsonified.com> wrote:
> On 30 June 2013 12:43, Marco Vito Moscaritolo <mav...@gmail.com> wrote:
>> We actually use git+git-crypt to encript / decrypt files (SSL Cert,
>> GPG Keys, ..) and use hashed variable for users password.
>>
>> Pros:
>> - Work from now
>> - Easy to implement
>> - Choise encrypt system that we prefer
>>
>> Cons:
>> - Only GIT compatible (don't work with SVN/BZR/...)
>> - File is TOTALLY crypted (no just a single variable)
>>
>> JM2C
>
> Sounds interesting! Can you share how to do this? A blog post would be PURRfect.
>
> A+
> Dave

>
> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--

Ciao
Marco
--
web: http://mavimo.org
msn: mav...@tiscali.it
gtalk: mav...@gmail.com
mob: +39 393 9249923
skype: marco.moscaritolo
twitter: http://twitter.com/mavimo

[Dylan Martin]

Sorry for coming in late, and sorry again if I'm missing the point or just generally being stupid and oafish.

Unattended decryption is tricky because a lot of the time you wind up with something that's no better than having your secrets saved in clear-text.

I can think of three ways this gets handled:

1) You have a password or key that gets loaded and used.  The problem here is there are few scenarios where a baddie has access to the encrypted data and not the key.  You can keep the key outside your git repo (or otherwise separated) but that isn't any more secure than keeping the plain text of whatever you want to encrypt separate.  EG you encrypt your db password and store it in your git repo and you use your keyczar key to decrypt the db password and access the db.  Which is more complex and no more secure than storing your db password wherever the keyczar key was kept.

2) You keep plaintext data outside your version control.  This offers protection when baddies getting ahold of your version control stuff, but does not protect you if they get access to your account or your data beyond the version control stuff (eg they get a complete copy of your home dir somehow and you kept that data in your home dir)  This is roughly just as secure as option 1 and a whole lot simpler.  I use this sometimes, where I'll have a file called .pass or something inside my working copy of a repo and I'll add .pass to the .gitignore file.  

3) You have an agent, like ssh-agent, that holds the password.   Said another way, you type one password into an agent and it makes a lot of encrypted stuff available to processes that can interact with it.  If a baddie hacks your account, they can submit encrypted stuff to the agent, but It does offer protection in a case where the baddie gets a copy of your data, including the keys, but does not have access to your account.  You could also use an agent that handles the encryption and produces clear text data (like your db password).  This is better than 1 and 2 because it only fails if your account is hacked and baddies can use it.

In conclusion, if you do add something to handle stored encrypted data, we should make sure it's actually more secure than just keeping the data in plain text in another directory.  Using an agent might be one way to do that.  There might be others.  

Sorry if I'm not making sense or saying something stupid or obvious.

-Dylan

[Michael DeHaan]

I'm very much in favor of the agent approach.

If there's an unlock "password" equivalent in the Ansible tree and the files are just obfuscated, we need to avoid that.

if the agent type approach involves just having Ansible ask for a password, that's ok... and I do like the idea of it spawning an editor to allow you to edit your password file in a transactional manner.

It is important that this need not rely on templates and lookup plugins and you can use regular vars_files type things, or if absolutely neccessary, something *like* vars_files, like a vault_files.

[Petros Moisiadis]

An agent-based solution does not make your global security necessarily
better because you still have a password, and that password should be
somewhere stored securely. Actually, you may add an extra security
measure on the systems on which you run ansible, but you should get more
concerned about the security of the systems that keep your passwords.

That said, the use of keyczar that Jen-Piet Mens introduced would work
fine with an agent-based password-protected session on a machine that
stores the keys. For example, you can put your keys on a (hardened) host
and access them from each managing machine through sshfs. Or you could
perhaps put the keys in a usb stick within an encrypted filesystem that
prompts for a password and gets automounted on the insertion of the
device into the managing hosts.

So, I think what we should discuss is not moving away from keyczar, but
what we can do to password-protect keyczar key stores and use them in an
agent-based fashion. Also see how ansible-vault can help us in
automating the involved processes.

[Michael DeHaan]

If I had a password to unlock my secret vault I wouldn't store it.

I think sshfs is a pretty 'bolted on' sort of thing (fuse?) and would like to avoid that.

What we need to do here is not discuss implementation and propose an implementation before we first arrive at a user interface and a specification.

I don't want to sound too much like I'm trying to dictate what goes on here, but as I am the one who gets the brunt of all the user questions and needs to support things, I do care very much that it presents as little questions as possible about how it works, and it works as smoothly as possible.

I need to dig through JP's code yet and see what I think about that approach and interface, and I'll share some comments here later.

It obviously does not eliminate the possibility of folks adding in their own modules and plugins and things outside of core, but if we do have something like this in core, the interface at which it presents itself is very important to me.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

[Petros Moisiadis]

On 07/03/13 16:39, Michael DeHaan wrote:

> What we need to do here is not discuss implementation and propose an
> implementation before we first arrive at a user interface and a
> specification.
>
> I don't want to sound too much like I'm trying to dictate what goes on
> here, but as I am the one who gets the brunt of all the user questions
> and needs to support things, I do care very much that it presents as
> little questions as possible about how it works, and it works as
> smoothly as possible.
>

See Jan-Piet Mens' recap message. Currently, it is the most well-shaped
approach which includes a suggested user interface and a discussion on
how it can be further improved. Of course it's ok to not like that idea,
but it would be better if there were alternative well-described
suggestions or specification drafts. In fact, I believe that this is
good time for this topic to be given a higher priority so that 1.3 be
the first version of Ansible that does offer core options for using it
with less fear in organizations with demanding security policies.

[Michael DeHaan]

I'm interested, but give me some time on this.

Smiting the bug queue down to 0 needs to happen first.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jan-Piet Mens]

This topic is keeping me awake BIG TIME :) I'd really like to see
encrypted vars in Ansible, and the simple way of doing it [1] appears to
be too simple; people want an agent.

How does the following sound?

A vault agent runs on a (protected) box on the network to which the
Ansible manager has access to (this need not be a dedicated box:
localhost will do very nicely). It's an RPC server which is responsible
for key creation and storage in files, and its main task is
encryption/decryption. Basically it does two things:

vault-agent --createkey mykey # create secret key & store it
vault-agent --run # start RPC server

On the Ansible side (ie within Ansible), we allow for *.vault files in
host_vars/ and group_vars/. These are YAML files which can only be
created/edited with a special command:

ansible-vault --key mykey --file host_vars/localhost.vault

The program (ansible-vault) checks with the RPC server whether 'mykey'
exists and drops the user into $EDITOR. Upon exiting $EDITOR,
ansible-vault sends the payload (i.e. what the human edited) to the
vault-agent for encryption with the specified key and obtains the
base64-encded result which is stored in host_vars/localhost.vault as
follows:

---
key: mykey # name of key used for encryption
payload: "Akxkxkxkxx=" # base64-encoded encrypted content

That's it.

When Ansible (ansible or ansible-playbook) runs and finds *.vault files,
it opens the *.vault file, requests from the agent via RPC that the
payload be decrypted with the specified key and uses the resulting
plaintext to feed into vars.

Keys will be created with Keyzcar (which need only exist on the agent
box). To keep requirements to a bare minimum, the initial implementation
will use Python's SimpleXMLRPCServer (but don't worry: you don't need to
learn XML :)

This basically does away with any kind of lookup plugin or template
filters -- as long as the vault-agent is running and reachable, it
should be transparent to Ansible.

Comments?

-JP

[1] https://github.com/ansible/ansible/pull/3393

[shirou]

Hello,

I am not a user just know the name, but how about using gpg-agent (not
ssh-agent)?

I glanced the python-gnupg doc[1], it can handle a gpg-agent with
use_agent argument.

(Since I haven't read all discussion, apologize if someone already
discuss or I miss the point.)

[1] http://pythonhosted.org/python-gnupg/

Regards,
Shirou


2013/7/4 Jan-Piet Mens <jpm...@gmail.com>:

[Jan-Piet Mens]

> I glanced the python-gnupg doc[1], it can handle a gpg-agent with
> use_agent argument.

That would mean people need a GPG infrastructure, which not everybody
has ...

-JP

[shirou]

> > I glanced the python-gnupg doc[1], it can handle a gpg-agent with
> > use_agent argument.
> That would mean people need a GPG infrastructure, which not everybody has ...

yah. that is a problem...


2013/7/4 Jan-Piet Mens <jpm...@gmail.com>:

[Michael DeHaan]

Here's an interface I'd like.   (Meaning, one I'd be happy to accept).   Small variations fine of course but notice the degree of encapsulation I am looking for.

First off, it works on any file Ansible can read but the file has a header so Ansible can yell if it's not running in a way that could decrypt them.   There must be absolutely no special syntax in YAML files, no lookup plugins, no templates.

This would work with vars_files, playbooks, whatever your want.

ansible-vault agent <get prompted for password>

ansible-vault create <YAML FILE> # spawns $EDITOR but on save will encrpt file

ansible-vault edit <YAML FILE>  # spawns $EDITOR ... if the file is unencrypted, just edit it like normal YAML, otherwise edit as encrypted file, on save, re-encrypts.   Be careful of temp files of course

ansible-playbook ... # works fine because we started the vault agent, but transparently works on all files as unencrypted.  Magic code lives in utils.parse_yaml_from_file.

Sometimes you might want to re-encrypt things so you can do:

ansible-vault change-password <directory> # prompts for old and new password to change the password on the file and re-encrypt it, and works optionally on entire subdirectories, so you could change multiple locked YAML files all in one pass

- hosts: blarg

   vars_files:

        ...

Running ansible without ansible-vault first would produce an error "vault file used without agent connection, see ansible-vault documentation for details on how to access encrypted files" when you tried to load a file and it couldn't decrypt it.

Finally you can shell exit to leave the agent, just like ssh-agent.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Petros Moisiadis]

First, I would like to make a comment that has more to do with the data representation and less with the UI:
Both of the suggested interfaces are encrypting files as a whole, which destroys yaml structure and makes the files unreadable and changes on them untraceable. I would prefer a solution that encrypts only the variable values. That way, encrypted files could be committed in repositories and continue to have diff logs that are meaningful (for history, auditing and troobleshooting). For example, you could easily see that Bob changed "mysql_root_password" or that Alice removed some credentials. Also, I think that implementing this would have no impact on any of the suggested UIs.

Secondly, on the UI:
Michael's interface proposal looks simpler. However, it does not provide information on how multiple passwords can be used (in parallel) for different files/variables.

[Michael DeHaan]

Yep, there's nothing to say a YAML encryptor algorithm could not just encrypt the "leaf" values of every YAML data structure, and also insert some code in the top to indicate it is "vaulted", even if something is as simple as a top level YAML hash element called "vault".

[Brian Coca]

what about a yaml interpreter hook like:

var: !!encrypted $6$ASDFSDH$%^DFF#$@E23435fSFWEFsd

I would put key location, method and salt in ansible.cfg/env vars.

[Michael DeHaan]

We don't want to do anything that uses more YAML arcanery.   I want to keep us free to represent things in other data structures, so things should be done at the datastructure level.

Also any vault metadata stuff it should have to save, which should be basically nothing, should be in each file if possible.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[GW]

I would recommend that before a solution is prepared:

- most common usage scenarios of such a feature are known

- exactly against which security attacks the solution should protect you against

I also like the idea of not reinventing the wheel and using GPG agent (most OS have support for its keys) or something that is known to work and be secure. Imho those who want to use this advanced features should know how to set up and work with GPG keys.

Greetings,
   gw 

[Dylan Martin]

You know, you could probably do this right now, today, with a combination of gpg & shell/command/raw

Playbook contains crypted text.  Shell module calls out to gpg/agent/whatever and you register the decrypted result.  (Does that make a secret show up in ps?)

Remember, keys are only for public key crypto.  Symmetric crypto doesn't need 'em.

You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/jAX8N3RMr-I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

[Michael DeHaan]

Yeah, we're not going to have playbook commands be part of that interface.

Want something a lot smoother than that :)