Secrets management, encryption and git; beyond cf-secret

[Cameron Kerr]

Hi all, I've recently become involved in a long-established CFEngine deployment. My previous CFEngine ended about 2018, and we didn't use Git at the time, so I'm looking for some advice with regard to security posture.

I'm currently working in an environment predominantly 3.15.<varies>, with a few outliers in 3.9.x and 3.21.x.

The issue I'm trying to address is that currently there are a lot of unencrypted secrets in the Git repository for our CFEngine policies, which while highly objectionable, is understandable given that cf-secret landed in 3.16.0 (and maybe 3.15.3).

Reading into cf-secret I find it lacks some capabilities I would have expected. I'm looking for community experience for keeping unencrypted secrets out of Git. I will accept having encrypted secrets in Git as a suitable middle-ground.

I see no capability within CFEngine to have variables that are stored in the likes of a secrets-manager such as AWS Parameter Store, or Azure Key Vault, Hashicorp Vault.

Ideally I'd want the only unencrypted secrets to be in their rendered configuration files, which might get inserted by an edit_line, edit_template or file copy promise (similar to Ansible Vault)

I'm _not_ looking to introduce an application dependency on configured applications to use the likes of HashiCorp Vault, or even to require them to obtain credentials in any way different to how they currently do; writing unencrypted secrets into an application's configuration file is completely desirable for these requirements (and yes, this could be improved further). So long as the variables are not present unencrypted elsewhere, particularly in Git. Bonus points if they aren't present unencrypted on the policy hub.

Idea #1: cf-secret

* Upgrade the fleet to be 3.16.0 or later (preferably targeting latest LTS release)

* Ensure all secrets are embedded in a data variables that comes from a file that can be sourced (eg. data.json), and that each server has its own version, encrypted according to its key.

* Now we have potentially many copies for each original file, so we need to solve for securing that original file and updating the various encrypted copies.

eg. data-original.json, data-host1.json, data-host2.json

Doesn't feel like it would be particularly scalable.

Idea #1b: Use Mozilla SOPS instead of cf-secret

I've encountered this problem in other similar situations before, and had discovered (but not used much) SOPS: https://github.com/mozilla/sops

It is more general than cf-secret (particularly in the ways it can encrypt files).... but is not so different to cf-secret. Likely it just has a different set of trade-offs (particularly if you use GPG), although I'm likely to trust it more than cf-secret, considering the credentials of the person who created it.

Idea #2: Provision the secrets outside of CFEngine using CI/CD.

This environment already has a practice of using CI/CD deployment, where secrets are injected by the CD tooling. In this case, we could do the following:

* introduce some data file (eg. a data.json) or files (eg. one per module?) that are stored in a particular location on each server. These files could either be unencrypted, or encrypted via cf-secret.

* it would remain preferable if the variable definitions (eg. naming, structure) would remain in Git, but the values be injected elsewhere.

* CFEngine should be configured to abort (partially?) if the file has not been delivered.

* the secret data could be pushed by CD technology (eg. Jenkins, Team City, Bamboo).

* it would likely entail enrolling many more servers in CD

* an alternative to using CD would be to use another push-based mechanism such as Ansible.

I'm thinking this potential solution is best suited for an on-premises environment that has no pre-existing secrets-management API.

Idea #3: CFEngine pulls data from a Secrets Management API

For environments that do have a Secrets Management API (eg. Azure Key Vault, AWS Secrets Manager, Hashicorp Vault), then we could:

* deliver a script (using CFEngine) that will query the API to obtain the secrets and write the result into a JSON or YAML output that can be parsed into a data variable. If this in a cloud environment, use Instance Roles etc. to govern and audit access; otherwise you need to solve for seeding access key for API.

* CFEngine should be configured to abort (partially?) if the data could not be obtained.

This solution is most-attractive, particularly in an environment where technology such as AWS Secrets Manager is in play, where it can rotate database credentials etc.

For my client's current need, I suspect that Idea #2 (CI/CD) will be the most appropriate, given what they do (not) have available.

If anyone has any resources (worked examples) or deployment experience (gotchas, success-stories), I would be really happy to receive them.

One other question.... if you create a data variable (containing unencrypted secrets) from a command that emits yaml/json... does that variable get persisted anywhere? I'm thinking its likely that it will get remembered for a period of time, presumably somewhere under /var/cfengine/

Thank you very much for your time and expertise,

Cameron

[Aleksey Tsalolikhin]

Hi Cameron,

Sorry for the late reply.

I'm currently using option #3 for secrets distribution.  I'm in an AWS shop, so I'm using AWS Secrets Manager.

In a past life, to distribute secrets (and this was before cf-secret), we would manually put them on the hub (unencrypted but only readable by root) where the clients would pick them up, use them and delete them.  Not super secure but better than putting them in Git.

Best,

Aleksey

--

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/4f325327-fd6d-4b29-930a-d26b13d717b5n%40googlegroups.com.