How to best secure environment variables (secret key, passwords etc.) stored in .yml files?

[Tom Moore]

Hi there, I'm following the guidelines by making sure the environment variables are stored outside of the settings.py files.

The project is "dockerised" and so the environment variables have been stored in files docker-compose.yml and docker-compose-prod.yml.

This includes things like the project's secret key, API keys, and database passwords.

My question is: 

• Just because environment variables are stored in .yml files, won't they be equally insecure the moment I commit the project folder to a git repo (and especially if I push that repo to GitHub)?

e.g. the Secret Key will forevermore be stored in the git repo (in earlier versions, even if I later move it to another file in subsequent commits).

Is there an even more secure way of storing environment variables? Or am I overthinking it (as I'm the only developer and the GitHub repo is set to Private)?

Many thanks in advance for your help.

[Shaheed Haque]

I don't think you are overthinking this. 

On Thu, 30 Jan 2020, 12:40 Tom Moore, <he...@thomasmoore.me> wrote:

Hi there, I'm following the guidelines by making sure the environment variables are stored outside of the settings.py files.

The project is "dockerised" and so the environment variables have been stored in files docker-compose.yml and docker-compose-prod.yml.

What we do isn't perfect but does limit our exposure:

- we don't store the production key(s) in git at all. 

- instead, we fetch them at deployment time using a secure connection to the store, and then inject them into our runtime (we are still in a vm) 

The secure connection is the less than ideal part, since we require a human to provide the private key. In theory, we can replace that with automation at the cost of a short insecure connection (or some other complex key management). But we can change that at will as our scale needs evolve. 

This includes things like the project's secret key, API keys, and database passwords.

My question is: 

• Just because environment variables are stored in .yml files, won't they be equally insecure the moment I commit the project folder to a git repo (and especially if I push that repo to GitHub)?

e.g. the Secret Key will forevermore be stored in the git repo (in earlier versions, even if I later move it to another file in subsequent commits).

Is there an even more secure way of storing environment variables? Or am I overthinking it (as I'm the only developer and the GitHub repo is set to Private)?

Many thanks in advance for your help.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/887bcd5b-4525-4a54-a4e5-5eae32b20041%40googlegroups.com.

[Chris Wedgwood]

Hi Tom

You are definitely not overthinking this. it's important.

This is an area that has baked my noodle for a while now and I always am left wondering "Do I have this right?" "Am I vulnerable to attack?" ..... and I still haven't figured it out completely. It's like static files  I never really feeel like I get it entirely :)

Firstly you should never need to store a password/token/secret in Source Control ever. If you are stop and think there must be a better way.

I use environment variables .env to store my secrets but the trick is ALWAYS put that in your .gitignore  file. If you start a new git repository there is an option to create a .gitignore file 

for Python that is a great starting point.

To complement my .env file it has a .env.example file that I DO put in source control with a dummy password.

.env file:

MAILGUN_API_KEY =asjdhasds78dy9s8dy012287e210eu209e72

.env.example:

MAILGUN_API_KEY=ThisIsNotARealToken

So when I do local development  I can populate my .env fie with local dev secrets.

For production deployments, I use Ansible for which I provide production tokens and secrets in a separate file also not in source control.

The Ansible deployment requires an ssh password that I store in a Password Manager that has two-factor authentication.

The docker-compose file can read environment variables from the .env file.

Have a look at Django-Cookiecutter and see how they do it. That helped me a lot when I started out

cheers

Chris

[Michael Rohan]

Hi,

Just fyi, it was, in part, to solve this problem that I implemented the django-yamlconf module:

https://django-yamlconf.readthedocs.io/en/latest/

While getting the data into either a private yaml file or env variable in the container isn't addressed, it does allow injecting the values into the Django app without modifying the committed sources.

Take care,

Michael.

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/55f28dec-7c9a-4cae-b658-f89772aa1bd7%40googlegroups.com.

--

Michael Rohan
mro...@acm.org

[Bill Torcaso]

A couple of years ago I posted on this topic to say that using ENV variables is dangerously subject to human error.  If it ever happens that (1) you put a server on the public internet with DEBUG on, and (2) a visitor can provoke a 5xx server error response, then all of your secrets will be dumped in the stack-trace output.

(Yes, I know people will say "don't do that" regarding point #1, and similarly say "never publish code with a possible 500 error event".  But if it ever happens, you are screwed.  And you won't necessarily know that it happened.)

In general, if your secrets are kept in plaintext in your github repository and your repository gets compromised, then you have lost everything.

I've tried to think of a way to store the secrets in a plaintext file and encrypt the file's contents before committing the ciphertext file to the repository.  The question, as always, is where to store they key for decrypting the secrets file.

I have an idea in which the encryption key is manufactured on the fly from some non-obvious fact about the repository contents.  I'd like to get feedback from the Django community about it.

Summary:

• Put all secret info in a text file, but encrypt the file before it gets committed into the repository.
  
  
• Use an encryption key that is derived from some fact about the files in the repository.
  
  
• At runtime, use that same fact to derive the decryption key for the secrets file.
  
  
• Ensure that the runtime server can never reveal the fact that is the basis of the encryption key.

Suppose for example that the encryption key is the SHA-1 hash value of urls.py.  This is calculated during the release process, and used to encrypt the plaintext secrets file.  Then the encrypted secrets file is committed to the repository, and ultimately gets deployed to production.

urls.py obviously changes over time.  But it is constant for any given commit of the entire codebase, which is what gets deployed. 

At start-up time, the Django server can open urls.py and hash the contents, and use that hash value to decrypt the contents of the secrets file.  As a result, the secrets are available at runtime, but they are never stored in a plaintext file.

Why is this secure?  It relies on two assumptions.

1. Ensure that the contents of urls.py will never be published as part of an HTTPResponse.
   
   When this is true, the runtime server cannot reveal the decryption key for the encrypted secrets file.  That is to say, do not write a management command with name "getthesecretdecryptionkey".
   
   
2. Using security-through-obscurity, do not always use the same file (urls.py) as the basis of the hash value.  Instead, use some obscure selection criteria like "the seventh-largest .PY source file in the tree; in case of a tie, use the alphabetically last file".
   
   Similarly, do not always use the same hash method; instead, select a hash method from a list like [ "SHA-1, SHA-256, "MD5", ] by taking the size of urls.py modulo the length of that list.

I look forward to hearing comments about this approach.

Bill Torcaso

[Mike Dewhirst]

On 1/02/2020 2:24 am, Bill Torcaso wrote:
>
> A couple of years ago I posted on this topic to say that using ENV
> variables is dangerously subject to human error.  If it ever happens
> that (1) you put a server on the public internet with DEBUG on, and
> (2) a visitor can provoke a 5xx server error response, then all of
> your secrets will be dumped in the stack-trace output.
>
> (Yes, I know people will say "don't do that" regarding point #1, and
> similarly say "never publish code with a possible 500 error event". 
> But if it ever happens, you are screwed.  And you won't necessarily
> know that it happened.)
>
> In general, if your secrets are kept in plaintext in your github
> repository and your repository gets compromised, then you have lost
> everything.

I agree with all that. I hadn't considered the DEBUG gotcha but now you
mention it nothing is more certain. If it can go wrong it will. Don't
know whose law that is but I respect it. I always look at the worst case
and assume it will happen.

Please correct me if my assumptions are wrong but if the server itself
is compromised it doesn't matter about the secrets.

If I have access to the machine I could examine the code and find the
api which decrypts secrets and then write a script to fetch and decrypt
any secret. But why would I bother if I have already pwned the machine?

A real problem is spreading secrets around a multi-person dev team and
needing to change them all whenever a team member leaves.

Keeping them in editable text files in a standardised location for each
project lets the secrets be kept and easily updated by the minimum
number of people. Developers obviously know the standard location but
might require sudo to see them. That would be fine on their own machines
where their secrets can be mostly different than server secrets.

The layout of each secret in its own file named appropriately (eg.,
django_secret_staging.txt, django_secret_dev.txt, django_secret_prd.txt
etc) would permit the server software to retrieve the secret when
required using a simple method which works on all machines. I call mine
get_creds() and that lets me have different secrets between staging and
production. Compromising one machine doesn't undermine the other.

I think you are on the right track by keeping secrets in text files. I
just think encrypting them is a step too far.

Cheers

Mike

>
> I've tried to think of a way to store the secrets in a plaintext file
> and encrypt the file's contents before committing the ciphertext file
> to the repository.  The question, as always, is where to store they
> key for decrypting the secrets file.
>
> I have an idea in which the encryption key is manufactured on the fly
> from some non-obvious fact about the repository contents.  I'd like to
> get feedback from the Django community about it.
>
> Summary:
>

> * Put all secret info in a text file, but encrypt the file before it

> gets committed into the repository.
>

> * *Use an encryption key that is derived from some fact about the
> files in the repository.*
>
> * At runtime, use that same fact to derive the decryption key for
> the secrets file.
>
> * Ensure that the runtime server can never reveal the fact that is

> the basis of the encryption key.
>
>
> Suppose for example that the encryption key is the SHA-1 hash value of
> urls.py.  This is calculated during the release process, and used to
> encrypt the plaintext secrets file.  Then the encrypted secrets file
> is committed to the repository, and ultimately gets deployed to
> production.
>
> urls.py obviously changes over time.  But it is constant for any given
> commit of the entire codebase, which is what gets deployed.
>
> At start-up time, the Django server can open urls.py and hash the
> contents, and use that hash value to decrypt the contents of the
> secrets file.  As a result, the secrets are available at runtime, but
> they are never stored in a plaintext file.
>
> Why is this secure?  It relies on two assumptions.
>

> 1. Ensure that the contents of urls.py will never be published as

> part of an HTTPResponse.
>
> When this is true, the runtime server cannot reveal the decryption
> key for the encrypted secrets file.  That is to say, do not write
> a management command with name "getthesecretdecryptionkey".
>

> 2. Using security-through-obscurity, do not always use the same file

> (urls.py) as the basis of the hash value. Instead, use some
> obscure selection criteria like "the seventh-largest .PY source
> file in the tree; in case of a tie, use the alphabetically last file".
>
> Similarly, do not always use the same hash method; instead, select
> a hash method from a list like [ "SHA-1, SHA-256, "MD5", ] by
> taking the size of urls.py modulo the length of that list.
>
> I look forward to hearing comments about this approach.
>
> Bill Torcaso

> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com

> <mailto:django-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/91ae856c-8408-44c6-893d-12e1189a418d%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/91ae856c-8408-44c6-893d-12e1189a418d%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Tom Moore]

Thanks everyone for responding to this question. Nice to know (in a way) that I'm not overthinking it.. although now I'm worried about all the different ways a key could be compromised.

Your suggestions for separating out a .env file and using gitignore is really helpful.

Tom

To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/55f28dec-7c9a-4cae-b658-f89772aa1bd7%40googlegroups.com.

--

Michael Rohan
mro...@acm.org

[onlinejudge95]

+1 for the same

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

[Motaz Hejaze]

This discussion was very helpful

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAD%3DM5eTLPJKDendsP9DvYzi_bDXhOYFZgNG5ZEBsLg7bknGO2g%40mail.gmail.com.