Decoupling Postgres database credentials in django for deployment.

147 views
Skip to first unread message

prince gosavi

unread,
Mar 29, 2018, 4:24:40 PM3/29/18
to Django users
Hi,
I have made a django project and want to deploy it on cloud.
Before that i want to decouple all the private information.
I want to decouple the database info too, like the username password etc.
Any help is appreciated.

Andréas Kühne

unread,
Mar 30, 2018, 5:09:53 AM3/30/18
to django...@googlegroups.com
Hi, 

I am sorry, but this doesn't really make sense. What do you mean by decoupling the data? 

Deploying to the cloud, will mean that you will need to setup a new database for your project - there you will get a completly new database, that won't be connected to your development data in any way. 

You can then add system variables for the username and password for the database - and thats about all you need?

Regards,

Andréas

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4b09223d-d58f-4fda-b3f7-64b230960d94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

PASCUAL Eric

unread,
Mar 30, 2018, 6:07:53 AM3/30/18
to django...@googlegroups.com

Hi,


Environment variables are the best option for not embedding sensitive information in files which have chances to be stored in places such as external source repositories (GitHub et al.).


If you are working with orchestrators such as Kubernetes, you can use the "secrets", which are specialized shared configuration data, stored in encrypted form. As shared configuration data, they bring the extra benefit over env vars of being manageable from a single central place, and then distributed to all the replicas and services which need them.


Best regards


Eric

From: django...@googlegroups.com <django...@googlegroups.com> on behalf of Andréas Kühne <andrea...@hypercode.se>
Sent: Friday, March 30, 2018 11:08:52 AM
To: django...@googlegroups.com
Subject: Re: Decoupling Postgres database credentials in django for deployment.
 
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

Antonis Christofides

unread,
Mar 30, 2018, 8:11:57 AM3/30/18
to django...@googlegroups.com

Environment variables are the best option for not embedding sensitive information in files

However some people (me that is :-) think that environment variables, though widely used, are an ugly hack for this problem.


I explain the way I do it (TLDR; production settings must be stored in a different "deployment" repository, preferably with any secrets encrypted) in https://www.crowdcast.io/e/deploying-django, from 38m10s to 46m30s.


Regards,


Antonis

Antonis Christofides
http://djangodeployment.com

PASCUAL Eric

unread,
Mar 30, 2018, 10:16:31 AM3/30/18
to django...@googlegroups.com

Env vars are considered bad in contexts where everything runs in the same... environment. And I too have considered them as hacks or as options to avoid by all means.


But times are changing, and nowadays there is a move towards containerized apps, even outside the cloud context. And we are no more now in the traditional initial hype phase, which means that the approach has proven its benefits (and also its constraints and maybe weaknesses). As an example of a real life work project, I've deployed a docker compose stack on RaspberryPi's for running an instrumentation system, to avoid having to install each and every component and manage the possible conflicts or interferences between them. It works 24/7 for months now without any hiccup. Containers really shine for relieving you from DLLs/shared libs/packages/... hell.


That being said, in such an architecture, each container is an isolated environment, and thus you can use env vars without fearing that someone else could tampered with them.


So perhaps the time is coming for considering env vars are no more the evil :) It depends on when and how you use them. 


Best


Eric

From: django...@googlegroups.com <django...@googlegroups.com> on behalf of Antonis Christofides <ant...@djangodeployment.com>
Sent: Friday, March 30, 2018 2:11:02 PM

Bill Torcaso

unread,
Mar 30, 2018, 10:50:03 AM3/30/18
to Django users

I have a concern about using environment variables to hold secret information, and an opinion about it.  

IF

DEBUG is enabled, and there is a 500 server internal error, and the default 500 template is used to render the response, 

THEN

all of your secret information is shown in the browser output

Of course, DEBUG should never be enabled in production.  But a single human error might make it happen. 

I would prefer to trust Github security and long passwords than to think I am infallible about setting DEBUG. 

Note that this is certainly what happens when I run on a Vagrant VM, and I think it would be the same in a Docker-like container.

PASCUAL Eric

unread,
Mar 30, 2018, 11:53:00 AM3/30/18
to Django users

You're perfectly right about the "500 Error + DEBUG" case.


One solution is to set DEBUG to off by default, and turn it on by code in the setting module if detecting  that the app is executing in a dev or Q&A environment. Depending on your context, this can be done with rules based on the host name or some other properties of the target systems.


Best


Eric

From: django...@googlegroups.com <django...@googlegroups.com> on behalf of Bill Torcaso <torca...@gmail.com>
Sent: Friday, March 30, 2018 4:50:02 PM
To: Django users

Subject: Re: Decoupling Postgres database credentials in django for deployment.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

Phang Mulianto

unread,
Mar 30, 2018, 12:10:18 PM3/30/18
to django-users
I agree to set debug to false in your production configuration file. Never debug on production setup.

If need enable debug on, use in staging environment or with different django configuration files. 



To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/DB7P193MB03311541D366207EF76069C28CA10%40DB7P193MB0331.EURP193.PROD.OUTLOOK.COM.

Andréas Kühne

unread,
Mar 31, 2018, 8:29:38 AM3/31/18
to django...@googlegroups.com
All you should do is have production settings as default - all settings (like DEBUG) should be set the way you use them in production. Then you have a local setting on your development machine - that should enable DEBUG and also NEVER be stored in the repository. 

Development settings should not be stored as defaults in settings.py.

Med vänliga hälsningar,

Andréas

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2BSr5mTee6UdLUqXUnr8Pz1HNp5YTL%2BMME3-gjp_Mc-Q2CNJNQ%40mail.gmail.com.

Derek

unread,
Mar 31, 2018, 9:46:53 AM3/31/18
to Django users
How is github "security" going to help you keep your passwords safe and why is this better than ENV variables (which most of use without any problems).?

PASCUAL Eric

unread,
Apr 1, 2018, 5:36:50 AM4/1/18
to Django users

How is github "security" going to help you keep your passwords safe


IMHO, it does not.


As you wrote, env vars configured on the target system (be it a bare server or a Dockerized environment in the cloud) seem to be safer on this point since not stored anywhere but on the target.


Eric

From: django...@googlegroups.com <django...@googlegroups.com> on behalf of Derek <game...@gmail.com>
Sent: Saturday, March 31, 2018 3:46:53 PM

To: Django users
Subject: Re: Decoupling Postgres database credentials in django for deployment.
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

Cictani

unread,
Apr 2, 2018, 5:59:07 AM4/2/18
to Django users
Hi,

You could rename your dev settings file to for example 'settings_dev.py' and only commit this file (add settings.py to .gitignore).

On your Linux Server you could create a new directory in /etc

like:

/etc/django

/etc/django/app1
/etc/django/app2
...

There you store your production 'settings.py'

In your app directory you can create a symbolic link to these files:

ln -s /etc/django/app1/settings.py /path/to/app1/app1/

You have to make sure the directory in /etc is readably by www-data or whatever user you run your webserver with:

Now you can change the settings in the /etc directory and the settings are quite secure since only root will be able to change them by default. And you now do it the Linux way by storing all settings in /etc

Since you added settings.py to .gitignore it won't get overwritten. You should make backups of your whole /etc directory anyways so your django settings also get saved.

Mike Dewhirst

unread,
Apr 2, 2018, 6:51:14 PM4/2/18
to django...@googlegroups.com
On 2/04/2018 7:59 PM, Cictani wrote:
> Hi,
>
> You could rename your dev settings file to for example
> 'settings_dev.py' and only commit this file (add settings.py to
> .gitignore).

I wrote a tiny utility to read a file and retrieve credentials for any
purpose but especially for keeping database credentials out of the
repository.

It means I have to store the credentials files (separate file for each
eg database, email etc) in a place accessible to the web server but out
of the doc root.

The settings file only contains calls to the utility like this ...

# production.py

from .base import *



SITE_ID = 1  # prd



ALLOWED_HOSTS += ['redacted',]



# Databases  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

dbdefault = getcreds(fname='db.host', project="{0}-prd".format(PROJECT))

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.postgresql_psycopg2',

        'NAME': PROJECT,

        'USER': dbdefault[0],

        'PASSWORD': dbdefault[1],

        'HOST': dbdefault[2],

        'PORT': dbdefault[3],

    }

}



email_creds = getcreds(fname='smtp.host', project="{0}-prd".format(PROJECT))

EMAIL_HOST = email_creds[0]

EMAIL_PORT = email_creds[1]

EMAIL_HOST_USER = email_creds[2]

EMAIL_HOST_PASSWORD = email_creds[3]

DEFAULT_FROM_EMAIL = email_creds[4]

#EMAIL_USE_TLS = True

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

DEFAULT_CHARSET = 'utf-8'


And for local testing ...

# mike-test.py

from .local import *



SITE_ID = 3

DEBUG = True

SESSION_COOKIE_SECURE = False

CSRF_COOKIE_SECURE = False

SECURE_BROWSER_XSS_FILTER = False

SECURE_SSL_REDIRECT = False



TEMPLATES[0]['OPTIONS']['debug'] = DEBUG

TEST_RUNNER = 'django.test.runner.DiscoverRunner'



INSTALLED_APPS = DEFAULT_APPS + THIRD_PARTY_APPS + LOCAL_APPS



# # # # # # # # # # IN-MEMORY TEST DATABASE

dbdefault = getcreds(fname='db.host', project="{0}-local".format(PROJECT))

DATABASES = {

    'default': {

        "ENGINE": "django.db.backends.sqlite3",

        "NAME": ":memory:",

        "USER": "",

        "PASSWORD": "",

        "HOST": "",

        "PORT": "",

    }

}



EMAIL_BACKEND = 'django.core.mail.backends.locmem.EmailBackend'


And here is the utility ...

#getcreds.py

from __future__ import unicode_literals, absolute_import, division



import os





def getcreds(fname, project, credsroot='/var/www/creds', credsdir=None):

    """ return a list of userid and password and perhaps other data """

    if credsdir is None:

        credsdir = os.path.join(credsroot, project)

    creds = list()

    fname = os.path.join(credsdir, fname).replace("\\", "/")

    with open(fname, 'r') as f:

        for line in f:

            # remove leading/trailing whitespace and append to list

            creds.append(line.strip())

    assert creds, "The list of credentials is empty"

    return creds






>
> On your Linux Server you could create a new directory in /etc
>
> like:
>
> /etc/django
>
> /etc/django/app1
> /etc/django/app2
> ...
>
> There you store your production 'settings.py'
>
> In your app directory you can create a symbolic link to these files:
>
> |
> ln -s /etc/django/app1/settings.py /path/to/app1/app1/
> |
>
> You have to make sure the directory in /etc is readably by www-data or
> whatever user you run your webserver with:
>
> Now you can change the settings in the /etc directory and the settings
> are quite secure since only root will be able to change them by
> default. And you now do it the Linux way by storing all settings in /etc
>
> Since you added settings.py to .gitignore it won't get overwritten.
> You should make backups of your whole /etc directory anyways so your
> django settings also get saved.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com
> <mailto:django-users...@googlegroups.com>.
> To post to this group, send email to django...@googlegroups.com
> <mailto:django...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/8f3d6ebb-9db1-455e-99e3-d337490ffab5%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/8f3d6ebb-9db1-455e-99e3-d337490ffab5%40googlegroups.com?utm_medium=email&utm_source=footer>.

Mike Dewhirst

unread,
Apr 2, 2018, 7:18:20 PM4/2/18
to django...@googlegroups.com
Just realised I don't need to make that call for the in-memory database.
I should have posted the local.py settings which uses a local postgres
database. But you get the idea. Apologies for wasting space on the list.

M.

Bill Torcaso

unread,
Apr 22, 2018, 1:05:31 PM4/22/18
to Django users

I waited a while to answer this, and my answer comes in three distinct parts.

#---------------------

Question:  what is the danger is using environment variables to hold secret info?

Answer:  The Django runtime will dump secret info from environment variables into an HTTP response, in some circumstances.  Once secret info is in an HTTP response on the open internet, the secrecy is compromised.  As far as I know, the Django runtime will never dump the contents of the settings object into an HTTP response.

The defense of environment variables, as described in this thread are all vulnerable to human error.  Except possibly the one by Mike DeWhirst.

The human error is as simple as this:

On Elastic Beanstalk,

git checkout DEBUG_branch 
eb use  PRODUCTION_environment
eb deploy 
 
#---------------------

Question: what did I mean by "github security" as protection for secret information stored in a plaintext file in the github repository?

Answer:  There are two ways to access a file in a private github repository.  One way is through hacking my password, and the other is by breaching the protection that github applies to keep a private repository private.

My github password is random and has more than 120 bits of entropy.  In practical terms, is is safe from brute-force hacking.  As for the methods that github uses to keep a repository private, I have no specific information.  But I take it on faith that smart people have thought about it; and that's what I mean by "trusting github security".

#---------------------

Lastly,  I'm not trying to persuade anybody of the one true way to do things.  Every method of storing secrets has strengths and weaknesses.  I have an opinion: using environment variables is more dangerous than generally believed, and, in my opinion, keeping secrets in plaintext in the repository is less dangerous than generally believed.

Vinnicyus Gracindo

unread,
Apr 22, 2018, 6:48:48 PM4/22/18
to django...@googlegroups.com
Github: python-decouple 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5e34a2a5-e226-4688-9d53-d5151bb741be%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages