Config: DB or git?

[Thomas Güttler]

Hi,

of course we separate data from code:

- code belongs into version control (git)
- data belongs into a database (postgres)

But where does configuration belong?

Since I am a developer I like version control.

But the longer I think about this question, I think
the perfect configuration for an app just contains
one entry: How to connect to the DB.

Sooner or later you want things to editable via an admin interface.

And if you look at big systems like SAP. There is only very little config in files.

I think config should be done in the database, not files. It is hard to accept
this, since version control is great for files, and not so good for databases,
but I think it is the way to go.

What do you think?

Thomas



--
Thomas Güttler
http://thomas-guettler.de/

[Alejandro Varas G.]

Hi Thomas,

El 29/09/2014 10:05, "Thomas Güttler" <h...@tbz-pariv.de> escribió:
>
> Hi,
>
> of course we separate data from code:
>
>  - code belongs into version control (git)
>  - data belongs into a database (postgres)
>
> But where does configuration belong?

"To the environment"

A good practice is to place config in the environment[0] and load it from code.

0 - http://12factor.net/config

[Erik Cederstrand]

Den 29/09/2014 kl. 15.04 skrev Thomas Güttler <h...@tbz-pariv.de>:

> Hi,
>
> of course we separate data from code:
>
> - code belongs into version control (git)
> - data belongs into a database (postgres)
>
> But where does configuration belong?

Strictly speaking, configuration belongs in a configuration management system - Ansible, Chef and Puppet being just examples. The words option, setting, preference and configuration are often used interchangeably, but if you define "configuration" as "something that is loaded by the program on startup" and which is not universal for all installations, then you must store configuration independently from the VCS and DB.

Depending on your hosting environment, you could store your host-specific configuration in a versioned Fabric script. But keep in mind that storing cleartext passwords is bad practice, be it in a database or in Git.

Erik

[Derek]

So is it possible to store encrypted database passwords in the Django settings (or local_settings) file? 

[Javier Guerra Giraldez]

On Mon, Sep 29, 2014 at 8:15 AM, Alejandro Varas G.
<alej0...@gmail.com> wrote:
>> But where does configuration belong?
>
> "To the environment"

this is the currently fashionable answer, but I haven't seen any
justification for it. Unfortunately, the 12factors manifesto tells
the how, but very little of the why.

In my experience, environment variables are clumsy, fragile and a pain
to work with.

I guess a more balanced option would be to store somewhere else,
probably the database (although i'm not too opposed to VCS) with the
host-variable parts (or everything, who cares about a couple KB extra
storage?) as a set indexed by hostname, which is read from the
environment.

--
Javier

[Daniel Rus Morales]

I see configurable settings, like those in the settings module, as code settings, and those you would put in the DB as user customizable settings. The former are meant to be changed by developers and sys admins, and the latter by website users. In my opinion settings module changes are mostly related to staging or performance rather than user oriented. Apart from there are a number of interesting Django apps in Djangopackages [1] that allow you to put your settings in the DB.

[1] -> https://www.djangopackages.com/grids/g/configuration/

Regards

[Erik Cederstrand]

Den 29/09/2014 kl. 16.11 skrev Derek <game...@gmail.com>:
>
> So is it possible to store encrypted database passwords in the Django settings (or local_settings) file?

Yes. A simple solution would be to use a symmetric encryption like simple-crypt to encrypt and decrypt your passwords using a key that you provide externally. Here's a quick example:

import simplecrypt

SECRET_KEY = 'my_secret_key' # Get this from a file, enter it manually or by other sufficiently secure means

# Example code to generate the encrypred password
my_password = 'super_secret'
my_encrypted_password = simplecrypt.encrypt(SECRET_KEY, text)

# This is what you put in settings.py
my_encrypted_password = 'KLJJ543#€%'
my_password = simplecrypt.decrypt(SECRET_KEY, my_encrypted_password)


Erik

[Thomas Güttler]

....

> I see configurable settings, like those in the settings module, as code settings, and those you would put in the DB as
> user customizable settings. The former are meant to be changed by developers and sys admins, and the latter by website
> users. In my opinion settings module changes are mostly related to staging or performance rather than user oriented.
> Apart from there are a number of interesting Django apps in Djangopackages [1] that allow you to put your settings in
> the DB.
>
> [1] -> https://www.djangopackages.com/grids/g/configuration/
>
> Regards

Thank you for this link. Unfortunately the feature "data store" is not evaluated up to now. But I created
a feature request: https://github.com/pydanny/djangopackages/issues/293

[Tom Evans]

I don't want to start a holy war, but I 100% disagree with putting
configuration in to the environment.

First of all, how does it get in the environment in the first place?
You must write a shell script that puts it in the environment. It may
not be a complex shell script; but it is a shell script.

A main part of maintaining a software project is managing
configuration changes. The best way of managing configuration changes
is store your configuration in a VCS in order to track changes.

So you add your shell script that configures your environment to a VCS¹.

Now your configuration is still "in code", but its not in "your code",
and now it's not even in the same language. About the only good thing
that is achieved is that the configuration is in a separate
repository, but that is orthogonal to sticking your DB credentials in
to the environment - you could do that with any configuration file.

Finally, the environment is a very public place to put things. The
environment is passed to any subprocesses you spawn, and included in
any debug emails. Django goes to some trouble to scrub passwords from
the settings it puts in debug emails; I don't see anything in the
Django docs about it scrubbing the environment..

Cheers

Tom

¹ Now, some wise-ass is going to come along and say "Oh no, thats not
what you do, you put the environment settings in to puppet/chef/etc".
You don't store your puppet configuration in VCS? *How* are you
managing change exactly?

[Vernon D. Cole]

12-factor is all the rage, and they have some very good ideas that ought to be followed.  But insisting that environment variables are the only correct way of storing settings, is, well, just plain wrong. [ Note: I have 40 years of experience, I recognize a fad when I see it. ] All of those values have to get _into_ the environment variables somehow, and that means that somewhere there is a disk file which has them in it.  Is it really safer to have them written in some language other than Python so that you have to have a special program to read them?  I doubt it.

So my preference is to use a structured settings arrangement, where the file which is referenced by DJANGO_SETTINGS_MODULE imports settings.py (not vice-verso). 
That outside settings module, called salt_settings.py on my systems, is written by the SaltStack "state" which deploys my server.
The same "state" also creates the database, the nginx configulation, and the uWSGI configuration, so the same passwords, sockets, database names, etc, are used on both sides of an interface.
The sensitive data, such as passwords, are kept in a Salt Pillar structure, which is in a (highly protected) repo, separate from the salt "state" files, which are in a separate repo from the application.  Three purposes, three repos.

Why SaltStack and not one of the other configuration engines mentioned above?  Because Salt, like django, is written using a Python templating system.  You do not program your configuration in Ruby (Puppet and Chef) or Python (fabric), but lay it out in nice, readable, white-space sensitive YAML files, with Jinja templates where needed.
I have an over-simplified example at https://github.com/eHealthAfrica/salt_demo
--
Venon Cole

[Arthur Alvim]

There's a really nice project called python-decouple. (https://github.com/henriquebastos/python-decouple/)

You can store you vars on .ini or .env files and ignore then on git.

Install

pip install python-decouple

Usage

On your settings.py.

1. Import the config object:

   from decouple import config
   

2. Retrieve the configuration parameters:

   SECRET_KEY = config('SECRET_KEY')
   DEBUG = config('DEBUG', default=False, cast=bool)
   EMAIL_HOST = config('EMAIL_HOST', default='localhost')
   EMAIL_PORT = config('EMAIL_PORT', default=25, cast=int)

 

-- Atenciosamente
-  Arthur Alvim [ arth....@gmail.com ]

IFPE - Tecnólogo em Análise e Desenvolvimento de Sistemas
UFPE CIn - Mestre em Ciência da Computação

"Partilha os teus conhecimentos. É a forma de conseguires a imortalidade."
Dalai Lama

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f518e9ed-e22d-49d6-8f0f-5ef8981b6d88%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jorge Andrés Vergara Ebratt]

I've been checking out python-decouple and I have to say it's pretty easy, it replaces django-dotenv, django-getenv and django-autoenv for those working in a 12factorish way

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CANjJ3%3DMTW0euFcQiv3QPTV4oidHWay91uK7oviAU8_YF71CN4g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Jorge Andres Vergara Ebratt

#SoftwareDeveloper (Or at least trying to be)

@javebratt

facebook.com/javebratt

[Collin Anderson]

If you're a programmer or sysadmin, configuration should be done in files. If you're not a programmer or sysadmin, it should be done in the database.

[Mike Dewhirst]

On 1/10/2014 10:56 PM, Collin Anderson wrote:
> If you're a programmer or sysadmin, configuration should be done in
> files.

#getcreds.py

# -*- coding: utf-8 -*-

# this is the only django import permitted in settings files
from django.core.exceptions import ImproperlyConfigured

def getcreds(fname, project, credsroot='/var/www/creds'):
""" return a list of userid and password and perhaps other data """
credsdir = '%s/%s' % (credsroot, project)
creds = []
fname = '%s/%s' % (credsdir, fname)
with open(fname, 'r') as f:
for line in f:
creds.append(line.strip())
if not creds:
raise ImproperlyConfigured('Missing setting: %s' % fname)
return creds

If you're not a programmer or sysadmin, it should be done in the
> database.

Don't understand this ...

>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com

> <mailto:django-users...@googlegroups.com>.

> To post to this group, send email to django...@googlegroups.com

> <mailto:django...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/cf7129be-8eb2-4e3b-8938-50df669fa484%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/cf7129be-8eb2-4e3b-8938-50df669fa484%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Collin Anderson]

If you're not a programmer or sysadmin, it should be done in the
> database.

Basically if only users are configuring things (who don't know how to use revision control), then I think it makes sense to have it in the database.

[Michael Manfre]

I don't think your claim holds true for larger projects or at organizations that have more business processes. At my current company, the configuration for the service must be in a database (DynamoDB) to allow for centralized management, auditing, and the ability to change the configuration without a deployment or service restart.

Regards,

Michael Manfre

[Thomas Güttler]

Am 01.10.2014 um 14:56 schrieb Collin Anderson:
> If you're a programmer or sysadmin, configuration should be done in files. If you're not a programmer or sysadmin, it
> should be done in the database.

Why do you look the person for a decision like this?

Wouldn't it be better to look at the data which gets configured here?

I love git like most programmers love their version control system.

But for most stuff config in DB is much better.

example: INSTALLED_APPS

Wouldn't it be really cool if you could add an app by pressing a button?

[Tom Evans]

I have developed, maintained, managed and supported a generic ORM that
stores all details about how things are configured - models, fields,
classes, behaviours - in the DB for the past ten years, and I can tell
you that, no, it is not really cool, it is a massive pain in the
proverbial.

How your models and fields behave is not configuration it is code.
What models are part of a project is not configuration, it is code.
What folder a project is in, what credentials to connect to a
database, what proxy to talk to - these things are configuration.

They are all settings, but some settings are configuration, some
settings are part of your codebase. If you change what is in
INSTALLED_APPS, it is no longer the same project, but if you change
what DB server to connect to it is still the same project.

Cheers

Tom

[Thomas Güttler]

Am 02.10.2014 um 14:11 schrieb Tom Evans:
> On Thu, Oct 2, 2014 at 12:21 PM, Thomas Güttler <h...@tbz-pariv.de> wrote:
>> Am 01.10.2014 um 14:56 schrieb Collin Anderson:
>>>
>>> If you're a programmer or sysadmin, configuration should be done in files.
>>> If you're not a programmer or sysadmin, it
>>> should be done in the database.
>>
>>
>> Why do you look the person for a decision like this?
>>
>> Wouldn't it be better to look at the data which gets configured here?
>>
>> I love git like most programmers love their version control system.
>>
>> But for most stuff config in DB is much better.
>>
>> example: INSTALLED_APPS
>>
>> Wouldn't it be really cool if you could add an app by pressing a button?
>>
>
> I have developed, maintained, managed and supported a generic ORM that
> stores all details about how things are configured - models, fields,
> classes, behaviours - in the DB for the past ten years, and I can tell
> you that, no, it is not really cool, it is a massive pain in the
> proverbial.

Yes, you are right. Storing the ORM models in DB is too much.

[Lee]

Isnt convenience vs flexibility the difference between choosing to use a full CMS and choosing to use a web development framework? Id look for that in the former, but not the latter - though Im sure .net "programmers" would disagree ;-)

[Saurabh Kumar]

I like to handle things the way Heroku does.

1. Use fabric or ansible to define the machine configurations.

2. Use a .env file on remote server, where you can store the setting values, which can be loaded into django settings.

python-dotenv[1] is a nifty tool that works with fabric to manage your remote .env file. Changing a config value on a dev 

server can be as simple as:

fab dev config:set,DEBUG,False

[1]: https://github.com/theskumar/python-dotenv

​--​

​Saurabh Kumar

http://saurabh-kumar.com

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/542D5946.8020706%40tbz-pariv.de.

[Collin Anderson]

Wouldn't it be really cool if you could add an app by pressing a button?

I'd rather use the command line, like "apt-get install package" or "pip install package". I also like the "git config" command. dotenv looks promising too. "manage.py install" and "manage.py config" would be cool, but machine-editing settings.py would be hard to pull off in reality.

The issue I've had with database configuration, is that if I, or anyone else, wants to make changes in development, how do you push those changes live? You have to keep track of (or guess at, if there's no record) exactly what was changed and then copy and paste everything.

Drupal, for example, is really big on database configuration, however in the end, they've found that they need a way to save that configuration to a file: https://www.drupal.org/node/1585750