Announcing Montague: replace INI with YAML, JSON, etc

[Jon Rosebaugh]

Around last December, I got annoyed with the way PasteDeploy combines
the INI file parsing with the WSGI object loading. INI files are awfully
clumsy and there are many alternatives that offer features such as lists
and nested dicts, but as long as WSGI object loading was done through
PasteDeploy, Pyramid could not easily take advantage of these config
formats.

(Of course, I could write my own bin script to launch my Pyramid app
using my WSGI server of choice, loading config from a JSON file, but
this doesn't let me take advantage of the Pyramid tools such as pshell
or pviews.)

This kind of flexible configuration is particularly important as Docker
and similar tools become more popular for deploying apps. My Postgres
database may change IP from deployment to deployment, but the correct IP
is always available in environment variables. Of course, I can pull
values from environment variables in my Python code, but I have to
special case that everywhere that config settings might get loaded.
Montague provides a solution to this through "montague_mako", which
renders a Mako template with environment variables into a config file of
any kind.

Montague also provides full PasteDeploy backwards compatibility (through
an optional package). I'll be honest, figuring out how to do this sanely
is what took the bulk of the time.

I wrote a blog post about Montague here, which includes a demo loading a
WSGI app, middleware, and server from a TOML file:
https://metaclassical.com/announcing-montague-the-new-way-to-configure-python-applications/

It is also being used by at least one person other than me.

The project is still in early stages and the docs are kind of lacking.
But I think the code speaks for itself. I am using Montague while
developing my next Pyramid app and I am forking pserve/pshell/etc to
work with it.

I've encountered one particular difficulty with pshell and other Pyramid
scripts: Montague allows a config file to have more than one logging
config, which means I've combined development.ini and production.ini
into a single config.toml. pserve lets you specify which app and server
to load (while pshell does not, except with the PasteDeploy #app
syntax), but none of the scripts have a --logging-config argument.

It seems to me that it would be good for the p*scripts to standardize on
one particular way to specify the app, server, and logging config to
load from the config files. Would anyone be opposed to a pull request
making these changes?

[Mike Orr]

On Tue, Jun 23, 2015 at 12:55 PM, Jon Rosebaugh <j...@inklesspen.com> wrote:
> Around last December, I got annoyed with the way PasteDeploy combines the
> INI file parsing with the WSGI object loading. INI files are awfully clumsy
> and there are many alternatives that offer features such as lists and nested
> dicts, but as long as WSGI object loading was done through PasteDeploy,
> Pyramid could not easily take advantage of these config formats.

Hey, I was just thinking that last week, that maybe it would be worth
switching to YAML or JSON config files. I thought I would have to
write the glue code myself. I have a validator that I call from my
main function that validates the settings and converts their types. I
want any invalid settings to raise an exception at startup, not when
they're used in some obscure request. I've also got a fair amount of
list parsing and nested data structures and extra INI sections. It
takes a significant amount of code and deciding how to express these
in a setting or a section and write unit tests for them. I've found a
section works for something dict-like, and dotted key names can be
used for nested dicts or objects, but it's more cumbersome than
expressing them directly in YAML or JSON. I also have to create a
ConfigParser myself to read the section, and set the 'here' or
'__file__' key if I need them. Thanks for publishing Montague; I'll
try it out.

[Mike Orr]

> I've encountered one particular difficulty with pshell and other Pyramid
> scripts: Montague allows a config file to have more than one logging config,
> which means I've combined development.ini and production.ini into a single
> config.toml. pserve lets you specify which app and server to load (while
> pshell does not, except with the PasteDeploy #app syntax), but none of the
> scripts have a --logging-config argument.
>
> It seems to me that it would be good for the p*scripts to standardize on one
> particular way to specify the app, server, and logging config to load from
> the config files. Would anyone be opposed to a pull request making these
> changes?

I also have sometimes wanted to separate the logging configuration
from the rest of the file. The logging configuration is very verbose
with multiple stanzas and many lines, and sometimes I have several
files that differ in only one or two settings but the same logging
configuration. I also have multiple developers with their own file in
the version-control system, and often they all have the same logging
or would like me to manage the default logging options. Whenever I
change a default setting I have to replicate it in several files, and
that includes logging changes.

I also found that launching the application under uwsgi doesn't
configure the logging automatically. That's done by Waitress or
PasteDeploy Server or something like that, something that uwsgi
bypasses. Uwsgi has a '--paste-logger' option that's supposed to
configure a logging INI file, but it has never worked for me. So I've
defined a 'logging' key in my application, and if it's not empty, the
settings processor called by the main function calls FileConfig on the
file named in the value. Under Uwsgi the value is just "%(__file__)s"
or such, but this also allows me to use a different logging file under
Waitress if I want to.

[Domen Kožar]

I also have sometimes wanted to separate the logging configuration
from the rest of the file. The logging configuration is very verbose
with multiple stanzas and many lines, and sometimes I have several
files that differ in only one or two settings but the same logging
configuration. I also have multiple developers with their own file in
the version-control system, and often they all have the same logging
or would like me to manage the default logging options. Whenever I
change a default setting I have to replicate it in several files, and
that includes logging changes.

ConfigParser does allow "file inheritance", but sadly PasteDeploy never used the feature:

https://docs.python.org/2/library/configparser.html#ConfigParser.RawConfigParser.read 

[Jon Rosebaugh]

On 6/28/15 3:39 AM, Domen Kožar wrote:
> ConfigParser does allow "file inheritance", but sadly PasteDeploy never
> used the feature:
>
> https://docs.python.org/2/library/configparser.html#ConfigParser.RawConfigParser.read

Logging configuration isn't actually done with PasteDeploy;
logging.config provides a fileConfig function. I'm not surprised it
doesn't work with inheritance, though.

[Domen Kožar]

You can pass a list of filenames to logging.config.fileConfig and it will use them in order (including inheritance). Although that's not documented anywhere.

--
You received this message because you are subscribed to the Google Groups "pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-devel...@googlegroups.com.
To post to this group, send email to pylons...@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

[Mike Orr]

On Tue, Jun 23, 2015 at 12:55 PM, Jon Rosebaugh <j...@inklesspen.com> wrote:
> Around last December, I got annoyed with the way PasteDeploy combines the
> INI file parsing with the WSGI object loading. INI files are awfully clumsy
> and there are many alternatives that offer features such as lists and nested
> dicts, but as long as WSGI object loading was done through PasteDeploy,
> Pyramid could not easily take advantage of these config formats.
>
> (Of course, I could write my own bin script to launch my Pyramid app using
> my WSGI server of choice, loading config from a JSON file, but this doesn't
> let me take advantage of the Pyramid tools such as pshell or pviews.)
>
> This kind of flexible configuration is particularly important as Docker and
> similar tools become more popular for deploying apps. My Postgres database
> may change IP from deployment to deployment, but the correct IP is always
> available in environment variables. Of course, I can pull values from
> environment variables in my Python code, but I have to special case that
> everywhere that config settings might get loaded. Montague provides a
> solution to this through "montague_mako", which renders a Mako template with
> environment variables into a config file of any kind.
>
> Montague also provides full PasteDeploy backwards compatibility (through an
> optional package). I'll be honest, figuring out how to do this sanely is
> what took the bulk of the time.
>
> I wrote a blog post about Montague here, which includes a demo loading a
> WSGI app, middleware, and server from a TOML file:
> https://metaclassical.com/announcing-montague-the-new-way-to-configure-python-applications/

I found an issue with TOML files. It says you can't have a dot in a
key name, but some consumers look for dotted keys
("pyramid.reload_templates", "debugtoolbar.enabled"). Can those
settings not be used without postprocessing the settings?

[Jon Rosebaugh]

On 7/2/15 6:35 PM, Mike Orr wrote:
> I found an issue with TOML files. It says you can't have a dot in a
> key name, but some consumers look for dotted keys
> ("pyramid.reload_templates", "debugtoolbar.enabled"). Can those
> settings not be used without postprocessing the settings?

Dots are prohibited in bare keys, but all you have to do is wrap the key
in quotes. e.g.:

[some_section]
simple_value = "foo"
"pyramid.dotted.value" = "bar"

is parsed into this Python dict:

{u'some_section': {u'simple_value': u'foo', u'pyramid.dotted.value':
u'bar'}}

[Chris Withers]

As far as validating config schemas goes, I can highly recommend this
library:

https://pypi.python.org/pypi/voluptuous/

Flexible, powerful and easy to use.

Chris