Paste Deploy config format
Cito
stemming from the fact that Paste Deploy (particularly the paster
serve command) uses a config file format where config values are not
evaluated, i.e. all the config settings (including booleans, ints
etc.) are stored as a string. So you need to manually convert non-
string values (like it is done in paste.httpserver.server_runner).
This is inconvenient, less performant and you can easily forget to do
this.
The problem gets aggravated in TurboGears, since these config files
are merged together with the rest of the TurboGears config, which
contains settings made in app_cfg.py which are of course Python
objects.
We were actually using evaluated config files in Turbogears1 (based on
CherryPy and ConfigObj which both do this), but Turbogears2 is now
based on Pylons+Paste. The problem is that we now have two
incompatible config file formats, which is quite confusing, and even
worse, depending on in which config file a setting was made, the
applicatons needs to evaluate the config setting or not.
The best solution for us would be if Paste Deploy changes its config
format so that the values are evaluated, too.
I'm writing here because I saw this point also mentioned under
"outstanding issues": http://pythonpaste.org/deploy/#outstanding-issues
In order to harmonize the config formats in TurboGears, it is
important for us to know: Do you still consider changing the format?
Or is this already out of the question anyway?
-- Christoph
Jorge Vargas
for the record I'll love it if paste will grow python (optional) in
it's config files. But I don't want it to go all the say to something
like ConfigObj (nested [[sections]] ) and other non-default ini stuff.
Evaluation of values seems like a nice enough feature.
Max Ischenko
for the record I'll love it if paste will grow python (optional) in> The best solution for us would be if Paste Deploy changes its config
> format so that the values are evaluated, too.
>
> I'm writing here because I saw this point also mentioned under
> "outstanding issues": http://pythonpaste.org/deploy/#outstanding-issues
>
> In order to harmonize the config formats in TurboGears, it is
> important for us to know: Do you still consider changing the format?
> Or is this already out of the question anyway?
>
it's config files. But I don't want it to go all the say to something
like ConfigObj (nested [[sections]] ) and other non-default ini stuff.
Evaluation of values seems like a nice enough feature.
mod_wsgi app files and django settings files work quite nicely.
Make it simpler to make it more versatile. Making it even more complicated won't help, IMO.
Best,
Max
Christoph Zwerschke
> Make it simpler to make it more versatile. Making it even more
> complicated won't help, IMO.
I'm not talking about making anything more complicated. I'm just
discussing whether .ini settings should be read in raw (like raw_input()
does) or evaluated (like input() does).
Currently they are not evaluated, which leads to many problems: You will
then have to evaluated booleans, ints and other non-strings yourself
when you access the config setting. Some parts of the code also evaluate
these settings in place (server_runner() in paste.httpserver). If other
parts of the code then also want to access the same setting you must not
evaluate these. And then there is code like TG2 which merges settings
from pure Python configs into the same config. So you end up with a
config where settings may or may not be evaluated, you never know.
Therefore I think it is much better (and more performant for settings
which are accessed very often, like in each request), to *always*
evaluate settings at the very beginning when they are read in.
-- Christoph
Jasiu
I tried to look inside Paste's code and AFAICT Paste reads config
files using Python's built-in module ConfigParser. This module however
doesn't seem to be able to eval strings into Python objects. So
enabling this feature sounds like a lot of work. Am I right?
I'd also love to have this feature in Paste, but isn't it a little bit
of security threat? How about this config snippet:
[section]
malicious_key = shutils.rmtree('/')
I guess you see my point. I know, that's easy to fix, you just have to
give correct locals and globals arguments to eval function. But
someone would have to think it over and decide which objects should
get into locals/globals (datetime, timedelta for example), and which
shouldn't.
I'm also wondering what would happen with backward compatibility. Take
a look at that:
[section]
some_key = 'Evaluating requires strings to be put between
apostrophes.'
other_key = But currently you don't have to use apostrophes.
How to fix that? There's an obvious way - try to evaluate the string
read from the config file and if you get an exception, leave it as it
is. But is this enough? Maybe there are some ill special cases that
don't fit into this approach? I don't know.
Cheers,
Mike
Christoph Zwerschke
> I tried to look inside Paste's code and AFAICT Paste reads config
> files using Python's built-in module ConfigParser. This module however
> doesn't seem to be able to eval strings into Python objects. So
> enabling this feature sounds like a lot of work. Am I right?
Right, ConfigParser does not do this automatically. Instead, it provides
accessor methods for typical types such as booleans, ints or floats.
But it is very easy to do this. You only need to loop through all values
after loading and eval them.
My point is that it is much easier and less work to have this done once
by the library when a config file is read, since otherwise *you* have to
do it anyway when you access the value (and you may not be sure whether
the value *was* already evaluated by somebody accessing it before you,
or because it was merged in from another source).
> I'd also love to have this feature in Paste, but isn't it a little bit
> of security threat? How about this config snippet:
>
> [section]
> malicious_key = shutils.rmtree('/')
>
> I guess you see my point. I know, that's easy to fix, you just have to
> give correct locals and globals arguments to eval function. But
> someone would have to think it over and decide which objects should
> get into locals/globals (datetime, timedelta for example), and which
> shouldn't.
These are config files written by admins and deployers, not the casual
user. Also, as you say, you can restrict the namespace.
You can have a look at how CherryPy does it. They use a special
"unrepr()" function instead of eval().
> I'm also wondering what would happen with backward compatibility. Take
> a look at that:
>
> [section]
> some_key = 'Evaluating requires strings to be put between
> apostrophes.'
> other_key = But currently you don't have to use apostrophes.
>
> How to fix that? There's an obvious way - try to evaluate the string
> read from the config file and if you get an exception, leave it as it
> is. But is this enough? Maybe there are some ill special cases that
> don't fit into this approach? I don't know.
Yes, that's the main problem.
But I think the solution you mention is applicable. We could also output
a warning messages in this case, and the users can then adapt their
config files or see if their was really a syntax error.
-- Christoph
Sergey Schetinin
is to use functionality implemented in paste script. But the same
functionality could be accessed as a library. Just think about
distutils and setup.py -- that's a much better model.
Best Regards,
Sergey Schetinin
http://s3bk.com/ -- S3 Backup
http://word-to-html.com/ -- Word to HTML Converter
Kevin Dangoor
> I'm with Max on this one. The only reason to write paste config files
> is to use functionality implemented in paste script. But the same
> functionality could be accessed as a library. Just think about
> distutils and setup.py -- that's a much better model.
Depends on the problems you're trying to solve. If you look at what
pip has to do to gather package metadata, you wouldn't think that
setup.py is such a great model.
That said, I personally configure my WSGI components in Python code.
It's easy, and I provide the switches that I need to provide for my
different environments.
Kevin
Sergey Schetinin
> On Sat, Feb 28, 2009 at 9:17 AM, Sergey Schetinin <mal...@gmail.com> wrote:
>> I'm with Max on this one. The only reason to write paste config files
>> is to use functionality implemented in paste script. But the same
>> functionality could be accessed as a library. Just think about
>> distutils and setup.py -- that's a much better model.
>
> Depends on the problems you're trying to solve. If you look at what
> pip has to do to gather package metadata, you wouldn't think that
> setup.py is such a great model.
Could you please go into more detail on this?
I know about issues of extracting metadata from such a format, but
first of all it's of minor concern (if any) for serving webapps, and
it in no way justifies the pain of having to write so much of
otherwise unnecessary code and having to deal with all those
limitations. On balance it's just pointless.
And secondly, extracting metadata can be implemented as one of the
commands, so that "setup.py info" would print metadata in easily
parseable format for indexing packages, dependencies etc. I find it an
odd omission that distutils doesn't have it. Even setuptools' egg_info
doesn't print the info but writes it to a special folder.
I maintain that distutils model is way better.
Kevin Dangoor
I should say that I wasn't actually disagreeing with you. The way that
I configure my WSGI apps is like this:
* have a python module that holds some config values in a dictionary
(usually a Bunch)
* sometimes overlay some values pulled from an ini file into that dictionary
* have some python code that instantiates and plugs together the
middleware/app with options set in that dictionary
Also, I created Paver in large part because I found zc.buildout's ini
files to be very annoying.
And, of course, I tended to prefer python config files for TurboGears...
My point about pip and distutils/setuptools was just that that didn't
seem like such a great example to me, because pip has to do this awful
bit where it monkeypatches the setup function and then runs the setup
script in order to get the metadata. You're right that a setup.py info
command would have been handy. Even more relevant to this conversation
is your point that metadata matters to distutils, but it doesn't
really for deploying apps.
All of this is a long-winded way of saying "sorry I even mentioned it"
:) I think Python files are a great way to configure WSGI apps,
and that's exactly how I do it.
Kevin
--
Kevin Dangoor
work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com
Sergey Schetinin
I'm glad we agree on that. To clarify myself I don't think that
distutils doesn't have its problems, but it seems like the best model
for webapp config / serving solution. And I mentioned it specifically
because the way mod_wsgi scripts work, one has to store the WSGI app
in global namespace as 'application', I liked that approach a lot, and
I used to have a program that would load such scripts and serve them
locally. So it still was a serving program / config file setup almost
like `paster` command and its .ini files.
But then I stopped using mod_wsgi so it seemed that this system could
be improved, and now instead of just having 'application' in global
namespace of some `webapp.wsgi` file I am writing regular serve.py
scripts that end with something like this:
from wsgiserve import serve
serve(app, default_host='0.0.0.0')
This had an unexpected positive effect because extending my
wsgiserve.serve was way easier than before. Now my serve.py script are
small self-contained apps that are super-easy to serve, debug and
deploy. It's not that different from what `paster` can do but can also
register itself as init.d script, for example:
serve.py service install webapp-v10 debug autoreload --port=80 --host=127.0.0.1
This would add script named webapp-v10 to init.d. That script would be
serving the app from serve.py wrapped in debugprint and evalexceptions
middleware on localhost:80 with cherrypy.wsgiserver and with
autoreloading.
I can remove it with:
serve.py service remove webapp-v10
So this thing got even further from .ini files than .wsgi scripts of
mod_wsgi and that's why I mentioned distutils.
And the reason why I mention installing an init.d script is because it
fits well with deployment scripts. While I never particularly liked
setup.py files this showed me that it's a good way to get these kind
of things done. I ended up with a similar thing for deployment. So my
deploy scripts look like this:
deploy('host', 'webapp', 'serve.py',
require=['SQLAlchemy'', 'lxml' etc],
export=['/Websites/webapp/www/site_pkg',
'/Websites/webapp/www/serve.py',
],
)
'host' would be the name of the machine where it will be deployed.
'webapp' would be the prefix used for init.d script names
serve.py is the name of the serve script that will be exported from
svn checkout as specified by `export` argument.
This file would be named deploy.py. To deploy it I would run:
deploy.py v1.0 create 80
That would
* log into the host machine,
* create virtualenv on path based on name I passed (v1.0),
* install the requirements.
* update the svn checkout (which it knows where to find)
* then export the paths specified from that checkout
* install the app defined by serve.py into init.d under the name
webapp-v1.0 to serve on port 80 and with other parameters we might
want.
* start the server
There are more commands, for example `deploy.py dev update` would
* log in
* update the svn checkout
* do the export for 'dev' deployment
* restart its server
`deploy.py dev update deps` would do the same but also updating the
dependencies. You get the idea.
I was already using this for some time when I learned about Paver
which seems to take similar approach, so it looked like I might switch
to it, but in the end I decided against that. There are no problems w/
my custom-built solution, and I stuck to the principle that if it's
not broken don't fix it.
What I really like about Paver is the @task extension mechanism.
However, I don't get why you use `paver` command instead of the
classic distutils way? Another thing I don't understand is the actual
use distutils themselves -- it makes sense for generating docs etc but
seems to make site-specific deploy scripts awkward. In my case, if I
were using distutils, I imagine that the amount of code saved from
writing would be less than that written to hook into distutils.
After looking at sources of my module for this, I can tell you that
there's not much code that could be called boilerplate. But if you
look at the paste.deploy it's tons of code to load the config files,
tons of docs for it, almost an entire new language for configuration
that is also quite weak for what it should do. There's a requirement
to use separate app/middleware factories, setuptools entrypoints, more
documentation and all that for no justified reason as far as I can
see.
Anyway, sorry for the long message, I just wanted to illustrate that
distutils-like approach can work well and is easy to implement, extend
and use for serving and deploying webapps. Which, IMO, makes using
"config formats" for these purposes just plain wrong way to do it.
P.S. All this also reminds me of unit testing -- neither doctest nor
nose make it too simple to pick and choose what gets tested. It's a
PITA to get them to run only certain test / doctests and I blame it on
the "don't call us, we'll call you" design that comes with external
commands and "frameworkey" designs like that of unittest.
Kevin Dangoor
> What I really like about Paver is the @task extension mechanism.
> However, I don't get why you use `paver` command instead of the
> classic distutils way?
That's a good question. I actually can't remember if I asked myself
that specific question (it's been a while), but I can think of two
very minor reasons right now:
1. less typing "paver foo" rather than "python somescript.py foo" (and
yes, somescript.py could have a shebang line thus negating this
argument :)
2. a bit less code in the script (no __name__ == "__main__" stuff, and
I don't like the distutils "call a function" approach... a pavement
can be imported and nothing will happen automatically if you do)
> Another thing I don't understand is the actual
> use distutils themselves -- it makes sense for generating docs etc but
> seems to make site-specific deploy scripts awkward. In my case, if I
> were using distutils, I imagine that the amount of code saved from
> writing would be less than that written to hook into distutils.
I'm not sure I understand this one. If you use Paver, you can work
with distutils for free. There are certain things that
distutils/setuptools make easier than rolling your own.
Paver does some other things than handrolled scripts don't... You
could include path.py in your pythonpath, but Paver throws it in
because it's so darn useful. Paver also makes it easy to handle
command line options of a couple different forms.
Paver's not a lot of code, though, and doesn't seek to solve the
world's problems :)
Sergey Schetinin
>> use distutils themselves -- it makes sense for generating docs etc but
>> seems to make site-specific deploy scripts awkward. In my case, if I
>> were using distutils, I imagine that the amount of code saved from
>> writing would be less than that written to hook into distutils.
>
> I'm not sure I understand this one. If you use Paver, you can work
> with distutils for free. There are certain things that
> distutils/setuptools make easier than rolling your own.
Hm.. I guess my point is that if Paver is meant to automate
site-specific tasks (and deployment is such a task), then integration
with distutils doesn't seem to make sense. For tasks such as
non-trivial build processes, docs generation and uploading those docs
to server Paver seems to fit well and integration with distutils is
very welcome for those purposes. But I see deployment as an entirely
different and separate thing, so I'd say it should be a different
script too.
Paver does makes integration of that script with distutils easy, but I
don't see the purpose of doing that. I mean, writing the deploy script
with Paver / distutils is not much different that writing one the
straightforward way, isn't it? And the deployment script surely
doesn't need the build / sdist / etc commands.
Sure, writing your own option parsing would seem like repeating what
distutils already does, but in fact it's not -- it's basically one
line per option -- not much code to be saved. Anyway, using the @task
to define deployment commands would be really nice, but only if
distutils had nothing to do with it.
What I'm saying is that Paver is great for custom operations on
packages, but seems to be a bit mismatched for deploying webapps.
> Paver does some other things than handrolled scripts don't... You
> could include path.py in your pythonpath, but Paver throws it in
> because it's so darn useful. Paver also makes it easy to handle
> command line options of a couple different forms.
>
> Paver's not a lot of code, though, and doesn't seek to solve the
> world's problems :)
Sure :)
Kevin Dangoor
> Hm.. I guess my point is that if Paver is meant to automate
> site-specific tasks (and deployment is such a task), then integration
> with distutils doesn't seem to make sense. For tasks such as
> non-trivial build processes, docs generation and uploading those docs
> to server Paver seems to fit well and integration with distutils is
> very welcome for those purposes. But I see deployment as an entirely
> different and separate thing, so I'd say it should be a different
> script too.
I guess it depends on how you do deployment :)
Personally, I build an sdist of my code and deploy that tarball and
its requirements into a virtualenv using pip.
Jon Nelson
A concern - one which appears to be overlooked - is that quite a few
folks in highly siloed environments (think: large corporate
environments, etc...) are justifyably afraid of using what amounts to
executable code as configuration data. Under normal circumstances,
editing a configuration file might change or break the way some
software works but it does not normally execute arbitrary code under a
different user's privileges. This is a security concern and
code-as-configuration certainly falls under the "a bit
scary/undesirable" category. Whenever possible please keep code
(executable) and configuration (just data meant to be edited by a
human) apart.
Just my 2 cents.
--
Jon
Ian Bicking
I've thought about making significant changes to Paste Deploy before,
as I think application and middleware composition is a bit awkward,
and it doesn't really give enough space for particularly complicated
application configuration, all while making single-file full-site
setups fairly easy, even though that's not a very important thing to
optimize.
But as for the values... I like strings. I think it was a good
choice. In a very early version of Paste it used Python configuration
files, and things got out of hand even for simple applications. Which
isn't to say things *have* to get out of hand, but it didn't make me
that happy with the format.
Best practice for handling configuration -- and admittedly this isn't
documented much, my fault -- is something like this:
def make_app(global_conf, conf1=default, conf2=default, ...):
conf1 = asbool(conf1)
conf2 = int(conf2)
if isinstance(conf3, basestring):
conf3 = conf3.splitlines()
return MyApplication(conf1=conf1, conf2=conf2, conf3=conf3, ...)
class MyApplication(object):
def __init__(self, conf1, conf2, conf3):
...
That is, there is a function (make_app) that specifically converts all
values, checks the configuration (raising errors as appropriate), and
invokes some other application factory which only deals with Python
values. Of course, many people convert the values when they use them,
not at application instantiation time, and I don't really have a good
suggestion about that, but so it goes.
I personally find it easy to write these functions, and to put in
whatever logic I want in the function, as well as error checking. Of
course, I know the pattern and I know where the function lives, other
people might have other experiences.
For most applications, I think there is considerable value in keeping
the number of configuration parameters low. I know with some larger
libraries there are reasons to pierce the opaqueness of this
application factory (for instance, on deployment you really might want
to tweak how Beaker works, or how Mako does its caching, etc). But
generally I think there's value in trying to keep stuff simple, and
for many applications just a small number of parameters should be
sufficient, and so manually coercing non-string parameters isn't
really that hard. I do wonder if there are problems here with
deployment configuration (which is all Paste Deploy is intended for)
is being confused with framework configuration.
--
Ian Bicking | http://blog.ianbicking.org
Sergey Schetinin
The thing than annoyed me most about using PasteDeploy ini files was
how hard or impossible it was to make config files use any kind of
parameters. For example I wanted to use the same config for local and
deployed servers but with different databases.
With scripts it's a trivial matter, with paste deploy it's suddenly an
issue, so I'd say PasteDeploy is not simple but rather underpowered.
Ian Bicking
>> But generally I think there's value in trying to keep stuff simple, [...]
>
> The thing than annoyed me most about using PasteDeploy ini files was
> how hard or impossible it was to make config files use any kind of
> parameters. For example I wanted to use the same config for local and
> deployed servers but with different databases.
> With scripts it's a trivial matter, with paste deploy it's suddenly an
> issue, so I'd say PasteDeploy is not simple but rather underpowered.
Or perhaps underdocumented. To parameterize:
[app:generic]
use = egg:MyApp
param1 = value1
etc...
[app:live]
use = generic
database = mysql:some_live_database
[app:development]
use = generic
database = mysql:some_live_database
[app:testing]
use = generic
database = sqlite:memory
These can live in one file, or in several files, and you can select
them with paster serve.
Max Ischenko
Max
Fiid
Sorry to resurrect an old thread; and as a complete n00b. We're
investigating improving our development configuration setup.
Currently we are using a [composite:main] app configuration with a few
child applications, and we're passing each developer's config
variables in on the command line to paste and substituting in the .ini
file by using %(variable)s style substitution.
Is there an easy way to do something similar to the following, where
we can pass in just a username and use it to swap a number of
variables out throughout the file??
[globals:fiid]
http_port=9999
facebook_key=123ABC
facebook_secret=DEFGHI
[server:main]
use globals:%(username)s
[app:subapp]
use globals:%(username)s
other.configs=otherstuff
If we approach this from the override style shown below - is there a
way to substitute the name of the application when it is mapped in the
composite?
Please forgive me if I've missed a key building block here - I'm
definitely in the newbie category here :)
Thanks in advance,
Fiid.
Ian Bicking
>
> Hey,
>
> Sorry to resurrect an old thread; and as a complete n00b. We're
> investigating improving our development configuration setup.
>
> Currently we are using a [composite:main] app configuration with a few
> child applications, and we're passing each developer's config
> variables in on the command line to paste and substituting in the .ini
> file by using %(variable)s style substitution.
>
> Is there an easy way to do something similar to the following, where
> we can pass in just a username and use it to swap a number of
> variables out throughout the file??
>
> [globals:fiid]
> http_port=9999
> facebook_key=123ABC
> facebook_secret=DEFGHI
>
> [server:main]
> use globals:%(username)s
>
> [app:subapp]
> use globals:%(username)s
> other.configs=otherstuff
Well, I think you can do something like:
# in user1.ini
[DEFAULT]
key = value
[app:main]
use = default.ini
# in default.ini
[app:main]
use = egg:MyApp
foo = %(key)s
Then you will do paster serve user1.ini. Not quite what you asked
for, but I think the closest thing possible.
> If we approach this from the override style shown below - is there a
> way to substitute the name of the application when it is mapped in the
> composite?
You should be able to do substitution in the names, so you could have:
[composite:main]
use = egg:Paste#urlmap
/ = %(mainapp)s
I don't know if that would help that much -- you can take out or add
apps, just switch them up some.