Accessing app config files outside of requests

[Hamid Elaosta]

Hi all,

Apologies, I'm both a Python and Cherrypy newbie (but otherwise an experiences developer)

I'm looking for a tidy way to access app specific config files from within my classes but outside of requests:

The answer here provides a solution but it seems messy, surely there is a better way?

http://stackoverflow.com/questions/2929790/how-do-you-access-config-outside-of-a-request-in-cherrypy

With the following in my "main" class:

application = cherrypy.tree.mount(MainApp.SomeClass(), '/somemountpoint', app_conf)

This is the most upvoted solution (albeit only 3 votes at present):

application = cherrypy.tree.apps['']
host = application.config['database']['host']

The problem, for me, is that this doesn't work for a start, the key [''] (blank) is not valid, and the key "/somemountpoint" is not only empty of my custom configs, but even were it not empty, would cause the class MainApp.SomeClass() to be dependent on the mointpoint specified by the parent that mounted it. This is clearly bad design to do.

Does cherrypy present the specific config file for that mountpoint from without the mounted app anywhere? What am I missing, or do I need to use other methods of getting access to the config for that specific app?

[David Bolen]

Hamid Elaosta <alias....@gmail.com> writes:

> I'm looking for a tidy way to access app specific config files from within
> my classes but outside of requests:

(...)

> application = cherrypy.tree.apps['']
> host = application.config['database']['host']
>
> The problem, for me, is that this doesn't work for a start, the key ['']
> (blank) is not valid, and the key "/somemountpoint" is not only empty of my
> custom configs,

The key '' was used in the Stack Overflow case since the application
was mounted at /, which in turn was stored under '' in the tree apps
dictionary (as Tree.mount() removes trailing slashes). So you are
correct that in your case you'd want a key of "/somemountpoint"
instead. If in doubt, just dump tree.apps after initialization.

I can't say why the config was empty on that application object.
That's the application object servicing that mount point, so assuming
you supplied an application configuration to mount(), it should be
available in the application's config attribute.

Or, put another way, any use of "cherrypy.request.app.config" within a
request should be no different than cherrypy.tree.apps[mountpoint].config
at any other point, assuming the app was mounted at mountpoint.

If that's not true in your case, it might be helpful to see how you
are specifying your configuration - perhaps what you consider
application configuration is, in reality, global configuration (in
which case you can use cherrypy.config), or handler specific
configuration, in which case you can probably use find_config() on
your application object.

> but even were it not empty, would cause the class
> MainApp.SomeClass() to be dependent on the mointpoint specified by the
> parent that mounted it. This is clearly bad design to do.

Not sure if I see the "clearly bad" part. Outside of a CherryPy
request (which knows the application by serving URL), you have to have
some criteria by which the proper application can be found, for which
a mount point seems as good to me as anything else. Or, put another
way, without an active request or mount point, how would you propose
to identify the "right" application - what criteria would you use?

If it were me, depending on what those criteria were, I'd probably
just track the application objects myself outside of CherryPy, along
with any necessary state or identifying information, and then use my
own information to locate the application when needed. No dependency
on CherryPy needed at all. This could even be as simple as a single
module level global variable or dictionary.

Of course, that's essentially what cherrypy.tree is - a package-level
singleton to translate a URL to an application. In this case its
designed for URL-based lookups, so yes, anything using it will need to
have some knowledge of mount points.

But perhaps you're creating a new set of object instances for a given
application. In that case, I might have my application object pass
itself into (or set an attribute on) each object it creates. That
provides tight coupling of those objects with their creating
application, permitting each object belonging to that application to
find its way back to its creating application without any help. Of
course you have to be sure to have a 1:1 relationship between each
object instance and a single application.

It might help to see more of your design and object hierarchy to
better understand the scenario you are trying to solve. I'm assuming
you have classes used both to serve CherryPy request but also getting
called from some other top level code. It's most likely in the higher
level code that you'd want to manage the application tracking if you
don't want your object instances to have any coupling with information
that is needed to find the application object.

-- David

[Hamid Elaosta]

Hi Dave,

Thank you for the thorough reply.

I don't have my code to hand but essentially, I'm planning to create a REST API based application/program.

MyProgram/
|
+-MyProgram.py
|
+-ClassSetupDatabase.py
|
+-ClassDoSomeWork.py
|
+-API/+
     +__init__.py    
     |
     +-v1/+
     |   |
     |   +EndpointSetupDatabase.py
     |   +EndpointDoSomeWork.py
     |   +Endpoint1.py  
     |   +Endpoint2.py
     |   +v1config.conf
     |  
     |
     +-v2/
     |
     :
     +-etc/

So, my v1 API should have, in its config the database filename, lets say for an SQlite DB, that I want to be a global variable within each of the endpoints that they can access to connect to the DB.

At the moment, they're all anonymously mounted in __init_.py or somewhere:

cherrypy.tree.mount(Endpoint1.EndPoint1(),"/endpoint1","v1config.conf")
cherrypy.tree.mount(Endpoint2.Endpoint2(),"/endpoint2", "v1config.conf")
etc

Perhaps I should be holding on to references of Endpoint1.Endpoint1() in __init__ and importing v1, v2, etc. If I wanted to mount a specific endpoint with its own config file though, I don't see a way to reference the configs from inside that endpoint without accessing the parent and asking it for my own config, or I have to set the app on the class after mounting?

[Hamid Elaosta]

Apologies, I took liberties with your name there David and only noticed after posting.

On Monday, 30 March 2015 15:54:19 UTC+1, Hamid Elaosta wrote:

Hi Dave,

[David Bolen]

Hamid Elaosta <alias....@gmail.com> writes:

(...)

> +-API/+
> +__init__.py
> |
> +-v1/+
> | |
> | +EndpointSetupDatabase.py
> | +EndpointDoSomeWork.py
> | +Endpoint1.py
> | +Endpoint2.py
> | +v1config.conf

(...)

> So, my v1 API should have, in its config the database filename, lets say
> for an SQlite DB, that I want to be a global variable within each of the
> endpoints that they can access to connect to the DB.

I guess what's still not clear to me is when - and more importantly
how - your objects are being called from outside of a CherryPy request
cycle.

> At the moment, they're all anonymously mounted in __init_.py or somewhere:
>
> cherrypy.tree.mount(Endpoint1.EndPoint1(),"/endpoint1","v1config.conf")
> cherrypy.tree.mount(Endpoint2.Endpoint2(),"/endpoint2", "v1config.conf")
> etc

For completeness, while you are handing these instances off to
CherryPy here and not maintaining any references yourself, they're
still easy to locate, as cherrypy.tree.apps maintains the mapping
between endpoint and Application instance (that CherryPy creates) and
your endpoints.

So after those two calls, the cherrypy.tree.apps dictionary will have
keys at /endpoint1 and /endpoint2, each of which points to an
Application object instance where the root attribute is your endpoint
instance.

(You can also choose to mount already created Application instances)

>
> Perhaps I should be holding on to references of Endpoint1.Endpoint1() in
> __init__ and importing v1, v2, etc.

(...)

Well, if you're not already holding such references, I'm unclear how
anything other than CherryPy is calling those endpoints? As written
above, the only way that I can see to get into that object instance is
through CherryPy as part of a request, so cherrypy.request.app should
always be valid.

> If I wanted to mount a specific
> endpoint with its own config file though, I don't see a way to reference
> the configs from inside that endpoint without accessing the parent and
> asking it for my own config, or I have to set the app on the class after
> mounting?

I'm not positive what "parent" references here, but yes, I see no way
around tracking your own reference to the application object - or its
configuration - if you don't want to depend on a cherrypy reference
(either in cherrypy.request or cherrypy.tree). Whether you store your
own reference within your endpoints or external and pass it to the
endpoints when needed is a design question.

So yes, assuming some other code gets references to those EndPoint1/2
instances, an alternate means is needed to find the enclosing
application. I can think of several ways to do it, such as:

1. Call some sort of setup method or set an attribute on the endpoint
to configuration an application reference at start up after which
the endpoint instance will always have its own reference. This is
the "set the app on the class" you suggest above.
2. Make the endpoint objects themselves Application subclasses in which
case the config dictionary will be an instance attribute "for free".
3. Having the calling code store and then pass the application object
(or whatever specific configuration data is needed) into the
methods called on the end point. This can be done either by
tracking the applications at setup, or as previously discussed,
using cherrypy.tree to locate the endpoint, and in turn the
application at call time.
4. Pull your endpoint configuration outside of CherryPy and just make
it part of the endpoint object construction itself. E.g., if the
API is the only consumer of the database configuration, it's not
clear it must be in the CherryPy configuration to start with.

Any of these can work (and probably other options as well).

There's so many permutations of how one might handle this at a design
level that I'm not really sure even how to start. But here's one
possible sample of the first option, configuring the endpoints with
their resulting application after mounting. This keeps application
setup close to the CherryPy setup, but afterwards it's just endpoint
instance data so doesn't need to be tracked by other callers. I've
included some debugging prints so you can see the configuration
dictionaries at various points.

Given the following module:

- - - - - - - - - - - - - - - - - - - - - - - - -

import cherrypy
import time

def dump_cpy_configs():
print 'CherryPy Configs:'
print ' Global:', cherrypy.config
print ' App:', cherrypy.request.app.config
print ' Request:', cherrypy.request.config


class Nested(object):

@cherrypy.expose
def entry(self):
print '[Nested.entry]'
dump_cpy_configs()

def tick(self, app):
print '[Nested.tick]'
print ' Global:', cherrypy.config
print ' App:', app.config


class EndPoint(object):

app = None

def __init__(self, name):
self.name = name
self.nested = Nested()

@cherrypy.expose
def index(self):
print '[%s.index]' % self.name
dump_cpy_configs()

def tick(self):
print '[%s.tick]' % self.name
# cherrypy.request is not valid here
print ' Global:', cherrypy.config
print ' App:', self.app.config
self.nested.tick(self.app)


if __name__ == "__main__":

cherrypy.config.update('global.conf')
cherrypy.tree.mount(EndPoint('EndPoint1'), '/endpoint1', 'v1config.conf')
cherrypy.tree.mount(EndPoint('EndPoint2'), '/endpoint2', 'v1config.conf')

# Configure each endpoint with its application
for app in cherrypy.tree.apps.values():
app.root.app = app

# Call endpoints outside of CherryPy request
last_tick = time.time()
def tick():
global last_tick
# Execute application ticks at 5s interval
cur_tick = time.time()
if cur_tick - last_tick > 5:
print 'TICK:', time.ctime(cur_tick)
last_tick = cur_tick
for app in cherrypy.tree.apps.values():
app.root.tick()
cherrypy.engine.subscribe('main', tick)

# CherryPy mainloop
cherrypy.engine.start()
cherrypy.engine.block()

- - - - - - - - - - - - - - - - - - - - - - - - -

and the following global configuration file global.conf:

- - - - - - - - - - - - - - - - - - - - - - - - -
[global]
server.socket_port = 9000
global_setting = 'global'
- - - - - - - - - - - - - - - - - - - - - - - - -

and the following application configuration file v1config.conf:

- - - - - - - - - - - - - - - - - - - - - - - - -

[database]
host='somehost'
port=5432

[/]
app_setting='app value'

[/nested]
nested_setting='value'
- - - - - - - - - - - - - - - - - - - - - - - - -

When run, this sets up URLs for:

http://localhost:9000/endpoint1/
http://localhost:9000/endpoint1/nested/entry/
http://localhost:9000/endpoint2/
http://localhost:9000/endpoint2/nested/entry/

where the /endpoint{1,2} trees are handled by independent EndPoint
instances, each with their own Nested instances.

In terms of configuration, the [global] stanza from global.conf will
always be available in cherrypy.config. (You could keep the global
stanza in the same v1config.conf if you wanted). All stanzas from
v1config.conf will be available in your application configuration
(cherrypy.app.config), while the URL stanzas (/ and /nested) will be
in the request config (cherrypy.request.config) merged with the global
config when serving a request beneath those URLs.

Since I don't know how you expect to call your endpoints independent
of CherryPy I just used an engine subscription to call a non-published
method every 5 seconds. I also opted to take advantage of the
CherryPy tree to get access to all of the applications/endpoints both
when configuring them with their own application reference and later
to call the tick methods.

If you retrieve one of the URLs above, the appropriate published
method will run, using cherrypy.request to get to the application and
configuration.

In the background, the tick entry point will be called every 5s with
access to the global configuration (since, well, it's global and works
everywhere) but using its own application reference for the
application configuration. You might also choose to configure your
endpoints with the app.config dictionary directly rather than having
to reference the application object at all.

From within a non-CherryPy entry point, if a nested object is being
called, the application needs to be supplied (such as I do here), or
you could have the endpoints configure the nested objects just as they
themselves are being configured. Sort of depends just what
configuration information the nested objects need.

Note that I believe referencing cherrypy.request from within tick()
will still technically have a value, but it'll point to an arbitrary
application.


To modify this for option 2, where the endpoint itself is also the
application object just means changing the EndPoint class to be an
Application sub-class and then to instantiate it differently (since
the application class is supplied with its mount point). So for
example:

- - - - - - - - - - - - - - - - - - - - - - - - -

# Application version of EndPoint
class EndPointApp(cherrypy.Application):

def __init__(self, name, script_name, config):
# Supply myself as the root object
cherrypy.Application.__init__(self, self, script_name, config)
self.name = name
self.nested = Nested()

@cherrypy.expose
def index(self):
print '[%s.index]' % self.name
dump_cpy_configs()

def tick(self):
print '[%s.tick]' % self.name
# cherrypy.request is not valid here
print ' Global:', cherrypy.config
print ' App:', self.config
self.nested.tick(self)


# Replace existing mount, and endpoint configuration with:

cherrypy.tree.mount(EndPointApp('EndPoint1', '/endpoint1', 'v1config.conf'))
cherrypy.tree.mount(EndPointApp('EndPoint2', '/endpoint2', 'v1config.conf'))

- - - - - - - - - - - - - - - - - - - - - - - - -

In this case the application configuration becomes self.config in the
endpoint instance since it itself is also the application. Of course,
this approach requires you be willing to have the endpoint be an
application-subclass (as opposed to the default behavior of being
referenced by a separate Application instance).


Hopefully this will at least give you some thoughts on how to approach
things or some code to play with further.

-- David

[Hamid Elaosta]

Thanks Again.

There is a lot to take in there so I'll go through it all and test out those snippets when I'm home. Just to respond quickly, one thing that seems apparent to me from your responses is that I may have mis-interpreted the purpose and scope of CherryPy.

If I'm not misunderstanding, it seems CherryPy configs should really be used for request specific configuration.

From my example, my intention was not to access the CherryPy endpoints from other locations, but, for example, to have, DB_FILENAME in Endpoint1 as a field on the class, that is set in the __init__ (again if I don't misunderstand, this equates roughly to a constructor in python?) of the class. This "field" (global variable?) should then be accessible from within requests in order to make their database connections (in this case).

It was my hope that when I mount and application with a config per "endpoint", code within the mounted application would have access to that configuration, but outside of requests (i.e. in the "constructor).

I see many references to the "index" request in documentation, as well as in your examples, is this equivalent to a constructor in CherryPy? Is index called at the start of all requests that, by my logic, are nested to it, or only when index itself at the top level is requested? If the former were the case, I can see how I might obtain the config there and set my "field".

[David Bolen]

Hamid Elaosta <alias....@gmail.com> writes:

> If I'm not misunderstanding, it seems CherryPy configs should really be
> used for request specific configuration.

While there's nothing inherently wrong with using CherryPy's config, I
was merely suggesting out that if we're talking about configuration data
that is consumed solely by the endpoint instance, and not benefiting
from any of CherryPy's configuration features, then you may be causing
some of your own headache by choosing to store it in the application
configuration.

That is, your original problem of locating a CherryPy application object
from outside a request is a consequence of storing the configuration in
CherryPy's application.

Of course, given this next paragraph...

> From my example, my intention was not to access the CherryPy endpoints from
> other locations, but, for example, to have, DB_FILENAME in Endpoint1 as a
> field on the class, that is set in the __init__ (again if I don't
> misunderstand, this equates roughly to a constructor in python?) of the
> class. This "field" (global variable?) should then be accessible from
> within requests in order to make their database connections (in this case).

I've now become confused, as this would seem to invalidate your original
question (locating an application object outside of a request). If
you're not accessing the endpoints from other locations, and the field
needs to be accessible from within requests, that seems to be a perfect
match for CherryPy's configuration, and your original problem would seem
to go away entirely. Just use cherrypy.request.app.config in the
request and you're done.

For the sake of argument, though, let's say you do opt to bypass
CherryPy's configuration. You might modify your earlier setup code as:

cfg = somehow_load_configuration()
cherrypy.tree.mount(Endpoint1.EndPoint1(cfg), "/endpoint1", "v1config.conf")
cherrypy.tree.mount(Endpoint2.EndPoint2(cfg), "/endpoint2", "v1config.conf")

which passes the configuration into each EndPoint at instantiation time,
so they can store whatever they need locally. Then, whenever any method on
those objects runs it'll have access to the necessary configuration data
whether it was called from a CherryPy request or any other code.

> It was my hope that when I mount and application with a config per
> "endpoint", code within the mounted application would have access to that
> configuration, but outside of requests (i.e. in the "constructor).

Ah, ok, so now this might be the key issue - is "outside of requests"
only within respect to __init__? That's not at all what I envisioned
when I read that phrase. True, creating the initial object is
technically outside of a request, but I certainly assumed you mean
post-construction.

So, this is more a plain programming question than anything CherryPy
specific. Given your original example setup:

cherrypy.tree.mount(Endpoint1.EndPoint1(), "/endpoint1", "v1config.conf")

The parsing of the v1config.conf configuration file is going to take
place in the cherrypy.tree.mount() method. It's impossible for any of
that configuration information to be available when EndPoint1 is being
instantiated (e.g., during __init__), because the EndPoint1 object is
created before cherrypy.tree.mount is called. The resulting Endpoint1
instance is then passed into tree.mount as the first parameter. The
configuration hasn't even been parsed when EndPoint1 is constructed.

If you need certain data when EndPoint1 is being instantiated, you must
pass it in as a parameter (like the example above), or have it globally
accessible in advance of creating the object. (Note that the global
cherrypy.config can be used for this if you wanted)

I can't help but think there's some missing information, since I can't
see a need for EndPoint1 to have access to the config from within its
__init__. You say above you only need to access it from within
requests. Which would seem to be a perfect match for how CherryPy's
configuration naturally works.

In other words, with the above call to tree.mount, whenever EndPoint1 is
handling a request, you can use cherrypy.config, cherrypy.request.config,
or cherrypy.request.app.config to access configuration information, just
as in my sample, and it would seem to do exactly what you need, without
ever needing to see that configuration in EndPoint1.__init__.

But perhaps still part of your design that you haven't covered yet that
would explain what EndPoint1.__init__() is doing with the database
configuration information.

> I see many references to the "index" request in documentation, as well as
> in your examples, is this equivalent to a constructor in CherryPy? Is index
> called at the start of all requests that, by my logic, are nested to it, or
> only when index itself at the top level is requested? If the former were
> the case, I can see how I might obtain the config there and set my "field".

Index is just a method like any other on the class - never called
automatically, just in response to a request. Sorry for any confusion.
It's only special in that CherryPy will look for that method as a
default handler, much like web servers will automatically look for
index.html in a directory.

I'd suggest running the code, making test requests and observing the
debug print output (or adding your own) to aid in understanding. You
can strip out the timed calls bit since it sounds like you don't
actually have independent (non-request) code using your endpoints.

-- David