Debugging failed tests with pylons globals
Justin Tulloss
I realize this is similar to a couple other recent threads, but I did
not find what I was looking for in them.
I refer to the session, cache, and other pylons globals from time to
time in many controllers. When testing, I have sometimes enabled the
nose pdb flag to debug a test that's failing for an unknown reason.
Unfortunately, I cannot access the session, the cache, g, or h from
the debugger simply because "No object (name: Session) has been
registered for this thread". I'm resorting to print statements to
verify the global state while debugging.
Why I can't access the session doesn't really make sense to me. Why
shouldn't the testing framework be registered? Ignoring that for a
minute, how do I go about registering my debugger thread to access the
global? I have to admit that despite my digging through the source, I
do not have a good understanding of the StackedObjectProxy class and
am constantly doing battle with it.
Justin
Mike Orr
>I have to admit that despite my digging through the source, I
> do not have a good understanding of the StackedObjectProxy class and
> am constantly doing battle with it.
I don't think there's anybody who doesn't do battle with StackedObjectProxy.
--
Mike Orr <slugg...@gmail.com>
Alberto Valverde
Hahaha :) I fully agree. However, I also think they're the necessary evil
that makes Pylons so powerful, IMHO.
What I *love* about Pylons (opposed to TG1, CP2, Django...) is that
applications are not singletons but instances so you can have several of
them happily sharing the same process.
This has changed the way I design my webapps since I can now compose them
out of several smaller, easily testable, apps and route to them with yet
another app (thanks to the "stacked" part).
For example, first Pylons app was a control panel for an ISP which I
migrated from TG1 and Pylons helped me solve the issue of giving some
clients their own panel which they can sublicense very elegantly: just
give them a different instance of the whole app pointing to their database
schema with their own auth policy, etc...
StackedObjectProxy might not be the best implementation (although I can't
imagine how to improve it since there are so many edge cases involved...,
eg: those that have been tripping people over when testing) but it is
clear that something like this is needed as there are other packages
implementing similar functionality for other frameworks/systems (ie: PJE's
Contextual, and I'm sure Zope has their own SOP thingie somewhere).
A way I can think to fix the root of the problem is to remove all stacked
object proxies, bind the app globals to the app instance (whose lifecycle
is the same as the process) and pass the per-request stuff around as
function arguments. Perhaps the app instance as well, or at least bind it
to the controllers when they're instantiated. Obviously, this will make
the API much less attractive due to the param-passing nightmare (but will
make FPers very happy ;)
An intermediate solution, which I favor and have been experimenting with
lately, is to limit the SOPS to just one: pylons.app (the app instance
that holds database connections, cache, etc). Perhaps another SOP at
pylons.app.request could be useful so the request is easily accessible
from helpers intented to be used inside the context of a request.
pylons.c would then become an attribute of the controller (whose lifecycle
only spans one request as it does now) as well as the response, the
request, the session, etc... (so they don't have to be passed around as
controller action's arguments and routing vars can be passed instead).
Bonus points if the app that instantiated the controller is also bound to
the controller instance since then the controller can refrain form
accessing any SOP at all which simplifies the implementation of async apps
or apps that stream content which need access to the context from the WSGI
iterator.
What's the advantage of this: Much easier to understand the whole system
and to explain it: pylons.app is magic, it does this and that and you can
only access it while a request is in place or when you've mocked it for
tests.
Alberto
Alberto Valverde
> For example, first Pylons app was a control panel ...
I typoed a "my" (...first Pylons app...) out of that phrase, didn't want
to imply I wrote the first Pylons app... :D
Alberto
Mike Orr
> StackedObjectProxy might not be the best implementation (although I can't
> imagine how to improve it since there are so many edge cases involved...,
> eg: those that have been tripping people over when testing) but it is
> clear that something like this is needed as there are other packages
> implementing similar functionality for other frameworks/systems (ie: PJE's
> Contextual, and I'm sure Zope has their own SOP thingie somewhere).
>
> A way I can think to fix the root of the problem is to remove all stacked
> object proxies, bind the app globals to the app instance (whose lifecycle
> is the same as the process) and pass the per-request stuff around as
> function arguments. Perhaps the app instance as well, or at least bind it
> to the controllers when they're instantiated. Obviously, this will make
> the API much less attractive due to the param-passing nightmare (but will
> make FPers very happy ;)
>
> An intermediate solution, which I favor and have been experimenting with
> lately, is to limit the SOPS to just one: pylons.app (the app instance
> that holds database connections, cache, etc). Perhaps another SOP at
> pylons.app.request could be useful so the request is easily accessible
> from helpers intented to be used inside the context of a request.
The Pylons developers have been debating the value of SOPs repeatedly
since they were first implemented. The most straightforward and
failsafe solution would be to attach the Pylons globals to the
controller instance because it's instantiated for each request. But
that means they would have to be passed to every other routine that
uses them, which brings up nightmare scenarios of dozens of extra
arguments, or even passing arguments through a function that it
doesn't need but something it calls needs... or maybe something it
calls is reimplemented and suddenly it needs a variable it didn't
previously. We don't know what variety of routines should ideally
access the Pylons globals, but putting them at the module level allows
any routine that needs them to access them without argument passing.
(Of course, this makes the routines dependent on the Pylons
infrastructure, which is why I personally use the globals only in
controllers and templates unless I have to (such as a 'c' variable in
a validator).)
Different implementations have been proposed for the SOPs. They could
be turned into ordinary functions, which would signal to the caller
that some kind of special processing is going on. That's how Quixote
does it:
"import quixote; req = quixote.get_request()". But the non-function
syntax is too well established in Pylons to change it now.
The SOPs could be changed to ordinary thread-local variables. The
safety of this is a matter of debate, which is why it hasn't been
done. Theoretically it should be OK for 98% of deployment cases that
have only one Pylons application instance running in the process. No
aggregating two applications with paste.composite, or having multiple
instances of a blog app for each blog. After all, the SQLAlchmey
contextual session (the Session object, capital s) is a thread-local
and it's working fine in Pylons apps. And each request runs from
beginning to end in its thread. But when you add the complexity of
middleware, things get more complicated. Ian says SOPs are necessary
for thread-middleware integrity, and I barely understand what SOPs and
the Registry are so I don't want to second-guess him.
Having one SOP, pylons.app, containing all the Pylons globals has long
been suggested by me. The value could also be attached to the
controller instance -- one attribute instead of several. That would
be for users who object to getting request info from a module global,
and because I've heard there are some situations (some mod_wsgi
applications?) that can't use the SOPs.
I did come up with a couple situations in my latest app when I was
glad SOPs were SOPs, but like everything SOP-related I can't remember
precisely what it was, just the feeling I had.
A new SOP is on the way too: Routes 2 is planning pylons.url, to
replace url_for.
So SOPs are here to stay for now but nobody thinks they're ideal. The
trouble is, we don't know what would be better.
--
Mike Orr <slugg...@gmail.com>
Graham Dumpleton
On Jul 9, 6:22 am, "Mike Orr" <sluggos...@gmail.com> wrote:
> Having one SOP, pylons.app, containing all the Pylons globals has long
> been suggested by me. The value could also be attached to the
> controller instance -- one attribute instead of several. That would
> be for users who object to getting request info from a module global,
> and because I've heard there are some situations (somemod_wsgi
> applications?) that can't use the SOPs.
be a problem in mod_wsgi.
I keep seeing comments by various people suggesting that they don't
really understand how mod_wsgi works. I'd either like to understand
what the issue with SOPs is, or if it is a misunderstanding about how
mod_wsgi works, then correct that misunderstanding.
Thanks.
Graham
Ian Bicking
>
>
> On Jul 9, 6:22 am, "Mike Orr" <sluggos...@gmail.com> wrote:
>> Having one SOP, pylons.app, containing all the Pylons globals has long
>> been suggested by me. The value could also be attached to the
>> controller instance -- one attribute instead of several. That would
>> be for users who object to getting request info from a module global,
>> and because I've heard there are some situations (somemod_wsgi
>> applications?) that can't use the SOPs.
>
> Not understand what SOPs are all about, can you explain why they would
> be a problem in mod_wsgi.
The StackedObjectProxy -- it's a proxy facade over objects, like the
request, registered in a thread-local manner. I don't really see why
mod_wsgi would care. In non-threaded situations the stacking still
comes into play -- that is, you can register new requests that overlap
the old requests, then pop them off the stack. But again, it shouldn't
matter to mod_wsgi at all.
--
Ian Bicking : ia...@colorstudy.com : http://blog.ianbicking.org
Mike Orr
I heard somewhere that SOPs don't work with mod_wsgi, and that
request/response/session/c et al were also attached to the controller
instance because of that. I've never used mod_wsgi so I don't know if
it's true or not.
--
Mike Orr <slugg...@gmail.com>
Lawrence Oluyede
We use mod_wsgi + pylons in production and I've never noticed a
problem like that before. Maybe is a configuration problem?
--
Lawrence, stacktrace.it - oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
Graham Dumpleton
comment by you:
http://markmail.org/message/wz4yeooo3v3l4ngf
Overall there is really nothing special about how mod_wsgi works that
would invalidate any specific Python programming techniques. As far as
a WSGI application goes, how threading is used should be no different
to something like paste server.
Since you don't know the reasons why SOPs may be a problem, I'd very
much suggest there probably isn't a problem, especially with Ian not
really understanding how there would be either.
Anyone else know any actual detail about this suggestion that SOPs and
mod_wsgi may be a problem.
Graham
Mike Orr
<Graham.D...@gmail.com> wrote:
> The only reference I can find to SOP and mod_wsgi together is a prior
> comment by you:
I tried to find the original message earlier too but I couldn't. So
maybe I misunderstood it, or maybe the original person was wrong about
it.
--
Mike Orr <slugg...@gmail.com>
Mike Orr
SOPs and you added self.request et al to the controllers because of
it? Or did I dream the whole thing?
--
Mike Orr <slugg...@gmail.com>
Ben Bangert
> Ben, do you remember? Wasn't there an environment that couldn't use
> SOPs and you added self.request et al to the controllers because of
> it? Or did I dream the whole thing?
Only async environments can't use SOP's, though since eventlet will
monkey patch thread-locals, even under those conditions the SOP's
work. Unless twisted monkey patches thread locals, they wouldn't work
under that async environment.
Cheers,
Ben
Graham Dumpleton
thread locals etc on WEB-SIG recently:
http://groups.google.com/group/python-web-sig/browse_frm/thread/e1152aafde377aa2?hl=en
Graham
> 3KDownload
Justin Tulloss
> So SOPs are here to stay for now but nobody thinks they're ideal. The
> trouble is, we don't know what would be better.
That might be a good time to look at what might be a better option. If
98% of cases are served by threadlocals, that might be the way to go.
It's rarely wise to introduce this kind of complexity for 2% of the
use cases.
Of course, there's also the possibility that SOPs aren't that
complicated and the problem is really that nobody knows how to deal
with them. The documentation for SOPs is somewhat lackluster; the API
is described in the paste docs, but not the motivation or concepts.
In the end, there is no technical reason why a program should not be
able to access an SOP from anywhere. What I would like to see is a way
of registering a thread to access the SOP that another thread is
refering to. This obviously isn't safe, and shouldn't be the default,
but for testing it's quite valuable to be able to access SOPs that
were registered to a thread that is not your own.
Justin
Alberto Valverde
>
>> So SOPs are here to stay for now but nobody thinks they're ideal. The
>> trouble is, we don't know what would be better.
>>
>
> I understand that pylons 1.0 will be breaking backwards compatibility.
> That might be a good time to look at what might be a better option. If
> 98% of cases are served by threadlocals, that might be the way to go.
> It's rarely wise to introduce this kind of complexity for 2% of the
> use cases.
>
pylons apps that dynamically instantiate other pylons apps in a
controller action and dispatch to them. Given that the "complexity" is
already implemented I think it's wiser to leave it there :)
Mostly because I believe that a simpler threadlocal won't make
debugging/testing any easier IMHO since you still have to populate them
to access them outside of a request.
I honestly don't find them *that* hard to understand, and I'm not that
bright either. Granted, I've fought them a lot and have implemented
several libraries (one of them is open source, ToscaWidgets) that use
them so I probably know more about them than the average Pylons user.
I believe some good docs explaining them, even better if they are
written or reviewed by whoever designed/implemented them, would be
enough to help understand them better.
Alberto
Jonathan Vanasco
do with how and where items are instantiated and stored.
I think if the load order were different ( i'm not going to make any
suggestions ;) ), and how we get at them, then many of the end-user
issues could disappear.
Justin Tulloss
On Jul 11, 8:23 am, Alberto Valverde <albe...@toscat.net> wrote:
>
> I honestly don't find them *that* hard to understand, and I'm not that
> bright either. Granted, I've fought them a lot and have implemented
> several libraries (one of them is open source, ToscaWidgets) that use
> them so I probably know more about them than the average Pylons user.
their behavior is. The documentation describes how to register a new
one, but not how to affect the values that are in there. In addition,
it's very difficult to inspect the object that an SOP refers to since
it obfuscates the object's __dict__. In the end, they're just a little
too much magic for me.
> I believe some good docs explaining them, even better if they are
> written or reviewed by whoever designed/implemented them, would be
> enough to help understand them better.
Justin
Alberto Valverde
> (....)
> In addition,
> it's very difficult to inspect the object that an SOP refers to since
> it obfuscates the object's __dict__.
This trick can help:
req = pylons.request._current_obj() # get current request object wrapped
by SOP
print dir(req)
Again, it's just a matter of documentation IMHO. The alternative would
be to make Pylons apps singletons like TurboGears 1, CherryPy 2,
Django... are and that would solve the root of the "problem". A stacked
proxy is only needed when the possibility exists of a WSGI
app/middleware dispatching to another application that makes use of the
same SOPs (a pylons app that calls another pylons app, or an app that
uses "pseudo-middleware" [1] X that calls an app that also uses X).
However, this alternative would be a *huge* step backwards IMO since it
would make this pseudo-middleware less interoperable After all, from
what I understand, one of Pylons design goals is to let you build your
app from all this tiny wsgi pieces that play well together.
Alberto
[1] By pseudo WSGI middleware I mean middleware that is not really WSGI
middleware since middleware should be transparent to the application it
wraps, that is, if it's missing from the stack then the application
could not care less about it. These are more like a fancy function
decorator which usually sets setup some context for some library
downstream that needs it, for example: toscawidgets, beaker, etc...
Mike Orr
>
> Justin Tulloss wrote:
>> (....)
>> In addition,
>> it's very difficult to inspect the object that an SOP refers to since
>> it obfuscates the object's __dict__.
>
> This trick can help:
> req = pylons.request._current_obj() # get current request object wrapped
> by SOP
> print dir(req)
>
> Again, it's just a matter of documentation IMHO.
It's not "just" documentation. Fundamentally it behaves differently
from a normal Python object, so you have to memorize not only the
._current_obj() syntax but also which methods require it and which
don't. So it's not a very good proxy. But at least it follows the
80/20 rule of proxying ordinary access if you know what you're looking
for. But it falls apart when you don't know what you're looking for,
which is why you typed dir(request). You get a bunch of unfamiliar
methods instead of what you were expecting. So you call help(request)
and find out it's a StackedObjectProxy... but nowhere does it say to
call request._current_obj() to get the real object.
Somehow help(config) prints help for the real object while
help(request) doesn't. I wonder why that is.
> [1] By pseudo WSGI middleware I mean middleware that is not really WSGI
> middleware since middleware should be transparent to the application it
> wraps, that is, if it's missing from the stack then the application
> could not care less about it. These are more like a fancy function
> decorator which usually sets setup some context for some library
> downstream that needs it, for example: toscawidgets, beaker, etc...
In other words, the most useful middleware. It's much too late to
call these "framework component objects" when thousands of people have
learned that a middleware is anything that uses the WSGI protocol on
both ends. It's hard even to think of a "true" middleware that would
be useful, just the gzip one and TransLogger. "Pseudo" middleware
provides a generic service to any application, and that should be
lauded, not disparaged as the proponents of true vs false middleware
tend to do.
--
Mike Orr <slugg...@gmail.com>
Justin Tulloss
On Jul 13, 8:16 am, Alberto Valverde <albe...@toscat.net> wrote:
> JustinTullosswrote:
> > In addition,
> > it's very difficult to inspect the object that an SOP refers to since
> > it obfuscates the object's __dict__.
>
> This trick can help:
> req = pylons.request._current_obj() # get current request object wrapped
> by SOP
> print dir(req)
I found what I was really looking for in the code for setting up the
test app when you execute "paster shell". I adapted it for the my
TestController as follows:
class TestController(TestCase):
def __init__(self):
self.app = paste.fixture.TestApp(wsgiapp)
# Query the test app to setup the environment
tresponse = self.app.get('/_test_vars')
request_id = int(tresponse.body)
# Disable restoration during test_app requests
self.app.pre_request_hook = lambda self: \
paste.registry.restorer.restoration_end()
self.app.post_request_hook = lambda self: \
paste.registry.restorer.restoration_begin(request_id)
self.request_id = request_id
def setUp(self):
paste.registry.restorer.restoration_begin(self.request_id)
def tearDown(self):
paste.registry.restorer.restoration_end()
This allows you to access the pylons globals outside of your self.app
requests. You can inspect them and put things in there to your heart's
content. Be careful though, the SOPs won't get cleared between
consecutive requests in the same test.
Justin