Test environment woes

[Yang Zhang]

When attempting various things from our unit tests, such as calling
render_mako or calling pylons.url or looking up certain parameters in
pylons.config, we've run into problems with the environment not being
fully set up, such as "No object (name: tmpl_context or C) has been
registered for this thread".

So far, we've been reading through the pylons source and then hacking
our way around each problem, such as by calling directly into mako's
render method or manually calling pylons.config._push_object. Is there
a better way to get a non-crippled environment in our tests? Thanks.
--
Yang Zhang
http://yz.mit.edu/

[Hans Lellelid]

I'm sorry to add a "me too!" to this thread (instead of an answer),
but this is by far the most frustrating thing about Pylons (well,
Paste...). In fact it's one of the only really frustrating things
about Pylons. I hate the stacked object proxies. They are completely
useless for us and cause _so much_ pain for unit testing. Can't we
please just have thread-locals?! I can only assume from the sorry
state of testing support in Pylons/Paste that no one is doing it.
(And the WebTest stuff really only applies for the most basic
applications, which are not the ones we are using Pylons to build.)

Hans

[trogdor]

I believe this was discussed about a month ago. I personally wound up
applying the patch here:
http://pastie.org/984565
posted by Eugueny Kontsevoy.

It seems to be a matter of philosophy, but I'm currently lazy and
putting my config params into the .ini file and need to do things like
returning properties on objects.

-jack

http://groups.google.com/group/pylons-discuss/browse_thread/thread/e3db187e9c98f7e6

[Yang Zhang]

On Thu, Jul 1, 2010 at 5:56 PM, trogdor <konoh...@gmail.com> wrote:
> I believe this was discussed about a month ago. I personally wound up
> applying the patch here:
> http://pastie.org/984565
> posted by Eugueny Kontsevoy.

This only addresses the pylons.config issue; e.g., rendering via
pylons.templating is still having problems.

[Didip Kerabat]

I have similar problem with app_globals. I ended up moving everything in app_globals to meta module, which is available when running nosetests.

- Didip -

http://yz.mit.edu/

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.

[Wyatt Baldwin]

I use nose with the --with-pylons setting, then do this in
load_environment right before `return config` (I have the line
`testing = true` in my test.ini):

if asbool(config['testing']):
pylons.config._push_object(config)
url._push_object(URLGenerator(config['routes.map'], {}))

I haven't needed it, but you could add a line in there for
app_globals, too:

pylons.app_globals._push_object(config['pylons.app_globals'])

So, I guess that's not really "better", but it works and seems clean
enough to me.

[Hans Lellelid]

> I use nose with the --with-pylons setting, then do this in
> load_environment right before `return config` (I have the line
> `testing = true` in my test.ini):
>
>     if asbool(config['testing']):
>         pylons.config._push_object(config)
>         url._push_object(URLGenerator(config['routes.map'], {}))
>
> I haven't needed it, but you could add a line in there for
> app_globals, too:
>
>         pylons.app_globals._push_object(config['pylons.app_globals'])
>
> So, I guess that's not really "better", but it works and seems clean
> enough to me.

Yeah, I just spend the morning tracing through the Pylons/Paste code
so that I could fully understand what was going on. I think the
result is that I'll end up doing something very similar to the above.

I had some line of code like this in pylons.test nose plugin begin()
method:
pylons.config.current_conf().update(self.app.config)

But obviously the better approach would be to keep the changes in the
code I control :)

Granted, this doesn't address other similar objects (though
app_globals is addressed above). I think it's unfortunate that Pylons
(with the help of Paste) have chosen to make this so unobvious /
confusing. In my opinion this really discourages people from writing
tests. A developer on my team came to me the other day after porting
from CherryPy to Pylons with the frustration that none of the tests
would run due to pylons.config not containing needed values. She had
tried updating test.ini, development.ini, etc. but the code being
tested would never pull in the configuration files. Having just done
it, I can sympathize that to understand what's going on here takes a
good deal of time. Typical developers getting started with Pylons
aren't going to be able to figure out why it feels like the framework
is completely "broken" for testing. So they're either not going to
test their code -- or they'll go find another framework that works
better. (Unfortunately, and frustratingly, Paste seems to be the
popular choice in the non-Django full-featured-frameworks ... so I can
only assume that they're all going to get mired down in SOP confusion
& misdirection.)

I love Pylons for everything else ... so I'm really hoping that these
problems get the attention of the developers and they address this in
future releases.

Hans

[Ben Bangert]

On Jul 3, 2010, at 6:57 AM, Hans Lellelid wrote:

> But obviously the better approach would be to keep the changes in the
> code I control :)

This is exactly the approach I'm taking to fix testing up for Pylons 1.1. Here is what the tests/__init__.py looks like in the 1.1 branch for example (which will work fine for projects right now using Pylons):
"""Pylons application test package

This package assumes the Pylons environment is already loaded, such as
when this script is imported from the `nosetests --with-pylons=test.ini`
command.

This module initializes the application via ``websetup`` (`paster
setup-app`) and provides the base testing objects.
"""
from unittest import TestCase
import os
import sys

import pylons
from pylons.i18n.translation import _get_translator
from paste.deploy import loadapp
from pylons import url
from routes.util import URLGenerator
from webtest import TestApp

from {{package}}.config.environment import load_environment

__all__ = ['environ', 'url', 'TestController']

environ = {}
here_dir = os.path.dirname(os.path.abspath(__file__))
conf_dir = os.path.dirname(os.path.dirname(here_dir))

sys.path.insert(0, conf_dir)

class TestController(TestCase):
def __init__(self, *args, **kwargs):
wsgiapp = loadapp('config:test.ini', relative_to=conf_dir)
config = wsgiapp.config
pylons.app_globals._push_object(config['pylons.app_globals'])
pylons.config._push_object(config)

# Initialize a translator for tests that utilize i18n
translator = _get_translator(pylons.config.get('lang'))
pylons.translator._push_object(translator)

url._push_object(URLGenerator(config['routes.map'], environ))
self.app = TestApp(wsgiapp)
TestCase.__init__(self, *args, **kwargs)

Replace {{package}} with the name of your project. Note that this sets up the following globals:
config, app_globals, url, and translator (needed for set_lang/get_lang/i18n)

Using this, you should be able to remove the 'with-pylons' line from the setup.cfg, as the plugin shouldn't be needed (unless you're doing doc testing of your project...).

Making testing great is definitely a top priority for Pylons, and some of the other new stuff coming in 1.1 will lead to substantially easier unit testing in addition to the functional/integrated testing Pylons emphasizes with the usage of TestApp.

Cheers,
Ben

[Wyatt Baldwin]

The only thing I don't like about this approach is that you have to do
`app.config = config` at the end of `make_app`. It feels clunky for
one thing, but more importantly, it won't work in every case--for
example, when the app is wrapped in filtering middleware that's
specified in a config file; in this case the app returned from the
`loadapp` call is not the same app that's returned from `make_app`.

Possible solutions:

- At the end of load_environment, set config as a global in the
environment module, which can then be imported into the tests module.

- In load_environment, call into a test setup function

# environment.py
from my_pkg.tests import setup_test_environment
def load_environment(...):
# ...
if testing:
setup_test_environment(config)
return config

# my_pkg/tests/__init__.py
def setup_test_environment(app_config):
global config
config = app_config
# maybe setup the test environ, too:
environ[key] = app_config[key]

In both cases, the `config = wsgiapp.config` would be removed from
TestController.__init__.


> [...]

[Mike Orr]

On Sat, Jul 3, 2010 at 6:57 AM, Hans Lellelid <ha...@velum.net> wrote:
> I think it's unfortunate that Pylons
> (with the help of Paste) have chosen to make this so unobvious /
> confusing.  In my opinion this really discourages people from writing
> tests.

Yes. It keeps improving incrementally but there's always something
left out, and that ends up being what somebody needs for their tests.
The problem is that while custom test code can look someplace else for
the objects, anything it calls will expect things in their traditional
place. I generally use the SOPs only in controllers and templates, to
minimize my functions depending on them.

Ben wrote:
> fix testing up for Pylons 1.1

Hooray, hooray. That code still doesn't address 'tmpl_context' which
was the original question. Although 'tmpl_context' shouldn't really be
used outside a request. However, for the sake of completeness, we
should probably either create an empty context or push a dummy object
that says "don't use me outside of a request".

Didip wrote:
> I have similar problem with app_globals. I ended up moving everything in app_globals to meta module, which is available when running nosetests.

I did essentially the same thing, making a separate model object that
can be put on app_globals during runtime, but also instantiated
standalone for testing the object itself.

Wyatt wrote:
> pylons.app_globals._push_object(config['pylons.app_globals'])

I was reluctant to do that myself without an "official" recommendation
that this was the right method, but I suppose it works.

> (Unfortunately, and frustratingly, Paste seems to be the
> popular choice in the non-Django full-featured-frameworks ... so I can
> only assume that they're all going to get mired down in SOP confusion
> & misdirection.)

Paste is a collection of independent things (which Ian is gradually
moving out of Paste to their own packages). Most Paste frameworks do
not use SOPs. SOPs were created for Pylons to address the rare case of
multiple WSGI applications (or instances of the same application)
using shared globals in the same process. E.g., 'app_globals' is
specific to the application instance, not to the worker thread, so it
has to be switched as each instance is requested. Pylons chose to use
shared globals due to their popularity in CherryPy, but they do raise
this problem when combining applications. So if a framework avoids
shared globals, it doesn't need SOPs. In some frameworks, the context
is passed as action arguments, or instance attributes on the
controller, or global functions that find the context and return it.

--
Mike Orr <slugg...@gmail.com>

[Marius Gedminas]

On Sat, Jul 03, 2010 at 10:53:05AM -0700, Ben Bangert wrote:
> This is exactly the approach I'm taking to fix testing up for Pylons
> 1.1. Here is what the tests/__init__.py looks like in the 1.1 branch
> for example (which will work fine for projects right now using
> Pylons):

<...>

> Using this, you should be able to remove the 'with-pylons' line from
> the setup.cfg, as the plugin shouldn't be needed (unless you're doing
> doc testing of your project...).

This is inconsistent with the module docstring you just pasted:

> """Pylons application test package
>
> This package assumes the Pylons environment is already loaded, such as
> when this script is imported from the `nosetests --with-pylons=test.ini`
> command.

Marius Gedminas
--
When I die I want to go peacefully
in my sleep like my ol' Grand
Dad...not screaming in terror like
his passengers.
-- seen in a sig of Branden Robinson

[Ben Bangert]

On Jul 7, 2010, at 1:27 PM, Marius Gedminas wrote:

> This is inconsistent with the module docstring you just pasted:

Good catch, I'll remove that from the doc string as I updated the code and failed to read the doc string. :)

Cheers,
Ben