need a namespace package guru....

[Michael Bayer]

I think I released the dogpile.cache stuff incorrectly, actually installing the packages it seems like I got the namespace package stuff wrong. I have the "pkg_resources.declare_namespace" directive in the dogpile/__init__.py of dogpile.cache but not of dogpile, and it appears that this needs to be exactly the opposite.

Can someone help me set up dogpile and dogpile.cache correctly? Here's a paste of what *seems* to work:

http://paste.pocoo.org/show/581532/

the questions I have are:

1. Is it OK if I have "__version__" and a few other things in dogpile/__init__.py of the root project ? this seems to work, as "dogpile" is always imported first, but the docs at http://packages.python.org/distribute/setuptools.html#namespace-packages might suggest otherwise:

"You must NOT include any other code and data in a namespace package’s __init__.py. Even though it may appear to work during development, or when projects are installed as .egg files, it will not work when the projects are installed using “system” packaging tools – in such cases the __init__.py files will not be installed, let alone executed."

2. since "dogpile" is always imported first it seems like I don't need anything in the dogpile/__init__.py of that of dogpile.cache ?

3. or is "dogpile" not always imported first in this scenario ? how would I see that ?

[Michael Bayer]

it seems like my entire concept of having "root" and "root.subpackage", the way "sqlalchemy" and "sqlalchemy.orm" do, is just entirely wrong from a "namespace package" point of view. While I think what I'm doing does actually work, this is not what anyone had in mind with "namespace packages".

I'm tempted to just merge these two things together. But I think I'll just move "dogpile" into "dogpile.core" and just get in line with everyone else.

> --
> You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
> To post to this group, send email to pylons-...@googlegroups.com.
> To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
>

[Mike Orr]

I don't know much about namespace packages, but I'd avoid them as they
lead to occasional problems on some systems. For instance, I can't
install Pyramid without --no-site-packages on some versions of Ubuntu,
because the "zope." namespace is split between the system
site-packages and the virtualenv's, and Python apparently can't handle
this. Py2exe is not Setuptools aware yet, so it may not be compatible
with namespace packages. (So far I haven't had a problem with Pylons
and Py2exe -- including SQLAlchemy -- as long as all imports are
listed explicitly rather than using Routes "autoload", but it may be
when I switch an application to Pyramid.)

--
Mike Orr <slugg...@gmail.com>

[Michael Bayer]

that seems odd, do you have that same issue even if you use pip ? namespace packages are supported without setuptools as it falls back onto pkgutil.extend_path.

also --no-site-packages is the default now and not using it is a little crazy...

[Chris Lambacher]

On Sun, Apr 15, 2012 at 3:01 PM, Mike Orr <slugg...@gmail.com> wrote:

Py2exe is not Setuptools aware yet, so it may not be compatible
with namespace packages. (So far I haven't had a problem with Pylons
and Py2exe -- including SQLAlchemy -- as long as all imports are
listed explicitly rather than using Routes "autoload", but it may be
when I switch an application to Pyramid.)

 

I have been successful with Pyramid and Py2Exe, the only problem I recall is that config.scan() does not find @view_config decorators in files included in the library.zip file. I think this is related to there not being .pyc files in the zip file. I only had a couple of views for that app so I just used config.add_view(). I may have also be some fiddling to get egg-info directories to work so that entry_points work with paster config files. If someone is interested in the details of how I got it to work they should post on a separate thread or find me on IRC as lambacck)

-Chris

--
Christopher Lambacher
ch...@kateandchris.net

[Wolfgang Schnerring]

* Michael Bayer <mik...@zzzcomputing.com> [2012-04-14 22:08]:

> 1. Is it OK if I have "__version__" and a few other things in
> dogpile/__init__.py of the root project ? this seems to work, as
> "dogpile" is always imported first, but the docs at
> http://packages.python.org/distribute/setuptools.html#namespace-packages
> might suggest otherwise:
>
> "You must NOT include any other code and data in a namespace package’s
> __init__.py. Even though it may appear to work during development, or
> when projects are installed as .egg files, it will not work when the
> projects are installed using “system” packaging tools – in such cases
> the __init__.py files will not be installed, let alone executed."

IIRC, the issue is that it is implementation dependent (read: totally
random) *which* dogplie/__init__.py will be used, that of egg "dogpile"
or that of egg "dogpile.cache".

Thus, unless all your dogpile/__init__.py's are exactly the same (which
is easiest when they're empty ;), you can't be sure which one will be
used.

So, yes, while namespace packages have the huge benefit of actually
working (most of the time), they are a huge hack. But until the PEP
people get the issue designed properly[1] that's what we're stuck with.

Wolfgang

[1] http://mail.python.org/pipermail/import-sig/2012-March/000421.html

[Mike Orr]

On Sun, Apr 15, 2012 at 12:23 PM, Michael Bayer
<mik...@zzzcomputing.com> wrote:
> also --no-site-packages is the default now and not using it is a little crazy...

It's the standard for Pyramid because chrism has been promoting it,
and I sometimes have to use it because my Pyramid imports won't work
otherwise. But it's sad that it's undermining the ability to use OS
packages for libraries that are stable enough that you're happy with
whichever version the OS package is. (Like Mako, BeautifulSoup, Nose,
etc. I have a list of Python DEB packages that's the baseline for all
my servers and workstations. It includes those libraries but not
Pyramid, Pylons or SQLAlchemy because those have been changing too
rapidly for the OS packages to keep up.

The biggest problem with not having access to OS packages is that you
have to locally compile C extensions, which sometimes fails. If you're
making a checklist for other developers or sysadmins, you have to list
all the header dependencies they'll have to install, and what to do if
one error or another happens. This can turn a half-page simple
checklist into a two-page complex article, and even then you still get
called in because "it won't install" and the person doesn't understand
why or what to do.

The other issue is you have to install the same packages into every
virtualenv. If you don't take the time to set up a download cache or
PyPI mirror, it's downloading the same packages repeatedly.

So I don't see the point in recommending --no-site-packages or
practically forcing people to use it.

--
Mike Orr <slugg...@gmail.com>

[Michael Merickel]

OS packages can be linked into a virtualenv using virtualenvwrapper's
"add2virtualenv" script, or by simply adding a link to a .pth file in
the virtualenv's site-packages directory. The issues brought about by
*not* using --no-site-packages are much worse than the overhead of
adding the one or two packages you do actually want from the OS
site-packages, in my experience.

Also, in terms of recommending people use it, --no-site-packages is
the default in virtualenv now.