Developing modules with ‘pkgutil’
Ben Finney
With all the current fuss about namespace packages, I'm dealing with a
similar goal that I hope is possible with stock Python 2.x.
When developing a module intended to be part of an existing package
installed on the system, I want to write programs that will import the
*in-development* module from my current working tree, but the
*system-installed* modules from the rest of the package.
The specific case is that of a new writer for Docutils. The module will
eventually be installed such that it is accessed via ‘import
docutils.writers.foo’, but while developing it I explicitly *don't* want
the module to come from the system-installed ‘docutils/writers/’
directory since that will fail to get the one I'm developing.
But the rest of the Docutils infrastructure I *need* to come from the
system-installed package when I do ‘docutils.bar.baz’. So I need to
somehow have a module in one location appear, for the purpose of
imports, as though it's part of a package defined in a different
location.
A naive first try would be:
/usr/lib/python2.5/site-packages/
docutils/
__init__.py
core.py
…
readers/
__init__.py
…
transforms/
__init__.py
…
writers/
__init__.py
…
$HOME/projects/foo-writer/
tools/
rst2foo
docutils/
__init__.py
writers/
__init__.py
foo.py
So, while developing the ‘foo.py’ module, I need to run ‘tools/rst2foo’
in order to make use of the module and exercise its in-development
functionality (and *not* whatever ‘foo.py’ might exist in the system
library location). But whenever I import anything *else* from the
‘docutils’ package, it's not going to find the system-installed package.
The ‘pkgutil’ module <URL:http://docs.python.org/library/pkgutil>, which
I only partly understand (with thanks to Doug Hellman for
<URL:http://blog.doughellmann.com/2008/02/pymotw-pkgutil.html>), appears
to offer a solution for this, by allowing my in-development ‘docutils’
shim package to also include the real one:
$ cd $HOME/projects/foo-writer/
$ cat docutils/__init__.py
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
__path__.reverse()
Which, according to Doug, should allow me to have ‘tools/rst2foo’ say:
from docutils.core import publish_cmdline, default_description
from docutils.writers import foo
as it will look when installed for the end user; and the ‘docutils’
namespace will, because of the in-development shim package, be searched
both there and the system-installed directory, finding both the
system-installed ‘docutils.core’ and the in-development
‘docutils.writers.manpage’.
Then I should be able to run the program while developing, like this:
$ PYTHONPATH=. python ./tools/rst2foo
and it should find everything it needs.
However, this approach doesn't seem to be complete: there is code in the
system-installed ‘docutils/__init__.py’ that is not being found:
Traceback (most recent call last):
File "./tools/rst2foo", line 21, in <module>
from docutils.core import publish_cmdline, default_description
File "/usr/lib/python2.5/site-packages/docutils/core.py", line 20, in <module>
from docutils import __version__, __version_details__, SettingsSpec
ImportError: cannot import name __version__
What's happening, of course, is that the name ‘__version__’ is defined
in the system-installed ‘docutils/__init__.py’, but that's not being
found because the in-development ‘docutils/__init__.py’ is found first,
which doesn't have that name.
A simple ‘from docutils import *’ clearly doesn't work, for the same
reason. Catch-22.
At this point I'm stuck. I can't see how to have the
‘docutils/__init__.py’ stop shadowing the names in the system-installed
‘docutils/__init__.py’, while still doing the namespace shuffle
necessary to have my in-development module appear part of the wider
package namespace. What should I be doing instead?
--
\ “Smoking cures weight problems. Eventually.” —Steven Wright |
`\ |
_o__) |
Ben Finney
Peter Otten
> At this point I'm stuck. I can't see how to have the
> ‘docutils/__init__.py’ stop shadowing the names in the system-installed
> ‘docutils/__init__.py’, while still doing the namespace shuffle
> necessary to have my in-development module appear part of the wider
> package namespace. What should I be doing instead?
Weird idea. Try putting the following in your __init__.py files:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
__path__.reverse()
import __init__
globals().update(vars(__init__))
__path__.reverse()
If that doesn't work add
import docutils
docutils.__path__.insert(0, path_to_modified_version)
to your main script. Repeat for every subpackage.
Peter
Ben Finney
> Weird idea. Try putting the following in your __init__.py files:
>
> import pkgutil
> __path__ = pkgutil.extend_path(__path__, __name__)
> __path__.reverse()
> import __init__
> globals().update(vars(__init__))
> __path__.reverse()
That's rather astounding. It doesn't *quite* work, but here's what did
work:
=====
# docutils/__init__.py
# Python package for in-development docutils packages.
import pkgutil
# Ensure we can also reach modules in the system package
__path__ = pkgutil.extend_path(__path__, __name__)
# Make this directory come last when importing from this package
__path__.reverse()
# Make this package gain all the attributes of the system package
_path_prev = __path__
import __init__
globals().update(vars(__init__))
__path__ = _path_prev
del _path_prev
# Make this directory come first when importing from this package
__path__.reverse()
=====
Plus the same thing in ‘docutils/writers/__init__.py’.
Pretty ugly. I hope namespace packages can come along and save us from
this.
> If that doesn't work add
>
> import docutils
> docutils.__path__.insert(0, path_to_modified_version)
>
> to your main script. Repeat for every subpackage.
No, modifying the program is exactly what I'm trying to avoid; that
file, unlike these development-only shims, will actually be installed on
the end-user's system. The point of the exercise is to set up the
development working tree so that it presents the in-development module
transparently to the program, and the program remains untainted by these
ugly hacks :-)
But, since the above re-worked package module does the job, I'm able to
continue (though appalled at the hackiness required). Thank you!
--
\ “Pinky, are you pondering what I'm pondering?” “I think so, |
`\ Brain, but isn't a cucumber that small called a gherkin?” |
_o__) —_Pinky and The Brain_ |
Ben Finney