structuring a package for network protocols.
Anthony Baxter
I'm currently trying to figure out the best way to structure a package
system for a bunch of network protocol modules.
For example, say I have the following:
Multicast/__init__.py - general multicast support
Multicast/SAP.py - Session Announcement Protocol
Multicast/SDP.py - Session Description Protocol
Multicast/...
ldap.py - Lightweight Directory Access Protocol
_ldapmodule.so - lowlevel module
snmp.py - SNMP
snmpymodule.so - lowlevel SNMP module
DNS/__init__.py - DNS
DNS/... - other files for this
RADIUS/__init__.py
RADIUS/...
...and so on. I don't like the idea of these all being toplevel modules,
but if I was to make a site-packages toplevel Net module, then it would
require all the networking modules to be installed under it. (so you couldn't
have a different 'Net' package in a different directory in the python path
with different modules in it...)
I guess what I'm sort of looking at here is moving towards some sort of
general structuring for packages into a tree, but I'm not sure how to do it.
Andrew Kuchling
Anthony Baxter <a...@connect.com.au> wrote:
>I guess what I'm sort of looking at here is moving towards some sort of
>general structuring for packages into a tree, but I'm not sure how to do it.
These are problems we'll have to solve over 1.5's lifetime.
In your position, I'd go ahead and create the Net package, since it
seems generally useful enough to be a significant package.
Two things we'll have to work out:
* This cheats the community of the opportunity to debate about
whether the package should be named "Net" or "Network" or "Comm" or
... Seriously, one wonders if the package namespace will be carved
out in an ad hoc fashion by whoever implements packages first.
* What happens if I write a module for AX25 networking, and
want to put it in the package as Net/AX25.py? "import Net.AX25" will
work, but "from Net import *" won't work, because Net/__init__.py
doesn't know about the AX25 module. The AX25 distribution can't just
dump its own __init__.py into the Net/ directory. (This is similar to
the original problem that sparked the addition of packages, but it's
reduced in severity, since it only occurs if several entities want to
dump modules into the same package's namespace.)
Writing an __init__.py that loads all the *.py files and
shared modules would probably solve this problem, since then AX25.py
just gets dropped into the Net directory and is automatically picked
up. The several developers sharing the package's namespace would then
need to agree on a convention for files that don't get loading by
"from ... import *", such as a leading underscore or whatever.
Andrew Kuchling
a...@magnet.com
http://starship.skyport.net/crew/amk/
Anthony Baxter
>>> Andrew Kuchling wrote
> * This cheats the community of the opportunity to debate about
> whether the package should be named "Net" or "Network" or "Comm" or
> ... Seriously, one wonders if the package namespace will be carved
> out in an ad hoc fashion by whoever implements packages first.
I was thinking about something more like 'Internet', or 'Protocol'.
> * What happens if I write a module for AX25 networking, and
> want to put it in the package as Net/AX25.py? "import Net.AX25" will
Worse yet - unless you have write access to where the current Net tree
is, you won't be able to install it. You can't have two Net packages,
each with their own sub-packages (well, that's what it seems, anyway)
Anthony
Skip Montanaro
>>>>> "Marc" == M -A Lemburg <lem...@uni-duesseldorf.de> writes:
Marc> I think it is a good idea to make the packages names have mixed
Marc> case names -- keeping in mind though, that on some platforms there
Marc> is no distinction between upper/lower case filenames.
I thought Guido's convention was that package names should not be identifier
names. Isn't that why they have hyphens in them in 1.5?
Skip Montanaro | Musi-Cal: http://concerts.calendar.com/
sk...@calendar.com | Python Support: http://www.pythonpros.com/
(518)372-5583 | XEmacs: http://www.automatrix.com/~skip/xemacs/tip.html
M.-A. Lemburg
Andrew Kuchling wrote:
>
> Anthony Baxter <a...@connect.com.au> wrote:
> >I guess what I'm sort of looking at here is moving towards some sort of
> >general structuring for packages into a tree, but I'm not sure how to do it.
>
> These are problems we'll have to solve over 1.5's lifetime.
> In your position, I'd go ahead and create the Net package, since it
> seems generally useful enough to be a significant package.
>
> Two things we'll have to work out:
>
> whether the package should be named "Net" or "Network" or "Comm" or
> ... Seriously, one wonders if the package namespace will be carved
> out in an ad hoc fashion by whoever implements packages first.
I think we should "carve out" the namespace before the package authors
do. At least to a certain extent. Here is a first try at such a
structure:
Python
Datatype
AVL
Stack
Queue
PriorityQueue
DateTime
Graph
Network
Protocol
Multicast
SAP
SDP
LDAP
DNS
RADIUS
HTTP
FTP
SMTP
Address
URL
RFC822
Interface
CGI
Numeric
<packages from NumPy>
Database
dbi
Oracle
Informix
ODBC
Language
HTML
SGML
Encoding
Mime
Audio
MIDI
Video
???
GUI
Tkinter
Graphics
GL
PIL
Crypto
<Andrew's packages>
Text
Parsing
Search
Tools
Misc
Profiler
Debugger
Mac
<Macintosh platform specific packages>
Win32
<...>
Unix
<...>
[The package Python should contain the modules from the
standard lib too]
and so on. I think it is a good idea to make the packages
names have mixed case names -- keeping in mind though, that
on some platforms there is no distinction between upper/lower
case filenames.
The structure should be published in a well known place,
e.g. in the docs and on www.python.org (there maybe even
browsable).
Since the introduction of such a structure causes incompatible
changes we now have a prime time chance to do the Right Thing.
We should be very careful not to make the mistakes that Sun
did with Java (with every company having it's own subtree).
A note on user side extensions:
The main package (Python) is intended to seperate the site installation
from a user's private package tree. The user's packages then have
a possibility to access the site's packages (even if they have the
same name, e.g. to extend functionality). The Python path needs
two entry points into the above tree to make this work: one that
points to the Python directory and one that points to the parent
directory of Python (in that order).
Example: the user wants to add a protocol
He has a directory Willy for his Python packages which mimics the
main structure:
Network
Protocol
MyProtocol
The __init__ files in Network and Protocol mimic the original files
with 'from Python.Network import *' and '...Network.Protocol import *'.
If one of his modules does an 'import Network.Protocol.MyProtocol',
it'll find his package. If it does 'import Network.Protocol.DNS'
then the lookup in his dir will fail and the site's installation
will get a chance.
That should make the private installation extend the site's
installation -- if I didn't miss anything :-)
> * What happens if I write a module for AX25 networking, and
> want to put it in the package as Net/AX25.py? "import Net.AX25" will
> work, but "from Net import *" won't work, because Net/__init__.py
> doesn't know about the AX25 module. The AX25 distribution can't just
> dump its own __init__.py into the Net/ directory. (This is similar to
> the original problem that sparked the addition of packages, but it's
> reduced in severity, since it only occurs if several entities want to
> dump modules into the same package's namespace.)
>
> Writing an __init__.py that loads all the *.py files and
> shared modules would probably solve this problem, since then AX25.py
> just gets dropped into the Net directory and is automatically picked
> up. The several developers sharing the package's namespace would then
> need to agree on a convention for files that don't get loading by
> "from ... import *", such as a leading underscore or whatever.
I don't think this is a good idea. You'll end up importing
the whole package subtree if you e.g. import Datatype.
With all those file IOs startup time will become poor.
BTW: what use would 'from package import *' have ? If you
need the AX25 subpackage then just do a
'from Network.Protocol import AX25'.
--
Marc-Andre Lemburg
M.-A. Lemburg
Skip Montanaro wrote:
>
> >>>>> "Marc" == M -A Lemburg <lem...@uni-duesseldorf.de> writes:
>
> Marc> I think it is a good idea to make the packages names have mixed
> Marc> case names -- keeping in mind though, that on some platforms there
> Marc> is no distinction between upper/lower case filenames.
>
> I thought Guido's convention was that package names should not be identifier
> names. Isn't that why they have hyphens in them in 1.5?
Directories with hyphens in them can't be used as package
directories (how would you import them ?). The hyphen
trick was invented by Guido before the __init__.py files were
used as "is package" criterion. Previously all subdirectories on the
path were treated as potential packages and imported as such, only
subdirs with hyphens in their names were omitted.
What do you think of the structure ?
--
Marc-Andre Lemburg
Andrew Kuchling
"M.-A. Lemburg" <lem...@uni-duesseldorf.de> wrote:
> Datatype
> AVL
> Stack
> Queue
I assume the leaves here are the actual modules? (AVL.py,
Stack.py, ...) I can quibble with your arrangement -- why are HTML
and SGML under Language and not Text or Text.Parsing, for example? --
but it seems reasonable. The scary thing is that there's really only
one chance to get things right; changing things later will be traumatic.
>Since the introduction of such a structure causes incompatible
>changes we now have a prime time chance to do the Right Thing.
>We should be very careful not to make the mistakes that Sun
>did with Java (with every company having it's own subtree).
But then we still have the difficult problem of letting
several people, such as both the administrator and the user, be able
to create subpackages of existing packages. Your suggested solution
doesn't work; I didn't look into the code to see how difficult it
would be to make it work.
>If one of his modules does an 'import Network.Protocol.MyProtocol',
>it'll find his package. If it does 'import Network.Protocol.DNS'
>then the lookup in his dir will fail and the site's installation
>will get a chance.
This doesn't seem to work; the lookup in the user's directory
fails, and Python raises an ImportError. It doesn't keep looking
further along sys.path .
>BTW: what use would 'from package import *' have ? If you
>need the AX25 subpackage then just do a
>'from Network.Protocol import AX25'.
Actually, I've found that I have to use 'from ... import *'
more with packages. The crypto test suite used to contain code like
this to get an arbitrary module:
module = __import__('MD5')
But __import__('A.B.C') returns the top-level A module, not A.B.C, so
all code like the above had to be changed to:
from Crypto.Hash import * ; module = eval('MD5')
Is there a more elegant solution to this?
M.-A. Lemburg
Here is a more elaborate version of require that also handles
the parent packages correctly and merges the module's dictionaries
if necessary. The packages in User. should always do a
from Python.My.Package.Name import *
in their __init__.py.
At least one problem remains: since from...import * doesn't import the
names with a leading underscore, these are not imported and thus
could either be from the Python.-tree or the User.-tree.
--
Marc-Andre Lemburg
---------------------------------------------------------------------
""" Package import helper. This makes it possible to have two
trees of packages, one starting with 'Python.' for the site
installation of Python and Python's standard packages and one
named 'User.' to which the user can add his/her own packages.
"""
__version__ = '0.1'
__debug__ = 1
import sys,string
def require(pkg,pkgpath=('Python.','User.')):
# Get locals & globals from caller
try:
1/0
except:
tb = sys.exc_info()[2]
parent_frame = tb.tb_frame.f_back
locals = parent_frame.f_locals
globals = parent_frame.f_globals
del tb
# Look for the module and try to import it
m = None
for prefix in pkgpath:
try:
m = __import__(prefix + pkg,locals,globals,['*'])
break
except ImportError:
pass
if not m:
raise ImportError,'No package named %s in %s' % (pkg,pkgpath)
if __debug__:
print 'found in',prefix
# Register the module and all it's parent modules too
path = string.split(pkg,'.')
addpkg = None
for p in path:
if addpkg:
addpkg = '%s.%s' % (addpkg,p)
else:
addpkg = p
m = __import__(prefix + addpkg,locals,globals,['*'])
# Merge the modules dictionaries if necessary
if sys.modules.has_key(addpkg):
if __debug__:
print 'merging the module dictionary for',addpkg
sys.modules[addpkg].__dict__.update(m.__dict__)
else:
sys.modules[addpkg] = m
def example_of_usage():
# This module needs Datatype.Stack:
require('Datatype.Stack')
# Now get the module
import Datatype.Stack
from Datatype import Stack
M.-A. Lemburg
Andrew Kuchling wrote:
>
> "M.-A. Lemburg" <lem...@uni-duesseldorf.de> wrote:
> > Datatype
> > AVL
> > Stack
> > Queue
>
> I assume the leaves here are the actual modules? (AVL.py,
> Stack.py, ...)
No, they are package directories. This is meant for easy upgrading
-- the package authors can provide an __init__.py file that imports
the most important parts and still add new subpackages as they like,
e.g. if I want to use my stack implementation, I write
from Datatype.Stack import Stack
s = Stack()
...
The __init__.py file provides a good way to do indirection
too, e.g. I could add a Python version of the Stack code and
import that if the C module isn't found. That Python module
could also be explicitly available as Datatype.Stack.UserStack
module.
> I can quibble with your arrangement -- why are HTML
> and SGML under Language and not Text or Text.Parsing, for example? --
> but it seems reasonable.
That's a point I wasn't sure about: the packages for HTML
and SGML could contain parsers, writers, special tools for
conversions, etc. specific to that language. BTW: Python
could be added just as well, containing converters to
HTML, Latex, etc.
> The scary thing is that there's really only
> one chance to get things right; changing things later will be traumatic.
That's why we should start discussing this. I'm beginning to
work with such a structure and find it most useful. The question
is: who is going to administer the structure ? To prevent
naming conflicts there will have to be a place to look for
the current setting. A possibility would be to open up a
site similar to Aaron's PyModuleFAQ -- since Guido's faqwiz
is organized as hierachy this should provide a good starting
point. Another possibility is using a BSCW server.
> >Since the introduction of such a structure causes incompatible
> >changes we now have a prime time chance to do the Right Thing.
> >We should be very careful not to make the mistakes that Sun
> >did with Java (with every company having it's own subtree).
>
> But then we still have the difficult problem of letting
> several people, such as both the administrator and the user, be able
> to create subpackages of existing packages. Your suggested solution
> doesn't work; I didn't look into the code to see how difficult it
> would be to make it work.
I'm afraid you're right :( It doesn't work since backtracking
would be needed... and that's just about impossible to do
(unloading modules, etc.).
> >If one of his modules does an 'import Network.Protocol.MyProtocol',
> >it'll find his package. If it does 'import Network.Protocol.DNS'
> >then the lookup in his dir will fail and the site's installation
> >will get a chance.
>
> This doesn't seem to work; the lookup in the user's directory
> fails, and Python raises an ImportError. It doesn't keep looking
> further along sys.path .
Hmm, another way would be:
""" Package import helper. This makes it possible to have two
trees of packages, one starting with 'Python.' for the site
installation of Python and Python's standard packages and one
named 'User.' to which the user can add his/her own packages.
"""
import sys
def require(pkg):
# Get locals & globals from caller
try:
1/0
except:
tb = sys.exc_info()[2]
parent_frame = tb.tb_frame.f_back
locals = parent_frame.f_locals
globals = parent_frame.f_globals
del tb
# Look for the module and import it under the given name
# (plus under it's original name).
try:
m = __import__('Python.'+pkg,locals,globals,['*'])
except ImportError:
m = __import__('User.'+pkg,locals,globals,['*'])
sys.modules[pkg] = m
def example_of_usage():
# This module needs Datatype.Stack:
require('Datatype.Stack')
# Now get the module
import Datatype.Stack
# Note that the following fails, since the parent package
# of Datatype.Stack is not registered in sys.modules.
from Datatype import Stack
> >BTW: what use would 'from package import *' have ? If you
> >need the AX25 subpackage then just do a
> >'from Network.Protocol import AX25'.
>
> Actually, I've found that I have to use 'from ... import *'
> more with packages. The crypto test suite used to contain code like
> this to get an arbitrary module:
> module = __import__('MD5')
>
> But __import__('A.B.C') returns the top-level A module, not A.B.C, so
> all code like the above had to be changed to:
> from Crypto.Hash import * ; module = eval('MD5')
>
> Is there a more elegant solution to this?
Yes, I posted the following a couple of weeks ago:
1. Importing a submodule of a package by name
-----------------------------------------------------------
In the past you could always do this to import a module
"by hand":
m = __import__('module',globals(),locals(),['*'])
With the advent of builtin packages this still works, but you
have two options now:
packagemodule = __import__('package.submodule',globals(),locals(),[])
and
submodule = __import__('package.submodule',globals(),locals(),['*'])
The fromlist entry '*' makes the difference.
You may also have use for this one:
2. Preventing double initialisation of modules
-----------------------------------------------------------
Modules in a package can import themselves by using just
the module name without the package name. They only get
initialized once. But say you are starting a module from
the package directory that imports a module from some
other package which in return needs some of the modules
from the package dir you're currently working in, then those
modules will get loaded *and* initialized again if they were
previously loaded (and registered once with and without the
package name).
--
Marc-Andre Lemburg
Dirk Heise
M.-A. Lemburg wrote:
> and so on. I think it is a good idea to make the packages
> names have mixed case names -- keeping in mind though, that
> on some platforms there is no distinction between upper/lower
> case filenames.
Do you mean that on DOS or the like "HTML" would address the
same file "html" would address?
If so, i agree with using mixed-case names.
Dirk Heise, Braunschweig, Germany
dhe...@metronet.de
M.-A. Lemburg
Dirk Heise wrote:
>
> M.-A. Lemburg wrote:
> > and so on. I think it is a good idea to make the packages
> > names have mixed case names -- keeping in mind though, that
> > on some platforms there is no distinction between upper/lower
> > case filenames.
>
> same file "html" would address?
>
> If so, i agree with using mixed-case names.
Yep. Modules and packages are identified by their filename
and if the OS doesn't make a difference, neither does Python.
Back to the subject:
So far I have only heard some comments on the structure from
Andrew... is this topic really that uninteresting ? I thought
it would be a good idea to start discussing this, since it
will come sooner or later anyway and better have the fights
now than having them when someone already uses his/her
own private structure [and no, I'm not calling for a SIG :-)].
We should at least fix some of the top-level package names...
--
Marc-Andre Lemburg
Guido van Rossum
> So far I have only heard some comments on the structure from
> Andrew... is this topic really that uninteresting ? I thought
> it would be a good idea to start discussing this, since it
> will come sooner or later anyway and better have the fights
> now than having them when someone already uses his/her
> own private structure [and no, I'm not calling for a SIG :-)].
>
> We should at least fix some of the top-level package names...
I find it fascinating but hard to fathom, and have refrained from
cdommenting so far because it requires much thought. I'm not sure
about the best solution. I don't like Java's solution very much
(reversed domain names, e.g. com.digicool.bobo) -- it solves the
problem of avoiding name clashes but you end up with renaming problems
when companies merge, or when the software changes hands.
Perhaps a different approach is to use arbitrary naming (i.e. the name
is up to the author of the software) combined with common sense and a
registry.
Common sense dictates that top-level names are a scarce resource and
should thus be allocated prudently. In particular, if you have a
large number of related modules, don't allocate a top-level module
name for each one -- instead, put them all in one package. If you are
a software vendor, you *may* decide to use a single top-level package
name for all software you produce -- but you could also use the
product names (if you don't have too many, and if they are distinctive
enough). Common sense also dictates that you don't use generic words
-- top-level package names like "net" are bad (but as subpackage names
within one product they are fine). Of course one can argue about the
proper choice of names but I don't see a need to decide on all this
beforehand. If someone uses an improper word as a package name, they
will get flamed and that is that.
To avoid name clashes between independently developed packages, I
propose to use a registry. Fortunately, a registration service for
Python module names already exists. It was created by Don Libes at
NIST, as an example of a general name registration service he runs.
To experiment with the service, go to
Click on "Click here to use the service now!", and select "Python
modules" from the box of available domains. From then on usage is
straightforward. (In order to add a name, you must first register
yourself as an "owner" to the system. Anybody can be an owner, as
long as you have an email address.)
I should fix the description somewhat to explain that the namespace
actually covers top-level modules as well as packages. I also want to
make the registry case insensitive (since it would be a real pain to
have two packages that only differ in case on non-Unix platforms).
But the registry is usable right away!
A separate issue is the assignment of the standard library modules to
packages. Eventually, most standard modules will be moved into
specific packages. I will be the final arbitrator for the package
names, but I sollicit proposals. A system of aliases or additional
path entries will provide backward compatibility. This should be in
place when Python 1.6 comes out (that's at least a year in the future,
probably).
It is important to understand that these package names will *only* be
used for "standard" modules. Some modules that are currently in the
distribution will probably lose their standard status or be moved to a
"misc" or "obsolete" package. Others may be moved to
platform-specific or domain-specific packages (e.g. "multimedia" or
"web").
--Guido van Rossum (home page: http://www.python.org/~guido/)
Gordon McMillan
M.-A. Lemburg wrote:
[...]
> So far I have only heard some comments on the structure from
> Andrew... is this topic really that uninteresting ? I thought
> it would be a good idea to start discussing this, since it
> will come sooner or later anyway and better have the fights
> now than having them when someone already uses his/her
> own private structure [and no, I'm not calling for a SIG :-)].
>
> We should at least fix some of the top-level package names...
Well I, for one, am still puzzling about the implications. So far my
only experience creating a package had to do with breaking up an
overly large module. The __init__.py for the package just says
'pass'.
At one point you said that the java package naming convention was a
mistake. While I agree that it can be cumbersome (and would be
particularly so in interactive use) because names can be arbitrarily
obtuse, it certainly avoids name collisions.
Point 1: Your proposal to have "Python." and "User." is just the
first step down the same path - you've just postponed the conflict to
"User.". (What are those anyway - invisible package-name prefixes?).
Point 2: I can distribute my stuff in a directory named (say)
"mcmillan", and have my users put it on their path. Then, if I name
my packages with obtuse prefixes, I can be fairly sure they'll work
no matter what someone else has done. So, I can recreate the java
situation, even if it is a mistake, (I think it's a trade-off).
I see nothing wrong with a benevolent despot (Guido) decreeing a
structure for blessed packages, but I don't see anything to be gained
by imposing (or bureaucratizing) any order on the rest of the
universe.
Or maybe I've misunderstood your proposal?
- Gordon
Andrew Kuchling
Guido van Rossum wrote:
>> I still have a big big problem with the way that 1.5 packages work -
>> if a site ends up with a standard python package, a site version, and
>> a user version, all called the same thing, it's going to be nightmarish.
>
>But you're not supposed to do that!!! Did anyone suggest this?
I think I did; here's a concrete example. Someone is working
on a Python/SSLeay interface. A natural place for it would be
Crypto.Protocol, a package I "own", but I have no intention of
including the code in my own distribution. If the system
administrator installs both my code and the SSLeay interface,
everything's fine; from "Crypto.Protocol import *" may not import the
SSLeay module, but that's a minor point.
However, what if the sysadmin installs my code, and a user
wishes to install the SSLeay module in their private directory? *This
won't work*, because when the user writes "import Crypto.Hash.MD5",
Python will find the user's private Crypto package, containing only
Protocol/SSLeay.so, and the import will fail. The user would have to
duplicate the whole Crypto tree in their private directory to make
things work.
The first solution that comes to mind is to make import keep
looking along sys.path for more Crypto packages than just the first.
However, what does "from Crypto import *" do, in that case? Run the
__init__.py for both packages in the same namespace? Or can the
language be left alone, and a smart __init__.py be written that will
handle this automatically?
Guido van Rossum
> I still have a big big problem with the way that 1.5 packages work -
> if a site ends up with a standard python package, a site version, and
> a user version, all called the same thing, it's going to be nightmarish.
But you're not supposed to do that!!! Did anyone suggest this?
--Guido van Rossum (home page: http://www.python.org/~guido/)
Anthony Baxter
>>> "M.-A. Lemburg" wrote
> So far I have only heard some comments on the structure from
> Andrew... is this topic really that uninteresting ? I thought
Nope, I've just been busy. :)
I still have a big big problem with the way that 1.5 packages work -
if a site ends up with a standard python package, a site version, and
a user version, all called the same thing, it's going to be nightmarish.
Anthony
Andrew Kuchling
Guido van Rossum wrote:
> ... I don't like the idea very much to have one
>global hierarchy of packages structured by some semantic
>classification scheme (like Crypto.Protocol). Designing
>classification trees is very hard, and assigning items to a place in
>the tree is even harder. ...
...
>See my previous post on the subject for an alternative; basically, you
>would give your Crypto kit a more ideosyncratic toplevel package name
>(e.g. AMKcrypto or CryptoKit) and register it with the Python
>module registration at NIST (http://pitch.nist.gov/nics/).
So if I have two little WWW modules, and Dave Mitchell has two
little WWW modules, we have to have AMKWWW/state.py and
DaveMWWW/cookie.py, or some other trivial distinction? While I could
probably learn to live with this, my initial reaction isn't favorable.
I think this would both reduce the use of packages--small groups of
one or two modules will just get left at the top-level--and larger
packages can't be organized into yet larger hierarchies.
BTW, looking at the perlfunc man page, I think Perl's
behaviour is: import Foo.Bar looks all along sys.path for Foo/Bar.py;
"from Foo import *" uses the contents of the first Foo package found,
and you lose if you were expecting the Foo package that's later on the
path. (Which is fine with me, since as you all know, I intensely
dislike "from ... import *".) I may well have misinterpreted the man
page, but have sent e-mail to a Perl wizard asking for enlightenment.
Just a data point...
Guido van Rossum
> Guido van Rossum wrote:
> >> I still have a big big problem with the way that 1.5 packages work -
> >> if a site ends up with a standard python package, a site version, and
> >> a user version, all called the same thing, it's going to be nightmarish.
> >
Andrew Kuchling <a...@magnet.com>:
> I think I did; here's a concrete example. Someone is working
> on a Python/SSLeay interface. A natural place for it would be
> Crypto.Protocol, a package I "own", but I have no intention of
> including the code in my own distribution. If the system
> administrator installs both my code and the SSLeay interface,
> everything's fine; from "Crypto.Protocol import *" may not import the
> SSLeay module, but that's a minor point.
>
> However, what if the sysadmin installs my code, and a user
> wishes to install the SSLeay module in their private directory? *This
> won't work*, because when the user writes "import Crypto.Hash.MD5",
> Python will find the user's private Crypto package, containing only
> Protocol/SSLeay.so, and the import will fail. The user would have to
> duplicate the whole Crypto tree in their private directory to make
> things work.
>
> The first solution that comes to mind is to make import keep
> looking along sys.path for more Crypto packages than just the first.
> However, what does "from Crypto import *" do, in that case? Run the
> __init__.py for both packages in the same namespace? Or can the
> language be left alone, and a smart __init__.py be written that will
> handle this automatically?
My take on this is that someone else has no business placing their
modules in your package. I don't like the idea very much to have one
global hierarchy of packages structured by some semantic
classification scheme (like Crypto.Protocol). Designing
classification trees is very hard, and assigning items to a place in
the tree is even harder. Plus, there's the problem that you're
pointing out. Even if it were technically possible to do that (a
smart __init__.py can indeed do this by manipulating its own
__path__), it can have all sorts of painful consequences, and it would
slow down the module searches.
See my previous post on the subject for an alternative; basically, you
would give your Crypto kit a more ideosyncratic toplevel package name
(e.g. AMKcrypto or CryptoKit) and register it with the Python
module registration at NIST (http://pitch.nist.gov/nics/).
--Guido van Rossum (home page: http://www.python.org/~guido/)
Anthony Baxter
>>> Guido van Rossum wrote
> > I still have a big big problem with the way that 1.5 packages work -
> > if a site ends up with a standard python package, a site version, and
> > a user version, all called the same thing, it's going to be nightmarish.
>
> But you're not supposed to do that!!! Did anyone suggest this?
Indirectly, yes - if there's one big standard tree of packages, as has
been suggested, then you're going to end up either having to make
the tree writable to everyone, having some sort of horrible hack that
knows what other trees are on the system, or not being able to install
stuff in the standard tree at all.
I'd _like_ for the package structure, if it gets a request of "import
Package.Subpackage", and it can't find Subpackage in the first instance of
Package, to keep searching the path for other occurances of Package. It
just seems more logical to me (with the obvious caveat that it's 5am now,
and my brain is fried from too much RADIUS coding...)
Whether this is actually implementable is another matter entirely. :)
Anthony
Scott
On Thu, Jan 15, 1998 at 01:17:13PM -0500, Guido van Rossum wrote:
[...]
| I don't like the idea very much to have one
| global hierarchy of packages structured by some semantic
| classification scheme (like Crypto.Protocol). Designing
| classification trees is very hard, and assigning items to a place in
| the tree is even harder.
I agree that hierarhical classification of lots of data (like the
standard library modules) would be very confusing and would lead more and
more difficulties finding things as more data is added. It would also
probably eventually lead to cross-subject dummy entries making the
whole thing even worse.
However, It seems to me that semantic classification schemes are
rather ideal for self contained smaller 'packages' where a coherent
classification tree can be planned thoroughly and ahead of time.
I'd also like to show my support for AMK naming his top level module
Crypto instead of AMKCrypto or something. To me, if someone else
wants to write a cryptographic package, and AMK has registered his,
then that person should just pick another name. Too much caution in
picking top level module names can result in problems and less
interesting names.
Just my .02$
scott
Guido van Rossum
> I agree that hierarhical classification of lots of data (like the
> standard library modules) would be very confusing and would lead more and
> more difficulties finding things as more data is added. It would also
> probably eventually lead to cross-subject dummy entries making the
> whole thing even worse.
No contest :-)
> However, It seems to me that semantic classification schemes are
> rather ideal for self contained smaller 'packages' where a coherent
> classification tree can be planned thoroughly and ahead of time.
Agreed -- *inside* one package you can do a good simple
subclassification.
> I'd also like to show my support for AMK naming his top level module
> Crypto instead of AMKCrypto or something. To me, if someone else
> wants to write a cryptographic package, and AMK has registered his,
> then that person should just pick another name. Too much caution in
> picking top level module names can result in problems and less
> interesting names.
Sure. Since Andrew is the main provider of Python Crypto modules,
that's fine (as long as we don't decide to use "Crypto" for the few
standard crypto modules). However, in general, you're better off
giving your package a non-generic name. I like the example of the RE
package JPython is using -- it's called "OROMatcher". I'll never
forget that one :-)
M.-A. Lemburg
Andrew Kuchling wrote:
>
> Guido van Rossum wrote:
> >> I still have a big big problem with the way that 1.5 packages work -
> >> if a site ends up with a standard python package, a site version, and
> >> a user version, all called the same thing, it's going to be nightmarish.
> >
> >But you're not supposed to do that!!! Did anyone suggest this?
>
> on a Python/SSLeay interface. A natural place for it would be
> Crypto.Protocol, a package I "own", but I have no intention of
> including the code in my own distribution. If the system
> administrator installs both my code and the SSLeay interface,
> everything's fine; from "Crypto.Protocol import *" may not import the
> SSLeay module, but that's a minor point.
>
> However, what if the sysadmin installs my code, and a user
> wishes to install the SSLeay module in their private directory? *This
> won't work*, because when the user writes "import Crypto.Hash.MD5",
> Python will find the user's private Crypto package, containing only
> Protocol/SSLeay.so, and the import will fail. The user would have to
> duplicate the whole Crypto tree in their private directory to make
> things work.
If the user calls the requires() function I posted earlier on
this thread, then he won't have to copy anything: the function
will look (per default) in two places: Python.Crypto.Hash.MD5
and if it can't find it there in User.Crypto.Hash.MD5. It then
registers the found module as 'Crypto.Hash.MD5' and subsequent
imports a la 'import Crypto.Hash.MD5' work as expected.
--
Marc-Andre Lemburg
M.-A. Lemburg
Andrew Kuchling wrote:
>
> Guido van Rossum wrote:
> >> I still have a big big problem with the way that 1.5 packages work -
> >> if a site ends up with a standard python package, a site version, and
> >> a user version, all called the same thing, it's going to be nightmarish.
> >
> >But you're not supposed to do that!!! Did anyone suggest this?
>
> I think I did; here's a concrete example. Someone is working
> on a Python/SSLeay interface. A natural place for it would be
> Crypto.Protocol, a package I "own", but I have no intention of
> including the code in my own distribution. If the system
> administrator installs both my code and the SSLeay interface,
> everything's fine; from "Crypto.Protocol import *" may not import the
> SSLeay module, but that's a minor point.
>
> However, what if the sysadmin installs my code, and a user
> wishes to install the SSLeay module in their private directory? *This
> won't work*, because when the user writes "import Crypto.Hash.MD5",
> Python will find the user's private Crypto package, containing only
> Protocol/SSLeay.so, and the import will fail. The user would have to
> duplicate the whole Crypto tree in their private directory to make
> things work.
>
> looking along sys.path for more Crypto packages than just the first.
How would you do that ? Python will already have initialized all
parent packages, e.g. Crypto and Crypto.Hash. If it tries a second
path, it will find them again...
Have you tried the function I posted earlier on this thread ?
I think it solves at least some aspects.
> However, what does "from Crypto import *" do, in that case? Run the
> __init__.py for both packages in the same namespace? Or can the
> language be left alone, and a smart __init__.py be written that will
> handle this automatically?
It would have to know where to look for the user installation.
The site module could provide a hint in the right direction...
But there still is the problem with import failing. The __init__.py
module of the package where import tried to find a submodule
is not informed about this and thus cannot react in some appropriate
way. Maybe we could add a string to sys that only exists when an
import is underway and then contains the module name that import
is trying to find.
--
Marc-Andre Lemburg
ross andrus
Scott wrote:
>
> However, It seems to me that semantic classification schemes are
> rather ideal for self contained smaller 'packages' where a coherent
> classification tree can be planned thoroughly and ahead of time.
>
> Crypto instead of AMKCrypto or something. To me, if someone else
> wants to write a cryptographic package, and AMK has registered his,
> then that person should just pick another name. Too much caution in
> picking top level module names can result in problems and less
> interesting names.
This sounds right to me - a balance between anarchy and theocracy.
Plus, the incentive to "get the good names" may spur lurkers (like
myself) to get their packages into common use. Sort of a land-grab.
ross
M.-A. Lemburg
Guido van Rossum wrote:
>
> > So far I have only heard some comments on the structure from
> > Andrew... is this topic really that uninteresting ? I thought
> > will come sooner or later anyway and better have the fights
> > now than having them when someone already uses his/her
> > own private structure [and no, I'm not calling for a SIG :-)].
> >
> > We should at least fix some of the top-level package names...
>
> cdommenting so far because it requires much thought.
True.
> I'm not sure
> about the best solution. I don't like Java's solution very much
> (reversed domain names, e.g. com.digicool.bobo) -- it solves the
> problem of avoiding name clashes but you end up with renaming problems
> when companies merge, or when the software changes hands.
>
> Perhaps a different approach is to use arbitrary naming (i.e. the name
> is up to the author of the software) combined with common sense and a
> registry.
The problem is: how would you share package namespace among different
people ? Who is going to maintain the __init__.py file of the parent
package (it should probably remain empty, IMHO) ? It's more a management
problem than a technical one.
>
> Common sense dictates that top-level names are a scarce resource and
> should thus be allocated prudently. In particular, if you have a
> large number of related modules, don't allocate a top-level module
> name for each one -- instead, put them all in one package. If you are
> a software vendor, you *may* decide to use a single top-level package
> name for all software you produce -- but you could also use the
> product names (if you don't have too many, and if they are distinctive
> enough). Common sense also dictates that you don't use generic words
> -- top-level package names like "net" are bad (but as subpackage names
> within one product they are fine). Of course one can argue about the
> proper choice of names but I don't see a need to decide on all this
> beforehand. If someone uses an improper word as a package name, they
> will get flamed and that is that.
>
The problem with this is that packages won't interoperate very
well: if the user wants all his installed datatypes in one package,
he can't just go and pack them into a new package, since other packages
under other top level nodes won't be able to see this change.
The package authors will pretty much have total control of what the
user can do with them and what not. That's why I propose to fix
some of the obvious names like Datatype, Database, Crypto, Network,
Network.Protocol, etc. For some of them defining the second level
too wouldn't hurt either (e.g. for Network and Crypto).
> To avoid name clashes between independently developed packages, I
> propose to use a registry. Fortunately, a registration service for
> Python module names already exists. It was created by Don Libes at
> NIST, as an example of a general name registration service he runs.
> To experiment with the service, go to
>
> http://pitch.nist.gov/nics/
>
> Click on "Click here to use the service now!", and select "Python
> modules" from the box of available domains. From then on usage is
> straightforward. (In order to add a name, you must first register
> yourself as an "owner" to the system. Anybody can be an owner, as
> long as you have an email address.)
I would have preferred an adapted version of your faqwiz -- apart
from having a nicer user interface (just a personal opinion), it
also offers a hierachical view, so you could not only register
the top level packages, but also some of the second and third
level ones. Name collision detection should be easy to add.
>
> I should fix the description somewhat to explain that the namespace
> actually covers top-level modules as well as packages. I also want to
> make the registry case insensitive (since it would be a real pain to
> have two packages that only differ in case on non-Unix platforms).
> But the registry is usable right away!
>
> A separate issue is the assignment of the standard library modules to
> packages. Eventually, most standard modules will be moved into
> specific packages. I will be the final arbitrator for the package
> names, but I sollicit proposals.
Only one wish: please use mixed case names -- since Python is
moving in that direction anyway it would be a step backward if
packages would start to use lowercase names.
> A system of aliases or additional
> path entries will provide backward compatibility. This should be in
> place when Python 1.6 comes out (that's at least a year in the future,
> probably).
>
> It is important to understand that these package names will *only* be
> used for "standard" modules. Some modules that are currently in the
> distribution will probably lose their standard status or be moved to a
> "misc" or "obsolete" package. Others may be moved to
> platform-specific or domain-specific packages (e.g. "multimedia" or
> "web").
Then you should place all of them inside a special top level package
e.g. 'Lib' or 'Standard'. The standard lib already cover great
parts of what one could think of a top-level packages. If you take over
control of all these names, there'd not be much left for package authors
to things into.
--
Marc-Andre Lemburg
Konrad Hinsen
Guido van Rossum <gu...@CNRI.Reston.Va.US> writes:
> direction. Authors have the right to choose a name they like. In any
> case I'd like to propose a rule that module names need to be unique
> when mapped to a single case. (Thank goodness we don't really have to
But how should such a rule be enforced? At best I can make sure that
there is no such conflict among my own modules (and the standard
library). And even that is difficult for Unix users.
A while ago a Windows user complained about my PDB module whose name
conflicts with pdb (the Python debugger) on Windows. Of course I had
never noticed or even suspected such a problem, even though I use both
modules frequently. Since everything in Python is case sensitive, it
is not easy to constantly be aware of the fact that modules on some
systems are the exception.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hin...@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
Guido van Rossum
[me]
> > Perhaps a different approach is to use arbitrary naming (i.e. the name
> > is up to the author of the software) combined with common sense and a
> > registry.
[Marc-Andre Lemburg]
> The problem is: how would you share package namespace among different
> people ? Who is going to maintain the __init__.py file of the parent
> package (it should probably remain empty, IMHO) ? It's more a management
> problem than a technical one.
My proposal is not to share packages between people (except when
they're part of a close-knit team or perhaps when they are in the same
organization).
> The problem with this is that packages won't interoperate very
> well: if the user wants all his installed datatypes in one package,
> he can't just go and pack them into a new package, since other packages
> under other top level nodes won't be able to see this change.
There is no way that a user can rearrange the namespace. It would
break dependencies. You don't do this with module names either -- why
should you do it with packages?
> The package authors will pretty much have total control of what the
> user can do with them and what not.
Huh? The authors have control over the name of their packages.
That's normal. The same is true for any other language with package
support that I'm aware of -- Java, Perl, Tcl.
> That's why I propose to fix
> some of the obvious names like Datatype, Database, Crypto, Network,
> Network.Protocol, etc. For some of them defining the second level
> too wouldn't hurt either (e.g. for Network and Crypto).
And that's exactly what I think won't work. That's trying to come up
with a universal classification scheme, and I think it'll fail. Does
the PIL image data type belong in Datatype or in Multimedia? Does
urllib belong in Network.Protocol? And so on.
> I would have preferred an adapted version of your faqwiz -- apart
> from having a nicer user interface (just a personal opinion), it
> also offers a hierachical view, so you could not only register
> the top level packages, but also some of the second and third
> level ones. Name collision detection should be easy to add.
A gift horse... NICS has a whole bunch of useful functionality that
faqwiz doesn't have and that would be a lot of work to add --
e.g. individual items have owners who can set their own password,
there is email notification of conflicts and comments, and so on.
For subpackages, simply register a name with dot in it.
> Only one wish: please use mixed case names -- since Python is
> moving in that direction anyway it would be a step backward if
> packages would start to use lowercase names.
This is a very religious issue. I don't like mixed case much, and I'm
not sure where you get the idea that Python is moving in that
direction. Authors have the right to choose a name they like. In any
case I'd like to propose a rule that module names need to be unique
when mapped to a single case. (Thank goodness we don't really have to
worry about 8+3 names any more...)
> Then you should place all of them inside a special top level package
> e.g. 'Lib' or 'Standard'. The standard lib already cover great
> parts of what one could think of a top-level packages. If you take over
> control of all these names, there'd not be much left for package authors
> to things into.
I expect to grab maybe 10 rather generic words as top-level standard
packages. I'll try to be careful. Maybe they will all end up in a
package "std". Then you can simply place "from std" in front of all
your import statements and change the existing "from foo import"
into "from std.foo import" to make old programs work.
Guido van Rossum
[me]
> > direction. Authors have the right to choose a name they like. In any
> > case I'd like to propose a rule that module names need to be unique
> > when mapped to a single case. (Thank goodness we don't really have to
[Konrad Hinsen]
> But how should such a rule be enforced? At best I can make sure that
> there is no such conflict among my own modules (and the standard
> library). And even that is difficult for Unix users.
>
> A while ago a Windows user complained about my PDB module whose name
> conflicts with pdb (the Python debugger) on Windows. Of course I had
> never noticed or even suspected such a problem, even though I use both
> modules frequently. Since everything in Python is case sensitive, it
> is not easy to constantly be aware of the fact that modules on some
> systems are the exception.
I sympathize with your position. I don't know how to enforce it
either. But it has to be a rule anyway. I see no other way to
support Windows and Mac. Perhaps the next release of your modules can
take the form of a package, so you only reserve one top-level name?
Andrew Kuchling
Guido van Rossum <gu...@CNRI.Reston.Va.US> wrote:
>My proposal is not to share packages between people (except when
>they're part of a close-knit team or perhaps when they are in the same
>organization).
Such shared packages could also be handled by having an editor
who collects contributions and maintains the package as a whole. For
example, there'd be one person who owned the "Web" package, and
coordinated with the people who maintain cookie.py, etc..
>This is a very religious issue. I don't like mixed case much, and I'm
>not sure where you get the idea that Python is moving in that
>direction.
Perhaps from the Great Renaming, which went from getintvalue
to PyInt_AsLong? But most Python modules are given lowercase-only
names, with a few exceptions.
BTW, from
http://www.yahoo.com/text/headlines/980115/entertainment/stories/culture_comedy_1.html:
===
HOLLYWOOD (Variety) - The five surviving partners of the legendary
British comedy troupe Monty Python are set to reunite for the first
time since the 1989 death of member Graham Chapman.
John Cleese, Terry Gilliam, Eric Idle, Terry Jones and Michael Palin
will appear at the U.S. Comedy Arts Festival in Aspen, Colo., which
runs March 4-8.
...
Indeed, as Palin has come to understand, being part of Python means
never really knowing what may lurk around the corner.
"We've never really followed any rules at all with Python," he said.
"We're a spontaneous lot. It's more fun that way."
===
I find the last paragraph highly amusing.
Christopher G. Petrilli
Guido van Rossum wrote
> > I still have a big big problem with the way that 1.5 packages work -
> > if a site ends up with a standard python package, a site version, and
> > a user version, all called the same thing, it's going to be nightmarish.
>
> But you're not supposed to do that!!! Did anyone suggest this?
It seems to me the best way to do this, and avoid localized naming problems
is to load packages similar to Perl, which means you handle them this way:
from blah import *
Here you look along the load path, and grab the first match,
and follow the __init__.py, and if IT wants to continue on,
fine, but it's responsible for "correct behavior."
from blah import foo
Follow the path until you find the package. If you don't find
the package in the "local" (user), site or standard library
locations, THEN you return the exception.
This seems overly reasonable, and really quite unlikely to break much. Also,
it places the responsibility on the package writer to do the right thing,
which is very situation dependent... rather than just dictatign it
from "on high".
Thoughts?
Chris
--
| Christopher Petrilli
| petr...@amber.org
Konrad Hinsen
Guido van Rossum <gu...@CNRI.Reston.Va.US> writes:
> I sympathize with your position. I don't know how to enforce it
> either. But it has to be a rule anyway. I see no other way to
> support Windows and Mac. Perhaps the next release of your modules can
> take the form of a package, so you only reserve one top-level name?
Moving to packages is nice, but it also breaks all existing code (even
if it's only a few import statement) and forces all users to upgrade
to 1.5. I guess it will happen at some point, but not immediately.
BTW, is there any statistics about the percentage of Python users
who have switched to 1.5 or will do so shortly?
Guido van Rossum
> It seems to me the best way to do this, and avoid localized naming problems
> is to load packages similar to Perl, which means you handle them this way:
>
> from blah import *
> Here you look along the load path, and grab the first match,
> and follow the __init__.py, and if IT wants to continue on,
> fine, but it's responsible for "correct behavior."
>
> from blah import foo
> Follow the path until you find the package. If you don't find
> the package in the "local" (user), site or standard library
> locations, THEN you return the exception.
Please clarify. Is blah a package or a module? And foo? Suppose
blah is a package -- can there be more than one (e.g. a user blah and
a site blah?)
Christopher G. Petrilli
Sure Guido,
As I understand everything (and I'm no doubt somewhat wrong), blah would
be a package---perhaps a dotted notation---and foo is the module. The
module would then enter the namespace simply as foo, rather than as
blah.foo (hense the purpose of from X import Y).
There COULD be more than one, but only the first one found would be loaded
unless it (thru __init__.py) loaded the parent. I guess this is a bit
like inheritance, with the child overriding the parent...
The compiler would take the first package matching the requested package
in the user's search path (user -> site -> python-standard) and load it.
If it wasn't what the use wanted, that's a different issue.
Hope this makes sense, I'm writing this between packing bags :-) Maybe
this should be discussed at the DC PUG? (I live in Reston)
Guido van Rossum
["Christopher G. Petrilli" <petr...@amber.org>]
> > > It seems to me the best way to do this, and avoid localized naming problems
> > > is to load packages similar to Perl, which means you handle them this way:
> > >
> > > from blah import *
> > > Here you look along the load path, and grab the first match,
> > > and follow the __init__.py, and if IT wants to continue on,
> > > fine, but it's responsible for "correct behavior."
> > >
> > > from blah import foo
> > > Follow the path until you find the package. If you don't find
> > > the package in the "local" (user), site or standard library
> > > locations, THEN you return the exception.
[me]
> > Please clarify. Is blah a package or a module? And foo? Suppose
> > blah is a package -- can there be more than one (e.g. a user blah and
> > a site blah?)
[Christopher again]
> Sure Guido,
>
> As I understand everything (and I'm no doubt somewhat wrong), blah would
> be a package---perhaps a dotted notation---and foo is the module. The
> module would then enter the namespace simply as foo, rather than as
> blah.foo (hense the purpose of from X import Y).
>
> There COULD be more than one, but only the first one found would be loaded
> unless it (thru __init__.py) loaded the parent. I guess this is a bit
> like inheritance, with the child overriding the parent...
>
> The compiler would take the first package matching the requested package
> in the user's search path (user -> site -> python-standard) and load it.
> If it wasn't what the use wanted, that's a different issue.
>
> Hope this makes sense, I'm writing this between packing bags :-) Maybe
> this should be discussed at the DC PUG? (I live in Reston)
What you describe seems to be exactly what Python 1.5 does: only the
first package found would be loaded. Am I missing anything?
Andrew Kuchling
Christopher G. Petrilli wrote:
> > > from blah import foo
> > > Follow the path until you find the package. If you don't find
> > > the package in the "local" (user), site or standard library
> > > locations, THEN you return the exception.
Guido van Rossum wrote:
>What you describe seems to be exactly what Python 1.5 does: only the
>first package found would be loaded. Am I missing anything?
Maybe the first paragraph should distinguish between package and
module more.
Follow the path until you find the module in a corresponding
package direcctory. If you don't find
the module in the "local" (user), site or standard library
locations, THEN you return the exception.
1.5 doesn't do this; it will find the first corresponding package
directory, look for the module, fail to find it, and raise
ImportError; it won't look further along the package for more
corresponding package directories, and I think Christopher is
proposing changing this behaviour.
Barry A. Warsaw
>>>>> "KH" == Konrad Hinsen <hin...@ibs.ibs.fr> writes:
KH> But how should such a rule be enforced?
By user intimidation, as you found out with PDB! :-)
-Barry
M.-A. Lemburg
First something technical: I have the strange feeling that the
last few messages on this thread did not make it through the
pytho...@python.org - USENET gateway... or is my news
host broken again ?
Guido van Rossum wrote:
>
> > Ok, I give up. I was only proposing an empty structure for package
> > authors to use or not. Nothing more. I never said anything about
> > the authors having to use this structure under all circumstances.
>
> There's a second reason why this is problematic. This was explained
> before and I'll try again.
>
> If two different authors both place a (different) module in the same
> package, the following can happen. Assume a sysadmin installs module
> P.A in a directory that's not user-writable. Now a user wants to use
> module P.B, but can't convince the sysadmin to install it. There's a
> directory P somewhere as a subdirectory of a module on his path (P
> itself is *not* on its path, since it is a package), but because it is
> owned by the sysadmin, the user can't copy B.py into directory P. The
> only thing the user can do is create another directory P and place
> B.py there, and insert the parent of this P in his sys.path. But now
> he can no longer import P.A, since Python finds the first
> subndirectory P of a directory on sys.path, and looks only there for
> submodules of package P!
>
> There's a solution for this, but it involves writing a long and
> complicated __init__.py for the user's P directory, which adds the
> other P directory to the package variable __path__.
>
It is actually not all that complicated. I posted a function
called 'requires()' a few days ago on this thread that covers
some of this problem area. All it needs is a way to differentiate
between the site installation and the user installation, e.g.
the site installation could have a prefix 'Site.' and the user
installation a prefix 'User.' (easy to do + needs some adjustments
to sys.path). The function takes care of locating any packages
missing from the site installation.
BTW: These __init__.py files should be handled with great
care, e.g. they should either be empty or only define
very few symbols (solely defined there) -- with
maybe one execption: leaf nodes in the package tree.
> > While it's a good idea to set up a central index of available
> > third party software, I don't see much of a chance for it, since
> > it's full time job administoring such a site.
>
> Agreed -- the central index must be automatically maintained, like
> Gamelan. (Strangely enough, I believe that CPAN is *not*
> automatically updated when new packages are added.)
>
> > That is why I proposed an automated approach some time ago
> > (remember, the Python Source Map entries). This minimizes
> > registration efforts on the authors side and maximizes
> > responsiveness to changes, since a search engine would visit the
> > packages home pages on a regular basis. Of course the engine will
> > have to be written by someone (that someone probably being me, since
> > I proposed it).
>
> Yes -- this would be a great thing to have!
In a few weeks maybe...
>
> > The main idea of the structure wasn't preventing naming conflicts.
> > If just think that it makes programs look more intuitive:
> >
> > from Database.ODBC import Connection
> > dbc = Connection(...,...,...)
> > c = dbc.cursor()
> > c.execute(...)
> >
> > If you don't know what ODBC is then you'd certainly be quite
> > puzzled by a line saying 'from ODBC import Connection'.
>
> Well, I don't think that it will make much of a difference, but this
> is rather personal. As far as I'm concerned, you're just making the
> names longer with little added info in most cases. E.g. do we really
> need UI.Pmw instead of Pmw?
I could ask: 'What's Pmw ?' just as I could ask 'What's PIL ?'.
These names are short (ie. easier to write), but not very meaningful.
Yet the extensions behind those names are very useful and deserve
to be found by beginners too.
>
> > I would register the names 'Datatype', 'Database' and in a few
> > months probably also 'Text' and then release my extensions as subpackages
> > of these top level packages and I'd invite everybody to put their extensions
> > and packages under these if they feel like it is the right thing
> > to do -- fair enough ?
>
> As I explained above, this is probably a bad idea unless you are
> willing to also serve as a central distribution point ("editor") for
> those packages.
There's no need for an editor: the package directory names are
well defined, the __init__.py file is empty (that is, everybody
can overwrite it with another empty file) and thus simple unzipping
over the structure will give the expected results.
>
> As far as grabbing generic words like these, I'd feel more comfortable
> if you didn't. Please don't do it just out of spite.
I wouldn't do it out of spite and I'm not taking them away
from anybody (see above). All I'm doing is posing a restriction
as to what can go into the __init__.py file (it must be empty).
>
> > Well, I guess that's the end of this thread. Strange, when I started
> > it I thought there'd be discussion about the names and maybe some
> > of the organization... not complete rejection. IMHO, it's a lost chance.
>
> Let's lay it to rest.
Ok.
For everybody that followed this thread until now, here is a little
something to cheer you up. It's a shell script I use to wrap
packages in one go (put things not intended for the packages into
subdirectories named 'private/'):
pypkgcreate----------------------------------------------------------
#!/bin/sh
find $2 -follow \( \
-name '*.h' -or -name '*.c' -or -name '*.html' -or \
-name 'Setup.in' -or -name 'Makefile.pre.in' -or -name '*.py' \
\) -and -not \( \
-name 'config.c' -or -path '*private*' \
\) \
| zip -@ $1
--
Marc-Andre Lemburg
Guido van Rossum
> Ok, I give up. I was only proposing an empty structure for package
> authors to use or not. Nothing more. I never said anything about
> the authors having to use this structure under all circumstances.
There's a second reason why this is problematic. This was explained
before and I'll try again.
If two different authors both place a (different) module in the same
package, the following can happen. Assume a sysadmin installs module
P.A in a directory that's not user-writable. Now a user wants to use
module P.B, but can't convince the sysadmin to install it. There's a
directory P somewhere as a subdirectory of a module on his path (P
itself is *not* on its path, since it is a package), but because it is
owned by the sysadmin, the user can't copy B.py into directory P. The
only thing the user can do is create another directory P and place
B.py there, and insert the parent of this P in his sys.path. But now
he can no longer import P.A, since Python finds the first
subndirectory P of a directory on sys.path, and looks only there for
submodules of package P!
There's a solution for this, but it involves writing a long and
complicated __init__.py for the user's P directory, which adds the
other P directory to the package variable __path__.
> While it's a good idea to set up a central index of available
> third party software, I don't see much of a chance for it, since
> it's full time job administoring such a site.
Agreed -- the central index must be automatically maintained, like
Gamelan. (Strangely enough, I believe that CPAN is *not*
automatically updated when new packages are added.)
> That is why I proposed an automated approach some time ago
> (remember, the Python Source Map entries). This minimizes
> registration efforts on the authors side and maximizes
> responsiveness to changes, since a search engine would visit the
> packages home pages on a regular basis. Of course the engine will
> have to be written by someone (that someone probably being me, since
> I proposed it).
Yes -- this would be a great thing to have!
> The main idea of the structure wasn't preventing naming conflicts.
> If just think that it makes programs look more intuitive:
>
> from Database.ODBC import Connection
> dbc = Connection(...,...,...)
> c = dbc.cursor()
> c.execute(...)
>
> If you don't know what ODBC is then you'd certainly be quite
> puzzled by a line saying 'from ODBC import Connection'.
Well, I don't think that it will make much of a difference, but this
is rather personal. As far as I'm concerned, you're just making the
names longer with little added info in most cases. E.g. do we really
need UI.Pmw instead of Pmw?
> I would register the names 'Datatype', 'Database' and in a few
> months probably also 'Text' and then release my extensions as subpackages
> of these top level packages and I'd invite everybody to put their extensions
> and packages under these if they feel like it is the right thing
> to do -- fair enough ?
As I explained above, this is probably a bad idea unless you are
willing to also serve as a central distribution point ("editor") for
those packages.
As far as grabbing generic words like these, I'd feel more comfortable
if you didn't. Please don't do it just out of spite.
> Well, I guess that's the end of this thread. Strange, when I started
> it I thought there'd be discussion about the names and maybe some
> of the organization... not complete rejection. IMHO, it's a lost chance.
Let's lay it to rest. I would still like to hear proposals for a
package structure for the standard library (only). Unfortunately, I'm
worried that discussions about "what to call things" often generate a
lot of suggestions witout much of a conclusion (see e.g. the
discussion on what to name the new starhip domain :-).
Sjoerd Mullender
On Tue, Jan 20 1998 "M.-A. Lemburg" wrote:
> First something technical: I have the strange feeling that the
> last few messages on this thread did not make it through the
> pytho...@python.org - USENET gateway... or is my news
> host broken again ?
Our news server has a bit of a backlog. It'll all clear up. No
messages have been lost (that I am aware of).
-- Sjoerd Mullender <Sjoerd.M...@cwi.nl>
<URL:http://www.cwi.nl/~sjoerd/>
Cedric Adjih
Anthony Baxter <a...@connect.com.au> wrote:
:>>> Guido van Rossum wrote
:> > I still have a big big problem with the way that 1.5 packages work -
:> > if a site ends up with a standard python package, a site version, and
:> > a user version, all called the same thing, it's going to be nightmarish.
:>
:> But you're not supposed to do that!!! Did anyone suggest this?
: Indirectly, yes - if there's one big standard tree of packages, as has
: been suggested, then you're going to end up either having to make
: the tree writable to everyone, having some sort of horrible hack that
: knows what other trees are on the system, or not being able to install
: stuff in the standard tree at all.
Sorry, to jump in by answering randomly to a post, but since there is
discussion about global namespace management, can't a Plan9/Inferno type
of approach be used ?
In these two OSes, there is no such thing as a global (file) namespace;
basically each process can "bind"("mount") directories without affecting
some other process. Binding: if a process binds "/i386/bin" on "/bin" then
every time it would access to for instance "/bin/cat" it would actually
be an access "/i386/bin/cat" while "/bin/cat" would stay the same for
(some?) other processes.
I found this very elegant (this way you could have for instance,
by judicious directories mount, differents versions of emacs/xemacs,
different versions of Linux distributions, or whatever, running
at the same time with each its view of the namespace).
In Python, this would mean that each package may have a different view
of the global namespace, and there should a "bind"-like function.
For instance, each package (comprising different python files) may
(or may not), bind by some means 'mydistribution' to point to its own
files, so that 'import mydistribution.module1' uses its own file 'module1' ;
it could also follow a convention that its type of files should be
found in 'std.goodies.*', and use/test the global one 'std.goodies.module1'
instead. Other packages could also bind the package 'module1' on
'std.goodies.module1' if they wish.
I haven't given much thought to this, since I don't know exactly how
directory/package management is done Python 1.5, but can this be useful
at all ? Won't have it a too big (runtime/semantic) overhead ?
Would it be impossible to do right, because of semantic difficulties ?
Is it already possible in Python1.5 ? Won't it be simply a mess :-) ?
-- Cedric Adjih