organizing your scripts, with plenty of re-use
bukzor
using pieces of them would be called a package, but since this is an
"anti-pattern" according to Guido, apparently I'm wrong-headed here.
(Reference: http://mail.python.org/pipermail/python-3000/2007-April/006793.html
)
Say you have ~50 scripts or so with lots of re-use (importing from
each other a lot) and you want to organize them into folders. How do
you do this simply?
The intent is to have people be able to check out the directly from
CVS and have the scripts "just work", even if they're not directly on
the PYTHONPATH.
This seems to be the best discussion on the topic, but the conclusion
seemed to be that there's no good way. That seems unthinkable
considering python's dedication to simplicity and elegance.
http://groups.google.com/group/comp.lang.python/browse_thread/thread/c44c769a72ca69fa/
It looks like I'm basically restating this post, which sadly got
dropped without further comment:
http://mail.python.org/pipermail/python-3000/2007-April/006814.html
Stef Mientki
> I would assume that putting scripts into a folder with the aim of re-
> using pieces of them would be called a package, but since this is an
> "anti-pattern" according to Guido, apparently I'm wrong-headed here.
> (Reference: http://mail.python.org/pipermail/python-3000/2007-April/006793.html
> )
>
> Say you have ~50 scripts or so with lots of re-use (importing from
> each other a lot) and you want to organize them into folders. How do
> you do this simply?
>
>
... and although I've a working situation, I would like to see other
answers.
In my situation I've an estimate of about 2000 scripts (in fact every
script I ever wrote),
with about zero redundancy.
I still don't use (because I don't fully understand them) packages,
but by trial and error I found a reasonable good working solution,
with the following specifications
- (Almost) any script (what ever it uses from one of the other scripts
can run standalone
- libraries that need another main program ( e.g. a grid component needs
a GUI) can launch another main program to test itself
- All __init__ files are generated automatically
Although not containing the last ideas, here's an idea of what I do:
http://mientki.ruhosting.nl/data_www/pylab_works/pw_importing.html
cheers,
Stef
Steven D'Aprano
> I would assume that putting scripts into a folder with the aim of re-
> using pieces of them would be called a package,
A package is a special arrangement of folder + modules. To be a package,
there must be a file called __init__.py in the folder, e.g.:
parrot/
+-- __init__.py
+-- feeding/
+-- __init__.py
+-- eating.py
+-- drinking.py
+-- fighting.py
+-- flying.py
+-- sleeping.py
+-- talking.py
This defines a package called "parrot" which includes a sub-package
feeding and modules fighting, flying, sleeping and talking. You can use
it by any variant of the following:
import parrot # loads parrot/__init__.py
import parrot.talking # loads parrot/talking.py
from parrot import sleeping
import parrot.feeding
from parrot.feeding.eating import eat_cracker
and similar.
Common (but not compulsory) behaviour is for parrot/__init__.py to import
all the modules within the package, so that the caller can do this:
import parrot
parrot.feeding.give_cracker()
without requiring to manually import sub-packages. The os module behaves
similarly: having imported os, you can immediately use functions in
os.path without an additional import.
Just dumping a bunch of modules in a folder doesn't make it a package, it
just makes it a bunch of modules in a folder. Unless that folder is in
the PYTHONPATH, you won't be able to import the modules because Python
doesn't look inside folders. The one exception is that it will look
inside a folder for a __init__.py file, and if it finds one, it will
treat that folder and its contents as a package.
> but since this is an
> "anti-pattern" according to Guido, apparently I'm wrong-headed here.
> (Reference:
> http://mail.python.org/pipermail/python-3000/2007-April/006793.html )
Guido's exact words were:
"The only use case seems to be running scripts that happen
to be living inside a module's directory, which I've always seen as an
antipattern."
I'm not sure precisely what he means by that, because modules don't have
directories, they are in directories. Perhaps he meant package.
In that case, the anti-pattern according to Guido is not to put modules
in a folder, but to have modules inside a package be executable scripts.
To use the above example, if the user can make the following call from
the shell:
$ python ./parrot/talking.py "polly want a cracker"
and have the module talking do something sensible, that's an anti-
pattern. Modules inside a package aren't intended to be executable
scripts called by the user. There should be one (or more) front-end
scripts which are called by the user. Since they aren't intended to be
imported, they can be anywhere, not just on the PYTHONPATH. But they
import the modules in the package, and that package *is* in the
PYTHONPATH.
Using the above example, you would install the parrot folder and its
contents somewhere on the PYTHONPATH, and then have a front-end script
(say) "talk-to-parrot" somewhere else. Notice that the script doesn't
even need to be a legal name for a module, since you're never importing
it.
> Say you have ~50 scripts or so with lots of re-use (importing from each
> other a lot) and you want to organize them into folders. How do you do
> this simply?
Of course you can have a flat hierarchy: one big folder, like the
standard library, with a mixed back of very loosely connected modules:
eating.py
drinking.py
feeding.py
fighting.py
flying.py
parrot.py
sleeping.py
talking.py
You can do that, of course, but it's a bit messy -- what if somebody
installs parrot.py and eating.py, but not drinking.py, and as a
consequence parrot.py fails to work correctly? Or what if the user
already has a completely unrelated module talking.py? Chaos.
The std library can get away with dumping (nearly) everything in the one
directory, because it's managed chaos. Users aren't supposed to pick and
choose which bits of the standard library get installed, or install other
modules in the same location.
Three alternatives are:
* put your modules in a sub-folder, and tell the user to change the
Python path to include your sub-folder, and hope they know what you're
talking about;
* put your modules in a package, tell the user to just place the entire
package directory where they normally install Python code, and importing
will just work; or
* have each and every script manually manipulate the PYTHONPATH so that
when the user calls that script, it adds its parent folder to the
PYTHONPATH before importing what it needs. Messy and ugly.
--
Steven
Steven D'Aprano
> I still don't use (because I don't fully understand them) packages, but
> by trial and error I found a reasonable good working solution, with the
> following specifications
I find that fascinating. You haven't used packages because you don't
understand them, but you've used another technique that you *also* don't
understand well enough to generate a solution, and had to rely on trial
and error.
Packages are quite well documented. Since the alternative was trial-and-
error on something you also don't fully understand, why did you avoid
packages?
--
Steven
Donn
\d
Stef Mientki
<quote from your other answer>
parrot/
+-- __init__.py
+-- feeding/
+-- __init__.py
+-- eating.py
+-- drinking.py
+-- fighting.py
+-- flying.py
+-- sleeping.py
+-- talking.py
import parrot # loads parrot/__init__.py
import parrot.talking # loads parrot/talking.py
from parrot import sleeping
import parrot.feeding
from parrot.feeding.eating import eat_cracker
</quote>
Instead of the above:
from sleeping import sleeping_in_a_bed
from eating import eat_cracker
anything wrong with that (knowing I've no redundancy) ?
cheers,
Stef
Robert Kern
>
> <quote from your other answer>
>
> parrot/
> +-- __init__.py
> +-- feeding/
> +-- __init__.py
> +-- eating.py
> +-- drinking.py
> +-- fighting.py
> +-- flying.py
> +-- sleeping.py
> +-- talking.py
> import parrot # loads parrot/__init__.py
> import parrot.talking # loads parrot/talking.py
> from parrot import sleeping
> import parrot.feeding
> from parrot.feeding.eating import eat_cracker
>
> </quote>
>
> Instead of the above:
> from sleeping import sleeping_in_a_bed
> from eating import eat_cracker
>
> anything wrong with that (knowing I've no redundancy) ?
With the package layout, you would just do:
from parrot.sleeping import sleeping_in_a_bed
from parrot.feeding.eating import eat_cracker
This is really much more straightforward than you are making it out to be.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
Buck
> With the package layout, you would just do:
>
> from parrot.sleeping import sleeping_in_a_bed
> from parrot.feeding.eating import eat_cracker
>
> This is really much more straightforward than you are making it out to be.
>
As in the OP, I need things to "Just Work" without installation
requirements.
The reason for this is that I'm in a large corporate environment
servicing many groups with their own custom environments.
Your solution requires work and knowledge on the user's part, but Stef
seems to be actually solving the complexity problem. It may seem
trivial to you, but it's beyond many people's grasp and brings the
usability and reliability of the system way down.
Like Stef, I was unable to grasp how to properly use python packages
in my environment even after reading the documentation repeatedly over
the course of several months.
The purpose of this thread is to discover and discuss how to use
packages in a user-space (as opposed to python-installation)
environment.
Robert Kern
>>
>> With the package layout, you would just do:
>>
>> from parrot.sleeping import sleeping_in_a_bed
>> from parrot.feeding.eating import eat_cracker
>>
>> This is really much more straightforward than you are making it out to be.
>>
>
> As in the OP, I need things to "Just Work" without installation
> requirements.
> The reason for this is that I'm in a large corporate environment
> servicing many groups with their own custom environments.
The more ad hoc hacks you use rather than the standard approaches, the harder it
is going to be for you to support those custom environments.
> Your solution requires work and knowledge on the user's part,
*All* solutions require work and knowledge. There is no free lunch. The
advantage of standard Python packages is that they are understood the best and
the most widely.
> but Stef
> seems to be actually solving the complexity problem. It may seem
> trivial to you, but it's beyond many people's grasp and brings the
> usability and reliability of the system way down.
>
> Like Stef, I was unable to grasp how to properly use python packages
> in my environment even after reading the documentation repeatedly over
> the course of several months.
I do believe that you and Stef are exceptions. The vast majority of Python users
seem to be able to grasp packages well enough.
Buck
> On 2009-10-05 12:42 PM, Buck wrote:
>
>
>
> >> With the package layout, you would just do:
>
> >> from parrot.sleeping import sleeping_in_a_bed
> >> from parrot.feeding.eating import eat_cracker
>
> >> This is really much more straightforward than you are making it out to be.
>
> > As in the OP, I need things to "Just Work" without installation
> > requirements.
> > The reason for this is that I'm in a large corporate environment
> > servicing many groups with their own custom environments.
>
> The more ad hoc hacks you use rather than the standard approaches, the harder it
> is going to be for you to support those custom environments.
I too would prefer a standard approach but there doesn't seem to be an
acceptable one.
> I do believe that you and Stef are exceptions. The vast majority of Python users
> seem to be able to grasp packages well enough.
You're failing to differentiate between python programmer and a
system's users. I understand packages well enough, but I need to
reduce the users' requirements down to simply running a command. I
don't see a way to do that as of now without a large amount of
boilerplate code in every script.
I've considered installing the thing to the PYTHONPATH as most people
suggest, but this has two drawbacks:
* Extremely hard to push thru my IT department. Possibly impossible.
* Local checkouts of scripts use the main installation, rather than
the local, possibly revised package code. This necessitates the
boilerplate that installation to the PYTHONPATH was supposed to avoid.
* We can work around the previous point by requiring a user-owned
dev installation of Python, but this raises the bar to entry past most
of my co-developers threshold. They are more comfortable with tcsh and
perl...
I think the issue here is that the current python-package system works
well enough for the core python devs but leaves normal python
developers without much options beyond "all scripts in one directory"
or "tons of boilerplate everywhere".
Robert Kern
> On Oct 5, 11:29 am, Robert Kern<robert.k...@gmail.com> wrote:
>> On 2009-10-05 12:42 PM, Buck wrote:
>>
>>
>>
>>>> With the package layout, you would just do:
>>
>>>> from parrot.sleeping import sleeping_in_a_bed
>>>> from parrot.feeding.eating import eat_cracker
>>
>>>> This is really much more straightforward than you are making it out to be.
>>
>>> As in the OP, I need things to "Just Work" without installation
>>> requirements.
>>> The reason for this is that I'm in a large corporate environment
>>> servicing many groups with their own custom environments.
>>
>> The more ad hoc hacks you use rather than the standard approaches, the harder it
>> is going to be for you to support those custom environments.
>
> I too would prefer a standard approach but there doesn't seem to be an
> acceptable one.
>
>> I do believe that you and Stef are exceptions. The vast majority of Python users
>> seem to be able to grasp packages well enough.
>
> You're failing to differentiate between python programmer and a
> system's users. I understand packages well enough, but I need to
> reduce the users' requirements down to simply running a command. I
> don't see a way to do that as of now without a large amount of
> boilerplate code in every script.
I would like to see an example of such boilerplate. I do not understand why
packages would require more than any other organization scheme.
> I've considered installing the thing to the PYTHONPATH as most people
> suggest, but this has two drawbacks:
> * Extremely hard to push thru my IT department. Possibly impossible.
> * Local checkouts of scripts use the main installation, rather than
> the local, possibly revised package code. This necessitates the
> boilerplate that installation to the PYTHONPATH was supposed to avoid.
> * We can work around the previous point by requiring a user-owned
> dev installation of Python, but this raises the bar to entry past most
> of my co-developers threshold. They are more comfortable with tcsh and
> perl...
Are you sure that you are not referring to site-packages/ when you say "PYTHONPATH"?
PYTHONPATH an environment variable that the user can set. He can add whatever
directories he wants to that environment variable. The user can make a directory
for his checkouts:
$ mkdir ~/LocalToolCheckouts
$ cd ~/LocalToolCheckouts
$ cvs ...
Now add that directory to the front of his PYTHONPATH:
$ export PYTHONPATH=~/LocalToolCheckouts/:$PYTHONPATH
Now everything works fine. Packages in ~/LocalToolCheckouts will get picked up
before anything else. This is a simple no-installation way to use the normal
Python package mechanism that works well if you don't actually need to build
anything.
> I think the issue here is that the current python-package system works
> well enough for the core python devs but leaves normal python
> developers without much options beyond "all scripts in one directory"
> or "tons of boilerplate everywhere".
The "vast majority" I am talking about *are* the normal Python developers.
Margie
I think that Buck's issue with your above example is that he doesn't
want his users to have to type
$ export PYTHONPATH=~/LocalToolCheckouts/:$PYTHONPATH
For example, say that the developer does this:
$ mkdir ~/LocalToolCheckouts
$ cd ~/LocalToolCheckouts
$ cvs ...
Which results in the following directory structure:
~/LocalToolCheckouts
+-- scripts/
+-- myscript.py
+-- parrot/
+-- __init__.py
+-- feeding/
+-- __init__.py
+-- eating.py
+-- drinking.py
I think he is looking for a way for users to be able to use scripts/
myscript.py (which imports parrot) without having to change their
PYTHON path with something like this:
$ export PYTHONPATH=~/LocalToolCheckouts/:$PYTHONPATH
I'm guessing that Buck has users that are running out of a cvs
repository. Although many would say those users are now "developers",
they really are not. They probably don't even know they are running
from a cvs repository. They in fact may think of it as their own
personal installation, and all they know is that they have scripts
directory and that that scripts directory has some scripts they want
to run.
As Buck said, it can often be very difficult to get things properly
and quickly installed in a large corporate environment, and providing
the user with a way to check out a cvs repository can be very quick
and easy. The problem is that once the user has access to that cvs
repository, it is difficult to tell them "hey, every time you run from
it, you need to execute this special command to set up your PYTHONPATH
environment variable."
I don't really see any good solution to this other than some
boilerplate code at the beginning of each script that overrides
PYTHON_PATH. I think what I'd do in this situation is have a dot file
that indicates that you are running out of a "dev" (ie cvs repository)
area. Then some boilerplate code like this at the beginning of your
scripts:
if os.path.exists(os.path.dirname(sys.argv[0]) + "/.dev"):
sys.path.append(os.path.abspath(os.path.dirname(sys.argv[0])) +
"/..")
This basically says "if the .dev file exists, put the parent directory
of the script into sys.path."
When an installation is done, as part of the install the .dev file
should be removed. In that case the libraries should be installed
into the standard site-packages location and the user running from an
install would automatically get their packages from the installation
area.
While having boilerplate code like this at the beginning of the script
is not great, I do think that it will save a lot of user questions/
confusion if the users are frequently running from one or more cvs
repositories. For example, if requiring the user to explictly set
PYTHONPATH, if the user has two cvs repositories, with different
versions of the code, a user that is not very python-knowlegable will
frequently forget to update their PYTHONPATH and will end up running
new scripts with an old library. Having this boilerplate code should
avoid a problem like that. So I think the pros probably outweigh the
cons.
Margie
Buck
To restate my problem more simply:
My core goal is to have my scripts in some sort of organization better
than a single directory, and still have plenty of re-use between them.
The only way I can see to implement this is to have 10+ lines of
unintelligible hard-coded boilerplate in every runnable script.
That doesn't seem reasonable or pythonic.
On Oct 5, 12:34 pm, Robert Kern <robert.k...@gmail.com> wrote:
> I would like to see an example of such boilerplate. I do not understand why
> packages would require more than any other organization scheme.
This example is from the 2007 post I referenced in my OP. I'm pretty
sure he meant 'dirname' rather than 'basename', and even then it
doesn't quite work.
http://mail.python.org/pipermail/python-3000/2007-April/006814.html
import os,sys
sys.path.insert(1, os.path.basename(os.path.basename(__file__)))
This is from a co-worker trying to address this topic:
import os, sys
binpath = binpath or os.path.dirname(os.path.realpath(sys.argv[0]))
libpath = os.path.join(binpath, 'lib')
verinfo = sys.version_info
pythonver = 'python%d.%d' % (verinfo[0], verinfo[1])
sys.path.append(os.path.join(libpath, pythonver, 'site-packages'))
sys.path.append(libpath)
This is my personal code:
from sys import path
from os.path import abspath, islink, realpath, dirname, normpath,
join
f = __file__
#continue working even if the script is symlinked and then compiled
if f.endswith(".pyc"): f = f[:-1]
if islink(f): f = realpath(f)
here = abspath(dirname(f))
libpath = join(here, "..", "lib")
libpath = normpath(libpath)
path.insert(1, libpath)
> $ export PYTHONPATH=~/LocalToolCheckouts/:$PYTHONPATH
> This is a simple no-installation way to use the normal
> Python package mechanism that works well if you don't actually need to build
> anything.
This seems simple to you, but my users are electrical engineers and
know just enough UNIX commands to get by. Most are afraid of Python.
Half of them will assume the script is borked when they see a
"ImportError: No module named foo". Another 20% will then read the
README and
set their environment wrong (setenv PYTHONPATH foo). The rest will get
it to work after half an hour but never use it again because it was
too complicated. I could fix the error message to tell them exactly
what to do, but at that point I might as well re-write the above
boilerplate code.
I'm overstating my case here for emphasis, but it's essentially true.
--Buck
Rami Chowdhury
> Thanks. I think we're getting closer to the core of this.
>
> To restate my problem more simply:
>
> My core goal is to have my scripts in some sort of organization better
> than a single directory, and still have plenty of re-use between them.
> The only way I can see to implement this is to have 10+ lines of
> unintelligible hard-coded boilerplate in every runnable script.
> That doesn't seem reasonable or pythonic.
>
Perhaps I've simply not followed this thread closely enough but could you
let us know a little bit more about how you intend / expect the scripts to
be used?
If there's a standard directory you expect them to be dropped into by your
users (e.g. $HOME/scripts) then surely you could do something like:
import mypathmunge
at the top of every script, and then have a mypathmunge.py in
site-packages that goes:
# mypathmunge.py
import sys, os
sys.path.insert(0, os.path.join(os.getenv('HOME'), 'scripts'))
?
--
Rami Chowdhury
"Never attribute to malice that which can be attributed to stupidity" --
Hanlon's Razor
408-597-7068 (US) / 07875-841-046 (UK) / 0189-245544 (BD)
Robert Kern
> I think he is looking for a way for users to be able to use scripts/
> myscript.py (which imports parrot) without having to change their
> PYTHON path with something like this:
>
> $ export PYTHONPATH=~/LocalToolCheckouts/:$PYTHONPATH
>
> I'm guessing that Buck has users that are running out of a cvs
> repository. Although many would say those users are now "developers",
> they really are not.
Since he called them "co-developers", I was operating under the assumption that
they were, in fact, developers.
> They probably don't even know they are running
> from a cvs repository. They in fact may think of it as their own
> personal installation, and all they know is that they have scripts
> directory and that that scripts directory has some scripts they want
> to run.
>
> As Buck said, it can often be very difficult to get things properly
> and quickly installed in a large corporate environment, and providing
> the user with a way to check out a cvs repository can be very quick
> and easy. The problem is that once the user has access to that cvs
> repository, it is difficult to tell them "hey, every time you run from
> it, you need to execute this special command to set up your PYTHONPATH
> environment variable."
No, you tell them: "Add the line
source ~/LocalToolCheckouts/tool-env.csh
to your ~/.tcshrc and start a new shell."
They each do this once, and it works for all of your scripts with no
boilerplate. You can add features to this shell script, too, like adding the
script directory to their $PATH.
Or you can write a simple "installation" script, which doesn't really install
anything, just adds to their ~/.tcshrc files like so.
Or if this still is beyond your users, put a module into your scripts directory
that does the prepends ~/LocalToolCheckouts/ to sys.path . The only
"boilerplate" you need is one line in each script that imports this module.
But this is all stuff you have to do whether you organize your code into
packages or not. The organization of the code really has no effect on the
installation problems the OP faces.
Carl Banks
> Thanks. I think we're getting closer to the core of this.
>
> To restate my problem more simply:
>
> My core goal is to have my scripts in some sort of organization better
> than a single directory, and still have plenty of re-use between them.
> The only way I can see to implement this is to have 10+ lines of
> unintelligible hard-coded boilerplate in every runnable script.
> That doesn't seem reasonable or pythonic.
Well, ignoring Python packaging conventions isn't reasonable or
Pythonic either, but oh well.
Here's what you should do: Create ONE script that is a single entry
point to the everything. The script would accept a subcommand, set up
the Python path, and invoke the appropirate Python module (NOT script)
corresponding to that subcommand, say by calling a main() function
you've defined within.
Users would invoke the appropirate tool using the appropriate
subcommand (similar to tools like Subversion and GIT).
If you really can't use subcommands, make symlinks to the ONE script,
and have the ONE script check what name it was invoked with.
Notice the key idea in all of this: ONE script. When you design it
that a file can be used either as a script or as a module, you are
asking for trouble.
Carl Banks
Steven D'Aprano
> Notice the key idea in all of this: ONE script. When you design it that
> a file can be used either as a script or as a module, you are asking for
> trouble.
I agree with everything you said in your post *except* that final
comment. The basic idea of modules usable as scripts is a fine, reliable
one, provided all the modules each script calls are visible in the
PYTHONPATH. (You still have problems with recursive imports, but that can
happen with any module, not just scripts.)
The Original Poster is confusing installation difficulties with code
organization -- his problem is that users have special requirements for
installation, and he's trying to work around those requirements by
organizing his code differently.
As far as I can see, whether or not he uses a package, he will still have
the same problem with installation, namely, that his users aren't
developers, plus for non-technical reasons he can't provide an installer
and has to have users check code out of CVS. Using a package will solve
his internal code organization problems, and a simple setup script that
modifies the user's .tcshrc to include the appropriate PYTHONPATH will
solve his other problem. The solution is to work with the language,
instead of fighting the language.
--
Steven
Processor-Dev1l
packages), so I am used to manage such messy files as described above.
But it is true that for building larger solution I always use
packages, it just looks better :).
So my solution:
Packages for job tasks and one big messy folder for my private usage
(system administration tasks, automation, etc).
Gabriel Genellina
<rami.ch...@gmail.com> escribiￜ:
> On Mon, 05 Oct 2009 13:46:09 -0700, Buck <workit...@gmail.com> wrote:
>
>> Thanks. I think we're getting closer to the core of this.
>>
>> To restate my problem more simply:
>>
>> My core goal is to have my scripts in some sort of organization better
>> than a single directory, and still have plenty of re-use between them.
>> The only way I can see to implement this is to have 10+ lines of
>> unintelligible hard-coded boilerplate in every runnable script.
>> That doesn't seem reasonable or pythonic.
>
> If there's a standard directory you expect them to be dropped into by
> your users (e.g. $HOME/scripts) then surely you could do something like:
>
> import mypathmunge
>
> at the top of every script, and then have a mypathmunge.py in
> site-packages that goes:
>
> # mypathmunge.py
> import sys, os
> sys.path.insert(0, os.path.join(os.getenv('HOME'), 'scripts'))
Since Python 2.6 and up, you don't even need that. There exists a per-user
site directory (~/.local/lib/python2.6/site-packages on Linux; under
%APPDATA% on Windows) that is prepended to sys.path automatically; .pth
files found there are honored too (see PEP 370 [1]).
So there is no need to set the PYTHONPATH variable, nor alter the standard
site-packages directory, nor play tricks with sys.path - just install the
modules/packages inside the user site directory (or any other directory
named in a .pth file found there).
[1] http://www.python.org/dev/peps/pep-0370/
--
Gabriel Genellina
catafest
__init__.py
If you want this script may include functions to reading yours scripts
from folder.
Put this script in each folder and you use Margie solution .
This allow you to use import from each folder .
Buck
> On Mon, 05 Oct 2009 13:46:09 -0700, Buck <workithar...@gmail.com> wrote:
> > Thanks. I think we're getting closer to the core of this.
>
> > To restate my problem more simply:
>
> > My core goal is to have my scripts in some sort of organization better
> > than a single directory, and still have plenty of re-use between them.
> > The only way I can see to implement this is to have 10+ lines of
> > unintelligible hard-coded boilerplate in every runnable script.
> > That doesn't seem reasonable or pythonic.
>
> Perhaps I've simply not followed this thread closely enough but could you
> let us know a little bit more about how you intend / expect the scripts to
> be used?
>
> If there's a standard directory you expect them to be dropped into by your
We don't have any such convention. The production code is at a well-
known path, but I need the code to be fully relocatable (cp -r /known/
path ~/mydir) for testing purposes.
Here's a scenario. A user does a cvs checkout into some arbitrary
directory and sees this:
project/
+-- python/
+-- animals.py
+-- mammals/
+-- horse.py
+-- otter.py
+-- reptiles/
+-- gator.py
+-- newt.py
+-- misc/
+-- lungs.py
+-- swimming.py
These are all runnable scripts that "just work" with no extra effort
or knowlege, both in the testing scenario above, and for normal users
that run it from some central location (maybe "/tools/mycompany/bin/
mammals").
The frustrating thing, for me, is that all these requirements are met
if you leave the scripts in jumbled into a flat directory. As soon as
you start organizing things, you need a good amount of boilerplate in
each script to make things work anything like they did with the flat
directory.
I wouldn't mind having to write 'from mammals.horse import Horse' or
'from .animals import Animal' but it's not that simple.
Buck
<ste...@REMOVE.THIS.cybersource.com.au> wrote:
> The Original Poster is confusing installation difficulties with code
> organization -- his problem is that users have special requirements for
> installation, and he's trying to work around those requirements by
> organizing his code differently.
I believe you have it backwards here. My installation requirements (no
installation at all) are met with a flat organization, but as soon as
I try to improve that organization, I can't meet the installation
requirements.
> As far as I can see, whether or not he uses a package, he will still have
> the same problem with installation, namely, that his users aren't
> developers, plus for non-technical reasons he can't provide an installer
> and has to have users check code out of CVS.
That's probably true... I was hoping it wasn't.
> a simple setup script that
> modifies the user's .tcshrc to include the appropriate PYTHONPATH will
> solve his other problem.
I haven't seen a program that needs separate installation procedure
for each user. Maybe mail clients?
But I have seen plenty of programs that can be installed with a simple
copy. Maybe that's just not true for Python projects except with the
simplest of organizations
Carl Banks
Ok, presumably each of these files has something like the following
code inside them, since they can all be both imported and run as
scripts, right?
if __name__ == '__main__':
main()
So follow these steps.
1. Remove this section from all of the files. If the section contains
anything nontrivial, move the nontrivial stuff into your main()
function, or whatever function it is that call. I'll just assume that
the function is called main.
2. In every directory and subdirectory below python, touch an
__init__.py file. It can be empty. You don't need one for the python
directory itself.
3. Change all your imports to package imports. I recommend using
absolute imports only, but that's up to you. "import otter" would
become "from mammals import otter", and so on.
4. Add the following basic script to your Python directory. This is a
script, NOT A MODULE, so don't ever import it. In fact, don't even
give it a py extension if you're not on Windows. I'll call the script
"sa", for "simulate animals". This script accepts the name of a
module to import on the command line, checks whether the top level
module is acceptable, imports the module, then calls the main function
in that module. Enhance as required.
#!/usr/bin/python
import sys
allowed_top_levels = set(('animals','mammals','reptiles','misc'))
modname = sys.argv[1]
modpath = modname.split('.')
if modpath[0] not in allowed_top_levels:
raise RuntimeError('invalid top level module specified')
mod = __import__(modname)
for pkgname in modpath[1:]:
mod = getattr(mod,pkgname)
mod.main() # or main(*sys.argv[2:]), or however you call it
5. Tell your users to stop entering commands like this:
$ ./python/mammal/otter.py
And start entering commands like this:
$ sa mammal.otter
With a little care this method will work fine, and will be installable
with cp -r.
Carl Banks
Steven D'Aprano
> Here's a scenario. A user does a cvs checkout into some arbitrary
> directory and sees this:
>
> project/
> +-- python/
> +-- animals.py
> +-- mammals/
> +-- horse.py
> +-- otter.py
> +-- reptiles/
> +-- gator.py
> +-- newt.py
> +-- misc/
> +-- lungs.py
> +-- swimming.py
>
> These are all runnable scripts that "just work" with no extra effort or
> knowlege, both in the testing scenario above, and for normal users that
> run it from some central location (maybe "/tools/mycompany/bin/
> mammals").
>
> The frustrating thing, for me, is that all these requirements are met if
> you leave the scripts in jumbled into a flat directory.
I bet that's not true. I bet that they Just Work only if the user cd's
into the directory first. In other words, if you have all your scripts in
the directory /tools/mycompany/bin/scripts, this will work:
$ cd /tools/mycompany/bin/scripts
$ animals.py
but this won't:
$ cd /home/username
$ /tools/mycompany/bin/scripts/animals.py
In the first case, it works because the current working directory is
included in the PYTHONPATH, and all the modules you need are there. In
the second, it doesn't because the modules aren't in either the current
directory or any other directory in the PYTHONPATH.
That's my prediction.
> As soon as you
> start organizing things, you need a good amount of boilerplate in each
> script to make things work anything like they did with the flat
> directory.
You shouldn't need that much boilerplate. A little, perhaps, but not that
much.
Although I have defended the practice of making modules executable, I do
recognise that for complex packages this becomes difficult quickly. It
sounds like you would benefit greatly from separating the interface from
the backend. You should arrange matters so that the users see something
like this:
project/
+-- animal
+-- mammal
+-- reptile
+-- backend/
+-- __init__.py
+-- animals.py
+-- mammals/
+-- __init__.py
+-- horse.py
+-- otter.py
+-- reptiles/
+-- __init__.py
+-- gator.py
+-- newt.py
+-- misc/
+-- __init__.py
+-- lungs.py
+-- swimming.py
where the front end is made up of three scripts "animal", "mammal" and
"reptile", and the entire backend is in a package. Each front end script
manages a small amount of boilerplate, something like this:
#!/usr/bin/python
import os, sys
if __name__ == '__main__':
# find out where we are, and add it to the path
location = __import__('__main__').__file__
location = os.path.dirname(location)
if location not in sys.path:
sys.path.append(location)
import animals
animals.main()
That's not a lot of boilerplate for a script.
The backend modules rely on the path being setup correctly. For example,
animals.py might do:
import mammals.horse
horse.ride('like the wind')
Calling the backend modules directly is not supported.
--
Steven
Gabriel Genellina
<st...@remove-this-cybersource.com.au> escribiￜ:
Mmm, I predict you won't have much success in your new fortune teller
career... :)
You got it backwards. At least on Windows, the current directory *isn't*
on the Python path, but the directory containing the script *is* included.
So both alternatives above work.
>> As soon as you
>> start organizing things, you need a good amount of boilerplate in each
>> script to make things work anything like they did with the flat
>> directory.
>
> You shouldn't need that much boilerplate. A little, perhaps, but not that
> much.
>
>
> project/
> +-- animal
> +-- mammal
> +-- reptile
> +-- backend/
> +-- __init__.py
> +-- animals.py
> +-- mammals/
> +-- __init__.py
> +-- horse.py
> +-- otter.py
> +-- reptiles/
> +-- __init__.py
> +-- gator.py
> +-- newt.py
> +-- misc/
> +-- __init__.py
> +-- lungs.py
> +-- swimming.py
>
>
> where the front end is made up of three scripts "animal", "mammal" and
> "reptile", and the entire backend is in a package. Each front end script
> manages a small amount of boilerplate, something like this:
[code omited]
You need even less code. In fact, you don't need any boilerplate code.
This Just Works (tm):
<code>
#!/usr/bin/python
from backend import animals
animals.main()
</code>
That's all. No need to set PYTHONPATH, nor alter sys.path, nor install the
whole tree in a specific place.
The good thing is that, if the backend package is properly installed
somewhere in the Python path, and the animal script is copied onto any
suitable place, it still works with no modifications.
--
Gabriel Genellina
Gabriel Genellina
<st...@remove-this-cybersource.com.au> escribiￜ:
Mmm, I predict you won't have much success in your new fortune teller
career... :)
You got it backwards. At least on Windows, the current directory *isn't*
on the Python path, but the directory containing the script *is* included.
So both alternatives above work.
>> As soon as you
>> start organizing things, you need a good amount of boilerplate in each
>> script to make things work anything like they did with the flat
>> directory.
>
> You shouldn't need that much boilerplate. A little, perhaps, but not that
> much.
>
>
> project/
> +-- animal
> +-- mammal
> +-- reptile
> +-- backend/
> +-- __init__.py
> +-- animals.py
> +-- mammals/
> +-- __init__.py
> +-- horse.py
> +-- otter.py
> +-- reptiles/
> +-- __init__.py
> +-- gator.py
> +-- newt.py
> +-- misc/
> +-- __init__.py
> +-- lungs.py
> +-- swimming.py
>
>
> where the front end is made up of three scripts "animal", "mammal" and
> "reptile", and the entire backend is in a package. Each front end script
> manages a small amount of boilerplate, something like this:
[code omited]
Ethan Furman
> En Sat, 10 Oct 2009 05:57:08 -0300, Steven D'Aprano
> career... :)
> You got it backwards. At least on Windows, the current directory
> *isn't* on the Python path, but the directory containing the script
> *is* included.
> So both alternatives above work.
>
Are you sure?
------
Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', <-- current directory?
'C:\\WINDOWS\\system32\\python25.zip',
'C:\\Python25\\DLLs',
'C:\\Python25\\lib',
'C:\\Python25\\lib\\plat-win',
'C:\\Python25\\lib\\lib-tk',
'C:\\Python25',
'C:\\Python25\\lib\\site-packages',
'C:\\Python25\\lib\\site-packages\\win32',
'C:\\Python25\\lib\\site-packages\\win32\\lib',
'C:\\Python25\\lib\\site-packages\\Pythonwin']
------
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
c:\temp>type print_path.py
import sys
print sys.path
c:\temp>cd \
C:\>python \temp\print_path.py
['C:\\temp', <-- hmmm, not the current directory
'C:\\WINDOWS\\system32\\python25.zip',
'c:\\python25\\DLLs',
'c:\\python25\\lib',
'c:\\python25\\lib\\plat-win',
'c:\\python25\\lib\\lib-tk',
'c:\\python25',
'c:\\python25\\lib\\site-packages',
'c:\\python25\\lib\\site-packages\\win32',
'c:\\python25\\lib\\site-packages\\win32\\lib',
'c:\\python25\\lib\\site-packages\\Pythonwin']
-----
I just love learning new things! Thanks, Gabriel!
Steven D'Aprano
>>> The frustrating thing, for me, is that all these requirements are met
>>> if you leave the scripts in jumbled into a flat directory.
>>
>> I bet that's not true. I bet that they Just Work only if the user cd's
>> into the directory first. In other words, if you have all your scripts
>> in the directory /tools/mycompany/bin/scripts, this will work:
>>
>> $ cd /tools/mycompany/bin/scripts
>> $ animals.py
>>
>> but this won't:
>>
>> $ cd /home/username
>> $ /tools/mycompany/bin/scripts/animals.py
>>
>>
>> In the first case, it works because the current working directory is
>> included in the PYTHONPATH, and all the modules you need are there. In
>> the second, it doesn't because the modules aren't in either the current
>> directory or any other directory in the PYTHONPATH.
>>
>> That's my prediction.
>
> Mmm, I predict you won't have much success in your new fortune teller
> career... :)
> You got it backwards. At least on Windows, the current directory *isn't*
> on the Python path, but the directory containing the script *is*
> included. So both alternatives above work.
Oops. Serves me right for making what I thought was a sure bet before
testing :)
It's the same for Linux too, and it seems to hold at least back to Python
1.5. On the ignominy of it all! I guess I'll have to give up the fortune-
telling and get a proper job :(
--
Steven
Buck
wrote:
> The good thing is that, if the backend package is properly installed
I'd like to get to zero-installation if possible. It's easy with
simple python scripts, why not packages too? I know the technical
reasons, but I haven't heard any practical reasons.
If the reasons are purely technical, it smells like a PEP to me.
Buck
cybersource.com.au> wrote:
> #!/usr/bin/python
> import os, sys
>
> if __name__ == '__main__':
> # find out where we are, and add it to the path
> location = __import__('__main__').__file__
> location = os.path.dirname(location)
> if location not in sys.path:
> sys.path.append(location)
>
> import animals
> animals.main()
>
> That's not a lot of boilerplate for a script.
Does this keep working if a user decides to symlink it to their ~/
bin/? I think it doesn't (untested). Adding that capability adds a few
lines (from my boilerplate previously):
#continue working even if the script is symlinked and then compiled
if f.endswith(".pyc"): f = f[:-1]
if islink(f): f = realpath(f)
My issues with boilerplate are several. It adds possible points of
bugs to the script. It's done differently by everybody, which has
various implications. The supported features are (obviously)
unstandardized. It's untested by the community and so may have bugs
even after a year or two of use.
It also violates the Don't Repeat Yourself (DRY) principle. To fix
this, I might make a packagehelper module or some such, but as soon as
I start organizing my scripts into a hierarchy (the topic of the OP),
I have a catch 22 in that packagehelper is a module that helps me find
modules. This means I need a copy of packagehelper in each script
directory, again violating DRY.
There's a couple other things, but I'll stop myself here.
--Buck
Gabriel Genellina
> On Oct 10, 9:44ï¿œam, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> wrote:
>> The good thing is that, if the backend package is properly installed ￜ
>> somewhere in the Python path ... it still works with no modifications.
>
> I'd like to get to zero-installation if possible. It's easy with
> simple python scripts, why not packages too? I know the technical
> reasons, but I haven't heard any practical reasons.
>
> If the reasons are purely technical, it smells like a PEP to me.
That's what I meant to say. It IS a zero-installation schema, and it also
works if you properly install the package. Quoting Steven D'Aprano
(changing names slightly):
"""You would benefit greatly from separating the interface from
the backend. You should arrange matters so that the users see something
like this:
project/
+-- animal
+-- mammal
+-- reptile
+-- somepackagename/
+-- __init__.py
+-- animals.py
+-- mammals/
+-- __init__.py
+-- horse.py
+-- otter.py
+-- reptiles/
+-- __init__.py
+-- gator.py
+-- newt.py
+-- misc/
+-- __init__.py
+-- lungs.py
+-- swimming.py
where the front end is made up of three scripts "animal", "mammal" and
"reptile", and the entire backend is in a package.""" [ignore the rest]
By example, the `animal` script would contain:
from somepackagename import animals
animals.main()
or perhaps something more elaborate, but in any case, the script imports
whatever it needs from the `somepackagename` package.
The above script can be run:
a) directly from the `project` directory; this could be a checked out copy
from svn, or a tar file extracted in /tmp, or whatever. No need to install
anything, it just works.
b) alternatively, you may install somepackagename into site-packages (or
the user site directory, or any other location along the Python path), and
copy the scripts into /usr/bin (or any other location along the system
PATH), and it still works.
The key is to put all the core functionality into a package, and place the
package where Python can find it. Also, it's a good idea to use relative
imports from inside the package. There is no need to juggle with sys.path
nor even set PYTHONPATH nor import __main__ nor play any strange games; it
Just Works (tm).
--
Gabriel Genellina
Gabriel Genellina
> On Oct 10, 9:44ï¿œam, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> wrote:
>> The good thing is that, if the backend package is properly installed ￜ
>> somewhere in the Python path ... it still works with no modifications.
>
> I'd like to get to zero-installation if possible. It's easy with
> simple python scripts, why not packages too? I know the technical
> reasons, but I haven't heard any practical reasons.
>
> If the reasons are purely technical, it smells like a PEP to me.
That's what I meant to say. It IS a zero-installation schema, and it also
works if you properly install the package. Quoting Steven D'Aprano
(changing names slightly):
"""You would benefit greatly from separating the interface from
the backend. You should arrange matters so that the users see something
like this:
project/
+-- animal
+-- mammal
+-- reptile
+-- somepackagename/
+-- __init__.py
+-- animals.py
+-- mammals/
+-- __init__.py
+-- horse.py
+-- otter.py
+-- reptiles/
+-- __init__.py
+-- gator.py
+-- newt.py
+-- misc/
+-- __init__.py
+-- lungs.py
+-- swimming.py
where the front end is made up of three scripts "animal", "mammal" and
Stef Mientki
> En Mon, 12 Oct 2009 15:24:34 -0300, Buck <workit...@gmail.com>
> escribiￜ:
>
> also works if you properly install the package. Quoting Steven
> D'Aprano (changing names slightly):
>
> the backend. You should arrange matters so that the users see something
> like this:
>
> +-- animal
> +-- mammal
> +-- reptile
> +-- __init__.py
> +-- animals.py
> +-- mammals/
> +-- __init__.py
> +-- horse.py
> +-- otter.py
> +-- reptiles/
> +-- __init__.py
> +-- gator.py
> +-- newt.py
> +-- misc/
> +-- __init__.py
> +-- lungs.py
> +-- swimming.py
>
>
> where the front end is made up of three scripts "animal", "mammal" and
>
> By example, the `animal` script would contain:
>
> from somepackagename import animals
> animals.main()
>
> or perhaps something more elaborate, but in any case, the script
> imports whatever it needs from the `somepackagename` package.
>
> The above script can be run:
>
> a) directly from the `project` directory; this could be a checked out
> copy from svn, or a tar file extracted in /tmp, or whatever. No need
> to install anything, it just works.
>
> b) alternatively, you may install somepackagename into site-packages
> (or the user site directory, or any other location along the Python
> path), and copy the scripts into /usr/bin (or any other location along
> the system PATH), and it still works.
>
> The key is to put all the core functionality into a package, and place
> the package where Python can find it. Also, it's a good idea to use
> relative imports from inside the package. There is no need to juggle
> with sys.path nor even set PYTHONPATH nor import __main__ nor play any
> strange games; it Just Works (tm).
>
I'm not a programmer, I'm just a human ;-)
Hierarchical choices are done on todays knowledge, tomorrow we might
have different views and want/need to arrange things in another way.
An otter may become a reptile ;-)
So from the human viewpoint the following should be possible (and is for
example possible in Delphi)
- I can move the complete project anywhere I like and it should still
work without any modifications (when I move my desk I can still do my work)
- I can move any file in he project to any other place in the project
and again everything should work without any modifications ( when I
rearrange my books, I can still find a specific book)
In my humble opinion if these actions are not possible, there must be
redundant information in the collection. The only valid reason for
redundant information is to perform self healing (or call it error
correction), and here we have a catch-22.
cheers,
Stef Mientki
Gabriel Genellina
> On Oct 10, 1:57ￜam, Steven D'Aprano <st...@REMOVE-THIS->> ￜ ￜ # find out where we are, and add it to the path
>> ￜ ￜ [...]
>>
>> That's not a lot of boilerplate for a script.
>
As pointed out before, all this boilerplate code is actually unnecesary.
Just import the module you want from its package, and you're done
(usually).
--
Gabriel Genellina
Carl Banks
> On Oct 10, 9:44 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> wrote:
>
> > The good thing is that, if the backend package is properly installed
> > somewhere in the Python path ... it still works with no modifications.
>
> I'd like to get to zero-installation if possible. It's easy with
> simple python scripts, why not packages too? I know the technical
> reasons, but I haven't heard any practical reasons.
No it's purely technical. Well mostly technical (there's a minor
issue of how a script would figure out its "root"). No language is
perfect, not even Python, and sometimes you just have to deal with
things the way they are.
We're trying to help you with workarounds, but it seems like you just
want to vent more than you want an actual solution.
If it makes you feel better, go ahead and vent. If you want a
solution, try instead to sit down and implement the advice Steven,
Gabriel, or I gave you.
> If the reasons are purely technical, it smells like a PEP to me.
Good luck with that. I'd wholeheartedly support a good alternative, I
just want to warn you that it's not a simple issue to fix, it would be
involve spectacular and highly backwards-incompatible changes.
Carl Banks
Ethan Furman
> Gabriel Genellina wrote:
[snip]
>>
>> That's what I meant to say. It IS a zero-installation schema, and it
>> also works if you properly install the package. Quoting Steven
>> D'Aprano (changing names slightly):
>>
>> """You would benefit greatly from separating the interface from
>> the backend. You should arrange matters so that the users see something
>> like this:
>>
>> project/
>> +-- animal
>> +-- mammal
>> +-- reptile
>> +-- somepackagename/
>> +-- __init__.py
>> +-- animals.py
>> +-- mammals/
>> +-- __init__.py
>> +-- horse.py
>> +-- otter.py
>> +-- reptiles/
>> +-- __init__.py
>> +-- gator.py
>> +-- newt.py
>> +-- misc/
>> +-- __init__.py
>> +-- lungs.py
>> +-- swimming.py
>>
[snip]
>> The key is to put all the core functionality into a package, and place
>> the package where Python can find it. Also, it's a good idea to use
>> relative imports from inside the package. There is no need to juggle
>> with sys.path nor even set PYTHONPATH nor import __main__ nor play any
>> strange games; it Just Works (tm).
>>
> please don't get angry,
> I'm not a programmer, I'm just a human ;-)
>
> Hierarchical choices are done on todays knowledge, tomorrow we might
> have different views and want/need to arrange things in another way.
> An otter may become a reptile ;-)
> So from the human viewpoint the following should be possible (and is for
> example possible in Delphi)
> - I can move the complete project anywhere I like and it should still
> work without any modifications (when I move my desk I can still do my work)
Move a complete package anywhere along the PYTHONPATH and it will still
work. Check.
> - I can move any file in he project to any other place in the project
> and again everything should work without any modifications ( when I
> rearrange my books, I can still find a specific book)
Move any file in any directory to any other spot in that same directory
and it will still work. Check. ;-)
Humans are a lot smarter than computers. Even 'just humans'. ;-) If
you move your book, then can't find it on the last shelf it used to be
on, you look on other shelves, you look on your desk, you look on the
coffe table, you look in your car, etc, etc, and so forth. If you move
a file in a package to somewhere else, and you don't tell the package
where it's at, it's not going to start looking all over the hard-drive
for it. If that were the case you would have to be extra careful to
have every module's name be distinct, and then what's the point of
having packages?
~Ethan~
greg
> - I can move the complete project anywhere I like and it should still
> work without any modifications (when I move my desk I can still do my work)
Gabriel's organisation satisfies that.
> - I can move any file in he project to any other place in the project
> and again everything should work without any modifications
That's not a reasonable thing to expect from Python. The
position of a module in the module naming hierarchy is
implied by the location of its file in the directory
hierarchy. When you move the file, you change the name
of the module, so you have to change any import statements
that refer to it accordingly.
For example, if you decide that otters are now lizards
and move mammals/otter.py into the lizards directory,
the module is now called lizards.otter instead of
mammals.otter.
> ( when I
> rearrange my books, I can still find a specific book)
Only because you've updated the index in your brain that
maps book titles to their locations. If you don't do that,
you lose track of the book and have to search for it.
The Python equivalent of this is updating import statements
to reflect the new location of the module.
> In my humble opinion if these actions are not possible, there must be
> redundant information in the collection.
Actually, it's because Python *doesn't* keep redundant
information. To be able to move files around without changing
anything else, Python would have to keep an index somewhere
mapping module names to files. But there is no such index --
the directory structure *is* the index.
--
Greg
Gabriel Genellina
<wlf...@ix.netcom.com> escribiￜ:
> On Mon, 12 Oct 2009 16:36:58 -0700, Ethan Furman <et...@stoneleaf.us>
> declaimed the following in gmane.comp.python.general:
>
>> coffe table, you look in your car, etc, etc, and so forth. If you move
>> a file in a package to somewhere else, and you don't tell the package
>> where it's at, it's not going to start looking all over the hard-drive
>> for it. If that were the case you would have to be extra careful to
>> have every module's name be distinct, and then what's the point of
>> having packages?
>>
> some earlier incarnations too) WOULD search all active drives if no
> drive letter was specified (no subdirectories, of course -- and I never
> had a hard drive on mine [$5000 for a 5MB drive?], just the two
> floppies). And for creating files, again if no drive were specified, it
> would create the file on the first drive that was not write-protected.
In the old MSDOS era, there was the APPEND command. It was used to set a
list of directories or subdirectories that were searched for data files
(in a way similar as PATH works for executable files). For example, after
APPEND c:\pirulo\data;doc
you could edit a file like c:\foo\doc\readme.txt directly from the c:\foo
directory:
c:\foo> edit readme.txt
and because of "doc" being in the APPEND path, edit would find it. I think
Stef would enjoy using it - just listing each subdirectory would make the
disk tree completely flat as seen by the application.
I think this functionality was removed by the time Windows 95 came out
because it was very dangerous. It was extremely easy to open (or even
remove!) the wrong file.
--
Gabriel Genellina
Buck
> On Oct 12, 11:24 am, Buck <workithar...@gmail.com> wrote:
>
> > On Oct 10, 9:44 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> > wrote:
>
> > > The good thing is that, if the backend package is properly installed
> > > somewhere in the Python path ... it still works with no modifications.
>
> > I'd like to get to zero-installation if possible. It's easy with
> > simple python scripts, why not packages too? I know the technical
> > reasons, but I haven't heard any practical reasons.
>
> No it's purely technical. Well mostly technical (there's a minor
> issue of how a script would figure out its "root"). No language is
> perfect, not even Python, and sometimes you just have to deal with
> things the way they are.
Python is the closest I've seen. I'd like to deal with this wart if we
can.
> We're trying to help you with workarounds, but it seems like you just
> want to vent more than you want an actual solution.
Steven had the nicest workaround (with the location = __import__
('__main__').__file__ trick), but none of them solve the problem of
the OP: organization of runnable scripts. So far it's been required to
place all runnable scripts directly above any used packages. The
workaround that Gabriel has been touting requires this too.
Maybe it seems like I'm venting when I shoot down these workarounds,
but my real aim is to find some sort of consensus, either that there
is a solution, or an unsolved problem. I'd be delighted with a
solution, but none have been acceptable so far (as I explained in
aggravating detail earlier).
If I can find consensus that this is a real problem, not just my
personal nit-pick, then I'd be willing to donate my time to design,
write and push through a PEP for this purpose. I believe it can be
done neatly with just three new standard functions, but it's premature
to discuss that.
> > If the reasons are purely technical, it smells like a PEP to me.
>
> Good luck with that. I'd wholeheartedly support a good alternative,
Thanks.
> I just want to warn you that it's not a simple issue to fix, it would be
> involve spectacular and highly backwards-incompatible changes.
> --Carl Banks
I don't believe that's true, but I think that's a separate discussion.
Buck
wrote:
> En Mon, 12 Oct 2009 15:24:34 -0300, Buck <workithar...@gmail.com> escribió:
>
> > On Oct 10, 9:44 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> > wrote:
> >> The good thing is that, if the backend package is properly installed
Hi Gabriel. This is very thoughtful. Thanks.
As in the OP, when I have 50 different runnable scripts, it become
necessary to arrange them in directories. How would you do that in
your scheme? Currently it looks like they're required to live directly
above the package containing their code.
--Buck
Ethan Furman
[snip]
> Steven had the nicest workaround (with the location = __import__
> ('__main__').__file__ trick), but none of them solve the problem of
> the OP: organization of runnable scripts. So far it's been required to
> place all runnable scripts directly above any used packages. The
> workaround that Gabriel has been touting requires this too.
[snip]
Wha? "Place all runnable scripts directly above any used packages?" I
must have missed something major in this thread. The only thing
necessary is to have the package being imported to be somewhere in
PYTHONPATH.
Previous post:
>I'd like to get to zero-installation if possible. It's easy with
>simple python scripts, why not packages too? I know the technical
>reasons, but I haven't heard any practical reasons.
I don't think we mean the same thing by "zero-installation"... seems to
me that if you have to copy it, check it out, or anything to get the
code from point A to point 'usable on your computer', then you have done
some sort of installation.
~Ethan~
Gabriel Genellina
> On Oct 12, 3:34ï¿œpm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> wrote:
>> Quoting Steven D'Aprano ￜ
>> (changing names slightly):
>>
>> """You would benefit greatly from separating the interface from
>> the backend. You should arrange matters so that the users see something
>> like this:
>>
>> project/
>> +-- animal
>> +-- mammal
>> +-- reptile
>> +-- somepackagename/
>> ￜ ￜ ￜ+-- animals.py
>> ￜ ￜ ￜ+-- mammals/
>> ￜ ￜ ￜ ￜ ￜ+-- __init__.py
>> ￜ ￜ ￜ ￜ ￜ+-- horse.py
>> ￜ ￜ ￜ ￜ ￜ+-- otter.py
>> ￜ ￜ ￜ+-- reptiles/
>> ￜ ￜ ￜ ￜ ￜ+-- __init__.py
>> ￜ ￜ ￜ ￜ ￜ+-- gator.py
>> ￜ ￜ ￜ ￜ ￜ+-- newt.py
>> ￜ ￜ ￜ+-- misc/
>> ￜ ￜ ￜ ￜ ￜ+-- __init__.py
>> ￜ ￜ ￜ ￜ ￜ+-- lungs.py
>> ￜ ￜ ￜ ￜ ￜ+-- swimming.py
>>
>> where the front end is made up of three scripts "animal", "mammal" and
>> "reptile", and the entire backend is in a package.""" [ignore the rest]
>>
>> By example, the `animal` script would contain:
>>
>> animals.main()
>>
>> or perhaps something more elaborate, but in any case, the script
>> whatever it needs from the `somepackagename` package.
>>
>> The above script can be run:
>>
>> a) directly from the `project` directory; this could be a checked out
>> ï¿œfrom svn, or a tar file extracted in /tmp, or whatever. No need to
>> install ￜ
>> anything, it just works.
>>
>> b) alternatively, you may install somepackagename into site-packages
>> the user site directory, or any other location along the Python path),
>> copy the scripts into /usr/bin (or any other location along the system ￜ
>> PATH), and it still works.
>>
>> The key is to put all the core functionality into a package, and place
>> package where Python can find it. Also, it's a good idea to use
>> imports from inside the package. There is no need to juggle with
>> nor even set PYTHONPATH nor import __main__ nor play any strange games;
>> Just Works (tm).
>
> As in the OP, when I have 50 different runnable scripts, it become
> necessary to arrange them in directories. How would you do that in
> your scheme? Currently it looks like they're required to live directly
> above the package containing their code.
I'd say that, if your application contains 50 different runnable scripts
in several directories, it deserves an install script. Being able to
execute something from a checked-out copy is fine for a small application
but in your case I'd say we are off limits...
Python must be able to import the package. In case a) above, this happens
because the directory containing the script (and the package) is in
sys.path by default. In case b), because you copy the package into some
place that is already in sys.path. There is another option: add the
directory containing the package to sys.path, by setting the environment
variable PYTHONPATH. Robert Kern already explained how to do that:
<http://permalink.gmane.org/gmane.comp.python.general/639964>
--
Gabriel Genellina
Stef Mientki
>
> [snip]
>
>>> The key is to put all the core functionality into a package, and
>>> to use relative imports from inside the package. There is no need to
>>> juggle with sys.path nor even set PYTHONPATH nor import __main__ nor
>>> play any strange games; it Just Works (tm).
>>>
>> please don't get angry,
>> I'm not a programmer, I'm just a human ;-)
>>
>> Hierarchical choices are done on todays knowledge, tomorrow we might
>> have different views and want/need to arrange things in another way.
>> An otter may become a reptile ;-)
>> So from the human viewpoint the following should be possible (and is
>> for example possible in Delphi)
>
>> work without any modifications (when I move my desk I can still do my
>> work)
>
> still work. Check.
>
>
> directory and it will still work. Check. ;-)
>
> Humans are a lot smarter than computers. Even 'just humans'. ;-) If
> you move your book, then can't find it on the last shelf it used to be
> on, you look on other shelves, you look on your desk, you look on the
> move a file in a package to somewhere else, and you don't tell the
> package where it's at, it's not going to start looking all over the
> hard-drive for it.
> If that were the case you would have to be extra careful to have
> every module's name be distinct, and then what's the point of having
> packages?
cheers,
Stef
>
> ~Ethan~
>
>> In my humble opinion if these actions are not possible, there must be
Stef Mientki
>
>
> On Mon, Oct 12, 2009 at 4:15 PM, Stef Mientki <stef.m...@gmail.com
> <mailto:stef.m...@gmail.com>> wrote:
>
> Hierarchical choices are done on todays knowledge, tomorrow we
> might have different views and want/need to arrange things in
> another way.
> An otter may become a reptile ;-)
> So from the human viewpoint the following should be possible (and
> is for example possible in Delphi)
> - I can move the complete project anywhere I like and it should
> still work without any modifications (when I move my desk I can
> still do my work)
>
>
> only need to ensure where you move it ends up in the PYTHONPATH-- but
> that'll take at most just one change, once. You can use batch scripts
> to alter it if you really need to, but that's all you would need to do.
>
>
> - I can move any file in he project to any other place in the
> project and again everything should work without any modifications
> ( when I rearrange my books, I can still find a specific book)
>
> be redundant information in the collection. The only valid reason
> for redundant information is to perform self healing (or call it
> error correction), and here we have a catch-22.
>
>
> programming in Python needs to be excised. Things don't work like
> that, and trying to make it work like that will result in you going
> against the grain of /how/ Python works, and life will become difficult.
I agree it's difficult at the start, but after that,
you can create a new script / module everywhere you like ( within the
projects), and every module is available,
and you can use (almost) any file as a self running program or as a
library module.
>
> Packages are meaningful in Python. They aren't just directories which
> you're organizing for your human-programmer's convenience and
> understanding. They are a fundamental structure of objects and must be
> treated as such.
Well I missed that, I can't see the "higher" meaning of packages,
so if someone has a good link that explains the "higher" meaning of
packages,
I'ld be much obliged.
>
> Python doesn't think in terms of files. You can't just move files
> around within a project because that's a semantically significant change.
>
> It's just like you can't move a method on a class to another class,
> and expect everything to just work.
No, that's like tearing a page from a book and gluing it in another book
and the story get's
So the difference in my reasoning might be a mis-concept of the largest
atomic part, which in my view is a class / function.
>
> A package is just like a class, its a significant, structural
> component in the code itself. It contains files, but those files
> aren't what's significant to Python-- what's significant is that they
> are modules, which are also just like a class. They're objects which
> contain other objects.
>
> Every package creates a namespace of objects it contains; those
> objects are modules (and other things that may be defined in
> __init__.py). Every module creates a namespace of objects it contains.
> Classes create a namespace of objects they contain.
>
> These are all objects. A package isn't a directory you're organizing
> -- its an object which contains other objects.
Well I'm completely lost now, but that's probably because I'm not a
programmer,
and I don't know the exact meaning of these words.
So I don't see a class as an object, just as a definition of
"functionality".
A module, a file, a package, a directory (with it's subdirectories) is
just a collector for classes.
I can freely move a class around within any of these containers without
affecting the functionality of that class or anything that really uses
that class.
>
> If you try to change this so that there's no object organization of an
> object hierarchy, then you're going to create significant issues for
> maintenance and have to jump through all kinds of hoops to try to
> accomplish it. It sounds like you want a system where each file is a
> unique module, and its particular location isn't important-- the
> filename creates a unique identity, a top-level namespace which can be
> accessed from anywhere.
yes that's about the view I've now.
>
> Python's object model just doesn't work like that. There's nothing
> wrong with how Python works in this regard, its simply different, and
> if you try to program Delphi in Python, you'll run into lots of
> problems. Just like if you try to program Java or C in Python. They're
> different languages with different object models, and you have to
> adapt to the object model of the language if you want to use the
> language effectively.
>
I would love to hear one (and preferable more ;-) feature that a package
adds to a collection of classes.
cheers,
Stef
> HTH,
>
> --S
Buck
> Buck wrote:
> >I'd like to get to zero-installation if possible. It's easy with
> >simple python scripts, why not packages too? I know the technical
> >reasons, but I haven't heard any practical reasons.
>
> I don't think we mean the same thing by "zero-installation"... seems to
> me that if you have to copy it, check it out, or anything to get the
> code from point A to point 'usable on your computer', then you have done
> some sort of installation.
I think most people would agree that installation is whatever you need
to do between downloading the software and being able to use it. For
GNU packages, it's './configure && make && make install'. For Python
packages, it's usually './setup.py install'.
> > Steven had the nicest workaround (with the location = __import__
> > ('__main__').__file__ trick), but none of them solve the problem of
> > the OP: organization of runnable scripts. So far it's been required to
> > place all runnable scripts directly above any used packages. The
> > workaround that Gabriel has been touting requires this too.
>
> Wha? "Place all runnable scripts directly above any used packages?" I
> must have missed something major in this thread. The only thing
> necessary is to have the package being imported to be somewhere in
> PYTHONPATH.
The only way to get your packages on the PYTHONPATH currently is to:
* install the packages to site-packages (I don't have access)
* edit the PYTHONPATH all users' environment (again, no access)
* create some boilerplate that edits sys.path at runtime (various
problems in previous post)
* put your scripts directly above the package (this seems best so
far, but forces a flat hierarchy of scripts)
Gabriel Genellina
> The only way to get your packages on the PYTHONPATH currently is to:
> * install the packages to site-packages (I don't have access)
> * edit the PYTHONPATH all users' environment (again, no access)
> * create some boilerplate that edits sys.path at runtime (various
> problems in previous post)
> * put your scripts directly above the package (this seems best so
> far, but forces a flat hierarchy of scripts)
Not exactly - in short, you have to place the package under some directory
that is eventually listed in sys.path.
By default, one of such directories is site-packages, but there are also
per-user directories. On Windows, %APPDATA%\Python\PythonNN\site-packages
(see PEP 370 [1]). If you don't want to copy the package there, you can
even add a .pth file and it will be processed.
[1] http://www.python.org/dev/peps/pep-0370/
--
Gabriel Genellina
bukzor
wrote:
> En Tue, 13 Oct 2009 17:38:44 -0300, Buck <workithar...@gmail.com> escribió:
>
> > The only way to get your packages on the PYTHONPATH currently is to:
> > * install the packages to site-packages (I don't have access)
> > * edit the PYTHONPATH all users' environment (again, no access)
> > * create some boilerplate that edits sys.path at runtime (various
> > problems in previous post)
> > * put your scripts directly above the package (this seems best so
> > far, but forces a flat hierarchy of scripts)
>
> Not exactly - in short, you have to place the package under some directory
> that is eventually listed in sys.path.
> By default, one of such directories is site-packages, but there are also
> per-user directories. On Windows, %APPDATA%\Python\PythonNN\site-packages
> (see PEP 370 [1]). If you don't want to copy the package there, you can
> even add a .pth file and it will be processed.
>
The .pth files are intriguing. Is this the best reference?
http://docs.python.org/library/site.html
My current solution very closely resembles the .pth system, but uses
the scripts' directory and accepts relative paths.
If I'm reading the doc correctly, the .pth system currently doesn't
support either of these. Does anyone know of the rationale for this?
bukzor
> On Oct 13, 3:20 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> wrote:
>
>
>
>
>
> > En Tue, 13 Oct 2009 17:38:44 -0300, Buck <workithar...@gmail.com> escribió:
>
> > > The only way to get your packages on the PYTHONPATH currently is to:
> > > * install the packages to site-packages (I don't have access)
> > > * edit the PYTHONPATH all users' environment (again, no access)
> > > * create some boilerplate that edits sys.path at runtime (various
> > > problems in previous post)
> > > * put your scripts directly above the package (this seems best so
> > > far, but forces a flat hierarchy of scripts)
>
> > Not exactly - in short, you have to place the package under some directory
> > that is eventually listed in sys.path.
> > By default, one of such directories is site-packages, but there are also
> > per-user directories. On Windows, %APPDATA%\Python\PythonNN\site-packages
> > (see PEP 370 [1]). If you don't want to copy the package there, you can
> > even add a .pth file and it will be processed.
>
>
> My current solution very closely resembles the .pth system, but uses
> the scripts' directory and accepts relative paths.
> If I'm reading the doc correctly, the .pth system currently doesn't
> support either of these. Does anyone know of the rationale for this?
I thought this was one of the least objectionable of my posts.
Does anyone know how to find a discussion of the .pth implementation?