Function for the path of the script?

[Peter Cacioppi]

Am I the only one who finds this function super useful?

def _code_file() :
return os.path.abspath(inspect.getsourcefile(_code_file))


I've got one in every script. It's the only one I have to copy around. For my workflow ... so handy.

I've got os.path.dirname aliased to dn, so its dn(_code_file()) that I find myself reaching for fairly often...

[Skip Montanaro]

Strange. I almost never want to know the location of the file that's executing, and when I do, I just reference the __file__ module level attribute.

Skip

[Gary Herron]

Huh? In what kind of a workflow are you running a python file without
knowing *what* file you are runnung?

Or am I just misinterpreting what this code does?

Confused but curious,
Gary Herron

[Chris Angelico]

On Sun, Oct 27, 2013 at 1:52 PM, Gary Herron
<gary....@islandtraining.com> wrote:
> Huh? In what kind of a workflow are you running a python file without
> knowing *what* file you are runnung?

It's very common to want to know what directory you're in - it's a
good way to find data files.

ChrisA

[Steven D'Aprano]

I'm afraid I don't find it useful. I won't say that I *never* need to
check the location of a module, but it's certainly very rare. I'm curious
-- what are you doing that you copy this function into every script?

Also, as given, the above function fails if the source code is not
available (say, you provide only the .pyc file without any corresponding
source). A better way is to inspect the module object's __file__
attribute:

py> import math
py> math.__file__
'/usr/local/lib/python2.7/lib-dynload/math.so'

(Built-in modules which don't exist as a separate file, like sys, have no
__file__ attribute, so be prepared to catch the exception.)


Inside the module itself, the easiest way to get your own path is to
check the global variable __file__.


--
Steven

[Ben Finney]

Peter Cacioppi <peter.c...@gmail.com> writes:

> Am I the only one who finds this function super useful?
>
> def _code_file() :
> return os.path.abspath(inspect.getsourcefile(_code_file))

I've used ‘os.path.dirname(os.path.abspath(__file__))’ to find the
directory containing the current file, in the past. But that was before
Python's ‘unittest’ module could discover where the test code lives.

> I've got one in every script. It's the only one I have to copy around.
> For my workflow ... so handy.

What workflow requires you to know the filename of the module, within
the module?

--
\ “Time wounds all heels.” —Groucho Marx |
`\ |
_o__) |
Ben Finney

[Ben Finney]

Chris Angelico <ros...@gmail.com> writes:

> It's very common to want to know what directory you're in - it's a
> good way to find data files.

That's a naive way to do it (though it's often good enough, for a
program only used on one system).

For programs intending to be used across many systems, the data files
often shouldn't be placed in the same directory as the program.

On systems conforming to the Filesystem Hierarchy Standard, it's
forbidden: programs go in a platform-specific location
<URL:http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA>,
while platform-independent data files go in a separate location
<URL:http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA>.

The ‘package_data’ feature of Distutils (originally from Setuptools)
<URL:http://docs.python.org/2/distutils/setupscript.html#installing-package-data>
doesn't help with this. The specified files are simply installed in a
subdirectory of the platform-specific program location, which violates
the FHS as described above.

So, what facility is there in Python to:

* allow the operating system to set separate installation locations for
the program versus the data files; and

* allow the program to interrogate the operating system for where its
data files are installed

in a way that lets the programmer ignore the specifics of each operating
system, while leaving it up to the system administrator to decide to
place data files separate from program files at install time?

--
\ “I still have my Christmas Tree. I looked at it today. Sure |
`\ enough, I couldn't see any forests.” —Steven Wright |
_o__) |
Ben Finney

[ru...@yahoo.com]

On 10/26/2013 10:28 PM, Ben Finney wrote:
> Chris Angelico <ros...@gmail.com> writes:
>
>> It's very common to want to know what directory you're in - it's a
>> good way to find data files.
>
> That's a naive way to do it (though it's often good enough, for a
> program only used on one system).
>
> For programs intending to be used across many systems, the data files
> often shouldn't be placed in the same directory as the program.
>
> On systems conforming to the Filesystem Hierarchy Standard, it's
> forbidden: programs go in a platform-specific location
> <URL:http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA>,
> while platform-independent data files go in a separate location
> <URL:http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA>.

What about Python applications installed in /opt?
On my systems, unless a third party app is packaged and
distributed by the OS package manager, it goes in /opt
and AFAIK, it is blessed practice to keep all the app
files in the same subdirectory tree there.

[Chris Angelico]

On Sun, Oct 27, 2013 at 3:28 PM, Ben Finney <ben+p...@benfinney.id.au> wrote:
> Chris Angelico <ros...@gmail.com> writes:
>
>> It's very common to want to know what directory you're in - it's a
>> good way to find data files.
>
> That's a naive way to do it (though it's often good enough, for a
> program only used on one system).

I never said it was right for a properly-installed program, just that
it's common - which it definitely is, and this not a Python thing,
it's across all languages. As you say, often good enough for simple
things.

Your main questions definitely still stand though, and I don't know
any answer to them.

ChrisA

[Gregory Ewing]

Ben Finney wrote:
> On systems conforming to the Filesystem Hierarchy Standard, it's
> forbidden: programs go in a platform-specific location
> <URL:http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA>,
> while platform-independent data files go in a separate location
> <URL:http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA>.

But Python code itself is platform-independent, so it
should count as data for the purposes of the FHS,
shouldn't it?

--
Greg

[Roy Smith]

In article <mailman.1636.1382847...@python.org>,

Ben Finney <ben+p...@benfinney.id.au> wrote:

> Peter Cacioppi <peter.c...@gmail.com> writes:
>
> > Am I the only one who finds this function super useful?
> >
> > def _code_file() :
> > return os.path.abspath(inspect.getsourcefile(_code_file))
>
> I've used ‘os.path.dirname(os.path.abspath(__file__))’ to find the
> directory containing the current file, in the past. But that was before
> Python's ‘unittest’ module could discover where the test code lives.
>
> > I've got one in every script. It's the only one I have to copy around.
> > For my workflow ... so handy.
>
> What workflow requires you to know the filename of the module, within
> the module?

We use:

config_dir = os.path.abspath(os.path.dirname(__file__))
songza_dir, _ = os.path.split(config_dir)
top_dir, _ = os.path.split(songza_dir)
assert os.path.exists(top_dir + "/.hg")
return top_dir

in our config files. This gives us the absolute path to the top of our
source tree (I don't remember why we do the double os.path.split()
instead of the more obvious "../.."). We know the file where this code
lives is two levels down from the top of the source tree.

Once we've got the absolute path to the top of the tree, that's used in
all sorts of places to locate data files, and to generate configurations
for other applications (nginx, etc).

[Ian Kelly]

I don't see why Python files should be treated any differently than
other non-binary executables, e.g. shell scripts.

[Ben Finney]

True in most cases, but not all (think extension modules, compiled to
architecture-specific bytecode). But you're right, I over-simplified.

The FHS specifies that the first location isn't only for
platform-specific files, but is also for “object files, libraries, and
internal binaries that are not intended to be executed directly by users
or shell scripts”.

The second location is for “all read-only architecture independent data
files” (where the context makes it clear that “data files” is exclusive
of program files).

So, whether Python bytecode is platform-specific or not, it is relegated
by the FHS to a separate location from data files. The separation
requirement remains, and Python's distutils doesn't support it.

What Python facilities do we have for supporting both install-time
decisions about package install layout, and run-time requirements to
find the files wherever they were installed?

--
\ “Two paradoxes are better than one; they may even suggest a |
`\ solution.” —Edward Teller |
_o__) |
Ben Finney

[Ben Finney]

Ian Kelly <ian.g...@gmail.com> writes:

> I don't see why Python files should be treated any differently than
> other non-binary executables, e.g. shell scripts.

It is an unfortunate artefact of Unix history that “binary” has an
established connotation of “executable”, encompassing even executable
text files.

So the separation I'm drawing attention to in the FHS has nothing to do
with whether the files are text files, and everything to do with whether
they're executable programs and code libraries.

According to the FHS (with which, of course, not every operating system
is bound to conform), executable program libraries belong in an entirely
separate location from non-executable data files.

This thread is about how to best use Python's standard tools to support
that separation on systems following the FHS.

--
\ “If [a technology company] has confidence in their future |
`\ ability to innovate, the importance they place on protecting |
_o__) their past innovations really should decline.” —Gary Barnett |
Ben Finney

[Chris Angelico]

On Mon, Oct 28, 2013 at 10:31 AM, Ben Finney <ben+p...@benfinney.id.au> wrote:
> It is an unfortunate artefact of Unix history that “binary” has an
> established connotation of “executable”, encompassing even executable
> text files.

That's a lot broader than Unix - people talk about "binaries" meaning
executables in Windows and OS/2 too. Unix is, if anything, _less_
inclined that way - the executable segment is called "text", which
always struck me as a bit odd.

> So the separation I'm drawing attention to in the FHS has nothing to do
> with whether the files are text files, and everything to do with whether
> they're executable programs and code libraries.

Yup. Unix does a fairly good job of blurring the line between
"executables that can be loaded and jumped to" and "scripts that get
loaded by an interpreter". I actually have a few scripts that take
several levels of interpreter, something like:

foo.pike
#!/usr/local/bin/pike

bar.pike:
#!/.../foo.pike --parameter

fum.pike:
#!/.../bar.pike --otherparameter

Unix will happily execute ./fum.pike as "/usr/local/bin/pike
/.../foo.pike --parameter /.../bar.pike --otherparameter ./fum.pike".
There's a limit on the number of interpreters (to prevent loops), but
I haven't hit it :)

There is one important place, though, where scripts are called data
files, and that's licensing. The GPL, for instance, does NOT cover
your scripts, even if it covers the interpreter, because *to the
language interpreter*, your scripts are just data files. But that's
more of a legal distinction than a filesystem hierarchical one.

ChrisA

[Mark Lawrence]

On 27/10/2013 23:58, Chris Angelico wrote:
> On Mon, Oct 28, 2013 at 10:31 AM, Ben Finney <ben+p...@benfinney.id.au> wrote:

>> It is an unfortunate artefact of Unix history that �binary� has an
>> established connotation of �executable�, encompassing even executable

Quoting from another thread

"What is the difference between "script" code (like Javascript and
visual) made for the screen (where such magic values are utilized) and
compiled source (made for the machine)?"

This obviously impacts on the discussion above, so how does Unix,
Windows and other operating systems distinguish these with respect to
binary, executable, code library or whatever?

--
Python is the second best programming language in the world.
But the best has yet to be invented. Christian Tismer

Mark Lawrence

[Chris Angelico]

On Mon, Oct 28, 2013 at 11:34 AM, Mark Lawrence <bream...@yahoo.co.uk> wrote:
> "What is the difference between "script" code (like Javascript and visual)
> made for the screen (where such magic values are utilized) and compiled
> source (made for the machine)?"
>
> This obviously impacts on the discussion above, so how does Unix, Windows
> and other operating systems distinguish these with respect to binary,
> executable, code library or whatever?

Suppose you pull up a shell - for argument's sake, let's say it's
Debian GNU/Linux and you're running bash. You get a prompt that ends
with a dollar sign, and you type "ls". What's going to get executed?

* An alias? You might have an internal function defined in your
.bashrc, or maybe a wrapper that adds parameters to your command.

* A bash internal command? The shell might directly interpret what you
specified. (I don't think ls is like that, but time is, on my
systems.)

* An external binary? On my systems, /bin/ls is an executable binary,
compiled and ready to run.

* A script? Another alternative to the shell alias, you could have
/usr/local/bin/ls that does something different, then maybe drops
through to /bin/ls. If it starts with "#!/usr/bin/python", it'll get
dropped through to Python for execution.

Chances are you wouldn't know the difference, as a human executing the
commands. And you shouldn't need to care, except in really weird
circumstances (maybe you broke your Python install and need to type
"/bin/ls" to figure out what's going on).

Most programs, trying to execute code, won't care about the difference
between binaries and scripts, though of course exec*() won't parse
bash aliases or internals. But if you need to distinguish for whatever
reason, the easiest way is to look at the magic numbers, which can be
done with the 'file' command:

rosuav@sikorsky:~$ file `which ls`
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.26,
BuildID[sha1]=0x55f1e005df252708d4c456dcc2c7dccea1006553, stripped

rosuav@sikorsky:~$ file `which zcat`
/bin/zcat: Bourne-Again shell script, ASCII text executable

Executables happily together, regardless of type.

ChrisA

[Roy Smith]

In article <mailman.1678.1382943...@python.org>,

Chris Angelico <ros...@gmail.com> wrote:

> If it starts with "#!/usr/bin/python", it'll get
> dropped through to Python for execution.

Even better (for most purposes), use "#!/usr/bin/env python". What that
does is (slight handwave here) search your PATH to find the same version
of Python you would get if you typed "python" at a shell prompt.

If you've only got a single version of python installed, it doesn't
matter. But it'll matter a lot if you have multiple versions (or even
multiple installations of the same version, but with different sets of
installed modules).

[Chris Angelico]

On Mon, Oct 28, 2013 at 11:51 PM, Roy Smith <r...@panix.com> wrote:
> In article <mailman.1678.1382943...@python.org>,
> Chris Angelico <ros...@gmail.com> wrote:
>
>> If it starts with "#!/usr/bin/python", it'll get
>> dropped through to Python for execution.
>
> Even better (for most purposes), use "#!/usr/bin/env python". What that
> does is (slight handwave here) search your PATH to find the same version
> of Python you would get if you typed "python" at a shell prompt.

Yeah, I'm aware of that... but I dodged a bit of complexity by
hard-coding the path :) The shebang you quote drops it through to env,
which waves its hands vigorously and says "Abracadabra", before
dropping it through to the Python that you forgot you ran 'make
install' instead of 'make altinstall' on. :)

ChrisA

[Grant Edwards]

On 2013-10-27, Chris Angelico <ros...@gmail.com> wrote:
> On Sun, Oct 27, 2013 at 1:52 PM, Gary Herron
><gary....@islandtraining.com> wrote:

>> Huh? In what kind of a workflow are you running a python file without
>> knowing *what* file you are runnung?

I've been writing Python programs for 10+ years. Never have I felt a
need to know the location of the file that's running. I do sometimes
want to know the _name_ of the file (for multipurpose programs), but I
don't ever remember caring where that file was.

> It's very common to want to know what directory you're in - it's a
> good way to find data files.

From a Unix point of view, that's also very wrong. Data files don't
belong in the same directory as the executable.

--
Grant Edwards grant.b.edwards Yow! Is this an out-take
at from the "BRADY BUNCH"?
gmail.com

[Grant Edwards]

On 2013-10-27, Ben Finney <ben+p...@benfinney.id.au> wrote:

> What workflow requires you to know the filename of the module, within
> the module?

If you have a utility that can be used to do several related things,
one way to tell that utility which you want to do is with command line
arguments. For example your utility checks sys.argv[1] for a command
or option flag. Another way is to give the file multiple names, and
check sys.argv[0] to see what name you've been invoked under. The
latter approach not unusual in the Unix world, and has been carried to
a rather astounding extent by the "busybox" project:

http://www.busybox.net/about.html

It's all the command-line utilities you need for a small Unix system
in a single binary: one executable with dozens and dozens of links
(hard or symbolic).

--
Grant Edwards grant.b.edwards Yow! I request a weekend in
at Havana with Phil Silvers!
gmail.com

[Chris Angelico]

On Tue, Oct 29, 2013 at 1:49 AM, Grant Edwards <inv...@invalid.invalid> wrote:
> On 2013-10-27, Chris Angelico <ros...@gmail.com> wrote:
>> It's very common to want to know what directory you're in - it's a
>> good way to find data files.
>
> From a Unix point of view, that's also very wrong. Data files don't
> belong in the same directory as the executable.

For installed programs, maybe; but we run a lot of scripts out of
source control, like a sanity checker that gets run immediately before
a commit. Some of them need data files that are also source-managed,
so we have them in the same directory. Locating those files is usually
easiest done by tailing onto the script's path.

ChrisA

[Ben Finney]

Grant Edwards <inv...@invalid.invalid> writes:

> On 2013-10-27, Ben Finney <ben+p...@benfinney.id.au> wrote:
>
> > What workflow requires you to know the filename of the module, within
> > the module?
>
> If you have a utility that can be used to do several related things,
> one way to tell that utility which you want to do is with command line
> arguments.

That's a case for inspecting the command line.

> For example your utility checks sys.argv[1] for a command or option
> flag. Another way is to give the file multiple names, and check
> sys.argv[0] to see what name you've been invoked under.

Exactly so. This isn't a use case for finding the filesystem location of
the module.

--
\ “We have to go forth and crush every world view that doesn't |
`\ believe in tolerance and free speech.” —David Brin |
_o__) |
Ben Finney

[Peter Cacioppi]

Ben Finney asked

"What workflow requires you to know the filename of the module, within
the module? "

So what I have is a project utility script in my scripts directory. I have a distinct file structure that holds my .py source, my test.py unit tests, and the file based data associated with the unit tests.

Each test.py is in a dedicated directory.

My utility script can be easily tweaked to do a variety of useful things, one of which is leave the interactive session with a variable that points to a testing directory whose unit test failed. It relies on each test.py having a same named function that knows its directory.

It sounds like there is a more pythonic way to do what I am doing, but also that I am not completely out to lunch.

Sounds about par for my python code at this point. As my code used to be almost exclusively out to lunch, I think I am improving (thanks guys!).

[ru...@yahoo.com]

On 10/28/2013 08:49 AM, Grant Edwards wrote:
> > On 2013-10-27, Chris Angelico <ros...@gmail.com> wrote:
>> >> On Sun, Oct 27, 2013 at 1:52 PM, Gary Herron

>> >> It's very common to want to know what directory you're in - it's a
>> >> good way to find data files.
> >
> > From a Unix point of view, that's also very wrong. Data files don't
> > belong in the same directory as the executable.

In the case of applications installed in /opt, data files
quite often go in a subdirectory of the executable (or a
subdirectory of the executable's parent dir) and getting
the executable's dir is the first step to find the data
files.

This is not only *not wrong* but the FHS prohibits locating
such files outside of the /opt subdirectory tree, with some
minor exceptions.

This was pointed out before but since you said you ignore
posts from GG you probably missed it, and will probably miss
this one too, and thus continue to post bad information.

This is a small but illustrative example of why such broad-
brush filtering is a bad idea. But it is your choice.

[Steven D'Aprano]

On Mon, 28 Oct 2013 21:00:39 -0700, rurpy wrote:

> This was pointed out before but since you said you ignore posts from GG
> you probably missed it, and will probably miss this one too, and thus
> continue to post bad information.
>
> This is a small but illustrative example of why such broad- brush
> filtering is a bad idea. But it is your choice.

Yes, it is unfortunate that somebody such as yourself who takes the time
to post well using Google Groups gets tared with the same brush as those
who don't, but then as my granny used to say, if you lie down with dogs
you'll get fleas.

If you don't want to be associated with the typical GG posts, you have
the choice to stop using GG. And just as you make the choice that the
convenience of GG outweighs the annoyance of being filtered, I expect
Grant has made the choice that the convenience of avoiding unwanted GG
posts outweighs the risk of throwing out a useful post or two.

Isn't freedom of choice wonderful? No matter what you choose, you'll
annoy somebody, possibly yourself :-)


--
Steven

[Peter Cacioppi]

Steven said

"Isn't freedom of choice wonderful?"

Didn't somebody once say "we're all adults here". I forget who said that. Eddard Stark? The guy always did keep his head in a crisis.

[Grant Edwards]

On 2013-10-28, Ben Finney <ben+p...@benfinney.id.au> wrote:
> Grant Edwards <inv...@invalid.invalid> writes:
>
>> On 2013-10-27, Ben Finney <ben+p...@benfinney.id.au> wrote:
>>
>> > What workflow requires you to know the filename of the module, within
>> > the module?
>>
>> If you have a utility that can be used to do several related things,
>> one way to tell that utility which you want to do is with command line
>> arguments.
>
> That's a case for inspecting the command line.
>
>> For example your utility checks sys.argv[1] for a command or option
>> flag. Another way is to give the file multiple names, and check
>> sys.argv[0] to see what name you've been invoked under.
>
> Exactly so. This isn't a use case for finding the filesystem location
> of the module.

Indeed. I was answering a question about a use case knowing the
_filename_, not the path for the file.


--
Grant Edwards grant.b.edwards Yow! I wonder if I should
at put myself in ESCROW!!
gmail.com

[Grant Edwards]

On 2013-10-29, Steven D'Aprano <st...@pearwood.info> wrote:
> On Mon, 28 Oct 2013 21:00:39 -0700, rurpy wrote:
>
>> This was pointed out before but since you said you ignore posts from GG
>> you probably missed it, and will probably miss this one too, and thus
>> continue to post bad information.
>>
>> This is a small but illustrative example of why such broad- brush
>> filtering is a bad idea. But it is your choice.
>

[...]

> If you don't want to be associated with the typical GG posts, you
> have the choice to stop using GG. And just as you make the choice
> that the convenience of GG outweighs the annoyance of being filtered,
> I expect Grant has made the choice that the convenience of avoiding
> unwanted GG posts outweighs the risk of throwing out a useful post or
> two.

At the time I made that choice there was a significant problem with
spam being sent using GG posts. I'm told that the list server
attempted to filter out those posts, but those of us who read this via
Usenet got the full broadside of ads and scams. Based on the S/N
ratio of non-spam GG posts, the choice was pretty obvious.

--
Grant Edwards grant.b.edwards Yow! Edwin Meese made me
at wear CORDOVANS!!
gmail.com