Proper shebang for python3

[Manfred Lotz]

Hi there,
Pretty new to python I've got a question regarding the proper shebang
for Python 3.

I use
#!/usr/bin/python3

which works fine.

Today I saw
#!/usr/bin/python3 -tt

and was wondering what -tt means.

Being on Fedora 30, Python 3.7.3 the man page of python3 doesn't even
mention -t.

python 2 man page mentions

-t Issue a warning when a source file mixes tabs and spaces
for indentation in a way that makes it depend on the worth
of a tab expressed in spaces. Issue an error when the option is
given twice.

I guess that -t has the same meaning with python 3.7.3.


My questions:

1. Is my guess correct?

2. Is it a bug that it is not mentioned? python3 --help doesn't mention
it either.


--
Manfred

[Chris Angelico]

On Sun, Jul 21, 2019 at 3:41 AM Manfred Lotz <ml_...@posteo.de> wrote:
>
> Hi there,
> Pretty new to python I've got a question regarding the proper shebang
> for Python 3.
>
> I use
> #!/usr/bin/python3
>
> which works fine.
>
> Today I saw
> #!/usr/bin/python3 -tt
>
> and was wondering what -tt means.
>
> Being on Fedora 30, Python 3.7.3 the man page of python3 doesn't even
> mention -t.
>
> python 2 man page mentions
>
> -t Issue a warning when a source file mixes tabs and spaces
> for indentation in a way that makes it depend on the worth
> of a tab expressed in spaces. Issue an error when the option is
> given twice.
>
> I guess that -t has the same meaning with python 3.7.3.

Far as I know, the -tt option to Python 3 is accepted for 2/3
compatibility, but does absolutely nothing. Having it on the shebang
probably means that it used to be /usr/bin/python -tt, and got
migrated to python3 without removing it.

ChrisA

[Manfred Lotz]

This is a plausible explanation.

Thanks, Manfred

[Michael Speer]

You may want to use `#!/usr/bin/env python3` instead.

There is a concept in python called the virtual environment. This used to
be done with a tool called virtualenv in python2, and is now done mainly
through a venv module in python3.

A virtual environment goes into a directory of your choosing and will have
its own python3 executable, and pip3 executable, and when you add
dependencies, they are also placed into the directory structure under your
chosen directory.

When you do a `. <directory>/bin/activate` the included source will places
the virtual environment's bin/ folder at the beginning of your PATH
environment variable, making it the default python3 when you type it
without a full path.

This allows you to run scripts that need different, or even conflicting,
sets of dependencies without bothering with the underlying linux
distribution's python installation's modules.

If you use `#!/usr/bin/python3`, it will always use exactly the system
version that is installed, and the system's installed modules.

Your scripts will still default to the system installation if a virtual
environment is not activated. So you lose nothing by doing it this way, but
gain a little control from it.

> --
> https://mail.python.org/mailman/listinfo/python-list
>

[Chris Angelico]

On Sun, Jul 21, 2019 at 4:13 AM Michael Speer <knom...@gmail.com> wrote:
>
> You may want to use `#!/usr/bin/env python3` instead.
>
> There is a concept in python called the virtual environment. This used to
> be done with a tool called virtualenv in python2, and is now done mainly
> through a venv module in python3.
>
> A virtual environment goes into a directory of your choosing and will have
> its own python3 executable, and pip3 executable, and when you add
> dependencies, they are also placed into the directory structure under your
> chosen directory.
>
> When you do a `. <directory>/bin/activate` the included source will places
> the virtual environment's bin/ folder at the beginning of your PATH
> environment variable, making it the default python3 when you type it
> without a full path.
>
> This allows you to run scripts that need different, or even conflicting,
> sets of dependencies without bothering with the underlying linux
> distribution's python installation's modules.
>
> If you use `#!/usr/bin/python3`, it will always use exactly the system
> version that is installed, and the system's installed modules.
>
> Your scripts will still default to the system installation if a virtual
> environment is not activated. So you lose nothing by doing it this way, but
> gain a little control from it.
>

ONLY if you actually want this script to behave differently inside a
venv. When you're setting a shebang on a script, you often do not want
that. You potentially LOSE a lot of control.

ChrisA

[Tim Daneliuk]

Disagree strongly. The environment should always define where you want to find key
binaries, whether in a venv or not. There are many use cases where you want to
override system binaries. For example, you may be running on an older OS release
and want to point to a newer one installed elsewhere. Similarly, the DevOps workflow
may demand particular configurations for the pipelines to run properly.

I have spent way too much time in my career trying to undo this kind of imperious
programming that thinks the coder knows best, when in the actual runtime environment,
they actually don't. Most recently, I have been working to produce a configuration
for linuxbrew that can be installed by an unprivileged user at any location they like.
It is maddening to spend hours getting things working only to find out that some genius
decided that the only version of, say, perl or autoconf, is to be found exclusively in
/usr/bin.

So, no, do NOT encode the hard location - ever. Always use env to discover the one that
the user has specified. The only exception is /bin/sh which - for a variety of reasons -
can reliably counted upon.

We don't need to bikeshed this. All we need is people who disagree with this view to spend
a year in software packaging, operations, deployment and DevOps ... then get back to us...

Grrrrrrrrrr......

[Manfred Lotz]

On Sat, 20 Jul 2019 14:11:21 -0400
Michael Speer <knom...@gmail.com> wrote:

> You may want to use `#!/usr/bin/env python3` instead.
>

In my case it doesn't matter. However, I agree that your suggestion is
usually preferable.

> There is a concept in python called the virtual environment. This
> used to be done with a tool called virtualenv in python2, and is now
> done mainly through a venv module in python3.
>
> A virtual environment goes into a directory of your choosing and will
> have its own python3 executable, and pip3 executable, and when you add
> dependencies, they are also placed into the directory structure under
> your chosen directory.
>
> When you do a `. <directory>/bin/activate` the included source will
> places the virtual environment's bin/ folder at the beginning of your
> PATH environment variable, making it the default python3 when you
> type it without a full path.
>
> This allows you to run scripts that need different, or even
> conflicting, sets of dependencies without bothering with the
> underlying linux distribution's python installation's modules.
>

New to me. Interesting.

> If you use `#!/usr/bin/python3`, it will always use exactly the system
> version that is installed, and the system's installed modules.
>
> Your scripts will still default to the system installation if a
> virtual environment is not activated. So you lose nothing by doing it
> this way, but gain a little control from it.
>

Ok, understood.


--
Manfred

[Peter J. Holzer]

On 2019-07-20 14:11:44 -0500, Tim Daneliuk wrote:
> So, no, do NOT encode the hard location - ever. Always use env to
> discover the one that the user has specified. The only exception is
> /bin/sh which - for a variety of reasons - can reliably counted upon.
>
> We don't need to bikeshed this. All we need is people who disagree
> with this view to spend a year in software packaging, operations,
> deployment and DevOps ... then get back to us...

After 25 years in software packaging, operations, deployment and DevOps
I disagree: A program should not behave differently because of a
different path, #!/usr/bin/env is a total no-no.

There is a nice way to achieve this: Just use the interpreter of the
virtual environment in the shebang.
(That requires rewriting the shebang during installation, but that's a
minor inconvenience)

hp

--
_ | Peter J. Holzer | we build much bigger, better disasters now
|_|_) | | because we have much more sophisticated
| | | h...@hjp.at | management tools.
__/ | http://www.hjp.at/ | -- Ross Anderson <https://www.edge.org/>

[Tim Daneliuk]

On 7/20/19 2:56 PM, Peter J. Holzer wrote:
> On 2019-07-20 14:11:44 -0500, Tim Daneliuk wrote:
>> So, no, do NOT encode the hard location - ever. Always use env to
>> discover the one that the user has specified. The only exception is
>> /bin/sh which - for a variety of reasons - can reliably counted upon.
>>
>> We don't need to bikeshed this. All we need is people who disagree
>> with this view to spend a year in software packaging, operations,
>> deployment and DevOps ... then get back to us...
>
> After 25 years in software packaging, operations, deployment and DevOps
> I disagree: A program should not behave differently because of a
> different path, #!/usr/bin/env is a total no-no.
>
> There is a nice way to achieve this: Just use the interpreter of the
> virtual environment in the shebang.
> (That requires rewriting the shebang during installation, but that's a
> minor inconvenience)
>
> hp
>

And what happens with programs that have no virtenv equivalent?
perl, go, ruby, awk, sed, grep, etc. have no simple way to
get installed virtual short of insisting the everything live in a
docker container or VM?

The fact is that most large compute environments are slow to upgrade
the OS. That means core tools also lag considerably behind as well.
Being able to install newer versions along side the OS' own and then
use them by default is manifestly necessary. That's why users have
the ability to modify $PATH to suit their own needs. All /usr/bin/env
does is to tell the interpreter, "honor the intent of the spawning shell".
This shouldn't even be a question ... and it's why so much garbage continues
to live forever. Having to constantly code around old systems versions of
tools with not other recourse is just crazy.

In actual fact, the very best way to write portable, reliable, and operable
systems of code is to divorce them entirely (or as a nearly as possible) for
the OS tools as you can. That's why docker works so well. That's why go
avoids dynamic linking.

In my case, that's why I compile my own version of
languages and daily use utilities to live outside the OS. I get absolutely
predicable behavior irrespective of my distro and whatever backleveled cruft
it has laying around. I _know_ every version of every important tool my code
will be using ... all by setting up $PATH properly and using /usr/bin/env in
my interpreters.

If you want really big fun, try going into an older CentOS or RedHat instances and, say,
upgrading system python to python3. It's super fun. Yes, in that case, you COULD
use a venv. But there are tons of other tools for which this is not an option -
gcc, autoconf, perl, go awk, sed, bash, ad infinitum, ad nauseum are invariably
backleveled on production OS instances. My way fixes that problem. Your way
forces me to code around it ... which is a way worse solution.

You may have 25 years at this but I have 40 - does that make me nearly twice
as right? Arguments from authority are silly.

P.S. https://www.tundraware.com/TechnicalNotes/Divorce-Your-Linux-Admin/

[Brian Oney]

Nice.

Emacs (well spacemacs) is my authority.

The key sequence 'SPC i !' inserts

#!/usr/bin/env python

as specified in the insert-shebang package
https://github.com/psachin/insert-shebang

My limited experience tells me to expect the user (me) to know what
they're doing, hence env.

Why not make a compromise? What would be a potential pitfall of the
following spitbang?

#!python

[Chris Angelico]

On Sun, Jul 21, 2019 at 5:26 AM Tim Daneliuk <in...@tundraware.com> wrote:
> So, no, do NOT encode the hard location - ever. Always use env to discover the one that
> the user has specified. The only exception is /bin/sh which - for a variety of reasons -
> can reliably counted upon.

A quick grep through my $PATH shows that there are a number of
executable Python scripts there, including add-apt-repository,
calibre, some lilypond stuff, trash-can management, samba-tool, iotop,
and youtube-dl. If I have a venv active with, say, Python 3.9, then
`/usr/bin/env python` is going to point to Python 3.9. What are the
odds that all those scripts will work with Python 3.9 with no
libraries installed? Why should typing "youtube-dl B7xai5u_tnk" be
affected by a virtual environment, when typing "man youtube-dl"
wouldn't be?? Using env for everything is a terrible idea and one that
will basically make virtual environments useless.

ChrisA

[Tim Daneliuk]

On 7/20/19 5:14 PM, Chris Angelico wrote:
> Using env for everything is a terrible idea and one that
> will basically make virtual environments useless.

Not if you manage them properly.

Everyone's mileage is different, but when I enter a venv, I ensure everything I do
there is packaged to work together. I pip install things in there and ensure
that the tools I invoke will work in that environment.

In the example you cite, I would simply run the tools in question outside any
environment where I know they don't work.

[Tim Daneliuk]

On 7/20/19 4:28 PM, Brian Oney wrote:
> Why not make a compromise? What would be a potential pitfall of the
> following spitbang?
>
> #!python

Not sure this really changes the discussion.

[Tim Daneliuk]

I guess I should clarify something:

1) I pip install everything locally inside my own $HOME for that set of things I need
generally and will not be using in a venv.

2) I pip install everything I need in each venv, even if it already exists under 1) above.

I effectively treat venvs as fully self-contained environments to the degree possible.

[Chris Angelico]

Are you aware of every systemwide command that happens to be
implemented in Python, such that you won't execute any of them while
you have the venv active? The ones I mentioned are all installed into
/usr/local/bin or /usr/bin or some other directory on my main $PATH. I
only know they're Python scripts because I just ran "file
/usr/local/bin/*" and had it check them all. If they all reacted to
the activation of a venv, I would have to be extremely careful of
which commands I ever run while a venv is active. Which basically
means I would never activate a venv ever again - not worth it.

ChrisA

[Cameron Simpson]

On 21Jul2019 08:14, Chris Angelico <ros...@gmail.com> wrote:
>On Sun, Jul 21, 2019 at 5:26 AM Tim Daneliuk <in...@tundraware.com> wrote:
>> So, no, do NOT encode the hard location - ever. Always use env to discover the one that
>> the user has specified. The only exception is /bin/sh which - for a variety of reasons -
>> can reliably counted upon.
>
>A quick grep through my $PATH shows that there are a number of
>executable Python scripts there, including add-apt-repository,
>calibre, some lilypond stuff, trash-can management, samba-tool, iotop,
>and youtube-dl. If I have a venv active with, say, Python 3.9, then
>`/usr/bin/env python` is going to point to Python 3.9. What are the
>odds that all those scripts will work with Python 3.9 with no
>libraries installed? Why should typing "youtube-dl B7xai5u_tnk" be
>affected by a virtual environment, when typing "man youtube-dl"

>wouldn't be?? Using env for everything is a terrible idea and one that

>will basically make virtual environments useless.

I'm with Tim Daneliuk. The environment matters and should be honoured
except in extremely weird cases.

If you require a specific outcoming, set a specific environment. It is
under your control. Control it.

For your specific example, "man youtube-dl" _is_ affected by the
environment. It honours the $MANPATH variable.

For Peter J. Holzer, if we must play the "I've been doing this forever"
game: I've been sysadmining etc longer than your 25 years and disagree
with you. If your tools are strongly affected by version, run them with
an environment that finds the right version.

Installers, particularly those run as root or in other low level setup
situations, should have a WELL DEFINED environment.

If you have a tool that isn't portable, put it in a wrapper that coddles
its behaviour.

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Chris Angelico]

So.... you mean that a tool that depends on running on a consistent
environment, it should use a shebang of "/usr/bin/python3.6" instead
of "/usr/bin/env python3"? Because, wow, that would be exactly what is
already happening on my system. Why use /usr/bin/env and then wrap
something around it to force the environment, when you could just set
a correct shebang so it properly defines its execution environment?

ChrisA

[Cameron Simpson]

On 21Jul2019 09:31, Chris Angelico <ros...@gmail.com> wrote:
>On Sun, Jul 21, 2019 at 9:15 AM Cameron Simpson <c...@cskk.id.au> wrote:
>So.... you mean that a tool that depends on running on a consistent
>environment, it should use a shebang of "/usr/bin/python3.6" instead
>of "/usr/bin/env python3"?

Jeez. No. That is the _opposite_ of what I'm saying.

>Because, wow, that would be exactly what is
>already happening on my system. Why use /usr/bin/env and then wrap
>something around it to force the environment, when you could just set
>a correct shebang so it properly defines its execution environment?

Because the shebang is hardwired and inflexible.

Because it means hand patching (even scripted) a bazillion scripts to
that they know their physical install.

Because it presumes detailed hardwired knowledge of the target system in
a script which should work anywhere.

Instead a tiny _common_ shell script resembling this:

#!/bin/sh
# Run command in the official environment.
exec env - PATH=/path/to/3.6venv/bin:/usr/sbin:/bin exec ${1+"$@"}

arranges things. The "env -" is aimed at "clean" daemon or install
environments. You can do subtler or less intrusive things in other
settings.

There are any number of variations available here that don't hardwire
the environment into the various _scripts_, but into the invocation
setting. Or just start an environment (eg a shell) with the desired
environment.

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Manfred Lotz]

On Sat, 20 Jul 2019 23:28:35 +0200
Brian Oney <brian....@googlemail.com> wrote:

> Why not make a compromise? What would be a potential pitfall of the
> following spitbang?
>
> #!python

I think that per definition a path in a shebang has to be absolute.

Actually, your suggestion won't work for people who use the fish
shell (just for the given reason).

[Tim Daneliuk]

On 7/20/19 6:04 PM, Chris Angelico wrote:
> Are you aware of every systemwide command that happens to be
> implemented in Python, such that you won't execute any of them while
> you have the venv active?

No, but this has never been a problem because the newer versions of
python tend to be pretty good - within a major release tree - of being
backward compatible. Certainly, if I were running, say, RedHat 4 with
a very new version of python in my path first, this could theoretically
be an issue. I've just not run into it.

[Tim Daneliuk]

On 7/20/19 6:04 PM, Cameron Simpson wrote:
> If you require a specific outcoming, set a specific environment. It is under your control. Control it.

Exactly right. I have just had the REALLY irritating experience of trying to bootstrap a
location insensitive version of linuxbrew that mostly works, but is crippled by brain damage
that insists that the best version of perl will always be found in /usr/bin.

I have been a hardware engineer (analog and digital), software implementor, systems designer,
devops person, and large scale (physical data center) platform engineer and this kind of
bad thinking just kills site reliability.

I stipulate that there are corner cases where Chris A. is correct - it is certainly
possible to clobber a system with incorrect /usr/bin/env findings. But I'd argue
that there is a simple work around - run another terminal session (terminator, tmux,
screen, ... whatever floats your boat) that has $PATH set up to find things in /usr/bin
first. Voila! Problem solved and everyone can use env the way they want to.

In some respects, this problem does get a little simpler when you docker-ize your
software distributions until ... you need newer versions of tools than even the latest
docker OS implementations support. Then you're back to the same old noise ...

[Chris Angelico]

Many Python packages are incompatible with CPython 3.9 at the moment. MANY.

ChrisA

[Christian Gollwitzer]

Am 21.07.19 um 07:31 schrieb Tim Daneliuk:

It's not about the core language, but these system tools depend on
packages, sometimes specialist packages, which might not be available in
your venv.

Christian

[Barry Scott]

Well said.

The fedora packaging system replaces all shebang lines with the full path to either python2 or python3 on the system
to make sure that scripts installed in the system run as intended and do not break because of the users PATH.

How you think about the shebang lines depends on where the scripts will be used.

If its as an installed system component the exact path is usually the right thing to do.

If it you personal scripts then you may well find /usr/bin/env python3 makes you life
simpler.

For code I'm testing I usual do not depend no shebang lines at all. Often I'm testing
on many python versions. And end up with something like this to test on all interesting python
versions.

for PTYHON in python3.6 python3.7 python3.8
do
$PYTHON my_script.py
done

Barry

[Manfred Lotz]

On Sun, 21 Jul 2019 10:21:55 +1000
Cameron Simpson <c...@cskk.id.au> wrote:

> On 21Jul2019 09:31, Chris Angelico <ros...@gmail.com> wrote:
> >On Sun, Jul 21, 2019 at 9:15 AM Cameron Simpson <c...@cskk.id.au>
> >wrote: So.... you mean that a tool that depends on running on a
> >consistent environment, it should use a shebang of
> >"/usr/bin/python3.6" instead of "/usr/bin/env python3"?
>
> Jeez. No. That is the _opposite_ of what I'm saying.
>
> >Because, wow, that would be exactly what is
> >already happening on my system. Why use /usr/bin/env and then wrap
> >something around it to force the environment, when you could just set
> >a correct shebang so it properly defines its execution environment?
>
> Because the shebang is hardwired and inflexible.
>
> Because it means hand patching (even scripted) a bazillion scripts to
> that they know their physical install.
>
> Because it presumes detailed hardwired knowledge of the target system
> in a script which should work anywhere.
>
> Instead a tiny _common_ shell script resembling this:
>
> #!/bin/sh
> # Run command in the official environment.
> exec env - PATH=/path/to/3.6venv/bin:/usr/sbin:/bin exec ${1+"$@"}
>
> arranges things. The "env -" is aimed at "clean" daemon or install
> environments. You can do subtler or less intrusive things in other
> settings.
>

I took a look and found that Fedora 30 and Debian Jessie both use
hard-wired paths for python in the rpm resp. deb packages.

I'm being new to Python and I am not acquainted in any way with
virtualenv resp. venv so cannot currently judge its pro and cons.

So I will stick to:
#!/usr/bin/env python3

as shebang for my scripts.


--
Manfred

[Brian Oney]

I think that's a good decision. Most of the conversation applies to sysadmins concerned with answering your question in absolute terms. When you start writing scripts which are restricted to a specific environment and are intended to be distributed, you may revisit this thread.

Be blissful until then :). Chris et al have "fixed" things for you.

[eryk sun]

On 7/20/19, Michael Speer <knom...@gmail.com> wrote:
>
> You may want to use `#!/usr/bin/env python3` instead.

This is partially supported in Windows, but only if .py files are
associated with the py.exe launcher and the shebang runs "python"
instead of "python3".

py.exe supports four builtin virtual commands: "/usr/bin/env python",
"/usr/bin/python", "/usr/local/bin/python", and "python". The "python"
command can be qualified with a version specification of the form
"X[.Y][-32|-64]", such as "python3". The "/usr/bin/env python" virtual
command searches PATH, but only if the command matches "python"
exactly. If there's a version specified, or if it's not an "env"
virtual command, the launcher looks for a registered Python
installation instead of searching PATH.

For all other commands, the launcher skips a virtual Unix prefix (i.e.
"/usr/bin/env", "/usr/bin", and "/usr/local/bin"), if present, and
searches PATH. For example, given "/usr/bin/pypy", it searches PATH
for "pypy" plus the file extensions in PATHEXT (i.e. .COM, .EXE, etc).
If the command isn't found in PATH, it checks in the "[commands]"
section of "%LocalAppData%\py.ini". For example, py.ini could define
"pypy=C:\PyPy71\pypy.exe".

A virtual environment should have a "python" executable, so shebangs
that use "/usr/bin/env python" will prefer an active virtual
environment. But a lot of scripts use "python3" instead of "python".
This is a stumbling block in Windows since we don't install versioned
"pythonX[.Y].exe" binaries. Thus the "env" virtual command is special
cased to find a qualified Python version in the list of registered
Python installations instead of searching PATH. (The new store app
distribution does install versioned app links, but, even in this case,
virtual environments lack versioned binaries in the "Scripts"
directory.)

That said, some developers manually create versioned binaries as
symlinks, hardlinks, or copies. Also, maybe a future version of the
full (non-app) Python installation will do the same. Thus I think the
launcher should search PATH for all "/usr/bin/env python*" shebangs,
but without the binary-type specification (i.e. "-32" or "-64") if
present. In this case, instead of a regular search based on WINAPI
SearchPath, we'll need a custom search that tokenizes and walks PATH
and checks the binary type via GetBinaryTypeW. If no matching version
is found in PATH, the "env" search should fall back on the list of
registered versions.

[Peter J. Holzer]

On 2019-07-20 15:26:46 -0500, Tim Daneliuk wrote:
> On 7/20/19 2:56 PM, Peter J. Holzer wrote:
> > On 2019-07-20 14:11:44 -0500, Tim Daneliuk wrote:
> >> So, no, do NOT encode the hard location - ever. Always use env to
> >> discover the one that the user has specified. The only exception is
> >> /bin/sh which - for a variety of reasons - can reliably counted upon.
> >>
> >> We don't need to bikeshed this. All we need is people who disagree
> >> with this view to spend a year in software packaging, operations,
> >> deployment and DevOps ... then get back to us...
> >
> > After 25 years in software packaging, operations, deployment and DevOps
> > I disagree: A program should not behave differently because of a
> > different path, #!/usr/bin/env is a total no-no.
> >
> > There is a nice way to achieve this: Just use the interpreter of the
> > virtual environment in the shebang.
> > (That requires rewriting the shebang during installation, but that's a
> > minor inconvenience)
>
>

> And what happens with programs that have no virtenv equivalent?
> perl, go, ruby, awk, sed, grep, etc. have no simple way to
> get installed virtual short of insisting the everything live in a
> docker container or VM?

Perl is an excellent example: When you install a Perl script which has
been packaged with one of the usual tools (e.g., Makemaker or
Module::Build), the shebang is set to the interpreter used for
installation. The user of the script doesn't have to know what
environment to use. (Perl also has perlbrew which is similar to virtual
environments, but I haven't used that much).

Go is a compiled language: The executables are binaries (no shebang) and
even statically linked. Again, the user doesn't have to care about a
language environment.

awk, grep, sed, etc. are often used in shell scripts, and they were
probably the reason why I first developed a "a program's behaviour must
not depend on the environment /except as documented/" policy: Back then
in the 1990s one usually wanted to have GNU awk, sed, grep etc. in
addition to the system utilities. Having scripts randomly fail or
(worse) produce wrong results depending on the order of /usr/bin and
/usr/local/bin in the user's PATH and the exact contents of
/usr/local/bin wasn't fun (oh, and don't forget cron).

> The fact is that most large compute environments are slow to upgrade
> the OS. That means core tools also lag considerably behind as well.
> Being able to install newer versions along side the OS' own and then
> use them by default is manifestly necessary.

I agree. But (for me at least) it is important to do that robustly, and
#!/usr/bin/env is not robust.

> That's why users have the ability to modify $PATH to suit their own
> needs. All /usr/bin/env does is to tell the interpreter, "honor the
> intent of the spawning shell".

You probably just phrased that badly, but as written this is completely
wrong: /usr/bin/env doesn't tell the interpreter anything. It *chooses*
the interpreter to invoke.

> If you want really big fun, try going into an older CentOS or RedHat instances and, say,
> upgrading system python to python3.

Don't do that. Install python3 somewhere else, e.g. into /usr/local.

> It's super fun. Yes, in that case, you COULD use a venv. But there
> are tons of other tools for which this is not an option - gcc,
> autoconf, perl, go awk, sed, bash, ad infinitum, ad nauseum are
> invariably backleveled on production OS instances. My way fixes that
> problem.

Your way only fixes your problem only if environments where your script
may be called. You may know that /usr/local/bin/foo is a python script
that requires /usr/local/python3.8/bin/python to be in the path before
any other python interpreter and you may rememmber that in every
instance where that script is called directly or indirectly. But do your
colleagues? Especially if they aren't programmers?

And what happens if /usr/local/bin/bar requires
/usr/local/python3.4/bin/python? Do you explicitely set the PATH before
invoking each script?

By putting #!/usr/local/python3.8/bin/python at the top of
/usr/local/bin/foo and #!/usr/local/python3.4/bin/python at the top of
/usr/local/bin/bar I can just invoke both scripts. Even better, my users
can invoke both scripts and they don't even have to know they are
written in Python.

> You may have 25 years at this but I have 40 - does that make me nearly twice
> as right? Arguments from authority are silly.

No, but it invalidates your notion that anybody with a bit of experience
would obviously use #!/usr/bin/env. I do have a bit more than one year
of experience and I think #!/usr/bin/env is a terrible idea. (I use it
for throwaway scripts which need a venv, but not for anything in
production.)

[Peter J. Holzer]

On 2019-07-21 09:04:43 +1000, Cameron Simpson wrote:
> On 21Jul2019 08:14, Chris Angelico <ros...@gmail.com> wrote:

> > On Sun, Jul 21, 2019 at 5:26 AM Tim Daneliuk <in...@tundraware.com> wrote:
> > > So, no, do NOT encode the hard location - ever. Always use env to
> > > discover the one that the user has specified. The only exception
> > > is /bin/sh which - for a variety of reasons - can reliably counted
> > > upon.
> >

> > A quick grep through my $PATH shows that there are a number of
> > executable Python scripts there, including add-apt-repository,
> > calibre, some lilypond stuff, trash-can management, samba-tool, iotop,
> > and youtube-dl. If I have a venv active with, say, Python 3.9, then
> > `/usr/bin/env python` is going to point to Python 3.9. What are the
> > odds that all those scripts will work with Python 3.9 with no
> > libraries installed? Why should typing "youtube-dl B7xai5u_tnk" be
> > affected by a virtual environment, when typing "man youtube-dl"
> > wouldn't be?? Using env for everything is a terrible idea and one that
> > will basically make virtual environments useless.
>
> I'm with Tim Daneliuk. The environment matters and should be honoured except
> in extremely weird cases.

I don't think that all the scripts in /usr/bin are extremely weird
cases.

> If you require a specific outcoming, set a specific environment. It is under
> your control. Control it.
>

> For your specific example, "man youtube-dl" _is_ affected by the
> environment. It honours the $MANPATH variable.

MANPATH is explicitely intended to control man.

But man doesn't fail if you set your PATH to something weird. It will
still invoke /usr/bin/groff even if that isn't in the PATH.
(I expected that there is also an environment variable to control that
but the manpage doesn't mention one).

> For Peter J. Holzer, if we must play the "I've been doing this forever"
> game: I've been sysadmining etc longer than your 25 years and disagree with
> you.

That's fine. Unlike Tim I don't claim that anybody who disagrees with me
must be a newbie.

[Tim Daneliuk]

On 7/21/19 8:47 AM, Peter J. Holzer wrote:
> That's fine. Unlike Tim I don't claim that anybody who disagrees with me
> must be a newbie.

Peter, that's ad hominem and unfair. I never said anything close to that.
What I said is that if someone were to spend an extended period of time
in devops and systems engineering at large scale, they'd quickly come to
hate hardwired paths.

The truth is that there is no single answer to this problem until you
hermetically seal an environment. Docker comes close, a freestanding
VM does this most completely. But short of that, the ability to decide
to use something other than system-provided tools absolutely IS a requirement
in systems of any scale.

[Chris Angelico]

Your final paragraph does not justify your preceding. Yes, you need
the ability to decide to use something other than the system-provided
tool; that's why "/usr/bin/env python3" is a valid shebang. No, that
does NOT mean that people with any decent experience will "hate
hardwired paths". The hardwired path ("#!/usr/bin/python3") has its
own value. Thanks to an absolute path in its shebang, I can run "sudo
iotop" regardless of whether I have a venv active. I cannot run "sudo
python3 `which iotop`" reliably, because that command uses the
environment to look things up.

You're absolutely right that there's no single answer to the problem.
That's why script authors have the option to use any path they like,
including /usr/bin/env.

ChrisA

[Peter J. Holzer]

On 2019-07-21 10:26:17 -0500, Tim Daneliuk wrote:
> On 7/21/19 8:47 AM, Peter J. Holzer wrote:
> > That's fine. Unlike Tim I don't claim that anybody who disagrees with me
> > must be a newbie.
>
> Peter, that's ad hominem and unfair.

No, it isn't. Please read the first paragraph of
https://en.wikipedia.org/wiki/Ad_hominem

> I never said anything close to that.

Well, let's see what you wrote:

| We don't need to bikeshed this.

Translation: This is not a matter of opinion or taste. There is one
objectively true answer.

| All we need is people who disagree with this view to spend a year in
| software packaging, operations, deployment and DevOps ... then get back
| to us...

Translation: People who disagree with your opinion obviously haven't
worked even a single year in this field because otherwise they would
have discovered the one objectively true answer.

| Grrrrrrrrrr......

I don't have to translate that :-)

> What I said is that if someone were to spend an extended period of time
> in devops and systems engineering at large scale, they'd quickly come to
> hate hardwired paths.

You are just stating the same point again: Everybody who has experience
in the field agrees with you, therefore anybody who disagrees can't have
experience.

> The truth is that there is no single answer to this problem until you
> hermetically seal an environment.

If you can hermetically seal the environment, the argument is moot. I
thought (and still think) we are talking about the case where you can't.
Therefore I took exception to your insistence that there is in fact one
single answer and that this single answer is one which I know from
bitter experience to be quite fragile.

> But short of that, the ability to decide to use something other than
> system-provided tools absolutely IS a requirement in systems of any
> scale.

I completely agree with that. I just don't think that using
#!/usr/bin/env is a good way to achieve that.

[Cameron Simpson]

On 21Jul2019 15:47, Peter J. Holzer <hjp-p...@hjp.at> wrote:
>On 2019-07-21 09:04:43 +1000, Cameron Simpson wrote:
>> I'm with Tim Daneliuk. The environment matters and should be honoured
>> except
>> in extremely weird cases.
>
>I don't think that all the scripts in /usr/bin are extremely weird
>cases.

I think I'd better add some nuance to my stance.

I've no problem with all the scripts shipped from an OS vendor having
#!/usr/bin/python (or whatever fixed path) in them. They have been
released tested against the system python and should _expect_ to run
against it. My position here is that the entire OS distribution
constitutes a working (and via the #! controlled) environment.

But consider third party tools. Including personal tools, but basicly
anything from _outside_ the local system in terms of authorship.

Particularly with a language like Python which is strongly backwards
compatible, they should generally use "#!/usr/bin/env python" (or
python2 or python3 as appropriate, when that matters) so that they can
run in the environment they find themselves in.

1: You can can always execute a script via a specific interpreter
explicitly.

2: If you want to test a script it is easier to provide an environment
that exercises the script in a particular way than to hand patch the
shebang lines on every run/reconfig.

3: If the script (per one of Chris' examples) requires a specific python
such as 3.6, you can always go "#!usr/bin/env python3.6" in the script
to express the target version and provide a executable "python3.6" name
in you environment. I keep a personal ~/bin-local directory for just
this kind of per-host stuff myself, and of course one can do the same
thing in places like venvs or /usr/local/bin etc. And thus _still_ leave
the script itself without a hardwired path.

[...]
>> If you require a specific outcoming, set a specific environment. It
>> is under

>> your control. Control it.
>>
>> For your specific example, "man youtube-dl" _is_ affected by the
>> environment. It honours the $MANPATH variable.
>
>MANPATH is explicitely intended to control man.
>
>But man doesn't fail if you set your PATH to something weird. It will
>still invoke /usr/bin/groff even if that isn't in the PATH.
>(I expected that there is also an environment variable to control that
>but the manpage doesn't mention one).

Heh.

I wrote my own "man" yonks ago for various reasons. Guess what? I expect
to type "man" and get mine most of the time, but type "man" when not me
and get /usr/bin/man (absent weirdness). That applies interactively and
also in scripts.

Same philosophy. Use the command name to express intent and the
environment to choose the implementation of the intent. And so also in
the shebang lines.

>> For Peter J. Holzer, if we must play the "I've been doing this
>> forever" game: I've been sysadmining etc longer than your 25 years and disagree with
>> you.
>

>That's fine. Unlike Tim I don't claim that anybody who disagrees with me
>must be a newbie.

Aye; sorry for the snarkiness. Which is why I'm disagreeing on some
things instead of asserting that you're wrong, because you're not
"wrong".

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Eli the Bearded]

In comp.lang.python, Tim Daneliuk <in...@tundraware.com> wrote:
> On 7/20/19 1:20 PM, Chris Angelico wrote:

> > On Sun, Jul 21, 2019 at 4:13 AM Michael Speer <knom...@gmail.com> wrote:
> >> You may want to use `#!/usr/bin/env python3` instead.

I no longer have one to verify, but I recall Solaris boxen used /bin/env
not /usr/bin/env.

> So, no, do NOT encode the hard location - ever. Always use env to
> discover the one that the user has specified.

But wait, you just hard coded the location of env...

> The only exception is
> /bin/sh which - for a variety of reasons - can reliably counted upon.

BZZZZ! Fully half of my work porting trn4 to my cellphone was fixing all
the places that ancient build system believed /bin/sh was the name of
sh. In that environment (Termux shell on an Android phone) the location
is /data/data/com.termux/files/usr/bin/sh (and env is also in
/data/data/com.termux/files/usr/bin hahaha).

Even on more traditional environments -cough-Solaris-cough- /bin/sh may
exist but be so ancient as to break things that work elsewhere. "^" as
a synonym for "|", is a noteworthy gotcha.

Figuring out where things are on the user's path is a laudable goal, but
do it only at install time, not run time, for consistent runs.

Elijah
------
pathological edge cases -r- us

[אורי‎]

We are using `#!/usr/bin/env python`, for example on
https://github.com/speedy-net/speedy-net/blob/master/speedy/core/manage.py

For bash we are using `#!/usr/bin/env bash`. I don't know if those are the
best but they work.

אורי
u...@speedy.net

On Sat, Jul 20, 2019 at 9:12 PM Michael Speer <knom...@gmail.com> wrote:

> You may want to use `#!/usr/bin/env python3` instead.
>

> There is a concept in python called the virtual environment. This used to
> be done with a tool called virtualenv in python2, and is now done mainly
> through a venv module in python3.
>
> A virtual environment goes into a directory of your choosing and will have
> its own python3 executable, and pip3 executable, and when you add
> dependencies, they are also placed into the directory structure under your
> chosen directory.
>
> When you do a `. <directory>/bin/activate` the included source will places
> the virtual environment's bin/ folder at the beginning of your PATH
> environment variable, making it the default python3 when you type it
> without a full path.
>
> This allows you to run scripts that need different, or even conflicting,
> sets of dependencies without bothering with the underlying linux
> distribution's python installation's modules.
>
> If you use `#!/usr/bin/python3`, it will always use exactly the system
> version that is installed, and the system's installed modules.
>
> Your scripts will still default to the system installation if a virtual
> environment is not activated. So you lose nothing by doing it this way, but
> gain a little control from it.
>
>
> On Sat, Jul 20, 2019 at 1:41 PM Manfred Lotz <ml_...@posteo.de> wrote:
>
> > Hi there,
> > Pretty new to python I've got a question regarding the proper shebang
> > for Python 3.
> >
> > I use
> > #!/usr/bin/python3
> >
> > which works fine.
> >
> > Today I saw
> > #!/usr/bin/python3 -tt
> >
> > and was wondering what -tt means.
> >
> > Being on Fedora 30, Python 3.7.3 the man page of python3 doesn't even
> > mention -t.
> >
> > python 2 man page mentions
> >
> > -t Issue a warning when a source file mixes tabs and spaces
> > for indentation in a way that makes it depend on the worth
> > of a tab expressed in spaces. Issue an error when the option is
> > given twice.
> >
> > I guess that -t has the same meaning with python 3.7.3.
> >
> >
> > My questions:
> >
> > 1. Is my guess correct?
> >
> > 2. Is it a bug that it is not mentioned? python3 --help doesn't mention
> > it either.
> >
> >
> > --
> > Manfred
> >
> >
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
> --
> https://mail.python.org/mailman/listinfo/python-list
>

[Cameron Simpson]

On 22Jul2019 21:14, Eli the Bearded <*@eli.users.panix.com> wrote:
>In comp.lang.python, Tim Daneliuk <in...@tundraware.com> wrote:
>> On 7/20/19 1:20 PM, Chris Angelico wrote:
>> > On Sun, Jul 21, 2019 at 4:13 AM Michael Speer <knom...@gmail.com> wrote:
>> >> You may want to use `#!/usr/bin/env python3` instead.
>
>I no longer have one to verify, but I recall Solaris boxen used /bin/env
>not /usr/bin/env.

That is my recollection too. A pain point. To the point that I'd hand
make a /usr/bin/env symlink.

Why _any_ modern system has anything other than /bin in the base install
escapes me. In the distant past /sbin and a distinct /usr with its own
bin had their values, but these days? Bah!

(I'm not complaining about /usr/local/bin here - keeping added stuff
distinct from the vendor/distributor stuff is very valuable.)

>> So, no, do NOT encode the hard location - ever. Always use env to
>> discover the one that the user has specified.
>
>But wait, you just hard coded the location of env...

Yeah. Too many boots, too many straps.

>> The only exception is
>> /bin/sh which - for a variety of reasons - can reliably counted upon.
>
>BZZZZ! Fully half of my work porting trn4 to my cellphone was fixing all
>the places that ancient build system believed /bin/sh was the name of
>sh. In that environment (Termux shell on an Android phone) the location
>is /data/data/com.termux/files/usr/bin/sh (and env is also in
>/data/data/com.termux/files/usr/bin hahaha).

I'd do the symlink thing there too, if feasible. (Counterpoint: I've a
DVR here where changes to / don't survive a reboot and there's no
/etc/rc.local like file which survives a reboot either.)

POSIX systems have a /bin/sh. I would move heaven itself to ensure this.

>Even on more traditional environments -cough-Solaris-cough- /bin/sh may
>exist but be so ancient as to break things that work elsewhere. "^" as
>a synonym for "|", is a noteworthy gotcha.

Aye. I still quote ^ in my scripts to this day for this reason. But
then, I learnt shell programming on V7 UNIX, well older than Solaris.

>Figuring out where things are on the user's path is a laudable goal, but
>do it only at install time, not run time, for consistent runs.

Perhaps. Consistent runs require consistent environments. That often
includes executables.

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Cameron Simpson]

On 23Jul2019 00:19, אורי <u...@speedy.net> wrote:
>We are using `#!/usr/bin/env python`, for example on
>https://github.com/speedy-net/speedy-net/blob/master/speedy/core/manage.py
>
>For bash we are using `#!/usr/bin/env bash`. I don't know if those are the
>best but they work.

Worthwhile. Plenty of platforms do not install bash in /bin.

Though personally I only VERY RARELY want bash for a shell script.
/bin/sh is portable and the extras in bash have relatively little value
in scripting; by the time your script really wants them it is usually
worth moving to a more expressive language. Like Python.

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Barry Scott]

> On 23 Jul 2019, at 00:13, Cameron Simpson <c...@cskk.id.au> wrote:
>
> Why _any_ modern system has anything other than /bin in the base install escapes me. In the distant past /sbin and a distinct /usr with its own bin had their values, but these days? Bah!

On fedora its all in /usr these days with symlinks to the old locations.

$ ls -l / | grep usr
lrwxrwxrwx 1 root root 7 Feb 11 13:47 bin -> usr/bin/
lrwxrwxrwx 1 root root 7 Feb 11 13:47 lib -> usr/lib/
lrwxrwxrwx 1 root root 9 Feb 11 13:47 lib64 -> usr/lib64/
lrwxrwxrwx 1 root root 8 Feb 11 13:47 sbin -> usr/sbin/
drwxr-xr-x. 13 root root 4096 May 5 17:22 usr/

You can read about why here https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ <https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/> and also https://fedoraproject.org/wiki/Features/UsrMove <https://fedoraproject.org/wiki/Features/UsrMove>

Barry

[Cameron Simpson]

On 24Jul2019 21:36, Barry Scott <ba...@barrys-emacs.org> wrote:
>> On 23 Jul 2019, at 00:13, Cameron Simpson <c...@cskk.id.au> wrote:
>> Why _any_ modern system has anything other than /bin in the base
>> install escapes me. In the distant past /sbin and a distinct /usr
>> with its own bin had their values, but these days? Bah!
>
>On fedora its all in /usr these days with symlinks to the old
>locations.
>
>$ ls -l / | grep usr
>lrwxrwxrwx 1 root root 7 Feb 11 13:47 bin -> usr/bin/
>lrwxrwxrwx 1 root root 7 Feb 11 13:47 lib -> usr/lib/
>lrwxrwxrwx 1 root root 9 Feb 11 13:47 lib64 -> usr/lib64/
>lrwxrwxrwx 1 root root 8 Feb 11 13:47 sbin -> usr/sbin/
>drwxr-xr-x. 13 root root 4096 May 5 17:22 usr/

That is some progress, hooray. Then there's just sbin -> bin to go. They
could merge lib and lib64 too if they embedded an architecture signature
in library filenames.

>You can read about why here
>https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/

>and also https://fedoraproject.org/wiki/Features/UsrMove

Thanks for these references.

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Michael Torrie]

On 7/24/19 4:20 PM, Cameron Simpson wrote:
> That is some progress, hooray. Then there's just sbin -> bin to go.

I suppose in the olden days sbin was for static binaries, usable in
single user mode for recovering the system without the main drive
mounted. In more recent times, binaries that are mostly applicable to
the super user go there. I don't see why you would want to merge those.
A normal user rarely has need of much in /sbin. Already /bin has way
too much stuff in it (although I don't see any other way to practically
do it without ridiculous PATHs searching all over the disk).

Having said that, I note that on my CentOS 7 workstation, sbin seems to
be in the path by default. So that negates my argument I suppose.
Although I might have made that change myself.

[Cameron Simpson]

On 24Jul2019 20:24, Michael Torrie <tor...@gmail.com> wrote:
>On 7/24/19 4:20 PM, Cameron Simpson wrote:
>> That is some progress, hooray. Then there's just sbin -> bin to go.
>
>I suppose in the olden days sbin was for static binaries, usable in
>single user mode for recovering the system without the main drive
>mounted.

Yep. Happy days.

>In more recent times, binaries that are mostly applicable to
>the super user go there. I don't see why you would want to merge those.
> A normal user rarely has need of much in /sbin.

I say unto to you "ifconfig". And, frankly, _any_ sbin command which can
be meaningfully run as nonroot, particularly for reporting.

I have always found this "oh its for root" distinction pretty vacuous,
and outstandingly annoying when basic system querying stuff isn't in the
default $PATH because of this. Maybe it is because I've been a sysadmin
for many years, but most physical machines are personal machines these
days anyway - we're our own sysadmins.

> Already /bin has way>too much stuff in it (although I don't see any other way to practically
>do it without ridiculous PATHs searching all over the disk).

Like modules with many names, the number of things in /bin or /usr/bin
is generally irrelevant. Nobody does an "ls" in there without expecting
a fair amount of stuff - like imports, we invoke commands by name. Who
_cares_ how many names there are?

>Having said that, I note that on my CentOS 7 workstation, sbin seems to
>be in the path by default. So that negates my argument I suppose.
>Although I might have made that change myself.

I have historically had to add it to my $PATH on most platforms.

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Dan Sommers]

On 7/24/19 10:24 PM, Michael Torrie wrote:

> ... In more recent times, binaries that are mostly applicable to the

> super user go there. I don't see why you would want to merge those.

> A normal user rarely has need of much in /sbin. Already /bin has way

> too much stuff in it (although I don't see any other way to
> practically do it without ridiculous PATHs searching all over the
> disk).

On ancient file systems (SysV?), ISTR something like 1400 files in a
directory being some sort of limit, or perhaps a major performance
issue. Separate directories (including /usr/X11/bin) mitigated that,
too.

> Having said that, I note that on my CentOS 7 workstation, sbin seems
> to be in the path by default. So that negates my argument I suppose.
> Although I might have made that change myself.

My .shrc file has all sorts of leftovers from the old days, but my
current Linux PATH is just $HOME/bin, $HOME/local/bin, and /usr/bin.

Get Off My Lawn,
Dan

[MRAB]

On 2019-07-25 03:57, Dan Sommers wrote:
> On 7/24/19 10:24 PM, Michael Torrie wrote:
>
> > ... In more recent times, binaries that are mostly applicable to the
> > super user go there. I don't see why you would want to merge those.
> > A normal user rarely has need of much in /sbin. Already /bin has way
> > too much stuff in it (although I don't see any other way to
> > practically do it without ridiculous PATHs searching all over the
> > disk).
>
> On ancient file systems (SysV?), ISTR something like 1400 files in a
> directory being some sort of limit, or perhaps a major performance
> issue. Separate directories (including /usr/X11/bin) mitigated that,
> too.
>

[snip]

At one time I used an MS-DOS application suite whose database component
stored predefined queries in individual files. There were _many_
predefined queries, so _many_ small files.

This took up a lot of disk space, and running a predefined query was
slow because of the need to search for the file (the FAT layout used a
linked list).

I put all of the predefined queries into a single file and called an
external program to look up the query on demand, copy it into a file,
and then performed the query.

That saved a _lot_ of disk space, and, despite the overhead of calling
an external program, was _much_ faster!

[Thomas 'PointedEars' Lahn]

Michael Torrie wrote:

> On 7/24/19 4:20 PM, Cameron Simpson wrote:
>> That is some progress, hooray. Then there's just sbin -> bin to go.
>

> I suppose in the olden days sbin was for static binaries, […]

No, “sbin” is short for “*system* binaries” which in general only the
superuser should be able to execute.

Which is why the above is a Very Bad Idea[tm].

<https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s16.html>

--
PointedEars

Twitter: @PointedEars2
Please do not cc me. /Bitte keine Kopien per E-Mail.

[Thomas 'PointedEars' Lahn]

Tim Daneliuk wrote:

> So, no, do NOT encode the hard location - ever. Always use env to
> discover the one that
> the user has specified.

By specifying /usr/bin/env instead of /usr/bin/python3, for example, you are
just hard-coding the path of another program, hoping that one path is more
likely to exist than the other. What is the/your rationale for that?

[Cameron Simpson]

On 24Jul2019 22:57, Dan Sommers <2QdxY4Rz...@potatochowder.com> wrote:
>On 7/24/19 10:24 PM, Michael Torrie wrote:
>> ... In more recent times, binaries that are mostly applicable to the
>> super user go there. I don't see why you would want to merge those.
>> A normal user rarely has need of much in /sbin. Already /bin has way
>> too much stuff in it (although I don't see any other way to
>> practically do it without ridiculous PATHs searching all over the
>> disk).
>
>On ancient file systems (SysV?), ISTR something like 1400 files in a
>directory being some sort of limit, or perhaps a major performance
>issue. Separate directories (including /usr/X11/bin) mitigated that,
>too.

I don't recall there being any specific hard limit though. What does
happen in decent modern filesystems is that directories have a hash
table in them for lookups, making a file name access constant time
instead of a linear search.

Also, shells like zsh keep a hash table of the available commands to
speed invocation, avoiding a linear $PATH search.

I don't recall stuff like /usr/X11/bin being aimed at performance (but I
would not necessarily know); it has always seemed more like partitioning
out various software groups - in this case the X11 Window System stuff.
If nothing else, this makes separate development easier.

>> Having said that, I note that on my CentOS 7 workstation, sbin seems
>> to be in the path by default. So that negates my argument I suppose.
>> Although I might have made that change myself.
>
>My .shrc file has all sorts of leftovers from the old days, but my
>current Linux PATH is just $HOME/bin, $HOME/local/bin, and /usr/bin.

Hmm. 21 componetent in my $PATH here :-)

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Eli the Bearded]

In comp.lang.python, Thomas 'PointedEars' Lahn <use...@PointedEars.de> wrote:
> Michael Torrie wrote:
>> On 7/24/19 4:20 PM, Cameron Simpson wrote:
>>> That is some progress, hooray. Then there's just sbin -> bin to go.
>> I suppose in the olden days sbin was for static binaries, […]
> No, “sbin” is short for “*system* binaries” which in general only the
> superuser should be able to execute.

I think Michael is confusing "sbin" with the statically linked utilities
some systems (particularly older ones, but also FreeBSD in /rescue/)
have for repairing the system when things start to go bad. You'd want
a shell (sh is great), a basic editor (eg, eg), and a smattering of
other tools, akin to the ones listed as "must be in /sbin" in your
linuxfoundation link.

But more than a few utilities in /sbin are useful for non-superusers.
Eg ip or ifconfig for informational purposes like identifying current
IP address and getting MAC.

> Which is why the above is a Very Bad Idea[tm].

Why? Programs that can *only* be usefully run by a privileged user
or in a system context (eg halt or getty) already *must* prevent non
privileged use. So why would it be a Very Bad Idea[tm] to have them in
a common directory like /bin/?

(Feel free to crosspost and set follow-ups to another group if you like.
But I would suggest *not* a Linux group, since this is something general
to all Unix-likes.)

> <https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s16.html>

Elijah
------
uses both netbsd and linux regularly

[Thomas 'PointedEars' Lahn]

Eli the Bearded wrote:
^^^^^^^^^^^^^^^
Please post here using your real name.

> In comp.lang.python, Thomas 'PointedEars' Lahn <use...@PointedEars.de>
> wrote:

Attribution _line_, not attribution novel.

>> Michael Torrie wrote:
>>> On 7/24/19 4:20 PM, Cameron Simpson wrote:
>>>> That is some progress, hooray. Then there's just sbin -> bin to go.
>>> I suppose in the olden days sbin was for static binaries, […]
>> No, “sbin” is short for “*system* binaries” which in general only the
>> superuser should be able to execute.
>
> I think Michael is confusing "sbin" with the statically linked utilities
> some systems (particularly older ones, but also FreeBSD in /rescue/)
> have for repairing the system when things start to go bad. You'd want
> a shell (sh is great), a basic editor (eg, eg), and a smattering of
> other tools, akin to the ones listed as "must be in /sbin" in your
> linuxfoundation link.

ACK.

> But more than a few utilities in /sbin are useful for non-superusers.
> Eg ip or ifconfig for informational purposes like identifying current
> IP address and getting MAC.

ACK. But what appears useful for a non-superuser can be viewed as
compromising system security, or opening ways to make that easier,
by superusers/system administrators.

Keep in mind that originally, and in fact still due to servers on the
Internet, the majority of Unices have not been/are not only used by one
person each which is both a non-superuser and a superuser of the computer
system:

<https://en.wikipedia.org/wiki/Usage_share_of_operating_systems>

>> Which is why the above is a Very Bad Idea[tm].
>
> Why? Programs that can *only* be usefully run by a privileged user
> or in a system context (eg halt or getty) already *must* prevent non
> privileged use. So why would it be a Very Bad Idea[tm] to have them in
> a common directory like /bin/?

Because that a file should only or usually be executed by a superuser does
not imply that the owner of that file must be a superuser, or that it has
execute permission only for one superuser group or more, and vice-versa.

Also, it is easier to keep files that usually should only be executed by
a superuser in a separate directory, so that they are not immediately
available or listed to or for other users [e.g. in shell command resolution
(along PATH), in a directory listing, or in tab completion].

> (Feel free to crosspost and set follow-ups to another group if you like.
> But I would suggest *not* a Linux group, since this is something general
> to all Unix-likes.)

ACK. X-Post & F’up2 comp.unix.misc.