shebanged script not executed right - wth

[fatty.merc...@aceecat.org]

I have the following setup, which works as intended on multiple quite
different penguin systems:

$ ls -la /opt/compat
total 16
drwxr-xr-x 2 root wheel 512 Aug 8 19:46 .
drwxr-xr-x 4 root wheel 512 Aug 3 13:17 ..
lrwxr-xr-x 1 root wheel 19 Aug 3 15:16 bash -> /usr/local/bin/bash
lrwxr-xr-x 1 root wheel 19 Aug 3 13:18 perl -> /usr/local/bin/perl
-rwxr-xr-x 1 root wheel 45 Aug 3 13:18 python3
-rwxr-xr-x 1 root wheel 1462 Aug 8 19:46 relaymail
$ cat /opt/compat/python3
#! /bin/sh

exec /usr/local/bin/python3 "$@"
$

On the penguin systems, of course it's /usr in place of /usr/local.

The idea is that scripts can put these locations for the interpreters
into their shebangs, so I can use literally the same script on
penguins and daemons.

Like relaymail here - that was supposed to be the first such script.
It is a python3 script:

$ head /opt/compat/relaymail
#! /opt/compat/python3

import sys
from argparse import (ArgumentParser, ArgumentDefaultsHelpFormatter)
import os
import subprocess
import json

CONFIG = '/usr/local/etc/relaymail.json'
BUFSIZE = 4096

Further details of relaymail aren't important.

And now the problem: this works:

$ /opt/compat/python3 /opt/compat/relaymail -h | head -3
usage: relaymail [-h] [-f ADDRESS] [-o OTHER] RCPT [RCPT ...]

simple mail submission agent with few options

but this fails mysteriously:

$ /opt/compat/relaymail -h
/opt/compat/relaymail: line 3: import: command not found
/opt/compat/relaymail: line 4: syntax error near unexpected token `('
/opt/compat/relaymail: line 4: `from argparse import (ArgumentParser, ArgumentDefaultsHelpFormatter)'

well, not completely mysteriously -- it sure seems like it's trying to
run it as a shell script ???

WTH WTH ??? IS THIS A KERNEL BUG ???

--
Ian

[Souji Thenria]

On Sat Aug 9, 2025 at 7:39 AM CEST, wrote:
> I have the following setup, which works as intended on multiple quite
> different penguin systems:
>
> $ ls -la /opt/compat
> total 16
> drwxr-xr-x 2 root wheel 512 Aug 8 19:46 .
> drwxr-xr-x 4 root wheel 512 Aug 3 13:17 ..
> lrwxr-xr-x 1 root wheel 19 Aug 3 15:16 bash -> /usr/local/bin/bash
> lrwxr-xr-x 1 root wheel 19 Aug 3 13:18 perl -> /usr/local/bin/perl
> -rwxr-xr-x 1 root wheel 45 Aug 3 13:18 python3
> -rwxr-xr-x 1 root wheel 1462 Aug 8 19:46 relaymail
> $ cat /opt/compat/python3
> #! /bin/sh
>
> exec /usr/local/bin/python3 "$@"
> $
>
> On the penguin systems, of course it's /usr in place of /usr/local.
>
> The idea is that scripts can put these locations for the interpreters
> into their shebangs, so I can use literally the same script on
> penguins and daemons.
>

You can use '#!/usr/bin/env python3' as shebang in your scripts. This
would work on Linux as well as FreeBSD, OpenBSD and so on.

Regards,
Souji

--
Souji Thenria
Website: www.souji-thenria.net

[Richard M Kreuter]

fatty.merc...@aceecat.org writes:

> $ cat /opt/compat/python3
> #! /bin/sh
>
> exec /usr/local/bin/python3 "$@"
> $

<snip>

> $ head /opt/compat/relaymail
> #! /opt/compat/python3

I believe that on FreeBSD (and probably most Unices, historically), the
pathname after "#!" must resolve to an executable binary, i.e., an
interpreted program cannot serve as an #!-line interpreter.

Could you have /opt/compat/python3 be a symlink to
/usr/local/bin/python3?

> IS THIS A KERNEL BUG ???

Unices have never agreed exactly on #!-line behavior. Here's one
person's attempt to document differences:

https://www.in-ulm.de/~mascheck/various/shebang/

The relevant section for this thread is

https://www.in-ulm.de/~mascheck/various/shebang/#interpreter-script

Regards,
Richard

[Dag-Erling Smørgrav]

Richard M Kreuter <kre...@progn.net> writes:
> I believe that on FreeBSD (and probably most Unices, historically), the
> pathname after "#!" must resolve to an executable binary, i.e., an
> interpreted program cannot serve as an #!-line interpreter.

Correct. As Souji points out, you can work around this limitation by
using `#!/usr/bin/env python3`, which has the advantage of working
regardless of where python3 is installed, as long as it's in your PATH.

Linux supports nested shebangs, and it wouldn't be hard to implement,
but it raises the question of what to do in case of a loop. Without
loop detection, we will eventually hit ARG_MAX (because argv grows with
each additional shebang level), but ARG_MAX is pretty big these days, so
it may take a while to fail.

DES
--
Dag-Erling Smørgrav - d...@FreeBSD.org

[Dag-Erling Smørgrav]

Dag-Erling Smørgrav <d...@FreeBSD.org> writes:
> Richard M Kreuter <kre...@progn.net> writes:
> > I believe that on FreeBSD (and probably most Unices, historically), the

> > pathname after "#!" must resolve to an executable binary, [...]
> Correct.

OK, that _used_ to be the answer, but on closer inspection, there is
nothing in the code to prevent it from working, and it does seem to work
on 13, 14, and 15. I'll have to take a closer look at the OP.

[Dag-Erling Smørgrav]

Dag-Erling Smørgrav <d...@FreeBSD.org> writes:
> [...] there is nothing in the code to prevent [double indirection]
> from working, and it does seem to work on 13, 14, and 15 [...]

Right, so in my testing, I was able to do double indirection (the
interpreter can be interpreted) but not triple indirection (the
interpreter's interpreter must be a binary).

I think the reason double indirection works is that the first level of
indirection is handled by the shell (this will obviously vary from one
shell to another), so the kernel only sees the second. Thus double
indirection becomes single indirection, which is permitted, and triple
indirection becomes double indirection, which is not.

I found the place in the code that prevents double indirection, in
sys/kern/imgact_shell.c:

/*
* Don't allow a shell script to be the shell for a shell
* script. :-)
*/
if (imgp->interpreted & IMGACT_SHELL)
return (ENOEXEC);

We could remove this with no ill effects, but I'm not sure we want to.

[Michael Sierchio]

On Sat, Aug 9, 2025 at 9:05 AM Dag-Erling Smørgrav <d...@freebsd.org> wrote:

I found the place in the code that prevents double indirection, in
sys/kern/imgact_shell.c:

        /*
         * Don't allow a shell script to be the shell for a shell
         *      script. :-)
         */
        if (imgp->interpreted & IMGACT_SHELL)
                return (ENOEXEC);

We could remove this with no ill effects, but I'm not sure we want to.

Loop detection?  Privilege escalation prevention?  I might be entirely wrong, but the potential hazards seem to outweigh any possibly benefit. 

[Dag-Erling Smørgrav]

Michael Sierchio <ku...@tenebras.com> writes:

> Dag-Erling Smørgrav <d...@freebsd.org> writes:
> > We could remove this with no ill effects, but I'm not sure we want
> > to.
> Loop detection?

Not strictly necessary; we will eventually run into ARG_MAX, and the
only thing that grows while we loop is argv.

> Privilege escalation prevention?

Completely orthogonal.

> I might be entirely wrong, but the potential hazards seem to outweigh
> any possibly benefit.

Did you miss the second half of my sentence?

[fatty.merc...@aceecat.org]

On Fri, Aug 08, 2025 at 10:39:12PM -0700, fatty.merc...@aceecat.org wrote:

...

> #! /bin/sh

> exec /usr/local/bin/python3 "$@"

Thanks for the discussion, it has been very useful and clarifying.

First, why not /usr/bin/env and PATH: this has to do with the reliance
in the python world for "virtual environments" (don't blame me for the
poor term!). Roughly, you can have all kinds of pythons slithering
about and showing up in your PATH, and you don't want *those* to run
generic scripts, you want the system python to do so.

This is discussed for example here:

https://utcc.utoronto.ca/~cks/space/blog/unix/UsingEnvRarelyUseful

So then there is the option of a direct symlink instead of a shim
script. In fact, that's what I do for bash and perl, so why not for
python? And the "because" was the same as above: I wasn't sure *how*
python detects it's in one of the so called virtual environments
(venvs), and I was afraid it might be confused by a symlink into
feeling it's in one. If you look inside a venv you see that the python
executable there is in fact a symlink to the system python.

But as it turns out, python must be using some other information to
make this decision. I suspect it is the presence (and perhaps format)
of the file bin/../pyvenv.cfg , but so far I haven't been able to find
a definitive confirmation of this; I'll be grateful if anyone provides
a confirmation. Maybe it's only clear from python source code.

Anyway, a symlink from /opt/compat/python3 to the system python seems
to work, so thanks.

--
Ian

[Richard M Kreuter]

Dag-Erling Smørgrav <d...@FreeBSD.org> wrote:
> Dag-Erling Smørgrav <d...@FreeBSD.org> writes:
> > [...] there is nothing in the code to prevent [double indirection]
> > from working, and it does seem to work on 13, 14, and 15 [...]
>
> Right, so in my testing, I was able to do double indirection (the
> interpreter can be interpreted) but not triple indirection (the
> interpreter's interpreter must be a binary).

Could you share how you're testing for this behavior? (If it's not true
that script interpreters cannot be scripts, I would like to find a
correct explanation of the OP's observed behavior.)

Thanks,
Richard

[Dag-Erling Smørgrav]

Richard M Kreuter <kre...@progn.net> writes:

> Could you share how you're testing for this behavior?

I just created three scripts.

[Richard M Kreuter]

Dag-Erling Smørgrav <d...@FreeBSD.org> wrote:
> Richard M Kreuter <kre...@progn.net> writes:
> > Could you share how you're testing for this behavior?
>
> I just created three scripts.

Here's how I tried to emulate the OP's report. It seems consistent with
the idea that a script cannot be a shebang interpreter. But I must be
mistaken, as you've found otherwise. What am I missing or
misunderstanding?

--
$ freebsd-version -urk
14.3-RELEASE
14.3-RELEASE
14.3-RELEASE-p1
$ pwd
/home/me
$ cat > script1
#!/bin/sh
exec /usr/local/bin/python3.11 "$@"
^D
$ cat > script2
#!/home/me/script1
print("foo")
^D
$ chmod 755 script1 script2
# Verify that script2 works
$ python3.11 script2
foo
# Verify that script1 works
$ ./script1 script2
foo
# But the shebang line doesn't work
$ ./script2
./script2: 2: Syntax error: word unexpected (expecting ")")
# And for clarity, that error comes from the shell:
$ sh -c 'print("foo")'
sh: Syntax error: word unexpected (expecting ")")
# Now let's see what's really going on
$ truss -a sh -c './script2'
--

It seems to me the relevant lines from truss are:

--
execve("./script2",[ "./script2" ],0x2a7626840208) ERR#8 'Exec format error'
openat(AT_FDCWD,"./script2",O_RDONLY|O_NONBLOCK,00) = 3 (0x3)
pread(3,"#!/home/me/script1\nprint(""...,256,0x0) = 38 (0x26)
close(3) = 0 (0x0)
execve("/bin/sh",[ "/bin/sh", "./script2" ],0x2a7626840208) EJUSTRETURN
--

The failure of the first execve is consistent with the theory that a
script cannot be a shebang interpreter. So I'm not sure how to reconcile
that with your finding.

Thanks in advance,
Richard

[Dag-Erling Smørgrav]

Richard M Kreuter <kre...@progn.net> writes:
> Here's how I tried to emulate the OP's report. It seems consistent
> with the idea that a script cannot be a shebang interpreter. But I
> must be mistaken, as you've found otherwise. What am I missing or
> misunderstanding?

Nothing. I didn't find otherwise. The kernel returns ENOEXEC if it
encounters a second level of indirection, but the shell (or libc) may
react to that by trying to execute the program as a shell script, which
may or may not work.