Survey: do you not have python2 in the PATH on relevant hosts?

45 views

Skip to first unread message

Rob Browning

unread,

Jul 18, 2015, 4:15:04 PM7/18/15

to bup-...@googlegroups.com

Could anyone who doesn't have python2 in the PATH reply (on systems
where you want to run bup)?

i.e. I'm looking for any relevant systems where the equivalent of "type
-p python2" doesn't return something useful, or more to the point, where

#!/usr/bin/env python2

doesn't work as well as, or better than

#!/usr/bin/env python

for programs that require python version 2.*.

cf. https://www.python.org/dev/peps/pep-0394/

Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Greg Troxel

unread,

Jul 18, 2015, 8:54:07 PM7/18/15

to Rob Browning, bup-...@googlegroups.com

Rob Browning <r...@defaultvalue.org> writes:

> Could anyone who doesn't have python2 in the PATH reply (on systems
> where you want to run bup)?
>
> i.e. I'm looking for any relevant systems where the equivalent of "type
> -p python2" doesn't return something useful, or more to the point, where
>
> #!/usr/bin/env python2
>
> doesn't work as well as, or better than
>
> #!/usr/bin/env python
>
> for programs that require python version 2.*.

On pkgsrc systems, only the fully-qualified versioned path is present.

$ ls -l /usr/pkg/bin/python*
-rwxr-xr-x 1 root wheel 3908 Sep 18 2014 /usr/pkg/bin/python2.7
-rwxr-xr-x 1 root wheel 1672 Sep 18 2014 /usr/pkg/bin/python2.7-config

The referenced PEP does not seem unreasonable, but on the other hand
having a python2 reference that can move versions seems unsound, as in
general checking python for usability involves checking pyXX-foo scripts
as well. FWIW, this is the first I've heard of the "python2"
convention.

So in my view, the only ok thing is to search for python versions in
some longish list, and actually subsitute paths.

Rob Browning

unread,

Jul 19, 2015, 11:41:08 AM7/19/15

to Greg Troxel, bup-...@googlegroups.com

Greg Troxel <g...@lexort.com> writes:

> On pkgsrc systems, only the fully-qualified versioned path is present.
>
> $ ls -l /usr/pkg/bin/python*
> -rwxr-xr-x 1 root wheel 3908 Sep 18 2014 /usr/pkg/bin/python2.7
> -rwxr-xr-x 1 root wheel 1672 Sep 18 2014 /usr/pkg/bin/python2.7-config

Thanks. For reference, here's why I asked.

A bit back, I tried out a "fix" where we put this at the top of every
relevant file:

#!/bin/sh
"""":
exec "${PYTHON:-python}" $0 ${1+"$@"}
"""
# End preamble.
...

which works fine from the source tree, respecting PYTHON. Then during
install, sed replaces the preamble with any PYTHON setting, or with
"/usr/bin/env python".

Ignoring for the moment whether that has exactly the right semantics
(i.e. the current version doesn't capture the PYTHON value at configure
time the way autoconf would), I asked the question because I'd wondered
if python2 might be a better default now than python, and whether it
might "good enough" for an interim fix, i.e. just:

#!/usr/bin/env python2

In any case, appreciate the report.

Greg Troxel

unread,

Jul 19, 2015, 1:25:14 PM7/19/15

to Rob Browning, bup-...@googlegroups.com

Rob Browning <r...@defaultvalue.org> writes:

> A bit back, I tried out a "fix" where we put this at the top of every
> relevant file:
>
> #!/bin/sh
> """":
> exec "${PYTHON:-python}" $0 ${1+"$@"}
> """
> # End preamble.
> ...
>
> which works fine from the source tree, respecting PYTHON. Then during
> install, sed replaces the preamble with any PYTHON setting, or with
> "/usr/bin/env python".

What if finding a python interpreter and binding to it (via a
fully-qualified path) was just non-optional in configure? If no path
given, search for a number of names in some order, basically
fully-qualified first, then newer first but also factoring in what bup
is known to work with, Then finally look for python2 and python3, and
then python.

The list would be, I think:

python3.4
python3.3
python2.7
python3
python2
python

(or perhaps flip 2/3 in priority).

This way we get as specific as we can, if the user/packaging-system
didn't instruct with PYTHON.

> Ignoring for the moment whether that has exactly the right semantics
> (i.e. the current version doesn't capture the PYTHON value at configure
> time the way autoconf would), I asked the question because I'd wondered
> if python2 might be a better default now than python, and whether it
> might "good enough" for an interim fix, i.e. just:
>
> #!/usr/bin/env python2
>
> In any case, appreciate the report.

If PYTHON in the configure env is respected and results in binding to
the fully-qualified name, then the "/usr/bin/env python2" scheme seems
like it's not relevant to environments that want tight binding and set
PYTHON.

But, why would you want an installed bup to usr /usr/bin/env python2
instead of finding the actual path at configure time and substituting
it?

These files:

/usr/pkg/lib/bup/bup/__init__.py
/usr/pkg/lib/bup/bup/_helpers.so
/usr/pkg/lib/bup/bup/_version.py

are surely compiled against particular includes/libs. Does it really
work to build against 2.6 and run with 2.7, or 3.3 and 3.4?

ldd shows:

/usr/pkg/lib/bup/bup/_helpers.so:
-lpython2.7.1.0 => /usr/pkg/lib/libpython2.7.so.1.0
-lutil.7 => /usr/lib/libutil.so.7
-lc.12 => /usr/lib/libc.so.12
-lm.0 => /usr/lib/libm.so.0
-lpthread.0 => /usr/lib/libpthread.so.0

Rob Browning

unread,

Jul 21, 2015, 12:31:04 AM7/21/15

to Greg Troxel, bup-...@googlegroups.com

Greg Troxel <g...@lexort.com> writes:

> The list would be, I think:
>
> python3.4
> python3.3
> python2.7
> python3
> python2
> python
>
> (or perhaps flip 2/3 in priority).

Of course for us at the moment it'd be

python2.7
python2
python

and possibly even

python2.7
python2.6
python2.5
python2
python

Since we claim to support 2.5 in the README, though I'm a bit skeptical
it still works.

I vaguely recall someone mentioning 2.6 in the not too distant past
(i.e. stuck on a machine where they had no python alternative), but it's
certainly closer to the point where we can just require 2.7.

Regardless, if we end up finding only python2 or python, I suppose we
might want to probe a bit further to make sure it satisfies our minimum
version requirement.

> If PYTHON in the configure env is respected and results in binding to
> the fully-qualified name, then the "/usr/bin/env python2" scheme seems
> like it's not relevant to environments that want tight binding and set
> PYTHON.
>
> But, why would you want an installed bup to usr /usr/bin/env python2
> instead of finding the actual path at configure time and substituting
> it?

I can imagine scenarios where the admin might want to track python2, so
that when they upgrade and 2.7 becomes the stable python2 (over say
2.6), bup just continues to work. But...

> These files:
>
> /usr/pkg/lib/bup/bup/__init__.py
> /usr/pkg/lib/bup/bup/_helpers.so
> /usr/pkg/lib/bup/bup/_version.py
>
> are surely compiled against particular includes/libs. Does it really
> work to build against 2.6 and run with 2.7, or 3.3 and 3.4?

An excellent point -- bup's not a pure Python application.

Which makes me wonder about embedding the X.Y version in our Python code
at build time, so that we can check it at startup and issue a friendly
error message instead of (presumably) just dying with a module
load/linker error.

In any case, I'm leaning toward the idea of embedding the absolute path
of the version of Python we find, since there would always be a
workaround:

PYTHON='/usr/bin/env blarg' ./configure

Thanks

Greg Troxel

unread,

Jul 21, 2015, 7:32:57 AM7/21/15

to Rob Browning, bup-...@googlegroups.com

Rob Browning <r...@defaultvalue.org> writes:

> Greg Troxel <g...@lexort.com> writes:
> Of course for us at the moment it'd be
>
> python2.7
> python2
> python
>
> and possibly even
>
> python2.7
> python2.6
> python2.5
> python2
> python
>
> Since we claim to support 2.5 in the README, though I'm a bit skeptical
> it still works.

> I vaguely recall someone mentioning 2.6 in the not too distant past
> (i.e. stuck on a machine where they had no python alternative), but it's
> certainly closer to the point where we can just require 2.7.

FWIW, pkgsrc just deleted 2.6. The only situation I'm aware of where
people have old python is red hat, and there they should just build bup
and all its dependencies (including python 2.7) from pkgsrc.

I would definitely adjust the README to limit support to either 2.6/2.7
or just 2.7.

> Regardless, if we end up finding only python2 or python, I suppose we
> might want to probe a bit further to make sure it satisfies our minimum
> version requirement.

Are there packaging systems that have only python or python2 and not
python2.7? Does that happen with any sane/instruction-following
install?

What I'm getting at, is do we need to look for python or python2 at all?

>> But, why would you want an installed bup to usr /usr/bin/env python2
>> instead of finding the actual path at configure time and substituting
>> it?
>
> I can imagine scenarios where the admin might want to track python2, so
> that when they upgrade and 2.7 becomes the stable python2 (over say
> 2.6), bup just continues to work. But...

If using a packaging system, the recorded dependencies will force a bup
rebuild (as they should).

>> These files:
>>
>> /usr/pkg/lib/bup/bup/__init__.py
>> /usr/pkg/lib/bup/bup/_helpers.so
>> /usr/pkg/lib/bup/bup/_version.py
>>
>> are surely compiled against particular includes/libs. Does it really
>> work to build against 2.6 and run with 2.7, or 3.3 and 3.4?
>
> An excellent point -- bup's not a pure Python application.
>
> Which makes me wonder about embedding the X.Y version in our Python code
> at build time, so that we can check it at startup and issue a friendly
> error message instead of (presumably) just dying with a module
> load/linker error.

I don't see that as necessary if:

> In any case, I'm leaning toward the idea of embedding the absolute path
> of the version of Python we find, since there would always be a
> workaround:

If the absolute path is substituted in, then bup will fail hard if that
path is missing. I think that's the right thing, and we are then as
safe as we can be against underlying changes.

> PYTHON='/usr/bin/env blarg' ./configure

Sure, that's a scary/flexible escape hatch.

Rob Browning

unread,

Jul 24, 2015, 2:03:02 AM7/24/15

to Greg Troxel, bup-...@googlegroups.com

Greg Troxel <g...@lexort.com> writes:

> FWIW, pkgsrc just deleted 2.6. The only situation I'm aware of where
> people have old python is red hat, and there they should just build bup
> and all its dependencies (including python 2.7) from pkgsrc.

If I recall correctly, the users in question were on NAS boxes with
fairly limited options. In any case, we should at least make sure the
docs reflect reality.

> If the absolute path is substituted in, then bup will fail hard if that
> path is missing. I think that's the right thing, and we are then as
> safe as we can be against underlying changes.

Here's what I'm currently toying with on that front. I have something
like this in configure:

bup_python="$PYTHON"
test -z "$bup_python" && bup_python="$(bup_find_prog python2.7 '')"
test -z "$bup_python" && bup_python="$(bup_find_prog python2.6 '')"
test -z "$bup_python" && bup_python="$(bup_find_prog python2 '')"
test -z "$bup_python" && bup_python="$(bup_find_prog python '')"
if test -z "$bup_python"; then
AC_FAIL "ERROR: unable to find python"
else
echo "#!/bin/sh" > ../cmd/bup-python
printf "exec %q \"\$@\"" "$bup_python" >> ../cmd/bup-python
chmod u+x ../cmd/bup-python
fi

And then perhaps as a preamble in (for example) main.py:

#!/bin/sh
"""":
export PATH="$(dirname "$0")/cmd:$PATH"
exec bup-python $0 ${1+"$@"}
"""
# End preamble.

along with a version guard that will check the actual Python major/minor
pair at startup and exit with an error if it's unsupported (say it's
actually Python 3).

The reason for bup-python, which would not be installed, and the
preamble, which would be replaced during "make install" with
"#!$bup_python", is so that we don't have to immediately add a large
number of foo.in templates (or something similar).

In the longer run, we may want other changes, but this might be a way to
start.

Greg Troxel

unread,

Jul 24, 2015, 6:43:10 AM7/24/15

to Rob Browning, bup-...@googlegroups.com

Rob Browning <r...@defaultvalue.org> writes:

> Here's what I'm currently toying with on that front. I have something
> like this in configure:
>
> bup_python="$PYTHON"
> test -z "$bup_python" && bup_python="$(bup_find_prog python2.7 '')"
> test -z "$bup_python" && bup_python="$(bup_find_prog python2.6 '')"
> test -z "$bup_python" && bup_python="$(bup_find_prog python2 '')"
> test -z "$bup_python" && bup_python="$(bup_find_prog python '')"
> if test -z "$bup_python"; then
> AC_FAIL "ERROR: unable to find python"
> else
> echo "#!/bin/sh" > ../cmd/bup-python
> printf "exec %q \"\$@\"" "$bup_python" >> ../cmd/bup-python
> chmod u+x ../cmd/bup-python
> fi
>
> And then perhaps as a preamble in (for example) main.py:
>
> #!/bin/sh
> """":
> export PATH="$(dirname "$0")/cmd:$PATH"
> exec bup-python $0 ${1+"$@"}
> """
> # End preamble.
>
> along with a version guard that will check the actual Python major/minor
> pair at startup and exit with an error if it's unsupported (say it's
> actually Python 3).

That looks good to me.

> The reason for bup-python, which would not be installed, and the
> preamble, which would be replaced during "make install" with
> "#!$bup_python", is so that we don't have to immediately add a large
> number of foo.in templates (or something similar).
>
> In the longer run, we may want other changes, but this might be a way to
> start.

Both the bup_python install-time substitution and \forall scripts (have
foo.in) are icky, but I can certainly see picking install-time changes.
It would be really nice if the install operation installed a modified
script but did not change anything in the source tree. Presumably
avoiding writing source files (a good goal) is the point.

Rob Browning

unread,

Jul 24, 2015, 7:31:06 PM7/24/15

to Greg Troxel, bup-...@googlegroups.com

Greg Troxel <g...@lexort.com> writes:

> Both the bup_python install-time substitution and \forall scripts (have
> foo.in) are icky, but I can certainly see picking install-time changes.
> It would be really nice if the install operation installed a modified
> script but did not change anything in the source tree. Presumably
> avoiding writing source files (a good goal) is the point.

Exactly, and already working here. We just sed the files during install
to replace the preamble.

Rob Browning

unread,

Jul 24, 2015, 8:48:01 PM7/24/15

to bup-...@googlegroups.com, Greg Troxel

Search for a "reasonable" python in ./configure (or respect PYTHON), and
use that to create a source tree cmd/python-cmd.sh that all
of the relevant invocations in the source tree respect.

During install, rewrite all of the relevant #! lines to refer to the
./configure selected Python.

(NOTE: this patch is unfinished. At a minimum, it doesn't cover the
test tools yet.)

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---

Plausible approach (roughly speaking)?

Makefile | 18 +++++++++++++-----
README.md | 6 ++++++
cmd/bloom-cmd.py | 7 ++++++-
cmd/cat-file-cmd.py | 7 ++++++-
cmd/daemon-cmd.py | 7 ++++++-
cmd/damage-cmd.py | 7 ++++++-
cmd/drecurse-cmd.py | 7 ++++++-
cmd/fsck-cmd.py | 7 ++++++-
cmd/ftp-cmd.py | 7 ++++++-
cmd/fuse-cmd.py | 7 ++++++-
cmd/help-cmd.py | 7 ++++++-
cmd/import-duplicity-cmd.py | 7 ++++++-
cmd/index-cmd.py | 7 ++++++-
cmd/init-cmd.py | 7 ++++++-
cmd/join-cmd.py | 7 ++++++-
cmd/list-idx-cmd.py | 7 ++++++-
cmd/ls-cmd.py | 7 ++++++-
cmd/margin-cmd.py | 7 ++++++-
cmd/memtest-cmd.py | 7 ++++++-
cmd/meta-cmd.py | 7 ++++++-
cmd/midx-cmd.py | 7 ++++++-
cmd/mux-cmd.py | 7 ++++++-
cmd/newliner-cmd.py | 7 ++++++-
cmd/on--server-cmd.py | 7 ++++++-
cmd/on-cmd.py | 7 ++++++-
cmd/random-cmd.py | 7 ++++++-
cmd/restore-cmd.py | 7 ++++++-
cmd/save-cmd.py | 7 ++++++-
cmd/server-cmd.py | 7 ++++++-
cmd/split-cmd.py | 7 ++++++-
cmd/tag-cmd.py | 7 ++++++-
cmd/tick-cmd.py | 7 ++++++-
cmd/version-cmd.py | 7 ++++++-
cmd/web-cmd.py | 7 ++++++-
cmd/xstat-cmd.py | 7 ++++++-
config/configure | 12 +++++++++++-
main.py | 8 +++++++-
37 files changed, 235 insertions(+), 40 deletions(-)

diff --git a/Makefile b/Makefile
index 988a9cc..6260553 100644
--- a/Makefile
+++ b/Makefile
@@ -30,8 +30,15 @@ Documentation/all: $(bup_deps)
$(current_sampledata):
t/configure-sampledata --setup

+PYTHON = $(shell cmd/bup-python -c 'import sys; print sys.executable')
+
+define install-python-bin
+ set -e; \
+ sed -e '1 s|.*|#!$(PYTHON)|; 2,/^# End preamble\.$$/d' $1 > $2; \
+ chmod 0755 $2;
+endef
+
INSTALL=install
-PYTHON=python
PREFIX=/usr
MANDIR=$(DESTDIR)$(PREFIX)/share/man
DOCDIR=$(DESTDIR)$(PREFIX)/share/doc/bup
@@ -49,10 +56,11 @@ install: all
$(INSTALL) -m 0644 \
Documentation/*.html \
$(DOCDIR)
- $(INSTALL) -pm 0755 bup $(BINDIR)
- $(INSTALL) -pm 0755 \
- cmd/bup-* \
- $(LIBDIR)/cmd
+ $(call install-python-bin,bup,"$(BINDIR)/bup")
+ set -e; \
+ for cmd in $$(ls cmd/bup-* | grep -v cmd/bup-python); do \
+ $(call install-python-bin,"$$cmd","$(LIBDIR)/$$cmd") \
+ done
$(INSTALL) -pm 0644 \
lib/bup/*.py \
$(LIBDIR)/bup
diff --git a/README.md b/README.md
index 7eb4c2d..01c2e5d 100644
--- a/README.md
+++ b/README.md
@@ -154,6 +154,12 @@ From source

make install DESTDIR=/opt/bup PREFIX=''

+ - The Python executable that bup will use is chosen by ./configure,
+ which will search for a reasonable version unless PYTHON is set in
+ the environment, in which case, bup will use that path. You can
+ see which Python executable was chosen by looking at the
+ configure output, or examining cmd/python-cmd.sh, and you can
+ change the selection by re-running ./configure.

From binary packages
--------------------
diff --git a/cmd/bloom-cmd.py b/cmd/bloom-cmd.py
index cd4506d..7e208cc 100755
--- a/cmd/bloom-cmd.py
+++ b/cmd/bloom-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, glob, tempfile
from bup import options, git, bloom
from bup.helpers import *
diff --git a/cmd/cat-file-cmd.py b/cmd/cat-file-cmd.py
index 8948ff2..9489429 100755
--- a/cmd/cat-file-cmd.py
+++ b/cmd/cat-file-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, stat
from bup import options, git, vfs
from bup.helpers import *
diff --git a/cmd/daemon-cmd.py b/cmd/daemon-cmd.py
index c90e3e4..c9f4493 100755
--- a/cmd/daemon-cmd.py
+++ b/cmd/daemon-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, getopt, socket, subprocess, fcntl
from bup import options, path
from bup.helpers import *
diff --git a/cmd/damage-cmd.py b/cmd/damage-cmd.py
index 49dbed0..8bf5bdc 100755
--- a/cmd/damage-cmd.py
+++ b/cmd/damage-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os, random
from bup import options
from bup.helpers import *
diff --git a/cmd/drecurse-cmd.py b/cmd/drecurse-cmd.py
index cdf6680..504a63c 100755
--- a/cmd/drecurse-cmd.py
+++ b/cmd/drecurse-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

from os.path import relpath
from bup import options, drecurse
diff --git a/cmd/fsck-cmd.py b/cmd/fsck-cmd.py
index 794c755..d985647 100755
--- a/cmd/fsck-cmd.py
+++ b/cmd/fsck-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os, glob, subprocess
from bup import options, git
from bup.helpers import *
diff --git a/cmd/ftp-cmd.py b/cmd/ftp-cmd.py
index baa1e08..fa6304b 100755
--- a/cmd/ftp-cmd.py
+++ b/cmd/ftp-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os, stat, fnmatch
from bup import options, git, shquote, vfs, ls
from bup.helpers import *
diff --git a/cmd/fuse-cmd.py b/cmd/fuse-cmd.py
index 30776cb..577c4e4 100755
--- a/cmd/fuse-cmd.py
+++ b/cmd/fuse-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os, errno
from bup import options, git, vfs, xstat
from bup.helpers import *
diff --git a/cmd/help-cmd.py b/cmd/help-cmd.py
index 56a4148..9dd9036 100755
--- a/cmd/help-cmd.py
+++ b/cmd/help-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os, glob
from bup import options, path

diff --git a/cmd/import-duplicity-cmd.py b/cmd/import-duplicity-cmd.py
index 163d4fa..fc815b9 100755
--- a/cmd/import-duplicity-cmd.py
+++ b/cmd/import-duplicity-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

from calendar import timegm
from pipes import quote
diff --git a/cmd/index-cmd.py b/cmd/index-cmd.py
index 6f2adf4..e1349cb 100755
--- a/cmd/index-cmd.py
+++ b/cmd/index-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import sys, stat, time, os, errno, re
from bup import metadata, options, git, index, drecurse, hlinkdb
diff --git a/cmd/init-cmd.py b/cmd/init-cmd.py
index 2e4a151..d3796b4 100755
--- a/cmd/init-cmd.py
+++ b/cmd/init-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys

from bup import git, options, client
diff --git a/cmd/join-cmd.py b/cmd/join-cmd.py
index 042b402..e7c83fd 100755
--- a/cmd/join-cmd.py
+++ b/cmd/join-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys
from bup import git, options, client
from bup.helpers import *
diff --git a/cmd/list-idx-cmd.py b/cmd/list-idx-cmd.py
index a3e3dc9..5d3c030 100755
--- a/cmd/list-idx-cmd.py
+++ b/cmd/list-idx-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os
from bup import git, options
from bup.helpers import *
diff --git a/cmd/ls-cmd.py b/cmd/ls-cmd.py
index 94a7131..0b02519 100755
--- a/cmd/ls-cmd.py
+++ b/cmd/ls-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys
from bup import git, vfs, ls
from bup.helpers import *
diff --git a/cmd/margin-cmd.py b/cmd/margin-cmd.py
index 9b7fd60..0f4380e 100755
--- a/cmd/margin-cmd.py
+++ b/cmd/margin-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, struct, math
from bup import options, git, _helpers
from bup.helpers import *
diff --git a/cmd/memtest-cmd.py b/cmd/memtest-cmd.py
index 0e3cf0c..38fd7a9 100755
--- a/cmd/memtest-cmd.py
+++ b/cmd/memtest-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, re, struct, time, resource
from bup import git, bloom, midx, options, _helpers
from bup.helpers import *
diff --git a/cmd/meta-cmd.py b/cmd/meta-cmd.py
index cac26b5..bf96b29 100755
--- a/cmd/meta-cmd.py
+++ b/cmd/meta-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

# Copyright (C) 2010 Rob Browning
#
diff --git a/cmd/midx-cmd.py b/cmd/midx-cmd.py
index 6876751..8f7604e 100755
--- a/cmd/midx-cmd.py
+++ b/cmd/midx-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, math, struct, glob, resource
import tempfile
from bup import options, git, midx, _helpers, xstat
diff --git a/cmd/mux-cmd.py b/cmd/mux-cmd.py
index b70c0a8..7a7d8e4 100755
--- a/cmd/mux-cmd.py
+++ b/cmd/mux-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import os, sys, subprocess, struct
from bup import options
from bup.helpers import *
diff --git a/cmd/newliner-cmd.py b/cmd/newliner-cmd.py
index 68d327b..f95decc 100755
--- a/cmd/newliner-cmd.py
+++ b/cmd/newliner-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os, re
from bup import options
from bup import _helpers # fixes up sys.argv on import
diff --git a/cmd/on--server-cmd.py b/cmd/on--server-cmd.py
index a6f3e21..fbdc86a 100755
--- a/cmd/on--server-cmd.py
+++ b/cmd/on--server-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os, struct
from bup import options, helpers

diff --git a/cmd/on-cmd.py b/cmd/on-cmd.py
index 9d6f683..e5b1d6b 100755
--- a/cmd/on-cmd.py
+++ b/cmd/on-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, os, struct, getopt, subprocess, signal
from subprocess import PIPE
from bup import options, ssh, path
diff --git a/cmd/random-cmd.py b/cmd/random-cmd.py
index 4be3660..9ff30ee 100755
--- a/cmd/random-cmd.py
+++ b/cmd/random-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys
from bup import options, _helpers
from bup.helpers import *
diff --git a/cmd/restore-cmd.py b/cmd/restore-cmd.py
index d527489..fa532c5 100755
--- a/cmd/restore-cmd.py
+++ b/cmd/restore-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import copy, errno, sys, stat, re
from bup import options, git, metadata, vfs
from bup.helpers import *
diff --git a/cmd/save-cmd.py b/cmd/save-cmd.py
index fc93408..f7d8c5d 100755
--- a/cmd/save-cmd.py
+++ b/cmd/save-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, stat, time, math
from cStringIO import StringIO
from errno import EACCES
diff --git a/cmd/server-cmd.py b/cmd/server-cmd.py
index 43a3fce..12f6a69 100755
--- a/cmd/server-cmd.py
+++ b/cmd/server-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import os, sys, struct
from bup import options, git
from bup.helpers import *
diff --git a/cmd/split-cmd.py b/cmd/split-cmd.py
index 70daf7c..c724694 100755
--- a/cmd/split-cmd.py
+++ b/cmd/split-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import os, sys, time
from bup import hashsplit, git, options, client
from bup.helpers import *
diff --git a/cmd/tag-cmd.py b/cmd/tag-cmd.py
index f33855e..e29bdfb 100755
--- a/cmd/tag-cmd.py
+++ b/cmd/tag-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import sys
import os
diff --git a/cmd/tick-cmd.py b/cmd/tick-cmd.py
index 4d462ad..f86455e 100755
--- a/cmd/tick-cmd.py
+++ b/cmd/tick-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, time
from bup import options

diff --git a/cmd/version-cmd.py b/cmd/version-cmd.py
index 645f25f..1f8c38c 100755
--- a/cmd/version-cmd.py
+++ b/cmd/version-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys
from bup import options
from bup import _version
diff --git a/cmd/web-cmd.py b/cmd/web-cmd.py
index 1c515e2..0a61f90 100755
--- a/cmd/web-cmd.py
+++ b/cmd/web-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
import sys, stat, urllib, mimetypes, posixpath, time, webbrowser
import urllib
from bup import options, git, vfs
diff --git a/cmd/xstat-cmd.py b/cmd/xstat-cmd.py
index 85071d2..472a797 100755
--- a/cmd/xstat-cmd.py
+++ b/cmd/xstat-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0"):$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
# Copyright (C) 2010 Rob Browning
#
# This code is covered under the terms of the GNU Library General
diff --git a/config/configure b/config/configure
index 8003241..a2afb83 100755
--- a/config/configure
+++ b/config/configure
@@ -56,8 +56,18 @@ if [ -z "$MAKE_VERSION" ]; then
fi
expr "$MAKE_VERSION" '>=' '3.81' || AC_FAIL "ERROR: $MAKE must be >= version 3.81"

-if test -z "$(bup_find_prog python '')"; then
+bup_python="$PYTHON"
+test -z "$bup_python" && bup_python="$(bup_find_prog python2.7 '')"
+test -z "$bup_python" && bup_python="$(bup_find_prog python2.6 '')"
+test -z "$bup_python" && bup_python="$(bup_find_prog python2 '')"
+test -z "$bup_python" && bup_python="$(bup_find_prog python '')"
+if test -z "$bup_python"; then

AC_FAIL "ERROR: unable to find python"

+else
+ echo "#!/bin/sh" > ../cmd/python-cmd.sh
+ printf "exec %q \"\$@\"" "$bup_python" >> ../cmd/python-cmd.sh
+ chmod u+x ../cmd/python-cmd.sh
+ cd ../cmd && ln -sf python-cmd.sh bup-python
fi

if test -z "$(bup_find_prog git '')"; then
diff --git a/main.py b/main.py
index 45a0e8a..c6d77ce 100755
--- a/main.py
+++ b/main.py
@@ -1,4 +1,10 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":
+export PATH="$(dirname "$0")/cmd:$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.
+
import sys, os, subprocess, signal, getopt

argv = sys.argv
--
2.1.4

Rob Browning

unread,

Jul 26, 2015, 12:18:15 PM7/26/15

to bup-...@googlegroups.com, Greg Troxel

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---

This version should be more complete, hopefully converting the
remaining #! files (in t/ etc.).

Makefile | 21 ++++++++++++++++-----

@@ -142,6 +150,9 @@ test: all

check: test

+cmd/python-cmd.sh: config/configure
+ ./config/configure
+
cmds: \
$(patsubst cmd/%-cmd.py,cmd/bup-%,$(wildcard cmd/*-cmd.py)) \
$(patsubst cmd/%-cmd.sh,cmd/bup-%,$(wildcard cmd/*-cmd.sh))

diff --git a/t/cleanup-mounts-under b/t/cleanup-mounts-under
index a8ba611..9a72ded 100755
--- a/t/cleanup-mounts-under
+++ b/t/cleanup-mounts-under
@@ -1,5 +1,8 @@
#!/usr/bin/env python

+# This cannot rely on bup-python because it runs during clean, and so
+# it also needs to be compatible with Python 2 and 3.
+
import os.path, re, subprocess, sys

def mntent_unescape(x):
diff --git a/t/hardlink-sets b/t/hardlink-sets
index 653b4d2..913ec6c 100755
--- a/t/hardlink-sets
+++ b/t/hardlink-sets

@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":

+export PATH="$(dirname "$0")/../cmd:$PATH"

+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import os, stat, sys

diff --git a/t/id-other-than b/t/id-other-than
index cfa219d..0190be0 100755
--- a/t/id-other-than
+++ b/t/id-other-than

@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":

+export PATH="$(dirname "$0")/../cmd:$PATH"

+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import grp
import pwd
diff --git a/t/mksock b/t/mksock
index ee2dbfe..70b5ce8 100755
--- a/t/mksock
+++ b/t/mksock

@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":

+export PATH="$(dirname "$0")/../cmd:$PATH"

+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import socket, sys

diff --git a/t/ns-timestamp-resolutions b/t/ns-timestamp-resolutions
index d1bb785..5512769 100755
--- a/t/ns-timestamp-resolutions
+++ b/t/ns-timestamp-resolutions

@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":

+export PATH="$(dirname "$0")/../cmd:$PATH"

+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import os, sys

diff --git a/t/root-status b/t/root-status
index 9fb12cb..40fb401 100755
--- a/t/root-status
+++ b/t/root-status

@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":

+export PATH="$(dirname "$0")/../cmd:$PATH"

+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import sys

diff --git a/t/subtree-hash b/t/subtree-hash
index 691c207..eae8acc 100755
--- a/t/subtree-hash
+++ b/t/subtree-hash

@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":

+export PATH="$(dirname "$0")/../cmd:$PATH"

+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import os, sys

diff --git a/t/unknown-owner b/t/unknown-owner
index 00442a4..5135544 100755
--- a/t/unknown-owner
+++ b/t/unknown-owner

@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":

+export PATH="$(dirname "$0")/../cmd:$PATH"

+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

import grp
import pwd
diff --git a/wvtest.py b/wvtest.py
index cc68601..d4c7d92 100755
--- a/wvtest.py
+++ b/wvtest.py

@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh
+"""":

+export PATH="$(dirname "$0")/cmd:$PATH"
+exec bup-python $0 ${1+"$@"}
+"""
+# End preamble.

#
# WvTest:
# Copyright (C)2007-2012 Versabanq Innovations Inc. and contributors.
--
2.1.4

Rob Browning

unread,

Jul 26, 2015, 12:32:56 PM7/26/15

to bup-...@googlegroups.com, Greg Troxel

Rob Browning <r...@defaultvalue.org> writes:

> +#!/bin/sh
> +"""":
> +export PATH="$(dirname "$0"):$PATH"
> +exec bup-python $0 ${1+"$@"}
> +"""
> +# End preamble.

...realized I'll want an Emacs mode line, so I'll include that in any
future versions, and I also wondered if additional safety might be
advisable, i.e.:

#!/bin/sh
"""": # -*-python-*-
PATH="$(dirname "$0"):$PATH" || exit $?
export PATH

exec bup-python $0 ${1+"$@"}
"""

# End preamble.

I'll include that too unless there's a good reason against it.

Thanks

Rob Browning

unread,

Jul 26, 2015, 12:40:06 PM7/26/15

to bup-...@googlegroups.com, Greg Troxel

Rob Browning <r...@defaultvalue.org> writes:

> #!/bin/sh
> """": # -*-python-*-
> PATH="$(dirname "$0"):$PATH" || exit $?
> export PATH
> exec bup-python $0 ${1+"$@"}
> """
> # End preamble.

And perhaps altering the PATH is wrong, instead:

#!/bin/sh
"""": # -*-python-*-
bup_python="$(dirname "$0")/cmd/bup-python" || exit $?
exec "$bup_python" $0 ${1+"$@"}
"""
# End preamble.

Otherwise the PATH grows with each sub-invocation, not to mention other
possible unintentional side-effects.

Rob Browning

unread,

Aug 27, 2015, 1:53:29 PM8/27/15

to bup-...@googlegroups.com, Greg Troxel

[r...@defaultvalue.org: add python modeline]
---

This version should incorporate the improvements I mentioned
(i.e. add a python mode line, don't change the PATH, and check
bup_python assignment).

I'm leaning toward including this soon, unless there are problems or
objections.

index cd4506d..3c23385 100755

--- a/cmd/bloom-cmd.py
+++ b/cmd/bloom-cmd.py
@@ -1,4 +1,9 @@
-#!/usr/bin/env python
+#!/bin/sh