[PATCH 00/14] Pushed git defaultBranch and pylint support; python-config ci fixes

[Rob Browning]

I've pushed these patches that fix an allocation bug, a number of
problems in CI, add the the first bits of Johannes' pylint support,
and make our tests insensitive to any git config defaultBranch setting
(or future upstream changes there).

Johannes Berg (1):
test: add pylint and test imports

Rob Browning (13):
test_prune_older: be independent of user's git defaultBranch
test-misc: be independent of user's git defaultBranch
test-save-restore: be independent of user's git defaultBranch
test_git: be independent of user's git defaultBranch
bup: remove redundant allocation doubling
dev/system-info: include df, mount, and git version
cirrus: adjust test names (root vs non-root, etc.)
prep-for-freebsd-build: switch to py38
cirrus: adjust for new python-config based builds
GNUmakefile: run "bup features" during test
configure: add --with-pylint=[yes|no|maybe] defaulting to maybe
cirrus: run pylint for python 2 and python 3
system-info: report freebsd cpu and mem info

.cirrus.yml | 39 +++++++++++++++------------
.pylintrc | 5 ++++
GNUmakefile | 14 ++++++----
config/configure | 17 ++++++++++++
configure | 7 ++---
dev/have-pylint | 17 ++++++++++++
dev/prep-for-debianish-build | 4 +--
dev/prep-for-freebsd-build | 2 +-
dev/system-info | 17 ++++++++++--
lib/bup/bloom.py | 2 +-
lib/bup/client.py | 4 +--
lib/bup/cmd/bloom.py | 4 +--
lib/bup/cmd/cat_file.py | 2 +-
lib/bup/cmd/daemon.py | 4 +--
lib/bup/cmd/damage.py | 2 +-
lib/bup/cmd/fsck.py | 2 +-
lib/bup/cmd/ftp.py | 4 +--
lib/bup/cmd/fuse.py | 2 +-
lib/bup/cmd/gc.py | 2 +-
lib/bup/cmd/get.py | 8 +++---
lib/bup/cmd/import_duplicity.py | 12 +++------
lib/bup/cmd/index.py | 4 +--
lib/bup/cmd/init.py | 2 +-
lib/bup/cmd/list_idx.py | 2 +-
lib/bup/cmd/ls.py | 4 +--
lib/bup/cmd/memtest.py | 1 -
lib/bup/cmd/meta.py | 4 +--
lib/bup/cmd/midx.py | 4 +--
lib/bup/cmd/mux.py | 2 +-
lib/bup/cmd/on__server.py | 2 +-
lib/bup/cmd/prune_older.py | 1 -
lib/bup/cmd/restore.py | 10 +++----
lib/bup/cmd/rm.py | 2 +-
lib/bup/cmd/server.py | 1 -
lib/bup/cmd/split.py | 2 +-
lib/bup/cmd/tag.py | 4 +--
lib/bup/cmd/tick.py | 2 +-
lib/bup/cmd/version.py | 2 +-
lib/bup/cmd/web.py | 5 ++--
lib/bup/cmd/xstat.py | 6 ++---
lib/bup/compat.py | 4 ++-
lib/bup/gc.py | 2 +-
lib/bup/git.py | 9 +++----
lib/bup/hashsplit.py | 4 +--
lib/bup/helpers.py | 4 +--
lib/bup/index.py | 2 +-
lib/bup/ls.py | 8 +++---
lib/bup/main.py | 6 ++---
lib/bup/metadata.py | 6 ++---
lib/bup/midx.py | 2 +-
lib/bup/pwdgrp.py | 2 +-
lib/bup/rm.py | 3 +--
lib/bup/ssh.py | 2 +-
lib/bup/tree.py | 2 --
lib/bup/version.py | 1 -
lib/bup/vfs.py | 8 +++---
lib/bup/vint.py | 1 -
lib/cmd/bup.c | 1 -
pylint | 47 +++++++++++++++++++++++++++++++++
pytest | 2 ++
test/ext/test-misc | 18 +++++++------
test/ext/test-save-restore | 8 +++---
test/ext/test_prune_older.py | 11 ++++----
test/int/test_bloom.py | 1 -
test/int/test_client.py | 3 +--
test/int/test_git.py | 9 ++++---
test/int/test_helpers.py | 5 ++--
test/int/test_index.py | 2 +-
test/int/test_metadata.py | 4 +--
test/int/test_resolve.py | 3 ---
test/int/test_vfs.py | 3 ---
test/int/test_xstat.py | 1 -
72 files changed, 247 insertions(+), 167 deletions(-)
create mode 100644 .pylintrc
create mode 100755 dev/have-pylint
create mode 100755 pylint

--
2.30.2

[Rob Browning]

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---
.cirrus.yml | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 4cb4a4b8..fd92f24c 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,6 +1,6 @@

task:
- name: debian (py2 check root)
+ name: debian (py2 check/lint root)
container:
image: debian:buster
cpu: 4
@@ -9,7 +9,8 @@ task:
set -xe
dev/prep-for-debianish-build python2
dev/system-info
- make -j6 BUP_PYTHON_CONFIG=python2.7-config check
+ BUP_PYTHON_CONFIG=python2.7-config ./configure --with-pylint=yes
+ make -j6 check

task:
name: debian (py2 long-check)
@@ -42,7 +43,7 @@ task:
"$(pwd)" | su -l bup

task:
- name: debian (py3 check root)
+ name: debian (py3 check/lint root)
container:
image: debian:buster
cpu: 4
@@ -51,7 +52,8 @@ task:
set -xe
dev/prep-for-debianish-build python3
dev/system-info
- make -j6 BUP_PYTHON_CONFIG=python3-config check
+ BUP_PYTHON_CONFIG=python3-config ./configure --with-pylint=yes
+ make -j6 check

task:
name: debian (py3 long-check)
--
2.30.2

[Rob Browning]

From: Johannes Berg <joha...@sipsolutions.net>

Add pylint and test for unnecessary imports as a first step, which
requires cleaning them all up.

[r...@defaultvalue.org: add and use ./pylint]

Signed-off-by: Johannes Berg <joha...@sipsolutions.net>
Reviewed-by: Rob Browning <r...@defaultvalue.org>
Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---
.pylintrc | 5 +++++
GNUmakefile | 5 ++++-
lib/bup/bloom.py | 2 +-
lib/bup/client.py | 4 ++--
lib/bup/cmd/bloom.py | 4 ++--
lib/bup/cmd/cat_file.py | 2 +-
lib/bup/cmd/daemon.py | 4 ++--

lib/bup/cmd/damage.py | 2 +-
lib/bup/cmd/fsck.py | 2 +-

lib/bup/cmd/ftp.py | 4 ++--

lib/bup/cmd/fuse.py | 2 +-
lib/bup/cmd/gc.py | 2 +-

lib/bup/cmd/get.py | 8 +++-----
lib/bup/cmd/import_duplicity.py | 12 ++++--------
lib/bup/cmd/index.py | 4 ++--

lib/bup/cmd/init.py | 2 +-
lib/bup/cmd/list_idx.py | 2 +-

lib/bup/cmd/ls.py | 4 ++--
lib/bup/cmd/memtest.py | 1 -
lib/bup/cmd/meta.py | 4 ++--
lib/bup/cmd/midx.py | 4 ++--

lib/bup/cmd/mux.py | 2 +-
lib/bup/cmd/on__server.py | 2 +-
lib/bup/cmd/prune_older.py | 1 -

lib/bup/cmd/restore.py | 10 +++++-----

lib/bup/cmd/rm.py | 2 +-
lib/bup/cmd/server.py | 1 -
lib/bup/cmd/split.py | 2 +-

lib/bup/cmd/tag.py | 4 ++--

lib/bup/cmd/tick.py | 2 +-
lib/bup/cmd/version.py | 2 +-

lib/bup/cmd/web.py | 5 ++---
lib/bup/cmd/xstat.py | 6 +++---
lib/bup/compat.py | 4 +++-
lib/bup/gc.py | 2 +-
lib/bup/git.py | 9 +++------
lib/bup/hashsplit.py | 4 ++--
lib/bup/helpers.py | 4 ++--
lib/bup/index.py | 2 +-
lib/bup/ls.py | 8 ++++----
lib/bup/main.py | 6 ++----
lib/bup/metadata.py | 6 ++----

lib/bup/midx.py | 2 +-
lib/bup/pwdgrp.py | 2 +-
lib/bup/rm.py | 3 +--
lib/bup/ssh.py | 2 +-
lib/bup/tree.py | 2 --
lib/bup/version.py | 1 -

lib/bup/vfs.py | 8 ++++----
lib/bup/vint.py | 1 -
pylint | 20 ++++++++++++++++++++
pytest | 2 ++

test/int/test_bloom.py | 1 -
test/int/test_client.py | 3 +--

test/int/test_git.py | 2 +-
test/int/test_helpers.py | 5 ++---
test/int/test_index.py | 2 +-
test/int/test_metadata.py | 4 +---

test/int/test_resolve.py | 3 ---
test/int/test_vfs.py | 3 ---
test/int/test_xstat.py | 1 -

61 files changed, 114 insertions(+), 116 deletions(-)
create mode 100644 .pylintrc
create mode 100755 pylint

diff --git a/.pylintrc b/.pylintrc
new file mode 100644
index 00000000..74e7c1f9
--- /dev/null
+++ b/.pylintrc
@@ -0,0 +1,5 @@
+# -*-conf-*-
+
+[MESSAGES CONTROL]
+disable=all
+enable=unused-import,unused-wildcard-import
diff --git a/GNUmakefile b/GNUmakefile
index 228b9794..f648dfb6 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -206,7 +206,10 @@ get_parallel_n = $(patsubst -j%,%,$(parallel_opt))
maybe_specific_n = $(if $(filter -j%,$(parallel_opt)),-n$(get_parallel_n))
xdist_opt = $(if $(filter -j,$(parallel_opt)),-nauto,$(maybe_specific_n))

-test: all test/tmp dev/python
+lint: dev/bup-python
+ ./pylint
+
+test: all test/tmp dev/python lint
if test yes = "$$(dev/python -c 'import xdist; print("yes")' 2>/dev/null)"; then \
(set -x; ./pytest $(xdist_opt);) \
else \
diff --git a/lib/bup/bloom.py b/lib/bup/bloom.py
index 18ed4608..f8ae3dd0 100644
--- a/lib/bup/bloom.py
+++ b/lib/bup/bloom.py
@@ -81,7 +81,7 @@ Brandon Low <lost...@lostlogicx.com> 2011-02-04
"""

from __future__ import absolute_import
-import sys, os, math, mmap, struct
+import os, math, struct

from bup import _helpers
from bup.helpers import (debug1, debug2, log, mmap_read, mmap_readwrite,
diff --git a/lib/bup/client.py b/lib/bup/client.py
index b37b71d2..9372d609 100644
--- a/lib/bup/client.py
+++ b/lib/bup/client.py
@@ -3,7 +3,7 @@ from __future__ import print_function

from __future__ import absolute_import
from binascii import hexlify, unhexlify
-import errno, os, re, struct, sys, time, zlib
+import errno, os, re, struct, time, zlib
import socket

from bup import git, ssh, vfs
@@ -12,7 +12,7 @@ from bup.helpers import (Conn, atomically_replaced_file, chunkyreader, debug1,
debug2, linereader, lines_until_sentinel,
mkdirp, progress, qprogress, DemuxConn)
from bup.io import path_msg
-from bup.vint import read_bvec, read_vuint, write_bvec
+from bup.vint import write_bvec


bwlimit = None
diff --git a/lib/bup/cmd/bloom.py b/lib/bup/cmd/bloom.py
index 996bea2b..27d26b14 100755
--- a/lib/bup/cmd/bloom.py
+++ b/lib/bup/cmd/bloom.py
@@ -3,9 +3,9 @@ from __future__ import absolute_import

import os, glob, sys

-from bup import compat, options, git, bloom
+from bup import options, git, bloom
from bup.compat import argv_bytes, hexstr
-from bup.helpers import (add_error, debug1, handle_ctrl_c, log, progress, qprogress,
+from bup.helpers import (add_error, debug1, log, progress, qprogress,
saved_errors)
from bup.io import path_msg

diff --git a/lib/bup/cmd/cat_file.py b/lib/bup/cmd/cat_file.py
index e45d1ea5..302d8e3d 100755
--- a/lib/bup/cmd/cat_file.py
+++ b/lib/bup/cmd/cat_file.py
@@ -5,7 +5,7 @@ import re, stat, sys

from bup import options, git, vfs
from bup.compat import argv_bytes
-from bup.helpers import chunkyreader, handle_ctrl_c, log, saved_errors
+from bup.helpers import chunkyreader, log, saved_errors
from bup.io import byte_stream
from bup.repo import LocalRepo

diff --git a/lib/bup/cmd/daemon.py b/lib/bup/cmd/daemon.py
index d82e1e01..dca17b40 100755
--- a/lib/bup/cmd/daemon.py
+++ b/lib/bup/cmd/daemon.py
@@ -1,9 +1,9 @@

from __future__ import absolute_import
-import fcntl, getopt, os, socket, subprocess, sys
+import fcntl, getopt, os, socket, subprocess, sys, select

from bup import options, path
-from bup.helpers import *
+from bup.helpers import log, debug1


optspec = """
diff --git a/lib/bup/cmd/damage.py b/lib/bup/cmd/damage.py
index d3a17782..1bd2aa19 100755
--- a/lib/bup/cmd/damage.py
+++ b/lib/bup/cmd/damage.py
@@ -1,6 +1,6 @@

from __future__ import absolute_import
-import os, random, sys
+import os, random

from bup import options
from bup.compat import argv_bytes, bytes_from_uint, range
diff --git a/lib/bup/cmd/fsck.py b/lib/bup/cmd/fsck.py
index c195f77e..0725f539 100755
--- a/lib/bup/cmd/fsck.py
+++ b/lib/bup/cmd/fsck.py
@@ -1,7 +1,7 @@

from __future__ import absolute_import, print_function
from shutil import rmtree
-from subprocess import PIPE, Popen
+from subprocess import PIPE
from tempfile import mkdtemp
from binascii import hexlify
import glob, os, subprocess, sys
diff --git a/lib/bup/cmd/ftp.py b/lib/bup/cmd/ftp.py
index 4982384c..8b7fe9d8 100755
--- a/lib/bup/cmd/ftp.py
+++ b/lib/bup/cmd/ftp.py
@@ -8,8 +8,8 @@ from __future__ import absolute_import, print_function
import os, fnmatch, stat, sys

from bup import _helpers, options, git, shquote, ls, vfs
-from bup.compat import argv_bytes, fsdecode
-from bup.helpers import chunkyreader, handle_ctrl_c, log
+from bup.compat import argv_bytes
+from bup.helpers import chunkyreader, log
from bup.io import byte_stream, path_msg
from bup.repo import LocalRepo

diff --git a/lib/bup/cmd/fuse.py b/lib/bup/cmd/fuse.py
index 96bf6179..63071197 100755
--- a/lib/bup/cmd/fuse.py
+++ b/lib/bup/cmd/fuse.py
@@ -27,7 +27,7 @@ if sys.version_info[0] > 2:
sys.exit(2)

from bup import options, git, vfs, xstat
-from bup.compat import argv_bytes, fsdecode, py_maj
+from bup.compat import argv_bytes, fsdecode
from bup.helpers import log
from bup.repo import LocalRepo

diff --git a/lib/bup/cmd/gc.py b/lib/bup/cmd/gc.py
index e1b995a3..a3733faa 100755
--- a/lib/bup/cmd/gc.py
+++ b/lib/bup/cmd/gc.py
@@ -3,7 +3,7 @@ from __future__ import absolute_import

from bup import git, options
from bup.gc import bup_gc
-from bup.helpers import die_if_errors, handle_ctrl_c, log
+from bup.helpers import die_if_errors


optspec = """
diff --git a/lib/bup/cmd/get.py b/lib/bup/cmd/get.py
index 8844544c..6c9b8ed2 100755
--- a/lib/bup/cmd/get.py
+++ b/lib/bup/cmd/get.py
@@ -2,22 +2,20 @@
from __future__ import absolute_import, print_function
from binascii import hexlify, unhexlify
from collections import namedtuple
-from functools import partial
from stat import S_ISDIR
import os, sys, textwrap, time

-from bup import compat, git, client, helpers, vfs
+from bup import compat, git, client, vfs
from bup.compat import (
argv_bytes,
bytes_from_byte,
environ,
hexstr,
items,
- wrap_main
)
from bup.git import get_cat_data, parse_commit, walk_object
-from bup.helpers import add_error, debug1, handle_ctrl_c, log, saved_errors
-from bup.helpers import hostname, shstr, tty_width
+from bup.helpers import add_error, debug1, log, saved_errors
+from bup.helpers import hostname, tty_width
from bup.io import path_msg
from bup.pwdgrp import userfullname, username
from bup.repo import LocalRepo, RemoteRepo
diff --git a/lib/bup/cmd/import_duplicity.py b/lib/bup/cmd/import_duplicity.py
index b2b011af..78ed1ee2 100755
--- a/lib/bup/cmd/import_duplicity.py
+++ b/lib/bup/cmd/import_duplicity.py
@@ -1,19 +1,15 @@

from __future__ import absolute_import
from calendar import timegm
-from pipes import quote
from subprocess import check_call
-from time import strftime, strptime
+from time import strptime
import os, sys, tempfile

from bup import git, helpers, options
-from bup.compat import argv_bytes, str_type
-from bup.helpers import (handle_ctrl_c,
- log,
- readpipe,
+from bup.compat import argv_bytes
+from bup.helpers import (log,
shstr,
- saved_errors,
- unlink)
+ saved_errors)
import bup.path


diff --git a/lib/bup/cmd/index.py b/lib/bup/cmd/index.py
index 86335857..35d393ad 100755
--- a/lib/bup/cmd/index.py
+++ b/lib/bup/cmd/index.py
@@ -3,10 +3,10 @@ from __future__ import absolute_import, print_function
from binascii import hexlify
import errno, os, stat, sys, time

-from bup import compat, metadata, options, git, index, drecurse, hlinkdb
+from bup import metadata, options, git, index, hlinkdb
from bup.compat import argv_bytes
from bup.drecurse import recursive_dirlist
-from bup.hashsplit import GIT_MODE_TREE, GIT_MODE_FILE
+from bup.hashsplit import GIT_MODE_FILE
from bup.helpers import (add_error, handle_ctrl_c, log, parse_excludes, parse_rx_excludes,
progress, qprogress, saved_errors)
from bup.io import byte_stream, path_msg
diff --git a/lib/bup/cmd/init.py b/lib/bup/cmd/init.py
index c354584a..4be83a28 100755
--- a/lib/bup/cmd/init.py
+++ b/lib/bup/cmd/init.py
@@ -3,7 +3,7 @@ from __future__ import absolute_import
import sys

from bup import git, options, client
-from bup.helpers import log, saved_errors
+from bup.helpers import log
from bup.compat import argv_bytes


diff --git a/lib/bup/cmd/list_idx.py b/lib/bup/cmd/list_idx.py
index 8bf940ef..15e8b4c8 100755
--- a/lib/bup/cmd/list_idx.py
+++ b/lib/bup/cmd/list_idx.py
@@ -1,7 +1,7 @@

from __future__ import absolute_import, print_function
from binascii import hexlify, unhexlify
-import os, sys
+import sys

from bup import git, options
from bup.compat import argv_bytes
diff --git a/lib/bup/cmd/ls.py b/lib/bup/cmd/ls.py
index a7fc6dbe..29323ccf 100755
--- a/lib/bup/cmd/ls.py
+++ b/lib/bup/cmd/ls.py
@@ -1,8 +1,8 @@

from __future__ import absolute_import, print_function
-import os.path, sys
+import sys

-from bup import compat, git, ls
+from bup import git, ls
from bup.io import byte_stream

def main(argv):
diff --git a/lib/bup/cmd/memtest.py b/lib/bup/cmd/memtest.py
index f62c64f1..831ec54b 100755
--- a/lib/bup/cmd/memtest.py
+++ b/lib/bup/cmd/memtest.py
@@ -4,7 +4,6 @@ import re, resource, sys, time

from bup import git, bloom, midx, options, _helpers
from bup.compat import range
-from bup.helpers import handle_ctrl_c
from bup.io import byte_stream


diff --git a/lib/bup/cmd/meta.py b/lib/bup/cmd/meta.py
index eab91014..b0d477d8 100755
--- a/lib/bup/cmd/meta.py
+++ b/lib/bup/cmd/meta.py
@@ -8,11 +8,11 @@
from __future__ import absolute_import
import sys

-from bup import compat, metadata
+from bup import metadata
from bup import options
from bup.compat import argv_bytes
from bup.io import byte_stream
-from bup.helpers import handle_ctrl_c, log, saved_errors
+from bup.helpers import log, saved_errors


def open_input(name):
diff --git a/lib/bup/cmd/midx.py b/lib/bup/cmd/midx.py
index e80afa1a..0d429583 100755
--- a/lib/bup/cmd/midx.py
+++ b/lib/bup/cmd/midx.py
@@ -1,12 +1,12 @@

from __future__ import absolute_import, print_function
from binascii import hexlify
-import glob, os, math, resource, struct, sys, tempfile
+import glob, os, math, resource, struct, sys

from bup import options, git, midx, _helpers, xstat
from bup.compat import argv_bytes, hexstr, range
from bup.helpers import (Sha1, add_error, atomically_replaced_file, debug1, fdatasync,
- handle_ctrl_c, log, mmap_readwrite, qprogress,
+ log, mmap_readwrite, qprogress,
saved_errors, unlink)
from bup.io import byte_stream, path_msg

diff --git a/lib/bup/cmd/mux.py b/lib/bup/cmd/mux.py
index 606f027b..a804f72e 100755
--- a/lib/bup/cmd/mux.py
+++ b/lib/bup/cmd/mux.py
@@ -1,6 +1,6 @@

from __future__ import absolute_import
-import os, struct, subprocess, sys
+import os, subprocess, sys

from bup import options
from bup.helpers import debug1, debug2, mux
diff --git a/lib/bup/cmd/on__server.py b/lib/bup/cmd/on__server.py
index c04b3b0c..13e2f847 100755
--- a/lib/bup/cmd/on__server.py
+++ b/lib/bup/cmd/on__server.py
@@ -3,7 +3,7 @@ from __future__ import absolute_import
import os, struct, sys

from bup import options, helpers, path
-from bup.compat import environ, py_maj
+from bup.compat import environ
from bup.io import byte_stream

optspec = """
diff --git a/lib/bup/cmd/prune_older.py b/lib/bup/cmd/prune_older.py
index b3df6e9f..e0f8545f 100755
--- a/lib/bup/cmd/prune_older.py
+++ b/lib/bup/cmd/prune_older.py
@@ -3,7 +3,6 @@ from __future__ import absolute_import, print_function
from binascii import hexlify, unhexlify
from collections import defaultdict
from itertools import groupby
-from sys import stderr
from time import localtime, strftime, time
import sys

diff --git a/lib/bup/cmd/restore.py b/lib/bup/cmd/restore.py
index 07643554..825493ff 100755
--- a/lib/bup/cmd/restore.py
+++ b/lib/bup/cmd/restore.py
@@ -3,12 +3,12 @@ from __future__ import absolute_import
from stat import S_ISDIR
import copy, errno, os, re, stat, sys

-from bup import options, git, metadata, vfs
+from bup import options, git, vfs
from bup._helpers import write_sparsely
-from bup.compat import argv_bytes, fsencode, wrap_main
-from bup.helpers import (add_error, chunkyreader, die_if_errors, handle_ctrl_c,
- log, mkdirp, parse_rx_excludes, progress, qprogress,
- saved_errors, should_rx_exclude_path, unlink)
+from bup.compat import argv_bytes, fsencode
+from bup.helpers import (add_error, chunkyreader, die_if_errors,
+ mkdirp, parse_rx_excludes, progress, qprogress,
+ should_rx_exclude_path)
from bup.io import byte_stream
from bup.repo import LocalRepo, RemoteRepo

diff --git a/lib/bup/cmd/rm.py b/lib/bup/cmd/rm.py
index 75d35a2a..2867657f 100755
--- a/lib/bup/cmd/rm.py
+++ b/lib/bup/cmd/rm.py
@@ -4,7 +4,7 @@ from __future__ import absolute_import
from bup.compat import argv_bytes
from bup.git import check_repo_or_die
from bup.options import Options
-from bup.helpers import die_if_errors, handle_ctrl_c, log
+from bup.helpers import die_if_errors
from bup.repo import LocalRepo
from bup.rm import bup_rm

diff --git a/lib/bup/cmd/server.py b/lib/bup/cmd/server.py
index 3ff8a191..7cb5a2a1 100755
--- a/lib/bup/cmd/server.py
+++ b/lib/bup/cmd/server.py
@@ -5,7 +5,6 @@ import os, struct, subprocess, sys

from bup import options, git, vfs, vint
from bup.compat import environ, hexstr
-from bup.git import MissingObject
from bup.helpers import (Conn, debug1, debug2, linereader, lines_until_sentinel,
log)
from bup.io import byte_stream, path_msg
diff --git a/lib/bup/cmd/split.py b/lib/bup/cmd/split.py
index 87ad8875..11392e20 100755
--- a/lib/bup/cmd/split.py
+++ b/lib/bup/cmd/split.py
@@ -5,7 +5,7 @@ import sys, time

from bup import compat, hashsplit, git, options, client
from bup.compat import argv_bytes, environ
-from bup.helpers import (add_error, handle_ctrl_c, hostname, log, parse_num,
+from bup.helpers import (add_error, hostname, log, parse_num,
qprogress, reprogress, saved_errors,
valid_save_name,
parse_date_or_fatal)
diff --git a/lib/bup/cmd/tag.py b/lib/bup/cmd/tag.py
index aaff7550..0f5475d5 100755
--- a/lib/bup/cmd/tag.py
+++ b/lib/bup/cmd/tag.py
@@ -1,11 +1,11 @@

from __future__ import absolute_import
from binascii import hexlify
-import os, sys
+import sys

from bup import git, options
from bup.compat import argv_bytes
-from bup.helpers import debug1, handle_ctrl_c, log
+from bup.helpers import debug1, log
from bup.io import byte_stream, path_msg


diff --git a/lib/bup/cmd/tick.py b/lib/bup/cmd/tick.py
index e2dfa395..82de69ba 100755
--- a/lib/bup/cmd/tick.py
+++ b/lib/bup/cmd/tick.py
@@ -1,6 +1,6 @@

from __future__ import absolute_import
-import os, sys, time
+import time

from bup import options

diff --git a/lib/bup/cmd/version.py b/lib/bup/cmd/version.py
index 853fe4fe..10684613 100755
--- a/lib/bup/cmd/version.py
+++ b/lib/bup/cmd/version.py
@@ -2,7 +2,7 @@
from __future__ import absolute_import, print_function
import re, sys

-from bup import compat, options, version
+from bup import options, version
from bup.io import byte_stream

version_rx = re.compile(r'^[0-9]+\.[0-9]+(\.[0-9]+)?(-[0-9]+-g[0-9abcdef]+)?$')
diff --git a/lib/bup/cmd/web.py b/lib/bup/cmd/web.py
index cae0d46e..ccac6faa 100755
--- a/lib/bup/cmd/web.py
+++ b/lib/bup/cmd/web.py
@@ -1,14 +1,13 @@

from __future__ import absolute_import, print_function
from collections import namedtuple
-import mimetypes, os, posixpath, signal, stat, sys, time, urllib, webbrowser
+import mimetypes, os, posixpath, signal, stat, sys, time, webbrowser
from binascii import hexlify


from bup import options, git, vfs
-from bup.helpers import (chunkyreader, debug1, format_filesize, handle_ctrl_c,
+from bup.helpers import (chunkyreader, debug1, format_filesize,
log, saved_errors)
-from bup.metadata import Metadata
from bup.path import resource_path
from bup.repo import LocalRepo
from bup.io import path_msg
diff --git a/lib/bup/cmd/xstat.py b/lib/bup/cmd/xstat.py
index 0cf8bdc6..246af24f 100755
--- a/lib/bup/cmd/xstat.py
+++ b/lib/bup/cmd/xstat.py
@@ -5,11 +5,11 @@

from __future__ import absolute_import, print_function

-import errno, stat, sys
+import errno, sys

-from bup import compat, metadata, options, xstat
+from bup import metadata, options
from bup.compat import argv_bytes
-from bup.helpers import add_error, handle_ctrl_c, parse_timestamp, saved_errors, \
+from bup.helpers import add_error, parse_timestamp, saved_errors, \
add_error, log
from bup.io import byte_stream

diff --git a/lib/bup/compat.py b/lib/bup/compat.py
index a06ffe8e..fb45ead5 100644
--- a/lib/bup/compat.py
+++ b/lib/bup/compat.py
@@ -1,6 +1,5 @@

from __future__ import absolute_import, print_function
-from array import array
from binascii import hexlify
from traceback import print_exception
import os, sys
@@ -13,6 +12,7 @@ py3 = py_maj >= 3

if py3:

+ # pylint: disable=unused-import
from os import environb as environ
from os import fsdecode, fsencode
from shlex import quote
@@ -88,8 +88,10 @@ else: # Python 2
return x

from pipes import quote
+ # pylint: disable=unused-import
from os import environ, getcwd

+ # pylint: disable=unused-import
from bup.py2raise import reraise

input = raw_input
diff --git a/lib/bup/gc.py b/lib/bup/gc.py
index 4009e815..b757034c 100644
--- a/lib/bup/gc.py
+++ b/lib/bup/gc.py
@@ -5,7 +5,7 @@ from os.path import basename
import glob, os, subprocess, sys, tempfile

from bup import bloom, git, midx
-from bup.compat import hexstr, pending_raise, range
+from bup.compat import hexstr, pending_raise
from bup.git import MissingObject, walk_object
from bup.helpers import Nonlocal, log, progress, qprogress
from bup.io import path_msg
diff --git a/lib/bup/git.py b/lib/bup/git.py
index ec84dd7b..cccc44ef 100644
--- a/lib/bup/git.py
+++ b/lib/bup/git.py
@@ -4,14 +4,13 @@ interact with the Git data structures.
"""

from __future__ import absolute_import, print_function
-import errno, os, sys, zlib, time, subprocess, struct, stat, re, tempfile, glob
+import os, sys, zlib, subprocess, struct, stat, re, tempfile, glob
from array import array
from binascii import hexlify, unhexlify
from collections import namedtuple
from itertools import islice
-from numbers import Integral

-from bup import _helpers, compat, hashsplit, path, midx, bloom, xstat
+from bup import _helpers, hashsplit, path, midx, bloom, xstat
from bup.compat import (buffer,
byte_int, bytes_from_byte, bytes_from_uint,
environ,
@@ -22,15 +21,13 @@ from bup.io import path_msg
from bup.helpers import (Sha1, add_error, chunkyreader, debug1, debug2,
exo,
fdatasync,
- hostname, localtime, log,
+ log,
merge_dict,
merge_iter,
mmap_read, mmap_readwrite,
- parse_num,
progress, qprogress, stat_if_exists,
unlink,
utc_offset_str)
-from bup.pwdgrp import username, userfullname


verbose = 0
diff --git a/lib/bup/hashsplit.py b/lib/bup/hashsplit.py
index 94d93ed3..16ccbb15 100644
--- a/lib/bup/hashsplit.py
+++ b/lib/bup/hashsplit.py
@@ -2,9 +2,9 @@
from __future__ import absolute_import
import io, math, os

-from bup import _helpers, compat, helpers
+from bup import _helpers, helpers
from bup._helpers import cat_bytes
-from bup.compat import buffer, py_maj
+from bup.compat import buffer
from bup.helpers import sc_page_size


diff --git a/lib/bup/helpers.py b/lib/bup/helpers.py
index a0679452..b0cff727 100644
--- a/lib/bup/helpers.py
+++ b/lib/bup/helpers.py
@@ -7,8 +7,8 @@ from ctypes import sizeof, c_void_p
from math import floor
from os import environ
from subprocess import PIPE, Popen
-import sys, os, pwd, subprocess, errno, socket, select, mmap, stat, re, struct
-import hashlib, heapq, math, operator, time, grp, tempfile
+import sys, os, subprocess, errno, select, mmap, stat, re, struct
+import hashlib, heapq, math, operator, time, tempfile

from bup import _helpers
from bup import compat
diff --git a/lib/bup/index.py b/lib/bup/index.py
index 6b773cea..a2abfa7a 100644
--- a/lib/bup/index.py
+++ b/lib/bup/index.py
@@ -2,7 +2,7 @@
from __future__ import absolute_import, print_function
import errno, os, stat, struct, tempfile

-from bup import compat, metadata, xstat
+from bup import metadata, xstat
from bup._helpers import UINT_MAX, bytescmp
from bup.compat import range
from bup.helpers import (add_error, log, merge_iter, mmap_readwrite,
diff --git a/lib/bup/ls.py b/lib/bup/ls.py
index 4bd67302..755c9981 100644
--- a/lib/bup/ls.py
+++ b/lib/bup/ls.py
@@ -3,16 +3,16 @@
from __future__ import absolute_import
from binascii import hexlify
from itertools import chain
-from stat import S_ISDIR, S_ISLNK
-import copy, locale, os.path, stat, sys
+from stat import S_ISDIR
+import os.path
import posixpath

-from bup import metadata, options, vfs, xstat
+from bup import metadata, vfs, xstat
from bup.compat import argv_bytes
from bup.io import path_msg
from bup.options import Options
from bup.repo import LocalRepo, RemoteRepo
-from bup.helpers import columnate, istty1, last, log
+from bup.helpers import columnate, istty1, log

def item_hash(item, tree_for_commit):
"""If the item is a Commit, return its commit oid, otherwise return
diff --git a/lib/bup/main.py b/lib/bup/main.py
index e24bee9b..0234a1b6 100755
--- a/lib/bup/main.py
+++ b/lib/bup/main.py
@@ -14,23 +14,21 @@ from importlib import import_module
from pkgutil import iter_modules
from subprocess import PIPE
from threading import Thread
-import errno, re, select, signal, subprocess
+import re, select, signal, subprocess

from bup import compat, path, helpers
from bup.compat import (
ModuleNotFoundError,
add_ex_ctx,
add_ex_tb,
- argv_bytes,
environ,
fsdecode,
int_types,
wrap_main
)
-from bup.compat import add_ex_tb, add_ex_ctx, argv_bytes, wrap_main
+from bup.compat import add_ex_tb, add_ex_ctx, wrap_main
from bup.helpers import (
columnate,
- debug1,
handle_ctrl_c,
log,
merge_dict,
diff --git a/lib/bup/metadata.py b/lib/bup/metadata.py
index b68c9851..a20b67b4 100644
--- a/lib/bup/metadata.py
+++ b/lib/bup/metadata.py
@@ -6,15 +6,13 @@
# Public License as described in the bup LICENSE file.

from __future__ import absolute_import, print_function
-from binascii import hexlify
from copy import deepcopy
from errno import EACCES, EINVAL, ENOTTY, ENOSYS, EOPNOTSUPP
from io import BytesIO
from time import gmtime, strftime
-import errno, os, sys, stat, time, pwd, grp, socket, struct
+import errno, os, sys, stat, time, socket, struct

-from bup import compat, vint, xstat
-from bup.compat import py_maj
+from bup import vint, xstat
from bup.drecurse import recursive_dirlist
from bup.helpers import add_error, mkdirp, log, is_superuser, format_filesize
from bup.io import path_msg
diff --git a/lib/bup/midx.py b/lib/bup/midx.py
index 5edf88ec..a18981da 100644
--- a/lib/bup/midx.py
+++ b/lib/bup/midx.py
@@ -1,6 +1,6 @@

from __future__ import absolute_import, print_function
-import glob, mmap, os, struct
+import glob, os, struct

from bup import _helpers
from bup.compat import range
diff --git a/lib/bup/pwdgrp.py b/lib/bup/pwdgrp.py
index a5b6d8b0..a41c4429 100644
--- a/lib/bup/pwdgrp.py
+++ b/lib/bup/pwdgrp.py
@@ -1,6 +1,6 @@

from __future__ import absolute_import, print_function
-import os, pwd, grp
+import os

from bup import _helpers
from bup.helpers import cache_key_value
diff --git a/lib/bup/rm.py b/lib/bup/rm.py
index b1f78898..844bdf15 100644
--- a/lib/bup/rm.py
+++ b/lib/bup/rm.py
@@ -1,11 +1,10 @@

from __future__ import absolute_import
from binascii import hexlify, unhexlify
-import sys

from bup import compat, git, vfs
from bup.client import ClientError
-from bup.compat import add_ex_ctx, add_ex_tb, hexstr, pending_raise
+from bup.compat import hexstr, pending_raise
from bup.git import get_commit_items
from bup.helpers import add_error, die_if_errors, log, saved_errors
from bup.io import path_msg
diff --git a/lib/bup/ssh.py b/lib/bup/ssh.py
index 2d539e8a..b6352293 100644
--- a/lib/bup/ssh.py
+++ b/lib/bup/ssh.py
@@ -5,7 +5,7 @@ Connect to a remote host via SSH and execute a command on the host.
from __future__ import absolute_import, print_function
import sys, os, re, subprocess

-from bup import helpers, path
+from bup import path
from bup.compat import environ

def connect(rhost, port, subcmd, stderr=None):
diff --git a/lib/bup/tree.py b/lib/bup/tree.py
index 277a839c..df1ccaf9 100644
--- a/lib/bup/tree.py
+++ b/lib/bup/tree.py
@@ -1,8 +1,6 @@

from __future__ import absolute_import, print_function

-from bup.hashsplit import GIT_MODE_TREE, GIT_MODE_FILE, GIT_MODE_SYMLINK
-

class TreeItem:
__slots__ = 'name', 'mode', 'gitmode', 'oid', 'meta'
diff --git a/lib/bup/version.py b/lib/bup/version.py
index 967702ee..b19f905c 100644
--- a/lib/bup/version.py
+++ b/lib/bup/version.py
@@ -1,6 +1,5 @@

from __future__ import absolute_import, print_function
-import sys

from bup.compat import ModuleNotFoundError

diff --git a/lib/bup/vfs.py b/lib/bup/vfs.py
index 9c8b9b4b..015209d7 100644
--- a/lib/bup/vfs.py
+++ b/lib/bup/vfs.py
@@ -49,16 +49,16 @@ item.coid.
from __future__ import absolute_import, print_function
from binascii import hexlify, unhexlify
from collections import namedtuple
-from errno import EINVAL, ELOOP, ENOENT, ENOTDIR
-from itertools import chain, dropwhile, groupby, tee
+from errno import EINVAL, ELOOP, ENOTDIR
+from itertools import chain, groupby, tee
from random import randrange
from stat import S_IFDIR, S_IFLNK, S_IFREG, S_ISDIR, S_ISLNK, S_ISREG
from time import localtime, strftime
import re, sys

-from bup import git, metadata, vint
+from bup import git, vint
from bup.compat import hexstr, range
-from bup.git import BUP_CHUNKED, cp, get_commit_items, parse_commit, tree_decode
+from bup.git import BUP_CHUNKED, parse_commit, tree_decode
from bup.helpers import debug2, last
from bup.io import path_msg
from bup.metadata import Metadata
diff --git a/lib/bup/vint.py b/lib/bup/vint.py
index fd8a8f0b..8179f53b 100644
--- a/lib/bup/vint.py
+++ b/lib/bup/vint.py
@@ -9,7 +9,6 @@

from __future__ import absolute_import
from io import BytesIO
-import sys

from bup import compat
from bup import _helpers
diff --git a/pylint b/pylint
new file mode 100755
index 00000000..c04af785
--- /dev/null
+++ b/pylint
@@ -0,0 +1,20 @@
+#!/bin/sh
+
+# Changes here might also be appropriate for ./pytest
+
+set -eu
+
+script_home="$(cd "$(dirname "$0")" && pwd -P)"
+testlibdir="$script_home/test/lib"
+
+export PYTHONPATH="$testlibdir${PYTHONPATH:+:$PYTHONPATH}"
+
+if test "$#" -eq 0; then
+ set -x
+ dev/bup-python -m pylint lib
+ # unused-wildcard-import: we always "import * from wvpytest"
+ dev/bup-python -m pylint -d unused-wildcard-import test/lib test/int
+else
+ set -x
+ exec dev/bup-python -m pylint "$@"
+fi
diff --git a/pytest b/pytest
index f35f626b..d64d766a 100755
--- a/pytest
+++ b/pytest
@@ -1,5 +1,7 @@
#!/bin/sh

+# Changes here might also be appropriate for ./pylint
+
set -eu

script_home="$(cd "$(dirname "$0")" && pwd -P)"
diff --git a/test/int/test_bloom.py b/test/int/test_bloom.py
index 69a5e460..5f75d7e6 100644
--- a/test/int/test_bloom.py
+++ b/test/int/test_bloom.py
@@ -5,7 +5,6 @@ import errno, platform, tempfile
import logging

from bup import bloom
-from bup.helpers import mkdirp

def test_bloom(tmpdir):
hashes = [os.urandom(20) for i in range(100)]
diff --git a/test/int/test_client.py b/test/int/test_client.py
index f5199ec6..b3cdba4b 100644
--- a/test/int/test_client.py
+++ b/test/int/test_client.py
@@ -1,11 +1,10 @@

from __future__ import absolute_import
-import sys, os, stat, time, random, subprocess, glob
+import os, time, random, subprocess, glob
import pytest

from bup import client, git, path
from bup.compat import bytes_from_uint, environ, range
-from bup.helpers import mkdirp

def randbytes(sz):
s = b''
diff --git a/test/int/test_git.py b/test/int/test_git.py
index 95e8671d..7965c60c 100644
--- a/test/int/test_git.py
+++ b/test/int/test_git.py
@@ -4,7 +4,7 @@ import sys
from binascii import hexlify, unhexlify
from subprocess import check_call
from functools import partial
-import struct, os, time
+import struct, os
import pytest

from wvpytest import *
diff --git a/test/int/test_helpers.py b/test/int/test_helpers.py
index 8bfea77f..82469a90 100644
--- a/test/int/test_helpers.py
+++ b/test/int/test_helpers.py
@@ -1,18 +1,17 @@

from __future__ import absolute_import
from time import tzset
-import math, os, os.path, re, subprocess
+import os, os.path, re
from bup import helpers

from wvpytest import *

from bup.compat import bytes_from_byte, bytes_from_uint, environ
from bup.helpers import (atomically_replaced_file, batchpipe, detect_fakeroot,
- grafted_path_components, mkdirp, parse_num,
+ grafted_path_components, parse_num,
path_components, readpipe, stripped_path_components,
shstr,
utc_offset_str)
-import bup._helpers as _helpers


def test_parse_num():
diff --git a/test/int/test_index.py b/test/int/test_index.py
index 1ab4828d..d00260ea 100644
--- a/test/int/test_index.py
+++ b/test/int/test_index.py
@@ -6,7 +6,7 @@ from wvpytest import *

from bup import index, metadata
from bup.compat import fsencode
-from bup.helpers import mkdirp, resolve_parent
+from bup.helpers import resolve_parent
import bup.xstat as xstat


diff --git a/test/int/test_metadata.py b/test/int/test_metadata.py
index 62d15c9d..fd3a00c5 100644
--- a/test/int/test_metadata.py
+++ b/test/int/test_metadata.py
@@ -1,6 +1,6 @@

from __future__ import absolute_import, print_function
-import errno, glob, grp, pwd, stat, tempfile, subprocess
+import errno, glob, stat, subprocess
import os, sys
import pytest

@@ -263,8 +263,6 @@ def test_restore_over_existing_target(tmpdir):
WVEXCEPT(Exception, dir_m.create_path, path, create_symlinks=True)


-from bup.metadata import read_acl
-
from bup.metadata import xattr
if xattr:
def remove_selinux(attrs):
diff --git a/test/int/test_resolve.py b/test/int/test_resolve.py
index cf66eb32..c9a222a8 100644
--- a/test/int/test_resolve.py
+++ b/test/int/test_resolve.py
@@ -4,7 +4,6 @@ from binascii import unhexlify
from errno import ELOOP, ENOTDIR
from os import symlink
from stat import S_IFDIR
-from sys import stderr
import os
from time import localtime, strftime

@@ -12,8 +11,6 @@ from wvpytest import *

from bup import git, path, vfs
from bup.compat import environ
-from bup.io import path_msg
-from bup.metadata import Metadata
from bup.repo import LocalRepo, RemoteRepo
from buptest import ex, exo
from buptest.vfs import tree_dict
diff --git a/test/int/test_vfs.py b/test/int/test_vfs.py
index 8c3e9272..f9b3f65c 100644
--- a/test/int/test_vfs.py
+++ b/test/int/test_vfs.py
@@ -1,8 +1,6 @@

from __future__ import absolute_import, print_function
from binascii import unhexlify
-from collections import namedtuple
-from errno import ELOOP, ENOTDIR
from io import BytesIO
from os import symlink
from random import Random, randint
@@ -17,7 +15,6 @@ from wvpytest import *
from bup._helpers import write_random
from bup import git, metadata, vfs
from bup.compat import environ, fsencode, items, range
-from bup.git import BUP_CHUNKED
from bup.helpers import exc, shstr
from bup.metadata import Metadata
from bup.repo import LocalRepo
diff --git a/test/int/test_xstat.py b/test/int/test_xstat.py
index b35a57f5..660a231f 100644
--- a/test/int/test_xstat.py
+++ b/test/int/test_xstat.py
@@ -1,6 +1,5 @@

from __future__ import absolute_import
-import math, tempfile, subprocess

from wvpytest import *

--
2.30.2

[Rob Browning]

The su tests run as non-root (bup), not root.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---

.cirrus.yml | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 712f953f..879c2bf7 100644

--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,6 +1,6 @@

task:

- name: debian (py2)
+ name: debian (py2 check root)

container:
image: debian:buster
cpu: 4

@@ -12,7 +12,7 @@ task:
eatmydata make -j6 PYTHON=python2 check

task:
- name: debian (long py2)
+ name: debian (py2 long-check)

container:
image: debian:buster
cpu: 4

@@ -26,7 +26,7 @@ task:
su -l bup -c "eatmydata make -j6 -C '$(pwd)' PYTHON=python2 long-check"

task:
- name: debian (root py2)
+ name: debian (py2 check)

container:
image: debian:buster
cpu: 4

@@ -40,7 +40,7 @@ task:
su -l bup -c "eatmydata make -j6 -C '$(pwd)' PYTHON=python2 check"

task:
- name: debian (py3)
+ name: debian (py3 check root)

container:
image: debian:buster
cpu: 4

@@ -52,7 +52,7 @@ task:
eatmydata make -j6 PYTHON=python3 check

task:
- name: debian (long py3)
+ name: debian (py3 long-check)

container:
image: debian:buster
cpu: 4

@@ -66,7 +66,7 @@ task:
su -l bup -c "eatmydata make -j6 -C '$(pwd)' PYTHON=python3 long-check"

task:
- name: debian (root py3)
+ name: debian (py3 check)

container:
image: debian:buster
cpu: 4

@@ -80,7 +80,7 @@ task:
su -l bup -c "eatmydata make -j6 -C '$(pwd)' PYTHON=python3 check"

task:
- name: freebsd (py3)
+ name: freebsd (py3 check)
freebsd_instance:
image: freebsd-12-2-release-amd64
cpu: 4
@@ -92,7 +92,7 @@ task:
gmake -j6 PYTHON=python3.7 check

task:
- name: macos (py3)
+ name: macos (py3 check)
osx_instance:
image: catalina-base
script: |
--
2.30.2

[Rob Browning]

Provide additional information about what's being tested, particularly
in CI.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
---

GNUmakefile | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/GNUmakefile b/GNUmakefile
index f648dfb6..2ed5b7ca 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -210,11 +210,12 @@ lint: dev/bup-python
./pylint

test: all test/tmp dev/python lint
- if test yes = "$$(dev/python -c 'import xdist; print("yes")' 2>/dev/null)"; then \
+ ./bup features
+ if test yes = "$$(dev/python -c 'import xdist; print("yes")' 2>/dev/null)"; then \

(set -x; ./pytest $(xdist_opt);) \

- else \
- (set -x; ./pytest;) \
- fi
+ else \
+ (set -x; ./pytest;) \
+ fi

stupid:
PATH=/bin:/usr/bin $(MAKE) test
--
2.30.2

[Rob Browning]

Adjust to accommodate the switch from PYTHON to BUP_PYTHON_CONFIG
cf. 4d9dd65141326b0f3ffa1658a0535348940ed017

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---
.cirrus.yml | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 879c2bf7..4cb4a4b8 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -9,7 +9,7 @@ task:

set -xe
dev/prep-for-debianish-build python2
dev/system-info

- eatmydata make -j6 PYTHON=python2 check
+ make -j6 BUP_PYTHON_CONFIG=python2.7-config check

task:
name: debian (py2 long-check)

@@ -23,7 +23,8 @@ task:
dev/system-info
adduser --disabled-password --gecos '' bup
chown -R bup:bup .
- su -l bup -c "eatmydata make -j6 -C '$(pwd)' PYTHON=python2 long-check"
+ printf "make -j6 -C %q BUP_PYTHON_CONFIG=python2.7-config long-check" \
+ "$(pwd)" | su -l bup

task:
name: debian (py2 check)
@@ -37,7 +38,8 @@ task:
dev/system-info
adduser --disabled-password --gecos '' bup
chown -R bup:bup .
- su -l bup -c "eatmydata make -j6 -C '$(pwd)' PYTHON=python2 check"
+ printf "make -j6 -C %q BUP_PYTHON_CONFIG=python2.7-config check" \
+ "$(pwd)" | su -l bup

task:

name: debian (py3 check root)

@@ -49,7 +51,7 @@ task:

set -xe
dev/prep-for-debianish-build python3
dev/system-info

- eatmydata make -j6 PYTHON=python3 check
+ make -j6 BUP_PYTHON_CONFIG=python3-config check

task:
name: debian (py3 long-check)

@@ -63,7 +65,8 @@ task:
dev/system-info
adduser --disabled-password --gecos '' bup
chown -R bup:bup .
- su -l bup -c "eatmydata make -j6 -C '$(pwd)' PYTHON=python3 long-check"
+ printf "make -j6 -C %q BUP_PYTHON_CONFIG=python3-config long-check" \
+ "$(pwd)" | su -l bup

task:
name: debian (py3 check)
@@ -77,7 +80,8 @@ task:
dev/system-info
adduser --disabled-password --gecos '' bup
chown -R bup:bup .
- su -l bup -c "eatmydata make -j6 -C '$(pwd)' PYTHON=python3 check"
+ printf "make -j6 -C %q BUP_PYTHON_CONFIG=python3-config check" \
+ "$(pwd)" | su -l bup

task:
name: freebsd (py3 check)
@@ -89,7 +93,7 @@ task:
set -xe
dev/prep-for-freebsd-build python3
dev/system-info
- gmake -j6 PYTHON=python3.7 check
+ BUP_PYTHON_CONFIG=python3.8-config make -j6 check

task:
name: macos (py3 check)
@@ -99,6 +103,5 @@ task:
set -xe
dev/prep-for-macos-build python3
export PKG_CONFIG_PATH=/usr/local/opt/readline/lib/pkgconfig
- export PYTHON=python3
dev/system-info
- make -j4 PYTHON=python3 check
+ make -j6 BUP_PYTHON_CONFIG=python3-config LDFLAGS=-L/usr/local/lib check
--
2.30.2

[Rob Browning]

When set to no, don't run pylint from ./pylint, just exit
successfully. When set to maybe, use dev/have-pylint to figure out if
pylint is available, and if so, run it, otherwise exit successfully
after describing the situation. When set to yes, always try to run
pylint.

This may be useful more generally, but in particular, it makes it
possible to run this:

./configure --with-pylint=maybe
make check-both

in situations where pylint is available for say python 3, but not
python 2.

Signed-off-by: Rob Browning <r...@defaultvalue.org>
Tested-by: Rob Browning <r...@defaultvalue.org>
---

GNUmakefile | 2 +-
config/configure | 17 +++++++++++++++++
configure | 7 ++-----
dev/have-pylint | 17 +++++++++++++++++
dev/prep-for-debianish-build | 4 ++--
pylint | 29 ++++++++++++++++++++++++++++-
6 files changed, 67 insertions(+), 9 deletions(-)
create mode 100755 dev/have-pylint

diff --git a/GNUmakefile b/GNUmakefile
index 2ed5b7ca..30f1b3a1 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -206,7 +206,7 @@ get_parallel_n = $(patsubst -j%,%,$(parallel_opt))

maybe_specific_n = $(if $(filter -j%,$(parallel_opt)),-n$(get_parallel_n))
xdist_opt = $(if $(filter -j,$(parallel_opt)),-nauto,$(maybe_specific_n))

-lint: dev/bup-python
+lint: dev/bup-exec dev/bup-python

./pylint

test: all test/tmp dev/python lint

diff --git a/config/configure b/config/configure
index 31a43e10..f1faa6e8 100755
--- a/config/configure
+++ b/config/configure
@@ -1,5 +1,7 @@
#!/usr/bin/env bash

+ac_help="--with-pylint[=yes|no|maybe] require and run pylint (maybe)"
+
bup_find_prog()
{
# Prints prog path to stdout or nothing.
@@ -35,6 +37,20 @@ bup_try_c_code()

TARGET=bup

+argv=()
+with_pylint=maybe
+while test $# -gt 0; do
+ case "$1" in
+ --with-pylint=yes) with_pylint=yes; shift;;
+ --with-pylint=maybe) with_pylint=maybe; shift;;
+ --with-pylint=no) with_pylint=no; shift;;
+ *) argv+=("$1"); shift;;
+ esac
+done
+
+# Set $@ to the adjusted args
+set - "${argv[@]}"
+
. ./configure.inc

# FIXME: real tmpdir
@@ -341,6 +357,7 @@ set -euo pipefail
mkdir -p config.var.tmp
echo -n "$MAKE" > config.var.tmp/bup-make
echo -n "$bup_python_config" > config.var.tmp/bup-python-config
+echo -n "$with_pylint" > config.var.tmp/with-pylint
mv config.var.tmp config.var

if test -e bin; then rm -r bin; fi
diff --git a/configure b/configure
index f7cb9289..9d6ca151 100755
--- a/configure
+++ b/configure
@@ -1,9 +1,6 @@
#!/bin/sh

-if test "$#" -gt 0; then
- echo "Usage: configure" 1>&2
- exit 1
-fi
+set -e

-cd config &&
+cd config
exec ./configure "$@"
diff --git a/dev/have-pylint b/dev/have-pylint
new file mode 100755
index 00000000..15262b47
--- /dev/null
+++ b/dev/have-pylint
@@ -0,0 +1,17 @@
+#!/bin/sh
+"""": # -*-python-*-
+bup_exec="$(dirname "$0")/bup-exec" || exit $?
+exec "$bup_exec" "$0" ${1+"$@"}
+"""
+
+from __future__ import absolute_import, print_function
+
+import sys
+
+try:
+ import pylint
+except ImportError as ex:
+ sys.exit(1)
+except BaseException as ex:
+ print(ex, file=sys.stderr)
+ sys.exit(2)
diff --git a/dev/prep-for-debianish-build b/dev/prep-for-debianish-build
index d45ce303..0ded7001 100755
--- a/dev/prep-for-debianish-build
+++ b/dev/prep-for-debianish-build
@@ -25,14 +25,14 @@ case "$pyver" in
$common_debs \
python2.7-dev python-fuse \
python-"$xattr" python-tornado python-pytest \
- python-pytest-xdist
+ python-pytest-xdist pylint
;;
python3)
apt-get install -y \
$common_debs \
python3-dev python3-distutils python3-fuse \
python3-"$xattr" python3-tornado python3-pytest \
- python3-pytest-xdist
+ python3-pytest-xdist pylint3
;;
*)
usage 1>&2
diff --git a/pylint b/pylint
index c04af785..d0474b4f 100755
--- a/pylint
+++ b/pylint
@@ -1,9 +1,36 @@
-#!/bin/sh
+#!/usr/bin/env bash

# Changes here might also be appropriate for ./pytest

set -eu

+with_pylint=$(cat config/config.var/with-pylint)
+
+case "$with_pylint" in
+ yes) ;;
+ no)
+ echo "./pylint: doing nothing given ./configure --with-pylint=no" 1>&2
+ exit 0
+ ;;
+ maybe)
+ rc=0
+ dev/have-pylint || rc=$?
+ case "$rc" in
+ 0) ;;
+ 1)
+ echo "./pylint: doing nothing (pylint not found)" 1>&2
+ exit 0
+ ;;
+ *) exit "$rc" ;;
+ esac
+ ;;
+ *)
+ printf "./pylint: unexpected config/config.var/with-pylint value %q\n" \
+ "$with_pylint" 1>&2
+ exit 2
+ ;;
+esac
+

script_home="$(cd "$(dirname "$0")" && pwd -P)"

testlibdir="$script_home/test/lib"

--
2.30.2