argparse eating more than one '--'

[Benjamin]

I am converting some scripts from optparse to argparse, and discovered
a problem. Argparse seems to be eating more double-dashes than it
should, preventing the nesting of several commands on a cmdline, each
of which take '--' to mean "all the rest of the cmdline are
positionals, not options". Consider this toy reimplementation of
'nice' in python 2.7.1:

=====================================
#!/usr/local/python27-2.7.1/bin/python

import argparse, os, sys

parser = argparse.ArgumentParser(
formatter_class=argparse.RawDescriptionHelpFormatter,
description="""Nice a command""")
parser.add_argument("-v", "--verbose", action="store_true",
default=False, help="verbose mode")
parser.add_argument('cmdline', nargs='+', help="The command to run,
niced.")
args = parser.parse_args()

os.nice(9)

if args.verbose:
sys.stderr.write(sys.argv[0] + ' executing: ' + str(args.cmdline) +
'\n')
os.execvp(args.cmdline[0], args.cmdline)
=====================================

Here's a scenario where we end up using the above script.

1) We are on linux. We want to create a file called '-r' to measure
the worth of our interview candidates to see if they can remove it
without doing a rm -r. Evil, I know, but bear with me.

2) We use 'touch':

$ touch -r
touch: option requires an argument -- r
Try `touch --help' for more information.

3) Oops. Touch is treating '-r' like an option, not a filename, so we
separate touch's options from positional args with the standard
mechanism '--' as is documented in the 'Common options' section of the
'info coreutils' docs:

$ touch -- -r
$ ls
-r

4) success! Now, we're going to do this often, because as I said
we're evil, but yet in a bit of a paradox, we want to be friendly to
other users on the system so we want to use our above nice script to
nice it as we create these files. Here we go:

$ /tmp/nicer -v -- touch -- -r
/tmp/nicer executing: ['touch', '-r']
touch: option requires an argument -- r
Try `touch --help' for more information.

Oops! Argparse ate not just the -- intended for it to separate the -v
from the positional args, but the -- intended for touch!

http://docs.python.org/release/2.7.1/library/argparse.html#arguments-containing
states that:

=====================================
If you have positional arguments that must begin with '-' and don't
look like negative numbers, you can insert the pseudo-argument '--'
which tells parse_args that everything after that is a positional
argument
=====================================

I think this bit of code in argparse.py is to blame:

def _get_values(self, action, arg_strings):
# for everything but PARSER args, strip out '--'
if action.nargs not in [PARSER, REMAINDER]:
arg_strings = [s for s in arg_strings if s != '--']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

5) Here's how I'd write my 'nicer' program with optparse:

=====================================
#!/usr/local/python27-2.7.1/bin/python

import os, sys

from optparse import OptionParser
parser = OptionParser(usage=os.path.basename(sys.argv[0]) + '
<cmd> ...')
parser.disable_interspersed_args()
parser.add_option("-v", "--verbose", dest="verbose",
action="store_true", default=False, help="verbose mode")
(options, args) = parser.parse_args()
if (len(args) < 1):
parser.print_help()
sys.exit(1)

os.nice(9)

if options.verbose:
sys.stderr.write(sys.argv[0] + ' executing: ' + str(args) + '\n')
os.execvp(args[0], args)
=====================================

Since I am using disable_interspersed_args, that works with or without
the '--' to separate "nicer's" options from positionals:

$ /tmp/nicer.optparse -v -- touch -- -r
/tmp/nicer.optparse executing: ['touch', '--', '-r']
$ /tmp/nicer.optparse -v touch -- -r
/tmp/nicer.optparse executing: ['touch', '--', '-r']

With disable_interspersed_args, optparse treats anything in sys.argv,
starting with the first element not beginning with a -, as a
positional.

So is there any way to achieve what I want using argparse?

Thanks,
Benjamin

[akira]

On 02/11/2012 11:59 PM, Benjamin wrote:
> I am converting some scripts from optparse to argparse, and discovered
> a problem. Argparse seems to be eating more double-dashes than it
> should, preventing the nesting of several commands on a cmdline, each
> of which take '--' to mean "all the rest of the cmdline are
> positionals, not options". Consider this toy reimplementation of
> 'nice' in python 2.7.1:
>
> =====================================
> #!/usr/local/python27-2.7.1/bin/python
>
> import argparse, os, sys
>
> parser = argparse.ArgumentParser(
> formatter_class=argparse.RawDescriptionHelpFormatter,
> description="""Nice a command""")
> parser.add_argument("-v", "--verbose", action="store_true",
> default=False, help="verbose mode")
> parser.add_argument('cmdline', nargs='+', help="The command to run,
> niced.")
> args = parser.parse_args()
>
> os.nice(9)
>
> if args.verbose:
> sys.stderr.write(sys.argv[0] + ' executing: ' + str(args.cmdline) +
> '\n')
> os.execvp(args.cmdline[0], args.cmdline)
> =====================================

From the docs:

argparse.REMAINDER. All the remaining command-line arguments are
gathered into a list. This is commonly useful for command line utilities
that dispatch to other command line utilities.

parser.add_argument('cmdline', nargs=REMAINDER)

And run it as:

$ /tmp/nicer -v touch -- -r
/tmp/nicer executing: ['touch', '--', '-r']

Note: unlike getopt_long the REMAINDER leaves the first '--' intact:

$ /tmp/nicer -v -- touch -- -r

/tmp/nicer executing: ['--', 'touch', '--', '-r']

To be compatible with the coreutils nice command you could remove it:

if args.cmdline[0] == '--':
del args.cmdline[0]

I think the *first* '--' should have higher priority than REMAINDER and
it should always be removed inside argparse. In rare cases when the
first positional argument is '--' it can be provided twice on the
command-line: -- --

--
akira.