argparse: use of double dash to separate options and positional arguments

[Amit Ramon]

Hello,

I'm trying to use argparse in a program that gets some options and
positional arguments. I want to collect all the positional arguments
in a single argument using the REMAINDER option to add_argument() as
shown bellow:

parser = argparse.ArgumentParser(description='Test argparse')
parser.add_argument('-v', '--verbose', action='store_true')
parser.add_argument('cmd_args', nargs=argparse.REMAINDER)
print parser.parse_args()

This works well unless the first positional argument starts with a
'-'.

For example, with the above code

my_prog -v hello world

works well, but

my_prog -v -x hello world

Fails with an error message 'error: unrecognized arguments: -x'.

If I invoke the program with a '--' at the end of the options (as I
understand is a common standard and a POSIX recommendation), as in:

my_prog -v -- -x hello world

It works well again.

However, a few questions remain:

Firstly, as far as I can tell the use of '--' is not documented in the
argparse documentation (but the code of argparse clearly shows that it
is a feature).

Secondly, when using '--', this string itself is inserted into the
list of the collected positional arguments (in the above example in
the cmd_args variable):

$ ./my_prog -v -- -x hello world
Namespace(cmd_args=['--', '-x', 'hello', 'world'], verbose=True)

It's quiet easy to check for it and remove it if it exists, but it
still seems like it shouldn't be there - it is not really an argument,
just a delimiter.

Am I doing anything wrong, or is this a bug? I hope someone here can
shed some light on this.

Thanks,

Amit



--

[Peter Otten]

Amit Ramon wrote:

> Hello,
>
> I'm trying to use argparse in a program that gets some options and
> positional arguments. I want to collect all the positional arguments
> in a single argument using the REMAINDER option to add_argument() as
> shown bellow:
>
> parser = argparse.ArgumentParser(description='Test argparse')
> parser.add_argument('-v', '--verbose', action='store_true')
> parser.add_argument('cmd_args', nargs=argparse.REMAINDER)
> print parser.parse_args()
>
> This works well unless the first positional argument starts with a
> '-'.
>
> For example, with the above code
>
> my_prog -v hello world
>
> works well, but
>
> my_prog -v -x hello world
>
> Fails with an error message 'error: unrecognized arguments: -x'.

This looks like a bug to me. Please report it on bug.python.org.

> If I invoke the program with a '--' at the end of the options (as I
> understand is a common standard and a POSIX recommendation), as in:
>
> my_prog -v -- -x hello world
>
> It works well again.
>
> However, a few questions remain:
>
> Firstly, as far as I can tell the use of '--' is not documented in the
> argparse documentation (but the code of argparse clearly shows that it
> is a feature).
>
> Secondly, when using '--', this string itself is inserted into the
> list of the collected positional arguments (in the above example in
> the cmd_args variable):
>
> $ ./my_prog -v -- -x hello world
> Namespace(cmd_args=['--', '-x', 'hello', 'world'], verbose=True)
>
> It's quiet easy to check for it and remove it if it exists, but it
> still seems like it shouldn't be there - it is not really an argument,
> just a delimiter.

I'm not sure about this one; one purpose of REMAINDER is to pass on the
unprocessed arguments to another program/script, and this might follow the
same convention. Should

parser.add_argument('-v', '--verbose', action='store_true')
parser.add_argument('cmd_args', nargs=argparse.REMAINDER)

args = parser.parse_args()
subprocess.call(["rm"] + args.cmd_args)

$ my_prog -v -- -r foo

attempt to delete two files "-r" and "foo" or remove the "foo" directory?
The first is the safer alternative, and as you say stripping the "--" is
easy.

[Amit Ramon]

Peter Otten <__pet...@web.de> [2015-11-06 11:57 +0100]:

>>
>> For example, with the above code
>>
>> my_prog -v hello world
>>
>> works well, but
>>
>> my_prog -v -x hello world
>>
>> Fails with an error message 'error: unrecognized arguments: -x'.
>
>This looks like a bug to me. Please report it on bug.python.org.

Why does it looks like a bug? To me it seems okay - argparse expects
that an argument that starts with '-' is an option, and it also expects
to be told about all possible options, so it complains on
encountering an unrecognized 'option'. This is why a special delimiter
is needed to mark the 'end of options'.

I'm using the REMAINDER exactly for passing on arguments to another
program, as in your example. I would expect that everything after
the '--' should be passed on as is - it should be up to the other
program to decide what to do with the arguments it receives (So in
your examples, rm would actually try to remove the "foo" directory).

And although stripping the '--' is easy, why should the user of
argparse need to handle it? Still unclear to me.

Cheers,

Amit

[Peter Otten]

My interpretation was that REMAINDER should give you the rest of the
commandline args, no matter what special meaning these args might have for
argparse.

[Random832]

Peter Otten <__pet...@web.de> writes:

> I'm not sure about this one; one purpose of REMAINDER is to pass on the
> unprocessed arguments to another program/script, and this might follow the
> same convention. Should
>
> parser.add_argument('-v', '--verbose', action='store_true')
> parser.add_argument('cmd_args', nargs=argparse.REMAINDER)
> args = parser.parse_args()
> subprocess.call(["rm"] + args.cmd_args)
>
> $ my_prog -v -- -r foo
>
> attempt to delete two files "-r" and "foo" or remove the "foo" directory?
> The first is the safer alternative, and as you say stripping the "--" is
> easy.

I think it should be the second. If the caller wants it to be treated as
a list of files rather than a list of arguments, it should be using
subprocess.call(["rm", "--"] + args.cmd_args).

The purpose is, as you said, to pass the *unprocessed* argument. -- has
been processed by being interpreted as a terminator for the argument
list.

I have a script (it's a shell script, not a python script) where I
actually do something similar... it does some setup, then calls ssh. If
I want to pass some options directly to ssh, I call it as:
"scriptname -script-option -- -ssh-option remote-command"