Hey all,

I recently was looking for a way to add bash completion to a management
command that I made. With changeset 11526[0] during the djangocon sprints,
bash completion was moved from bash into Python. Now there is a super basic
bash script that calls django-admin.py with the correct environment for it
to autocomplete.

Now that the code is in Python, this lets us do a lot more with it. As
implemented, a custom management command's options (--settings, --list) will
be autocompleted. There is currently no way to define arguments to your
function that will be autocompleted. I went ahead and looked through the
code today and wrote up some proof of concept code that does just this.

What I did
========

First thing I did was write tests for the current behavior[1]. No tests were
written for the original commit, so if nothing else, these tests should be
commited. The link there works for the current environment, then I added a
few more tests that test my changes as well.

After that, I implemented a basic API for declaring a completion in a
Command class[2]. I will describe here the implemented API. I'm hoping that
people have some ideas about the correct way to implement this.

Currently, your Command class will define a complete() function along with
your handle() function. The complete() function can then return two
different things. In the simple case, it can return a simple list of
arguments that it expects to be able to handle. These will be passed along
to the bash completion, and complete appropriately.

So for example if your custom command 'check_site' had a complete() command
that returned ['ljworld', 'kusports', 'lcom'], and on the command line you
did `django-admin.py check_site ljw<tab>`, it would return ljworld.

The more complex case is where you want to be able to define multiple
positional arguments for a command. Currently, this is implemented by
returning a dict with the key being the number that you want to complete
(this sucks. So you could do something like:

    def complete(self):
        return {
            '0': ['ljworld', 'kusports']
            '1': ['year', 'week', 'day']
        }

Then you would be able to do `django-admin.py ljworld ye<tab>` and it would
return `year`.

Currently there is also special casing for returning All Installed Apps, and
All Installed Commands. I went ahead and made a magic symlbol "APP_NAME" and
"COMMAND_NAME" that will evaluate to these lists. Both of these APIs seem a
bit hacky.

So currently I am thinking about making the following changes to make this
stuff a bit better.

Proposal
=======

I think that instead of special casing[3] the commands that take an APP_NAME
etc., we would put the complete() function on the BaseCommand, and then for
built-in commands that want custom bash completion, use the proposed API to
define it.

Instead of using simple strings for APP_NAME and COMMAND_NAME, we put these
as constants on the base command class. These could either be special cased
in the bash completion script, like currently, or actually make these
evaluate to the actual list they represent. Computing both of these values
isn't processor intensive, so it would make sense to just have them there.

I don't know what exactly we should do to represent commands that want
control over specific arguments. The current dict with keys of ints seems
silly, but I don't know a much better way. Perhaps represent this as a list
of lists, where index 0 would be the same as the dict['0'].

This would make a basic class end up looking something like this:

    def complete(self):
        return [
             ['awesome', 'sweet'],
             BaseCommand.SUBCOMMANDS,
             BaseCommand.INSTALLED_APPS,
        ]

Please let me know what you all think, and what I have missed. Implementing
these changes would resolve a lot of the special casing in the bash
completion, and turn it into a real API that is useful for management
command authors. I think that this is a big win. Bash completion is one of
those things that is super useful, but a damned pain to implement.
Abstracting it this way would make a lot of our commands grow bash
completion I would bet.

[0] http://code.djangoproject.com/changeset/11526
[1]
http://github.com/ericholscher/django/commit/eceda439ab1a950230bd5f79...
[2]
http://github.com/ericholscher/django/commit/b48b261d2533b45fd7bb955e...
[3]
http://code.djangoproject.com/browser/django/trunk/django/core/manage...

--
Eric Holscher
Web Developer at The World Company in Lawrence, Ks
http://ericholscher.com

Hey Eric,

That's a very good idea. I've looked through the current management commands to see what arguments they take. The most used variants are:

- no arguments
- custom list
- appname(s)
- fixture(s)

The problem with your proposal is the handling of multiple appnames and fixtures (e.g. "django-admin.py sqlreset [options] <appname appname ...>" since each position must have a value assigned in the iterable returned by `complete`. I think a better idea would be to pass the current cursor position to the `complete` function:

    def complete(self, pos):
        if pos == 1:
            return ('foo', 'bar',)
        elif pos >= 2:
            from django.conf import settings
            return [(a.split('.')[-1], 0) for a in settings.INSTALLED_APPS]

That way we could easily handle cases where one or multiple appnames are passed and avoid dealing with complex data structures. Displaying the directory listing for fixtures wouldn't be a problem either since the function could just return None. The `-o default` setting of the bash-completion would then default to displaying the directory listing.

In addition to that, I'd also like to change the way options and arguments are displayed. If you currently type "./manage.py dumpdata <tab>" it will display options as well as arguments. All other bash-completion scripts[1] display only the arguments. The options are displayed if the user entered a dash and pressed tab e.g. "./manage.py dumpdata -<tab>". I think we should do the same to be consistent with other bash-completion scripts.

[1]: http://git.debian.org/?p=bash-completion/bash-completion.git;a=tree;f...

Arthur

On Nov 15, 2009, at 11:35 PM, Eric Holscher wrote:

also argument can be a subcommand, and i don't understand how you
intended to complete options. i.e. "manage.py migrate show -l --indent
4 h<cursor_here>", and so that means like everywhere you need previous
arguments sometimes to be accounted when making choices for the next
one, and few more tweaks, so
def complete(self, args, current_arg_start='', opts={}):
    """args: are previous args,
       current_arg_start: to be completed now, and will be overwritten,
           you don't have to return arguments only starting with
current_arg_start,
           but this one might be used to reduce output and make faster search.
       opts: someone might pass options explicitly there. or it might
happen automatically.
    """
    args, opts = parseopts(args, opts) # remove completed --opts (in
either short or long form)
    if args == ['']:
        return ['show', 'list', 'add', 'rename', 'help', 'whatever']
    if args == ['migrate']:
        return migrate_complete(args, current_arg_start, opts)

is much better.

Next, since 99% of commands use optparse, they can be automated entirely.

I believe there are projects out there supporting bash completion for
optparsed python scripts.
Say, http://furius.ca/optcomplete/ is one of those.
Please look carefully what it provides, what not, and why it does so.

Generally speaking, usually,

main ::= head* body
head ::= word | '-' shortopt_with_0_or_1_args | '--' longopt_with_0_or_1_args
body = type_A type_B ... type_N

where argument types are usually fixed; for options usually there are
0 or 1 arguments.

I believe you all know this... Just you shared with your understanding
on the problem.
And I expressed my feeling that problem is much more complex than both
of you think.

I'm also agreed that it would be great to be able to abstract the task
of command completion to inside the command definition itself. Having
the list of app-name-accepting commands hardcoded is an annoying
limitation.

However, I'm not completely sold on your API proposal. The second
version is certainly better than the first (the dict approach is way
off), but I agree with Yuri - this problem is a bit more complex than
your API allows for.

For example, what if the subcommand 'awesome' allows one set of
arguments, and 'sweet' allows a different set? What if the order of
arguments is significant? What if the order of arguments with relation
to command flags is significant (e.g., any command given after the -x
option will be handled differently)?

At the very least, I suspect that the completion method would need to
be given some sort of context on which to base completions. This would
need to include (but isn't necessarily limited to) the previous
command line arguments that have been parsed, or some analogous
description of the command line context.

Alternatively, the completion command could pass back a grammar of
some kind that described the order in which arguments can be accepted,
and providing callbacks to describe how each token in the grammar can
be completed. In some ways, the syntax you have proposed follows this
direction - but the grammar you have provided isn't expressive enough
to cover anything but really simple examples. For example, I'm not
sure how you would express the current completion syntax for sqlall
(which allows any number of arguments, each of which must be an app)
in a way that is unambiguous to your ljworld example which appears to
allow exactly 3 arguments.

Of course, the risk here is that you end up re-implementing the whole
of optparse just to enable the parsing of individual options - which
isn't really an appealing option.

Yours,
Russ Magee %-)