Improving (and testing!) bash completion
30 views
Skip to first unread message
Eric Holscher
Nov 15, 2009, 5:35:33 PM11/15/09
to django-d...@googlegroups.com
Hey all,
I recently was looking for a way to add bash completion to a management command that I made. With changeset 11526[0] during the djangocon sprints, bash completion was moved from bash into Python. Now there is a super basic bash script that calls django-admin.py with the correct environment for it to autocomplete.
Now that the code is in Python, this lets us do a lot more with it. As implemented, a custom management command's options (--settings, --list) will be autocompleted. There is currently no way to define arguments to your function that will be autocompleted. I went ahead and looked through the code today and wrote up some proof of concept code that does just this.
What I did
========
First thing I did was write tests for the current behavior[1]. No tests were written for the original commit, so if nothing else, these tests should be commited. The link there works for the current environment, then I added a few more tests that test my changes as well.
After that, I implemented a basic API for declaring a completion in a Command class[2]. I will describe here the implemented API. I'm hoping that people have some ideas about the correct way to implement this.
Currently, your Command class will define a complete() function along with your handle() function. The complete() function can then return two different things. In the simple case, it can return a simple list of arguments that it expects to be able to handle. These will be passed along to the bash completion, and complete appropriately.
So for example if your custom command 'check_site' had a complete() command that returned ['ljworld', 'kusports', 'lcom'], and on the command line you did `django-admin.py check_site ljw<tab>`, it would return ljworld.
The more complex case is where you want to be able to define multiple positional arguments for a command. Currently, this is implemented by returning a dict with the key being the number that you want to complete (this sucks. So you could do something like:
def complete(self):
return {
'0': ['ljworld', 'kusports']
'1': ['year', 'week', 'day']
}
Then you would be able to do `django-admin.py ljworld ye<tab>` and it would return `year`.
Currently there is also special casing for returning All Installed Apps, and All Installed Commands. I went ahead and made a magic symlbol "APP_NAME" and "COMMAND_NAME" that will evaluate to these lists. Both of these APIs seem a bit hacky.
So currently I am thinking about making the following changes to make this stuff a bit better.
Proposal
=======
I think that instead of special casing[3] the commands that take an APP_NAME etc., we would put the complete() function on the BaseCommand, and then for built-in commands that want custom bash completion, use the proposed API to define it.
Instead of using simple strings for APP_NAME and COMMAND_NAME, we put these as constants on the base command class. These could either be special cased in the bash completion script, like currently, or actually make these evaluate to the actual list they represent. Computing both of these values isn't processor intensive, so it would make sense to just have them there.
I don't know what exactly we should do to represent commands that want control over specific arguments. The current dict with keys of ints seems silly, but I don't know a much better way. Perhaps represent this as a list of lists, where index 0 would be the same as the dict['0'].
This would make a basic class end up looking something like this:
def complete(self):
return [
['awesome', 'sweet'],
BaseCommand.SUBCOMMANDS,
BaseCommand.INSTALLED_APPS,
]
Please let me know what you all think, and what I have missed. Implementing these changes would resolve a lot of the special casing in the bash completion, and turn it into a real API that is useful for management command authors. I think that this is a big win. Bash completion is one of those things that is super useful, but a damned pain to implement. Abstracting it this way would make a lot of our commands grow bash completion I would bet.
[0] http://code.djangoproject.com/changeset/11526
[1] http://github.com/ericholscher/django/commit/eceda439ab1a950230bd5f792d3f8baef86a56a7
[2] http://github.com/ericholscher/django/commit/b48b261d2533b45fd7bb955e50869aa1f41bab7b#L0R330
[3] http://code.djangoproject.com/browser/django/trunk/django/core/management/__init__.py?rev=11526#L313
--
Eric Holscher
Web Developer at The World Company in Lawrence, Ks
http://ericholscher.com
I recently was looking for a way to add bash completion to a management command that I made. With changeset 11526[0] during the djangocon sprints, bash completion was moved from bash into Python. Now there is a super basic bash script that calls django-admin.py with the correct environment for it to autocomplete.
Now that the code is in Python, this lets us do a lot more with it. As implemented, a custom management command's options (--settings, --list) will be autocompleted. There is currently no way to define arguments to your function that will be autocompleted. I went ahead and looked through the code today and wrote up some proof of concept code that does just this.
What I did
========
First thing I did was write tests for the current behavior[1]. No tests were written for the original commit, so if nothing else, these tests should be commited. The link there works for the current environment, then I added a few more tests that test my changes as well.
After that, I implemented a basic API for declaring a completion in a Command class[2]. I will describe here the implemented API. I'm hoping that people have some ideas about the correct way to implement this.
Currently, your Command class will define a complete() function along with your handle() function. The complete() function can then return two different things. In the simple case, it can return a simple list of arguments that it expects to be able to handle. These will be passed along to the bash completion, and complete appropriately.
So for example if your custom command 'check_site' had a complete() command that returned ['ljworld', 'kusports', 'lcom'], and on the command line you did `django-admin.py check_site ljw<tab>`, it would return ljworld.
The more complex case is where you want to be able to define multiple positional arguments for a command. Currently, this is implemented by returning a dict with the key being the number that you want to complete (this sucks. So you could do something like:
def complete(self):
return {
'0': ['ljworld', 'kusports']
'1': ['year', 'week', 'day']
}
Then you would be able to do `django-admin.py ljworld ye<tab>` and it would return `year`.
Currently there is also special casing for returning All Installed Apps, and All Installed Commands. I went ahead and made a magic symlbol "APP_NAME" and "COMMAND_NAME" that will evaluate to these lists. Both of these APIs seem a bit hacky.
So currently I am thinking about making the following changes to make this stuff a bit better.
Proposal
=======
I think that instead of special casing[3] the commands that take an APP_NAME etc., we would put the complete() function on the BaseCommand, and then for built-in commands that want custom bash completion, use the proposed API to define it.
Instead of using simple strings for APP_NAME and COMMAND_NAME, we put these as constants on the base command class. These could either be special cased in the bash completion script, like currently, or actually make these evaluate to the actual list they represent. Computing both of these values isn't processor intensive, so it would make sense to just have them there.
I don't know what exactly we should do to represent commands that want control over specific arguments. The current dict with keys of ints seems silly, but I don't know a much better way. Perhaps represent this as a list of lists, where index 0 would be the same as the dict['0'].
This would make a basic class end up looking something like this:
def complete(self):
return [
['awesome', 'sweet'],
BaseCommand.SUBCOMMANDS,
BaseCommand.INSTALLED_APPS,
]
Please let me know what you all think, and what I have missed. Implementing these changes would resolve a lot of the special casing in the bash completion, and turn it into a real API that is useful for management command authors. I think that this is a big win. Bash completion is one of those things that is super useful, but a damned pain to implement. Abstracting it this way would make a lot of our commands grow bash completion I would bet.
[0] http://code.djangoproject.com/changeset/11526
[1] http://github.com/ericholscher/django/commit/eceda439ab1a950230bd5f792d3f8baef86a56a7
[2] http://github.com/ericholscher/django/commit/b48b261d2533b45fd7bb955e50869aa1f41bab7b#L0R330
[3] http://code.djangoproject.com/browser/django/trunk/django/core/management/__init__.py?rev=11526#L313
--
Eric Holscher
Web Developer at The World Company in Lawrence, Ks
http://ericholscher.com
Arthur Koziel
Nov 16, 2009, 2:36:24 PM11/16/09
to django-d...@googlegroups.com
Hey Eric,
That's a very good idea. I've looked through the current management commands to see what arguments they take. The most used variants are:
- no arguments
- custom list
- appname(s)
- fixture(s)
The problem with your proposal is the handling of multiple appnames and fixtures (e.g. "django-admin.py sqlreset [options] <appname appname ...>" since each position must have a value assigned in the iterable returned by `complete`. I think a better idea would be to pass the current cursor position to the `complete` function:
def complete(self, pos):
if pos == 1:
return ('foo', 'bar',)
elif pos >= 2:
from django.conf import settings
return [(a.split('.')[-1], 0) for a in settings.INSTALLED_APPS]
That way we could easily handle cases where one or multiple appnames are passed and avoid dealing with complex data structures. Displaying the directory listing for fixtures wouldn't be a problem either since the function could just return None. The `-o default` setting of the bash-completion would then default to displaying the directory listing.
In addition to that, I'd also like to change the way options and arguments are displayed. If you currently type "./manage.py dumpdata <tab>" it will display options as well as arguments. All other bash-completion scripts[1] display only the arguments. The options are displayed if the user entered a dash and pressed tab e.g. "./manage.py dumpdata -<tab>". I think we should do the same to be consistent with other bash-completion scripts.
Arthur
--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=.
Yuri Baburov
Nov 16, 2009, 7:04:31 PM11/16/09
to django-d...@googlegroups.com
Hi Eric, Arthur,
also argument can be a subcommand, and i don't understand how you
intended to complete options. i.e. "manage.py migrate show -l --indent
4 h<cursor_here>", and so that means like everywhere you need previous
arguments sometimes to be accounted when making choices for the next
one, and few more tweaks, so
def complete(self, args, current_arg_start='', opts={}):
"""args: are previous args,
current_arg_start: to be completed now, and will be overwritten,
you don't have to return arguments only starting with
current_arg_start,
but this one might be used to reduce output and make faster search.
opts: someone might pass options explicitly there. or it might
happen automatically.
"""
args, opts = parseopts(args, opts) # remove completed --opts (in
either short or long form)
if args == ['']:
return ['show', 'list', 'add', 'rename', 'help', 'whatever']
if args == ['migrate']:
return migrate_complete(args, current_arg_start, opts)
is much better.
Next, since 99% of commands use optparse, they can be automated entirely.
I believe there are projects out there supporting bash completion for
optparsed python scripts.
Say, http://furius.ca/optcomplete/ is one of those.
Please look carefully what it provides, what not, and why it does so.
Generally speaking, usually,
main ::= head* body
head ::= word | '-' shortopt_with_0_or_1_args | '--' longopt_with_0_or_1_args
body = type_A type_B ... type_N
where argument types are usually fixed; for options usually there are
0 or 1 arguments.
I believe you all know this... Just you shared with your understanding
on the problem.
And I expressed my feeling that problem is much more complex than both
of you think.
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com
also argument can be a subcommand, and i don't understand how you
intended to complete options. i.e. "manage.py migrate show -l --indent
4 h<cursor_here>", and so that means like everywhere you need previous
arguments sometimes to be accounted when making choices for the next
one, and few more tweaks, so
def complete(self, args, current_arg_start='', opts={}):
"""args: are previous args,
current_arg_start: to be completed now, and will be overwritten,
you don't have to return arguments only starting with
current_arg_start,
but this one might be used to reduce output and make faster search.
opts: someone might pass options explicitly there. or it might
happen automatically.
"""
args, opts = parseopts(args, opts) # remove completed --opts (in
either short or long form)
if args == ['']:
return ['show', 'list', 'add', 'rename', 'help', 'whatever']
if args == ['migrate']:
return migrate_complete(args, current_arg_start, opts)
is much better.
Next, since 99% of commands use optparse, they can be automated entirely.
I believe there are projects out there supporting bash completion for
optparsed python scripts.
Say, http://furius.ca/optcomplete/ is one of those.
Please look carefully what it provides, what not, and why it does so.
Generally speaking, usually,
main ::= head* body
head ::= word | '-' shortopt_with_0_or_1_args | '--' longopt_with_0_or_1_args
body = type_A type_B ... type_N
where argument types are usually fixed; for options usually there are
0 or 1 arguments.
I believe you all know this... Just you shared with your understanding
on the problem.
And I expressed my feeling that problem is much more complex than both
of you think.
MSN: bu...@live.com
Russell Keith-Magee
Nov 18, 2009, 6:14:14 AM11/18/09
to django-d...@googlegroups.com
On Mon, Nov 16, 2009 at 6:35 AM, Eric Holscher <er...@ericholscher.com> wrote:
> Hey all,
- I've just committed it.
> After that, I implemented a basic API for declaring a completion in a
> Command class[2]. I will describe here the implemented API. I'm hoping that
> people have some ideas about the correct way to implement this.
...
but very useful when it is.
I'm also agreed that it would be great to be able to abstract the task
of command completion to inside the command definition itself. Having
the list of app-name-accepting commands hardcoded is an annoying
limitation.
However, I'm not completely sold on your API proposal. The second
version is certainly better than the first (the dict approach is way
off), but I agree with Yuri - this problem is a bit more complex than
your API allows for.
For example, what if the subcommand 'awesome' allows one set of
arguments, and 'sweet' allows a different set? What if the order of
arguments is significant? What if the order of arguments with relation
to command flags is significant (e.g., any command given after the -x
option will be handled differently)?
At the very least, I suspect that the completion method would need to
be given some sort of context on which to base completions. This would
need to include (but isn't necessarily limited to) the previous
command line arguments that have been parsed, or some analogous
description of the command line context.
Alternatively, the completion command could pass back a grammar of
some kind that described the order in which arguments can be accepted,
and providing callbacks to describe how each token in the grammar can
be completed. In some ways, the syntax you have proposed follows this
direction - but the grammar you have provided isn't expressive enough
to cover anything but really simple examples. For example, I'm not
sure how you would express the current completion syntax for sqlall
(which allows any number of arguments, each of which must be an app)
in a way that is unambiguous to your ljworld example which appears to
allow exactly 3 arguments.
Of course, the risk here is that you end up re-implementing the whole
of optparse just to enable the parsing of individual options - which
isn't really an appealing option.
Yours,
Russ Magee %-)
> Hey all,
>
> What I did
> ========
>
> First thing I did was write tests for the current behavior[1]. No tests were
> written for the original commit, so if nothing else, these tests should be
> commited. The link there works for the current environment, then I added a
> few more tests that test my changes as well.
This bit is super-mega-awesome. Thanks for taking the time to do this
> What I did
> ========
>
> First thing I did was write tests for the current behavior[1]. No tests were
> written for the original commit, so if nothing else, these tests should be
> commited. The link there works for the current environment, then I added a
> few more tests that test my changes as well.
- I've just committed it.
> After that, I implemented a basic API for declaring a completion in a
> Command class[2]. I will describe here the implemented API. I'm hoping that
> people have some ideas about the correct way to implement this.
> Please let me know what you all think, and what I have missed. Implementing
> these changes would resolve a lot of the special casing in the bash
> completion, and turn it into a real API that is useful for management
> command authors. I think that this is a big win. Bash completion is one of
> those things that is super useful, but a damned pain to implement.
> Abstracting it this way would make a lot of our commands grow bash
> completion I would bet.
I'm agreed that this is one of those things that is hard to implement,
> these changes would resolve a lot of the special casing in the bash
> completion, and turn it into a real API that is useful for management
> command authors. I think that this is a big win. Bash completion is one of
> those things that is super useful, but a damned pain to implement.
> Abstracting it this way would make a lot of our commands grow bash
> completion I would bet.
but very useful when it is.
I'm also agreed that it would be great to be able to abstract the task
of command completion to inside the command definition itself. Having
the list of app-name-accepting commands hardcoded is an annoying
limitation.
However, I'm not completely sold on your API proposal. The second
version is certainly better than the first (the dict approach is way
off), but I agree with Yuri - this problem is a bit more complex than
your API allows for.
For example, what if the subcommand 'awesome' allows one set of
arguments, and 'sweet' allows a different set? What if the order of
arguments is significant? What if the order of arguments with relation
to command flags is significant (e.g., any command given after the -x
option will be handled differently)?
At the very least, I suspect that the completion method would need to
be given some sort of context on which to base completions. This would
need to include (but isn't necessarily limited to) the previous
command line arguments that have been parsed, or some analogous
description of the command line context.
Alternatively, the completion command could pass back a grammar of
some kind that described the order in which arguments can be accepted,
and providing callbacks to describe how each token in the grammar can
be completed. In some ways, the syntax you have proposed follows this
direction - but the grammar you have provided isn't expressive enough
to cover anything but really simple examples. For example, I'm not
sure how you would express the current completion syntax for sqlall
(which allows any number of arguments, each of which must be an app)
in a way that is unambiguous to your ljworld example which appears to
allow exactly 3 arguments.
Of course, the risk here is that you end up re-implementing the whole
of optparse just to enable the parsing of individual options - which
isn't really an appealing option.
Yours,
Russ Magee %-)
Reply all
Reply to author
Forward
0 new messages