Strange compgen behaviour

[Mathias Dahl]

Hi fellow bashers!

I am trying to add some completion to a command. The completion should
list all movies I have in a certain folder, regardless if I am in that
folder or not. I have kind of got it to work in several variants but
all have some issue. The current problem I am looking at is very
strange. I have isolated it down to a strange behaviour with compgen.
Let's see if I can describe it clearly enough:

zf contains the list of movie file names
zc is the current input, in my case "/home/mathias/Videos/movies/H"

$ compgen -W "${zf}" -- ${zc}

Here is the output:

/home/mathias/Videos/movies/Harry.Potter...
/home/mathias/Videos/movies/Harry.Potter...
/home/mathias/Videos/movies/Harry.Potter...
/home/mathias/Videos/movies/Harry.Potter...
/home/mathias/Videos/movies/Harry.Potter...
/home/mathias/Videos/movies/Brazil (Terry Gilliam, 1985).avi
/home/mathias/Videos/movies/Ice.Age.2...
/home/mathias/Videos/movies/True.Blood.S01...
/home/mathias/Videos/movies/True.Blood.S01...

I have shortended the names to make it more readable and also removed
some hits.

The main issue here is how for example the line with Brazil can match
the input. Same goes for True Blood. And, to make it even more scary,
an input such as "/home/mathias/Videos/movies/B" does not match the
Brazil line.

What's up with this?

My bash version:

$ bash --version
GNU bash, version 3.2.39(1)-release (i486-pc-linux-gnu)

Running under Ubuntu 8.04.

Thanks!

/Mathias

PS. If you like I can post the whole function that takes care of the
completion, but I think it is not important here.

PPS. The main cause of the issues I have seens to be that some file
names have spaces, and as I said earlier, every way I have tried, even
the ones that workes so so, have some issues. Any tips to get this
relatively fool proof?

[Bernd Eggink]

Mathias Dahl schrieb:

> Hi fellow bashers!
>
> I am trying to add some completion to a command. The completion should
> list all movies I have in a certain folder, regardless if I am in that
> folder or not. I have kind of got it to work in several variants but
> all have some issue. The current problem I am looking at is very
> strange. I have isolated it down to a strange behaviour with compgen.
> Let's see if I can describe it clearly enough:
>
> zf contains the list of movie file names
> zc is the current input, in my case "/home/mathias/Videos/movies/H"
>
> $ compgen -W "${zf}" -- ${zc}
>
> Here is the output:
>
> /home/mathias/Videos/movies/Harry.Potter...
> /home/mathias/Videos/movies/Harry.Potter...
> /home/mathias/Videos/movies/Harry.Potter...
> /home/mathias/Videos/movies/Harry.Potter...
> /home/mathias/Videos/movies/Harry.Potter...
> /home/mathias/Videos/movies/Brazil (Terry Gilliam, 1985).avi
> /home/mathias/Videos/movies/Ice.Age.2...
> /home/mathias/Videos/movies/True.Blood.S01...
> /home/mathias/Videos/movies/True.Blood.S01...

It depends heavily on how the variables IFS and zf are set. From 'man bash':

-W wordlist
The wordlist is split using the characters in the IFS special
variable as delimiters, and each resultant word is expanded.
The possible completions are the members of the resultant
list which match the word being completed.

You didn't say how you assigned the variable zf. If you simply did
zf=$(ls /home/mathias/Videos/movies/*), the "Brazil" line will be split
into 4 words instead of 1. However, your output suggest that you somehow
managed to combine all file names to a single word starting with
Harry.Potter.

Try this: Choose a character which doesn't appear in any file name,
e.g., ':'.

list=$(printf "%s:" /home/mathias/Videos/movies/*)
IFS=: compgen -W "$list" -- $zc

Regards,
Bernd

--
Bernd Eggink
http://sudrala.de

[Mathias Dahl]

> It depends heavily on how the variables IFS and zf are set. From 'man bash':
>
> -W wordlist
>     The  wordlist is split using the characters in the IFS special
>     variable as delimiters, and each resultant word is expanded.
>     The possible completions are the members  of  the  resultant
>     list which match the word being completed.

I used a newline since the original listing comes from `find'.

> You didn't say how you assigned the variable zf. If you simply did
> zf=$(ls /home/mathias/Videos/movies/*), the "Brazil" line will be split
> into 4 words instead of 1. However, your output suggest that you somehow
> managed to combine all file names to a single word starting with
> Harry.Potter.

Yes, that could be the case.

> Try this: Choose a character which doesn't appear in any file name,
> e.g., ':'.
>
>      list=$(printf "%s:" /home/mathias/Videos/movies/*)
>      IFS=: compgen -W "$list" -- $zc

That works, thanks! However, I also want files from sub folders to be
found, so I use `find' to list them.

Here is my latest attempt, using the idea of setting IFS to `:':

_mm2() {
local cur files
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
files=$(find /home/mathias/Videos/movies/ -iname '*.avi' -type f -
printf "%p:")
OLDIFS=$IFS
IFS=:
COMPREPLY=($(compgen -W "${files}" -- ${cur}))
IFS=$OLDIFS
}
complete -o filenames -F _mm2 mm

Looks like it should work but it does not. Typing mm<SPC><TAB> gives
the listing and completes all the way to the path, but if I add B
again it does not match Brazil.

Any ideas?

[Bernd Eggink]

Mathias Dahl schrieb:

Hm, compgen appears to behave strange if words contain whitespace.
However, you don't need it, as you build the list yourself. Try this:

_mm2() {
local cur files
cur=${COMP_WORDS[COMP_CWORD]}
files=$(find /home/mathias/Videos/movies/ -iname "$cur*.avi" -type
f -printf "%P\n")
local IFS=$'\n'
COMPREPLY=($files)

[Chet Ramey]

> Hm, compgen appears to behave strange if words contain whitespace.

Well, it splits the argument to -W on $IFS as documented. What other
strange behavior do you see?

Chet

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU ch...@case.edu http://cnswww.cns.cwru.edu/~chet/

[Bernd Eggink]

Chet Ramey schrieb:

>> Hm, compgen appears to behave strange if words contain whitespace.
>
> Well, it splits the argument to -W on $IFS as documented. What other
> strange behavior do you see?

For example, this:

function _aha
{
local list="a b:c d:e f"
COMPREPLY=($(IFS=: compgen -W "$list"))
}

complete -F _aha aha

Typing aha <Tab> cycles through 6 items a, b, c, d, e, f, whereas I
would expect 3 items 'a b', 'c d', 'e f'. It looks like compgen splits
the argument to -W on $IFS _and_ whitespace. Or am I missing something?

[Chet Ramey]

Hmmm...let me take a look.

[Chris F.A. Johnson]

On Thu, 24 Sep 2009, Bernd Eggink wrote:

> Chet Ramey schrieb:
> > > Hm, compgen appears to behave strange if words contain whitespace.
> >
> > Well, it splits the argument to -W on $IFS as documented. What other
> > strange behavior do you see?
>
> For example, this:
>
> function _aha
> {
> local list="a b:c d:e f"
> COMPREPLY=($(IFS=: compgen -W "$list"))

That expands to:

COMPREPLY=( a b c d e f )

> }
>
> complete -F _aha aha
>
> Typing aha <Tab> cycles through 6 items a, b, c, d, e, f, whereas I would
> expect 3 items 'a b', 'c d', 'e f'. It looks like compgen splits the argument
> to -W on $IFS _and_ whitespace. Or am I missing something?

--
Chris F.A. Johnson, webmaster <http://woodbine-gerrard.com>
===================================================================
Author:
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)

[Mathias Dahl]

> Hm, compgen appears to behave strange if words contain whitespace.
> However, you don't need it, as you build the list yourself. Try this:
>
>   _mm2() {
>       local cur files
>       cur=${COMP_WORDS[COMP_CWORD]}
>       files=$(find /home/mathias/Videos/movies/ -iname "$cur*.avi" -type
> f -printf "%P\n")
>       local IFS=$'\n'
>       COMPREPLY=($files)
>   }

Ah, you're right of course, I can do the matching myself. I have yet
another version working now (had to change your latest suggestion and
use grep for matching because -name does not like full paths which
becomes the case here):

_mm() {

local cur files
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
files=$(find /home/mathias/Videos/movies/ -iname "*.avi" -type f -

printf "%p\n" | grep "${cur}")
local IFS=$'\n'
COMPREPLY=(${files})
}
complete -o filenames -F _mm mm

Now, this works almost. The remaining problem is that because `find'
finds file in subdirs (which I want, otherwise I could add the -
maxdepth option) as well, the `-o filenames' argument to `complete'
does not play well with it. I see the names of files in subdirs listed
when I type TAB (without path) but can never pick them without knowing
the name of the folder they are in. So I have to get rid of that
option, but then I have to shell quote the file name myself to handle
spaces, brackets of various sorts, comma characters etc. Will hunt for
such a function and see. There are all sorts of crazy helper functions
in /etc/bash_completion, of which I barely understand anything.

Thanks!

[Mathias Dahl]

> ...but then I have to shell quote the file name myself to handle

> spaces, brackets of various sorts, comma characters etc. Will hunt for
> such a function and see. There are all sorts of crazy helper functions
> in /etc/bash_completion, of which I barely understand anything.

I did not find any generic way to quote/escape file names so I
hardcoded some chars I know exist in my file names and used sed:

_mm() {
local cur files
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
files=$(find /home/mathias/Videos/movies/ -iname "*.avi" -type f -

printf "%p\n" | grep "${cur}" | sed 's/\([\(\) ,]\)/\\\1/g')

local IFS=$'\n'
COMPREPLY=(${files})
}

complete -F _mm mm

However, I don't like it, is feels ugly and one of these days there
will be some other funky char in some filename... :(

/Mathias

[Bernd Eggink]

Mathias Dahl schrieb:

Hm, I can't see any problem here. My version lets you pick any file in
any subdir by simply typing the name (or part of it) without the
directory part. After all, 'find -name' matches names, not paths (if you
want to match full paths, use 'find -path'). I'd also rather use printf
"%P\n" (capital P) instead of %p, the results look nicer (IMHO).

[Mathias Dahl]

> Hm, I can't see any problem here. My version lets you pick any file in
> any subdir by simply typing the name (or part of it) without the
> directory part. After all, 'find -name' matches names, not paths (if you
> want to match full paths, use 'find -path'). I'd also rather use printf
> "%P\n" (capital P) instead of %p, the results look nicer (IMHO).

When I try with %P it works (better). However, when I do that and use
the -o filenames option I see only the filenames of the sub folders
and sometimes that name does not reflect what it is as well as the
folder does (if for example I have a folder called Cool Movie 2 and in
it a file called thefile.avi). So, it seems I did not really want what
I thought I wanted from the beginning :) The output I get now looks
like this:

...
filie_in_root_folder.avi
Some sub folder/file_in_sub_folder.avi
...

The only annoyance now is the hardcoded quoting (to handle spaces and
other "bad" chars) I do using sed, but I guess I have to live with it.

This has been an interesting topic!

/Mathias

[Mathias Dahl]

> This has been an interesting topic!

I thought I should share the final version as well:

_mm() {
local cur files
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
files=$(find /home/mathias/Videos/movies/ -iname "*.avi" -type f -

printf "%P\n" | grep "${cur}" | sed "s/\\([][\\(\\) ,\']\\)/\\\\\\1/
g")

local IFS=$'\n'
COMPREPLY=(${files})
}

complete -F _mm mm

And the `mm' script needs to append the root folder, of course, like
so:

#!/bin/bash
mplayer "/home/mathias/Videos/movies/$1"

Works quite well. Maybe it will be useful to someone else too.

/Mathias

[Greg Wooledge]

On Thu, Sep 24, 2009 at 03:05:07PM -0700, Mathias Dahl wrote:
> I did not find any generic way to quote/escape file names so I
> hardcoded some chars I know exist in my file names and used sed:

printf %q "$filename"

will either insert backslashes in front of all the shell metacharacters,
or $'...' quote the whole thing, or take some other action which renders
a string "safe". It's basically the opposite of eval.

[Mathias Dahl]

> printf %q "$filename"
>
> will either insert backslashes in front of all the shell metacharacters,
> or $'...' quote the whole thing, or take some other action which renders
> a string "safe".  It's basically the opposite of eval.

Interesting! Can I make it part of my pipe (in place of `sed') or need
I change it to a for or while loop? This is what I have now:

files=$(find /home/mathias/Videos/movies/ -iname "*.avi" -type f -

printf "%P\n" | grep "${cur}" | sed "s/\\([][\\(\\) ,\']\\)/\\\\\\1/
g")

/Mathias

[Mathias Dahl]

> Interesting! Can I make it part of my pipe (in place of `sed') or need
> I change it to a for or while loop? This is what I have now:
>
> files=$(find /home/mathias/Videos/movies/ -iname "*.avi" -type f -
> printf "%P\n" | grep "${cur}" | sed "s/\\([][\\(\\) ,\']\\)/\\\\\\1/
> g")

Got this to work:

files=$(find /home/mathias/Videos/movies/ -iname "*.avi" -type f -

printf "%P\n" | grep "${cur}" | while read file; do
printf %q "$file"
echo
done)

With the %q option to printf it no longer accepts a \n so I needed the
extra echo.

/Mathias

[Andreas Schwab]

Mathias Dahl <mathia...@gmail.com> writes:

> Got this to work:
>
> files=$(find /home/mathias/Videos/movies/ -iname "*.avi" -type f -
> printf "%P\n" | grep "${cur}" | while read file; do
> printf %q "$file"
> echo
> done)
>
> With the %q option to printf it no longer accepts a \n so I needed the
> extra echo.

printf "%q\n" works fine here.

Also, grep "${cur}" could probably be replaced by -path "*/${cur}*".

Andreas.

--
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."

[Chet Ramey]

Bernd Eggink wrote:
> Chet Ramey schrieb:

>>> Hm, compgen appears to behave strange if words contain whitespace.
>>

>> Well, it splits the argument to -W on $IFS as documented. What other
>> strange behavior do you see?
>
> For example, this:
>
> function _aha
> {
> local list="a b:c d:e f"
> COMPREPLY=($(IFS=: compgen -W "$list"))
> }
>

> complete -F _aha aha
>
> Typing aha <Tab> cycles through 6 items a, b, c, d, e, f, whereas I
> would expect 3 items 'a b', 'c d', 'e f'. It looks like compgen splits
> the argument to -W on $IFS _and_ whitespace. Or am I missing something?

I don't think you're considering the interaction between compgen,
command substitution, and compound array assignment.

I'll use the assignment to `list' you used above for the explanation.

compgen performs word splitting using $IFS and then outputs the results,
one word per line:

$ IFS=: compgen -W "$list"

a b
c d
e f

Command substitution takes that, chops off the last newline, and, since
it's unquoted, performs word splitting using $IFS on the result:

$ recho $(IFS=: compgen -W "$list")
argv[1] = <a>
argv[2] = <b>
argv[3] = <c>
argv[4] = <d>
argv[5] = <e>
argv[6] = <f>

If you want to preserve the embedded newlines, you have to quote the
expansion (but see below):

$ recho "$(IFS=: compgen -W "$list")"
argv[1] = <a b^Jc d^Je f>

When performing compound array assignments, though, the words between
the parens undergo all the shell word expansions, including word splitting:

$ COMPREPLY=( $(IFS=: compgen -W "$list") )
$ recho "${COMPREPLY[@]}"
argv[1] = <a>
argv[2] = <b>
argv[3] = <c>
argv[4] = <d>
argv[5] = <e>
argv[6] = <f>

So you can split on just the embedded newlines by setting $IFS:

$ oifs="$IFS"
$ IFS=$'\n' COMPREPLY=( $(IFS=: compgen -W "$list") )
$ IFS="$oifs"
$ recho "${COMPREPLY[@]}"
argv[1] = <a b>
argv[2] = <c d>
argv[3] = <e f>

Remember to save and restore $IFS, though, since the assignment to it
preceding the assignment to COMPREPLY will persist.

Chet

[Mathias Dahl]

> printf "%q\n" works fine here.

It does here too :), I misunderstood the syntax.

> Also, grep "${cur}" could probably be replaced by -path "*/${cur}*".

Indeed it can.

Smaller = better, thanks!

/Mathias

[Bernd.Eggink]

Chet Ramey schrieb:

Chet,

thanks for the clarification. It's actually quite obvious and I should
have seen it myself.

Regards,
Bernd

[Mathias Dahl]

I decided to make this into a blog post to in some way contribute back
to the community for all the help I got.

http://muublog.blogspot.com/2009/10/bash-completion-for-playing-movies.html

Enjoy!

/Mathias