reportbug: don't know: bug in apt [list] or in grep

[js-...@online.de]

Hello together,
listing packages in apt with ”sudo“ in the title returns different output (bash commands at the end of the email). I would fill a bug report, but I'm not sure whether to address it to grep or apt. How do you see this?

Kind regards
Julian Schreck
--
$ apt list sudo* vs. $ apt list | grep "^sudo[a-z-]"
$ apt list *sudo vs. $ apt list | grep "[a-z-]sudo/"
$ apt list *sudo* vs. $ apt list | grep "sudo"

[Markus Schönhaber]

19.01.23, 09:10 +0100, js-...@online.de:

> Hello together,
> listing packages in apt with ”sudo“ in the title returns different output (bash commands at the end of the email). I would fill a bug report, but I'm not sure whether to address it to grep or apt. How do you see this?

To me it seems there's neither a bug in apt nor in grep but rather in
your regular expressions.

> Kind regards
> Julian Schreck
> --
> $ apt list sudo* vs. $ apt list | grep "^sudo[a-z-]"

The former also matches "sudo", the latter RE does not - it matches ex.
"sodoa" or "sudo-".

> $ apt list *sudo vs. $ apt list | grep "[a-z-]sudo/"

The former also matches "sudo", the latter RE does not - it matches ex.
"bsudo" or "-sudo".

> $ apt list *sudo* vs. $ apt list | grep "sudo"

This might give the same results.

BTW: using unquoted wildcards in parameters to shell commands is most
often a bad idea (unless they are really meant to be file name patterns).

--
Regards
mks

[Christoph Brinkhaus]

Am Thu, Jan 19, 2023 at 09:10:30AM +0100 schrieb js-...@online.de:

Hello Julian,

It seems to be more a shell topic or how man 7 glob is handled.
Please try the first patterns with "" signs as
apt-list "sudo*" instead of apt-list sudo* and so on.
It made the difference for me using the fish shell.

Kind regards,
Christoph
--
Ist die Katze gesund
schmeckt sie dem Hund.

[to...@tuxteam.de]

Doing "apt list sudo*" without any quoting is asking for trouble anyway,
regardless of the shell you use.

What the command apt will ultimately see will depend on what files are
in your current directory.

Don't do that. It will drive you crazy.

Cheers
--
t

[Christoph Brinkhaus]

Am Thu, Jan 19, 2023 at 11:40:50AM +0100 schrieb to...@tuxteam.de:

Hello Tomas,

> On Thu, Jan 19, 2023 at 11:31:23AM +0100, Christoph Brinkhaus wrote:
> > Am Thu, Jan 19, 2023 at 09:10:30AM +0100 schrieb js-...@online.de:
> >
> > Hello Julian,
> >
> > > Hello together,
> > > listing packages in apt with ”sudo“ in the title returns different output (bash commands at the end of the email). I would fill a bug report, but I'm not sure whether to address it to grep or apt. How do you see this?
> > >
> > > Kind regards
> > > Julian Schreck
> > > --
> > > $ apt list sudo* vs. $ apt list | grep "^sudo[a-z-]"
> > > $ apt list *sudo vs. $ apt list | grep "[a-z-]sudo/"
> > > $ apt list *sudo* vs. $ apt list | grep "sudo"
> >
> > It seems to be more a shell topic or how man 7 glob is handled.
> > Please try the first patterns with "" signs as
> > apt-list "sudo*" instead of apt-list sudo* and so on.
> > It made the difference for me using the fish shell.
>
> Doing "apt list sudo*" without any quoting is asking for trouble anyway,
> regardless of the shell you use.

Yes, quoting is the correct term. And it is of superior improtance,
especially if the input is not perdefined in terms of the structure.
E.g. existance of special signs as the asterisk *, white spaces and so
on.

>
> What the command apt will ultimately see will depend on what files are
> in your current directory.

For curiosity If have done a small test as below.
Unfortunately there are a few outputs in German. For this comparisons
the exact meanings of the German text has no importance at all.

1. The first command of the original poster:
chris@lenovo ~> apt list sudo*
fish: No matches for wildcard 'sudo*'. See `help expand`.
apt list sudo*
^
2. Create an empty file to see the effect:
chris@lenovo ~> touch sudo

3. The first command of the original poster:
chris@lenovo ~> apt list sudo*
Auflistung… Fertig
sudo/stable-security,now 1.9.5p2-3+deb11u1 amd64 [Installiert,automatisch]
N: Es gibt 1 zusätzliche Version. Bitte verwenden Sie die Option »-a«, um sie anzuzeigen.

4. Run apt list sudo
The result is the same as 3.

5. Delete the original empty file and create a modified one:
mv sudo sudolala

6. The first command of the original poster:
chris@lenovo ~> apt list sudo*
Auflistung… Fertig

There is no sudolala.

> Don't do that. It will drive you crazy.

Sure. Thanks for triggering me to do the small tests.
I did not want to hijack the thread. But I think it makes sense to
show the effect of missing quoting.

[Greg Wooledge]

On Thu, Jan 19, 2023 at 12:11:43PM +0100, Christoph Brinkhaus wrote:
> For curiosity If have done a small test as below.
> Unfortunately there are a few outputs in German. For this comparisons
> the exact meanings of the German text has no importance at all.
>
> 1. The first command of the original poster:
> chris@lenovo ~> apt list sudo*
> fish: No matches for wildcard 'sudo*'. See `help expand`.
> apt list sudo*
> ^
> 2. Create an empty file to see the effect:
> chris@lenovo ~> touch sudo
>
> 3. The first command of the original poster:
> chris@lenovo ~> apt list sudo*
> Auflistung… Fertig
> sudo/stable-security,now 1.9.5p2-3+deb11u1 amd64 [Installiert,automatisch]
> N: Es gibt 1 zusätzliche Version. Bitte verwenden Sie die Option »-a«, um sie anzuzeigen.

Your examples are excellent, but there's one more piece to this story.
The behavior of a glob that doesn't match any files (e.g. your 1.)
depends on the shell, and on the settings that are chosen within that
shell.

In bash, with default settings, a glob that doesn't match any files
is passed on literally as a command argument. The classic example
of this is demonstrated by "ls":

unicorn:~$ ls *.ttx
ls: cannot access '*.ttx': No such file or directory

If I misspell "*.txt" as "*.ttx" I get this message. Bash (my shell)
saw the *.ttx glob, and tried to expand it to the list of matching
filenames in my directory. There aren't any, so it passed the glob
along without expanding it. This allowed ls to see the original glob
just as I had typed it, and to include it in its error message.

If I did the same thing with apt list sudo* I would get this:

unicorn:~$ ls sudo*
ls: cannot access 'sudo*': No such file or directory
unicorn:~$ apt list sudo*
Listing... Done
sudo-ldap/stable-security 1.9.5p2-3+deb11u1 amd64
sudo-ldap/stable-security 1.9.5p2-3+deb11u1 i386
sudo/stable-security,now 1.9.5p2-3+deb11u1 amd64 [installed]
sudo/stable-security 1.9.5p2-3+deb11u1 i386
sudoku-solver/stable 1.0.1-2 amd64
sudoku-solver/stable 1.0.1-2 i386
sudoku/stable 1.0.5-2+b3 amd64
sudoku/stable 1.0.5-2+b3 i386

Since there are no files matching the sudo* glob in my directory, bash
passes it along untouched, and apt uses it as a matching pattern against
package names.

The fact that this *appears* to work is what causes so much confusion.
It will "work" some of the time, but not all of the time, and you'll
get different results depending on which directory you're in, on which
computer.

Bash has two other settings for handling unmatched globs. The first one
is called "nullglob", and if it's turned on, an unmatched glob is simply
discarded from the command argument list.

unicorn:~$ bash
unicorn:~$ shopt | grep glob
dotglob off
extglob off
failglob off
globasciiranges on
globstar off
nocaseglob off
nullglob off
unicorn:~$ shopt -s nullglob
unicorn:~$ apt list sudo* | head

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Listing...
0ad-data-common/stable,stable 0.0.23.1-1.1 all
0ad-data/stable,stable 0.0.23.1-1.1 all
0ad/stable 0.0.23.1-5+b1 amd64
0ad/stable 0.0.23.1-5+b1 i386
0install-core/stable 2.16-1 amd64
0install-core/stable 2.16-1 i386
0install/stable 2.16-1 amd64
0install/stable 2.16-1 i386
0xffff/stable 0.9-1 amd64

In this case, the unmatched sudo* glob is dropped altogether, and the
resulting command is simply "apt list". I knew what would happen, so I
piped it to head, to shorten the output.

The other setting is called "failglob", and if it's turned on, an
unmatched glob causes an error at the shell level, and prevents execution
of the command.

unicorn:~$ shopt -u nullglob; shopt -s failglob
unicorn:~$ apt list sudo*
bash: no match: sudo*
unicorn:~$ exit
exit

This is very much like what your fish example did, and what (t)csh does
by default, if I remember correctly.

So, just to add to the list of people who've already said it: always
quote the patterns that you pass to apt list, because you want apt
to use them directly, without your shell interfering.

[to...@tuxteam.de]

On Thu, Jan 19, 2023 at 07:13:46AM -0500, Greg Wooledge wrote:

[...]

> So, just to add to the list of people who've already said it: always
> quote the patterns that you pass to apt list, because you want apt
> to use them directly, without your shell interfering.

And, if in doubt, just replace the original command with "echo". This
will let you "see" what your command is going to see. I think it's a
good way to get a feeling of what's going on. Those examples are bash,
in its default setting:

tomas@trotzki:~$ echo foo*
foo foo.txt
tomas@trotzki:~$ echo blarg*
blarg*

Cheers
--
t

[to...@tuxteam.de]

On Thu, Jan 19, 2023 at 02:00:00PM +0100, DdB wrote:

[...]

> I was really curious, how Greg would put words to this one. And i gotta
> applaud: Such unambiguous explanations, and so circumspect at the same
> time. Even understanding the basis for confusion, i could learn
> something new from this (other settings ...).

Greg knows bash. Really. If you don't know his bash pages, you should:

http://mywiki.wooledge.org/

Cheers
--
t

[Greg Wooledge]

On Thu, Jan 19, 2023 at 02:00:00PM +0100, DdB wrote:

> Am 19.01.2023 um 13:13 schrieb Greg Wooledge:
> > The fact that this *appears* to work is what causes so much confusion.
> > It will "work" some of the time, but not all of the time, and you'll
> > get different results depending on which directory you're in, on which
> > computer.
> >
> > Bash has two other settings for handling unmatched globs. The first one
> > is called "nullglob", and if it's turned on, an unmatched glob is simply
> > discarded from the command argument list.
>

> I was really curious, how Greg would put words to this one. And i gotta
> applaud: Such unambiguous explanations, and so circumspect at the same
> time. Even understanding the basis for confusion, i could learn
> something new from this (other settings ...).

Regarding the nullglob and failglob settings:

failglob is pretty safe. You can play around with that, and see how
it affects things. Other shells have their equivalent enabled by
default, so it's basically a preference.

nullglob should not be used in an interactive shell. There are far too
many tools that rely on the shell's normal behavior (passing along an
unmatched glob), and which will break if you turn on nullglob and then
type something innocuous. The classic example, again, is ls. If you
have nullglob enabled, and you do "ls *.ttx" or whatever typo, you'll
get the behavior of "ls" with no arguments, which shows all the files,
rather than an error message.

The only place where nullglob is useful is in a script, and even then,
it has to be a script where extra care is used. I believe the intended
use case is something like this:

for f in *.txt; do
printf 'Processing "%s"...' "$f"
process -- "$f" && echo ' done.'
done

With nullglob enabled, if there are no matching *.txt files, the loop
is skipped. Without nullglob, the *.txt will be unchanged, and the
loop body will be executed once, passing the literal '*txt' to the
printf and process commands.

There are several reasons why nullglob isn't on by default in scripts,
and isn't commonly used. One of them is the same as with interactive
shells -- there are programs that may behave in a surprising way if
an unmatched glob is removed, rather than passed along.

Another reason is that you probably want to perform a basic sanity
check on your filenames before calling your "process" command. The
glob might match things that aren't regular files -- directories, for
example, or FIFOs, which could cause "process" to fail or hang. So,
you might prefer something like this:

for f in *.txt; do
test -f "$f" || continue
printf 'Processing "%s"...' "$f"
process -- "$f" && echo ' done.'
done

Adding that basic sanity check also prevents processing an umatched
*.txt glob as a filename, so nullglob just isn't needed here.

[Michael]

another excellent example why your posts are almost always worth reading.

thank you! :)