Bookworm: dash shell globs don't recognise [^...] to negate a character class

[David]

In Debian, shell scripts that have
#!/usr/bin/sh
as the first line are executed by the 'dash' shell.

If you write such scripts, you might be interested
to know that 'dash' currently has a behaviour
change in Debian version 12 Bookworm compared to
Debian version 11 Bullseye.

This is being discussed at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028002

Below is a demo of the change which shows an
example of possible consequences.

In Debian version 11 Bullseye,
Bash and 'dash' behave the same:
$ cat /etc/debian_version
11.6
$ mkdir eek
$ cd eek
$ touch aa bb 11 22
$ bash
$ echo [!0-9]*
aa bb
$ echo [^0-9]*
aa bb
$ sh
$ echo [!0-9]*
aa bb
$ echo [^0-9]*
aa bb

In Debian version 12 Bookworm,
Bash and 'dash' behave differently:
$ cat /etc/debian_version
12.0
$ mkdir eek
$ cd eek
$ touch aa bb 11 22
$ bash
$ echo [!0-9]*
aa bb
$ echo [^0-9]*
aa bb
$ sh
$ echo [!0-9]*
aa bb
$ echo [^0-9]*
11 22 <------ new behaviour by dash

[Jude DaShiell]

When I write bash scripts and I've done this for several debian versions I
use:
#!/usr/bin/env bash
That has worked in the past.


-- Jude <jdashiel at panix dot com> "There are four boxes to be used in
defense of liberty: soap, ballot, jury, and ammo. Please use in that
order." Ed Howdershelt 1940.

[Jude DaShiell]

Something else I've noticed with bash.
Those work when run in mate-terminal but not in console for some strange
reason.

-- Jude <jdashiel at panix dot com> "There are four boxes to be used in
defense of liberty: soap, ballot, jury, and ammo. Please use in that
order." Ed Howdershelt 1940.

[Greg Wooledge]

On Thu, Apr 13, 2023 at 12:12:23AM +0000, David wrote:
> $ echo [^0-9]*
> 11 22 <------ new behaviour by dash

The [^chars] syntax is a negation in Basic and Extended Regular
Expressions, and in bash's globs (it's a bash extension), but NOT in
POSIX globs.

The correct negation syntax in POSIX sh globs is [!chars].

This goes all the way back to the original Bourne shell, where the ^
character was a synonym for | (pipe), because many terminals did not
have a | character. Due to the way sh parses commands, using ^ in
globs would have caused problems. That's why sh uses [!chars].

[Vincent Lefevre]

On 2023-04-12 21:07:59 -0400, Greg Wooledge wrote:
> On Thu, Apr 13, 2023 at 12:12:23AM +0000, David wrote:
> > $ echo [^0-9]*
> > 11 22 <------ new behaviour by dash
>
> The [^chars] syntax is a negation in Basic and Extended Regular
> Expressions, and in bash's globs (it's a bash extension), but NOT in
> POSIX globs.
>
> The correct negation syntax in POSIX sh globs is [!chars].

It seems that [^chars] has undefined behavior (various POSIX shells
behave differently: bash, ksh93 and yash regard ^ as a negation, but
not mksh). In short, for portability, this form should not be used
in shells.

--
Vincent Lefèvre <vin...@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

[Max Nikulin]

On 13/04/2023 08:07, Greg Wooledge wrote:
> On Thu, Apr 13, 2023 at 12:12:23AM +0000, David wrote:
>> $ echo [^0-9]*
>> 11 22 <------ new behaviour by dash

[...]

> The correct negation syntax in POSIX sh globs is [!chars].

The shellcheck utility gives this suggestion as well. Their wiki have
concise description of the issue, but links describing it in more
details with precise references have not been added yet.

https://www.shellcheck.net/wiki/SC3026
> Help by adding links to BashFAQ, StackOverflow, man pages, POSIX, etc

[Greg Wooledge]

Well, it's not clear how to actually add such a link. I guess you're
supposed to do it through github or something?

Anyway, here's the POSIX documentation section:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_13

And the relevant piece of text:

[ If an open bracket introduces a bracket expression as in XBD RE
Bracket Expression, except that the <exclamation-mark> character
( '!' ) shall replace the <circumflex> character ( '^' ) in its
role in a non-matching list in the regular expression notation, it
shall introduce a pattern bracket expression. A bracket expression
starting with an unquoted <circumflex> character produces unspecified
results. Otherwise, '[' shall match the character itself.

[rhkr...@gmail.com]

On Thursday, April 13, 2023 10:36:08 PM Greg Wooledge wrote:
> Anyway, here's the POSIX documentation section:
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#t
> ag_18_13
>
> And the relevant piece of text:
>
> [ If an open bracket introduces a bracket expression as in XBD RE
> Bracket Expression, except that the <exclamation-mark> character
> ( '!' ) shall replace the <circumflex> character ( '^' ) in its
> role in a non-matching list in the regular expression notation, it
> shall introduce a pattern bracket expression. A bracket expression
> starting with an unquoted <circumflex> character produces unspecified
> results. Otherwise, '[' shall match the character itself.

Wow -- I thought this was an English language list :-(

But seriously, that seems very hard to interpret / understand.

--
rhk

| No entity has permission to use this email to train an AI.

Sig truncated.

[Greg Wooledge]

If it helps, you can ignore most of it. The pieces that we care about
are:

1) the <exclamation-mark> character ( '!' ) shall replace the <circumflex>
character ( '^' )

2) A bracket expression starting with an unquoted <circumflex> character
produces unspecified results.

[David Wright]

The quick answer: it's a Standards document, so it's a penalty
you pay for precision. Look at the opening of that section:

"2.13.1 Patterns Matching a Single Character

"The following patterns matching a single character shall match a single character:"

BTW, the authors of these documents might be the sort of people
referred to by the early RFC authors in RFC 1000:

"We weren't sure whether there was really room
to think hard about these problems; surely someone from the east
would be along by and by to bring the word. …

"We had no official charter. Most of us were graduate students
and we expected that a professional crew would show up eventually
to take over the problems we were dealing with. …

"I remember having great fear that we would offend whomever the
official protocol designers were, and I spent a sleepless night
composing humble words for our notes. The basic ground rules were
that anyone could say anything and that nothing was official."

Cheers,
David.

[Max Nikulin]

On 14/04/2023 09:36, Greg Wooledge wrote:
> On Fri, Apr 14, 2023 at 09:08:08AM +0700, Max Nikulin wrote:
>> https://www.shellcheck.net/wiki/SC3026
>>> Help by adding links to BashFAQ, StackOverflow, man pages, POSIX, etc
> Well, it's not clear how to actually add such a link. I guess you're
> supposed to do it through github or something?

Thank you, Greg. I edited
https://github.com/koalaman/shellcheck/wiki/SC3026 (the link below the
page header) and the shellcheck.net page was updated in several hours.
Sorry, web interface assumes brief commit message, so I did not added
credits to you there.

I think, shellcheck is not known well enough while it may help to avoid
a lot of troubles with shell scripts. It can provide some hints when
unexpected behavior is observed and prevent it if used regularly.

As to [^c] vs. [!c], unfortunately the latter can not be always used as
portable variant. It is treated as history expansion in the case of
interactive bash session. From my point of view it is an argument to
support [^c]. The enhancement request was rejected however, see

Stephane Chazelas. Re: [v2 PATCH] expand: Always quote caret when using
fnmatch. Sun, 20 Feb 2022 07:15:44 +0000
https://lore.kernel.org/dash/20220220071544....@chazelas.org/

and

https://www.austingroupbugs.net/view.php?id=1558
0001558: require [^...] in addition to [!...] for bracket expression
negation

[Greg Wooledge]

On Sat, Apr 15, 2023 at 09:44:03AM +0700, Max Nikulin wrote:
> As to [^c] vs. [!c], unfortunately the latter can not be always used as
> portable variant. It is treated as history expansion in the case of
> interactive bash session.

Ugh. That abomination. I've had history expansion disabled for *years*.

[to...@tuxteam.de]

You have to escape it with a backslash. Quoting with single quotes also
helps, although I don't know whether that is portable itself.

Yes, somewhat annoying, unless one's making use of it.

Cheers
--
t

[Max Nikulin]

On 15/04/2023 12:02, to...@tuxteam.de wrote:
> On Fri, Apr 14, 2023 at 11:02:03PM -0400, Greg Wooledge wrote:
>> On Sat, Apr 15, 2023 at 09:44:03AM +0700, Max Nikulin wrote:
>>> As to [^c] vs. [!c], unfortunately the latter can not be always used as
>>> portable variant. It is treated as history expansion in the case of
>>> interactive bash session.
>>
>> Ugh. That abomination. I've had history expansion disabled for *years*.
>
> You have to escape it with a backslash. Quoting with single quotes also
> helps, although I don't know whether that is portable itself.

The problem is to prevent history expansion while keeping pattern
matching (glob) active.

du -ks -- .[!.]* | sort -n | tail

> Yes, somewhat annoying, unless one's making use of it.

From search engine results my impression is that there enough users of
history expansion in bash and zsh. Certainly they would be angry due to
a breaking changes if history expansion were disabled by default. That
is why, when recommending [!c] instead of [^c], it is necessary to say
"besides interactive shells such as bash and zsh". It should not be
assumed that everybody has history expansion disabled.

[to...@tuxteam.de]

On Sat, Apr 15, 2023 at 01:14:53PM +0700, Max Nikulin wrote:
> On 15/04/2023 12:02, to...@tuxteam.de wrote:
> > On Fri, Apr 14, 2023 at 11:02:03PM -0400, Greg Wooledge wrote:
> > > On Sat, Apr 15, 2023 at 09:44:03AM +0700, Max Nikulin wrote:
> > > > As to [^c] vs. [!c], unfortunately the latter can not be always used as
> > > > portable variant. It is treated as history expansion in the case of
> > > > interactive bash session.
> > >
> > > Ugh. That abomination. I've had history expansion disabled for *years*.
> >
> > You have to escape it with a backslash. Quoting with single quotes also
> > helps, although I don't know whether that is portable itself.
>
> The problem is to prevent history expansion while keeping pattern matching
> (glob) active.
>
> du -ks -- .[!.]* | sort -n | tail

I see, thanks

Cheers
--
t

[davidson]

On Sat, 15 Apr 2023 Max Nikulin wrote:
> On 15/04/2023 12:02, to...@tuxteam.de wrote:
>> On Fri, Apr 14, 2023 at 11:02:03PM -0400, Greg Wooledge wrote:
>>> On Sat, Apr 15, 2023 at 09:44:03AM +0700, Max Nikulin wrote:
>>>> As to [^c] vs. [!c], unfortunately the latter can not be always used as
>>>> portable variant. It is treated as history expansion in the case of
>>>> interactive bash session.
>>>
>>> Ugh. That abomination. I've had history expansion disabled for *years*.
>>
>> You have to escape it with a backslash. Quoting with single quotes also
>> helps, although I don't know whether that is portable itself.
>
> The problem is to prevent history expansion while keeping pattern matching
> (glob) active.
>
> du -ks -- .[!.]* | sort -n | tail

Are there versions of bash that exhibit history expansion in the
example above?

davidson@parsnip:0 ~$ bash --version # In case it matters
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
davidson@parsnip:0 ~$ echo $- $SHELLOPTS # History expansion enabled, interactive
himBHs braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor
davidson@parsnip:0 ~$ . .bash_functions/apt-description # Populate history
davidson@parsnip:0 ~$ whee!. # Demonstrate history expansion
whee. .bash_functions/apt-description
bash: whee.: command not found
davidson@parsnip:127 ~$ du -ks -- .[!.]* | sort -n | tail # No problem!
472 .keymap.new
816 .dvdcss
1544 .config
1884 .bash_history.d
3340 .xsession-errors
19008 .lynx
66012 .mozilla
260528 .local
348360 .cache
980500 .cabal
davidson@parsnip:0 ~$

--
Hackers are free people. They are like artists. If they are in a good
mood, they get up in the morning and begin painting their pictures.
-- Vladimir Putin

[Greg Wooledge]

On Sat, Apr 15, 2023 at 11:02:12AM +0000, davidson wrote:
> On Sat, 15 Apr 2023 Max Nikulin wrote:
> > The problem is to prevent history expansion while keeping pattern
> > matching (glob) active.
> >
> > du -ks -- .[!.]* | sort -n | tail
>
> Are there versions of bash that exhibit history expansion in the
> example above?

Not that I've found. I tried 3.2 and 2.05b and they're both fine.

unicorn:~$ bash-2.05b
unicorn:~$ set -H
unicorn:~$ !xyz
bash-2.05b: !xyz: event not found
unicorn:~$ du -ks -- .[!.]* | sort -n | tail
101208 .mozilla
136244 .t-engine
171952 .kolproxy
180180 .old-crufty-mozilla
222572 .local
442240 .vscode
991928 .config
1047780 .nuget
1476376 .cache
5798668 .steam

[Max Nikulin]

On 15/04/2023 19:37, Greg Wooledge wrote:
> On Sat, Apr 15, 2023 at 11:02:12AM +0000, davidson wrote:
>> On Sat, 15 Apr 2023 Max Nikulin wrote:
>>> The problem is to prevent history expansion while keeping pattern
>>> matching (glob) active.
>>>
>>> du -ks -- .[!.]* | sort -n | tail
>>
>> Are there versions of bash that exhibit history expansion in the
>> example above?
>
> Not that I've found. I tried 3.2 and 2.05b and they're both fine.

I am really sorry. I have checked the terminal app buffer where I was
experimenting with history expansion and almost certainly I
misinterpreted some result.

I see that inside square brackets expansion of exclamation mark is
suppressed when (and only when) it immediately follows the opening
bracket.
- [!c] is safe
- [a-f!@1-2] and [0-9!] are affected by history expansion
(and it is likely the source of my confusion)

Useful shortcuts for experiments:
- M-^ history-expand-line
- C-/ undo

When writing that message I decided to use the command where I usually
need negation in interactive sessions. So thank you for drawing my
attention that I was wrong.

I decided that history expansion in [!c] is consistent with

echo "Hello World!" in BASH Pitfalls
https://mywiki.wooledge.org/BashPitfalls#echo_.22Hello_World.21.22

and statements in "History Expansion" section of the BASH manual
(and the man page as well)
info "(bash) History Interaction"
https://www.gnu.org/software/bash/manual/html_node/History-Interaction.html

> History expansion is performed immediately after a complete line is
> read, before the shell breaks it into words, and is performed on each
> line individually.
...
> the history expansion character is also treated as quoted if it
> immediately precedes the closing double quote in a double-quoted string.

No exception for square brackets is mentioned.

[Vincent Lefevre]

On 2023-04-16 11:04:09 +0700, Max Nikulin wrote:
> On 15/04/2023 19:37, Greg Wooledge wrote:
> > On Sat, Apr 15, 2023 at 11:02:12AM +0000, davidson wrote:
> > > On Sat, 15 Apr 2023 Max Nikulin wrote:
> > > > The problem is to prevent history expansion while keeping pattern
> > > > matching (glob) active.
> > > >
> > > > du -ks -- .[!.]* | sort -n | tail
> > >
> > > Are there versions of bash that exhibit history expansion in the
> > > example above?
> >
> > Not that I've found. I tried 3.2 and 2.05b and they're both fine.
>
> I am really sorry. I have checked the terminal app buffer where I was
> experimenting with history expansion and almost certainly I
> misinterpreted some result.
>
> I see that inside square brackets expansion of exclamation mark is
> suppressed when (and only when) it immediately follows the opening
> bracket.
> - [!c] is safe
> - [a-f!@1-2] and [0-9!] are affected by history expansion
> (and it is likely the source of my confusion)

Having such an exception is very ugly. Contrary to bash, ksh93 and zsh
do not have such an exception. But since this is for interactive use,
there is usually no need for portability, so that one can use ^
instead of ! in shells that have history expansion.

BTW, history expansion can be very useful, but IMHO, this should
have been interactive and triggered by control characters or
escape sequences, not by "normal" characters.

[Greg Wooledge]

On Tue, Apr 18, 2023 at 04:19:47PM +0200, Vincent Lefevre wrote:
> BTW, history expansion can be very useful, but IMHO, this should
> have been interactive and triggered by control characters or
> escape sequences, not by "normal" characters.

History expansion originated in csh, and was duplicated in bash, which
aimed to be accomodating and familiar to both ksh/sh and csh users.

If history expansion feels foreign or alien, or if it has unexpected
interactions with other shell features... well, that's why.

[Max Nikulin]

On 18/04/2023 21:19, Vincent Lefevre wrote:
> BTW, history expansion can be very useful, but IMHO, this should
> have been interactive and triggered by control characters or
> escape sequences, not by "normal" characters.

It would be great. Unfortunately disabling histexpand option in bash
blocks the M-^ shortcut as well.

shopt -s histverify

allows at least to confirm that expansion result is expected.

[davidson]

There is also the 'p' (print) word designator, that prints the
expansion but does not execute it. Used like so:

$ !.:p # See what the history expansion for 'dot' is
. .bash_functions/manterm

This places the expansion in history as last command, so up-arrow +
return will now execute it.

[davidson]

On Wed, 19 Apr 2023 davidson wrote:
> On Wed, 19 Apr 2023 Max Nikulin wrote:
>> On 18/04/2023 21:19, Vincent Lefevre wrote:
>>> BTW, history expansion can be very useful, but IMHO, this should
>>> have been interactive and triggered by control characters or
>>> escape sequences, not by "normal" characters.
>>
>> It would be great. Unfortunately disabling histexpand option in bash blocks
>> the M-^ shortcut as well.
>>
>> shopt -s histverify
>>
>> allows at least to confirm that expansion result is expected.
>
> There is also the 'p' (print) word designator, that prints the

Sorry. Not a "word designator", but a "modifier". (Modifiers follow
word designator, if any).

[Max Nikulin]

On 20/04/2023 03:18, davidson wrote:
> On Wed, 19 Apr 2023 Max Nikulin wrote:
>> On 18/04/2023 21:19, Vincent Lefevre wrote:
>>> BTW, history expansion can be very useful, but IMHO, this should
>>> have been interactive and triggered by control characters or
>>> escape sequences, not by "normal" characters.
>>
>> It would be great. Unfortunately disabling histexpand option in bash
>> blocks the M-^ shortcut as well.
>>
>> shopt -s histverify
>>
>> allows at least to confirm that expansion result is expected.
>

> There is also the 'p' (print) modifier, that prints the

I corrected the line above basing on your another message.

> expansion but does not execute it. Used like so:
>
>  $ !.:p # See what the history expansion for 'dot' is
>  . .bash_functions/manterm
>
> This places the expansion in history as last command, so up-arrow +
> return will now execute it.

It is nice to have such feature. Likely you actively use history
expansion, while I prefer interactive approach: search (C-r), edit,
including yank-last-arg (M-.), perhaps operate-and-get-next (C-o). I
find it safer even if it is not so fast.

Unfortunately :p modifier does not prevent unexpected history expansion,
e.g. when a file name contains "!". That is why I would prefer to
suppress history expansion for every typed command, but still have
possibility to perform expansion on request (M-^ or similar shortcut),
perhaps limiting it to a single "!" before cursor.

histverify is a means to cancel command if unsolicited expansion or
unexpected result noticed.