go1.21.0 panics when loading plugin

[Bernd Fix]

After switching from go1.20.7 to go1.21.0 one of my applications
compiles without warnings or errors, but fails at run-time with the
following panic when loading a plugin:

panic: regexp: Compile(`^([\w./]+)/((?:\w+)|[*])(.+)?$`): error parsing
regexp: invalid escape sequence: `\w`

goroutine 1 [running]:
regexp.MustCompile({0x7fee026ddc4f, 0x1e})
.../golang/src/regexp/regexp.go:319 +0xb4
google.golang.org/grpc/internal/binarylog.init()

.../ext/pkg/mod/google.golang.org/gr...@v1.57.0/internal/binarylog/env_config.go:135
+0xf9
plugin.open({0xc0000146b0, 0xe})
.../golang/src/plugin/plugin_dlopen.go:95 +0x51c
plugin.Open(...)
.../golang/src/plugin/plugin.go:80

The above regex compiles fine in go1.21.0 directly, so I wonder what is
happening here.

After switching back to 1.20.7 and a full recompilation, the application
works fine again...

Cheers, Bernd.

[Kurtis Rader]

Did you also recompile the plugin with the new Go toolchain? It's a requirement of plugin support that the main program and all plugins be compiled with the same toolchain.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d208fed7-dec3-8865-b332-1d693122edda%40hoi-polloi.org.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Bernd Fix]

On 8/8/23 18:16, Kurtis Rader wrote:
> Did you also recompile the plugin with the new Go toolchain? It's a
> requirement of plugin support that the main program and all plugins be
> compiled with the same toolchain.

Yes, the loading program and the module were both compiled with the same
toolchain. After the first occurence of the issue, I removed everything
below $GOPATH and cleaned the cache, just to make sure everything got
recompiled correctly - but the problem persists.

If loading and loaded module differ, I would expect a panic like "plugin
was built with a different version" (I remember seeing that once) - but
certainly not a panic in a regular expression parser.

"Es sind nicht die besten Massen, die für Brot und Spiele den Verlust
der Freiheit verschmerzen." (Kautsky, 1919, "Diktatur des Proletariats")

[Marcello H]

Can you isolate the problem?
If the issue is within the regex, you might be able to put that in a separate program to test if it happens there too.

Op woensdag 9 augustus 2023 om 00:51:01 UTC+2 schreef Bernd Fix:

[Bernd Fix]

On 8/9/23 07:09, Marcello H wrote:
> Can you isolate the problem?
> If the issue is within the regex, you might be able to put that in a
> separate program to test if it happens there too.

I did that and the regex compiles fine in go1.21.0 (as expected).

>
> Op woensdag 9 augustus 2023 om 00:51:01 UTC+2 schreef Bernd Fix:
>
>> On 8/8/23 18:16, Kurtis Rader wrote:
>>> Did you also recompile the plugin with the new Go toolchain? It's a
>>> requirement of plugin support that the main program and all plugins be
>>> compiled with the same toolchain.
>>
>> Yes, the loading program and the module were both compiled with the same
>> toolchain. After the first occurence of the issue, I removed everything
>> below $GOPATH and cleaned the cache, just to make sure everything got
>> recompiled correctly - but the problem persists.
>>
>> If loading and loaded module differ, I would expect a panic like "plugin
>> was built with a different version" (I remember seeing that once) - but
>> certainly not a panic in a regular expression parser.
>>
>>>
>>> On Tue, Aug 8, 2023 at 11:12 AM Bernd Fix <b...@hoi-polloi.org> wrote:
>>>
>>>> After switching from go1.20.7 to go1.21.0 one of my applications
>>>> compiles without warnings or errors, but fails at run-time with the
>>>> following panic when loading a plugin:
>>>>
>>>> panic: regexp: Compile(`^([\w./]+)/((?:\w+)|[*])(.+)?$`): error parsing
>>>> regexp: invalid escape sequence: `\w`
>>>>
>>>> goroutine 1 [running]:
>>>> regexp.MustCompile({0x7fee026ddc4f, 0x1e})
>>>> .../golang/src/regexp/regexp.go:319 +0xb4
>>>> google.golang.org/grpc/internal/binarylog.init()
>>>>
>>>> .../ext/pkg/mod/
>>>> google.golang.org/gr...@v1.57.0/internal/binarylog/env_config.go:135

>> <http://google.golang.org/gr...@v1.57.0/internal/binarylog/env_config.go:135>

[Wojciech S. Czarnecki]

Dnia 2023-08-08, o godz. 18:11:56
Bernd Fix <b...@hoi-polloi.org> napisał(a):

> After switching from go1.20.7 to go1.21.0 one of my applications
> compiles without warnings or errors, but fails at run-time with the
> following panic when loading a plugin:

IIRC release notes tools now are version aware, so likely you need to bring
all go.mod's version to state 1.21.0

hope this helps,

> Cheers, Bernd.
>

[Olivier Szika]

Hi,

Unexpectedly, unicode.Categories is an empty map when it's only used with a plugin.

I opened a ticket: https://github.com/golang/go/issues/62430

Regards,

[Google's Affiliate (Google Business Affiliate)]

Hello, this is Jonathan, I can help you with your question about the error parsing regexp in Go.

The error you are getting is caused by the use of \w in your regular expression. This is an invalid escape sequence in Go, because \w is not a predefined character class1. Go only supports the following escape sequences in regular expressions2:

• \t for tab
• \n for newline
• \r for carriage return
• \f for form feed
• \a for alert
• \b for word boundary
• \B for non-word boundary
• \d for decimal digit
• \D for non-decimal digit
• \s for whitespace character
• \S for non-whitespace character
• \w for word character (alphanumeric plus _)
• \W for non-word character

To match a word character in Go, you need to use [[:word:]], which is equivalent to [0-9A-Za-z_]. Alternatively, you can use [[:alnum:]]_, which is equivalent to [0-9A-Za-z]_. So, your regular expression should be:

regexp.Compile(`^([[:word:]./]+)/((?:[[:word:]]+)|[*])(.+)?$`)

or

regexp.Compile(`^([[:alnum:]_./]+)/((?:[[:alnum:]_]+)|[*])(.+)?$`)

This should fix the error and allow you to load the plugin successfully.

I hope this helps you with your problem. If you have any further questions, please feel free to ask. 😊

[Brian Candler]

I'm afraid I have to contradict that answer (from an AI chatbot perhaps?)

Go's regexp library, called re2, is documented here: https://github.com/google/re2/wiki/Syntax

It *does* support the \w character class. See the section headed "Perl character classes (all ASCII-only)".  Furthermore, the OP was successfully using \w with go 1.20.7.  The problem arose with 1.21 and is something to do with plugin loading or initialization.

[Brian Candler]

I have been able to replicate the OP's problem with go 1.21.0

==> main.go <==
package main

import (
"plugin"
//"regexp"
"regexp/syntax"
)

func main() {
//_, _ = regexp.Compile(`\w+`)

p, err := plugin.Open("p1.so")
if err != nil {
panic(err)
}
s, err := p.Lookup("F")
if err != nil {
panic(err)
}

f := s.(func() (syntax.Flags, error))
_, err = f()
if err != nil {
panic(err)
}
}

==> p1/plugin.go <==
package main

import (
"regexp"
"regexp/syntax"
)

func F() (syntax.Flags, error) {
_, err := regexp.Compile(`\w+`)
return syntax.Flags(0), err
}

func main() {}

$ go version

go version go1.21.0 linux/amd64

$ go build -buildmode=plugin ./p1 && go run main.go
panic: error parsing regexp: invalid escape sequence: `\w`

goroutine 1 [running]:
main.main()
/home/ubuntu/62430/main.go:24 +0x129
exit status 2

The problem goes away if you uncomment the two commented-out lines in main.go

I had an idea about what might be happening.  In this situation, MustCompile is behaving like MustCompilePOSIX (which doesn't accept the \w character class).

Package regexp/syntax defines a constant "Perl" which includes various flags, including PerlX which allows this class:

func Compile(expr string) (*Regexp, error) {
        return compile(expr, syntax.Perl, false)
}

I'm wondering if somehow this value hasn't been initialized, and is treated as zero.  However, if I print the value of syntax.Perl within main() or F(), I see that it is 212 as expected, even when the error occurs.

[Brian Candler]

I forgot to include

==> go.mod <==
module example.com

go 1.20

[Brian Candler]

Slightly simplified version:

==> go.mod <==
module example.com

go 1.20

==> main.go <==
package main

import (
"plugin"

"regexp/syntax"
)

func main() {
if syntax.Perl != 212 {
panic("Unexpected flags")

}

p, err := plugin.Open("p1.so")
if err != nil {
panic(err)
}
s, err := p.Lookup("F")
if err != nil {
panic(err)
}

f := s.(func())
f()

}

==> p1/plugin.go <==
package main

import (
"regexp"

)

func F() {
_ = regexp.MustCompile(`\w+`)
}

func main() {}

FWIW, it also fails in the same way on macOS:

$ go build -buildmode=plugin ./p1 && go run main.go

panic: regexp: Compile(`\w+`): error parsing regexp: invalid escape sequence: `\w`

goroutine 1 [running]:
regexp.MustCompile({0x107aa6eea, 0x3})
/usr/local/go/src/regexp/regexp.go:319 +0xb4
example.com/p1.F()
/Users/brian/tmp/go/62430/p1/plugin.go:8 +0x1f
main.main()
/Users/brian/tmp/go/62430/main.go:23 +0x103
exit status 2

[Howard C. Shaw III]

It looks to me like the regexp package has an func init() that never gets called when Re is only imported in a plugin, which is why it works when you uncomment the lines that use regexp in the main package.

Howard

[Brian Candler]

But if you remove the reference to regexp/syntax in the top-level main.go, then it also works fine (with no change to the plugin code itself).