Kernel panic on 12.2-STABLE-HBSD

[Théo BERTIN]

Hi everyone,

We're currently experiencing kernel panics and reboots on several machines updated recently :

/var/crash/info.0
Dump header from device: /dev/da1
 Architecture: amd64
 Architecture Version: 2
 Dump Length: 626298880
 Blocksize: 512
 Compression: none
 Dumptime: Tue Aug 31 14:22:24 2021
 Hostname: [redacted]
 Magic: FreeBSD Kernel Dump
 Version String: FreeBSD 12.2-STABLE-HBSD #0 : Tue Aug 10 20:14:33 UTC 2021
   ro...@ci-12.md.hardenedbsd.lan:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD
 Panic String: pf_free_state: timeout 0
 Dump Parity: 436929362
 Bounds: 0
 Dump Status: good

/var/log/messages :

Sep  1 05:23:08 [redacted] syslogd: kernel boot file is /boot/kernel/kernel
Sep  1 05:23:08 [redacted] kernel: [4048] panic: pf_free_state: timeout 0
Sep  1 05:23:08 [redacted] kernel: [4048] cpuid = 2
Sep  1 05:23:08 [redacted] kernel: [4048] time = 1630473734
Sep  1 05:23:08 [redacted] kernel: [4048] __HardenedBSD_version = 1200060 __FreeBSD_version = 1202508
Sep  1 05:23:08 [redacted] kernel: [4048] version = FreeBSD 12.2-STABLE-HBSD #0 : Tue Aug 10 20:14:33 UTC 2021
Sep  1 05:23:08 [redacted] kernel: [4048]     ro...@ci-12.md.hardenedbsd.lan:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD
Sep  1 05:23:08 [redacted] kernel: [4048] KDB: stack backtrace:
Sep  1 05:23:08 [redacted] kernel: [4048] #0 0xffffffff80b9f12b at kdb_backtrace+0x6b
Sep  1 05:23:08 [redacted] kernel: [4048] #1 0xffffffff80b581e0 at vpanic+0x180
Sep  1 05:23:08 [redacted] kernel: [4048] #2 0xffffffff80b57fe3 at panic+0x43
Sep  1 05:23:08 [redacted] kernel: [4048] #3 0xffffffff82923622 at pf_free_state+0xb2
Sep  1 05:23:08 [redacted] kernel: [4048] #4 0xffffffff8292cdde at pf_test_rule+0x312e
Sep  1 05:23:08 [redacted] kernel: [4048] #5 0xffffffff82930282 at pf_test6+0x772
Sep  1 05:23:08 [redacted] kernel: [4048] #6 0xffffffff8293abe9 at pf_check6_out+0x59
Sep  1 05:23:08 [redacted] kernel: [4048] #7 0xffffffff80c7095a at pfil_run_hooks+0xaa
Sep  1 05:23:08 [redacted] kernel: [4048] #8 0xffffffff80da192f at ip6_output+0x15af
Sep  1 05:23:08 [redacted] kernel: [4048] #9 0xffffffff80d67317 at tcp_output+0x1d37
Sep  1 05:23:08 [redacted] kernel: [4048] #10 0xffffffff80d7acb0 at tcp6_usr_connect+0x2f0
Sep  1 05:23:08 [redacted] kernel: [4048] #11 0xffffffff80be75fc at soconnectat+0xdc
Sep  1 05:23:08 [redacted] kernel: [4048] #12 0xffffffff80beea8e at kern_connectat+0xfe
Sep  1 05:23:08 [redacted] kernel: [4048] #13 0xffffffff80bee965 at sys_connect+0x75
Sep  1 05:23:08 [redacted] kernel: [4048] #14 0xffffffff81023556 at amd64_syscall+0x2b6
Sep  1 05:23:08 [redacted] kernel: [4048] #15 0xffffffff80ffa99e at fast_syscall_common+0xf8
Sep  1 05:23:08 [redacted] kernel: [4048] Uptime: 1h7m28s
Sep  1 05:23:08 [redacted] kernel: [4048] Dumping 580 out of 4056 MB:..3%..12%..23%..31%..42%..53%..61%..72%..83%..91%
Sep  1 05:23:08 [redacted] kernel: [4048] Dump complete
Sep  1 05:23:08 [redacted] kernel: [4048] Automatic reboot in 15 seconds - press a key on the console to abort

uname -a :

FreeBSD [redacted] 12.2-STABLE-HBSD FreeBSD 12.2-STABLE-HBSD #0 : Tue Aug 10 20:14:33 UTC 2021     ro...@ci-12.md.hardenedbsd.lan:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD  amd64

hbsd-update -C :

[+] Local version: hbsd-v1200060-41a52376cfffabfd43667d4f7ed01fc72959d2f7
[+] Remote version: hbsd-v1200060-1969e37a92d528ce30ddf9e5b19481127532e4cd sha256 39f5a3a823c2431117599256b441deb18c1f7698f320b103ea9e308c7c55322d

As I said, several machines have been reported to reboot following that same error, in them are several levels of update hashes (but always the same HBSD version 12.2-STABLE) using 'hbsd-update -C' :

-

hbsd-v1200060-1969e37a92d528ce30ddf9e5b19481127532e4cd (latest update to this day)

-

hbsd-v1200060-4fc0cb9290b0c219e2f981222d98a1fdc8eba97e

-

hbsd-v1200060-41a52376cfffabfd43667d4f7ed01fc72959d2f7

This issue is possibly linked to the existing conversation here : https://groups.google.com/a/hardenedbsd.org/g/users/c/i3OT7wcH8nQ/m/XU4LAFGFBwAJ
But the version is different (13.0 instead of 12.2)

All machines are virtualized, under several different engines and infrastructures.

It would seem some machines with the latest update patch (or close to it) and not under heavy network load are not impacted by this problem.

[Uwe Trenkner]

Hi,

sorry this will not help you, but just for information: I use the exact same version on a production server and experience nothing unusual. So it is not a problem affecting all users.

Best regards

Uwe

--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@hardenedbsd.org.

[Shawn Webb]

On Wed, Sep 01, 2021 at 01:19:15AM -0700, Théo BERTIN wrote:
>
> Hi everyone,
>
> We're currently experiencing kernel panics and reboots on several machines
> updated recently :

> */var/crash/info.0*

> Dump header from device: /dev/da1
> Architecture: amd64
> Architecture Version: 2
> Dump Length: 626298880
> Blocksize: 512
> Compression: none
> Dumptime: Tue Aug 31 14:22:24 2021
> Hostname: [redacted]
> Magic: FreeBSD Kernel Dump
> Version String: FreeBSD 12.2-STABLE-HBSD #0 : Tue Aug 10 20:14:33 UTC 2021
> ro...@ci-12.md.hardenedbsd.lan:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD
>
> Panic String: pf_free_state: timeout 0
> Dump Parity: 436929362
> Bounds: 0
> Dump Status: good
>

> */var/log/messages :*

> *uname -a :*

> FreeBSD [redacted] 12.2-STABLE-HBSD FreeBSD 12.2-STABLE-HBSD #0 : Tue Aug
> 10 20:14:33 UTC 2021
> ro...@ci-12.md.hardenedbsd.lan:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD
> amd64
>
>

> *hbsd-update -C : *

> [+] Local version: hbsd-v1200060-41a52376cfffabfd43667d4f7ed01fc72959d2f7
> [+] Remote version: hbsd-v1200060-1969e37a92d528ce30ddf9e5b19481127532e4cd
> sha256 39f5a3a823c2431117599256b441deb18c1f7698f320b103ea9e308c7c55322d
>
> As I said, several machines have been reported to reboot following that
> same error, in them are several levels of update hashes (but always the
> same HBSD version 12.2-STABLE) using 'hbsd-update -C' :
> -
> hbsd-v1200060-1969e37a92d528ce30ddf9e5b19481127532e4cd (latest update to
> this day)
> -
> hbsd-v1200060-4fc0cb9290b0c219e2f981222d98a1fdc8eba97e
> -
> hbsd-v1200060-41a52376cfffabfd43667d4f7ed01fc72959d2f7
>
>
> This issue is possibly linked to the existing conversation here :
> https://groups.google.com/a/hardenedbsd.org/g/users/c/i3OT7wcH8nQ/m/XU4LAFGFBwAJ
> But the version is different (13.0 instead of 12.2)
>
> All machines are virtualized, under several different engines and
> infrastructures.
> It would seem some machines with the latest update patch (or close to it)
> and not under heavy network load are not impacted by this problem.
>

From that panic string and the backtrace, it seems like the problem is
in pf's handling of outbound ipv6 tcp packets, specifically when
trying to connect to the remote host.

I'm wholly unfamiliar with pf's code, and HardenedBSD doesn't have any
changes that would impact pf. You might want to consider filing a bug
with FreeBSD.

Thanks,


--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc

[Théo BERTIN]

Thanks for the quick reply. This looks indeed to be a problem with IPv6, as all the concerned machines are using IPv6 stacks.

For future reference, a bug has been submitted to FreeBSD at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258196

[Théo BERTIN]

After some analysis on recent changes to PF's code and impacted versions of HardenedBSD, it seems problem should be located between commits bc6cf5a56 and 98b05b41e.

However, some later commits to pf are not yet included in a HardenedBSD patch, notably commits ff54f3762, 253d1f4e3, 4562d33c8, a37c697b8 (which looks very related to our actual problem), 21d8ee0cd, 4ef480e60 and e23d47cd9.

I would very much like to test a patch containing at least commits through a37c697b8 on our systems to determine if they could fix the current bug, can someone tell me when the next patch releases is scheduled for 12-stable ?

[Loic F]

Le mer. 22 sept. 2021 à 15:27, Théo BERTIN <berti...@gmail.com> a écrit :

After some analysis on recent changes to PF's code and impacted versions of HardenedBSD, it seems problem should be located between commits bc6cf5a56 and 98b05b41e.

Can you confirm that the problem does not affect Freebsd?

Have you tried to recompile the kernel by reversing the suspected commits?

 

However, some later commits to pf are not yet included in a HardenedBSD patch, notably commits ff54f3762, 253d1f4e3, 4562d33c8, a37c697b8 (which looks very related to our actual problem), 21d8ee0cd, 4ef480e60 and e23d47cd9.

I would very much like to test a patch containing at least commits through a37c697b8 on our systems to determine if they could fix the current bug, can someone tell me when the next patch releases is scheduled for 12-stable ?

You can download the latest kernel [1] to extract it to /boot/kernel-latest and select this at reboot.

[1] https://ci-01.nyi.hardenedbsd.org/pub/hardenedbsd/12-stable/amd64/amd64/BUILD-LATEST/kernel.txz

Best regards,

--

Loic
dev team
HardenedBSD

[Shawn Webb]

The good news is that my home firewall crashed with the same panic
string. I'm going to be busy over the next couple weeks, but
afterwards I'll take a look.

[Shawn Webb]

It looks like the problem might've been fixed by FreeBSD[0]. After our
next auto-sync, I'll kick off a new binary update and a package build.

[0]: https://cgit.freebsd.org/src/commit/?id=a0c64a443e4cae67a5eea3a61a47d746866de3ee

[Théo BERTIN]

Thanks for the new patch, the latest code from FreeBSD seems to have resolved the problem