Seeking for help on frequent kernel dumps possibly related to pf/connection tracking

[tobias.we...@skyline.link38.eu]

Hello HardenedBSD folks,

on two HardenedBSD VMs of mine, which are constantly under high network load, I am currently observing
frequent hiccups, when HBSD decides to dump its kernel onto disk.

Not being an expert on BSD kernel debugging/troubleshooting at all, the "Panic Strings" left in the
/var/crashes/info.* files look like this has something to do with pf in general or connection tracking
in particular - two other HBSD VMs handling significantly less traffic and connections are running stable.

root@x:/var/crash # cat info.1 info.last
Dump header from device: /dev/da0s1b
Architecture: amd64
Architecture Version: 2
Dump Length: 302120960
Blocksize: 512
Compression: none
Dumptime: 2021-08-05 18:05:28 +0200
Hostname: x
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 13.0-STABLE-HBSD #0 : Wed Jul 28 20:30:12 UTC 2021
ro...@ci-12.md.hardenedbsd.lan:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD
Panic String: pf_free_state: timeout 0 <<<<<
Dump Parity: 1024873034
Bounds: 1
Dump Status: good
Dump header from device: /dev/da0s1b
Architecture: amd64
Architecture Version: 2
Dump Length: 259280896
Blocksize: 512
Compression: none
Dumptime: 2021-08-06 11:35:13 +0200
Hostname: x
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 13.0-STABLE-HBSD #0 : Wed Jul 28 20:30:12 UTC 2021
ro...@ci-12.md.hardenedbsd.lan:/usr/obj/usr/src/amd64.amd64/sys/HARDENEDBSD
Panic String: tcp_hostcache: bucket length too high at 51: 30 <<<<<
Dump Parity: 1299402335
Bounds: 4
Dump Status: good

The latter leads to https://cgit.freebsd.org/src/commit/?id=d554522f6e687a365ebe935010298024fa2c1c9d, but
kind of leaves me unsure on what to do about it. Due to the massive amount of connections, I increased
pf's default by setting

> set limit states 1000000

in /etc/pf.conf (this was especially necessary for UDP). My first, uneducated guess would be to comment
out this directive, risking dropped connections but getting a stable system on the other hand.

Am I overlooking something? Is somebody else experiencing this on recent HBSD versions?

Thanks in advance,
Tobias

[Shawn Webb]

Hey Tobias,

I haven't experienced any pf-related crashes. Granted, my firewall's
running kernel isn't as up-to-date as yours. I've CC'd Gleb on this
email to see if he has any suggestions.

My firewall's `uname -a`: FreeBSD firewall-01 13.0-STABLE-HBSD FreeBSD
13.0-STABLE-HBSD #76 hardened/13-stable/master-n190237-4b6e6b2446c:
Thu May 13 11:42:17 EDT 2021
ro...@ci-01.md.hardenedbsd.org:/usr/obj/src/13-stable/amd64.amd64/sys/HARDENEDBSD
amd64

Thanks,

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc

[Gleb Smirnoff]

Tobias,

On Sat, Aug 07, 2021 at 09:25:31AM -0400, Shawn Webb wrote:
S> > Panic String: tcp_hostcache: bucket length too high at 51: 30 <<<<<
S> > Dump Parity: 1299402335
S> > Bounds: 4
S> > Dump Status: good
S> >
S> > The latter leads to https://cgit.freebsd.org/src/commit/?id=d554522f6e687a365ebe935010298024fa2c1c9d, but
S> > kind of leaves me unsure on what to do about it. Due to the massive amount of connections, I increased
S> > pf's default by setting
S> >
S> > > set limit states 1000000
S> >
S> > in /etc/pf.conf (this was especially necessary for UDP). My first, uneducated guess would be to comment
S> > out this directive, risking dropped connections but getting a stable system on the other hand.
S> >
S> > Am I overlooking something? Is somebody else experiencing this on recent HBSD versions?
S>
S> Hey Tobias,
S>
S> I haven't experienced any pf-related crashes. Granted, my firewall's
S> running kernel isn't as up-to-date as yours. I've CC'd Gleb on this
S> email to see if he has any suggestions.

This isn't related to pf, but it is related to my change d554522f6e687a365ebe935010298024fa2c1c9d.

Can I download the kernel dump + kernel?

--
Gleb Smirnoff