Kernel Panic, trying to find cause: Hardened BSD 15 and latest Hardened BSD 14.2

[B.C. Cotman]

Hello,

I have been trying to identify the cause of a few Hardened BSD servers
deciding to suddenly, ungracefully "reset" (reboot.)

We have a few Hardened BSD servers. Some on 14.1, others on 14.2 and
some on latest 15.

For public-facing server, we have observed this unexpected "reset."
However, for internal-only services, we have not seen this. (Example:
internal DB server, internal mail : working fine.)
The reset seems to be related to how much public traffic the servers see.
A Simple web server that is busy suffers resets more often than a less
busy web server that supports PHP.

When there is a panic, nothing is logged to the Filesystems.

On one server, where it was happening often enough, we started
recording video using BMC to capture this on-console (only) message:

############################################
Panic: sacked_bytes < 0
Cupid = 2
Time = 1742974752
__HardenedBSD_version = 1500001 _FreeBSD_version = 1500034 Version =
FreeBSD 15.0-CURRENT-HBSD #0
hardened/current/master-n194722-d386fe55efbd: Mon Mar 24 18:28:18 PDT
2025
Root2@$SERVER_HOSTNAME:$SRC_PATH/amd64/sys/$KERNEL_CONFIG_FILE
Uptime: 11h56m50s
Automatic reboot in 15 seconds - press any key on console to abort
############################################

This panic seems related to kernel network code.

We have not captured the console message at time of reset on other
servers suffering this yet, but someone else at work says he observed
this same Panic error when watching console on a different server as
it reset.

This recent discussion seems related and is fairly recent:
https://reviews.freebsd.org/D48652

Do any of you have any suggestions you all can provide on steps to
take on production servers to get more details about panic preferably
logged somewhere? Remote logging by kernel at panic?

Thanks!

[B.C. Cotman]

After disabling SACK, a kernel panic happened again, but moved to:

##############################################
Panic: tcp_do_segment:sent too much
Cupid = 3
Time = 1744102152

__HardenedBSD_version = 1500001 _FreeBSD_version = 1500034

version = FreeBSD 15.0-CURRENT-HBSD #1
hardened/current/master-n194722-d386fe55efbd: Mon Apr 7 03:34:30 PDT
2025
root2@$SERVER_HOSTNAME:$SRC_PATH/amd64/sys/$KERNEL_CONFIG_FILE
Uptime: 34m27s
##############################################

It appears this is a known issue with freebsd TCP stack since late last year:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282605

After switching to a different TCP stack, the problem has not returned
in over 24 hours while it has happened on another public facing server
which is still using the default FreeBSD TCP stack, since then.

HTH

[Shawn Webb]

Out of curiosity, which TCP stack did you move to?

--
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Signal Username: shawn_webb.74
Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc

[B.C. Cotman]

Sent reply with details, off-list.