practical experience of FreeBSD > 1TiB RAM

[Dave Cottlehuber]

hi,

Is anybody using production FreeBSD systems with 1-2 TiB+ RAM?
This is larger than most FreeBSD systems I've dealt with personally.

I'm interested in any aspects of tuning & performance you might have
encountered.

Workload is PostgreSQL on ZFS, possibly with co-located applications.

If you have info that can't be shared publically, feel free to
contact me directly.

A+
Dave

[Alan Somers]

Here's a few changes we've made for our servers that have up to 1 TB of RAM:

* Adjust zfs.vfs.arc_max to 2/3 of the total. But there wasn't much
science behind that decision, and I'm far from convinced that it's the
best possible value.
* freebsd-src commit 24938f9311c lib/libc/rpc: switch the per-fd
structs in clnt_{dg,vc}.c to RB Tree . Before this change, every sshd
process (among others) would consume 1GB of RAM if NIS was in use.
* Python formerly did a loop from 0 to kern.maxfilesperproc on every
exec. That would make exec very slow with large amounts of RAM.
Lowering kern.maxfilesperproc proved necessary until
https://github.com/python/cpython/commit/162c567d164b5742c0d1f3f8bd8c8bab9c117cd0
.
* With this much RAM, making a core dump onto spinning media takes too
much time. So we switched to textdumps for all servers with spinning
boot disks.
* debug.ddb.capture.bufsize is too small for textdumps with this much
RAM. We raised it to 5242880.
* We set vfs.fusefs.data_cache_mode=0, but ONLY for the sake of fuse
file systems that use libfuse-2 and are also exported via iSCSI (or
mmaped). Without that setting, they can consume far too much RAM.
And file systems that use libfuse-3 (or that don't use libfuse) have a
different way to specify their cache mode.