ZFS+find(1) wiring all RAM

[Peter Jeremy]

I've noticed that 11-stable/amd64 has been wiring seemingly excessive
amounts of RAM for some time (the problem goes back at least 6 months).
This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap
errors killing processes on a low-memory system. I'm not sure when it
started by it seems to hawe gotten worse between r331535 and r334494.

I can see the "excessive wired memory" on my main home system with 32GB RAM
but haven't seen it completely run out of RAM. After some gentle use and a
nightly run, there is 10GB more wired RAM than ARC.

My "low memory" system is a Google GCE f1-micro instance[*] (600MB RAM) with
about 723k inodes used and the following ZFS tuning:
vfs.zfs.arc_max="128M"
vfs.zfs.arc_meta_limit="50M"
vfs.zfs.arc_min="25M"

The following numbers were gatherer by looking at top(1). Running r334494,
after booting, to multi-user, the system has about 187MB wired (94MB ARC).
If I then run /etc/periodic/security/100.chksetuid, wired RAM increases to
about 580MB, with 380MB ARC, dropping to 467MB and 217MB ARC when the script
exits (this is still nearly twice arc_max). Free memory can drop to <10KB
whilst the find(1) is running.

I have several issues with this behaviour:
0) ARC usage can significantly exceed arc_max. I understand that arc_max is
a soft limit but IMO, 3x is unreasonable - especially when the system is
under extreme memory pressuse.
1) Significant amounts of wired memory are in use but I can't find anything
in "vmstat -mz" that would explain where it's going.

Does anyone have any suggestions for digging into this?

[1] I get the same behaviour using a VBox instance with similar dimensioning
and the same tuning)

--
Peter Jeremy

[Shane Ambler]

On 07/06/2018 16:09, Peter Jeremy wrote:
> I've noticed that 11-stable/amd64 has been wiring seemingly excessive
> amounts of RAM for some time (the problem goes back at least 6 months).
> This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap
> errors killing processes on a low-memory system. I'm not sure when it
> started by it seems to hawe gotten worse between r331535 and r334494.

Don't know if this will help you at all --

I have seen excess wired for a few years, since 10.1, I now run
11-stable, my experience has seen the severity varying over time.

I first reported this 28/10/2014 related to heavy disk use on a zpool.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194654

The inability to get solid repeatable steps to reproduce have prevented
me from chasing this more.

My desktop machine originally had 8G and when wired went over 7G there
was no choice but to reset. I now have 16G and have a script running
constantly to monitor the wired and react when needed, allocating a big
chunk of ram normally causes wired to be released, this has kept wired
under 10G for several months.

I see arc usage sitting at the arc_max setting with wired going more
than two times that.

My vm.max_wired is at 5G (default) and I think this is a limit that is
somehow bypassed, if it could be enforced we might be good. This could
be the point to chase in a bug report.

> 1) Significant amounts of wired memory are in use but I can't find anything
> in "vmstat -mz" that would explain where it's going.

I started logging my wired usage and have about 6 months of data, in the
following sample, allocating 2G more than free caused wired to drop 4G
in about 2 secs, you can also see that arc usage did not change in this
time. This leads me to think it may be related to changes made to add
support for zfs but I don't think zfs is the one allocating this wired
amount.

The values I am logging are from -
vm.max_wired
vm.stats.vm.v_wire_count
kstat.zfs.misc.arcstats.size
vm.stats.vm.v_free_count

2018/06/06_13:22:53 wired: 9779M max_wired: 5210M ARC: 4.1G Free:3.1G
2018/06/06_13:22:53 wired: 9781M max_wired: 5210M ARC: 4.1G Free:3.0G
2018/06/06_13:22:54 Excess wired detected. Allocating 4.9G to reduce wired.
2018/06/06_13:22:54 wired: 9787M max_wired: 5210M ARC: 4.1G Free:3.0G
2018/06/06_13:22:54 wired: 9796M max_wired: 5210M ARC: 4.1G Free:1.9G
2018/06/06_13:22:55 wired: 9809M max_wired: 5210M ARC: 4.1G Free:389.6M
2018/06/06_13:22:55 wired: 8169M max_wired: 5210M ARC: 4.1G Free:1.2G
2018/06/06_13:22:55 wired: 5917M max_wired: 5210M ARC: 4.1G Free:2.6G
2018/06/06_13:22:55 wired: 5566M max_wired: 5210M ARC: 4.1G Free:2.2G


--

Shane Ambler
Shane (at) ShaneWare (dot) Biz

http://ShaneWare.Biz
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

[Slawa Olhovchenkov]

On Thu, Jun 07, 2018 at 07:04:29PM +0930, Shane Ambler wrote:

> On 07/06/2018 16:09, Peter Jeremy wrote:
> > I've noticed that 11-stable/amd64 has been wiring seemingly excessive
> > amounts of RAM for some time (the problem goes back at least 6 months).
> > This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap
> > errors killing processes on a low-memory system. I'm not sure when it
> > started by it seems to hawe gotten worse between r331535 and r334494.
>
> Don't know if this will help you at all --
>
> I have seen excess wired for a few years, since 10.1, I now run
> 11-stable, my experience has seen the severity varying over time.
>
> I first reported this 28/10/2014 related to heavy disk use on a zpool.
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194654
>
> The inability to get solid repeatable steps to reproduce have prevented
> me from chasing this more.

Can you try https://reviews.freebsd.org/D7538 ?
I am try in this patch to resolve trouble w/ wired and unused memory
by ARC.