Freebsd 8.0 kmem map too small
Giulio Ferro
This server has been running for several months without problems.
Beginning last week, however, I'm experiencing panics (about 1 per day)
with the error in the subject
Current settings:
vm.kmem_size_scale: 3
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 2764046336
..
hw.physmem: 8568225792
hw.usermem: 6117404672
hw.realmem: 9395240960
..
vfs.zfs.l2arc_noprefetch: 0
vfs.zfs.l2arc_feed_secs_shift: 1
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 128
vfs.zfs.l2arc_write_boost: 67108864
vfs.zfs.l2arc_write_max: 67108864
vfs.zfs.arc_meta_limit: 431882240
vfs.zfs.arc_meta_used: 431874720
vfs.zfs.arc_min: 215941120
vfs.zfs.arc_max: 1727528960
I've set nothing in either /boot/loader.conf or sysctl.conf
What should I do?
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
Jeremy Chadwick
> NFS server amd64 Freebsd 8.0 recent (2 days ago)
>
> This server has been running for several months without problems.
> Beginning last week, however, I'm experiencing panics (about 1 per day)
> with the error in the subject
>
> Current settings:
>
>
> vm.kmem_size_scale: 3
> vm.kmem_size_max: 329853485875
> vm.kmem_size_min: 0
> vm.kmem_size: 2764046336
> vfs.zfs.l2arc_noprefetch: 0
> vfs.zfs.l2arc_feed_secs_shift: 1
> vfs.zfs.l2arc_feed_secs: 1
> vfs.zfs.l2arc_headroom: 128
> vfs.zfs.l2arc_write_boost: 67108864
> vfs.zfs.l2arc_write_max: 67108864
> vfs.zfs.arc_meta_limit: 431882240
> vfs.zfs.arc_meta_used: 431874720
> vfs.zfs.arc_min: 215941120
> vfs.zfs.arc_max: 1727528960
>
>
> I've set nothing in either /boot/loader.conf or sysctl.conf
>
>
> What should I do?
You need to adjust vm.kmem_size to provide more space for the ARC.
Below are ZFS-relevant entries in our /boot/loader.conf on production
RELENG_8 systems with 8GB of RAM. The reason we set kmem_size to half
our physical system memory is because I didn't want to risk other
processes which use a larger maxdsiz/dfldsiz/maxssiz to potentially
exhaust all memory.
# Increase vm.kmem_size to allow for ZFS ARC to utilise more memory.
vm.kmem_size="4096M"
vfs.zfs.arc_max="3584M"
# Disable ZFS prefetching
# http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html
# Increases overall speed of ZFS, but when disk flushing/writes occur,
# system is less responsive (due to extreme disk I/O).
# NOTE: 8.0-RC1 disables this by default on systems <= 4GB RAM anyway
# NOTE: System has 8GB of RAM, so prefetch would be enabled by default.
vfs.zfs.prefetch_disable="1"
# Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
vfs.zfs.txg.timeout="5"
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
Giulio Ferro
Thanks, I'll try these settings.
I'll keep you posted.
Giulio Ferro
> Thanks, I'll try these settings.
>
> I'll keep you posted.
Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
I'm really astounded at how unstable zfs is, it's causing me a lot of
problem.
Why isn't it stated in the handbook that zfs isn't up to production yet?
Jeremy Chadwick
> Giulio Ferro wrote:
> >Thanks, I'll try these settings.
> >
> >I'll keep you posted.
>
> Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to
something *less* than vm.kmem_size?
> I'm really astounded at how unstable zfs is, it's causing me a lot
> of problem.
>
> Why isn't it stated in the handbook that zfs isn't up to production yet?
I'm not at liberty to comment + answer this question.
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
_______________________________________________
Giulio Ferro
Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
> Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to
> something *less* than vm.kmem_size?
>
>
Yes.
After your suggestion, I set
vfs.zfs.arc_max: 3758096384
vm.kmem_size: 4G
Now:
vfs.zfs.arc_max: 3758096384
vm.kmem_size: 6392119296
>> I'm really astounded at how unstable zfs is, it's causing me a lot
>> of problem.
>>
>> Why isn't it stated in the handbook that zfs isn't up to production yet?
>>
> I'm not at liberty to comment + answer this question.
>
>
Why people responsible for ZFS on freebsd aren't saying anything?
What's the status of this issue? Is someone working on this???
Giulio Ferro
> ----- Original Message ----
>
>>> I'm really astounded at how unstable zfs is, it's
>>>
>> causing me a lot
>>
>>> of problem.
>>>
>>> Why isn't it
>>>
>> stated in the handbook that zfs isn't up to production yet?
>>
>>>
>>>
>> freebsd aren't saying anything? What's the status of this issue? Is someone
>> working on this???
>>
>
>
>
This is the first post of the current thread: it should tell you everything
--------------------------------------------------------------------
Current settings:
What should I do?
--------------------------------------------------------------------
Pawel Jakub Dawidek
> On 05.05.2010 09:52, Jeremy Chadwick wrote:
>
> Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
>
>
> >Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to
> >something *less* than vm.kmem_size?
> >
> >
>
> Yes.
> After your suggestion, I set
> vfs.zfs.arc_max: 3758096384
> vm.kmem_size: 4G
>
> Now:
> vfs.zfs.arc_max: 3758096384
> vm.kmem_size: 6392119296
Could you try to track down the commit that is causing your problems?
Could you try 8-STABLE kernel from before r206815?
--
Pawel Jakub Dawidek http://www.wheelsystems.com
p...@FreeBSD.org http://www.FreeBSD.org
FreeBSD committer Am I Evil? Yes, I Am!
Ben Kelly
On May 5, 2010, at 9:33 AM, Pawel Jakub Dawidek wrote:
> On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote:
>> On 05.05.2010 09:52, Jeremy Chadwick wrote:
>>
>> Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
>>
>>
>>> Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to
>>> something *less* than vm.kmem_size?
>>>
>>>
>>
>> Yes.
>> After your suggestion, I set
>> vfs.zfs.arc_max: 3758096384
>> vm.kmem_size: 4G
>>
>> Now:
>> vfs.zfs.arc_max: 3758096384
>> vm.kmem_size: 6392119296
>
> Could you try to track down the commit that is causing your problems?
> Could you try 8-STABLE kernel from before r206815?
Are others generally able to run ARC values so close to kmem size? My experience has been that you really need the ARC to be much smaller than the kmem limit (like 25% or less) due to fragmentation of kmem_map.
- Ben_______________________________________________
Steve Polyack
> On May 5, 2010, at 9:33 AM, Pawel Jakub Dawidek wrote:
>
>
>> On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote:
>>
>>> On 05.05.2010 09:52, Jeremy Chadwick wrote:
>>>
>>> Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
>>>
>>>
>>>
>>>> Did you set both vm.kmem_size and vfs.zfs.arc_max, setting the latter to
>>>> something *less* than vm.kmem_size?
>>>>
>>>>
>>>>
>>> Yes.
>>> After your suggestion, I set
>>> vfs.zfs.arc_max: 3758096384
>>> vm.kmem_size: 4G
>>>
>>> Now:
>>> vfs.zfs.arc_max: 3758096384
>>> vm.kmem_size: 6392119296
>>>
>> Could you try to track down the commit that is causing your problems?
>> Could you try 8-STABLE kernel from before r206815?
>>
>
> Are others generally able to run ARC values so close to kmem size? My experience has been that you really need the ARC to be much smaller than the kmem limit (like 25% or less) due to fragmentation of kmem_map.
>
>
multiple zdevs, little tuning was needed. The system is running
8-STABLE(amd64) as of a month or two ago. The only things I have set in
/boot/loader.conf are:
vm.kmem_size="12G"
vfs.zfs.arc_max="4G"
Setting kmem_size to 12G came from a combination of recommendations I
saw in various mailing list posts (you can probably dig them up).
Setting it to the physical memory size was the initial recommendation,
while others recommended 1.5x physical memory size to help prevent
fragmentation/wasted space in kmem.
Regardless, this has served us quite well for the ~6 months the system
has been in use. It has never crashed, even under intensive
multi-threaded benchmarking.
-Steve
Freddie Cash
> Giulio Ferro wrote:
>
>> Thanks, I'll try these settings.
>>
>> I'll keep you posted.
>>
>
> Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
> I'm really astounded at how unstable zfs is, it's causing me a lot of
> problem.
> Why isn't it stated in the handbook that zfs isn't up to production yet?
>
As with everything related to computers, it all depends on your uses.
We've been using FreeBSD+ZFS since it first hit 7.0 with very few problems.
The first month of use required a lot of tuning and testing, though, to find
the correct kmem, arc, etc settings for our workload. Once we found them,
though, things have been rockstable.
We rsync 120 remote Linux and FreeBSD servers to our 24-drive, 18 TB backup
server, every night. And then rsync that to another similar server. And
share out a couple of filesytems via NFS and another couple via Samba.
Expect to spend a week or two to tune ZFS, and to rebuild the pool at least
once to find the optimal vdev layout. But once that's done, expect it to be
solid.
--
Freddie Cash
fjw...@gmail.com
Mike Andrews
> On Tue, May 4, 2010 at 11:32 PM, Giulio Ferro<au...@zirakzigil.org> wrote:
>
>> Giulio Ferro wrote:
>>
>>> Thanks, I'll try these settings.
>>>
>>> I'll keep you posted.
>>>
>>
>> Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
>> I'm really astounded at how unstable zfs is, it's causing me a lot of
>> problem.
>> Why isn't it stated in the handbook that zfs isn't up to production yet?
>>
>
> As with everything related to computers, it all depends on your uses.
Sorry to semi-hijack this, but... I'm also running into frequent
"kmem_map too small" panics on 8-STABLE, such as:
panic: kmem_malloc(131072): kmem_map too small: 2023780352 total allocated
panic: kmem_malloc(131072): kmem_map too small: 2011525120 total allocated
panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated
panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated
panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated
panic: kmem_malloc(131072): kmem_map too small: 2020409344 total allocated
panic: kmem_malloc(536576): kmem_map too small: 2022957056 total allocated
(those are over the course of 3-4 days)
On this specific system, it has 32 GB physical memory and has
vfs.zfs.arc_max="2G" and vm.kmem_size="64G" in /boot/loader.conf. The
latter was added per earlier suggestions on this list, but appears to be
ignored as "sysctl vm.kmem_size" returns about 2 GB (2172452864) anyway.
Unfortunately I have not yet found a way to reliably reproduce the panic
on demand, so it's hard to iteratively narrow down which date the
potentially offending commit was on. It happened at least twice
overnight, where several memory-intensive jobs run. Backing out to a
previous 8-STABLE kernel from March 28 appears (so far) to have
stabilized things, so the offending code was likely somewhere between
then and May 5 or so. I do have KDB/DDB and serial console on this box,
if there's any more info I can give to help troubleshoot other than just
a one-month vague date range :)
This is happening to more than just one system, but I figure it's easier
to troubleshoot memory settings on the 32 GB i7 server instead of the 2
GB Atom one...
Jeremy Chadwick
> Sorry to semi-hijack this, but... I'm also running into frequent
> "kmem_map too small" panics on 8-STABLE, such as:
>
> panic: kmem_malloc(131072): kmem_map too small: 2023780352 total allocated
> panic: kmem_malloc(131072): kmem_map too small: 2011525120 total allocated
> panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated
> panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated
> panic: kmem_malloc(114688): kmem_map too small: 1849356288 total allocated
> panic: kmem_malloc(131072): kmem_map too small: 2020409344 total allocated
> panic: kmem_malloc(536576): kmem_map too small: 2022957056 total allocated
>
> (those are over the course of 3-4 days)
>
> On this specific system, it has 32 GB physical memory and has
> vfs.zfs.arc_max="2G" and vm.kmem_size="64G" in /boot/loader.conf.
> The latter was added per earlier suggestions on this list, but
> appears to be ignored as "sysctl vm.kmem_size" returns about 2 GB
> (2172452864) anyway.
>
In your case, vm.kmem_size being stuck at 2GB (despite you setting it
larger) is the problem. A workaround would be for you to set
vfs.zfs.arc_max to something smaller, such as 1G or 512M.
Can you please provide output from the following 5 commands:
uname -a
sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale
sysctl hw.physmem hw.usermem hw.realmem
sysctl hw.machine_arch
sysctl hw.pagesize hw.pagesizes hw.availpages
Thanks.
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
_______________________________________________
Artem Belevich
> vfs.zfs.arc_max="2G" and vm.kmem_size="64G" in /boot/loader.conf. �The
> latter was added per earlier suggestions on this list, but appears to be
> ignored as "sysctl vm.kmem_size" returns about 2 GB (2172452864) anyway.
Set vm.kmem_size to slightly below 2x the amount of physical memory
your kernel *sees* (sysctl hw.physmem) . Chances are that real amount
of physical memory available to kernel is slightly below 32G so your
tunable is ignored. My guess would be that vm.kmem_size=63G would work
much better.
--Artem
Steve Polyack
slightly under 2x the physical memory. The kernel will default to 2GB
if you pass this limit. 1.5x physical memory size should be sufficient,
so try "48G" and verify that it gets set correctly on the next boot.
Mike Andrews
OK, I've got vm.kmem_size set a bit lower and it now accepts it. It's
still not clear why this just recently (April?) became necessary to do at
all :)
Meanwhile, I'll see if things get more stable now...
Artem Belevich
What's changed recently is that ZFS ARC now uses UMA for its memory
allocations. If I understand it correctly, this would make ARC's
memory use more efficient as allocated chunks will end up in a zone
tuned for allocations of particular size.
Increased fragmentation could be the side effect of this change, but
I'm guessing here.
--Artem
Richard Perini
> On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote:
> > On 05.05.2010 09:52, Jeremy Chadwick wrote:
> >
> > Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
> >
[ ... ]
> Could you try to track down the commit that is causing your problems?
> Could you try 8-STABLE kernel from before r206815?
A quick note to say "same here", but on i386.
FreeBSD 8.0-STABLE as of 8/5/2010 paniced last night with same symptoms,
approx 48 hours uptime.
Previous kernel was FreeBSD 8.0-STABLE from Sun Mar 7 14:31:45 EST 2010,
perfectly stable for intervening 2 months, about 2 months uptime.
Please let me know if full details would help (as opposed to just adding noise :-)
--
Richard Perini Internet: r...@ci.com.au
Corinthian Engineering Pty Ltd PHONE: +61 2 9552 5500
Sydney, Australia FAX: +61 2 9552 5549
Artem Belevich
Comment out following line in that file:
CFLAGS+=-DZIO_USE_UMA
This should revert memory allocation method back to its previous mode.
Let us know whether it helps or not.
--Artem
On Mon, May 10, 2010 at 4:14 PM, Richard Perini <r...@ci.com.au> wrote:
> On Wed, May 05, 2010 at 03:33:02PM +0200, Pawel Jakub Dawidek wrote:
>> On Wed, May 05, 2010 at 10:46:31AM +0200, Giulio Ferro wrote:
>> > On 05.05.2010 09:52, Jeremy Chadwick wrote:
>> >
>> > Nope, it's happened again... Now I've tried to rise vm.kmem_size to 6G...
>> >
>
> [ ... ]
>
>> Could you try to track down the commit that is causing your problems?
>> Could you try 8-STABLE kernel from before r206815?
>
> A quick note to say "same here", but on i386.
>
> FreeBSD 8.0-STABLE as of 8/5/2010 paniced last night with same symptoms,
> approx 48 hours uptime.
>
> Previous kernel was FreeBSD 8.0-STABLE from Sun Mar �7 14:31:45 EST 2010,
> perfectly stable for intervening 2 months, about 2 months uptime.
>
> Please let me know if full details would help (as opposed to just adding noise :-)
>
> --
> Richard Perini � � � � � � � � � � � � � � � � � � � Internet: �r...@ci.com.au
> Corinthian Engineering Pty Ltd � � � � � � � � � � � PHONE: � +61 2 9552 5500
> Sydney, Australia � � � � � � � � � � � � � � � � � �FAX: � � +61 2 9552 5549
> _______________________________________________
> freeb...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"
Richard Perini
> You can try disabling ZIO_USE_UMA in sys/modules/zfs/Makefile
>
> Comment out following line in that file:
> CFLAGS+=-DZIO_USE_UMA
>
> This should revert memory allocation method back to its previous mode.
> Let us know whether it helps or not.
Thanks Artem, I'll try the suggestion and report back. I'll give it 72 hours.
The workload of the machine is fairly consistent day to day.
Cheers,
--
Richard Perini Internet: r...@ci.com.au
Corinthian Engineering Pty Ltd PHONE: +61 2 9552 5500
Sydney, Australia FAX: +61 2 9552 5549
_______________________________________________
Yoshiaki Kasahara
I'd like to add another instance of similar problems. I recently
updated my FreeBSD amd64 box with ZFS root and 8GB RAM from 8-STABLE
(as of Mar 1st) to 8.1-PRERELEASE (as of May 27th). After that, my
box started to crash every couple of days due to kmem_map too small.
Here is a (last week) screenshot of Munin graph about the memory usage
of the box:
http://eron.info/munin-memory.png
In "by month" graph, a white gap at the end of "Week 20" is the update
period from 8-STABLE to 8.1-PRERELEASE I mentioned above. Before the
upgrade, the system was rock solid without any kmem tuning in
loader.conf (vm.kmem_size was around 2.7GB IIRC).
After the update, I could see that more wired memory was assigned, and
then steep drop (crash) occured.
"by day" graph shows my experiment to bump vm.kmem_size=12G
(recommended somewhere in this thread) and explicitly limit
vfs.zfs.arc_max=2G. I was surprised because the wired memory quickly
increased over 5GB...
Then I noticed that the default value of vfs.zfs.zio.use_uma was 1 on
amd64, so I turned it off and removed other memory tunings (kmem_size
and arc_max) in loader.conf on Tuesday at 16:00 and rebooted. It
seems that the usage of wired memory was stablized and no crash since
then with the default kmem_size and arc_max.
Does anyone have any idea about this behavior?
My wild guess is that ZFS/UMA code is not stable on amd64 too, and it
should be turned off by default for 8.1-RELEASE maybe....
Regards,
--
Yoshiaki Kasahara
Research Institute for Information Technology, Kyushu University
kasa...@nc.kyushu-u.ac.jp
Jeremy Chadwick
I realise you're talking about amd64, but I'm not sure how UMA is
getting enabled on i386 to begin with. It does look like it's enabled
on amd64 by default.
src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c --
#if defined(__amd64__)
static int zio_use_uma = 1;
#else
static int zio_use_uma = 0;
#endif
SYSCTL_DECL(_vfs_zfs);
SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO");
TUNABLE_INT("vfs.zfs.zio.use_uma", &zio_use_uma);
SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, &zio_use_uma, 0,
"Use uma(9) for ZIO allocations");
Also note that this tunable does not appear in sysctl -a output; I
believe this is caused by the first parameter to SYSCTL_INT() being
incorrect -- should be _vfs_zfs, not _vfs_zfs_zio. Regardless of
that, it's still adjustable in loader.conf.
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
_______________________________________________
Pawel Jakub Dawidek
> SYSCTL_DECL(_vfs_zfs);
> SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO");
> TUNABLE_INT("vfs.zfs.zio.use_uma", &zio_use_uma);
> SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, &zio_use_uma, 0,
> "Use uma(9) for ZIO allocations");
>
> Also note that this tunable does not appear in sysctl -a output; I
> believe this is caused by the first parameter to SYSCTL_INT() being
> incorrect -- should be _vfs_zfs, not _vfs_zfs_zio. Regardless of
> that, it's still adjustable in loader.conf.
Why? I create vfs.zfs.zio node and create vfs.zfs.zio.use_uma sysctl to
match tunable name.
Jeremy Chadwick
> On Tue, Jun 08, 2010 at 02:54:44AM -0700, Jeremy Chadwick wrote:
> > SYSCTL_DECL(_vfs_zfs);
> > SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO");
> > TUNABLE_INT("vfs.zfs.zio.use_uma", &zio_use_uma);
> > SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, &zio_use_uma, 0,
> > "Use uma(9) for ZIO allocations");
> >
> > Also note that this tunable does not appear in sysctl -a output; I
> > believe this is caused by the first parameter to SYSCTL_INT() being
> > incorrect -- should be _vfs_zfs, not _vfs_zfs_zio. Regardless of
> > that, it's still adjustable in loader.conf.
>
> Why? I create vfs.zfs.zio node and create vfs.zfs.zio.use_uma sysctl to
> match tunable name.
Sorry, I was looking at some example code that used _vfs_zfs, not
realising the node delimiter is based on the underscore.
So to recap, vfs.zfs.zio.use_uma doesn't show up in sysctl output.
There are other pieces of code which declare their own node as well, and
those do appear. So the question is why this specific tunable doesn't.
Looking through comparative code (specifically
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c) --
81 SYSCTL_DECL(_vfs_zfs_vdev);
82 SYSCTL_NODE(_vfs_zfs_vdev, OID_AUTO, cache, CTLFLAG_RW, 0, "ZFS VDEV Cache");
83 TUNABLE_INT("vfs.zfs.vdev.cache.max", &zfs_vdev_cache_max);
84 SYSCTL_INT(_vfs_zfs_vdev_cache, OID_AUTO, max, CTLFLAG_RDTUN,
85 &zfs_vdev_cache_max, 0, "Maximum I/O request size that increase read size");
86 TUNABLE_INT("vfs.zfs.vdev.cache.size", &zfs_vdev_cache_size);
87 SYSCTL_INT(_vfs_zfs_vdev_cache, OID_AUTO, size, CTLFLAG_RDTUN,
88 &zfs_vdev_cache_size, 0, "Size of VDEV cache");
89 TUNABLE_INT("vfs.zfs.vdev.cache.bshift", &zfs_vdev_cache_bshift);
90 SYSCTL_INT(_vfs_zfs_vdev_cache, OID_AUTO, bshift, CTLFLAG_RDTUN,
91 &zfs_vdev_cache_bshift, 0, "Turn too small requests into 1 << this value");
While for sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c --
41 SYSCTL_DECL(_vfs_zfs);
42 SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO");
43 TUNABLE_INT("vfs.zfs.zio.use_uma", &zio_use_uma);
44 SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, &zio_use_uma, 0,
45 "Use uma(9) for ZIO allocations");
I think lines 41 and 42 here are incorrect and should have _vfs_zfs_zio
as their first parameter?
Pete French
Errr, does for me....
$ sysctl -a | grep vfs.zfs.zio.use_uma
vfs.zfs.zio.use_uma: 1
Thats from 8.1-PRERELEASE on June 2nd.
..but all the question of sysctls is a bit of a red herring to me ?I'm
more intyerested in whether we really need to disable UMA to get stable
ZFS on amd64 right now.
-pete.
Jeremy Chadwick
> On Tue, 8 Jun 2010 02:54:44 -0700,
> Jeremy Chadwick <fre...@jdc.parodius.com> said:
>
> > I realise you're talking about amd64, but I'm not sure how UMA is
> > getting enabled on i386 to begin with. It does look like it's enabled
> > on amd64 by default.
>
> introduced on May 24th. Before that, ZIO_USE_UMA was used to control
> the use of UMA, and it was enabled unconditionally in
> /usr/src/sys/modules/zfs.
Yoshiaki, thanks for the tip. I didn't check CVS commit logs to see
when this tunable was introduced. It appears that the systems I'm
looking at don't have the use_uma tunable because the kernel was built
from RELENG_8 code dated May 23rd. This would also explain why Pete
sees the tunable and I don't.
Wish this stuff was documented somewhere. :-)
Florian Smeets
I can confirm this. I just upgraded a system from late January to
yesterdays 8-STABLE. I had to set vfs.zfs.zio.use_uma="0", to get the
system survive buildworld again.
This is amd64 with only 1GB, (i know this is considered not enough for
ZFS) with the kernel from January it was stable without any tuning.
After disabling ZFSs UMA usage, it seems to be working fine again.
Just another data point.
Cheers,
Florian
Paul Schenkeveld
+1 for me, several of my servers crash *every night* on a script that
does csup from cvs.freebsd.org (cvs mode if that matters) followed by
cvs up -dP in some source trees and one ports tree.
Hardware has 12 GB RAM, dual nehalem processors; kernel is amd64, no
sysctl tuning related to ZFS. With kernels before april 18 everything
is rock solid here, one production server has been up since april 17.
Can do some limited testing on one of these servers on mondays if
needed.
> Cheers,
> Florian
Regards,
Paul Schenkeveld