Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
zfs arc and amount of wired memory
92 views
Skip to first unread message
Eugene M. Zheganin
Feb 7, 2012, 3:36:20 AM2/7/12
to
Hi.
I have a server with 9.0/amd64 and 4 Gigs of RAM.
Today's questions are about the amount of memory in 'wired' state and
the ARC size.
If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
===Cut===
ARC Size: 12.50% 363.14 MiB
Target Size: (Adaptive) 12.50% 363.18 MiB
Min Size (Hard Limit): 12.50% 363.18 MiB
Max Size (High Water): 8:1 2.84 GiB
===Cut===
At the same time I have 3500 megs in wired state:
===Cut===
Mem: 237M Active, 36M Inact, 3502M Wired, 78M Cache, 432K Buf, 37M Free
===Cut===
First question - what is the actual size of ARC, and how can it be
determined ?
Solaris version of the script is more comprehensive (ran on Solaris):
===Cut===
ARC Size:
Current Size: 6457 MB (arcsize)
Target Size (Adaptive): 6457 MB (c)
Min Size (Hard Limit): 2941 MB (zfs_arc_min)
Max Size (Hard Limit): 23534 MB (zfs_arc_max)
===Cut===
The arcstat script makes me think that the ARC size is about 380 megs
indeed:
===Cut===
Time read miss miss% dmis dm% pmis pm% mmis mm%
size tsize
14:33:35 170M 7466K 4 7466K 4 192 78 793K 3
380M 380M
===Cut===
Second question: if the size is 363 Megs, why do I have 3500 Megs in
wired state? From my experience this is directly related to the zfs, but
380 megs its like about ten times smaller than 3600 megs. At the same
time I have like 700 Megs in swap, so my guess - zfs isn't freeing
memory for current needs that easily.
Yeah, I can tune it down, but I just would like to know what is
happening on an untuned machine.
Thanks.
Eugene.
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
I have a server with 9.0/amd64 and 4 Gigs of RAM.
Today's questions are about the amount of memory in 'wired' state and
the ARC size.
If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
===Cut===
ARC Size: 12.50% 363.14 MiB
Target Size: (Adaptive) 12.50% 363.18 MiB
Min Size (Hard Limit): 12.50% 363.18 MiB
Max Size (High Water): 8:1 2.84 GiB
===Cut===
At the same time I have 3500 megs in wired state:
===Cut===
Mem: 237M Active, 36M Inact, 3502M Wired, 78M Cache, 432K Buf, 37M Free
===Cut===
First question - what is the actual size of ARC, and how can it be
determined ?
Solaris version of the script is more comprehensive (ran on Solaris):
===Cut===
ARC Size:
Current Size: 6457 MB (arcsize)
Target Size (Adaptive): 6457 MB (c)
Min Size (Hard Limit): 2941 MB (zfs_arc_min)
Max Size (Hard Limit): 23534 MB (zfs_arc_max)
===Cut===
The arcstat script makes me think that the ARC size is about 380 megs
indeed:
===Cut===
Time read miss miss% dmis dm% pmis pm% mmis mm%
size tsize
14:33:35 170M 7466K 4 7466K 4 192 78 793K 3
380M 380M
===Cut===
Second question: if the size is 363 Megs, why do I have 3500 Megs in
wired state? From my experience this is directly related to the zfs, but
380 megs its like about ten times smaller than 3600 megs. At the same
time I have like 700 Megs in swap, so my guess - zfs isn't freeing
memory for current needs that easily.
Yeah, I can tune it down, but I just would like to know what is
happening on an untuned machine.
Thanks.
Eugene.
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
Andriy Gapon
Feb 7, 2012, 5:46:45 AM2/7/12
to
on 07/02/2012 10:36 Eugene M. Zheganin said the following:
overview of the state and configuration of the system.
> The arcstat script makes me think that the ARC size is about 380 megs indeed:
>
> ===Cut===
> Time read miss miss% dmis dm% pmis pm% mmis mm% size tsize
> 14:33:35 170M 7466K 4 7466K 4 192 78 793K 3 380M 380M
> ===Cut===
>
> Second question: if the size is 363 Megs, why do I have 3500 Megs in wired
> state? From my experience this is directly related to the zfs, but 380 megs its
> like about ten times smaller than 3600 megs. At the same time I have like 700
> Megs in swap, so my guess - zfs isn't freeing memory for current needs that easily.
>
> Yeah, I can tune it down, but I just would like to know what is happening on an
> untuned machine.
--
Andriy Gapon
> Hi.
>
> I have a server with 9.0/amd64 and 4 Gigs of RAM.
> Today's questions are about the amount of memory in 'wired' state and the ARC size.
>
> If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
>
> ===Cut===
> ARC Size: 12.50% 363.14 MiB
> Target Size: (Adaptive) 12.50% 363.18 MiB
> Min Size (Hard Limit): 12.50% 363.18 MiB
> Max Size (High Water): 8:1 2.84 GiB
> ===Cut===
>
> At the same time I have 3500 megs in wired state:
>
> ===Cut===
> Mem: 237M Active, 36M Inact, 3502M Wired, 78M Cache, 432K Buf, 37M Free
> ===Cut===
>
> First question - what is the actual size of ARC, and how can it be determined ?
> Solaris version of the script is more comprehensive (ran on Solaris):
>
> ===Cut===
> ARC Size:
> Current Size: 6457 MB (arcsize)
> Target Size (Adaptive): 6457 MB (c)
> Min Size (Hard Limit): 2941 MB (zfs_arc_min)
> Max Size (Hard Limit): 23534 MB (zfs_arc_max)
> ===Cut===
Please try sysutils/zfs-stats; zfs-stats -a output should provide a good
>
> I have a server with 9.0/amd64 and 4 Gigs of RAM.
> Today's questions are about the amount of memory in 'wired' state and the ARC size.
>
> If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
>
> ===Cut===
> ARC Size: 12.50% 363.14 MiB
> Target Size: (Adaptive) 12.50% 363.18 MiB
> Min Size (Hard Limit): 12.50% 363.18 MiB
> Max Size (High Water): 8:1 2.84 GiB
> ===Cut===
>
> At the same time I have 3500 megs in wired state:
>
> ===Cut===
> Mem: 237M Active, 36M Inact, 3502M Wired, 78M Cache, 432K Buf, 37M Free
> ===Cut===
>
> First question - what is the actual size of ARC, and how can it be determined ?
> Solaris version of the script is more comprehensive (ran on Solaris):
>
> ===Cut===
> ARC Size:
> Current Size: 6457 MB (arcsize)
> Target Size (Adaptive): 6457 MB (c)
> Min Size (Hard Limit): 2941 MB (zfs_arc_min)
> Max Size (Hard Limit): 23534 MB (zfs_arc_max)
> ===Cut===
overview of the state and configuration of the system.
> The arcstat script makes me think that the ARC size is about 380 megs indeed:
>
> ===Cut===
> Time read miss miss% dmis dm% pmis pm% mmis mm% size tsize
> 14:33:35 170M 7466K 4 7466K 4 192 78 793K 3 380M 380M
> ===Cut===
>
> Second question: if the size is 363 Megs, why do I have 3500 Megs in wired
> state? From my experience this is directly related to the zfs, but 380 megs its
> like about ten times smaller than 3600 megs. At the same time I have like 700
> Megs in swap, so my guess - zfs isn't freeing memory for current needs that easily.
>
> Yeah, I can tune it down, but I just would like to know what is happening on an
> untuned machine.
Andriy Gapon
Eugene M. Zheganin
Feb 7, 2012, 6:34:50 AM2/7/12
to
Hi.
On 07.02.2012 16:46, Andriy Gapon wrote:
> on 07/02/2012 10:36 Eugene M. Zheganin said the following:
amount of memory:
[emz@taiga:~]# zfs-stats -A
------------------------------------------------------------------------
ZFS Subsystem Report Tue Feb 7 17:32:34 2012
------------------------------------------------------------------------
ARC Summary: (THROTTLED)
Memory Throttle Count: 2.62k
ARC Misc:
Deleted: 10.17m
Recycle Misses: 1.47m
Mutex Misses: 7.69k
Evict Skips: 7.69k
ARC Size: 12.50% 363.24 MiB
Recently Used Cache Size: 57.54% 209.01 MiB
Frequently Used Cache Size: 42.46% 154.22 MiB
ARC Hash Breakdown:
Elements Max: 191.41k
Elements Current: 33.76% 64.63k
Collisions: 27.09m
Chain Max: 17
Chains: 13.56k
------------------------------------------------------------------------
I still don't get why I have 3/5 Gigs in wired state.
Thanks.
Eugene.
On 07.02.2012 16:46, Andriy Gapon wrote:
> on 07/02/2012 10:36 Eugene M. Zheganin said the following:
>> If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
>>
>> ===Cut===
>> ARC Size: 12.50% 363.14 MiB
>> Target Size: (Adaptive) 12.50% 363.18 MiB
>> Min Size (Hard Limit): 12.50% 363.18 MiB
>> Max Size (High Water): 8:1 2.84 GiB
>> ===Cut===
>>
>> At the same time I have 3500 megs in wired state:
>>
>>
>>
>> ===Cut===
>> ARC Size: 12.50% 363.14 MiB
>> Target Size: (Adaptive) 12.50% 363.18 MiB
>> Min Size (Hard Limit): 12.50% 363.18 MiB
>> Max Size (High Water): 8:1 2.84 GiB
>> ===Cut===
>>
>> At the same time I have 3500 megs in wired state:
>>
>>
>> Please try sysutils/zfs-stats; zfs-stats -a output should provide a good
>> overview of the state and configuration of the system.
>>
Thanks. But it seems to be the same script. Anyway, it reports the same
>> overview of the state and configuration of the system.
>>
amount of memory:
[emz@taiga:~]# zfs-stats -A
------------------------------------------------------------------------
ZFS Subsystem Report Tue Feb 7 17:32:34 2012
------------------------------------------------------------------------
ARC Summary: (THROTTLED)
Memory Throttle Count: 2.62k
ARC Misc:
Deleted: 10.17m
Recycle Misses: 1.47m
Mutex Misses: 7.69k
Evict Skips: 7.69k
ARC Size: 12.50% 363.24 MiB
Target Size: (Adaptive) 12.50% 363.18 MiB
Min Size (Hard Limit): 12.50% 363.18 MiB
Max Size (High Water): 8:1 2.84 GiB
ARC Size Breakdown:
Min Size (Hard Limit): 12.50% 363.18 MiB
Max Size (High Water): 8:1 2.84 GiB
Recently Used Cache Size: 57.54% 209.01 MiB
Frequently Used Cache Size: 42.46% 154.22 MiB
ARC Hash Breakdown:
Elements Max: 191.41k
Elements Current: 33.76% 64.63k
Collisions: 27.09m
Chain Max: 17
Chains: 13.56k
------------------------------------------------------------------------
I still don't get why I have 3/5 Gigs in wired state.
Thanks.
Eugene.
Andriy Gapon
Feb 7, 2012, 6:43:49 AM2/7/12
to
on 07/02/2012 13:34 Eugene M. Zheganin said the following:
--
Andriy Gapon
> Hi.
>
> On 07.02.2012 16:46, Andriy Gapon wrote:
>> on 07/02/2012 10:36 Eugene M. Zheganin said the following:
>>> If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
>>>
>>> ===Cut===
>>> ARC Size: 12.50% 363.14 MiB
>>> Target Size: (Adaptive) 12.50% 363.18 MiB
>>> Min Size (Hard Limit): 12.50% 363.18 MiB
>>> Max Size (High Water): 8:1 2.84 GiB
>>> ===Cut===
>>>
>>> At the same time I have 3500 megs in wired state:
>>>
>>>
>>> Please try sysutils/zfs-stats; zfs-stats -a output should provide a good
>>> overview of the state and configuration of the system.
>>>
> Thanks. But it seems to be the same script. Anyway, it reports the same amount
> of memory:
>
> [emz@taiga:~]# zfs-stats -A
zfs-stats -A is not the same as zfs-stats -a
>
> On 07.02.2012 16:46, Andriy Gapon wrote:
>> on 07/02/2012 10:36 Eugene M. Zheganin said the following:
>>> If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
>>>
>>> ===Cut===
>>> ARC Size: 12.50% 363.14 MiB
>>> Target Size: (Adaptive) 12.50% 363.18 MiB
>>> Min Size (Hard Limit): 12.50% 363.18 MiB
>>> Max Size (High Water): 8:1 2.84 GiB
>>> ===Cut===
>>>
>>> At the same time I have 3500 megs in wired state:
>>>
>>>
>>> Please try sysutils/zfs-stats; zfs-stats -a output should provide a good
>>> overview of the state and configuration of the system.
>>>
> Thanks. But it seems to be the same script. Anyway, it reports the same amount
> of memory:
>
> [emz@taiga:~]# zfs-stats -A
Andriy Gapon
Eugene M. Zheganin
Feb 7, 2012, 10:35:05 AM2/7/12
to
Hi.
On 07.02.2012 17:43, Andriy Gapon wrote:
> on 07/02/2012 13:34 Eugene M. Zheganin said the following:
>> Hi.
>>
>> On 07.02.2012 16:46, Andriy Gapon wrote:
>>> on 07/02/2012 10:36 Eugene M. Zheganin said the following:
>>>> If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
>>>>
>>>> ===Cut===
>>>> ARC Size: 12.50% 363.14 MiB
>>>> Target Size: (Adaptive) 12.50% 363.18 MiB
>>>> Min Size (Hard Limit): 12.50% 363.18 MiB
>>>> Max Size (High Water): 8:1 2.84 GiB
>>>> ===Cut===
>>>>
>>>> At the same time I have 3500 megs in wired state:
>>>>
>>>>
>>>> Please try sysutils/zfs-stats; zfs-stats -a output should provide a good
>>>> overview of the state and configuration of the system.
>>>>
>> Thanks. But it seems to be the same script. Anyway, it reports the same amount
>> of memory:
>>
>> [emz@taiga:~]# zfs-stats -A
> zfs-stats -A is not the same as zfs-stats -a
>
Okay, thank you once again, it made me more clear what is happening to
the memory:
System Memory:
4.39% 172.47 MiB Active, 0.29% 11.53 MiB Inact
90.68% 3.48 GiB Wired, 3.54% 138.98 MiB Cache
0.10% 4.12 MiB Free, 0.99% 39.07 MiB Gap
Real Installed: 4.00 GiB
Real Available: 99.61% 3.98 GiB
Real Managed: 96.31% 3.84 GiB
Logical Total: 4.00 GiB
Logical Used: 96.22% 3.85 GiB
Logical Free: 3.78% 154.63 MiB
Kernel Memory: 393.82 MiB
Data: 96.14% 378.63 MiB
Text: 3.86% 15.20 MiB
Kernel Memory Map: 2.68 GiB
Size: 9.82% 269.77 MiB
Free: 90.18% 2.42 GiB
Looks like most of the 'wired' space is makred in kernel as free, though
still 'wired', and thus for me it doesn't look that free as 'free' or
'inactive', and, as the paged out amount continues to grow, I want to
ask if there's a way to make this address space availabe to a userland
processes ?
On 07.02.2012 17:43, Andriy Gapon wrote:
> on 07/02/2012 13:34 Eugene M. Zheganin said the following:
>> Hi.
>>
>> On 07.02.2012 16:46, Andriy Gapon wrote:
>>> on 07/02/2012 10:36 Eugene M. Zheganin said the following:
>>>> If I use the script from http://wiki.freebsd.org/ZFSTuningGuide , it says:
>>>>
>>>> ===Cut===
>>>> ARC Size: 12.50% 363.14 MiB
>>>> Target Size: (Adaptive) 12.50% 363.18 MiB
>>>> Min Size (Hard Limit): 12.50% 363.18 MiB
>>>> Max Size (High Water): 8:1 2.84 GiB
>>>> ===Cut===
>>>>
>>>> At the same time I have 3500 megs in wired state:
>>>>
>>>>
>>>> Please try sysutils/zfs-stats; zfs-stats -a output should provide a good
>>>> overview of the state and configuration of the system.
>>>>
>> Thanks. But it seems to be the same script. Anyway, it reports the same amount
>> of memory:
>>
>> [emz@taiga:~]# zfs-stats -A
> zfs-stats -A is not the same as zfs-stats -a
>
the memory:
System Memory:
4.39% 172.47 MiB Active, 0.29% 11.53 MiB Inact
90.68% 3.48 GiB Wired, 3.54% 138.98 MiB Cache
0.10% 4.12 MiB Free, 0.99% 39.07 MiB Gap
Real Installed: 4.00 GiB
Real Available: 99.61% 3.98 GiB
Real Managed: 96.31% 3.84 GiB
Logical Total: 4.00 GiB
Logical Used: 96.22% 3.85 GiB
Logical Free: 3.78% 154.63 MiB
Kernel Memory: 393.82 MiB
Data: 96.14% 378.63 MiB
Text: 3.86% 15.20 MiB
Kernel Memory Map: 2.68 GiB
Size: 9.82% 269.77 MiB
Free: 90.18% 2.42 GiB
Looks like most of the 'wired' space is makred in kernel as free, though
still 'wired', and thus for me it doesn't look that free as 'free' or
'inactive', and, as the paged out amount continues to grow, I want to
ask if there's a way to make this address space availabe to a userland
processes ?
Andriy Gapon
Feb 7, 2012, 10:51:46 AM2/7/12
to
on 07/02/2012 17:35 Eugene M. Zheganin said the following:
BTW, are you reluctant to share the full zfs-stats -a output? You don't have to
place it inline, you can upload it somewhere and provide a link.
--
Andriy Gapon
> Okay, thank you once again, it made me more clear what is happening to the memory:
>
>
> System Memory:
>
> 4.39% 172.47 MiB Active, 0.29% 11.53 MiB Inact
> 90.68% 3.48 GiB Wired, 3.54% 138.98 MiB Cache
> 0.10% 4.12 MiB Free, 0.99% 39.07 MiB Gap
>
> Real Installed: 4.00 GiB
> Real Available: 99.61% 3.98 GiB
> Real Managed: 96.31% 3.84 GiB
>
> Logical Total: 4.00 GiB
> Logical Used: 96.22% 3.85 GiB
> Logical Free: 3.78% 154.63 MiB
>
> Kernel Memory: 393.82 MiB
> Data: 96.14% 378.63 MiB
> Text: 3.86% 15.20 MiB
>
> Kernel Memory Map: 2.68 GiB
> Size: 9.82% 269.77 MiB
> Free: 90.18% 2.42 GiB
>
> Looks like most of the 'wired' space is makred in kernel as free, though still
> 'wired', and thus for me it doesn't look that free as 'free' or 'inactive', and,
> as the paged out amount continues to grow, I want to ask if there's a way to make
> this address space availabe to a userland processes ?
I am not sure that these conclusions are correct. Wired is wired, it's not free.
>
>
> System Memory:
>
> 4.39% 172.47 MiB Active, 0.29% 11.53 MiB Inact
> 90.68% 3.48 GiB Wired, 3.54% 138.98 MiB Cache
> 0.10% 4.12 MiB Free, 0.99% 39.07 MiB Gap
>
> Real Installed: 4.00 GiB
> Real Available: 99.61% 3.98 GiB
> Real Managed: 96.31% 3.84 GiB
>
> Logical Total: 4.00 GiB
> Logical Used: 96.22% 3.85 GiB
> Logical Free: 3.78% 154.63 MiB
>
> Kernel Memory: 393.82 MiB
> Data: 96.14% 378.63 MiB
> Text: 3.86% 15.20 MiB
>
> Kernel Memory Map: 2.68 GiB
> Size: 9.82% 269.77 MiB
> Free: 90.18% 2.42 GiB
>
> Looks like most of the 'wired' space is makred in kernel as free, though still
> 'wired', and thus for me it doesn't look that free as 'free' or 'inactive', and,
> as the paged out amount continues to grow, I want to ask if there's a way to make
> this address space availabe to a userland processes ?
BTW, are you reluctant to share the full zfs-stats -a output? You don't have to
place it inline, you can upload it somewhere and provide a link.
--
Andriy Gapon
Eugene M. Zheganin
Feb 7, 2012, 11:03:39 AM2/7/12
to
Hi.
On 07.02.2012 21:51, Andriy Gapon wrote:
>
> I am not sure that these conclusions are correct. Wired is wired, it's not free.
> BTW, are you reluctant to share the full zfs-stats -a output? You don't have to
> place it inline, you can upload it somewhere and provide a link.
>
Well... nothing secret in it (in case someone will be interested too and
so it stays in maillist archive):
===Cut===
[emz@taiga:~]> zfs-stats -a
------------------------------------------------------------------------
ZFS Subsystem Report Tue Feb 7 22:01:09 2012
------------------------------------------------------------------------
System Information:
Kernel Version: 900044 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 28
ZFS Filesystem Version: 5
FreeBSD 9.0-RELEASE #1: Mon Jan 23 13:36:16 YEKT 2012 emz
22:01 up 11 days, 2:44, 4 users, load averages: 0,30 0,26 0,32
------------------------------------------------------------------------
System Memory:
4.30% 169.09 MiB Active, 1.04% 40.95 MiB Inact
90.80% 3.48 GiB Wired, 2.17% 85.25 MiB Cache
0.69% 27.30 MiB Free, 0.99% 39.08 MiB Gap
Real Installed: 4.00 GiB
Real Available: 99.61% 3.98 GiB
Real Managed: 96.31% 3.84 GiB
Logical Total: 4.00 GiB
Logical Used: 96.25% 3.85 GiB
Logical Free: 3.75% 153.50 MiB
Kernel Memory: 397.53 MiB
Data: 96.18% 382.33 MiB
Text: 3.82% 15.20 MiB
Kernel Memory Map: 2.69 GiB
Size: 9.93% 273.20 MiB
Free: 90.07% 2.42 GiB
------------------------------------------------------------------------
ARC Summary: (THROTTLED)
Memory Throttle Count: 3.20k
ARC Misc:
Deleted: 10.83m
Recycle Misses: 1.55m
Mutex Misses: 7.80k
Evict Skips: 7.80k
ARC Size: 12.50% 363.20 MiB
Recently Used Cache Size: 56.17% 204.02 MiB
Frequently Used Cache Size: 43.83% 159.18 MiB
ARC Hash Breakdown:
Elements Max: 191.41k
Elements Current: 32.79% 62.76k
Collisions: 28.06m
Chain Max: 17
Chains: 12.77k
------------------------------------------------------------------------
ARC Efficiency: 179.54m
Cache Hit Ratio: 95.07% 170.68m
Cache Miss Ratio: 4.93% 8.86m
Actual Hit Ratio: 95.07% 170.68m
Data Demand Efficiency: 94.72% 152.53m
Data Prefetch Efficiency: 0.00% 20
CACHE HITS BY CACHE LIST:
Most Recently Used: 30.50% 52.05m
Most Frequently Used: 69.50% 118.63m
Most Recently Used Ghost: 0.30% 517.41k
Most Frequently Used Ghost: 1.02% 1.74m
CACHE HITS BY DATA TYPE:
Demand Data: 84.65% 144.48m
Prefetch Data: 0.00% 0
Demand Metadata: 15.35% 26.20m
Prefetch Metadata: 0.00% 52
CACHE MISSES BY DATA TYPE:
Demand Data: 90.88% 8.05m
Prefetch Data: 0.00% 20
Demand Metadata: 9.12% 807.98k
Prefetch Metadata: 0.00% 172
------------------------------------------------------------------------
L2ARC is disabled
------------------------------------------------------------------------
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 384
vm.kmem_size 4120326144
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 329853485875
vfs.zfs.l2c_only_size 0
vfs.zfs.mfu_ghost_data_lsize 114792960
vfs.zfs.mfu_ghost_metadata_lsize 69912064
vfs.zfs.mfu_ghost_size 184705024
vfs.zfs.mfu_data_lsize 26686464
vfs.zfs.mfu_metadata_lsize 13492736
vfs.zfs.mfu_size 45798912
vfs.zfs.mru_ghost_data_lsize 22662656
vfs.zfs.mru_ghost_metadata_lsize 142631424
vfs.zfs.mru_ghost_size 165294080
vfs.zfs.mru_data_lsize 149837312
vfs.zfs.mru_metadata_lsize 6439424
vfs.zfs.mru_size 215066624
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 1421824
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 761646080
vfs.zfs.arc_meta_used 202913864
vfs.zfs.arc_min 380823040
vfs.zfs.arc_max 3046584320
vfs.zfs.dedup.prefetch 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.write_limit_override 0
vfs.zfs.write_limit_inflated 12834803712
vfs.zfs.write_limit_max 534783488
vfs.zfs.write_limit_min 33554432
vfs.zfs.write_limit_shift 3
vfs.zfs.no_write_throttle 0
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 1
vfs.zfs.mg_alloc_failures 8
vfs.zfs.check_hostid 1
vfs.zfs.recover 0
vfs.zfs.txg.synctime_ms 1000
vfs.zfs.txg.timeout 5
vfs.zfs.scrub_limit 10
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.ramp_rate 2
vfs.zfs.vdev.time_shift 6
vfs.zfs.vdev.min_pending 4
vfs.zfs.vdev.max_pending 10
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_replay_disable 0
vfs.zfs.zio.use_uma 0
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 28
vfs.zfs.version.acl 1
vfs.zfs.debug 0
vfs.zfs.super_owner 0
------------------------------------------------------------------------
[emz@taiga:~]>
===Cut===
Thanks.
Eugene.
On 07.02.2012 21:51, Andriy Gapon wrote:
>
> I am not sure that these conclusions are correct. Wired is wired, it's not free.
> BTW, are you reluctant to share the full zfs-stats -a output? You don't have to
> place it inline, you can upload it somewhere and provide a link.
>
so it stays in maillist archive):
===Cut===
[emz@taiga:~]> zfs-stats -a
------------------------------------------------------------------------
ZFS Subsystem Report Tue Feb 7 22:01:09 2012
------------------------------------------------------------------------
System Information:
Kernel Version: 900044 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 28
ZFS Filesystem Version: 5
FreeBSD 9.0-RELEASE #1: Mon Jan 23 13:36:16 YEKT 2012 emz
22:01 up 11 days, 2:44, 4 users, load averages: 0,30 0,26 0,32
------------------------------------------------------------------------
System Memory:
4.30% 169.09 MiB Active, 1.04% 40.95 MiB Inact
90.80% 3.48 GiB Wired, 2.17% 85.25 MiB Cache
0.69% 27.30 MiB Free, 0.99% 39.08 MiB Gap
Real Installed: 4.00 GiB
Real Available: 99.61% 3.98 GiB
Real Managed: 96.31% 3.84 GiB
Logical Total: 4.00 GiB
Logical Free: 3.75% 153.50 MiB
Kernel Memory: 397.53 MiB
Data: 96.18% 382.33 MiB
Text: 3.82% 15.20 MiB
Kernel Memory Map: 2.69 GiB
Size: 9.93% 273.20 MiB
Free: 90.07% 2.42 GiB
------------------------------------------------------------------------
ARC Summary: (THROTTLED)
Memory Throttle Count: 3.20k
ARC Misc:
Deleted: 10.83m
Recycle Misses: 1.55m
Mutex Misses: 7.80k
Evict Skips: 7.80k
ARC Size: 12.50% 363.20 MiB
Target Size: (Adaptive) 12.50% 363.18 MiB
Min Size (Hard Limit): 12.50% 363.18 MiB
Max Size (High Water): 8:1 2.84 GiB
ARC Size Breakdown:
Min Size (Hard Limit): 12.50% 363.18 MiB
Max Size (High Water): 8:1 2.84 GiB
Recently Used Cache Size: 56.17% 204.02 MiB
Frequently Used Cache Size: 43.83% 159.18 MiB
ARC Hash Breakdown:
Elements Max: 191.41k
Collisions: 28.06m
Chain Max: 17
Chains: 12.77k
------------------------------------------------------------------------
ARC Efficiency: 179.54m
Cache Hit Ratio: 95.07% 170.68m
Cache Miss Ratio: 4.93% 8.86m
Actual Hit Ratio: 95.07% 170.68m
Data Demand Efficiency: 94.72% 152.53m
Data Prefetch Efficiency: 0.00% 20
CACHE HITS BY CACHE LIST:
Most Recently Used: 30.50% 52.05m
Most Frequently Used: 69.50% 118.63m
Most Recently Used Ghost: 0.30% 517.41k
Most Frequently Used Ghost: 1.02% 1.74m
CACHE HITS BY DATA TYPE:
Demand Data: 84.65% 144.48m
Prefetch Data: 0.00% 0
Demand Metadata: 15.35% 26.20m
Prefetch Metadata: 0.00% 52
CACHE MISSES BY DATA TYPE:
Demand Data: 90.88% 8.05m
Prefetch Data: 0.00% 20
Demand Metadata: 9.12% 807.98k
Prefetch Metadata: 0.00% 172
------------------------------------------------------------------------
L2ARC is disabled
------------------------------------------------------------------------
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 384
vm.kmem_size 4120326144
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 329853485875
vfs.zfs.l2c_only_size 0
vfs.zfs.mfu_ghost_data_lsize 114792960
vfs.zfs.mfu_ghost_metadata_lsize 69912064
vfs.zfs.mfu_ghost_size 184705024
vfs.zfs.mfu_data_lsize 26686464
vfs.zfs.mfu_metadata_lsize 13492736
vfs.zfs.mfu_size 45798912
vfs.zfs.mru_ghost_data_lsize 22662656
vfs.zfs.mru_ghost_metadata_lsize 142631424
vfs.zfs.mru_ghost_size 165294080
vfs.zfs.mru_data_lsize 149837312
vfs.zfs.mru_metadata_lsize 6439424
vfs.zfs.mru_size 215066624
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 1421824
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 761646080
vfs.zfs.arc_meta_used 202913864
vfs.zfs.arc_min 380823040
vfs.zfs.arc_max 3046584320
vfs.zfs.dedup.prefetch 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.write_limit_override 0
vfs.zfs.write_limit_inflated 12834803712
vfs.zfs.write_limit_max 534783488
vfs.zfs.write_limit_min 33554432
vfs.zfs.write_limit_shift 3
vfs.zfs.no_write_throttle 0
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 1
vfs.zfs.mg_alloc_failures 8
vfs.zfs.check_hostid 1
vfs.zfs.recover 0
vfs.zfs.txg.synctime_ms 1000
vfs.zfs.txg.timeout 5
vfs.zfs.scrub_limit 10
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.ramp_rate 2
vfs.zfs.vdev.time_shift 6
vfs.zfs.vdev.min_pending 4
vfs.zfs.vdev.max_pending 10
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_replay_disable 0
vfs.zfs.zio.use_uma 0
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 28
vfs.zfs.version.acl 1
vfs.zfs.debug 0
vfs.zfs.super_owner 0
------------------------------------------------------------------------
[emz@taiga:~]>
===Cut===
Thanks.
Eugene.
Andriy Gapon
Feb 7, 2012, 3:17:10 PM2/7/12
to
on 07/02/2012 18:03 Eugene M. Zheganin said the following:
Thank you. I don't see anything suspicious/unusual there.
Just case, do you have ZFS dedup enabled by a chance?
I think that examination of vmstat -m and vmstat -z outputs may provide some
clues as to what got all that memory wired.
--
Andriy Gapon
> Hi.
>
> On 07.02.2012 21:51, Andriy Gapon wrote:
>>
>> I am not sure that these conclusions are correct. Wired is wired, it's not free.
>> BTW, are you reluctant to share the full zfs-stats -a output? You don't have to
>> place it inline, you can upload it somewhere and provide a link.
>>
> Well... nothing secret in it (in case someone will be interested too and so it
> stays in maillist archive):
[output snipped]
>
> On 07.02.2012 21:51, Andriy Gapon wrote:
>>
>> I am not sure that these conclusions are correct. Wired is wired, it's not free.
>> BTW, are you reluctant to share the full zfs-stats -a output? You don't have to
>> place it inline, you can upload it somewhere and provide a link.
>>
> Well... nothing secret in it (in case someone will be interested too and so it
> stays in maillist archive):
Thank you. I don't see anything suspicious/unusual there.
Just case, do you have ZFS dedup enabled by a chance?
I think that examination of vmstat -m and vmstat -z outputs may provide some
clues as to what got all that memory wired.
--
Andriy Gapon
Eugene M. Zheganin
Feb 8, 2012, 5:31:44 AM2/8/12
to
Hi.
On 08.02.2012 02:17, Andriy Gapon wrote:
> [output snipped]
>
> Thank you. I don't see anything suspicious/unusual there.
> Just case, do you have ZFS dedup enabled by a chance?
>
> I think that examination of vmstat -m and vmstat -z outputs may provide some
> clues as to what got all that memory wired.
>
Nope, I don't have deduplication feature enabled.
By the way, today, after eating another 100M of wired memory this server
hanged out with multiple non-stopping messages
swap_pager: indefinite wait buffer
Since it's swapping on zvol, it looks to me like it could be the
mentioned in another thread here ("Swap on zvol - recommendable?")
resource starvation issue; may be it happens faster when the ARC isn't
limited.
So I want to ask - how to report it and what should I include in such pr ?
Thanks.
Eugene.
On 08.02.2012 02:17, Andriy Gapon wrote:
> [output snipped]
>
> Thank you. I don't see anything suspicious/unusual there.
> Just case, do you have ZFS dedup enabled by a chance?
>
> I think that examination of vmstat -m and vmstat -z outputs may provide some
> clues as to what got all that memory wired.
>
By the way, today, after eating another 100M of wired memory this server
hanged out with multiple non-stopping messages
swap_pager: indefinite wait buffer
Since it's swapping on zvol, it looks to me like it could be the
mentioned in another thread here ("Swap on zvol - recommendable?")
resource starvation issue; may be it happens faster when the ARC isn't
limited.
So I want to ask - how to report it and what should I include in such pr ?
Thanks.
Eugene.
Alexander Leidinger
Feb 8, 2012, 7:15:37 AM2/8/12
to
On Wed, 08 Feb 2012 16:31:44 +0600 "Eugene M. Zheganin"
<e...@norma.perm.ru> wrote:
> swap_pager: indefinite wait buffer
>
> Since it's swapping on zvol, it looks to me like it could be the
> mentioned in another thread here ("Swap on zvol - recommendable?")
> resource starvation issue; may be it happens faster when the ARC
> isn't limited.
>
> So I want to ask - how to report it and what should I include in such
> pr ?
I can't remember to have seen any mention of SWAP on ZFS being safe
<e...@norma.perm.ru> wrote:
> swap_pager: indefinite wait buffer
>
> Since it's swapping on zvol, it looks to me like it could be the
> mentioned in another thread here ("Swap on zvol - recommendable?")
> resource starvation issue; may be it happens faster when the ARC
> isn't limited.
>
> So I want to ask - how to report it and what should I include in such
> pr ?
now. So if nobody can provide a reference to a place which tells that
the problems with SWAP on ZFS are fixed:
1. do not use SWAP on ZFS
2. see 1.
3. check if you see the same problem without SWAP on ZFS (btw. see 1.)
Bye,
Alexander.
--
http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137
Eugene M. Zheganin
Feb 8, 2012, 1:25:15 PM2/8/12
to
Hi.
On 08.02.2012 18:15, Alexander Leidinger wrote:
> I can't remember to have seen any mention of SWAP on ZFS being safe
> now. So if nobody can provide a reference to a place which tells that
> the problems with SWAP on ZFS are fixed:
> 1. do not use SWAP on ZFS
> 2. see 1.
> 3. check if you see the same problem without SWAP on ZFS (btw. see 1.)
>
So, if a swap have to be used, and, it has to be backed up with
something like gmirror so it won't come down with one of the disks,
there's no need to use zfs for system.
This makes zfs only useful in cases where you need to store something on
a couple+ of terabytes, still having OS on ufs. Occam's razor and so on.
Thanks for explanation.
Eugene.
On 08.02.2012 18:15, Alexander Leidinger wrote:
> I can't remember to have seen any mention of SWAP on ZFS being safe
> now. So if nobody can provide a reference to a place which tells that
> the problems with SWAP on ZFS are fixed:
> 1. do not use SWAP on ZFS
> 2. see 1.
> 3. check if you see the same problem without SWAP on ZFS (btw. see 1.)
>
something like gmirror so it won't come down with one of the disks,
there's no need to use zfs for system.
This makes zfs only useful in cases where you need to store something on
a couple+ of terabytes, still having OS on ufs. Occam's razor and so on.
Thanks for explanation.
Eugene.
Freddie Cash
Feb 8, 2012, 1:40:12 PM2/8/12
to
On Wed, Feb 8, 2012 at 10:25 AM, Eugene M. Zheganin <e...@norma.perm.ru> wrote:
> On 08.02.2012 18:15, Alexander Leidinger wrote:
>> I can't remember to have seen any mention of SWAP on ZFS being safe
>> now. So if nobody can provide a reference to a place which tells that
>> the problems with SWAP on ZFS are fixed:
>> 1. do not use SWAP on ZFS
>> 2. see 1.
>> 3. check if you see the same problem without SWAP on ZFS (btw. see 1.)
>>
> So, if a swap have to be used, and, it has to be backed up with something
> like gmirror so it won't come down with one of the disks, there's no need to
> use zfs for system.
>
> This makes zfs only useful in cases where you need to store something on a
> couple+ of terabytes, still having OS on ufs. Occam's razor and so on.
Or, you plug a USB stick into the back (or even inside the case as a
> On 08.02.2012 18:15, Alexander Leidinger wrote:
>> I can't remember to have seen any mention of SWAP on ZFS being safe
>> now. So if nobody can provide a reference to a place which tells that
>> the problems with SWAP on ZFS are fixed:
>> 1. do not use SWAP on ZFS
>> 2. see 1.
>> 3. check if you see the same problem without SWAP on ZFS (btw. see 1.)
>>
> So, if a swap have to be used, and, it has to be backed up with something
> like gmirror so it won't come down with one of the disks, there's no need to
> use zfs for system.
>
> This makes zfs only useful in cases where you need to store something on a
> couple+ of terabytes, still having OS on ufs. Occam's razor and so on.
lot of mobos have internal USB connectors now) and use that for swap.
--
Freddie Cash
fjw...@gmail.com
Freddie Cash
Feb 8, 2012, 1:40:45 PM2/8/12
to
On Wed, Feb 8, 2012 at 10:40 AM, Freddie Cash <fjw...@gmail.com> wrote:
> On Wed, Feb 8, 2012 at 10:25 AM, Eugene M. Zheganin <e...@norma.perm.ru> wrote:
>> On 08.02.2012 18:15, Alexander Leidinger wrote:
>>> I can't remember to have seen any mention of SWAP on ZFS being safe
>>> now. So if nobody can provide a reference to a place which tells that
>>> the problems with SWAP on ZFS are fixed:
>>> 1. do not use SWAP on ZFS
>>> 2. see 1.
>>> 3. check if you see the same problem without SWAP on ZFS (btw. see 1.)
>>>
>> So, if a swap have to be used, and, it has to be backed up with something
>> like gmirror so it won't come down with one of the disks, there's no need to
>> use zfs for system.
>>
>> This makes zfs only useful in cases where you need to store something on a
>> couple+ of terabytes, still having OS on ufs. Occam's razor and so on.
>
> Or, you plug a USB stick into the back (or even inside the case as a
> lot of mobos have internal USB connectors now) and use that for swap.
That also works well for adding L2ARC (cache) to the ZFS pool as well.
> On Wed, Feb 8, 2012 at 10:25 AM, Eugene M. Zheganin <e...@norma.perm.ru> wrote:
>> On 08.02.2012 18:15, Alexander Leidinger wrote:
>>> I can't remember to have seen any mention of SWAP on ZFS being safe
>>> now. So if nobody can provide a reference to a place which tells that
>>> the problems with SWAP on ZFS are fixed:
>>> 1. do not use SWAP on ZFS
>>> 2. see 1.
>>> 3. check if you see the same problem without SWAP on ZFS (btw. see 1.)
>>>
>> So, if a swap have to be used, and, it has to be backed up with something
>> like gmirror so it won't come down with one of the disks, there's no need to
>> use zfs for system.
>>
>> This makes zfs only useful in cases where you need to store something on a
>> couple+ of terabytes, still having OS on ufs. Occam's razor and so on.
>
> Or, you plug a USB stick into the back (or even inside the case as a
> lot of mobos have internal USB connectors now) and use that for swap.
Andriy Gapon
Feb 8, 2012, 3:29:36 PM2/8/12
to
on 08/02/2012 12:31 Eugene M. Zheganin said the following:
> By the way, today, after eating another 100M of wired memory this server hanged
> out with multiple non-stopping messages
>
kernel itself is starved on memory.
> So I want to ask - how to report it and what should I include in such pr ?
I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be
ZFS-related. I suspect that you might be running into some kernel memory leak.
If you manage to reproduce the high wired value again, then vmstat -m and
vmstat -z may provide some useful information.
In this vein, do you use any out-of-tree kernel modules?
Also, can you try to monitor your system to see when wired count grows?
--
Andriy Gapon
> Hi.
>
> On 08.02.2012 02:17, Andriy Gapon wrote:
>> [output snipped]
>>
>> Thank you. I don't see anything suspicious/unusual there.
>> Just case, do you have ZFS dedup enabled by a chance?
>>
>> I think that examination of vmstat -m and vmstat -z outputs may provide some
>> clues as to what got all that memory wired.
>>
> Nope, I don't have deduplication feature enabled.
OK. So, did you have a chance to inspect vmstat -m and vmstat -z?
>
> On 08.02.2012 02:17, Andriy Gapon wrote:
>> [output snipped]
>>
>> Thank you. I don't see anything suspicious/unusual there.
>> Just case, do you have ZFS dedup enabled by a chance?
>>
>> I think that examination of vmstat -m and vmstat -z outputs may provide some
>> clues as to what got all that memory wired.
>>
> Nope, I don't have deduplication feature enabled.
> By the way, today, after eating another 100M of wired memory this server hanged
> out with multiple non-stopping messages
>
> swap_pager: indefinite wait buffer
>
> Since it's swapping on zvol, it looks to me like it could be the mentioned in
> another thread here ("Swap on zvol - recommendable?") resource starvation issue;
> may be it happens faster when the ARC isn't limited.
It could be very well possible that swap on zvol doesn't work well when the
>
> Since it's swapping on zvol, it looks to me like it could be the mentioned in
> another thread here ("Swap on zvol - recommendable?") resource starvation issue;
> may be it happens faster when the ARC isn't limited.
kernel itself is starved on memory.
> So I want to ask - how to report it and what should I include in such pr ?
ZFS-related. I suspect that you might be running into some kernel memory leak.
If you manage to reproduce the high wired value again, then vmstat -m and
vmstat -z may provide some useful information.
In this vein, do you use any out-of-tree kernel modules?
Also, can you try to monitor your system to see when wired count grows?
--
Andriy Gapon
Jeremy Chadwick
Feb 8, 2012, 3:50:00 PM2/8/12
to
On Wed, Feb 08, 2012 at 10:29:36PM +0200, Andriy Gapon wrote:
> on 08/02/2012 12:31 Eugene M. Zheganin said the following:
> > Hi.
> >
> > On 08.02.2012 02:17, Andriy Gapon wrote:
> >> [output snipped]
> >>
> >> Thank you. I don't see anything suspicious/unusual there.
> >> Just case, do you have ZFS dedup enabled by a chance?
> >>
> >> I think that examination of vmstat -m and vmstat -z outputs may provide some
> >> clues as to what got all that memory wired.
> >>
> > Nope, I don't have deduplication feature enabled.
>
> OK. So, did you have a chance to inspect vmstat -m and vmstat -z?
Andriy,
> on 08/02/2012 12:31 Eugene M. Zheganin said the following:
> > Hi.
> >
> > On 08.02.2012 02:17, Andriy Gapon wrote:
> >> [output snipped]
> >>
> >> Thank you. I don't see anything suspicious/unusual there.
> >> Just case, do you have ZFS dedup enabled by a chance?
> >>
> >> I think that examination of vmstat -m and vmstat -z outputs may provide some
> >> clues as to what got all that memory wired.
> >>
> > Nope, I don't have deduplication feature enabled.
>
> OK. So, did you have a chance to inspect vmstat -m and vmstat -z?
Politely -- recommending this to a user is a good choice of action, but
the problem is that no user, even an experienced user, is going to know
what all of the "Types" (vmstat -m) or "ITEMs" (vmstat -z) correlate
with on the system.
For example, for vmstat -m, the ITEM name is "solaris". For vmstat -z,
the Types are named zio_* but I have a feeling there are more than just
that which pertain to ZFS. I'm having to make *assumptions*.
The FreeBSD VM is highly complex and is not "easy to understand" even
remotely. It becomes more complex when you consider that we use terms
like "wired", "active", "inactive", "cache", and "free" -- and none of
them, in simple English terms, actually represent the words chosen for
what they do.
Furthermore, the only definition I've been able to find over the years
for how any of these work, what they do/mean, etc. is here:
http://www.freebsd.org/doc/en/books/arch-handbook/vm.html
And this piece of documentation is only useful for people who understand
VMs (note: it was written by Matt Dillon, for example). It is not
useful for end-users trying to track down what within the kernel is
actually eating up memory. "vmstat -m" is as best as it's going to get,
and like I said, with the ITEM names being borderline ambiguous
(depending on what you're looking for -- with VFS and so on it's spread
all over the place), this becomes a very tedious task, where the user or
admin have to continually ask developers on the mailing lists what it is
they're looking at.
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |
Andriy Gapon
Feb 8, 2012, 4:18:02 PM2/8/12
to
on 08/02/2012 22:50 Jeremy Chadwick said the following:
it. I do not know of any easier way to analyze problems like this one.
--
Andriy Gapon
> Politely -- recommending this to a user is a good choice of action, but
> the problem is that no user, even an experienced user, is going to know
> what all of the "Types" (vmstat -m) or "ITEMs" (vmstat -z) correlate
> with on the system.
I see no problem with users sharing the output and asking for help interpreting
> the problem is that no user, even an experienced user, is going to know
> what all of the "Types" (vmstat -m) or "ITEMs" (vmstat -z) correlate
> with on the system.
it. I do not know of any easier way to analyze problems like this one.
--
Andriy Gapon
Miroslav Lachman
Feb 8, 2012, 7:11:36 PM2/8/12
to
Andriy Gapon wrote:
> on 08/02/2012 12:31 Eugene M. Zheganin said the following:
>> Hi.
>>
>> On 08.02.2012 02:17, Andriy Gapon wrote:
>>> [output snipped]
>>>
>>> Thank you. I don't see anything suspicious/unusual there.
>>> Just case, do you have ZFS dedup enabled by a chance?
>>>
>>> I think that examination of vmstat -m and vmstat -z outputs may provide some
>>> clues as to what got all that memory wired.
>>>
>> Nope, I don't have deduplication feature enabled.
>
> OK. So, did you have a chance to inspect vmstat -m and vmstat -z?
>
> on 08/02/2012 12:31 Eugene M. Zheganin said the following:
>> Hi.
>>
>> On 08.02.2012 02:17, Andriy Gapon wrote:
>>> [output snipped]
>>>
>>> Thank you. I don't see anything suspicious/unusual there.
>>> Just case, do you have ZFS dedup enabled by a chance?
>>>
>>> I think that examination of vmstat -m and vmstat -z outputs may provide some
>>> clues as to what got all that memory wired.
>>>
>> Nope, I don't have deduplication feature enabled.
>
> OK. So, did you have a chance to inspect vmstat -m and vmstat -z?
>
>> By the way, today, after eating another 100M of wired memory this server hanged
>> out with multiple non-stopping messages
>>
>> swap_pager: indefinite wait buffer
>>
>> Since it's swapping on zvol, it looks to me like it could be the mentioned in
>> another thread here ("Swap on zvol - recommendable?") resource starvation issue;
>> may be it happens faster when the ARC isn't limited.
>
> It could be very well possible that swap on zvol doesn't work well when the
> kernel itself is starved on memory.
>
>> So I want to ask - how to report it and what should I include in such pr ?
>
> I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be
> ZFS-related. I suspect that you might be running into some kernel memory leak.
> If you manage to reproduce the high wired value again, then vmstat -m and
> vmstat -z may provide some useful information.
>
> In this vein, do you use any out-of-tree kernel modules?
> Also, can you try to monitor your system to see when wired count grows?
I am seeing something similar on one of our machine. This is old 7.3
>> out with multiple non-stopping messages
>>
>> swap_pager: indefinite wait buffer
>>
>> Since it's swapping on zvol, it looks to me like it could be the mentioned in
>> another thread here ("Swap on zvol - recommendable?") resource starvation issue;
>> may be it happens faster when the ARC isn't limited.
>
> It could be very well possible that swap on zvol doesn't work well when the
> kernel itself is starved on memory.
>
>> So I want to ask - how to report it and what should I include in such pr ?
>
> I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be
> ZFS-related. I suspect that you might be running into some kernel memory leak.
> If you manage to reproduce the high wired value again, then vmstat -m and
> vmstat -z may provide some useful information.
>
> In this vein, do you use any out-of-tree kernel modules?
> Also, can you try to monitor your system to see when wired count grows?
with ZFS v13, that's why I did not reported it.
The machine is used as storage for backups made by rsync. All is running
fine for about 107 days. Then backups are slower and slower because of
some strange memory situation.
Mem: 15M Active, 17M Inact, 3620M Wired, 420K Cache, 48M Buf, 1166M Free
ARC Size:
Current Size: 1769 MB (arcsize)
Target Size (Adaptive): 512 MB (c)
Min Size (Hard Limit): 512 MB (zfs_arc_min)
Max Size (Hard Limit): 3584 MB (zfs_arc_max)
The target size is going down to the min size and after few more days,
the system is so slow, that I must reboot the machine. Then it is
running fine for about 107 days and then it all repeat again.
You can see more on MRTG graphs
http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/
You can see links to other useful informations on top of the page
(arc_summary, top, dmesg, fs usage, loader.conf)
There you can see nightly backups (higher CPU load started at 01:13),
otherwise the machine is idle.
It coresponds with ARC target size lowering in last 5 days
http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html
And with ARC metadata cache overflowing the limit in last 5 days
http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html
I don't know what's going on and I don't know if it is something know /
fixed in newer releases. We are running a few more ZFS systems on 8.2
without this issue. But those systems are in different roles.
Miroslav Lachman
Jeremy Chadwick
Feb 8, 2012, 7:28:35 PM2/8/12
to
"counter" or "ticks" thing within the ZFS code that was discovered to
only begin happening after a certain period of time (which correlated to
some number of days, possibly 107). I'm sorry that I can't be more
specific, but it's been discussed heavily on the lists in the past, and
fixes for all of that were committed to RELENG_8. I wish I could
remember the name of the function or macro or variable name it pertained
to, something like LTHAW or TLOCK or something like that. I would say
"I don't know why I can't remember", but I do know why I can't remember:
because I gave up trying to track all of these problems.
Does someone else remember this issue? CC'ing Martin who might remember
for certain.
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |
Artem Belevich
Feb 8, 2012, 7:43:48 PM2/8/12
to
On Wed, Feb 8, 2012 at 4:28 PM, Jeremy Chadwick
<fre...@jdc.parodius.com> wrote:
> On Thu, Feb 09, 2012 at 01:11:36AM +0100, Miroslav Lachman wrote:
...
<fre...@jdc.parodius.com> wrote:
> On Thu, Feb 09, 2012 at 01:11:36AM +0100, Miroslav Lachman wrote:
And there was more than one related integer overflow. One of them
manifested itself as L2ARC feeding thread hogging CPU time after about
a month of uptime. Another one caused issue with ARC reclaim after 107
days. See more details in this thread:
http://lists.freebsd.org/pipermail/freebsd-fs/2011-May/011584.html
--Artem
Miroslav Lachman
Feb 8, 2012, 8:20:35 PM2/8/12
to
I will upgrade this last old machine in few weeks. :)
>> I wish I could
>> remember the name of the function or macro or variable name it pertained
>> to, something like LTHAW or TLOCK or something like that. I would say
>> "I don't know why I can't remember", but I do know why I can't remember:
>> because I gave up trying to track all of these problems.
>>
>> Does someone else remember this issue? CC'ing Martin who might remember
>> for certain.
>
> It's LBOLT. :-)
>
> And there was more than one related integer overflow. One of them
> manifested itself as L2ARC feeding thread hogging CPU time after about
> a month of uptime. Another one caused issue with ARC reclaim after 107
> days. See more details in this thread:
>
> http://lists.freebsd.org/pipermail/freebsd-fs/2011-May/011584.html
I am subscribed to freebsd-fs@ and I am reading it almost daily, but I
missed this one!
Thanks to both of you!
Miroslav Lachman
Charles Sprickman
Feb 8, 2012, 8:05:24 PM2/8/12
to
> ARC Size:
> Current Size: 1769 MB (arcsize)
> Target Size (Adaptive): 512 MB (c)
> Min Size (Hard Limit): 512 MB (zfs_arc_min)
> Max Size (Hard Limit): 3584 MB (zfs_arc_max)
>
> The target size is going down to the min size and after few more days, the system is so slow, that I must reboot the machine. Then it is running fine for about 107 days and then it all repeat again.
>
> You can see more on MRTG graphs
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/
> You can see links to other useful informations on top of the page (arc_summary, top, dmesg, fs usage, loader.conf)
>
> There you can see nightly backups (higher CPU load started at 01:13), otherwise the machine is idle.
>
> It coresponds with ARC target size lowering in last 5 days
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html
>
> And with ARC metadata cache overflowing the limit in last 5 days
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html
I'm not having luck finding it, but there's some known issue that exists even in 8.2 where some 32-bit counter overflows or something. I don't truly remember the logic in it, but when you hit it, it's around 110 days or so. Before it gets really bad (to the point where you either reboot or get some memory exhaustion panic), you can see zfs "evict skips" incrementing rapidly. Looking at that graph, that would be my guess as to what's happening to you. It's easy to check - run one of the arc stats scripts, look for "evict_skips", note the number and then run it a few minutes later. If it increases by more than a few hundred, you've hit the bug. You'll find at that point the kernel is no longer "evicting" ARC from the kernel and it will just continue to grow until bad things happen.
> Current Size: 1769 MB (arcsize)
> Target Size (Adaptive): 512 MB (c)
> Min Size (Hard Limit): 512 MB (zfs_arc_min)
> Max Size (Hard Limit): 3584 MB (zfs_arc_max)
>
> The target size is going down to the min size and after few more days, the system is so slow, that I must reboot the machine. Then it is running fine for about 107 days and then it all repeat again.
>
> You can see more on MRTG graphs
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/
> You can see links to other useful informations on top of the page (arc_summary, top, dmesg, fs usage, loader.conf)
>
> There you can see nightly backups (higher CPU load started at 01:13), otherwise the machine is idle.
>
> It coresponds with ARC target size lowering in last 5 days
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html
>
> And with ARC metadata cache overflowing the limit in last 5 days
> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html
Charles
>
> I don't know what's going on and I don't know if it is something know / fixed in newer releases. We are running a few more ZFS systems on 8.2 without this issue. But those systems are in different roles.
>
Charles Sprickman
Feb 8, 2012, 8:45:11 PM2/8/12
to
On Feb 8, 2012, at 7:43 PM, Artem Belevich wrote:
> On Wed, Feb 8, 2012 at 4:28 PM, Jeremy Chadwick
> <fre...@jdc.parodius.com> wrote:
>> On Thu, Feb 09, 2012 at 01:11:36AM +0100, Miroslav Lachman wrote:
> ...
>>> ARC Size:
>>> Current Size: 1769 MB (arcsize)
>>> Target Size (Adaptive): 512 MB (c)
>>> Min Size (Hard Limit): 512 MB (zfs_arc_min)
>>> Max Size (Hard Limit): 3584 MB (zfs_arc_max)
>>>
>>> The target size is going down to the min size and after few more
>>> days, the system is so slow, that I must reboot the machine. Then it
>>> is running fine for about 107 days and then it all repeat again.
>>>
>>> You can see more on MRTG graphs
>>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/
>>> You can see links to other useful informations on top of the page
>>> (arc_summary, top, dmesg, fs usage, loader.conf)
>>>
>>> There you can see nightly backups (higher CPU load started at
>>> 01:13), otherwise the machine is idle.
>>>
>>> It coresponds with ARC target size lowering in last 5 days
>>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html
>>>
>>> And with ARC metadata cache overflowing the limit in last 5 days
>>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html
>>>
>>> Current Size: 1769 MB (arcsize)
>>> Target Size (Adaptive): 512 MB (c)
>>> Min Size (Hard Limit): 512 MB (zfs_arc_min)
>>> Max Size (Hard Limit): 3584 MB (zfs_arc_max)
>>>
>>> The target size is going down to the min size and after few more
>>> days, the system is so slow, that I must reboot the machine. Then it
>>> is running fine for about 107 days and then it all repeat again.
>>>
>>> You can see more on MRTG graphs
>>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/
>>> You can see links to other useful informations on top of the page
>>> (arc_summary, top, dmesg, fs usage, loader.conf)
>>>
>>> There you can see nightly backups (higher CPU load started at
>>> 01:13), otherwise the machine is idle.
>>>
>>> It coresponds with ARC target size lowering in last 5 days
>>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcstats_size.html
>>>
>>> And with ARC metadata cache overflowing the limit in last 5 days
>>> http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_meta.html
>>>
>>> I don't know what's going on and I don't know if it is something
>>> know / fixed in newer releases. We are running a few more ZFS
>>> systems on 8.2 without this issue. But those systems are in
>>> different roles.
>>
>>> know / fixed in newer releases. We are running a few more ZFS
>>> systems on 8.2 without this issue. But those systems are in
>>> different roles.
>>
>> This sounds like the... damn, what is it called... some kind of internal
>> "counter" or "ticks" thing within the ZFS code that was discovered to
>> only begin happening after a certain period of time (which correlated to
>> some number of days, possibly 107). I'm sorry that I can't be more
>> specific, but it's been discussed heavily on the lists in the past, and
>> fixes for all of that were committed to RELENG_8. I wish I could
>> "counter" or "ticks" thing within the ZFS code that was discovered to
>> only begin happening after a certain period of time (which correlated to
>> some number of days, possibly 107). I'm sorry that I can't be more
>> specific, but it's been discussed heavily on the lists in the past, and
>> remember the name of the function or macro or variable name it pertained
>> to, something like LTHAW or TLOCK or something like that. I would say
>> "I don't know why I can't remember", but I do know why I can't remember:
>> because I gave up trying to track all of these problems.
>>
>> Does someone else remember this issue? CC'ing Martin who might remember
>> for certain.
>
> It's LBOLT. :-)
>
> And there was more than one related integer overflow. One of them
> manifested itself as L2ARC feeding thread hogging CPU time after about
> a month of uptime. Another one caused issue with ARC reclaim after 107
> days. See more details in this thread:
>
> http://lists.freebsd.org/pipermail/freebsd-fs/2011-May/011584.html
This would be an excellent piece of information to have on one of the ZFS
>> to, something like LTHAW or TLOCK or something like that. I would say
>> "I don't know why I can't remember", but I do know why I can't remember:
>> because I gave up trying to track all of these problems.
>>
>> Does someone else remember this issue? CC'ing Martin who might remember
>> for certain.
>
> It's LBOLT. :-)
>
> And there was more than one related integer overflow. One of them
> manifested itself as L2ARC feeding thread hogging CPU time after about
> a month of uptime. Another one caused issue with ARC reclaim after 107
> days. See more details in this thread:
>
> http://lists.freebsd.org/pipermail/freebsd-fs/2011-May/011584.html
wiki pages. The 107 day issue exists post-8.2, correct? Anyone on this
cc: list have permissions to edit those pages?
Thanks,
Charles
>
> --Artem
Gary Palmer
Feb 8, 2012, 9:45:43 PM2/8/12
to
On Wed, Feb 08, 2012 at 11:18:02PM +0200, Andriy Gapon wrote:
> on 08/02/2012 22:50 Jeremy Chadwick said the following:
> > Politely -- recommending this to a user is a good choice of action, but
> > the problem is that no user, even an experienced user, is going to know
> > what all of the "Types" (vmstat -m) or "ITEMs" (vmstat -z) correlate
> > with on the system.
>
> I see no problem with users sharing the output and asking for help interpreting
> it. I do not know of any easier way to analyze problems like this one.
Also, since we are looking for gigs of memory it should be relatively easy
> on 08/02/2012 22:50 Jeremy Chadwick said the following:
> > Politely -- recommending this to a user is a good choice of action, but
> > the problem is that no user, even an experienced user, is going to know
> > what all of the "Types" (vmstat -m) or "ITEMs" (vmstat -z) correlate
> > with on the system.
>
> I see no problem with users sharing the output and asking for help interpreting
> it. I do not know of any easier way to analyze problems like this one.
to look down the 'Size' or 'MemUse' columns and identify likely candidates
for "eating gobs of memory". The user doesn't need to know what the rest
of the data means, and can ask what that line means and how to fix it
Gary
Eugene M. Zheganin
Feb 8, 2012, 11:27:33 PM2/8/12
to
Hi.
On 09.02.2012 02:29, Andriy Gapon wrote:
> on 08/02/2012 12:31 Eugene M. Zheganin said the following:
>> Hi.
>>
>> On 08.02.2012 02:17, Andriy Gapon wrote:
>>> [output snipped]
>>>
>>> Thank you. I don't see anything suspicious/unusual there.
>>> Just case, do you have ZFS dedup enabled by a chance?
>>>
>>> I think that examination of vmstat -m and vmstat -z outputs may provide some
>>> clues as to what got all that memory wired.
>>>
>> Nope, I don't have deduplication feature enabled.
> OK. So, did you have a chance to inspect vmstat -m and vmstat -z?
I did. I didn't understand it, but kinda 'felt the atmosphere'. It was
pretty much similar to the output I supplied below. Most of the sizes
were used by 'solaris' and numerous 'zio' caches.
>
> It could be very well possible that swap on zvol doesn't work well when the
> kernel itself is starved on memory.
>
>> So I want to ask - how to report it and what should I include in such pr ?
> I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be
> ZFS-related. I suspect that you might be running into some kernel memory leak.
> If you manage to reproduce the high wired value again, then vmstat -m and
> vmstat -z may provide some useful information.
>
> In this vein, do you use any out-of-tree kernel modules?
> Also, can you try to monitor your system to see when wired count grows?
>
Nope, I don't have any 3rd party kernel modules.
Yes, I can monitor it, but I have no idea what should I exactly monitor.
This system is running squid with a dozens of authentication helpers,
freeradius + postgresql, sendmail and a perl squid log parser, which
uses postgresql too. net/isc-dhcp, quagga, net/mpd5, a bunch of sendmail
milters, net/samba35, bind. So it's some kind of a corporate production
zoo. As I write this letter, the wired amount of memory increases by 70
megs. Excuse me, 80 megs now.
The output I promised (if it's MORE acceptable in the form of a link to
a paste site, just say it):
[emz@taiga:etc/snmp]# vmstat -m
Type InUse MemUse HighUse Requests Size(s)
hhook 2 1K - 2 128
ithread 85 14K - 85 32,128,256
KTRACE 100 13K - 100 128
linker 280 226K - 384
16,32,64,128,256,512,1024,2048,4096
lockf 94 10K - 20264872 64,128
loginclass 3 1K - 367 64
pci_link 13 2K - 13 16,128
ip6ndp 55 5K - 78 64,128
ip6opt 23 6K - 142134 32,256
temp 146 20K - 114199
16,32,64,128,256,512,1024,2048,4096
devbuf 28285 56235K - 29225
16,32,64,128,256,512,1024,2048,4096
module 291 37K - 291 128
USBdev 39 10K - 39 64,128,512,1024
mtx_pool 2 16K - 2
USB 55 166K - 58 16,32,64,128,256,512,2048,4096
osd 22 1K - 10870 16,64
ddb_capture 1 48K - 1
subproc 831 1312K - 56233 512,4096
proc 2 16K - 2
session 66 9K - 16431 128
pgrp 73 10K - 16581 128
cred 650 102K - 818736 64,256
uidinfo 15 4K - 5420 128,2048
plimit 25 7K - 4948 256
kbdmux 8 18K - 8 16,512,1024,2048
sysctltmp 0 0K - 9741241 16,32,64,128,4096
sysctloid 4837 243K - 4950 16,32,64,128
sysctl 0 0K - 50230 16,32,64
tidhash 1 16K - 1
callout 3 1536K - 3
umtx 2712 339K - 2766 128
p1003.1b 1 1K - 1 16
SWAP 2 1097K - 2 64
bus-sc 84 686K - 2193
16,32,64,128,256,512,1024,2048,4096
bus 861 78K - 4641 16,32,64,128,256,512,1024
devstat 4 9K - 4 32,4096
eventhandler 83 7K - 83 64,128
kobj 194 776K - 231 4096
Per-cpu 1 1K - 1 32
aacbuf 241 72K - 273 64,128,512
rman 219 23K - 449 16,32,128
acpiintr 1 1K - 1 64
sbuf 1 1K - 967
16,32,64,128,256,512,1024,2048,4096
acpica 1641 174K - 50289
16,32,64,128,256,512,1024,2048,4096
DEVFS1 106 53K - 111 512
DEVFS3 261 66K - 269 256
stack 0 0K - 2 256
taskqueue 85 8K - 121 16,32,64,128,1024
Unitno 21 1K - 208557 32,64
DEVFS2 106 2K - 108 16
DEVFS_RULE 54 26K - 54 64,512
DEVFS 39 1K - 40 16,128
Witness 1 128K - 1
iov 0 0K - 12708587 16,32,64,128,256,512
select 1081 136K - 1108 128
ioctlops 0 0K - 5846280
16,32,64,128,256,512,1024,2048,4096
msg 4 30K - 4 2048,4096
sem 4 106K - 4 2048,4096
shm 11 40K - 13055 2048
tty 23 23K - 29 1024,2048
pts 5 2K - 9 256
mbuf_tag 133 13K - 425972406 32,64,128
shmfd 1 8K - 1
DEVFSP 8 1K - 8 64
pcb 618 176K - 1042950 16,32,64,128,1024,2048,4096
soname 221 27K - 70492663 16,32,128
acl 0 0K - 1575 4096
vfscache 1 2048K - 1
vfs_hash 1 1024K - 1
vnodes 6 1K - 17 64,256
vnodemarker 0 0K - 72943 512
mount 256 16K - 1498 16,32,64,128,256,512
BPF 65 74K - 66 128,512,4096
ether_multi 564 32K - 786 16,32,64
ifaddr 330 93K - 339 16,32,64,128,256,512,2048,4096
ifnet 34 67K - 37 128,256,512,2048
clone 8 32K - 8 4096
arpcom 23 1K - 23 16
gif 1 1K - 1 256
lltable 767 209K - 15188 256,512
vlan 103 8K - 465 64,128
routetbl 840 45K - 13455 32,64,128,256,512
igmp 33 9K - 34 256
CARP 32 13K - 32 64,256,1024
ipid 2 24K - 2
ip_moptions 2 1K - 2 64,256
in_multi 23 6K - 24 256
in_mfilter 1 1K - 1 1024
encap_export_host 1 1K - 1 1024
sctp_iter 0 0K - 29 256
sctp_ifn 16 2K - 16 128
sctp_ifa 44 6K - 44 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 29 16
hostcache 1 28K - 1
syncache 1 96K - 1
fragment 0 0K - 18 64,128
ip6_moptions 10 2K - 10 32,256
in6_multi 188 28K - 196 32,256
in6_mfilter 5 5K - 5 1024
mld 33 5K - 34 128
NLM 0 0K - 1 32
rpc 64 8K - 398 16,32,64,128,256,512,1024,2048
audit_evclass 179 6K - 218 32
freework 1 1K - 1 64
newblk 1 128K - 1
bmsafemap 1 8K - 1
inodedep 1 1024K - 1
pagedep 1 128K - 1
vm_pgdata 2 129K - 2 128
UMAHash 2 5K - 6 512,1024,2048,4096
NFSD srvcache 0 0K - 91 128
pfs_nodes 21 6K - 21 256
acpitask 1 2K - 1 2048
memdesc 1 4K - 1 4096
GEOM 61 12K - 297 16,32,64,128,256,512,1024,2048
atkbddev 2 1K - 2 64
ata_pci 1 1K - 1 64
CAM dev queue 3 1K - 3 128
CAM queue 11 1K - 56 16,32
acpisem 16 2K - 16 128
CAM SIM 3 1K - 3 256
scsi_cd 0 0K - 4 16
CAM periph 4 1K - 18 16,32,64,128,256
isadev 7 1K - 7 128
entropy 1024 64K - 1024 64
apmdev 1 1K - 1 128
CAM XPT 29 14K - 80 32,64,128,1024,2048
UART 3 2K - 3 16,512,1024
cdev 8 2K - 8 256
io_apic 1 2K - 1 2048
sigio 1 1K - 3 64
filedesc 544 566K - 60333
16,32,64,128,256,512,1024,2048,4096
msi 2 1K - 2 128
nexusdev 3 1K - 3 16
kenv 104 12K - 108 16,32,64,128
kqueue 24 39K - 405980 256,2048,4096
acpidev 27 2K - 27 64
proc-args 219 24K - 449753 16,32,64,128,256
solaris 427482 2357264K - 228638905
16,32,64,128,256,512,1024,2048,4096
kstat_data 4 1K - 4 64
netgraph_msg 0 0K - 17743 64,128,256,512,1024
netgraph_node 11 3K - 26 256
netgraph_hook 14 2K - 46 128
netgraph 3 1K - 9 64,256,512,1024
netgraph_sock 5 1K - 10 128
netgraph_path 0 0K - 8912 16,32
netgraph_mppc 0 0K - 3 1024
netgraph_l2tp 2 3K - 2 128,2048
netgraph_ksock 1 1K - 2 128
netgraph_iface 1 1K - 2 128
netgraph_ppp 1 12K - 2
[emz@taiga:etc/snmp]# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 199, 5, 199, 0, 0
UMA Zones: 896, 0, 199, 1, 199, 0, 0
UMA Slabs: 568, 0, 38730, 827, 2045963, 0, 0
UMA RCntSlabs: 568, 0, 2417, 152, 10161, 0, 0
UMA Hash: 256, 0, 79, 11, 81, 0, 0
16 Bucket: 152, 0, 18, 132, 156, 0, 0
32 Bucket: 280, 0, 35, 119, 299, 0, 0
64 Bucket: 536, 0, 54, 72, 644, 57, 0
128 Bucket: 1048, 0, 467, 262, 112841,14481, 0
VM OBJECT: 216, 0, 39763, 2267, 1794570, 0, 0
MAP: 232, 0, 7, 25, 7, 0, 0
KMAP ENTRY: 120, 150412, 3666, 2131, 4810417, 0, 0
MAP ENTRY: 120, 0, 16746, 8457, 4147673, 0, 0
fakepg: 120, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 283, 77, 283, 0, 0
16: 16, 0, 65795, 565,103575464, 0, 0
32: 32, 0, 14388, 964,32512329, 0, 0
64: 64, 0, 164377, 14879,486891319, 0, 0
128: 128, 0, 17439, 2571,112770466, 0, 0
256: 256, 0, 12848, 15832,32654808, 0, 0
512: 512, 0, 149161, 4727, 6242680, 0, 0
1024: 1024, 0, 1970, 314, 151802, 0, 0
2048: 2048, 0, 1717, 349, 465184, 0, 0
4096: 4096, 0, 7621, 669, 375642, 0, 0
Files: 80, 0, 2210, 805,47573314, 0, 0
TURNSTILE: 136, 0, 1357, 83, 1384, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1160, 0, 259, 314, 55680, 0, 0
THREAD: 1112, 0, 864, 492, 53330, 0, 0
SLEEPQUEUE: 88, 0, 1357, 122, 1384, 0, 0
VMSPACE: 392, 0, 239, 681, 55662, 0, 0
cpuset: 72, 0, 2, 98, 2, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 2107, 709,137988001, 0, 0
mbuf: 256, 0, 58204, 1065,523117709, 0, 0
mbuf_cluster: 2048, 25600, 2816, 914, 7273137, 0, 0
mbuf_jumbo_page: 4096, 12800, 9, 543, 7683953, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 672, 29126, 0, 0
g_bio: 232, 0, 0, 752, 5733840, 0, 0
ttyinq: 160, 0, 240, 120, 555, 0, 0
ttyoutq: 256, 0, 127, 83, 293, 0, 0
ata_request: 328, 0, 0, 36, 14, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
taskq_zone: 48, 0, 0, 792, 69760, 0, 0
VNODE: 480, 0, 37315, 2269, 429291, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
S VFS Cache: 108, 0, 16761, 28350, 349225, 0, 0
L VFS Cache: 328, 0, 18, 1086, 13342, 0, 0
NAMEI: 1024, 0, 0, 384,29340521, 0, 0
NCLNODE: 560, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 0, 0, 0, 0, 0
zio_cache: 880, 0, 1, 1135,40366913, 0, 0
zio_link_cache: 48, 0, 0, 1440,11333105, 0, 0
zio_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_data_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_data_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_data_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_data_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_data_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_data_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_data_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_data_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_data_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_data_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_data_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_data_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_data_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_data_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_data_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_data_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_data_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_data_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_data_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_data_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_data_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_data_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_data_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_data_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_data_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_data_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_data_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_data_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_data_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_data_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_data_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_data_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_data_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_data_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_data_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_data_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_data_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_data_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_data_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_data_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_data_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_buf_131072: 131072, 0, 0, 0, 0, 0, 0
zio_data_buf_131072: 131072, 0, 0, 0, 0, 0, 0
sa_cache: 80, 0, 37248, 2262, 429178, 0, 0
dnode_t: 856, 0, 151801, 1919, 249658, 0, 0
dmu_buf_impl_t: 224, 0, 177337, 2404, 600337, 0, 0
arc_buf_hdr_t: 216, 0, 105487, 1253, 519289, 0, 0
arc_buf_t: 104, 0, 38391, 5205, 559830, 0, 0
zil_lwb_cache: 192, 0, 9, 611, 80899, 0, 0
zfs_znode_cache: 400, 0, 37248, 2325, 429178, 0, 0
pipe: 728, 0, 42, 328, 39330, 0, 0
Mountpoints: 768, 0, 20, 20, 20, 0, 0
ksiginfo: 112, 0, 564, 657, 29202, 0, 0
itimer: 344, 0, 1, 21, 1, 0, 0
KNOTE: 128, 0, 877, 776,90925709, 0, 0
pfsrctrpl: 152, 10000, 0, 0, 0, 0, 0
pfrulepl: 936, 0, 325, 11, 325, 0, 0
pfstatepl: 288, 10010, 3360, 2568, 1170450, 0, 0
pfstatekeypl: 288, 0, 3983, 2621, 1916481, 0, 0
pfstateitempl: 288, 0, 3983, 2439, 1439637, 0, 0
pfaltqpl: 240, 0, 0, 0, 0, 0, 0
pfpooladdrpl: 88, 0, 71, 55, 71, 0, 0
pfrktable: 1296, 1002, 6, 9, 11, 0, 0
pfrkentry: 160, 200016, 21, 27, 21, 0, 0
pffrent: 32, 5050, 0, 0, 0, 0, 0
pffrag: 80, 0, 0, 0, 0, 0, 0
pffrcache: 80, 10035, 0, 0, 0, 0, 0
pffrcent: 24, 50022, 0, 0, 0, 0, 0
pfstatescrub: 40, 0, 0, 0, 0, 0, 0
pfiaddrpl: 120, 0, 0, 0, 0, 0, 0
pfospfen: 112, 0, 700, 26, 700, 0, 0
pfosfp: 40, 0, 410, 94, 410, 0, 0
pfsync: 88, 0, 0, 0, 0, 0, 0
socket: 680, 25602, 1726, 596,25837811, 0, 0
ipq: 56, 819, 0, 126, 396, 0, 0
udp_inpcb: 392, 25600, 152, 728,20355207, 0, 0
udpcb: 16, 25704, 152, 856,20355207, 0, 0
tcp_inpcb: 392, 25600, 1782, 4968, 5284799, 0, 0
tcpcb: 976, 25600, 982, 762, 5284799, 0, 0
tcptw: 72, 5150, 800, 4350, 306578,97901, 0
syncache: 152, 15375, 0, 325, 5052872, 0, 0
hostcache: 136, 15372, 2053, 467, 7364, 0, 0
tcpreass: 40, 1680, 3, 585, 216370, 0, 0
sackhole: 32, 0, 0, 909, 55622, 0, 0
sctp_ep: 1368, 25600, 0, 0, 0, 0, 0
sctp_asoc: 2280, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 288, 43, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 25600, 6, 34, 8, 0, 0
unpcb: 240, 25600, 570, 550, 197771, 0, 0
rtentry: 200, 0, 756, 137, 1395, 0, 0
selfd: 56, 0, 1504, 701,705385407, 0, 0
SWAPMETA: 288, 116519, 4, 113, 866, 0, 0
NetGraph items: 72, 4118, 0, 812, 115049, 0, 0
NetGraph data items: 72, 522, 0, 493, 189271, 0, 0
P.S. after I pasted this, the wired memory gone down by 70 megs, the
final output (I guess enough is enough can be seen here:
http://dpaste.org/iA5mP/ ).
Thanks.
Eugene.
On 09.02.2012 02:29, Andriy Gapon wrote:
> on 08/02/2012 12:31 Eugene M. Zheganin said the following:
>> Hi.
>>
>> On 08.02.2012 02:17, Andriy Gapon wrote:
>>> [output snipped]
>>>
>>> Thank you. I don't see anything suspicious/unusual there.
>>> Just case, do you have ZFS dedup enabled by a chance?
>>>
>>> I think that examination of vmstat -m and vmstat -z outputs may provide some
>>> clues as to what got all that memory wired.
>>>
>> Nope, I don't have deduplication feature enabled.
> OK. So, did you have a chance to inspect vmstat -m and vmstat -z?
pretty much similar to the output I supplied below. Most of the sizes
were used by 'solaris' and numerous 'zio' caches.
>
> It could be very well possible that swap on zvol doesn't work well when the
> kernel itself is starved on memory.
>
>> So I want to ask - how to report it and what should I include in such pr ?
> I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be
> ZFS-related. I suspect that you might be running into some kernel memory leak.
> If you manage to reproduce the high wired value again, then vmstat -m and
> vmstat -z may provide some useful information.
>
> In this vein, do you use any out-of-tree kernel modules?
> Also, can you try to monitor your system to see when wired count grows?
>
Yes, I can monitor it, but I have no idea what should I exactly monitor.
This system is running squid with a dozens of authentication helpers,
freeradius + postgresql, sendmail and a perl squid log parser, which
uses postgresql too. net/isc-dhcp, quagga, net/mpd5, a bunch of sendmail
milters, net/samba35, bind. So it's some kind of a corporate production
zoo. As I write this letter, the wired amount of memory increases by 70
megs. Excuse me, 80 megs now.
The output I promised (if it's MORE acceptable in the form of a link to
a paste site, just say it):
[emz@taiga:etc/snmp]# vmstat -m
Type InUse MemUse HighUse Requests Size(s)
hhook 2 1K - 2 128
ithread 85 14K - 85 32,128,256
KTRACE 100 13K - 100 128
linker 280 226K - 384
16,32,64,128,256,512,1024,2048,4096
lockf 94 10K - 20264872 64,128
loginclass 3 1K - 367 64
pci_link 13 2K - 13 16,128
ip6ndp 55 5K - 78 64,128
ip6opt 23 6K - 142134 32,256
temp 146 20K - 114199
16,32,64,128,256,512,1024,2048,4096
devbuf 28285 56235K - 29225
16,32,64,128,256,512,1024,2048,4096
module 291 37K - 291 128
USBdev 39 10K - 39 64,128,512,1024
mtx_pool 2 16K - 2
USB 55 166K - 58 16,32,64,128,256,512,2048,4096
osd 22 1K - 10870 16,64
ddb_capture 1 48K - 1
subproc 831 1312K - 56233 512,4096
proc 2 16K - 2
session 66 9K - 16431 128
pgrp 73 10K - 16581 128
cred 650 102K - 818736 64,256
uidinfo 15 4K - 5420 128,2048
plimit 25 7K - 4948 256
kbdmux 8 18K - 8 16,512,1024,2048
sysctltmp 0 0K - 9741241 16,32,64,128,4096
sysctloid 4837 243K - 4950 16,32,64,128
sysctl 0 0K - 50230 16,32,64
tidhash 1 16K - 1
callout 3 1536K - 3
umtx 2712 339K - 2766 128
p1003.1b 1 1K - 1 16
SWAP 2 1097K - 2 64
bus-sc 84 686K - 2193
16,32,64,128,256,512,1024,2048,4096
bus 861 78K - 4641 16,32,64,128,256,512,1024
devstat 4 9K - 4 32,4096
eventhandler 83 7K - 83 64,128
kobj 194 776K - 231 4096
Per-cpu 1 1K - 1 32
aacbuf 241 72K - 273 64,128,512
rman 219 23K - 449 16,32,128
acpiintr 1 1K - 1 64
sbuf 1 1K - 967
16,32,64,128,256,512,1024,2048,4096
acpica 1641 174K - 50289
16,32,64,128,256,512,1024,2048,4096
DEVFS1 106 53K - 111 512
DEVFS3 261 66K - 269 256
stack 0 0K - 2 256
taskqueue 85 8K - 121 16,32,64,128,1024
Unitno 21 1K - 208557 32,64
DEVFS2 106 2K - 108 16
DEVFS_RULE 54 26K - 54 64,512
DEVFS 39 1K - 40 16,128
Witness 1 128K - 1
iov 0 0K - 12708587 16,32,64,128,256,512
select 1081 136K - 1108 128
ioctlops 0 0K - 5846280
16,32,64,128,256,512,1024,2048,4096
msg 4 30K - 4 2048,4096
sem 4 106K - 4 2048,4096
shm 11 40K - 13055 2048
tty 23 23K - 29 1024,2048
pts 5 2K - 9 256
mbuf_tag 133 13K - 425972406 32,64,128
shmfd 1 8K - 1
DEVFSP 8 1K - 8 64
pcb 618 176K - 1042950 16,32,64,128,1024,2048,4096
soname 221 27K - 70492663 16,32,128
acl 0 0K - 1575 4096
vfscache 1 2048K - 1
vfs_hash 1 1024K - 1
vnodes 6 1K - 17 64,256
vnodemarker 0 0K - 72943 512
mount 256 16K - 1498 16,32,64,128,256,512
BPF 65 74K - 66 128,512,4096
ether_multi 564 32K - 786 16,32,64
ifaddr 330 93K - 339 16,32,64,128,256,512,2048,4096
ifnet 34 67K - 37 128,256,512,2048
clone 8 32K - 8 4096
arpcom 23 1K - 23 16
gif 1 1K - 1 256
lltable 767 209K - 15188 256,512
vlan 103 8K - 465 64,128
routetbl 840 45K - 13455 32,64,128,256,512
igmp 33 9K - 34 256
CARP 32 13K - 32 64,256,1024
ipid 2 24K - 2
ip_moptions 2 1K - 2 64,256
in_multi 23 6K - 24 256
in_mfilter 1 1K - 1 1024
encap_export_host 1 1K - 1 1024
sctp_iter 0 0K - 29 256
sctp_ifn 16 2K - 16 128
sctp_ifa 44 6K - 44 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 29 16
hostcache 1 28K - 1
syncache 1 96K - 1
fragment 0 0K - 18 64,128
ip6_moptions 10 2K - 10 32,256
in6_multi 188 28K - 196 32,256
in6_mfilter 5 5K - 5 1024
mld 33 5K - 34 128
NLM 0 0K - 1 32
rpc 64 8K - 398 16,32,64,128,256,512,1024,2048
audit_evclass 179 6K - 218 32
freework 1 1K - 1 64
newblk 1 128K - 1
bmsafemap 1 8K - 1
inodedep 1 1024K - 1
pagedep 1 128K - 1
vm_pgdata 2 129K - 2 128
UMAHash 2 5K - 6 512,1024,2048,4096
NFSD srvcache 0 0K - 91 128
pfs_nodes 21 6K - 21 256
acpitask 1 2K - 1 2048
memdesc 1 4K - 1 4096
GEOM 61 12K - 297 16,32,64,128,256,512,1024,2048
atkbddev 2 1K - 2 64
ata_pci 1 1K - 1 64
CAM dev queue 3 1K - 3 128
CAM queue 11 1K - 56 16,32
acpisem 16 2K - 16 128
CAM SIM 3 1K - 3 256
scsi_cd 0 0K - 4 16
CAM periph 4 1K - 18 16,32,64,128,256
isadev 7 1K - 7 128
entropy 1024 64K - 1024 64
apmdev 1 1K - 1 128
CAM XPT 29 14K - 80 32,64,128,1024,2048
UART 3 2K - 3 16,512,1024
cdev 8 2K - 8 256
io_apic 1 2K - 1 2048
sigio 1 1K - 3 64
filedesc 544 566K - 60333
16,32,64,128,256,512,1024,2048,4096
msi 2 1K - 2 128
nexusdev 3 1K - 3 16
kenv 104 12K - 108 16,32,64,128
kqueue 24 39K - 405980 256,2048,4096
acpidev 27 2K - 27 64
proc-args 219 24K - 449753 16,32,64,128,256
solaris 427482 2357264K - 228638905
16,32,64,128,256,512,1024,2048,4096
kstat_data 4 1K - 4 64
netgraph_msg 0 0K - 17743 64,128,256,512,1024
netgraph_node 11 3K - 26 256
netgraph_hook 14 2K - 46 128
netgraph 3 1K - 9 64,256,512,1024
netgraph_sock 5 1K - 10 128
netgraph_path 0 0K - 8912 16,32
netgraph_mppc 0 0K - 3 1024
netgraph_l2tp 2 3K - 2 128,2048
netgraph_ksock 1 1K - 2 128
netgraph_iface 1 1K - 2 128
netgraph_ppp 1 12K - 2
[emz@taiga:etc/snmp]# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 199, 5, 199, 0, 0
UMA Zones: 896, 0, 199, 1, 199, 0, 0
UMA Slabs: 568, 0, 38730, 827, 2045963, 0, 0
UMA RCntSlabs: 568, 0, 2417, 152, 10161, 0, 0
UMA Hash: 256, 0, 79, 11, 81, 0, 0
16 Bucket: 152, 0, 18, 132, 156, 0, 0
32 Bucket: 280, 0, 35, 119, 299, 0, 0
64 Bucket: 536, 0, 54, 72, 644, 57, 0
128 Bucket: 1048, 0, 467, 262, 112841,14481, 0
VM OBJECT: 216, 0, 39763, 2267, 1794570, 0, 0
MAP: 232, 0, 7, 25, 7, 0, 0
KMAP ENTRY: 120, 150412, 3666, 2131, 4810417, 0, 0
MAP ENTRY: 120, 0, 16746, 8457, 4147673, 0, 0
fakepg: 120, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 283, 77, 283, 0, 0
16: 16, 0, 65795, 565,103575464, 0, 0
32: 32, 0, 14388, 964,32512329, 0, 0
64: 64, 0, 164377, 14879,486891319, 0, 0
128: 128, 0, 17439, 2571,112770466, 0, 0
256: 256, 0, 12848, 15832,32654808, 0, 0
512: 512, 0, 149161, 4727, 6242680, 0, 0
1024: 1024, 0, 1970, 314, 151802, 0, 0
2048: 2048, 0, 1717, 349, 465184, 0, 0
4096: 4096, 0, 7621, 669, 375642, 0, 0
Files: 80, 0, 2210, 805,47573314, 0, 0
TURNSTILE: 136, 0, 1357, 83, 1384, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1160, 0, 259, 314, 55680, 0, 0
THREAD: 1112, 0, 864, 492, 53330, 0, 0
SLEEPQUEUE: 88, 0, 1357, 122, 1384, 0, 0
VMSPACE: 392, 0, 239, 681, 55662, 0, 0
cpuset: 72, 0, 2, 98, 2, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 2107, 709,137988001, 0, 0
mbuf: 256, 0, 58204, 1065,523117709, 0, 0
mbuf_cluster: 2048, 25600, 2816, 914, 7273137, 0, 0
mbuf_jumbo_page: 4096, 12800, 9, 543, 7683953, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 672, 29126, 0, 0
g_bio: 232, 0, 0, 752, 5733840, 0, 0
ttyinq: 160, 0, 240, 120, 555, 0, 0
ttyoutq: 256, 0, 127, 83, 293, 0, 0
ata_request: 328, 0, 0, 36, 14, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
taskq_zone: 48, 0, 0, 792, 69760, 0, 0
VNODE: 480, 0, 37315, 2269, 429291, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
S VFS Cache: 108, 0, 16761, 28350, 349225, 0, 0
L VFS Cache: 328, 0, 18, 1086, 13342, 0, 0
NAMEI: 1024, 0, 0, 384,29340521, 0, 0
NCLNODE: 560, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 0, 0, 0, 0, 0
zio_cache: 880, 0, 1, 1135,40366913, 0, 0
zio_link_cache: 48, 0, 0, 1440,11333105, 0, 0
zio_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_data_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_data_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_data_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_data_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_data_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_data_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_data_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_data_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_data_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_data_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_data_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_data_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_data_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_data_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_data_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_data_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_data_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_data_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_data_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_data_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_data_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_data_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_data_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_data_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_data_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_data_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_data_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_data_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_data_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_data_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_data_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_data_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_data_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_data_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_data_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_data_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_data_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_data_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_data_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_data_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_data_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_buf_131072: 131072, 0, 0, 0, 0, 0, 0
zio_data_buf_131072: 131072, 0, 0, 0, 0, 0, 0
sa_cache: 80, 0, 37248, 2262, 429178, 0, 0
dnode_t: 856, 0, 151801, 1919, 249658, 0, 0
dmu_buf_impl_t: 224, 0, 177337, 2404, 600337, 0, 0
arc_buf_hdr_t: 216, 0, 105487, 1253, 519289, 0, 0
arc_buf_t: 104, 0, 38391, 5205, 559830, 0, 0
zil_lwb_cache: 192, 0, 9, 611, 80899, 0, 0
zfs_znode_cache: 400, 0, 37248, 2325, 429178, 0, 0
pipe: 728, 0, 42, 328, 39330, 0, 0
Mountpoints: 768, 0, 20, 20, 20, 0, 0
ksiginfo: 112, 0, 564, 657, 29202, 0, 0
itimer: 344, 0, 1, 21, 1, 0, 0
KNOTE: 128, 0, 877, 776,90925709, 0, 0
pfsrctrpl: 152, 10000, 0, 0, 0, 0, 0
pfrulepl: 936, 0, 325, 11, 325, 0, 0
pfstatepl: 288, 10010, 3360, 2568, 1170450, 0, 0
pfstatekeypl: 288, 0, 3983, 2621, 1916481, 0, 0
pfstateitempl: 288, 0, 3983, 2439, 1439637, 0, 0
pfaltqpl: 240, 0, 0, 0, 0, 0, 0
pfpooladdrpl: 88, 0, 71, 55, 71, 0, 0
pfrktable: 1296, 1002, 6, 9, 11, 0, 0
pfrkentry: 160, 200016, 21, 27, 21, 0, 0
pffrent: 32, 5050, 0, 0, 0, 0, 0
pffrag: 80, 0, 0, 0, 0, 0, 0
pffrcache: 80, 10035, 0, 0, 0, 0, 0
pffrcent: 24, 50022, 0, 0, 0, 0, 0
pfstatescrub: 40, 0, 0, 0, 0, 0, 0
pfiaddrpl: 120, 0, 0, 0, 0, 0, 0
pfospfen: 112, 0, 700, 26, 700, 0, 0
pfosfp: 40, 0, 410, 94, 410, 0, 0
pfsync: 88, 0, 0, 0, 0, 0, 0
socket: 680, 25602, 1726, 596,25837811, 0, 0
ipq: 56, 819, 0, 126, 396, 0, 0
udp_inpcb: 392, 25600, 152, 728,20355207, 0, 0
udpcb: 16, 25704, 152, 856,20355207, 0, 0
tcp_inpcb: 392, 25600, 1782, 4968, 5284799, 0, 0
tcpcb: 976, 25600, 982, 762, 5284799, 0, 0
tcptw: 72, 5150, 800, 4350, 306578,97901, 0
syncache: 152, 15375, 0, 325, 5052872, 0, 0
hostcache: 136, 15372, 2053, 467, 7364, 0, 0
tcpreass: 40, 1680, 3, 585, 216370, 0, 0
sackhole: 32, 0, 0, 909, 55622, 0, 0
sctp_ep: 1368, 25600, 0, 0, 0, 0, 0
sctp_asoc: 2280, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 288, 43, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 25600, 6, 34, 8, 0, 0
unpcb: 240, 25600, 570, 550, 197771, 0, 0
rtentry: 200, 0, 756, 137, 1395, 0, 0
selfd: 56, 0, 1504, 701,705385407, 0, 0
SWAPMETA: 288, 116519, 4, 113, 866, 0, 0
NetGraph items: 72, 4118, 0, 812, 115049, 0, 0
NetGraph data items: 72, 522, 0, 493, 189271, 0, 0
P.S. after I pasted this, the wired memory gone down by 70 megs, the
final output (I guess enough is enough can be seen here:
http://dpaste.org/iA5mP/ ).
Thanks.
Eugene.
Andriy Gapon
Feb 9, 2012, 3:33:58 AM2/9/12
to
on 09/02/2012 06:27 Eugene M. Zheganin said the following:
Just one more hint on the reporting. The most useful reports are coherent
reports. That is, I now have your older reports from top and zfs-stat and I
have newer vmstat reports. But I do not have all the reports taken at about the
same time, so I don't have a coherent picture of a system state.
--
Andriy Gapon
> The output I promised (if it's MORE acceptable in the form of a link to a paste
> site, just say it):
I prefer links, but both ways are acceptable to me.
> site, just say it):
Just one more hint on the reporting. The most useful reports are coherent
reports. That is, I now have your older reports from top and zfs-stat and I
have newer vmstat reports. But I do not have all the reports taken at about the
same time, so I don't have a coherent picture of a system state.
--
Andriy Gapon
Andriy Gapon
Feb 9, 2012, 3:35:44 AM2/9/12
to
on 09/02/2012 10:33 Andriy Gapon said the following:
large enough, like e.g. 1GB. That's when they are useful.
> on 09/02/2012 06:27 Eugene M. Zheganin said the following:
>> The output I promised (if it's MORE acceptable in the form of a link to a paste
>> site, just say it):
>
> I prefer links, but both ways are acceptable to me.
> Just one more hint on the reporting. The most useful reports are coherent
> reports. That is, I now have your older reports from top and zfs-stat and I
> have newer vmstat reports. But I do not have all the reports taken at about the
> same time, so I don't have a coherent picture of a system state.
>
And please take the reports after discrepancy between ARC size an wired size is
>> The output I promised (if it's MORE acceptable in the form of a link to a paste
>> site, just say it):
>
> I prefer links, but both ways are acceptable to me.
> Just one more hint on the reporting. The most useful reports are coherent
> reports. That is, I now have your older reports from top and zfs-stat and I
> have newer vmstat reports. But I do not have all the reports taken at about the
> same time, so I don't have a coherent picture of a system state.
>
large enough, like e.g. 1GB. That's when they are useful.
Eugene M. Zheganin
Feb 9, 2012, 5:41:20 AM2/9/12
to
Hi.
On 09.02.2012 14:35, Andriy Gapon wrote:
> And please take the reports after discrepancy between ARC size an wired size is
> large enough, like e.g. 1GB. That's when they are useful.
>
Okay, I wrote a short script capturing sequence of top -b/zfs-stats
-a/vmstat -m/vmstat -z in a timestamped file and put it in a crontab
every hour.
I will provide the files it creates (or a subset of files, if there will
be too many) after the system will enter a deadlock again.
This time varies from one week to two.
Thanks.
Eugene.
On 09.02.2012 14:35, Andriy Gapon wrote:
> And please take the reports after discrepancy between ARC size an wired size is
> large enough, like e.g. 1GB. That's when they are useful.
>
-a/vmstat -m/vmstat -z in a timestamped file and put it in a crontab
every hour.
I will provide the files it creates (or a subset of files, if there will
be too many) after the system will enter a deadlock again.
This time varies from one week to two.
Thanks.
Eugene.
Eugene M. Zheganin
Feb 9, 2012, 6:10:03 AM2/9/12
to
Hi.
On 09.02.2012 14:35, Andriy Gapon wrote:
> And please take the reports after discrepancy between ARC size an wired size is
> large enough, like e.g. 1GB. That's when they are useful.
>
One more thing - this machine is running a debug/ddb kernel, so just in
On 09.02.2012 14:35, Andriy Gapon wrote:
> And please take the reports after discrepancy between ARC size an wired size is
> large enough, like e.g. 1GB. That's when they are useful.
>
order to save two weeks - when/if it will enter a deadlock, do you (or
anyone else) need crashdump or anything else I can provide using ddb in
a deadlock ?
0 new messages