Suggest changing dirhash defaults for FreeBSD 9.2.

[Robert Burmeister]

I have been experimenting with dirhash settings, and have scoured the internet
for other peoples' experience with it.
(I found the performance improvement in compiling has forestalled the need to
add an SSD drive. ;-)

I believe that increasing the following values by 10 would benefit most FreeBSD
users without disadvantage.

vfs.ufs.dirhash_maxmem: 2097152 to 20971520

vfs.ufs.dirhash_reclaimage: 5 to 50 or 60

FreeBSD administrators who have adjusted vfs.ufs.dirhash_maxmem for file server
use have found
8 megs to be about minimum to satisfy vfs.ufs.dirhash_mem usage,
while I have found compiling larger packages such as Firefox brings
vfs.ufs.dirhash_mem up to about 13 megs.

Setting vfs.ufs.dirhash_maxmem much larger than 20 megs results in
vfs.ufs.dirhash_lowmemcount events
scavenging excessive amounts of the directory cache, at 10% per event.

I believe vfs.ufs.dirhash_reclaimage of about a minute is sufficient to reclaim
memory without losing active directory cache,
particularly when compiling; however my experience in operations leads me to
believe this is also an optimal default for file servers.

As the worst case scenario is that the directory cache will be over cleansed and
the memory returned to the kernel,
I would like to suggest that these defaults be raised by a factor of 10 for
FreeBSD 9.2 or 10.0.


_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

[Mark Felder]

I've also toyed with dirhash on a few servers and received favorable
results. I've no idea where the defaults currently come from, but I'm
guessing probably around 1999 ;-)

[Sergey Kandaurov]

On 27 August 2013 16:41, Robert Burmeister

<Robert.B...@utoledo.edu> wrote:
>
> I have been experimenting with dirhash settings, and have scoured the internet for other peoples' experience with it.
> (I found the performance improvement in compiling has forestalled the need to add an SSD drive. ;-)
>
> I believe that increasing the following values by 10 would benefit most FreeBSD users without disadvantage.
>
> vfs.ufs.dirhash_maxmem: 2097152 to 20971520
>
> vfs.ufs.dirhash_reclaimage: 5 to 50 or 60

vfs.ufs.dirhash_maxmem is further autotuned based on available physical memory.
See r214359 for details.

--
wbr,
pluknet

[Robert Burmeister]

On 8/27/2013 9:40 AM, Sergey Kandaurov wrote:
> On 27 August 2013 16:41, Robert Burmeister
> <Robert.B...@utoledo.edu> wrote:
>> I have been experimenting with dirhash settings, and have scoured the internet for other peoples' experience with it.
>> (I found the performance improvement in compiling has forestalled the need to add an SSD drive. ;-)
>>
>> I believe that increasing the following values by 10 would benefit most FreeBSD users without disadvantage.
>>
>> vfs.ufs.dirhash_maxmem: 2097152 to 20971520
>>
>> vfs.ufs.dirhash_reclaimage: 5 to 50 or 60
> vfs.ufs.dirhash_maxmem is further autotuned based on available physical memory.
> See r214359 for details.
>

Sorry, the documentation has not been updated since 2008.
https://wiki.freebsd.org/DirhashDynamicMemory

What about bumping up vfs.ufs.dirhash_reclaimage
> all hashes older than 5 seconds (tunable via a sysctl,
> vfs.ufs.dirhash_reclaimage) will always be deleted when a vm_lowmem event
> occurs, and if that doesn't free up at least 10% of the memory currently being
> used by dirhash, more hashes will be destroyed from the head of the TAILQ
> until we do free up at least 10% of the memory initially used.
As the memory scavenger will keep nibbling until at least 10% of the cache is free,
eradicating all entries older than 5 seconds for the first chop seems overly
aggressive.

[Robert Burmeister]

On 8/27/2013 9:40 AM, Sergey Kandaurov wrote:

> On 27 August 2013 16:41, Robert Burmeister
> <Robert.B...@utoledo.edu> wrote:
>> I have been experimenting with dirhash settings, and have scoured the internet for other peoples' experience with it.
>> (I found the performance improvement in compiling has forestalled the need to add an SSD drive. ;-)
>>
>> I believe that increasing the following values by 10 would benefit most FreeBSD users without disadvantage.
>>
>> vfs.ufs.dirhash_maxmem: 2097152 to 20971520
>>
>> vfs.ufs.dirhash_reclaimage: 5 to 50 or 60
> vfs.ufs.dirhash_maxmem is further autotuned based on available physical memory.
> See r214359 for details.
>

[Spock Eyebrow of Thought]

I'm running FreeBSD i386 9.2, that allows a max of 4 Gigs of RAM.

I think the algorithm is still overly conservative for 32 bit systems,
which are more likely to be using UFS.

As 64 bit platforms tend to have more RAM and use ZFS,
is the same tuning algorithm appropriate for both?

[Ivan Voras]

On 28/08/2013 05:58, Robert Burmeister wrote:
>
> On 8/27/2013 9:40 AM, Sergey Kandaurov wrote:
>> On 27 August 2013 16:41, Robert Burmeister
>> <Robert.B...@utoledo.edu> wrote:
>>> I have been experimenting with dirhash settings, and have scoured the
>>> internet for other peoples' experience with it.
>>> (I found the performance improvement in compiling has forestalled the
>>> need to add an SSD drive. ;-)
>>>
>>> I believe that increasing the following values by 10 would benefit
>>> most FreeBSD users without disadvantage.
>>>
>>> vfs.ufs.dirhash_maxmem: 2097152 to 20971520
>>>
>>> vfs.ufs.dirhash_reclaimage: 5 to 50 or 60
>> vfs.ufs.dirhash_maxmem is further autotuned based on available
>> physical memory.
>> See r214359 for details.
>>
> [Spock Eyebrow of Thought]
>
> I'm running FreeBSD i386 9.2, that allows a max of 4 Gigs of RAM.

To what value does the algorithm tune in your case? On my 16 GB machine,
it's ~~ 25 MB:

vfs.ufs.dirhash_maxmem: 26968064

> I think the algorithm is still overly conservative for 32 bit systems,
> which are more likely to be using UFS.
>
> As 64 bit platforms tend to have more RAM and use ZFS,
> is the same tuning algorithm appropriate for both?

The policy is to use fractions of the installed RAM (though in a
roundabout way), so it should scale reasonably well to both systems with
large and small memories.

I'll bump vfs.ufs.dirhash_reclaimage to 60, it's worth it.

[Torfinn Ingolfsen]

On Tue, 27 Aug 2013 23:58:10 -0400
Robert Burmeister <Robert.B...@UToledo.edu> wrote:

>
> As 64 bit platforms tend to have more RAM and use ZFS,

Do you have any numbers for the "64 bit platforms tend to use ZFS"?
If not, I will suggest that this is just a theory.
FWIW, I haven't abandoned UFS on 64-bit platforms yet (I use ZFS were I need it).

Oh, and all people - please trim your quotes.

--
Torfinn Ingolfsen <torfinn....@getmail.no>

[Robert_Burmeister]

Torfinn Ingolfsen-5 wrote

> On Tue, 27 Aug 2013 23:58:10 -0400

> Robert Burmeister &lt;

> Robert.Burmeister@

> &gt; wrote:
>>
>> As 64 bit platforms tend to have more RAM and use ZFS,
> Do you have any numbers for the "64 bit platforms tend to use ZFS"?

PCBSD and OSX both default to ZFS for their 64 bit installs,
and anyone else wanting to use ZFS will choose 64 bit for the address space.

Do to the kernel memory constraints,
ZFS takes some configuration jiggering to work stably on 32 bit.




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5839745.html
Sent from the freebsd-stable mailing list archive at Nabble.com.

[Robert_Burmeister]

>>>> I believe that increasing the following values by 10 would benefit
>>>> most FreeBSD users without disadvantage.
>>>>
>>>> vfs.ufs.dirhash_maxmem: 2097152 to 20971520
>>>>
>>>> vfs.ufs.dirhash_reclaimage: 5 to 50 or 60
>>> vfs.ufs.dirhash_maxmem is further autotuned based on available
>>> physical memory.
>>> See r214359 for details.
>>>
>

> To what value does the algorithm tune in your case? On my 16 GB machine,
> it's ~~ 25 MB:
>
> vfs.ufs.dirhash_maxmem: 26968064
>

> The policy is to use fractions of the installed RAM (though in a
> roundabout way), so it should scale reasonably well to both systems with
> large and small memories.
>
> I'll bump vfs.ufs.dirhash_reclaimage to 60, it's worth it.

I'm running 2 Gigs of RAM, so it sets to 2 Megs.
As compiling most of the system uses ~1.5 Gigs, I can certainly use the
cache.

Note that allocated kernel memory does not scale linearly with physical RAM,
so a policy to use fractions of kernel memory may be more appropriate.

~25 Megs is a good value, however, as the reclaim value is a portion of
vfs.ufs.dirhash_maxmem, a much larger cache and it becomes pyric
as it can keep wiping out a small active cache.

(I tested vfs.ufs.dirhash_maxmem up to 60 megs.)




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5839755.html

Sent from the freebsd-stable mailing list archive at Nabble.com.

[Mark Saad]

On Aug 28, 2013, at 3:59 PM, Robert_Burmeister <robert.b...@utoledo.edu> wrote:

> Torfinn Ingolfsen-5 wrote
>> On Tue, 27 Aug 2013 23:58:10 -0400
>> Robert Burmeister &lt;
>
>> Robert.Burmeister@
>
>> &gt; wrote:
>>>
>>> As 64 bit platforms tend to have more RAM and use ZFS,
>> Do you have any numbers for the "64 bit platforms tend to use ZFS"?
>
> PCBSD and OSX both default to ZFS for their 64 bit installs,
> and anyone else wanting to use ZFS will choose 64 bit for the address space.
>

OSX abandoned zfs back in 10.7, I think you meant Solaris 10 update 5ish and newer use zfs root by default. fwiw I am using ufs on amd64 too and have only some machines using zfs .

> Do to the kernel memory constraints,
> ZFS takes some configuration jiggering to work stably on 32 bit.
>
>
>
>
> --
> View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5839745.html
> Sent from the freebsd-stable mailing list archive at Nabble.com.
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

Mark saad | mark...@longcount.org

[Robert_Burmeister]

Ivan Voras-7 wrote

> On 28/08/2013 05:58, Robert Burmeister wrote:
>>
>> On 8/27/2013 9:40 AM, Sergey Kandaurov wrote:
>>> On 27 August 2013 16:41, Robert Burmeister

>>> &lt;

> Robert.Burmeister@

> &gt; wrote:
>>>> I believe that increasing the following values by 10 would benefit
>>>> most FreeBSD users without disadvantage.
>>>> vfs.ufs.dirhash_maxmem: 2097152 to 20971520
>>>> vfs.ufs.dirhash_reclaimage: 5 to 50 or 60
>

> I'll bump vfs.ufs.dirhash_reclaimage to 60, it's worth it.

Note that vfs.ufs.dirhash_reclaimage is not used to shield ufs_dirhashmem
entries,
rather all entries older than reclaimage are evicted from the cache on
vm_lowmem
before evicting a percentage of the oldest remaining entries.

If it has been hours since a kernel vm_lowmem event,
most of the entries in the cache can be lost due to a momentary memory
spike.

Also note, under kernel memory pressure continuous vm_lowmem events
will effectively extinguish the ufs_dirhash cache, returning its memory to
the kernel.

I believe this prevents the ufsdirhash cache from in any way contributing to
a kernel panic,
so bumping vfs.ufs.dirhash_reclaimage to 60 should be riskless.




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5839769.html

Sent from the freebsd-stable mailing list archive at Nabble.com.

[Robert_Burmeister]

For previous benchmarks on the effect of the dirhash cache see:

https://wiki.freebsd.org/DirhashDynamicMemory




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5839775.html

Sent from the freebsd-stable mailing list archive at Nabble.com.

[Dewayne Geraghty]

Ivan,
From the analysis perforned in 2009, and referenced earlier by Robert, this
https://wiki.freebsd.org/DirhashDynamicMemory and other material at this site,
indicates that the reclaimage interval is workload dependent and that 5 to 8 seconds seems, on average, to be adequate.

We perform quite a detailed analysis before changing our sysctls for customers' servers that range from 32bit machines with between
1G and 4G of memory, and 64bit with 30G of memory, incidentally all are UFS2. Is the discussion, rather than (synthetic) workload
performance, sufficient to warrant changing the default settings by a factor of 12?

Regards, Dewayne.

[Robert_Burmeister]

Dewayne Geraghty-4 wrote

>> I'll bump vfs.ufs.dirhash_reclaimage to 60, it's worth it.
>

> From the analysis perforned in 2009, and referenced earlier by Robert,
> this
> https://wiki.freebsd.org/DirhashDynamicMemory and other material at this
> site,
> indicates that the reclaimage interval is workload dependent and that 5 to
> 8 seconds seems, on average, to be adequate.
>
> We perform quite a detailed analysis before changing our sysctls for
> customers' servers that range from 32bit machines with between
> 1G and 4G of memory, and 64bit with 30G of memory, incidentally all are
> UFS2. Is the discussion, rather than (synthetic) workload
> performance, sufficient to warrant changing the default settings by a
> factor of 12?
>
> Regards, Dewayne.

I recompile kernel, world and ~1200 ports for every FreeBSD i386 release
candidate to detect regressions.
Most ports use less than the 2 Gigs of RAM I have to compile,
but FireFox, LibreOffice, Sage and others will dig deep into virtual memory.

I have observed how the UFS dirhash cache performs during these tasks by
periodically running
sysctl -a | grep dirhash
and I believe it offers noticeable performance improvements for many common
tasks,
at no risk of loss of performance in other areas, or risk of a kernel panic.

I first noticed the UFS dirhash cache because it emulates an old feature of
Novel file servers.
Most OSs require one disk IO for file metadata and another to access the
file.
Novel used to store its file metadata in RAM to provide single disk IO file
access,
at the cost of long boot times.

Likewise, the UFS dirhash cache avoids a disk IO for file access when the
metadata is in the cache,
but only when there is excess memory available for the cache.
Since it is an odd on, it was not necessary to change the file system for
this increase in performance.

This is beneficial for any process requiring disk IO, but especially
compiling, mail servers, etc.
I have since migrated to SSDs and found the cache to leverage the low
latency of SSDs even more.

I have read the source code and observed the cache working with high, low
and borderline available RAM.
Hence determining a cache of > 13 megs is needed to avoid thrashing when
rebuilding the ports tree, for instance.

I have observed that
the UFS dirhash cache gracefully gives back its allocated memory to the
kernel before swapping begins,
abet with a noticeable drop in performance from dirhash_mem being reduced to
insignificance.
When kernel memory is again available, the cache will hold its entries
indefinitely.

Case: if 90% of the cache contains entries from five minutes ago,
why should the entries from the last minute that are being used by an active
job be lost
when dirhash_reclaimage = 60 would clear 90% of the cache on the first
vm_lowmem event?

In any case, clearing all cache entries older than 5 seconds on every
vm_lowmem event
keeps the cache from being useful in precisely the circumstances when it
would be of most benefit.

I suggested vfs.ufs.dirhash_reclaimage = 60 because most bursts of IO tend
to fit
within >5 - <60 seconds, and I can't think of a case for <=5 seconds.

I am interested in hearing other proposals,
but I think it shouldn't be difficult to agree that
vfs.ufs.dirhash_reclaimage = 5
is too short to really be useful for either servers or desktop users.

(Note that 5 seconds was chosen back when the feature was still experimental
in 2001.)





--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5839828.html

Sent from the freebsd-stable mailing list archive at Nabble.com.

[Ivan Voras]

On 29/08/2013 03:32, Dewayne Geraghty wrote:

> From the analysis perforned in 2009, and referenced earlier by Robert, this
> https://wiki.freebsd.org/DirhashDynamicMemory and other material at this site,
> indicates that the reclaimage interval is workload dependent and that 5 to 8 seconds seems, on average, to be adequate.

I'm having trouble understanding what the graphs are saying - you seem
to have almost consistently worse results with increasing dirhash memory
and/or reclaim age.

> Is the discussion, rather than (synthetic) workload
> performance, sufficient to warrant changing the default settings by a factor of 12?

Yes, and also for an additional reason on top of what I've already said
in @fs and @hackers: it will be at least 10 years before someone
remembers to bumb this tunable again, so I consider a little overkill to
be ok.

[Robert_Burmeister]

I have seen significant benefits from setting the UFS dirhash cache tuneables
to effective values,
and I believe all FreeBSD users will see at least a small benefit as well.

The ball for establishing production defaults appears to have been dropped
in 2008.
I am suggesting we pick it up and finish the job.
(But not necessarily in time for the impending release.)


> http://www.freebsd.org/projects/summerofcode-2008.html
> Project: Dynamic memory allocation for dirhash in UFS2
> Student: Nick Barkas
> Mentor: David Malone
> Summary:
>
> Modified dirhash code in perforce is now able to free up memory used by
> older dirhashes when the VM system invokes vm_lowmem events. This will
> allow the default dirhash_maxmem value to be increased, improving
> performance on large directory lookups when there is memory to spare on
> they system. There are versions of the low memory event handling code for
> both -CURRENT and 7-STABLE. A number of tests have been run showing the
> new event handler seems to work properly.
>
> I intend to do further testing and benchmarking to find the best default
> values to use for vfs.ufs.dirhash_reclaimage (the number of seconds a
> dirhash can sit unused before the dirhash low memeory event handler will
> unconditionally delete it) and the minimum percentage of memory that will
> be freed upon vm_lowmem events even if there are not enough hashes older
> than dirhash_reclaimage (currently this is hard coded to 10%). I would
> also like to add some code to choose a reasonable new default
> vfs.ufs.dirhash_maxmem value based upon the amount of memory in the
> system, set automatically at boot time and tunable via sysctl. Once these
> tweaks have been made I plan to ask for testing from more users to shake
> out any bugs or potential workloads where the new code may hurt overall
> performance.

If someone has more up to date information please post.




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5839900.html

Sent from the freebsd-stable mailing list archive at Nabble.com.

[Robert_Burmeister]

Here is a more recent dialog between the developers.

<quote
Nick Barkas
http://markmail.org/message/3sufphda2exjmhnq#query:+page:1+mid:3sufphda2exjmhnq+state:results

On Wed, May 27, 2009 at 12:36:49PM +0200, Nick Barkas wrote:

Some time during the next week or so, I plan on committing the attached
patch. It adds a vm_lowmem event handler to the dirhash code in UFS2 so that
dirhashes will be deleted when the system is low on memory. This allows one
to increase the maximum amount of memory available for dirhash on machines
that have memory to spare (via the vfs.ufs.dirhash_maxmem sysctl), and
hopefully just improving behaviour in low memory situations. I worked on
this last year for the summer of code with David Malone as my mentor.

cool! do you have any performance numbers? graphs? :) what value do you
recommend for the dirhash_maxmem sysctl?

Oh yes, I have many graphs: http://wiki.freebsd.org/DirhashDynamicMemory
When I ran those tests a few months ago, I used 64MB for dirhash_maxmem on a
system with 1GB of memory. I have not tried other amounts of memory besides
that, at least that I can recall, so please let me know what you find if you
experiment with other values. Performance improvements and sometimes
degradations changed depending on the type of work load, and the results on
7.x were also sometimes quite different from -current. I feel that the tests
I did were pretty artificial though, so it would be great to hear about any
results found with more realistic testing.

Nick
&lt;/quote>

<quote
David Malone
http://markmail.org/message/3sufphda2exjmhnq#query:+page:1+mid:aerievrnrmkezehk+state:results

On Wed, May 27, 2009 at 09:57:31AM -0700, Julian Elischer wrote:

I was initially impressed by the numbers until I saw the scales.. a
difference between 475.5 and 474 is not that significant, but if your graph
scale is from 473 to 477, it looks at first glace very impressive.

I think we felt the real gains here would be the ability to set the dirhash
memory limits to a much larger value without having to worry about it
chewing all your memory. The results basically confirm that we haven't
introduced any serious regressions ;-)

David.
&lt;/quote>



--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5840090.html

Sent from the freebsd-stable mailing list archive at Nabble.com.

[Robert_Burmeister]

After scouring the internet,
it seems that no one else has done a great deal of testing of UFS2 dirhash
defaults lately.

As the dirhash feature has effectively been tested for regressions,
I would like to propose setting the default dirhash values to my original
recommendation:

vfs.ufs.dirhash_maxmem: 20971520

vfs.ufs.dirhash_reclaimage: 60

The last benchmarks from 2008 only tested dirhash_maxmem at 2 and 64 megs.

I found in my testing that setting maxmem much over 32 megs makes the UFS
dirhash dysfunctional,
as scavenging 10% of maxmem on each vm_lowmem event becomes too aggressive.

While autotuning defaults is highly desirable, autotuning dirhash_maxmem
opens the can of worms of also tuning the hard coded 10% scavenging.

It may be better to hard code the maxmem default.


It may be appropriate to set the reclaimage to 5 minutes rather than 5
seconds;
I consider 60 seconds to still be a very conservative value, but at least an
effective one.




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5840115.html

Sent from the freebsd-stable mailing list archive at Nabble.com.

[Robert_Burmeister]

Observation relevant to tuning vfs.ufs.dirhash_maxmem:

When swap is in use
dirhash_mem hovers between 10% and 20% of dirhash_maxmem
due to frequent scavenging.

This indicates active dirhash_mem effectively behaves differently during low
memory,
and that the dirhash_reclaimage setting is effectively irrelevant during low
memory.




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5840292.html

[Robert_Burmeister]

Presentation describing the logic behind adding dynamic memory allocation to
UFS dirhash
can be found at:

"EuroBSDCon 2008 - Nick Barkas - Dynamic memory allocation for dirhash in
UFS2"
http://www.za.freebsd.org/multimedia/tag-nick_barkas.html




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5851625.html

Sent from the freebsd-stable mailing list archive at Nabble.com.

[Robert_Burmeister]

Additional observation:
When virtual memory is continually using swap
the vfs.ufs.dirhash_mem gets squished to 0
which is, I believe, the desired behavior.




--
View this message in context: http://freebsd.1045724.n5.nabble.com/Suggest-changing-dirhash-defaults-for-FreeBSD-9-2-tp5839351p5855675.html