0.5.1 & disk caches

[Sean Meacher]

After using zfs quite a bit at my last job, on the Sun thumpers with
48 disks & Sol10/x64.
What I discovered there was trying to compress a large number of very
small files ensures it will grind to a halt eventually...

Now I've been trying it out at home.
This machine has 6 disks (Samsung 1Tb, 32 Mb cache) for zfs usage
connected through an Adaptec 16 port SATA RAID card, with 8Gb RAM & an
AMD 5600+ AM2 proc running Ubuntu Jaunty (9.04)

From reading the list and other google-found info, I've now got the
techarcana patches which means the write performance isn't completely
abysmal but I've still got some questions.
fuse is 2.8.0-pre2-0techarcana0 (from source)
zfs-fuse is 0.5.1-1ubuntu2.1 (from source)

So far, using bonnie++ without extra options, the fastest results have
been zfs stripe over 3x hardware mirrors, read/write caches disabled -
so very redundantm, but 50% disk space)
Now, supposedly with the 0.5x release of zfs-fuse, the disk cache
settings didn't matter, but that's not what I'm seeing.

Last night I ran some basic tests with a 6xraidz2, with & without
compression & disk caches enabled/disabled.
What I could see from the activity lights and zpool iostat, was that
enabling the caches killed the performance and left the drives
twiddling their thumbs so to speak, waiting around for 'something',
which I thought they weren't supposed to do any more?..

I still want to try the raidz & raid10 equivalent again with the new
packages, but my question here is:

Should I go with what appears to be obvious and keep the disk caches
off & compression on? or is there some combination that uses the read/
write caches that can give me better write performance? (as reading
seems fine).
Given that the array will be mainly large a/v files that don't change
often but not wanting to spend days waiting for them to copy...

bonnie results:
zfs raidz2 over 6 drives, compression off, all drive caches off
storey,16G,
10271,29,13157,3,11015,4,39397,64,100346,11,139.5,0,16,5315,9,13915,4,6097,6,3417,8,20152,13,8727,10
as above, but with compression on
storey,16G,
9354,22,38643,6,23603,7,39716,66,148869,16,245.7,0,16,5604,8,12809,10,6884,5,5815,11,5011,3,1863,2
compression off, all caches enabled
storey,16G,
6869,16,7092,1,5555,2,38013,68,95000,11,105.6,0,16,5319,10,9517,13,6441,8,2831,5,4290,1,1569,1
compression on, all caches enabled
storey,16G,
6673,15,16552,4,10013,4,40495,71,154574,16,188.8,1,16,4519,9,10800,8,7908,11,4813,7,7942,6,2565,3

[Emmanuel Anne]

I can't reply to every point here, but it seems indeed a good idea to turn the cache off at least partially:

There is a very complex cache system in zfs, but it works only if it works in the kernel, here in fuse it doesn't have direct access to the cache memory and it does its work in around 200 Mb at max.
+ fuse does already some caching at the file level, which means that when zfs caches some user data it's very probably cached by fuse and the kernel at the same time.

So it's probably a good idea to use
zfs set primarycache=metadata
and
zfs set secondarycache=metadata
on all the zfs pools (inherited in all the datasets).

And it reduces the memory consumption of zfs-fuse, at least at the begining.
Use zfs set primarycache/secondarycache=none to be sure to save as much memory as possible.

All this would probably require some more testing though...

2009/7/13 Sean Meacher <gad...@gmail.com>

[Rudd-O]

If you have the FUSE caching patch on this list applied to your ZFS
sources, I think it would be wise to disable the ARC. After all,
looking up something in the ARC still costs you 2 context switches per
syscall

On 13 jul, 16:38, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> I can't reply to every point here, but it seems indeed a good idea to turn
> the cache off at least partially:
>
> There is a very complex cache system in zfs, but it works only if it works
> in the kernel, here in fuse it doesn't have direct access to the cache
> memory and it does its work in around 200 Mb at max.
> + fuse does already some caching at the file level, which means that when
> zfs caches some user data it's very probably cached by fuse and the kernel
> at the same time.
>
> So it's probably a good idea to use
> zfs set primarycache=metadata
> and
> zfs set secondarycache=metadata
> on all the zfs pools (inherited in all the datasets).
>
> And it reduces the memory consumption of zfs-fuse, at least at the begining.
> Use zfs set primarycache/secondarycache=none to be sure to save as much
> memory as possible.
>
> All this would probably require some more testing though...
>

> 2009/7/13 Sean Meacher <gadg...@gmail.com>

[Rudd-O]

I can confirm that disabling the caches in ZFS-FUSE dumps an enormous
amount of memory. It goes somewhere from 750 MB resident to 17 MB
resident. Couple this with the caching patch that we wrote, and you
have a *winner*.

However, if you are using the last code drop that vvatashki left on
this list, you won't be able to disable the primarycache and
secondarycache properties in your datasets, because your child
datasets will "vanish" once you have done that.

[Emmanuel Anne]

I experienced the children datasets vanishing in my version, I just remounted them manually, and everything was fine after that. (don't know yet if they will be mounted automatically after that).
My zfs-fuse still eats something like 70 Mb without the caches, I wonder where that comes from.

Not sure about the caching patch you are talking about, maybe I missed it...
This one maybe ?
Improved the FUSE kernel caching of attributes
But it's just about the timing of the cache, nothing huge here...

2009/7/14 Rudd-O <rud...@rudd-o.com>

[Rudd-O]

Yeah, with 0.5.0 there is no difference in RAM usage apparently.
Let's see if 0.5.1 heralds a change. I'm not interested in running
with ARC enabled anyway because my patches use the linux kernel cache
for dentries and blocks.

[Fajar A. Nugraha]

Do you need a patch to enable linux kernel cache? Didn't you post
something about double caching sometime ago?
http://groups.google.com/group/zfs-fuse/browse_thread/thread/4cba8d16a31f2d6e

--
Fajar

[Rudd-O]

Yes. The double caching thing -- ARC is caching, and the Linux kernel
is also caching. Supposedly the suggestions in this thread let us
turn off the primary and secondary ARC caches, but I see no difference
in RAM usage from ZFS (which we should obviously see grow if ZFS were
caching, which it is).