Double caching?

[Rudd-O]

Hey guys,

Double caching is the issue now. Here's the deal. As far as I can
assume, ZFS opens the partitions with O_DIRECT, right? OK, so I
modified the sources to make ZFS-FUSE cache 512 MB RAM (MAX_ARCSIZE)
and went into single-user mode to replace them files.

After I had replaced them, I catted three 350 MB files into /dev/
null. According to this, the ARC cache ought to be full to 512 and
indeed, zfs-fuse shows growth up to around 500 and so MBs RAM. But
that's not the problem. The problem is that the KERNEL CACHE went up
to 1.1 GB or so.

I 3 > drop_caches, try again. Same result.

I am using the FUSE patches for caching that I and Ricardo wrote in
collaboration, but as far as I understand them, this caching is only
for dentries. Or is it for actual block data too?

Ideas?

[rud...@rudd-o.com]

If I may reply to self,

I just tried the same cat trick with a recently compiled copy of ZFS
without the FUSE caching patch. same result, read data remains in
kernel cache. interesting to note, as soon as the zfsfuse process is
stopped, cache drops the entire gig of data read.

TLDR: double caching, data is being cached in kernel AND in the ARC,
whether FUSE is told to cache attr_timeout and entry_timeout, or not.

fuse 2.8.0pre1 here, for the big_writes thing.

[Jonathan Schmidt]

Interesting. Under FUSE I would assume it's more valuable to have the
kernel VFS cache block data instead of using the ZFS ARC (due to the
extra overhead getting data from userspace). Maybe the kernel cache is
misbehaving though? I don't really know what its expected behaviour is...

[rud...@rudd-o.com]

According to my understanding of Ricardo's assumptions, O_DIRECT ought
to prevent kernel caching. If this is false, as my experiments point
out, then I think it follows that ZFS-FUSE ought to ship with the ARC
disabled or only minimally enabled.

At the moment I have horrible I/O contention issues between the kernel
and ZFS fighting for I/O in their respective partitions; this is why I
am investigating the caching issue as a way to optimize performance.
I'm trying to see how I can fit yet another disk on this already very
very hardware-stuffed machine, just to see if I can run the root
filesystem on a disk completely separate from the ZFS-FUSE one. And,
of course, ionice doesn't help as I pointed out in my latest post
titled "Priority inversion?". It's unbearable. ZFS is a champ, don't
get me wrong, but the minute an app does heavy disk I/O, the machine
grinds to its knees.

You guys were really telling the naked truth when you recommended
against running ZFS along with UFS on the same disk.