After using zfs quite a bit at my last job, on the Sun thumpers with
48 disks & Sol10/x64.
What I discovered there was trying to compress a large number of very
small files ensures it will grind to a halt eventually...
Now I've been trying it out at home.
This machine has 6 disks (Samsung 1Tb, 32 Mb cache) for zfs usage
connected through an Adaptec 16 port SATA RAID card, with 8Gb RAM & an
AMD 5600+ AM2 proc running Ubuntu Jaunty (9.04)
From reading the list and other google-found info, I've now got the
techarcana patches which means the write performance isn't completely
abysmal but I've still got some questions.
fuse is 2.8.0-pre2-0techarcana0 (from source)
zfs-fuse is 0.5.1-1ubuntu2.1 (from source)
So far, using bonnie++ without extra options, the fastest results have
been zfs stripe over 3x hardware mirrors, read/write caches disabled -
so very redundantm, but 50% disk space)
Now, supposedly with the 0.5x release of zfs-fuse, the disk cache
settings didn't matter, but that's not what I'm seeing.
Last night I ran some basic tests with a 6xraidz2, with & without
compression & disk caches enabled/disabled.
What I could see from the activity lights and zpool iostat, was that
enabling the caches killed the performance and left the drives
twiddling their thumbs so to speak, waiting around for 'something',
which I thought they weren't supposed to do any more?..
I still want to try the raidz & raid10 equivalent again with the new
packages, but my question here is:
Should I go with what appears to be obvious and keep the disk caches
off & compression on? or is there some combination that uses the read/
write caches that can give me better write performance? (as reading
seems fine).
Given that the array will be mainly large a/v files that don't change
often but not wanting to spend days waiting for them to copy...
bonnie results:
zfs raidz2 over 6 drives, compression off, all drive caches off
storey,16G,
10271,29,13157,3,11015,4,39397,64,100346,11,139.5,0,16,5315,9,13915,4,6097, 6,3417,8,20152,13,8727,10
as above, but with compression on
storey,16G,
9354,22,38643,6,23603,7,39716,66,148869,16,245.7,0,16,5604,8,12809,10,6884, 5,5815,11,5011,3,1863,2
compression off, all caches enabled
storey,16G,
6869,16,7092,1,5555,2,38013,68,95000,11,105.6,0,16,5319,10,9517,13,6441,8,2 831,5,4290,1,1569,1
compression on, all caches enabled
storey,16G,
6673,15,16552,4,10013,4,40495,71,154574,16,188.8,1,16,4519,9,10800,8,7908,1 1,4813,7,7942,6,2565,3
I can't reply to every point here, but it seems indeed a good idea to turn
the cache off at least partially:
There is a very complex cache system in zfs, but it works only if it works
in the kernel, here in fuse it doesn't have direct access to the cache
memory and it does its work in around 200 Mb at max.
+ fuse does already some caching at the file level, which means that when
zfs caches some user data it's very probably cached by fuse and the kernel
at the same time.
So it's probably a good idea to use
zfs set primarycache=metadata
and
zfs set secondarycache=metadata
on all the zfs pools (inherited in all the datasets).
And it reduces the memory consumption of zfs-fuse, at least at the begining.
Use zfs set primarycache/secondarycache=none to be sure to save as much
memory as possible.
All this would probably require some more testing though...
> After using zfs quite a bit at my last job, on the Sun thumpers with
> 48 disks & Sol10/x64.
> What I discovered there was trying to compress a large number of very
> small files ensures it will grind to a halt eventually...
> Now I've been trying it out at home.
> This machine has 6 disks (Samsung 1Tb, 32 Mb cache) for zfs usage
> connected through an Adaptec 16 port SATA RAID card, with 8Gb RAM & an
> AMD 5600+ AM2 proc running Ubuntu Jaunty (9.04)
> From reading the list and other google-found info, I've now got the
> techarcana patches which means the write performance isn't completely
> abysmal but I've still got some questions.
> fuse is 2.8.0-pre2-0techarcana0 (from source)
> zfs-fuse is 0.5.1-1ubuntu2.1 (from source)
> So far, using bonnie++ without extra options, the fastest results have
> been zfs stripe over 3x hardware mirrors, read/write caches disabled -
> so very redundantm, but 50% disk space)
> Now, supposedly with the 0.5x release of zfs-fuse, the disk cache
> settings didn't matter, but that's not what I'm seeing.
> Last night I ran some basic tests with a 6xraidz2, with & without
> compression & disk caches enabled/disabled.
> What I could see from the activity lights and zpool iostat, was that
> enabling the caches killed the performance and left the drives
> twiddling their thumbs so to speak, waiting around for 'something',
> which I thought they weren't supposed to do any more?..
> I still want to try the raidz & raid10 equivalent again with the new
> packages, but my question here is:
> Should I go with what appears to be obvious and keep the disk caches
> off & compression on? or is there some combination that uses the read/
> write caches that can give me better write performance? (as reading
> seems fine).
> Given that the array will be mainly large a/v files that don't change
> often but not wanting to spend days waiting for them to copy...
> bonnie results:
> zfs raidz2 over 6 drives, compression off, all drive caches off
> storey,16G,
> 10271,29,13157,3,11015,4,39397,64,100346,11,139.5,0,16,5315,9,13915,4,6097, 6,3417,8,20152,13,8727,10
> as above, but with compression on
> storey,16G,
If you have the FUSE caching patch on this list applied to your ZFS
sources, I think it would be wise to disable the ARC. After all,
looking up something in the ARC still costs you 2 context switches per
syscall
On 13 jul, 16:38, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> I can't reply to every point here, but it seems indeed a good idea to turn
> the cache off at least partially:
> There is a very complex cache system in zfs, but it works only if it works
> in the kernel, here in fuse it doesn't have direct access to the cache
> memory and it does its work in around 200 Mb at max.
> + fuse does already some caching at the file level, which means that when
> zfs caches some user data it's very probably cached by fuse and the kernel
> at the same time.
> So it's probably a good idea to use
> zfs set primarycache=metadata
> and
> zfs set secondarycache=metadata
> on all the zfs pools (inherited in all the datasets).
> And it reduces the memory consumption of zfs-fuse, at least at the begining.
> Use zfs set primarycache/secondarycache=none to be sure to save as much
> memory as possible.
> All this would probably require some more testing though...
> > After using zfs quite a bit at my last job, on the Sun thumpers with
> > 48 disks & Sol10/x64.
> > What I discovered there was trying to compress a large number of very
> > small files ensures it will grind to a halt eventually...
> > Now I've been trying it out at home.
> > This machine has 6 disks (Samsung 1Tb, 32 Mb cache) for zfs usage
> > connected through an Adaptec 16 port SATA RAID card, with 8Gb RAM & an
> > AMD 5600+ AM2 proc running Ubuntu Jaunty (9.04)
> > From reading the list and other google-found info, I've now got the
> > techarcana patches which means the write performance isn't completely
> > abysmal but I've still got some questions.
> > fuse is 2.8.0-pre2-0techarcana0 (from source)
> > zfs-fuse is 0.5.1-1ubuntu2.1 (from source)
> > So far, using bonnie++ without extra options, the fastest results have
> > been zfs stripe over 3x hardware mirrors, read/write caches disabled -
> > so very redundantm, but 50% disk space)
> > Now, supposedly with the 0.5x release of zfs-fuse, the disk cache
> > settings didn't matter, but that's not what I'm seeing.
> > Last night I ran some basic tests with a 6xraidz2, with & without
> > compression & disk caches enabled/disabled.
> > What I could see from the activity lights and zpool iostat, was that
> > enabling the caches killed the performance and left the drives
> > twiddling their thumbs so to speak, waiting around for 'something',
> > which I thought they weren't supposed to do any more?..
> > I still want to try the raidz & raid10 equivalent again with the new
> > packages, but my question here is:
> > Should I go with what appears to be obvious and keep the disk caches
> > off & compression on? or is there some combination that uses the read/
> > write caches that can give me better write performance? (as reading
> > seems fine).
> > Given that the array will be mainly large a/v files that don't change
> > often but not wanting to spend days waiting for them to copy...
> > bonnie results:
> > zfs raidz2 over 6 drives, compression off, all drive caches off
> > storey,16G,
> > 10271,29,13157,3,11015,4,39397,64,100346,11,139.5,0,16,5315,9,13915,4,6097, 6,3417,8,20152,13,8727,10
> > as above, but with compression on
> > storey,16G,
I can confirm that disabling the caches in ZFS-FUSE dumps an enormous
amount of memory. It goes somewhere from 750 MB resident to 17 MB
resident. Couple this with the caching patch that we wrote, and you
have a *winner*.
However, if you are using the last code drop that vvatashki left on
this list, you won't be able to disable the primarycache and
secondarycache properties in your datasets, because your child
datasets will "vanish" once you have done that.
On 13 jul, 21:29, Rudd-O <rud...@rudd-o.com> wrote:
> If you have the FUSE caching patch on this list applied to your ZFS
> sources, I think it would be wise to disable the ARC. After all,
> looking up something in the ARC still costs you 2 context switches per
> syscall
> On 13 jul, 16:38, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> > I can't reply to every point here, but it seems indeed a good idea to turn
> > the cache off at least partially:
> > There is a very complex cache system in zfs, but it works only if it works
> > in the kernel, here in fuse it doesn't have direct access to the cache
> > memory and it does its work in around 200 Mb at max.
> > + fuse does already some caching at the file level, which means that when
> > zfs caches some user data it's very probably cached by fuse and the kernel
> > at the same time.
> > So it's probably a good idea to use
> > zfs set primarycache=metadata
> > and
> > zfs set secondarycache=metadata
> > on all the zfs pools (inherited in all the datasets).
> > And it reduces the memory consumption of zfs-fuse, at least at the begining.
> > Use zfs set primarycache/secondarycache=none to be sure to save as much
> > memory as possible.
> > All this would probably require some more testing though...
> > > After using zfs quite a bit at my last job, on the Sun thumpers with
> > > 48 disks & Sol10/x64.
> > > What I discovered there was trying to compress a large number of very
> > > small files ensures it will grind to a halt eventually...
> > > Now I've been trying it out at home.
> > > This machine has 6 disks (Samsung 1Tb, 32 Mb cache) for zfs usage
> > > connected through an Adaptec 16 port SATA RAID card, with 8Gb RAM & an
> > > AMD 5600+ AM2 proc running Ubuntu Jaunty (9.04)
> > > From reading the list and other google-found info, I've now got the
> > > techarcana patches which means the write performance isn't completely
> > > abysmal but I've still got some questions.
> > > fuse is 2.8.0-pre2-0techarcana0 (from source)
> > > zfs-fuse is 0.5.1-1ubuntu2.1 (from source)
> > > So far, using bonnie++ without extra options, the fastest results have
> > > been zfs stripe over 3x hardware mirrors, read/write caches disabled -
> > > so very redundantm, but 50% disk space)
> > > Now, supposedly with the 0.5x release of zfs-fuse, the disk cache
> > > settings didn't matter, but that's not what I'm seeing.
> > > Last night I ran some basic tests with a 6xraidz2, with & without
> > > compression & disk caches enabled/disabled.
> > > What I could see from the activity lights and zpool iostat, was that
> > > enabling the caches killed the performance and left the drives
> > > twiddling their thumbs so to speak, waiting around for 'something',
> > > which I thought they weren't supposed to do any more?..
> > > I still want to try the raidz & raid10 equivalent again with the new
> > > packages, but my question here is:
> > > Should I go with what appears to be obvious and keep the disk caches
> > > off & compression on? or is there some combination that uses the read/
> > > write caches that can give me better write performance? (as reading
> > > seems fine).
> > > Given that the array will be mainly large a/v files that don't change
> > > often but not wanting to spend days waiting for them to copy...
> > > bonnie results:
> > > zfs raidz2 over 6 drives, compression off, all drive caches off
> > > storey,16G,
> > > 10271,29,13157,3,11015,4,39397,64,100346,11,139.5,0,16,5315,9,13915,4,6097, 6,3417,8,20152,13,8727,10
> > > as above, but with compression on
> > > storey,16G,
I experienced the children datasets vanishing in my version, I just
remounted them manually, and everything was fine after that. (don't know yet
if they will be mounted automatically after that).
My zfs-fuse still eats something like 70 Mb without the caches, I wonder
where that comes from.
Not sure about the caching patch you are talking about, maybe I missed it...
This one maybe ?
Improved the FUSE kernel caching of
attributes<http://git.rudd-o.com/zfs/rev/87fc119c7256>
But it's just about the timing of the cache, nothing huge here...
> I can confirm that disabling the caches in ZFS-FUSE dumps an enormous
> amount of memory. It goes somewhere from 750 MB resident to 17 MB
> resident. Couple this with the caching patch that we wrote, and you
> have a *winner*.
> However, if you are using the last code drop that vvatashki left on
> this list, you won't be able to disable the primarycache and
> secondarycache properties in your datasets, because your child
> datasets will "vanish" once you have done that.
> On 13 jul, 21:29, Rudd-O <rud...@rudd-o.com> wrote:
> > If you have the FUSE caching patch on this list applied to your ZFS
> > sources, I think it would be wise to disable the ARC. After all,
> > looking up something in the ARC still costs you 2 context switches per
> > syscall
> > On 13 jul, 16:38, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> > > I can't reply to every point here, but it seems indeed a good idea to
> turn
> > > the cache off at least partially:
> > > There is a very complex cache system in zfs, but it works only if it
> works
> > > in the kernel, here in fuse it doesn't have direct access to the cache
> > > memory and it does its work in around 200 Mb at max.
> > > + fuse does already some caching at the file level, which means that
> when
> > > zfs caches some user data it's very probably cached by fuse and the
> kernel
> > > at the same time.
> > > So it's probably a good idea to use
> > > zfs set primarycache=metadata
> > > and
> > > zfs set secondarycache=metadata
> > > on all the zfs pools (inherited in all the datasets).
> > > And it reduces the memory consumption of zfs-fuse, at least at the
> begining.
> > > Use zfs set primarycache/secondarycache=none to be sure to save as much
> > > memory as possible.
> > > All this would probably require some more testing though...
> > > > After using zfs quite a bit at my last job, on the Sun thumpers with
> > > > 48 disks & Sol10/x64.
> > > > What I discovered there was trying to compress a large number of very
> > > > small files ensures it will grind to a halt eventually...
> > > > Now I've been trying it out at home.
> > > > This machine has 6 disks (Samsung 1Tb, 32 Mb cache) for zfs usage
> > > > connected through an Adaptec 16 port SATA RAID card, with 8Gb RAM &
> an
> > > > AMD 5600+ AM2 proc running Ubuntu Jaunty (9.04)
> > > > From reading the list and other google-found info, I've now got the
> > > > techarcana patches which means the write performance isn't completely
> > > > abysmal but I've still got some questions.
> > > > fuse is 2.8.0-pre2-0techarcana0 (from source)
> > > > zfs-fuse is 0.5.1-1ubuntu2.1 (from source)
> > > > So far, using bonnie++ without extra options, the fastest results
> have
> > > > been zfs stripe over 3x hardware mirrors, read/write caches disabled
> -
> > > > so very redundantm, but 50% disk space)
> > > > Now, supposedly with the 0.5x release of zfs-fuse, the disk cache
> > > > settings didn't matter, but that's not what I'm seeing.
> > > > Last night I ran some basic tests with a 6xraidz2, with & without
> > > > compression & disk caches enabled/disabled.
> > > > What I could see from the activity lights and zpool iostat, was that
> > > > enabling the caches killed the performance and left the drives
> > > > twiddling their thumbs so to speak, waiting around for 'something',
> > > > which I thought they weren't supposed to do any more?..
> > > > I still want to try the raidz & raid10 equivalent again with the new
> > > > packages, but my question here is:
> > > > Should I go with what appears to be obvious and keep the disk caches
> > > > off & compression on? or is there some combination that uses the
> read/
> > > > write caches that can give me better write performance? (as reading
> > > > seems fine).
> > > > Given that the array will be mainly large a/v files that don't change
> > > > often but not wanting to spend days waiting for them to copy...
> > > > bonnie results:
> > > > zfs raidz2 over 6 drives, compression off, all drive caches off
> > > > storey,16G,
> 10271,29,13157,3,11015,4,39397,64,100346,11,139.5,0,16,5315,9,13915,4,6097, 6,3417,8,20152,13,8727,10
> > > > as above, but with compression on
> > > > storey,16G,
Yeah, with 0.5.0 there is no difference in RAM usage apparently.
Let's see if 0.5.1 heralds a change. I'm not interested in running
with ARC enabled anyway because my patches use the linux kernel cache
for dentries and blocks.
On Wed, Jul 15, 2009 at 6:03 AM, Rudd-O<rud...@rudd-o.com> wrote:
> Yeah, with 0.5.0 there is no difference in RAM usage apparently. > Let's see if 0.5.1 heralds a change. I'm not interested in running > with ARC enabled anyway because my patches use the linux kernel cache > for dentries and blocks.
Yes. The double caching thing -- ARC is caching, and the Linux kernel
is also caching. Supposedly the suggestions in this thread let us
turn off the primary and secondary ARC caches, but I see no difference
in RAM usage from ZFS (which we should obviously see grow if ZFS were
caching, which it is).
