L2ARC and ZIL on same SSD?

[Josh Beard]

Hello,

I have a file server running 9.1-RELEASE with 24 GB of RAM and a 6 TB zpool
spread across 12 500GB disks in two raidz1 and 2 spares. This server
houses hundreds of home directories for end-users and stays pretty busy
with both read and writes.

I'm planning to add two SSDs soon (128 GB each) to help performance and was
planning to mirror them. I was curious if having the L2ARC and ZIL on the
same disks (mirrored) is okay. With some searching, I haven't seen any
"official" word on this. I've seen some folks say not to and others
recommend it. Adding a second set of SSDs isn't desirable, as I'll start
cutting into my available storage capacity (16 bay system).

Any advice or experience with this?

Thanks,
Josh
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[Dmitry Morozovsky]

On Mon, 20 May 2013, Josh Beard wrote:

> I have a file server running 9.1-RELEASE with 24 GB of RAM and a 6 TB zpool
> spread across 12 500GB disks in two raidz1 and 2 spares. This server
> houses hundreds of home directories for end-users and stays pretty busy
> with both read and writes.
>
> I'm planning to add two SSDs soon (128 GB each) to help performance and was
> planning to mirror them. I was curious if having the L2ARC and ZIL on the
> same disks (mirrored) is okay. With some searching, I haven't seen any
> "official" word on this. I've seen some folks say not to and others
> recommend it. Adding a second set of SSDs isn't desirable, as I'll start
> cutting into my available storage capacity (16 bay system).
>
> Any advice or experience with this?

Well, no official words, but -- I use the following setup for our new database
server, very similar to yours (but RAID10-alike as it's database, not generic
storage):

root@briareus:/usr/local/etc# zpool status
pool: br
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Fri Apr 26 15:29:32 2013
config:

NAME STATE READ WRITE CKSUM
br ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/br0 ONLINE 0 0 0
gpt/br4 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gpt/br1 ONLINE 0 0 0
gpt/br5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gpt/br6 ONLINE 0 0 0
gpt/br2 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
gpt/br3 ONLINE 0 0 0
gpt/br7 ONLINE 0 0 0
logs
mirror-4 ONLINE 0 0 0
gpt/br-zil0 ONLINE 0 0 0
gpt/br-zil1 ONLINE 0 0 0
cache
gpt/br-cache0 ONLINE 0 0 0
gpt/br-cache1 ONLINE 0 0 0

errors: No known data errors
root@briareus:/usr/local/etc# glabel status | egrep 'zil|cache'
gpt/br-zil0 N/A da8p1
gpt/br-cache0 N/A da8p2
gpt/br-zil1 N/A da9p1
gpt/br-cache1 N/A da9p2
root@briareus:/usr/local/etc# gpart show da8 da9
=> 34 234371453 da8 GPT (111G)
34 2014 - free - (1M)
2048 16777216 1 freebsd-zfs (8.0G)
16779264 217591808 2 freebsd-zfs (103G)
234371072 415 - free - (207k)

=> 34 234371453 da9 GPT (111G)
34 2014 - free - (1M)
2048 16777216 1 freebsd-zfs (8.0G)
16779264 217591808 2 freebsd-zfs (103G)
234371072 415 - free - (207k)

... and it's working pretty well, givins us some tens of kIOPs on database
writes.

--
Sincerely,
D.Marck [DM5020, MCK-RIPE, DM3-RIPN]
[ FreeBSD committer: ma...@FreeBSD.org ]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru ***
------------------------------------------------------------------------

[Daniel Kalchev]

On 20.05.13 17:53, Josh Beard wrote:
> Hello,

>
> I have a file server running 9.1-RELEASE with 24 GB of RAM and a 6 TB zpool
> spread across 12 500GB disks in two raidz1 and 2 spares. This server
> houses hundreds of home directories for end-users and stays pretty busy
> with both read and writes.
>
> I'm planning to add two SSDs soon (128 GB each) to help performance and was
> planning to mirror them. I was curious if having the L2ARC and ZIL on the
> same disks (mirrored) is okay. With some searching, I haven't seen any
> "official" word on this. I've seen some folks say not to and others
> recommend it. Adding a second set of SSDs isn't desirable, as I'll start
> cutting into my available storage capacity (16 bay system).
>
> Any advice or experience with this?
>

Indeed, there is no official position on how SSDs should be used with
ZFS, because SSDs are so much different from each other. Here is what
you should consider, however:

There is no need to mirror the L2ARC. On failure, it is simply ignored.
Therefore, for L2ARC in general you put as large SSD as you find, that
has good read rate. You might want to have this on a separate
controller, which provides the highest bandwidth.

There is suggestion to mirror the ZIL, in order to avoid pool damage,
but in post-v28 pools, the ZIL can be removed and replaced on the fly.
Since the ZIL is only ever read after crash, it all boils down what your
recovery scenario is. If you can tolerate the loss of most recent data,
you really do not need to mirror ZIL.

In any case, it is a bad idea to mirror the SSDs before giving them to
ZFS. Let ZFS manage it's redundancy, by giving it either the raw devices
or raw slices/partittions.

You also need to start thinking of SSDs in ZFS as "expense". You are
wearing it during use and at some point you will have to replace the
SSD. That is, you trade SSD wear for performance. L2ARC and ZIL have
quite different wearing characteristics.

In theory, ZIL would work best if it is either an high endurance SLC
flash drive, or if you use only small portion of it for writes. For
example, use only 8GB out of an 30GB (or larger) SSD for ZIL.

L2ARC, if sufficiently large would wear much less, because it will be
mostly used for reads. But, if your L2ARC is smaller than required -- it
will be overwriten many many times and thus wear faster. In both cases
you can monitor the "Media Wearout Indicator" of SMART to see how much
life you have left.

So considering especially the "expense" concept, you may want to have
one higher performance but small SSD for ZIL and another lower
performance but cheap SSD for L2ARC. If your SSD has sufficient
read/write performance, you might use slices of the same SSD for ZIL and
L2ARC. But consider consumer grade "performance" SSD wear faster. One
good things is that as SSDs wear, larger and cheaper SSDs will be
available :)

I don't know how much this helps -- you need to put it into perspective
for your workload and other constraints.

Daniel