ZFS UNMAP performance

[Danny Schales]

I'm seeing very slow performance with certain operations on a ZFS
filesystem built on ISCSI LUN's on a 10.0 system (new ISCSI
implementation). The issue appears to be with BIO_DELETE operations.
Monitoring the system with gstat shows expected times for read and write
operations, but deletes are in the multiple hundreds of milliseconds
under normal operation. Destroying a snapshot sends the times to
astronomical levels. sysctl says the system is using UNMAP for deletes:

kern.cam.da.0.delete_method: UNMAP

I searched and found where Oracle issued a performance alert for Solaris
11.1 where ZFS using UNMAP was in use. Here's a link to a blog
discussing it:

http://schalwad.blogspot.com/2013/12/solaris-111-zfs-write-performance.html


Is FreeBSD also impacted? If so, is there a fix or a workaround?

Thanks

Danny Schales

[Tom Evans]

FreeBSD is affected the same way Solaris is. IMHO, neither are
affected, it is merely a consequence of using a device with a poor
TRIM/UNMAP algorithm.

You can trivially tell ZFS to not use TRIM, or tweak the tunings to
suit your devices. I'm afraid I don't know what they all mean
precisely..

vfs.zfs.vdev.trim_on_init: 1
vfs.zfs.vdev.trim_max_bytes: 2147483648
vfs.zfs.vdev.trim_max_pending: 64
vfs.zfs.trim.enabled: 1
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.max_interval: 1

Of course, if you disable TRIM you will end up with new (but
different) poor performance characteristics. Probably.

Cheers

Tom
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

[Danny Schales]

The backend is an ISCSI LUN from a SAN (Dell Compellent). From
research, it seems the SAN *does* support SCSI UNMAP requests for
deletes, but the performance is horrible. I don't know if this is a
FreeBSD issue or a Compellent issue. I haven't seen the problem with
other devices, but I don't think anything else is using UNMAP (yet).

Danny

[Danny Schales]

On 03/11/2014 11:08, Danny Schales wrote:
> On 03/11/2014 10:48, Tom Evans wrote:
>> On Tue, Mar 11, 2014 at 3:28 PM, Danny Schales <d...@latech.edu> wrote:
>>> I'm seeing very slow performance with certain operations on a ZFS
>>> filesystem built on ISCSI LUN's on a 10.0 system (new ISCSI
>>> implementation). The issue appears to be with BIO_DELETE operations.
>>> Monitoring the system with gstat shows expected times for read and write
>>> operations, but deletes are in the multiple hundreds of milliseconds
>>> under normal operation. Destroying a snapshot sends the times to
>>> astronomical levels. sysctl says the system is using UNMAP for deletes:
>>>
>>> kern.cam.da.0.delete_method: UNMAP
>>>
>>> I searched and found where Oracle issued a performance alert for Solaris
>>> 11.1 where ZFS using UNMAP was in use. Here's a link to a blog
>>> discussing it:
>>>
>>> http://schalwad.blogspot.com/2013/12/solaris-111-zfs-write-performance.html
>>>
>>>
>>> Is FreeBSD also impacted? If so, is there a fix or a workaround?
>>

> The backend is an ISCSI LUN from a SAN (Dell Compellent). From
> research, it seems the SAN *does* support SCSI UNMAP requests for
> deletes, but the performance is horrible. I don't know if this is a
> FreeBSD issue or a Compellent issue. I haven't seen the problem with
> other devices, but I don't think anything else is using UNMAP (yet).
>
> Danny
>
>

Replying to myself...I note that the system is reporting that TRIM is
being used. Is this normal for non-SSD systems? There *is* SSD in the
system, but I'm pretty sure the system can't tell it's SSD (it's hidden
behind a Dell PERC card). The number of trim.successes is roughly
equivalent to the number of deletes reported by gstat for the ISCSI LUN
devices. Should the system be using TRIM for ISCSI LUNs?

kstat.zfs.misc.zio_trim.bytes: 232845656064
kstat.zfs.misc.zio_trim.success: 30810983
kstat.zfs.misc.zio_trim.unsupported: 809
kstat.zfs.misc.zio_trim.failed: 0

Danny

[Eric van Gyzen]

Sure, if the LUN (i.e. the storage controller) reports that it supports
TRIM/UNMAP. Note that this is completely unrelated to the type of disks
that provide the LUN's backing store.

> kstat.zfs.misc.zio_trim.bytes: 232845656064
> kstat.zfs.misc.zio_trim.success: 30810983
> kstat.zfs.misc.zio_trim.unsupported: 809
> kstat.zfs.misc.zio_trim.failed: 0
>
> Danny
>

[Danny Schales]

On 3/11/2014 5:31 PM, Eric van Gyzen wrote:
>> Replying to myself...I note that the system is reporting that TRIM is
>> being used. Is this normal for non-SSD systems? There *is* SSD in the
>> system, but I'm pretty sure the system can't tell it's SSD (it's hidden
>> behind a Dell PERC card). The number of trim.successes is roughly
>> equivalent to the number of deletes reported by gstat for the ISCSI LUN
>> devices. Should the system be using TRIM for ISCSI LUNs?
> Sure, if the LUN (i.e. the storage controller) reports that it supports
> TRIM/UNMAP. Note that this is completely unrelated to the type of disks
> that provide the LUN's backing store.
>
>> kstat.zfs.misc.zio_trim.bytes: 232845656064
>> kstat.zfs.misc.zio_trim.success: 30810983
>> kstat.zfs.misc.zio_trim.unsupported: 809
>> kstat.zfs.misc.zio_trim.failed: 0
>>
>> Danny
>>
>

Are there any risks to turning off TRIM to see if the performance
improves (other than the loss of space recovery)?

Danny

[Bob Bishop]

Hi,

If the backing store really is SSD then turning off TRIM should hurt write performance eventually (and read to a lesser extent).

--
Bob Bishop
r...@gid.co.uk

[Eric van Gyzen]

On 03/12/2014 05:01, Bob Bishop wrote:
> Hi,
>
> On 12 Mar 2014, at 00:28, Danny Schales <d...@LaTech.edu> wrote:
>
>> On 3/11/2014 5:31 PM, Eric van Gyzen wrote:
>>>> Replying to myself...I note that the system is reporting that TRIM is
>>>> being used. Is this normal for non-SSD systems? There *is* SSD in the
>>>> system, but I'm pretty sure the system can't tell it's SSD (it's hidden
>>>> behind a Dell PERC card). The number of trim.successes is roughly
>>>> equivalent to the number of deletes reported by gstat for the ISCSI LUN
>>>> devices. Should the system be using TRIM for ISCSI LUNs?
>>> Sure, if the LUN (i.e. the storage controller) reports that it supports
>>> TRIM/UNMAP. Note that this is completely unrelated to the type of disks
>>> that provide the LUN's backing store.
>>>
>>>> kstat.zfs.misc.zio_trim.bytes: 232845656064
>>>> kstat.zfs.misc.zio_trim.success: 30810983
>>>> kstat.zfs.misc.zio_trim.unsupported: 809
>>>> kstat.zfs.misc.zio_trim.failed: 0
>>>>
>>>> Danny
>>>>
>> Are there any risks to turning off TRIM to see if the performance
>> improves (other than the loss of space recovery)?
>>
>> Danny
> If the backing store really is SSD then turning off TRIM should hurt write performance eventually (and read to a lesser extent).

Correct. If it's not SSD, though, the loss of space recovery should be
the only risk. And I imagine your highest tier of storage is not SSD.

Eric