TRIM/UNMAP/DISCARD via ATA Passthrough

[Jason A. Donenfeld]

Hi folks,

There's a huge quantity of USB enclosures flooding the market for
quite sometime that support UASP, which is very nice. This allows for
higher speeds by using the SCSI command set via the uas driver, and
offers various other advantages over the classic usb-storage driver.

These USB enclosures do not support SCSI UNMAP. However, they do
support ATA-passthrough, and `hdparm -I` shows that TRIM is supported.

I was wondering if it'd be possible to have the uas driver -- or
perhaps somewhere else in the stack -- fall back to using
ATA-passthrough-TRIM for UNMAP, so that discard can work properly.
AFAIK, the Windows drivers do exactly this.

If the answer is, "yes we want this! but we don't have the man power,"
please tell me where I can start looking, so that I can submit the
patch myself.

Thanks,
Jason

[Henrique de Moraes Holschuh]

On Mon, 12 Sep 2016, Jason A. Donenfeld wrote:
> I was wondering if it'd be possible to have the uas driver -- or
> perhaps somewhere else in the stack -- fall back to using
> ATA-passthrough-TRIM for UNMAP, so that discard can work properly.
> AFAIK, the Windows drivers do exactly this.
>
> If the answer is, "yes we want this! but we don't have the man power,"
> please tell me where I can start looking, so that I can submit the
> patch myself.

Whomever implements this, please add a "chicken bit" to allow the user
to disable it if the enclosure misbehaves.

--
Henrique Holschuh

[Martin K. Petersen]

>>>>> "Jason" == Jason A Donenfeld <Ja...@zx2c4.com> writes:

Jason,

Jason> These USB enclosures do not support SCSI UNMAP. However, they do
Jason> support ATA-passthrough, and `hdparm -I` shows that TRIM is
Jason> supported.

How do they signal that they support the passthrough?

--
Martin K. Petersen Oracle Linux Engineering

[Jason A. Donenfeld]

On Wed, Sep 14, 2016 at 8:37 PM, Martin K. Petersen

> How do they signal that they support the passthrough?

Through the usual SCSI ATA-passthrough interface, "SAT" (SCSI-ATA
Command Translation) -- ATA PASS THROUGH SCSI (16) and ATA PASS
THROUGH SCSI (12).

I can use hdparm to treat /dev/sdb as an ATA interface:

zx2c4@thinkpad ~ $ sudo hdparm -I /dev/sdb | grep TRIM
* Data Set Management TRIM supported

[Martin K. Petersen]

>>>>> "Jason" == Jason A Donenfeld <Ja...@zx2c4.com> writes:

Jason,

>> How do they signal that they support the passthrough?

Jason> Through the usual SCSI ATA-passthrough interface, "SAT" (SCSI-ATA
Jason> Command Translation) -- ATA PASS THROUGH SCSI (16) and ATA PASS
Jason> THROUGH SCSI (12).

But how do they signal that ATA passthrough is possible? Is there an ATA
Information VPD page? Is REPORT SUPPORTED OPERATION CODES supported?

We need really solid discovery data before we can entertain enabling
something like this.

[Jason A. Donenfeld]

Hi Martin,

On Thu, Sep 15, 2016 at 6:07 PM, Martin K. Petersen

> But how do they signal that ATA passthrough is possible? Is there an ATA
> Information VPD page? Is REPORT SUPPORTED OPERATION CODES supported?
>
> We need really solid discovery data before we can entertain enabling
> something like this.

`sg_opcodes` said invalid request, so I think there isn't REPORT
SUPPORTED OPERATION CODES, and `sg_vpd -p ai` came up illegal too.

However, sg_sat_identify worked reliably, which means a solid way of
probing this would be to send IDENTIFY DEVICE ATA via SG_ATA_16 or
SG_ATA_12.

Let me know and I can give you access to the hardware if you're curious.

Regards,
Jason

[Hannes Reinecke]

Sadly, that's not sufficient.
linux is not the only provider of an SATL (mpt3sas being the most
prominent other one).
And while they might support ATA_12/ATA_16, there is no indication that
you can pass DSM TRIM that way.

Cheers,

Hannes
--
Dr. Hannes Reinecke zSeries & Storage
ha...@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

[Austin S. Hemmelgarn]

On 2016-09-16 07:16, Hannes Reinecke wrote:
> On 09/15/2016 10:52 PM, Jason A. Donenfeld wrote:
>> Hi Martin,
>>
>> On Thu, Sep 15, 2016 at 6:07 PM, Martin K. Petersen
>>> But how do they signal that ATA passthrough is possible? Is there an ATA
>>> Information VPD page? Is REPORT SUPPORTED OPERATION CODES supported?
>>>
>>> We need really solid discovery data before we can entertain enabling
>>> something like this.
>>
>> `sg_opcodes` said invalid request, so I think there isn't REPORT
>> SUPPORTED OPERATION CODES, and `sg_vpd -p ai` came up illegal too.
>>
>> However, sg_sat_identify worked reliably, which means a solid way of
>> probing this would be to send IDENTIFY DEVICE ATA via SG_ATA_16 or
>> SG_ATA_12.
>>
>> Let me know and I can give you access to the hardware if you're curious.
>>
> Sadly, that's not sufficient.
> linux is not the only provider of an SATL (mpt3sas being the most
> prominent other one).
> And while they might support ATA_12/ATA_16, there is no indication that
> you can pass DSM TRIM that way.

So it's better to not support it at all than to support it on hardware
we can reliably identify?

I get that having feature parity is a good thing, but the discussion
isn't about providing support for all SATL devices, it's specifically
about UAS connected SATL devices. Last I checked, mpt3sas doesn't do
anything with UAS, which means it's kind of irrelevant WRT supporting
this for UAS devices.

It's pretty easy to tell that something is a UAS device (the uas driver
wouldn't be bound to it otherwise), so if we check that and then check
whether or not IDENTIFY DEVICE ATA works when sent via SG_ATA_16 or
SG_ATA_12, it should be relatively safe (ignoring of course the fact
that there will inevitably be some brain-dead hardware that for some
obscure reason translates the command into something that will corrupt
data). I've got three USB 3.0 UAS SATA adapters (all ASMedia branded
chips) that behave pretty much identically to what Jason is describing,
so it appears that at least one brand behaves this way in a reliable and
reproducible manner.

[James Bottomley]

We're getting a bit off topic on mptsas and it's eccentric SATL.

The point is, you're asking for UAS devices which each have an internal
SATL which you say potentially doesn't support discard. The three
problems we have are

1. How do we identify if the UAS SATL doesn't support discard.  If it
does, we really don't want to cause further SATL related issues by
bypassing it, so we need a way of telling this.
2. If the SATL doesn't support discard, will it reliably support the
ATA_12 or ATA_16 pass through (and which one) .. we need a way of
checking this because there are known SATLs that don't do pass
through.
3. How do we actually configure it?  Presumably if the SATL doesn't
support discard, it also doesn't give us the useful mode page
indications we use to configure TRIM, so we're going to have to do
some pass through discovery as well.

James

[Austin S. Hemmelgarn]

I assume by 'discard' here you're referring to SCSI UNMAP, as anything
that supports ATA_12 or ATA_16 pass through correctly will support ATA
TRIM/DISCARD on drives that support it. If that's the case, then:
1. If SCSI UNMAP fails, it doesn't support UNMAP. This is of course
non-trivial to verify safely (we pretty much have to assume it is
supported if we have no clear indication it isn't, and then switch based
on what happens the first time we try to use it).
2. Unless there are SATL's out there that write garbage to the device or
die when sent an ATA_12 or ATA_16 pass through command encapsulating an
ATA DEVICE IDENTIFY command, this isn't an issue. Even if such SATL's
exist, they can easily be blacklisted.
3. This isn't hard, a SATL which actually supports ATA pass through will
almost always pass through the mode page unmodified.

On the note of UAS SATL's, all of them that I've seen fall into one of
four categories:
1. Supports one or both of ATA_12 or ATA_16 pass through, and supports
passing through ATA TRIM/DISCARD, but not SCSI UNMAP.
2. Supports one of ATA_12 or ATA_16 pass through, and does not support
passing through ATA TRIM/DISCARD or translating SCSI UNMAP. All devices
I've seen that fit this will modify the ATA DEVICE IDENTIFY data so it
doesn't report DISCARD support, or will simply return an error for
DISCARD requests. I haven't seen any like this that were manufactured
after UAS became standardized.
3. Supports neither ATA_12 or ATA_16 pass through, and doesn't support
UNMAP.
4. Like type 1, except it supports both pass through commands, and also
properly translates SCSI UNMAP commands (I've only ever seen one of
these, have no idea what chipset it had, and it was insanely expensive
(upside of 300 USD)).
All we really can do anything about is category 1. Category 4 works
with the current drivers, and we can't properly support it on category 2
or 3. All three devices I have right now are in category 1, I know a
number of other people in a similar situation, and it sounds like Jason
has at least one such device as well. Given that Windows does this
(I've confirmed this with a hardware USB analyzer I borrowed from a
friend), and that I've not seen anything since the UAS spec was actually
released that falls into category 2 (and if I understand the spec
correctly, such a device is actually not compliant with it anyway), I
think it's probably safe to do this in Linux and just base the check on:
1. UAS (not some other SCSI transport) without UNMAP support.
2. Supports ATA_12 or ATA_16 pass through.
3. ATA DEVICE IDENTIFY via SAT indicates that the device supports
DISCARD/TRIM.
Then we'd be matching behavior on Windows, and should probably be
relatively safe.

[Jason A. Donenfeld]

On Fri, Sep 16, 2016 at 7:06 PM, Austin S. Hemmelgarn
<ahfer...@gmail.com> wrote:
> 1. UAS (not some other SCSI transport) without UNMAP support.
> 2. Supports ATA_12 or ATA_16 pass through.
> 3. ATA DEVICE IDENTIFY via SAT indicates that the device supports
> DISCARD/TRIM.
> Then we'd be matching behavior on Windows, and should probably be relatively
> safe.

This very clearly seems like a sensible plan. It's nice to learn that
someone with a USB sniffer (and not just a désassembler :P) confirms
this is what Windows is doing. I see no reason for Linux not to do the
same. Are there any remaining objections to this plan?

[James Bottomley]

discard is the block layer terminology it's mapped per transport to
UNMAP or WRITE SAME on SCSI and TRIM on ATA.

> If that's the case, then:
> 1. If SCSI UNMAP fails, it doesn't support UNMAP. This is of course
> non-trivial to verify safely (we pretty much have to assume it is
> supported if we have no clear indication it isn't, and then switch
> based on what happens the first time we try to use it).

It's not quite that simple: to get us to configure discard in the first
place, you have to indicate support in READ CAPACITY (16): the LBPME
bit. The chances are your UAS SATL isn't setting this.

> 2. Unless there are SATL's out there that write garbage to the device
> or die when sent an ATA_12 or ATA_16 pass through command

Yes, there are; the problems with USB devices that fail to speak
standard versions of SCSI are legion.

> encapsulating an ATA DEVICE IDENTIFY command, this isn't an issue.
> Even if such SATL's exist, they can easily be blacklisted.
> 3. This isn't hard, a SATL which actually supports ATA pass through
> will almost always pass through the mode page unmodified.

You mean the ATA Information VPD page? Yes, that's feasible because we
already queried the supported VPD pages, so we can tell if this one's
there.

> On the note of UAS SATL's, all of them that I've seen fall into one
> of four categories:
> 1. Supports one or both of ATA_12 or ATA_16 pass through, and
> supports passing through ATA TRIM/DISCARD, but not SCSI UNMAP.
> 2. Supports one of ATA_12 or ATA_16 pass through, and does not
> support passing through ATA TRIM/DISCARD or translating SCSI UNMAP.
> All devices I've seen that fit this will modify the ATA DEVICE
> IDENTIFY data so it doesn't report DISCARD support, or will simply
> return an error for DISCARD requests. I haven't seen any like this
> that were manufactured after UAS became standardized.
> 3. Supports neither ATA_12 or ATA_16 pass through, and doesn't
> support UNMAP.
> 4. Like type 1, except it supports both pass through commands, and
> also properly translates SCSI UNMAP commands (I've only ever seen one
> of these, have no idea what chipset it had, and it was insanely
> expensive (upside of 300 USD)).
> All we really can do anything about is category 1. Category 4 works
> with the current drivers, and we can't properly support it on
> category 2 or 3.

Right, but you need to make sure it continues to work.

> All three devices I have right now are in category 1, I know a

> number of other people in a similar situation, and it sounds like
> Jason has at least one such device as well. Given that Windows does
> this (I've confirmed this with a hardware USB analyzer I borrowed
> from a friend), and that I've not seen anything since the UAS spec
> was actually released that falls into category 2 (and if I understand
> the spec correctly, such a device is actually not compliant with it

> anyway), think it's probably safe to do this in Linux and just base

> the check on:
> 1. UAS (not some other SCSI transport) without UNMAP support.
> 2. Supports ATA_12 or ATA_16 pass through.
> 3. ATA DEVICE IDENTIFY via SAT indicates that the device supports
> DISCARD/TRIM.
> Then we'd be matching behavior on Windows, and should probably be
> relatively safe.

OK, well I think we've all expressed the concerns, let's see the patch.

James

[Austin S. Hemmelgarn]

I actually didn't know this. I'm not quite as knowledgeable about the
block layer as I probably should be, and definitely not as up-to-date as
I could be on the ATA and SCSI specs.

>
>> If that's the case, then:
>> 1. If SCSI UNMAP fails, it doesn't support UNMAP. This is of course
>> non-trivial to verify safely (we pretty much have to assume it is
>> supported if we have no clear indication it isn't, and then switch
>> based on what happens the first time we try to use it).
>
> It's not quite that simple: to get us to configure discard in the first
> place, you have to indicate support in READ CAPACITY (16): the LBPME
> bit. The chances are your UAS SATL isn't setting this.

OK, that makes sense. Given that though, is it known how something like
that may react if you tried to issue an UNMAP or WRITE SAME command when
it's not supported?

>
>> 2. Unless there are SATL's out there that write garbage to the device
>> or die when sent an ATA_12 or ATA_16 pass through command
>
>
> Yes, there are; the problems with USB devices that fail to speak
> standard versions of SCSI are legion.

>
>> encapsulating an ATA DEVICE IDENTIFY command, this isn't an issue.
>> Even if such SATL's exist, they can easily be blacklisted.
>> 3. This isn't hard, a SATL which actually supports ATA pass through
>> will almost always pass through the mode page unmodified.
>
> You mean the ATA Information VPD page? Yes, that's feasible because we
> already queried the supported VPD pages, so we can tell if this one's
> there.

I kind of got my terminology confused here, and didn't proof-read
properly. I'm not sure exactly what I was trying to refer to
originally, but what I meant was that pretty much all UAS SATL's I've
seen that support ATA pass through either have a proper ATA Information
VPD page, or properly pass through ATA DEVICE IDENTIFY and related commands.

>
>> On the note of UAS SATL's, all of them that I've seen fall into one
>> of four categories:
>> 1. Supports one or both of ATA_12 or ATA_16 pass through, and
>> supports passing through ATA TRIM/DISCARD, but not SCSI UNMAP.
>> 2. Supports one of ATA_12 or ATA_16 pass through, and does not
>> support passing through ATA TRIM/DISCARD or translating SCSI UNMAP.
>> All devices I've seen that fit this will modify the ATA DEVICE
>> IDENTIFY data so it doesn't report DISCARD support, or will simply
>> return an error for DISCARD requests. I haven't seen any like this
>> that were manufactured after UAS became standardized.
>> 3. Supports neither ATA_12 or ATA_16 pass through, and doesn't
>> support UNMAP.
>> 4. Like type 1, except it supports both pass through commands, and
>> also properly translates SCSI UNMAP commands (I've only ever seen one
>> of these, have no idea what chipset it had, and it was insanely
>> expensive (upside of 300 USD)).
>> All we really can do anything about is category 1. Category 4 works
>> with the current drivers, and we can't properly support it on
>> category 2 or 3.
>
> Right, but you need to make sure it continues to work.

Indeed, but if we're only trying this when UNMAP (or WRITE SAME) isn't
supported, then there's no reason to expect that it won't continue to
work. There's also the fact that I've only ever seen one of these,
don't have access to the hardware in question, and I don't even think
it's commercially available, so it will be very non-trivial to test
unless someone wants to write a UAS emulator for QEMU (or some other
VMM) that supports operating like this.