Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Re: seeing data corruption with zfs trim functionality
2 views
Skip to first unread message
Steven Hartland
May 14, 2013, 5:10:10 PM5/14/13
to
----- Original Message -----
From: "Steven Hartland"
>>> What version are you porting the changes to?
>>>
>>> What SSD are you using?
>>>
>>> What LSI controller are you using?
>>
>> I'd also like to see "zpool status" (for every pool that involves this
>> SSD) and "gpart show" against the disk itself.
>
> Also:
> 1. What FW version is your LSI? You can get this from dmesg.
> 2. The exact command line your running iotest with?
Any update on this? I'd like to try and replicate your test here so
would appreciate as much information as possible.
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postm...@multiplay.co.uk.
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"
From: "Steven Hartland"
>>> What version are you porting the changes to?
>>>
>>> What SSD are you using?
>>>
>>> What LSI controller are you using?
>>
>> I'd also like to see "zpool status" (for every pool that involves this
>> SSD) and "gpart show" against the disk itself.
>
> Also:
> 1. What FW version is your LSI? You can get this from dmesg.
> 2. The exact command line your running iotest with?
Any update on this? I'd like to try and replicate your test here so
would appreciate as much information as possible.
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postm...@multiplay.co.uk.
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"
Steven Hartland
May 14, 2013, 6:21:20 PM5/14/13
to
----- Original Message -----
From: "Steven Hartland" <kil...@multiplay.co.uk>
>>>> What version are you porting the changes to?
>>>>
>>>> What SSD are you using?
>>>>
>>>> What LSI controller are you using?
>>>
>>> I'd also like to see "zpool status" (for every pool that involves this
>>> SSD) and "gpart show" against the disk itself.
>>
>> Also:
>> 1. What FW version is your LSI? You can get this from dmesg.
>> 2. The exact command line your running iotest with?
>
> Any update on this? I'd like to try and replicate your test here so
> would appreciate as much information as possible.
Oh and of course details on where to download your non-standard iotest
From: "Steven Hartland" <kil...@multiplay.co.uk>
>>>> What version are you porting the changes to?
>>>>
>>>> What SSD are you using?
>>>>
>>>> What LSI controller are you using?
>>>
>>> I'd also like to see "zpool status" (for every pool that involves this
>>> SSD) and "gpart show" against the disk itself.
>>
>> Also:
>> 1. What FW version is your LSI? You can get this from dmesg.
>> 2. The exact command line your running iotest with?
>
> Any update on this? I'd like to try and replicate your test here so
> would appreciate as much information as possible.
including version info required :)
Ajit Jain
May 15, 2013, 1:47:54 AM5/15/13
to
Hi Steven,
Thanks for the follow-up.
The code where I pulled in zfs trim patches is not updated to 9 stable
specially the cam directory.
I pulled in many dependent patches in order to apply the patches that you
gave. After that all da devices
CAM_PERIPH_INVALID in dadone() because read capability was returning a very
big number (bigger than MAXPHYS)
for the block size. I think this is because I have not update the code to 9
stable (only pulled in required patches and miss
some patches).
So, I am planning to first update my code to 9stable and then try the same
test again. That might take some time.
thanks again,
ajit
On Wed, May 15, 2013 at 2:40 AM, Steven Hartland <kil...@multiplay.co.uk>wrote:
> ----- Original Message ----- From: "Steven Hartland"
>
> Steve
>
> ==============================**==================
Thanks for the follow-up.
The code where I pulled in zfs trim patches is not updated to 9 stable
specially the cam directory.
I pulled in many dependent patches in order to apply the patches that you
gave. After that all da devices
CAM_PERIPH_INVALID in dadone() because read capability was returning a very
big number (bigger than MAXPHYS)
for the block size. I think this is because I have not update the code to 9
stable (only pulled in required patches and miss
some patches).
So, I am planning to first update my code to 9stable and then try the same
test again. That might take some time.
thanks again,
ajit
On Wed, May 15, 2013 at 2:40 AM, Steven Hartland <kil...@multiplay.co.uk>wrote:
> ----- Original Message ----- From: "Steven Hartland"
>
> What version are you porting the changes to?
>>>>
>>>> What SSD are you using?
>>>>
>>>> What LSI controller are you using?
>>>>
>>>
>>> I'd also like to see "zpool status" (for every pool that involves this
>>> SSD) and "gpart show" against the disk itself.
>>>
>>
>> Also:
>> 1. What FW version is your LSI? You can get this from dmesg.
>> 2. The exact command line your running iotest with?
>>
>
> Any update on this? I'd like to try and replicate your test here so
> would appreciate as much information as possible.
>
>
> Regards
>>>>
>>>> What SSD are you using?
>>>>
>>>> What LSI controller are you using?
>>>>
>>>
>>> I'd also like to see "zpool status" (for every pool that involves this
>>> SSD) and "gpart show" against the disk itself.
>>>
>>
>> Also:
>> 1. What FW version is your LSI? You can get this from dmesg.
>> 2. The exact command line your running iotest with?
>>
>
> Any update on this? I'd like to try and replicate your test here so
> would appreciate as much information as possible.
>
>
> Steve
>
> ==============================**==================
Steven Hartland
May 15, 2013, 3:26:09 AM5/15/13
to
Could you provide us with details on the tests your using so we can
run them here on current sources and see if we see any issues?
Regards
Steve
================================================
run them here on current sources and see if we see any issues?
Regards
Steve
Ajit Jain
May 15, 2013, 4:19:49 AM5/15/13
to
Hi Steven,
Please find the tar ball of src code and binary of the test utility
attached with the mail.
Steps of test:
1. Enable zfs trim in /boot/loader.conf (vfs.zfs.trim_disable=0)
2. Set the delete method of the ssd device as UNMAP or WS16.
3. Create a pool (and optionally dataset) on the device.
4. Run iotest utility with thread count 10 (-t option) file size as at
least 5GB
Second to execute as at least 500 sec (-T option) and write as 100% (W
option).
regards,
ajit
On Wed, May 15, 2013 at 12:56 PM, Steven Hartland
<kil...@multiplay.co.uk>wrote:
>> >**wrote:
Please find the tar ball of src code and binary of the test utility
attached with the mail.
Steps of test:
1. Enable zfs trim in /boot/loader.conf (vfs.zfs.trim_disable=0)
2. Set the delete method of the ssd device as UNMAP or WS16.
3. Create a pool (and optionally dataset) on the device.
4. Run iotest utility with thread count 10 (-t option) file size as at
least 5GB
Second to execute as at least 500 sec (-T option) and write as 100% (W
option).
regards,
ajit
On Wed, May 15, 2013 at 12:56 PM, Steven Hartland
<kil...@multiplay.co.uk>wrote:
>>
>> ----- Original Message ----- From: "Steven Hartland"
>>>
>>> What version are you porting the changes to?
>>>
>>>>
>>>>>> What SSD are you using?
>>>>>>
>>>>>> What LSI controller are you using?
>>>>>>
>>>>>>
>>>>> I'd also like to see "zpool status" (for every pool that involves this
>>>>> SSD) and "gpart show" against the disk itself.
>>>>>
>>>>>
>>>> Also:
>>>> 1. What FW version is your LSI? You can get this from dmesg.
>>>> 2. The exact command line your running iotest with?
>>>>
>>>>
>>> Any update on this? I'd like to try and replicate your test here so
>>> would appreciate as much information as possible.
>>>
>>>
>>> Regards
>>> Steve
>>>
>>> ==============================****==================
>> ----- Original Message ----- From: "Steven Hartland"
>>>
>>> What version are you porting the changes to?
>>>
>>>>
>>>>>> What SSD are you using?
>>>>>>
>>>>>> What LSI controller are you using?
>>>>>>
>>>>>>
>>>>> I'd also like to see "zpool status" (for every pool that involves this
>>>>> SSD) and "gpart show" against the disk itself.
>>>>>
>>>>>
>>>> Also:
>>>> 1. What FW version is your LSI? You can get this from dmesg.
>>>> 2. The exact command line your running iotest with?
>>>>
>>>>
>>> Any update on this? I'd like to try and replicate your test here so
>>> would appreciate as much information as possible.
>>>
>>>
>>> Regards
>>> Steve
>>>
Ajit Jain
May 15, 2013, 4:29:55 AM5/15/13
to
Hi Steve,
One more thing I am not seeing data corruption with SATA SSD (kingston).
The issue was seen on SAS SSD i.e. Seagate PULSAR ST100FM0002 .
regards,
ajit
One more thing I am not seeing data corruption with SATA SSD (kingston).
The issue was seen on SAS SSD i.e. Seagate PULSAR ST100FM0002 .
regards,
ajit
Steven Hartland
May 15, 2013, 10:07:44 AM5/15/13
to
Unless you have the latest CAM patches, which is in current, you wont be
doing TRIM on SATA disk connected to an LSI controller.
I've just tested using the following cmd under 8.3 with MFC'ed changes from
current, using ATA_TRIM, UNMAP & WS16 and have had no issues on a machine
with Intel SSD and LSI controller.
./iotest -t 20 -s 536870912 -W 100 -T 500 /test/iotest/
I did however notice that your test is hardly doing any disk access apart
from when its "Initializing test file....", instead it seems to be CPU bound,
so not sure if there's a problem with the iotest code or with my command
line args?
Given this my current thinking is either:
1. There's a problem with your patches
2. There's a bug in the FW of the Seagate disk
3. There's a problem with the UNMAP code which is being trigged by your disk
only.
I think #3 is quite unlikely.
If you could install a recent version of current and test with that it
should rule out #1, leaving #2.
================================================
doing TRIM on SATA disk connected to an LSI controller.
I've just tested using the following cmd under 8.3 with MFC'ed changes from
current, using ATA_TRIM, UNMAP & WS16 and have had no issues on a machine
with Intel SSD and LSI controller.
./iotest -t 20 -s 536870912 -W 100 -T 500 /test/iotest/
I did however notice that your test is hardly doing any disk access apart
from when its "Initializing test file....", instead it seems to be CPU bound,
so not sure if there's a problem with the iotest code or with my command
line args?
Given this my current thinking is either:
1. There's a problem with your patches
2. There's a bug in the FW of the Seagate disk
3. There's a problem with the UNMAP code which is being trigged by your disk
only.
I think #3 is quite unlikely.
If you could install a recent version of current and test with that it
should rule out #1, leaving #2.
Ajit Jain
May 15, 2013, 11:18:19 AM5/15/13
to
Hi Steven,
Thanks for the update.
It is surprising that there is no/less disk activity as the command is
correct. May be we need to enable
sync=always on the zfs dataset.
I will try again the once I update the cam code base.
regards,
ajit
>>>>> kil...@multiplay.co.uk>****wrote:
Thanks for the update.
It is surprising that there is no/less disk activity as the command is
correct. May be we need to enable
sync=always on the zfs dataset.
I will try again the once I update the cam code base.
regards,
ajit
>>>>>
>>>>>
>>>>> ----- Original Message ----- From: "Steven Hartland"
>>>>>
>>>>>>
>>>>>> What version are you porting the changes to?
>>>>>>
>>>>>>
>>>>>>> What SSD are you using?
>>>>>>>>>
>>>>>>>>> What LSI controller are you using?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'd also like to see "zpool status" (for every pool that involves
>>>>>>>> this
>>>>>>>> SSD) and "gpart show" against the disk itself.
>>>>>>>>
>>>>>>>>
>>>>>>>> Also:
>>>>>>> 1. What FW version is your LSI? You can get this from dmesg.
>>>>>>> 2. The exact command line your running iotest with?
>>>>>>>
>>>>>>>
>>>>>>> Any update on this? I'd like to try and replicate your test here so
>>>>>> would appreciate as much information as possible.
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>> Steve
>>>>>>
>>>>>> ==============================******==================
>>>>>
>>>>> ----- Original Message ----- From: "Steven Hartland"
>>>>>
>>>>>>
>>>>>> What version are you porting the changes to?
>>>>>>
>>>>>>
>>>>>>> What SSD are you using?
>>>>>>>>>
>>>>>>>>> What LSI controller are you using?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'd also like to see "zpool status" (for every pool that involves
>>>>>>>> this
>>>>>>>> SSD) and "gpart show" against the disk itself.
>>>>>>>>
>>>>>>>>
>>>>>>>> Also:
>>>>>>> 1. What FW version is your LSI? You can get this from dmesg.
>>>>>>> 2. The exact command line your running iotest with?
>>>>>>>
>>>>>>>
>>>>>>> Any update on this? I'd like to try and replicate your test here so
>>>>>> would appreciate as much information as possible.
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>> Steve
>>>>>>
Steven Hartland
May 16, 2013, 6:35:23 PM5/16/13
to
----- Original Message -----
From: "Ajit Jain" <ajit...@cloudbyte.com>
To: "Steven Hartland" <kil...@multiplay.co.uk>
Cc: "freebsd-fs" <freeb...@freebsd.org>
Sent: Wednesday, May 15, 2013 4:18 PM
Subject: Re: seeing data corruption with zfs trim functionality
> Hi Steven,
>
big time on the test box. So we checked and there was zpool corruption as well
as a missing boot loader and a corrupt GPT, so I believe we have reproduced
your issue.
After recovering the machine I created 3 pools on 3 different disks each
running a different delete_method.
We then re-ran the tests which resulted in the pool running with delete_method
WS16 being so broken it had suspended IO. A reboot resulted in it once again
reporting no partition table via gpart.
A third test run again produced a corrupt pool for WS16.
I've conducted a preliminary review of the CAM WS16 code path along with SBC-3
spec which didn't identify any obvious issues.
Given we're both using LSI 2008 based controllers it could be FW issue specific
to WS16 but that's just speculation atm, so I'll continue to investigate.
If you could re-test you end without using WS16 to see if you can reproduce the
problem with either UNMAP or ATA_TRIM that would be a very useful data point.
Regards
Steve
================================================
0 new messages