Issues on debian jessie and Dell MD3200i - connection pdu rejected

[Lars Schimmer]

Hi!

We jsut upgraded one of our old wheezy linux server to Debian jessie.
Now Open-iscsi spits out this error on load:


connection1:0: pdu (op 0x1 itt 0x33) rejected. Reason code 0x5
[ 649.368791] connection1:0: pdu (op 0x1 itt 0x1) rejected. Reason
code 0x5
[ 649.368928] connection1:0: pdu (op 0x1 itt 0x50) rejected. Reason
code 0x5
[ 748.430030] connection1:0: detected conn error (1020)
[ 754.695618] connection1:0: detected conn error (1021)

and the co9nnection is broken.
Any idea what could be the reason?

Thank you.

(the storage I connect to is a Dell MD3200i storage, no issues before of
this kind).

MfG,
Lars Schimmer
--
-------------------------------------------------------------
TU Graz, Institut für ComputerGraphik & WissensVisualisierung
Tel: +43 316 873-5405 E-Mail: l.sch...@cgv.tugraz.at
Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723

[Lars Schimmer]

On 2016-03-08 14:49, Lars Schimmer wrote:
> Hi!
>
> We jsut upgraded one of our old wheezy linux server to Debian jessie.
> Now Open-iscsi spits out this error on load:
>
>
> connection1:0: pdu (op 0x1 itt 0x33) rejected. Reason code 0x5
> [ 649.368791] connection1:0: pdu (op 0x1 itt 0x1) rejected. Reason
> code 0x5
> [ 649.368928] connection1:0: pdu (op 0x1 itt 0x50) rejected. Reason
> code 0x5
> [ 748.430030] connection1:0: detected conn error (1020)
> [ 754.695618] connection1:0: detected conn error (1021)
>
> and the co9nnection is broken.
> Any idea what could be the reason?
>
> Thank you.
>
> (the storage I connect to is a Dell MD3200i storage, no issues before of
> this kind).

Ok, some more information:

Error appeared in Debian with kernel linux-image-4.3.0-0.bpo.1-amd64
(jessie-backports) and open-iscsi 2.0.873+git0.3b4b4500-14 (also
appeared with open-iscsi 2.0.873+git0.3b4b4500-8+deb8u1).

Error is gone on debian jessie kernel 3.16+63.

FS is a ext4 on storage.

[Mike Christie]

On 03/08/2016 08:01 AM, Lars Schimmer wrote:
> On 2016-03-08 14:49, Lars Schimmer wrote:
>> Hi!
>>
>> We jsut upgraded one of our old wheezy linux server to Debian jessie.
>> Now Open-iscsi spits out this error on load:
>>
>>
>> connection1:0: pdu (op 0x1 itt 0x33) rejected. Reason code 0x5
>> [ 649.368791] connection1:0: pdu (op 0x1 itt 0x1) rejected. Reason
>> code 0x5
>> [ 649.368928] connection1:0: pdu (op 0x1 itt 0x50) rejected. Reason
>> code 0x5

The command is being rejected by the target because the target does not
support it.

>> [ 748.430030] connection1:0: detected conn error (1020)
>> [ 754.695618] connection1:0: detected conn error (1021)
>>
>> and the co9nnection is broken.
>> Any idea what could be the reason?
>>
>> Thank you.
>>
>> (the storage I connect to is a Dell MD3200i storage, no issues before of
>> this kind).
>
> Ok, some more information:
>
> Error appeared in Debian with kernel linux-image-4.3.0-0.bpo.1-amd64
> (jessie-backports) and open-iscsi 2.0.873+git0.3b4b4500-14 (also
> appeared with open-iscsi 2.0.873+git0.3b4b4500-8+deb8u1).
>
> Error is gone on debian jessie kernel 3.16+63.
>
> FS is a ext4 on storage.
>

I think this was caused due to newer ext4/block layer code sending a
command the storage did not support in newer kernels by default or maybe
the block/SCSI layer changed the format being used. I want to say it was
discard related, but I cannot remember for sure. Could you take a
tcpdump or writeshark trace when you run this test, so I can see what
SCSI command we are failing on?

Also, is there anything in the MD320 logs about a command being rejected
because it is not supported?

[Lars Schimmer]

On 2016-03-08 17:49, Mike Christie wrote:
> On 03/08/2016 08:01 AM, Lars Schimmer wrote:
>> On 2016-03-08 14:49, Lars Schimmer wrote:
>>> Hi!
>>>
>>> We jsut upgraded one of our old wheezy linux server to Debian jessie.
>>> Now Open-iscsi spits out this error on load:
>>>
>>>
>>> connection1:0: pdu (op 0x1 itt 0x33) rejected. Reason code 0x5
>>> [ 649.368791] connection1:0: pdu (op 0x1 itt 0x1) rejected. Reason
>>> code 0x5
>>> [ 649.368928] connection1:0: pdu (op 0x1 itt 0x50) rejected. Reason
>>> code 0x5
>
> The command is being rejected by the target because the target does not
> support it.

Interesting.
As I got another Debian box working with iSCSI
2.0.873+git0.3b4b4500-8+deb8u1 and debian kernel 4.3.0 on the same
target/debain MD3200i with no errors yet. Also ext4.
Strange

>>> [ 748.430030] connection1:0: detected conn error (1020)
>>> [ 754.695618] connection1:0: detected conn error (1021)
>>>
>>> and the co9nnection is broken.
>>> Any idea what could be the reason?
>>>
>>> Thank you.
>>>
>>> (the storage I connect to is a Dell MD3200i storage, no issues before of
>>> this kind).
>>
>> Ok, some more information:
>>
>> Error appeared in Debian with kernel linux-image-4.3.0-0.bpo.1-amd64
>> (jessie-backports) and open-iscsi 2.0.873+git0.3b4b4500-14 (also
>> appeared with open-iscsi 2.0.873+git0.3b4b4500-8+deb8u1).
>>
>> Error is gone on debian jessie kernel 3.16+63.
>>
>> FS is a ext4 on storage.
>>
>
> I think this was caused due to newer ext4/block layer code sending a
> command the storage did not support in newer kernels by default or maybe
> the block/SCSI layer changed the format being used. I want to say it was
> discard related, but I cannot remember for sure. Could you take a
> tcpdump or writeshark trace when you run this test, so I can see what
> SCSI command we are failing on?

Hm, currently not, it was our main storage server, I do not want to set
it offline again, but if I see the same error on another box, I´ll do my
very best.

> Also, is there anything in the MD320 logs about a command being rejected
> because it is not supported?

Not really, mostly this:

Date/Time: 08.03.16 13:19:38
Sequence number: 17388
Event type: 180C
Event category: Internal
Priority: Informational
Event needs attention: false
Event send alert: false
Event visibility: true
Description: iSCSI connection terminated unexpectedly
Event specific codes: 0/0/0
Component type: iSCSI Initiator
Component location: iqn.1993-08.org.debian:01:84455b760a
Logged by: RAID Controller Module in slot 0

There is a update available from Dell for that storage, not yet applied
due to "load on storage cannot be stopped yet". Maybe it could help.