iSer write speed problem

[ekha...@gmail.com]

I have last Esos 4.2.1 with 2 Mellanox Connectx4 cards 100GB. 

It connected to ESxi 7/0.3 host with same cards.

When I used iSCSI connections, it works stable both on read and write and provided about 1GB/sec read/write performance. It works stable, but much slower that actually server's disks system working. I configured iSER adapters on ESXi and got much better performance for read, I got about 5-6GB/sec, But write performance downgraded up to 5MB/sec. And I got lot of errors in ESOS log:

   Apr 24 01:53:01 localhost kernel: [ 7539.980156] [14580]: scst: TM fn ABORT_TASK/0 (mcmd 0000000043576396, initiator iqn.1998-01.com.vmware:  host36.smvmware.local:180811617:69, target iqn.2024-04.esos.localhost:24c98)

Apr 24 01:53:01 localhost kernel: [ 7539.980267] [1915]: scst: TM fn 0 (mcmd 0000000043576396) finished, status -1
Apr 24 01:53:01 localhost kernel: [ 7539.980279] [1915]: iscsi-scst: iSCSI TM fn 1 finished, status 0, dropped 0
Apr 24 01:53:15 localhost kernel: [ 7554.185151] [14580]: iscsi-scst: iSCSI TM fn 1
Apr 24 01:53:15 localhost kernel: [ 7554.185173] [14580]: scst: TM fn ABORT_TASK/0 (mcmd 00000000cef328c6, initiator iqn.1998-01.com.vmware:  host36.smvmware.local:180811617:69, target iqn.2024-04.esos.localhost:24c98)
Apr 24 01:53:15 localhost kernel: [ 7554.185311] [1915]: scst: cmd 000000001cf655ae (tag 124, sn 4294967276) being executed/xmitted (state XMIT_WAIT, op COMPARE AND WRITE, proc time 8 sec., timeout 60 sec.), deferring ABORT (cmd_done_wait_count 0, cmd_finish_wait_count 1, internal 0, mcmd fn 0 (mcmd 00000000cef328c6), initiator iqn.1998-01.com.vmware:  host36.smvmware.local:180811617:69, target iqn.2024-04.esos.localhost:24c98)
Apr 24 01:53:15 localhost kernel: [ 7554.243540] [14580]: scst: Aborted cmd 000000001cf655ae finished (tag 124, ref 1)
Apr 24 01:53:15 localhost kernel: [ 7554.243640] [1915]: scst: TM fn 0 (mcmd 00000000cef328c6) finished, status 0
Apr 24 01:53:15 localhost kernel: [ 7554.243652] [1915]: iscsi-scst: iSCSI TM fn 1 finished, status 0, dropped 0
Apr 24 01:54:10 localhost kernel: [ 7609.183886] [14580]: iscsi-scst: iSCSI TM fn 1
Apr 24 01:54:10 localhost kernel: [ 7609.183909] [14580]: scst: TM fn ABORT_TASK/0 (mcmd 00000000ad95626f, initiator iqn.1998-01.com.vmware:  host36.smvmware.local:180811617:69, target iqn.2024-04.esos.localhost:24c98)
Apr 24 01:54:10 localhost kernel: [ 7609.184017] [1915]: scst: cmd 00000000c0fb4d95 (tag 4, sn 4294967276) being executed/xmitted (state XMIT_WAIT, op WRITE(16), proc time 5 sec., timeout 60 sec.), deferring ABORT (cmd_done_wait_count 0, cmd_finish_wait_count 1, internal 0, mcmd fn 0 (mcmd 00000000ad95626f), initiator iqn.1998-01.com.vmware:  host36.smvmware.local:180811617:69, target iqn.2024-04.esos.localhost:24c98)
Apr 24 01:54:13 localhost kernel: [ 7612.316124] [14580]: scst: Aborted cmd 00000000c0fb4d95 finished (tag 4, ref 1)
Apr 24 01:54:13 localhost kernel: [ 7612.316221] [1915]: scst: TM fn 0 (mcmd 00000000ad95626f) finished, status 0
Apr 24 01:54:13 localhost kernel: [ 7612.316233] [1915]: iscsi-scst: iSCSI TM fn 1 finished, status 0, dropped 0
Apr 24 01:54:30 localhost kernel: [ 7629.186312] [14580]: iscsi-scst: iSCSI TM fn 1
Apr 24 01:54:30 localhost kernel: [ 7629.186335] [14580]: scst: TM fn ABORT_TASK/0 (mcmd 00000000ade847c0, initiator iqn.1998-01.com.vmware:  host36.smvmware.local:180811617:69, target iqn.2024-04.esos.localhost:24c98)
Apr 24 01:54:30 localhost kernel: [ 7629.186492] [1915]: scst: cmd 00000000463bef3a (tag 20, sn 4294967276) being executed/xmitted (state XMIT_WAIT, op COMPARE AND WRITE, proc time 5 sec., timeout 60 sec.), deferring ABORT (cmd_done_wait_count 0, cmd_finish_wait_count 1, internal 0, mcmd fn 0 (mcmd 00000000ade847c0), initiator iqn.1998-01.com.vmware:  host36.smvmware.local:180811617:69, target iqn.2024-04.esos.localhost:24c98)
Apr 24 01:54:30 localhost kernel: [ 7629.521795] [14580]: scst: Aborted cmd 00000000463bef3a finished (tag 20, ref 1)
Apr 24 01:54:30 localhost kernel: [ 7629.521895] [1915]: scst: TM fn 0 (mcmd 00000000ade847c0) finished, status 0
Apr 24 01:54:30 localhost kernel: [ 7629.521907] [1915]: iscsi-scst: iSCSI TM fn 1 finished, status 0, dropped 0
Apr 24 01:54:30 localhost kernel: [ 7629.522880] [13907]: scst: Aborted cmd 00000000616d7c31 finished (tag 22, ref 1)
Apr 24 01:54:30 localhost kernel: [ 7629.523391] [13907]: scst: Aborted cmd 00000000d4e3bb16 finished (tag 66, ref 1)
Apr 24 01:54:30 localhost kernel: [ 7629.523442] [13911]: scst: Aborted cmd 0000000060f030f7 finished (tag 61, ref 1)
Apr 24 01:54:30 localhost kernel: [ 7629.523455] [13908]: scst: Aborted cmd 00000000bc240d7b finished (tag 36, ref 1)
Apr 24 01:54:30 localhost kernel: [ 7629.523467] [13909]: scst: Aborted cmd 000000005b02e646 finished (tag 63, ref 1)
Apr 24 01:54:30 localhost kernel: [ 7629.523478] [13907]: scst: Aborted cmd 000000006bf28cdf finished (tag 93, ref 1)
Apr 24 01:54:30 localhost kernel: [ 7629.523759] [13907]: scst: Aborted cmd 00000000b5aa412b finished (tag 114, ref 1)
Apr 24 01:55:30 localhost kernel: [ 7689.114055] [14580]: iscsi-scst: iSCSI TM fn 1
Apr 24 01:55:30 localhost kernel: [ 7689.114076] [14580]: scst: TM fn ABORT_TASK/0 (mcmd 000000007f6e1531, initiator iqn.1998-01.com.vmware:  host36.smvmware.local:180811617:69, target iqn.2024-04.esos.localhost:24c98)
Apr 24 01:55:30 localhost kernel: [ 7689.114169] [1915]: scst: TM fn 0 (mcmd 000000007f6e1531) finished, status -1
Apr 24 01:55:30 localhost kernel: [ 7689.114181] [1915]: iscsi-scst: iSCSI TM fn 1 finished, status 0, dropped 0
Apr 24 01:55:30 localhost kernel: [ 7689.114310] [14580]: iscsi-scst: iSCSI TM fn 1

What can be a problem with poor write speed?

Thanks

E.

[Marc Smith]

Generally, I find performance issues with iSER are related to the
lossless Ethernet configuration (or lack of), especially when I see
I/O in one direction performs well (but not the other). So I'd start
with verifying your lossless configuration -- make sure no frames on
the Ethernet switch are being dropped for the target and initiator
ports. This should be easy to see when you start your write test and
then watch the counters on the switch -- seeing drops increase (or any
drops) is the likely cause. This means something is misconfigured on
lossless Ethernet (eg, PFC, or global-pause style configs) on either
the switch, the initiator (VMware), or the target (ESOS).


--Marc


> Thanks
> E.
>
> --
> You received this message because you are subscribed to the Google Groups "esos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/b9d1f2b0-25df-45c8-978e-08e78bf4d6ban%40googlegroups.com.

[ekha...@gmail.com]

Hi

Thanks for quick response. 

First of all, I have to note that on same physical adapters and same physical switch I have VSAN working with RDMA enabled with no errors. It means that on ESXi adapter side and switch all must be Ok. 

Can you recommend  some way to check stability of RDMA network from ESOS side? BTW - what ROCE version iSER on Esos configured for? Vsan require ROCE v2, so iSER adapters on ESXi configured for ROCE v2 also.

I also don't see any pauses on RDMA nics:

# vsish -e cat /net/pNics/vmnic4/stats | grep -e "Pause\|PerPrio"
rxPauseCtrlPhy: 0
txPauseCtrlPhy: 0
txPauseStormWarningEvents: 0
txPauseStormErrorEvents: 0
# vsish -e cat /net/pNics/vmnic5/stats | grep -e "Pause\|PerPrio"
rxPauseCtrlPhy: 0
txPauseCtrlPhy: 0
txPauseStormWarningEvents: 0
txPauseStormErrorEvents: 0

Regards

E.

[ekha...@gmail.com]

Yep, probably like some misconfiguration on ESOS side.

Port on Switch where connected ESXi shows no errors:

show interfaces hundredGigE 1/4/6
hundredGigE 1/4/6 is up, line protocol is up
Hardware is DellEMCEth, address is 34:17:eb:0f:4b:82
    Current address is 34:17:eb:0f:4b:82
Non-qualified pluggable media present, QSFP28 type is 100GBASE-CR4-1M
    AutoNegotiation is ON
    Forward Error Correction(FEC) configured is default
    FEC status is cl91
    Wavelength is 38nm
Interface index is 2112014
Internet address is not set
Mode of IPv4 Address Assignment : NONE
DHCP Client-ID :3417eb0f4b82
MTU 9216 bytes, IP MTU 9198 bytes
LineSpeed 100000 Mbit
Flowcontrol rx off tx off
ARP type: ARPA, ARP Timeout 04:00:00
Last clearing of "show interface" counters 3w4d7h
Queueing strategy: fifo
Input Statistics:
     302099604 packets, 558729546118 bytes
     6191801 64-byte pkts, 116747922 over 64-byte pkts, 29469220 over 127-byte pkts
     5027457 over 255-byte pkts, 4886898 over 511-byte pkts, 139776306 over 1023-byte pkts
     29538 Multicasts, 2017 Broadcasts, 302068049 Unicasts
     0 runts, 0 giants, 2 throttles
     0 CRC, 0 overrun, 0 discarded
     0 FEC bit errors, 18 FEC uncorrected code words
Output Statistics:
     377458338 packets, 731518759522 bytes, 0 underruns
     26765554 64-byte pkts, 115057075 over 64-byte pkts, 30670928 over 127-byte pkts
     3077514 over 255-byte pkts, 2044380 over 511-byte pkts, 199842887 over 1023-byte pkts
     6169155 Multicasts, 20131195 Broadcasts, 351157988 Unicasts
     0 throttles, 0 discarded, 0 collisions, 0 wreddrops
Rate info (interval 299 seconds):
     Input 00.00 Mbits/sec,        293 packets/sec, 0.00% of line-rate
     Output 01.00 Mbits/sec,        313 packets/sec, 0.00% of line-rate
Time since last interface status change: 3d3h3m

But where ESOS connected, shows the next:

show interfaces hundredGigE 1/1/4
hundredGigE 1/1/4 is up, line protocol is up
Hardware is DellEMCEth, address is 34:17:eb:0f:4b:82
    Current address is 34:17:eb:0f:4b:82
Non-qualified pluggable media present, QSFP28 type is 100GBASE-CR4-1M
    AutoNegotiation is ON
    Forward Error Correction(FEC) configured is default
    FEC status is cl91
    Wavelength is 38nm
Interface index is 2098702
Internet address is not set
Mode of IPv4 Address Assignment : NONE
DHCP Client-ID :3417eb0f4b82
MTU 9216 bytes, IP MTU 9198 bytes
LineSpeed 100000 Mbit
Flowcontrol rx off tx off
ARP type: ARPA, ARP Timeout 04:00:00
Last clearing of "show interface" counters 3w4d7h
Queueing strategy: fifo
Input Statistics:
     96101576 packets, 351686497863 bytes
     1738393 64-byte pkts, 18515270 over 64-byte pkts, 7873626 over 127-byte pkts
     6152 over 255-byte pkts, 10003 over 511-byte pkts, 67958132 over 1023-byte pkts
     5960 Multicasts, 2965 Broadcasts, 96092651 Unicasts
     0 runts, 0 giants, 0 throttles
     0 CRC, 0 overrun, 0 discarded
     4294967296 FEC bit errors, 102 FEC uncorrected code words
Output Statistics:
     86969045 packets, 237173302715 bytes, 0 underruns
     30913289 64-byte pkts, 16353137 over 64-byte pkts, 8896218 over 127-byte pkts
     93949 over 255-byte pkts, 61583 over 511-byte pkts, 30650869 over 1023-byte pkts
     6205308 Multicasts, 19894166 Broadcasts, 60869571 Unicasts
     0 throttles, 0 discarded, 0 collisions, 0 wreddrops
Rate info (interval 299 seconds):
     Input 46.00 Mbits/sec,       5585 packets/sec, 0.05% of line-rate
     Output 22.00 Mbits/sec,       5544 packets/sec, 0.02% of line-rate
Time since last interface status change: 1d21h15m

I mean for the next line: 4294967296 FEC bit errors - is it may be sign of problem? 

Can you help with idea how to manage this possible communication problem? ..

I really appreciate your help.

[ekha...@gmail.com]

Have to correct the info. The errors on port was old, I cleaned counters and now don't see any errors or drops on ports on both side - where the ESOS connected and where the ESXi connected.

No errors and no packet drops on both sides:

show interfaces hundredGigE 1/1/4
hundredGigE 1/1/4 is up, line protocol is up
Hardware is DellEMCEth, address is 34:17:eb:0f:4b:82
    Current address is 34:17:eb:0f:4b:82
Non-qualified pluggable media present, QSFP28 type is 100GBASE-CR4-1M
    AutoNegotiation is ON
    Forward Error Correction(FEC) configured is default
    FEC status is cl91
    Wavelength is 38nm
Interface index is 2098702
Internet address is not set
Mode of IPv4 Address Assignment : NONE
DHCP Client-ID :3417eb0f4b82
MTU 9216 bytes, IP MTU 9198 bytes
LineSpeed 100000 Mbit
Flowcontrol rx off tx off
ARP type: ARPA, ARP Timeout 04:00:00

Last clearing of "show interface" counters 00:02:46

Queueing strategy: fifo
Input Statistics:

     17085646 packets, 69845125455 bytes
     0 64-byte pkts, 151346 over 64-byte pkts, 144279 over 127-byte pkts
     0 over 255-byte pkts, 0 over 511-byte pkts, 16790021 over 1023-byte pkts
     7 Multicasts, 1 Broadcasts, 17085638 Unicasts

     0 runts, 0 giants, 0 throttles
     0 CRC, 0 overrun, 0 discarded

     0 FEC bit errors, 0 FEC uncorrected code words
Output Statistics:
     1400915 packets, 2443943302 bytes, 0 underruns
     2799 64-byte pkts, 684338 over 64-byte pkts, 140355 over 127-byte pkts
     2 over 255-byte pkts, 1387 over 511-byte pkts, 572034 over 1023-byte pkts
     646 Multicasts, 2305 Broadcasts, 1397964 Unicasts

     0 throttles, 0 discarded, 0 collisions, 0 wreddrops
Rate info (interval 299 seconds):

     Input 1959.00 Mbits/sec,      59906 packets/sec, 1.97% of line-rate
     Output 67.00 Mbits/sec,       4903 packets/sec, 0.07% of line-rate
Time since last interface status change: 4d13h59m

So, it looks like network is configure properly, isn't it?

What more may I check?

Regards

E.

[ekha...@gmail.com]

#show interfaces hundredGigE 1/1/4 dcbx detail

E-ETS Configuration TLV enabled                 e-ETS Configuration TLV disabled
R-ETS Recommendation TLV enabled                r-ETS Recommendation TLV disabled
P-PFC Configuration TLV enabled                 p-PFC Configuration TLV disabled
F-Application priority for FCOE enabled         f-Application Priority for FCOE disabled
I-Application priority for iSCSI enabled        i-Application Priority for iSCSI disabled
------------------------------------------------------------------------------------------

Interface hundredGigE 1/1/4
   Remote Mac Address 24:8a:07:5b:15:e8
    Port Role is Manual
   DCBX Operational Status is Enabled
   Is Configuration Source? FALSE
   Local DCBX Compatibility mode is IEEEv2.5
   Local DCBX Configured mode is AUTO
   Peer Operating version is IEEEv2.5
   Local DCBX TLVs Transmitted: ERPfI
   118 Input PFC TLV pkts, 41528 Output PFC TLV pkts, 0 Error PFC pkts
   0 PFC Pause Tx pkts, 0 Pause Rx pkts
   111 Input ETS Conf TLV Pkts, 41528 Output ETS Conf TLV Pkts, 7 Error ETS Conf TLV Pkts
   96 Input ETS Reco TLV pkts, 41528 Output ETS Reco TLV pkts, 7 Error ETS Reco TLV Pkts
   0 Input Appln Priority TLV pkts, 0 Output Appln Priority TLV pkts, 0 Error Appln Priority TLV Pkts
     Total DCBX Frames transmitted 82928
     Total DCBX Frames received 118
     Total DCBX Frame errors 0
     Total DCBX Frames unrecognized 10

[Andrei Wasylyk]

In esos, what is the version of the mellanox drivers that are included? You may be able to see this using modinfo if modinfo is indeed present. 

Mark is right in that many times users dont understand what lossless fabric means or mistake FC for PFC... I think the reason you aren't seeing a response is simply that we don't know why. We may have an idea, but those ideas are so low confidence that they dont merit sharing.

I have been excited about RDMA over Ethernet since iWarp and ROCE were released. Every few years I check in on them and have tested using intel, qlogic and most recently mlnx... I have yet to be successful in deploying a service (iser, nfsordma,...etc....) that used RDMA with the singular exception of qlogic with SMBDirect on Windows. But in that case, whats the point right?

So, to that end - if you want my long-shot advice of something to test, see below.

I have seen similar troubles with in-tree drivers for qedr/qede/qed. Especially where rdma is involved. And this is not an esos specific issue - it's present to varying degrees for most high end hardware in almost any distro. The only difference with esos is that it is a lean, purpose built OS. The advantage of using esos is that it's a way to leverage a small part of the knowledge and expertise of someone who (presumably) knows far more about the topic than you. The downside is that you are limited in what you can do to remediate the issue. Almost anything you do - if it requires making a big enough change to the OS then you'd better get acquainted with how the builds work.

So in that context, I would check the versions of mlnx_* modules that are loaded. Compare that to what is available in the mlnx_ofed packages from nvidia and if you notice a huge difference in version numbers OR if you notice that the module is completely lacking any settable parameters, you are running the basic in-tree driver. In any other distro I would say install the mlnx_ofed for your distro - but there won't be any for esos. So then I'd say well just download the mlnx_ofed source and compile it manually - but you cant do that either because you have no build tools. 

So your only option is to cross-compile for esos on a different machine. Which is really tough if youve never done that before. So your next best option is pick your distro of choice, install the kernel and headers that match esos as closely as possible, compile them against that kernel, copy the result to esos, do rmmod <all the mlnx modules> then do insmod --force /the/full/path/to/copied/modules/<mlnx module>.ko

Then, test your throughput on iser. 

Whether redhat or debian, intel or mellanox or qlogic... I have never seen scst+iser work correctly with the in-tree rdma drivers. Either i get weird throughput, or the machine crashes under load, or the adapter resets randomly.

So if you can manage to get a more recent /more feature complete version of the module to load and you manage to hang an iser target off it and get the transfer speeds you expected... Then I would say that might be your issue. And you can model yourself off of one of Mark's previous --with-<driver> build options to get it included into your esos proper.

Andrei

To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/0b0c76ce-f4d5-43c2-bf7b-2868c2305540n%40googlegroups.com.

[Marc Smith]

On Wed, May 1, 2024 at 7:07 PM Andrei Wasylyk <and...@gmail.com> wrote:
>
> In esos, what is the version of the mellanox drivers that are included? You may be able to see this using modinfo if modinfo is indeed present.
>
> Mark is right in that many times users dont understand what lossless fabric means or mistake FC for PFC... I think the reason you aren't seeing a response is simply that we don't know why. We may have an idea, but those ideas are so low confidence that they dont merit sharing.
>
> I have been excited about RDMA over Ethernet since iWarp and ROCE were released. Every few years I check in on them and have tested using intel, qlogic and most recently mlnx... I have yet to be successful in deploying a service (iser, nfsordma,...etc....) that used RDMA with the singular exception of qlogic with SMBDirect on Windows. But in that case, whats the point right?
>
> So, to that end - if you want my long-shot advice of something to test, see below.
>
>
> I have seen similar troubles with in-tree drivers for qedr/qede/qed. Especially where rdma is involved. And this is not an esos specific issue - it's present to varying degrees for most high end hardware in almost any distro. The only difference with esos is that it is a lean, purpose built OS. The advantage of using esos is that it's a way to leverage a small part of the knowledge and expertise of someone who (presumably) knows far more about the topic than you. The downside is that you are limited in what you can do to remediate the issue. Almost anything you do - if it requires making a big enough change to the OS then you'd better get acquainted with how the builds work.
>
> So in that context, I would check the versions of mlnx_* modules that are loaded. Compare that to what is available in the mlnx_ofed packages from nvidia and if you notice a huge difference in version numbers OR if you notice that the module is completely lacking any settable parameters, you are running the basic in-tree driver. In any other distro I would say install the mlnx_ofed for your distro - but there won't be any for esos. So then I'd say well just download the mlnx_ofed source and compile it manually - but you cant do that either because you have no build tools.
> So your only option is to cross-compile for esos on a different machine. Which is really tough if youve never done that before. So your next best option is pick your distro of choice, install the kernel and headers that match esos as closely as possible, compile them against that kernel, copy the result to esos, do rmmod <all the mlnx modules> then do insmod --force /the/full/path/to/copied/modules/<mlnx module>.ko
>
> Then, test your throughput on iser.
>
> Whether redhat or debian, intel or mellanox or qlogic... I have never seen scst+iser work correctly with the in-tree rdma drivers. Either i get weird throughput, or the machine crashes under load, or the adapter resets randomly.
>
> So if you can manage to get a more recent /more feature complete version of the module to load and you manage to hang an iser target off it and get the transfer speeds you expected... Then I would say that might be your issue. And you can model yourself off of one of Mark's previous --with-<driver> build options to get it included into your esos proper.

Yes, I'd recommend building ESOS with the latest MOFED package from
NVIDIA. The in-tree drivers might work fine for global pause style
lossless configurations, but if you're doing PFC (and you probably
will want to), IIRC you need the MOFED build option.

ESOS doesn't do anything automatically for lossless Ethernet
configurations, so it's up to the user to configure the NVRAM settings
on the Mellanox NICs, and figure out your lossless Ethernet setup
(depending on what the switch supports). Some switches only support TX
or RX flow control (global pause style), and even some switches that
have full TX/RX flow control, it still doesn't work the best and gives
odd performance results. I've found VLAN-based PFC configs work the
best, so you'll need to configure the VLAN-egress-map-priority value
on BOTH sides (the initaitor ports, and the target ports). And of
course configure the NVRAM settings if you're using newer Mellanox
NICs that can do DCBX via firmware (if not, you'll need to use lldpad,
or disable lldpad if doing the firmware method).

Hope this helps.

--Marc

> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/CAF9KdPvXRS5xWP-B%2BT%3Dx2PVfVcoVK9fF6VYmo0ZMAGbDuLEbcQ%40mail.gmail.com.

[inbusine...@gmail.com]

If you want to using iSER target, you must configure each end point of switch, HCA, host OS or Hypervisor and ESOS Storage.

Here is a 1MB 100% read test result with ESXi 8.0u3 and ESOS 4.2.1_z.

100% write test shows me a almost same level throughput performance.

Each host and storage configure with 2 of MCX455A-ECAT and Mellanox SX6036G 56Gb Ethernet port mode that configure PFC 3 & vlan.

TrueNAS SCALE show me a more good result that include more latest SCST target than ESOS.

Regards,

Jaehoon Choi

P.S

This post is a typo correction, only...:) 

[ekha...@gmail.com]

Hi
thanks for response. 
Can you please explain what exact configuration of  each end point of switch, HCA, host OS or Hypervisor you mean?
I have VSAN on this adapters, and I preformed all needed setup followed manuals. So, VSAN with RDMA enabled works properly. 
Can you please share your config of ESOS that works properly with iSER? How did you created the targets?

Thanks

E.

[ekha...@gmail.com]

Followed your post I installed TrueNAS Scale and trying to find out how to manage the iSER target, but without success yet. I see iSCSI only. Can you please to point to some instruction how to setup it? 
Thanks 

On Wednesday, May 22, 2024 at 12:05:10 PM UTC+3 inbusine...@gmail.com wrote:

[inbusine...@gmail.com]

Hi!

Did you enable iSER initiator on ESXi host?

At first you must clear iSCSI initiator configuration on iSCSI initiator.

(dynamic , static discovery and include vmk bindings)

At this point don't remove iSCSI initiator the add a iSER initiator with this command on ESXi host CLI.

 esxcli rdma iser add

  

Then you can configure iSER initiator configuration with iSCSI initiator in same.

At last you can add a dynamic discovery of TrueNAS SCALE Target.

(You must add a iSER initiator to TrueNAS SCALE Target configuration)

Good luck!

P.S

When next ESOS release with SCSI 3.8.0, I'll test it again.

TrueNAS SCALE include SCSI 3.8.0, but ESOS don't have it.

That cause problem with 128k block or above zvol connection.

[Evgeniy Khaskelberg]

Hi

Yes, sure I created an iser initiator and binded rdma network adapters on the ESXi side.

The problem is I can't find how to create iser target on TrueNAS.

Hard to believe that a regular iSCSI target will work as iSER also...

Regards

E.

To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/634d3f90-80de-4dac-beb8-4852e817e613n%40googlegroups.com.