Infiniband use iSer instead ib_srpt

[Giuseppe]

Hello, i couldnt understand how use the driver iSer instead ib_srpt, i enabled ISER module at /etc/infiniband/openib.conf
when im enabling the targer esos showing just ib_srpt.
someone can help me ?
Thanks

[Marc Smith]

Hi,

The iSER functionality is still very new, and its not actually in the "mainline" branch of SCST (trunk) or any release branches. I expect in the near future the iSER patches will be merged into SCST trunk.

For the time being, it should possible to build ESOS with this different branch... Here it is: http://sourceforge.net/p/scst/svn/HEAD/tree/branches/iser

It would require a bit of work to modify the rc scripts, and the build recipe in the Makefile.

I'm actually interested in this feature myself, so I'll be keeping a close eye on it, and when its available.

--Marc

--
You received this message because you are subscribed to the Google Groups "esos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Swartzendruber]

I will be awaiting this too.  The out-of-box Mellanox drivers on esxi 5.5 support 10gbe, not native IB, and present a 10gbe iSER storage adapter (which I currently have nothing to talk to - lol)

[Dan Swartzendruber]

So, I found out there is an scst branch (6210) from awhile back saying 'iser ready for testing...)  I decided to take a look at it, and after running the complete build sequence, I see there is ./work/build/scst-trunk_r6146.  What I am wondering is: how would I fiddle things to build with the 6210 codebase I pulled down?

[Dan Swartzendruber]

I note the scst devs pointed out a 'ready for testing' snapshot.  How would I build with this?  Following the docs, I ended up with a work/build/scst-trunk_r6146 directory.  What do I need to do to build against the 6210 snapshot I copied from sourceforge?

[Dan Swartzendruber]

There is a 6210 snapshot with iser support.  How do I build against this?  I did the complete build, and got a scst-trunk_r6146 directory.  What do I do to get mine instead?

[Marc Smith]

Replace the extracted directory in the vanilla ESOS build's
"work/build" directory with your 6210 directory.

--Marc

[Dan Swartzendruber]

Doesn't work.  It just re-creates the stock scst sub-directory.  I assume because the default target does this:

# all - The default goal; complete every target except image.
all:
        $(MAKE) -C $(BUILD_DIR) fetch
        $(MAKE) -C $(BUILD_DIR) extract
        $(MAKE) -C $(BUILD_DIR) build

So instead of 'make -j4' I did 'make build'.  It's doing it's thing now.  Stay tuned...

[Dan Swartzendruber]

Hmmm, I'm not sure if I understood right.  Do you mean replace with the exact same name?  I tried doing a 'make build' and that failed...

[Marc Smith]

It looks for the "work/build/scst-*" (wildcard match) so as long as
there is one directory matching "scst-*" it should find that
directory. What is the error output from building?

On Wed, May 27, 2015 at 1:32 PM, Dan Swartzendruber <dsw...@druber.com> wrote:
>
> Hmmm, I'm not sure if I understood right. Do you mean replace with the
> exact same name? I tried doing a 'make build' and that failed...
>

[Dan Swartzendruber]

On 2015-05-27 13:42, 'Marc Smith' via esos-users wrote:
> It looks for the "work/build/scst-*" (wildcard match) so as long as
> there is one directory matching "scst-*" it should find that
> directory. What is the error output from building?

it didn't even do the scst stuff. i don't know if in the process of
hacking on this, i screwed up something. i nuked the directory and am
redoing from scratch...

[Dan Swartzendruber]

It puked some messages about libgrcrypt.  Not sure if they are okay or not.  I just realized this is a centos7 VM I was using to do some ZoL development, and maybe there are some sharp edges there.  I am retrying the build on a ubuntu 12.04 VM that was sitting there - I'll post if I get the same anomalies...

[Dan Swartzendruber]

Well, I ran it and it pukes.  I don't know if the structure of the tree is different for the code drop they made available, but it pukes like this:

make[1]: Entering directory `/mnt/esos/esos'
make --directory=/mnt/esos/esos/work/build/scst-svn-6210-branches-iser /mnt/esos/esos/work/build/scst-trunk_r6146/iscsi-scst \
        SCST_INC_DIR=/mnt/esos/esos/work/build/scst-svn-6210-branches-iser /mnt/esos/esos/work/build/scst-trunk_r6146/scst/include progs
make[2]: Entering directory `/mnt/esos/esos/work/build/scst-svn-6210-branches-iser'
make[2]: Nothing to be done for `/mnt/esos/esos/work/build/scst-trunk_r6146/iscsi-scst'.
make[2]: Nothing to be done for `/mnt/esos/esos/work/build/scst-trunk_r6146/scst/include'.
make[2]: *** No rule to make target `progs'.  Stop.
make[2]: Leaving directory `/mnt/esos/esos/work/build/scst-svn-6210-branches-iser'
make[1]: *** [scst] Error 2
make[1]: Leaving directory `/mnt/esos/esos'
make: *** [all] Error 2

Looking at files in top level, Makefile and Makefile.in seem to have the 6416 hard-coded? I found it looked like I had to replace the 6146 tarball in the dist directory and did so, but now it blows up because it can't download the new tarball from your repo.  So if I change the name it doesn't fetch and blows up, but if I don't change it, it tries to extract and build the 6146 scst.  So I have no idea what needs to be done, but it sure doesn't look like just extracting the new tarball in work/build?

[Marc Smith]

This line:
make --directory=/mnt/esos/esos/work/build/scst-svn-6210-branches-iser
/mnt/esos/esos/work/build/scst-trunk_r6146/iscsi-scst \

It appears you didn't REPLACE the "scst-trunk_r6146" directory with
the "scst-svn-6210-branches-iser" you only added/copied it in addition
to. Try removing the original directory and only keeping the new one
in work/build/ and see how it goes.


--Marc

[Dan Swartzendruber]

On 2015-05-28 08:50, 'Marc Smith' via esos-users wrote:
> This line:
> make --directory=/mnt/esos/esos/work/build/scst-svn-6210-branches-iser
> /mnt/esos/esos/work/build/scst-trunk_r6146/iscsi-scst \
>
> It appears you didn't REPLACE the "scst-trunk_r6146" directory with
> the "scst-svn-6210-branches-iser" you only added/copied it in addition
> to. Try removing the original directory and only keeping the new one
> in work/build/ and see how it goes.

I may not have been clear. That was the first thing I tried. It
apparently refetches and extracts the original (6146) tarball. I will
try again from scratch and send you the relevant logfile excerpts...

[Dan Swartzendruber]

btw, see my update to the closed issue about enet support. turns out I
bought connectx-3 EN cards, not VPI like I thought, so they are enet
only (so it shouldn't be a surprise that they 'just worked'). I'm fine
with this, since SRP is not likely to be picking up any market share
going forward...

[Dan Swartzendruber]

Okay, I got it built. It wasn't as simple as one would have thought.
After the download and config, I had to:

1. 'make fetch'

2. 'make extract'

3. delete the 6146 directory and replace with the iser version.

4. 'make build'

if you do a vanilla 'make', it sees the missing 6146 directory and
regenerates it and everything goes off the rails. i will try deploying
this later this evening and see what happens...

[Marc Smith]

Right... if you look at the Makefile, running "make" by itself will
give you the default goal (the 'all' recipe) which calls the fetch,
extract, and build recipes serially. You'd also need to modify the
recipe so it builds/installs the iSER target driver module, if it is
in fact a different module (or maybe its part of the SRP driver) -- I
haven't looked at it, so I'm not sure.

--Marc

[Dan Swartzendruber]

On 2015-05-28 10:56, 'Marc Smith' via esos-users wrote:
> Right... if you look at the Makefile, running "make" by itself will
> give you the default goal (the 'all' recipe) which calls the fetch,
> extract, and build recipes serially. You'd also need to modify the
> recipe so it builds/installs the iSER target driver module, if it is
> in fact a different module (or maybe its part of the SRP driver) -- I
> haven't looked at it, so I'm not sure.

okay, i'll let you know what i find out...

[Dan Swartzendruber]

Hmm, okay. I ran 'make image' like the docs said. It ground away for a
bit and blew up. The last few lines:

/usr/bin/touch Python
cd
/mnt/esos/esos/work/build/scst-svn-6210-branches-iser/scstadmin/scstadmin/scst-0.9.10
&& PERL5LIB="/mnt/esos/esos/work/image/usr/lib/perl5/5.12.4" \
/mnt/esos/esos/work/image/usr/bin/perl Makefile.PL \
PREFIX=/usr DESTDIR=/mnt/esos/esos/work/image
PERL_LIB="/mnt/esos/esos/work/image/usr/lib/perl5/5.12.4" \

PERL_ARCHLIB="/mnt/esos/esos/work/image/usr/lib/perl5/5.12.4/x86_64-linux"
Checking if your kit is complete...
Looks good
Only one of PREFIX or INSTALL_BASE can be given. Not both.
make: *** [scstadmin] Error 255

now, i've never run 'make image' with a vanilla build, but it seems
doubtful my replacing the scst directory caused this?

[Dan Swartzendruber]

on the other hand, it *is* bitching about the 6210 dir :) then again, i
never set either of those variables? I suppose I can try again with the
stock 6146 in place and make sure that is working okay...

[Marc Smith]

Doesn't sound like your "make build" actually completed... all
$(build_targets) are a dependency for the 'image' recipe.

[Dan Swartzendruber]

On 2015-05-28 11:17, 'Marc Smith' via esos-users wrote:
> Doesn't sound like your "make build" actually completed... all
> $(build_targets) are a dependency for the 'image' recipe.

here is the last few lines in the build log:

LD [M]
/mnt/esos/esos/work/build/EnhanceIO-master_20140810/Driver/enhanceio/enhanceio.ko
CC
/mnt/esos/esos/work/build/EnhanceIO-master_20140810/Driver/enhanceio/enhanceio_fifo.mod.o
LD [M]
/mnt/esos/esos/work/build/EnhanceIO-master_20140810/Driver/enhanceio/enhanceio_fifo.ko
CC
/mnt/esos/esos/work/build/EnhanceIO-master_20140810/Driver/enhanceio/enhanceio_lru.mod.o
LD [M]
/mnt/esos/esos/work/build/EnhanceIO-master_20140810/Driver/enhanceio/enhanceio_lru.ko
CC
/mnt/esos/esos/work/build/EnhanceIO-master_20140810/Driver/enhanceio/enhanceio_rand.mod.o
LD [M]
/mnt/esos/esos/work/build/EnhanceIO-master_20140810/Driver/enhanceio/enhanceio_rand.ko
make[2]: Leaving directory `/mnt/esos/esos/work/build/linux-3.14.36'
make[1]: Leaving directory
`/mnt/esos/esos/work/build/EnhanceIO-master_20140810/Driver/enhanceio'
### Building the BTIER module...
make[1]: Entering directory `/mnt/esos/esos/work/build/linux-3.14.36'
CC [M]
/mnt/esos/esos/work/build/btier-tip_c3e46b/kernel/btier/btier_common.o
CC [M]
/mnt/esos/esos/work/build/btier-tip_c3e46b/kernel/btier/btier_sysfs.o
CC [M]
/mnt/esos/esos/work/build/btier-tip_c3e46b/kernel/btier/btier_main.o
LD [M]
/mnt/esos/esos/work/build/btier-tip_c3e46b/kernel/btier/btier.o
Building modules, stage 2.
MODPOST 1 modules
CC
/mnt/esos/esos/work/build/btier-tip_c3e46b/kernel/btier/btier.mod.o
LD [M]
/mnt/esos/esos/work/build/btier-tip_c3e46b/kernel/btier/btier.ko
make[1]: Leaving directory `/mnt/esos/esos/work/build/linux-3.14.36'
/usr/bin/touch esos_kernels

No indications anything failed? I checked for '*** ' in that log and
only see this:

*** These auxiliary programs are missing or incompatible versions:
makeinfo
*** some features will be disabled.
*** Check the INSTALL file for required versions.
*** errlist.c count 133 inflated to GLIBC_2.12 count 134 (old errno.h?)
checking for sys/stat.h... *** This configuration is not supported in
the following subdirectories:
*** Makeinfo is missing. Info documentation will not be built.
configure: WARNING: *** FAM support will not be built (FAM library not
found) ***
*** Makeinfo is missing. Info documentation will not be built.
*** Makeinfo is missing or too old.
*** Info documentation will not be built.
make: *** [Net-Telnet] Error 255
make: *** Waiting for unfinished jobs....
*** Makeinfo is missing. Info documentation will not be built.

[Marc Smith]

If you're looking for errors/issues in a build, I wouldn't use
parallel jobs with make (--jobs / -j). Based on that sparse output,
looks like the Net-Telnet recipe/package is you're culprit. If I were
you, I'd start there.

--Marc

[Dan Swartzendruber]

On 2015-05-28 11:29, 'Marc Smith' via esos-users wrote:
> If you're looking for errors/issues in a build, I wouldn't use
> parallel jobs with make (--jobs / -j). Based on that sparse output,
> looks like the Net-Telnet recipe/package is you're culprit. If I were
> you, I'd start there.

got it...

[Dan Swartzendruber]

Well, I give up.  I'm running ubuntu 12.04 which the docs say is supported.  Did the initial git and config steps (using 0.1.2 for checkout).  A completely clean build dir.  And after a ton of grinding, it fails here:

 k/image/lib -L/mnt/esos/esos/work/image/usr/lib -Wl,-rpath-link,/mnt/esos/esos/work/image/lib -Wl,-rpath-link,/mnt/esos/esos/work/image/usr/lib -o nut-scanner nut_scanner-nut-scanner.o libnutscan.la ../../common/libcommon.la -lpthread
libtool: link: gcc -specs=/mnt/esos/esos/work/esos_build.spec -nostdinc -idirafter /usr/lib/gcc/x86_64-linux-gnu/4.6/include -idirafter /usr/lib/gcc/x86_64-linux-gnu/4.6/include-fixed -idirafter /mnt/esos/esos/work/image/usr/include -idirafter /mnt/esos/esos/work/kernel_headers/include -I../../clients -I../../include -g -O2 -Wall -Wsign-compare -Wl,-rpath-link -Wl,/mnt/esos/esos/work/image/lib -Wl,-rpath-link -Wl,/mnt/esos/esos/work/image/usr/lib -o .libs/nut-scanner nut_scanner-nut-scanner.o  -L/mnt/esos/esos/work/image/lib -L/mnt/esos/esos/work/image/usr/lib ./.libs/libnutscan.so ../../common/.libs/libcommon.a -lpthread
./.libs/libnutscan.so: undefined reference to `__fdelt_chk'
collect2: ld returned 1 exit status
make[4]: *** [nut-scanner] Error 1
make[4]: Leaving directory `/mnt/esos/esos/work/build/nut-2.7.1/tools/nut-scanner'
make[3]: *** [all] Error 2
make[3]: Leaving directory `/mnt/esos/esos/work/build/nut-2.7.1/tools/nut-scanner'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/mnt/esos/esos/work/build/nut-2.7.1/tools'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/mnt/esos/esos/work/build/nut-2.7.1'
make: *** [nut] Error 2

I've spent literally hours trying to get this to build.  I don't know if I need to have the exact setup you use, but every iteration, I have to wait 2-3 hours to see yet another weird failure.  Here is all the log entries that look like errors:

*** These auxiliary programs are missing or incompatible versions: makeinfo
*** some features will be disabled.
*** Check the INSTALL file for required versions.
*** errlist.c count 133 inflated to GLIBC_2.12 count 134 (old errno.h?)

*** This configuration is not supported in the following subdirectories:

*** Makeinfo is missing or too old.
*** Info documentation will not be built.
*** Makeinfo is missing. Info documentation will not be built.
*** Makeinfo is missing. Info documentation will not be built.

*** Makeinfo is missing. Info documentation will not be built.
configure: WARNING: *** FAM support will not be built (FAM library not found) ***

*** Generating DH Parameters for SSL/TLS ***
*** Configuration summary for nrpe 2.15 09-06-2013 ***:
*** Compile finished ***
make[4]: *** [nut-scanner] Error 1
make[3]: *** [all] Error 2
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [nut] Error 2

[Dan Swartzendruber]

I'm giving this one more shot (giving up on f'ing ubuntu).  Back to my centos7 build VM.

[Dan Swartzendruber]

Anything new on the iser front?

[Marc Smith]

Its coming soon! Sorry, I've been swamped the last 1.5 months! Its
actually now in the version of SCST that we're using in both 0.1.x and
master, but its just not being built/installed. I'll enable it in the
master branch... would you mind putting in a issue on GitHub so I can
queue it up. =)

--Marc

On Mon, Oct 12, 2015 at 9:41 PM, Dan Swartzendruber <dsw...@druber.com> wrote:
> Anything new on the iser front?
>

[Dan Swartzendruber]

10-4 :)

>You received this message because you are subscribed to a topic in the Google Groups "esos-users" group.
>To unsubscribe from this topic, visit https://groups.google.com/d/topic/esos-users/yMtwm7UMTHU/unsubscribe.
>To unsubscribe from this group and all its topics, send an email to esos-users+...@googlegroups.com.

[Marc Smith]

Actually, its already in there... I honestly hadn't taken a look at
the iSER stuff until now, but its just part of the iSCSI target
driver. Here is a HOWTO document:
http://community.mellanox.com/docs/DOC-1479


--Marc

[Dan Swartzendruber]

Yeah, I had seen that some time ago, but I had thought the iSER stuff wasn't yet in SCST.  I'll give this a try - I  need to figure out what to do with vsphere to have it use iSER.

[Marc Smith]

It wasn't for quite a while, but then it was merged into SCST's 3.0.x branch and we moved to that version in ESOS (for both branches currently).

--Marc

[Dan Swartzendruber]

On 2015-10-15 09:34, 'Marc Smith' via esos-users wrote:
> It wasn't for quite a while, but then it was merged into SCST's 3.0.x
> branch and we moved to that version in ESOS (for both branches
> currently).

got it. turns out i can test all of your commits more quickly than i
thought, since i can do a lot of it with vmware virtual machine. e.g.
create a generic 64-bit linux vm with 4GB ram and 8GB 'disk'. leave
powered off. use vmware gui to add the 8GB disk to the esos buildbot.
do 'dd if=xxx.img of=/dev/sdb bs=1M'. use vmware gui to remove (but not
delete) the 8GB disk from the buildbot VM. power on the esos VM. watch
it do the 1st time initialization on boot.

[Dan Swartzendruber]

Turns out there will be an unspecified delay testing this.  Since upgrading vsphere from 5.5 to 6.0, I no longer see the iSER storage adapters on vsphere.  Some google fu reveals that they have not yet release iSER/RDMA for vsphere 6.0.  No timetable yet :(

[Sander Lahpor]

Dear Dan,

I've been looking into the iSER solution as well the last few weeks. I was wondering if you made any progress with the use of iSER towards your VSphere environment. I saw at the Mellanox site an update OFED driver that actually supports iSER on ESXi6. Could you please share your experiences on this with me?

Sander Lahpor.

[Dan Swartzendruber]

Ah well, I have up and am switching from mellanox...

[Sander Lahpor]

Dan,

When you're moving away from Mellanox, can you tell me towards what new brand / sollution?

Does that also mean that you didn't find a working solution with the Mellanox stuff in combination with ESOS?

rgds,

Sander.

[Dan Swartzendruber]

I got some Intel x520-da cards. Still setting it up...

[Marc Smith]

And to clarify: It appeared the iSER target side (ESOS part) would
function correctly, you just did not have a working solution (ESXi) on
the initiator side, right?

--Marc

> You received this message because you are subscribed to the Google Groups
> "esos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Dan Swartzendruber]

On 2016-02-18 08:40, 'Marc Smith' via esos-users wrote:
> And to clarify: It appeared the iSER target side (ESOS part) would
> function correctly, you just did not have a working solution (ESXi) on
> the initiator side, right?

Honestly? I have no idea. iSER is incredibly frustrating - I have
spent literally hours googling for how to set up a target and/or
initiator using this capability, and have come up (largely) dry. Closest
useful info I ever found was on oracle's solaris doc site, where they
say something like 'if the adapter is iSER capable, we will use that
automatically', but still hard to tell if you *are* using it :(