libibverbs: Warning: no userspace device-specific driver found

[daniel...@gmail.com]

Hi all,
Am getting the below messages when I try to check for install devices with the command ibv_devices.

libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

Am running ubuntu 16.4 with kernel 4.8.0-46-generic. 

Output of lsmod below looks OK:
Module                  Size  Used by
siw                   110592  0
rdma_ucm               28672  0
ib_uverbs              65536  1 rdma_ucm
rdma_cm                57344  1 rdma_ucm
iw_cm                  49152  1 rdma_cm
ib_cm                  45056  1 rdma_cm
ib_core               212992  6 ib_cm,rdma_cm,ib_uverbs,iw_cm,rdma_ucm,siw
configfs               40960  2 rdma_cm

Additionally when i run strace ibv_devices,  the output includes the below lines:
open("/sys/class/infiniband_verbs/uverbs1/device/vendor", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/sys/class/infiniband_verbs/uverbs0/device/vendor", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

I will be grateful if I can get some help on resolving this.   Thanks.

Regards

[Bernard Metzler]

Hi Daniel,
Did you install the siw user library? If yes, do you have it in your LD_LIBRARY_PATH? Per default, libsiw should land at /usr/local/lib

Best,
Bernard.


danielkwofie---04/07/2017 08:17:57 PM---Hi all, Am getting the below messages when I try to check for install devices with

--
You received this message because you are subscribed to the Google Groups "zrlio-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zrlio-users...@googlegroups.com.
To post to this group, send email to zrlio...@googlegroups.com.
Visit this group at https://groups.google.com/group/zrlio-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/zrlio-users/bfeecb5a-fcc5-4029-af44-8d5e33cb401f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[daniel...@gmail.com]

Thanks Bernard for the  response..
Sorry for the late feedback. I went off this for sometime.
I do have my LD_LIBRARY_PATH set  but i still get the same messages

libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

No IB devices found

Look forward to your feedback then. Thanks.

Regards,
Daniel

[karim.s...@gmail.com]

I was facing the same problem and this comment on github solved it for me. It basically downloads the libibverbs1 packge and compiles it from source then replaces the ibv_* utils with the compiled ones. I will write the commands below, adapted for Ubuntu:

First download libibverbs-1.2.1.tar.gz from here: https://www.openfabrics.org/downloads/verbs/

then run the following commands in the download location:

tar zxvf libibverbs-1.2.1.tar.gz
cd libibverbs-1.2.1

sudo apt-get install libnl-3-200 libnl-3-dev libnl-route-3-200 libnl-route-3-dev
./configure
make

cp examples/ibv_* /usr/bin/

Now you should be able to see the iwarp device when running ibv_devices.

[Rajesh S]

Hi Karim,

Thanks for the post. It helped in getting the libibverbs installed. However when I run ibv_devices I get this below error. Same for ibv_devinfo as well.

-sh-4.2$ cd /usr/bin

-sh-4.2$ ls -ltr

-rwxr-xr-x    1 root root       7626 Jun 17 12:32 ibv_devices
-rwxr-xr-x    1 root root       7655 Jun 17 12:32 ibv_asyncwatch
-rwxr-xr-x    1 root root       7622 Jun 17 12:32 ibv_devinfo
-rwxr-xr-x    1 root root       7686 Jun 17 12:32 ibv_rc_pingpong
-rwxr-xr-x    1 root root       7697 Jun 17 12:32 ibv_srq_pingpong
-rwxr-xr-x    1 root root       7686 Jun 17 12:32 ibv_uc_pingpong
-rwxr-xr-x    1 root root       7686 Jun 17 12:32 ibv_ud_pingpong
-rwxr-xr-x    1 root root       7708 Jun 17 12:32 ibv_xsrq_pingpong
-sh-4.2$ ibv_devices
Failed to get IB devices list: Function not implemented

I am not sure what the message is complaining about. Am I missing something here ? Any help is appreciated.

-Rajesh

[karim.s...@gmail.com]

Hi Rajesh,

To use SoftiWARP, you need to insert the kernel modules using the following commands:

• sudo modprobe  rdma_cm

• sudo modprobe ib_uverbs

• sudo modprobe rdma_ucm

• sudo insmod /lib/modules/$(uname -r)/extra/siw.ko

Note: make sure you have booted into the exact same kernel version as the one you have have installed SoftiWARP on (I currently choose older kernel versions from the boot menu since mine got updated).

To save the trouble of running these commands each time, I have created a config file to load them on system startup.

sudo nano /etc/modules-load.d/rdma.conf

You can name the file anything with the .conf extension. Then fill the file with these lines:

rdma_cm

ib_uverbs

rdma_ucm

siw

For the last module to be loaded correctly, I created a symbolic link to siw.ko:

• sudo ln -s /lib/modules/$(uname -r)/extra/siw.ko /lib/modules/$(uname -r)/

• sudo depmod -a

Now you just need to restart the system and you should be able to see the siw device with ibv_devices.

Best,

Karim

[Rajesh S]

Hi Karim,

Thanks much for the response and the help.

While inserting the kernel modules, I get this error. 

-sh-4.2$ sudo modprobe  rdma_cm
-sh-4.2$ sudo modprobe ib_uverbs
-sh-4.2$ sudo modprobe rdma_ucm
-sh-4.2$ sudo insmod /lib/modules/$(uname -r)/extra/siw.ko
insmod: ERROR: could not load module /lib/modules/3.10.0-862.el7.x86_64/extra/siw.ko: No such file or directory

-sh-4.2$ ls -ltr  /lib/modules/3.10.0-862.el7.x86_64/extra/
total 0

-sh-4.2$ ls -ltr  /lib/modules
total 8
drwxr-xr-x. 3 root root 4096 Apr 30 13:12 3.10.0-855.el7.x86_64
drwxr-xr-x. 7 root root 4096 Jun 19 02:46 3.10.0-862.el7.x86_64

By the way, I am yet to get the make on kernel and userlib. I have these errors when I do that.

cd /home/rsambandhan/rdma/git/softiwarp/userlib

./configure

checking for ibv_get_device_list in -libverbs... yes
checking sysfs/libsysfs.h usability... no
checking sysfs/libsysfs.h presence... no
checking for sysfs/libsysfs.h... no
checking infiniband/driver.h usability... no
checking infiniband/driver.h presence... no
checking for infiniband/driver.h... no
configure: error: <infiniband/driver.h> not found.  Is libibverbs installed?

-sh-4.2$ cd /home/rsambandhan/rdma/git/softiwarp/kernel
-sh-4.2$ make
/home/rsambandhan/rdma/git/softiwarp/kernel/siw_cm.h:165:8: error: ‘struct tcp_sock’ has no mem                                                    ber named ‘xmit_size_goal_segs’
  if (tp->xmit_size_goal_segs)
        ^
/home/rsambandhan/rdma/git/softiwarp/kernel/siw_cm.h:166:12: error: ‘struct tcp_sock’ has no me                                                    mber named ‘xmit_size_goal_segs’
   return tp->xmit_size_goal_segs * tp->mss_cache;
            ^
/home/rsambandhan/rdma/git/softiwarp/kernel/siw_main.c: At top level:
/home/rsambandhan/rdma/git/softiwarp/kernel/siw_main.c:125:2: error: unknown field ‘dma_ops’ sp                                                    ecified in initializer
  .archdata.dma_ops = &siw_dma_generic_ops,

Though libibverbs configure and make succeeded, when I run ibv_devices I now get this error.

-sh-4.2$ cd libibverbs-1.2.1
-sh-4.2$ sudo ibv_devices
libibverbs: Warning: couldn't open config directory '/usr/local/etc/libibverbs.d'.
    device                 node GUID
    ------              ----------------

Do you see anything that I am missing to hit these errors ? Or how to get through these errors ? Any help is highly appreciated.

Many Thanks...

[karim.s...@gmail.com]

Hi Rajesh,

You need to get the "make install" to work for the "siw.ko" file to be created. There still seems to be a problem with your libibverbs installation. Did you run this -from the libibverbs source folder- after configure and make?

sudo cp examples/ibv_* /usr/bin/

Best,

[Rajesh S]

Hi Karim,

With libibverbs the configure and make ran without error and I have the below files in examples/ibv* which I copied to /usr/bin as well. I am providing the details below, do you spot any anomaly ?

-sh-4.2$ pwd
/home/rsambandhan/rdma/git/utils/libibverbs-1.2.1
-sh-4.2$
-sh-4.2$ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
...

...

config.status: executing depfiles commands
config.status: executing libtool commands

-sh-4.2$ make
make  all-am
make[1]: Entering directory `/home/rsambandhan/rdma/git/utils/libibverbs-1.2.1'
make[1]: Leaving directory `/home/rsambandhan/rdma/git/utils/libibverbs-1.2.1'

-sh-4.2$ ls -ltr examples/ibv_*
-rwxrwxrwx 1 rsambandhan abp 7626 Jun 17 11:51 examples/ibv_devices
-rwxrwxrwx 1 rsambandhan abp 7622 Jun 17 11:51 examples/ibv_devinfo
-rwxrwxrwx 1 rsambandhan abp 7655 Jun 17 11:51 examples/ibv_asyncwatch
-rwxrwxrwx 1 rsambandhan abp 7686 Jun 17 11:51 examples/ibv_rc_pingpong
-rwxrwxrwx 1 rsambandhan abp 7686 Jun 17 11:51 examples/ibv_uc_pingpong
-rwxrwxrwx 1 rsambandhan abp 7686 Jun 17 11:51 examples/ibv_ud_pingpong
-rwxrwxrwx 1 rsambandhan abp 7697 Jun 17 11:51 examples/ibv_srq_pingpong
-rwxrwxrwx 1 rsambandhan abp 7708 Jun 17 11:51 examples/ibv_xsrq_pingpong

Thanks,

Rajesh

[Rajesh S]

Hi Karim,

I created a sym link and got the error resolved for now and ibv_devices does not throw any error. However I still have trouble getting the kernel make working. Sorry for hijacking this thread so far. I will post that specific problem in the other thread. Hope you have some inputs. Thanks for all the help so far :-)

Thanks much !!!

-Rajesh