RDMA about qemu live migrate and softiwarp 3.10.0 version issue
213 views
Skip to first unread message
邓艳山
Oct 24, 2017, 7:11:01 AM10/24/17
to zrlio-users, 黄勋, 明廷胜, 艾敬
hello,
can i use this open source code under centos 7.3 verison , which the kernel version is 3.10.0 , what should i do to port it back , now i can see it is used under linux kernel 4.8.0 from github version 4.8 , can i get the version 3.10.0 ? why can i only find the 4.8 verison of the softiwarp ?
how can i test the RDMA , can you give me some suggestions ? thanks
邓艳山(David Deng)
烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
网络产出线(Network Business Unit)
基础软件部/嵌入式公共技术领域/
Basic Software Engeneer Department/Embeded Public Domain/Support Team.
Tel: +86-27-81611804
Email: ysd...@fiberhome.com
Address: 高新四路6号 2号楼5楼
(No.67, Guanggu Chuangye Jie , Wuhan, Hubei)
Postcode: 430073
http://www.fiberhome.com/
Bernard Metzler
Oct 24, 2017, 10:48:26 AM10/24/17
to 邓艳山, zrlio-users, 黄勋, 明廷胜, 艾敬
-----zrlio...@googlegroups.com wrote: -----
>To: zrlio-users <zrlio...@googlegroups.com>
>From: 邓艳山
>Sent by: zrlio...@googlegroups.com
>Date: 10/24/2017 01:11PM
>Cc: 黄勋 <huang...@fiberhome.com>, 明廷胜 <nwos...@fiberhome.com>, 艾敬
><aijin...@fiberhome.com>
>Subject: [zrlio-users] RDMA about qemu live migrate and softiwarp
>3.10.0 version issue
In your case, the 'siw-pre4.4' branch should make it. Since moving forward
with the code aiming at mainline Linux acceptance of the driver, we
do not maintain old branches, but keep them for users of old OS versions.
I recommend updating your OS, if possible.
> how can i test the RDMA , can you give me some suggestions ?
> thanks
>
If things install fine, you might want to start with some simple RDMA
tests as included in the Infiniband/RDMA packages of the OS version of
your choice. E.g., start with rping to test end-to-end connectivity.
Bernard.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>邓艳山(David Deng)
>
>烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
>网络产出线(Network Business Unit)
>基础软件部/嵌入式公共技术领域/
>Basic Software Engeneer Department/Embeded Public Domain/Support
>Team.
>Tel: +86-27-81611804
>Email: ysd...@fiberhome.com
>Address: 高新四路6号 2号楼5楼
>(No.67, Guanggu Chuangye Jie , Wuhan, Hubei)
>Postcode: 430073
>http://www.fiberhome.com/
> --
> You received this message because you are subscribed to the Google
>Groups "zrlio-users" group.
> To unsubscribe from this group and stop receiving emails from it,
>send an email to zrlio-users...@googlegroups.com.
> To post to this group, send email to zrlio...@googlegroups.com.
> Visit this group at https://groups.google.com/group/zrlio-users.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/zrlio-users/2017102419095137606927%
>40fiberhome.com.
> For more options, visit https://groups.google.com/d/optout.
>
>To: zrlio-users <zrlio...@googlegroups.com>
>From: 邓艳山
>Sent by: zrlio...@googlegroups.com
>Date: 10/24/2017 01:11PM
>Cc: 黄勋 <huang...@fiberhome.com>, 明廷胜 <nwos...@fiberhome.com>, 艾敬
><aijin...@fiberhome.com>
>Subject: [zrlio-users] RDMA about qemu live migrate and softiwarp
>3.10.0 version issue
>
>
>hello,
> can i use this open source code under centos 7.3 verison , which
>the kernel version is 3.10.0 , what should i do to port it back ,
>now i can see it is used under linux kernel 4.8.0 from github version
>4.8 , can i get the version 3.10.0 ? why can i only find the 4.8
>verison of the softiwarp ?
You can get an older version of siw if you check out an appropriate branch.
>
>hello,
> can i use this open source code under centos 7.3 verison , which
>the kernel version is 3.10.0 , what should i do to port it back ,
>now i can see it is used under linux kernel 4.8.0 from github version
>4.8 , can i get the version 3.10.0 ? why can i only find the 4.8
>verison of the softiwarp ?
In your case, the 'siw-pre4.4' branch should make it. Since moving forward
with the code aiming at mainline Linux acceptance of the driver, we
do not maintain old branches, but keep them for users of old OS versions.
I recommend updating your OS, if possible.
> how can i test the RDMA , can you give me some suggestions ?
> thanks
>
tests as included in the Infiniband/RDMA packages of the OS version of
your choice. E.g., start with rping to test end-to-end connectivity.
Bernard.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>邓艳山(David Deng)
>
>烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
>网络产出线(Network Business Unit)
>基础软件部/嵌入式公共技术领域/
>Basic Software Engeneer Department/Embeded Public Domain/Support
>Team.
>Tel: +86-27-81611804
>Email: ysd...@fiberhome.com
>Address: 高新四路6号 2号楼5楼
>(No.67, Guanggu Chuangye Jie , Wuhan, Hubei)
>Postcode: 430073
>http://www.fiberhome.com/
> You received this message because you are subscribed to the Google
>Groups "zrlio-users" group.
> To unsubscribe from this group and stop receiving emails from it,
>send an email to zrlio-users...@googlegroups.com.
> To post to this group, send email to zrlio...@googlegroups.com.
> Visit this group at https://groups.google.com/group/zrlio-users.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/zrlio-users/2017102419095137606927%
>40fiberhome.com.
> For more options, visit https://groups.google.com/d/optout.
>
邓艳山
Oct 24, 2017, 11:26:34 PM10/24/17
to Bernard Metzler, zrlio-users, 黄勋, 明廷胜, 艾敬
thanks, how can i download the version siw-pre4.4 brach ?
邓艳山
Oct 25, 2017, 11:22:24 PM10/25/17
to Bernard Metzler, zrlio-users, 黄勋, 明廷胜, 艾敬
hello Bernard ,
now , i have this compile problem , how can i get to resolve it ?
root@host1 kernel]# make
make[1]: 进入目录“/home/tian/kernel-3.10.0/source/linux-3.10.0-514.16.1.el7”
CC [M] /root/download/softiwarp-4.8/kernel/siw_main.o
In file included from /root/download/softiwarp-4.8/kernel/siw_main.c:55:0:
/root/download/softiwarp-4.8/kernel/siw_cm.h: 在函数‘get_tcp_mss’中:
/root/download/softiwarp-4.8/kernel/siw_cm.h:165:8: 错误：‘struct tcp_sock’没有名为‘xmit_size_goal_segs’的成员
if (tp->xmit_size_goal_segs)
^
/root/download/softiwarp-4.8/kernel/siw_cm.h:166:12: 错误：‘struct tcp_sock’没有名为‘xmit_size_goal_segs’的成员
return tp->xmit_size_goal_segs * tp->mss_cache;
^
/root/download/softiwarp-4.8/kernel/siw_main.c: 在函数‘siw_device_create’中:
/root/download/softiwarp-4.8/kernel/siw_main.c:415:24: 警告：从不兼容的指针类型赋值 [默认启用]
ofa_dev->query_device = siw_query_device;
^
/root/download/softiwarp-4.8/kernel/siw_main.c:434:21: 警告：从不兼容的指针类型赋值 [默认启用]
ofa_dev->create_cq = siw_create_cq;
^
/root/download/softiwarp-4.8/kernel/siw_main.c:452:23: 警告：从不兼容的指针类型赋值 [默认启用]
ofa_dev->process_mad = siw_no_mad;
^
make[2]: *** [/root/download/softiwarp-4.8/kernel/siw_main.o] 错误 1
make[1]: *** [_module_/root/download/softiwarp-4.8/kernel] 错误 2
make[1]: 离开目录“/home/tian/kernel-3.10.0/source/linux-3.10.0-514.16.1.el7”
make: *** [modules] 错误 2
[root@host1 kernel]#
make[1]: 进入目录“/home/tian/kernel-3.10.0/source/linux-3.10.0-514.16.1.el7”
CC [M] /root/download/softiwarp-4.8/kernel/siw_main.o
In file included from /root/download/softiwarp-4.8/kernel/siw_main.c:55:0:
/root/download/softiwarp-4.8/kernel/siw_cm.h: 在函数‘get_tcp_mss’中:
/root/download/softiwarp-4.8/kernel/siw_cm.h:165:8: 错误：‘struct tcp_sock’没有名为‘xmit_size_goal_segs’的成员
if (tp->xmit_size_goal_segs)
^
/root/download/softiwarp-4.8/kernel/siw_cm.h:166:12: 错误：‘struct tcp_sock’没有名为‘xmit_size_goal_segs’的成员
return tp->xmit_size_goal_segs * tp->mss_cache;
^
/root/download/softiwarp-4.8/kernel/siw_main.c: 在函数‘siw_device_create’中:
/root/download/softiwarp-4.8/kernel/siw_main.c:415:24: 警告：从不兼容的指针类型赋值 [默认启用]
ofa_dev->query_device = siw_query_device;
^
/root/download/softiwarp-4.8/kernel/siw_main.c:434:21: 警告：从不兼容的指针类型赋值 [默认启用]
ofa_dev->create_cq = siw_create_cq;
^
/root/download/softiwarp-4.8/kernel/siw_main.c:452:23: 警告：从不兼容的指针类型赋值 [默认启用]
ofa_dev->process_mad = siw_no_mad;
^
make[2]: *** [/root/download/softiwarp-4.8/kernel/siw_main.o] 错误 1
make[1]: *** [_module_/root/download/softiwarp-4.8/kernel] 错误 2
make[1]: 离开目录“/home/tian/kernel-3.10.0/source/linux-3.10.0-514.16.1.el7”
make: *** [modules] 错误 2
[root@host1 kernel]#
邓艳山
Oct 25, 2017, 11:24:34 PM10/25/17
to Bernard Metzler, zrlio-users, 黄勋, 明廷胜, 艾敬
where can i get the softiwarp_old/ code to compile with linux kernel version 3.10.0 . which is the centos 7.3 kernel verson
7/21/2011
To ease development and maintenance of the siw kernel module
source code, it has been splitted into two independent directories
softiwarp/ and softiwarp_old/. both directories keep all files
needed to build a siw kernel module.
softiwarp/
contains the code aligned to the current linux kernel development
tree. it does not contain any legacy code to run with older kernels.
this code is updated frequently, but runs only on newer kernels.
it has been tested for kernels back to kernel version 2.6.36.2.
code documentation (IMPLEMENTATION.txt) will be updated within
the next days.
softiwarp_old/
contains code which compiles and runs on older kernels, but may
soon break on newer kernels. code maintenance is sloppy and
it is recommended to use the newer softiwarp/ code base if
possible.
06/13/2014
softiwarp_old/ has been removed.
01/20/2015
With kernel version 3.15 the OFA core changed the way it maintains
registered user communication buffers. The new code would result
in inefficient initialization of RDMA data source or sink location for
a software RDMA stack. Therefore, siw abandons using OFA core
user page management (ib_umem_get() etc.) and implememnts its
own simple, but better suited management of pinned user pages.
~
To ease development and maintenance of the siw kernel module
source code, it has been splitted into two independent directories
softiwarp/ and softiwarp_old/. both directories keep all files
needed to build a siw kernel module.
softiwarp/
contains the code aligned to the current linux kernel development
tree. it does not contain any legacy code to run with older kernels.
this code is updated frequently, but runs only on newer kernels.
it has been tested for kernels back to kernel version 2.6.36.2.
code documentation (IMPLEMENTATION.txt) will be updated within
the next days.
softiwarp_old/
contains code which compiles and runs on older kernels, but may
soon break on newer kernels. code maintenance is sloppy and
it is recommended to use the newer softiwarp/ code base if
possible.
06/13/2014
softiwarp_old/ has been removed.
01/20/2015
With kernel version 3.15 the OFA core changed the way it maintains
registered user communication buffers. The new code would result
in inefficient initialization of RDMA data source or sink location for
a software RDMA stack. Therefore, siw abandons using OFA core
user page management (ib_umem_get() etc.) and implememnts its
own simple, but better suited management of pinned user pages.
~
Bernard Metzler
Oct 26, 2017, 8:39:37 AM10/26/17
to 邓艳山, zrlio-users, 黄勋, 明廷胜, 艾敬
softiwarp_old/ does not exist anymore. We dropped that when moving
from gitorious to github.
I recommend installing a plain Linux kernel, as new as possible, maybe 4.12
or 4.13, and checkout/install the latest siw branch (dev-siw.mem_ext).
It is virtually impossible to provide a siw code version which
correctly deals with all distro-version-specific amendments/backports/...
to a plain kernel environment.
As said previously, you could try the 'siw-pre4.4' branch on your
system. just git checkout that branch from your clone and give it
a try. Let us know how compilation turns out.
Thanks
Bernard.
-----邓艳山 <ysd...@fiberhome.com> wrote: -----
>To: "Bernard Metzler" <B...@zurich.ibm.com>
>From: 邓艳山 <ysd...@fiberhome.com>
>Date: 10/26/2017 05:25AM
>Cc: zrlio-users <zrlio...@googlegroups.com>, 黄勋
>Subject: Re: [zrlio-users] RDMA about qemu live migrate and
>softiwarp 3.10.0 version issue
>
>
from gitorious to github.
I recommend installing a plain Linux kernel, as new as possible, maybe 4.12
or 4.13, and checkout/install the latest siw branch (dev-siw.mem_ext).
It is virtually impossible to provide a siw code version which
correctly deals with all distro-version-specific amendments/backports/...
to a plain kernel environment.
As said previously, you could try the 'siw-pre4.4' branch on your
system. just git checkout that branch from your clone and give it
a try. Let us know how compilation turns out.
Thanks
Bernard.
-----邓艳山 <ysd...@fiberhome.com> wrote: -----
>To: "Bernard Metzler" <B...@zurich.ibm.com>
>From: 邓艳山 <ysd...@fiberhome.com>
>Date: 10/26/2017 05:25AM
>Cc: zrlio-users <zrlio...@googlegroups.com>, 黄勋
>Subject: Re: [zrlio-users] RDMA about qemu live migrate and
>softiwarp 3.10.0 version issue
>
>
>邓艳山(David Deng)
>
>烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
>网络产出线(Network Business Unit)
>基础软件部/嵌入式公共技术领域/
>Basic Software Engeneer Department/Embeded Public Domain/Support
>Team.
>Tel: +86-27-81611804
>Email: ysd...@fiberhome.com
>Address: 高新四路6号 2号楼5楼
>(No.67, Guanggu Chuangye Jie , Wuhan, Hubei)
>Postcode: 430073
>http://www.fiberhome.com/
>
>From: ysd...@fiberhome.com
>Date: 2017-10-25 11:26
>To: Bernard Metzler
>CC: zrlio-users; 黄勋; 明廷胜(外包)明廷胜(外包); 艾敬aijing7065
>Subject: Re: Re: [zrlio-users] RDMA about qemu live migrate and
>softiwarp 3.10.0 version issue
>
>thanks, how can i download the version siw-pre4.4 brach ?
>
>
>
>烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
>网络产出线(Network Business Unit)
>基础软件部/嵌入式公共技术领域/
>Basic Software Engeneer Department/Embeded Public Domain/Support
>Team.
>Tel: +86-27-81611804
>Email: ysd...@fiberhome.com
>Address: 高新四路6号 2号楼5楼
>(No.67, Guanggu Chuangye Jie , Wuhan, Hubei)
>Postcode: 430073
>http://www.fiberhome.com/
>
>From: ysd...@fiberhome.com
>Date: 2017-10-25 11:26
>To: Bernard Metzler
>CC: zrlio-users; 黄勋; 明廷胜(外包)明廷胜(外包); 艾敬aijing7065
>Subject: Re: Re: [zrlio-users] RDMA about qemu live migrate and
>softiwarp 3.10.0 version issue
>
>thanks, how can i download the version siw-pre4.4 brach ?
>
>
黄勋
Oct 26, 2017, 11:30:43 PM10/26/17
to Bernard Metzler, zrlio-users, 明廷胜, 艾敬, 邓艳山
Hi, Bernard
Here is the operation we did under Linux 3.10.0-514.el7.x86_64 kernel, and siw version was softiwarp-siw-pre4.4.zip
#yum install epel-release -y
#yum install gcc gcc-c++ bc openssl-devel automake ncurses-devel libibverbs -y #yum install libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils perl-Switch elfutils-libelf-devel -y
# yum install -y rdma libocrdma libocrdma-static librdmacm-devel librdmacm-utils rdma-ndd libfabric libi40iw libibverbs libvma qperf binutils-devel zlib-devel libibverbs libibverbs-utils libibverbs-devel-static libibverbs-devel libipathverbs-static libipathverbs libmlx5 libnl libnl-devel
[root@localhost userlib]# yum install libibverbs*
# systemctl start rdma.service
# systemctl enable rdma.service
# systemctl enable rdma.service
# modprobe rdma_ucm
# modprobe ib_uverbs
# modprobe rdma_ucm
# sudo lsmod
# ls /dev/infiniband/
rdma_cm
# modprobe ib_uverbs
# modprobe rdma_ucm
# sudo lsmod
# ls /dev/infiniband/
rdma_cm
//here is the userlib part
[root@localhost userlib]# mv configure.in configure.ac
[root@localhost userlib]# ./autogen.sh
[root@localhost userlib]# yum install epel-release -y
[root@localhost userlib]# yum install libibverbs-devel
[root@localhost userlib]# ./configure
.............
checking for sysfs/libsysfs.h... no
checking infiniband/driver.h usability... no
checking infiniband/driver.h presence... no
checking for infiniband/driver.h... no
configure: error: <infiniband/driver.h> not found. Is libibverbs installed?
checking infiniband/driver.h usability... no
checking infiniband/driver.h presence... no
checking for infiniband/driver.h... no
configure: error: <infiniband/driver.h> not found. Is libibverbs installed?
[root@localhost userlib]# cp /usr/src/debug/libibverbs-1.2.1/include/infiniband/* /usr/include/infiniband/
then the "make & make install" seems work fine.
//Here is the kernel part, where errors occurred.
[root@localhost kernel]# make
make[1]: Entering directory `/usr/src/kernels/3.10.0-514.el7.x86_64'
CC [M] /home/softiwarp-siw-pre4.4/kernel/siw_main.o
In file included from /home/softiwarp-siw-pre4.4/kernel/siw_main.c:55:0:
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h: In function ‘get_tcp_mss’:
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h:165:8: error: ‘struct tcp_sock’ has no member named ‘xmit_size_goal_segs’
if (tp->xmit_size_goal_segs)
^
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h:166:12: error: ‘struct tcp_sock’ has no member named ‘xmit_size_goal_segs’
CC [M] /home/softiwarp-siw-pre4.4/kernel/siw_main.o
In file included from /home/softiwarp-siw-pre4.4/kernel/siw_main.c:55:0:
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h: In function ‘get_tcp_mss’:
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h:165:8: error: ‘struct tcp_sock’ has no member named ‘xmit_size_goal_segs’
if (tp->xmit_size_goal_segs)
^
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h:166:12: error: ‘struct tcp_sock’ has no member named ‘xmit_size_goal_segs’
return tp->xmit_size_goal_segs * tp->mss_cache;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c: In function ‘siw_device_create’:
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:415:24: warning: assignment from incompatible pointer type [enabled by default]
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:415:24: warning: assignment from incompatible pointer type [enabled by default]
ofa_dev->query_device = siw_query_device;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:434:21: warning: assignment from incompatible pointer type [enabled by default]
ofa_dev->create_cq = siw_create_cq;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:439:9: error: ‘struct ib_device’ has no member named ‘reg_phys_mr’
ofa_dev->reg_phys_mr = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:440:9: error: ‘struct ib_device’ has no member named ‘rereg_phys_mr’
ofa_dev->rereg_phys_mr = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:444:9: error: ‘struct ib_device’ has no member named ‘bind_mw’
ofa_dev->bind_mw = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:455:23: warning: assignment from incompatible pointer type [enabled by default]
ofa_dev->reg_phys_mr = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:440:9: error: ‘struct ib_device’ has no member named ‘rereg_phys_mr’
ofa_dev->rereg_phys_mr = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:444:9: error: ‘struct ib_device’ has no member named ‘bind_mw’
ofa_dev->bind_mw = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:455:23: warning: assignment from incompatible pointer type [enabled by default]
ofa_dev->process_mad = siw_no_mad;
^
make[2]: *** [/home/softiwarp-siw-pre4.4/kernel/siw_main.o] Error 1
make[1]: *** [_module_/home/softiwarp-siw-pre4.4/kernel] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.10.0-514.el7.x86_64'
make: *** [modules] Error 2
[root@localhost kernel]# make |less
[root@localhost kernel]# make > error.txt
In file included from /home/softiwarp-siw-pre4.4/kernel/siw_main.c:55:0:
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h: In function ‘get_tcp_mss’:
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h:165:8: error: ‘struct tcp_sock’ has no member named ‘xmit_size_goal_segs’
if (tp->xmit_size_goal_segs)
^
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h:166:12: error: ‘struct tcp_sock’ has no member named ‘xmit_size_goal_segs’
make[1]: *** [_module_/home/softiwarp-siw-pre4.4/kernel] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.10.0-514.el7.x86_64'
make: *** [modules] Error 2
[root@localhost kernel]# make |less
[root@localhost kernel]# make > error.txt
In file included from /home/softiwarp-siw-pre4.4/kernel/siw_main.c:55:0:
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h: In function ‘get_tcp_mss’:
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h:165:8: error: ‘struct tcp_sock’ has no member named ‘xmit_size_goal_segs’
if (tp->xmit_size_goal_segs)
^
/home/softiwarp-siw-pre4.4/kernel/siw_cm.h:166:12: error: ‘struct tcp_sock’ has no member named ‘xmit_size_goal_segs’
return tp->xmit_size_goal_segs * tp->mss_cache;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c: In function ‘siw_device_create’:
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:415:24: warning: assignment from incompatible pointer type [enabled by default]
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:415:24: warning: assignment from incompatible pointer type [enabled by default]
ofa_dev->query_device = siw_query_device;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:434:21: warning: assignment from incompatible pointer type [enabled by default]
ofa_dev->create_cq = siw_create_cq;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:439:9: error: ‘struct ib_device’ has no member named ‘reg_phys_mr’
ofa_dev->reg_phys_mr = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:440:9: error: ‘struct ib_device’ has no member named ‘rereg_phys_mr’
ofa_dev->rereg_phys_mr = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:444:9: error: ‘struct ib_device’ has no member named ‘bind_mw’
ofa_dev->bind_mw = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:455:23: warning: assignment from incompatible pointer type [enabled by default]
ofa_dev->reg_phys_mr = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:440:9: error: ‘struct ib_device’ has no member named ‘rereg_phys_mr’
ofa_dev->rereg_phys_mr = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:444:9: error: ‘struct ib_device’ has no member named ‘bind_mw’
ofa_dev->bind_mw = NULL;
^
/home/softiwarp-siw-pre4.4/kernel/siw_main.c:455:23: warning: assignment from incompatible pointer type [enabled by default]
ofa_dev->process_mad = siw_no_mad;
^
make[2]: *** [/home/softiwarp-siw-pre4.4/kernel/siw_main.o] Error 1
make[1]: *** [_module_/home/softiwarp-siw-pre4.4/kernel] Error 2
make: *** [modules] Error 2
make[1]: *** [_module_/home/softiwarp-siw-pre4.4/kernel] Error 2
make: *** [modules] Error 2
Then I searched the Kernel 3.10.0-514.16.1.el7 source code by keyword "tcp_sock", and have not found any structs named "tcp_sock", Does that mean the oldest version of softiwarp we can get, is not supported by Kernel 3.10? We want to use softiwarp RDMA to do Qemu RDMA live migration test, which chooses your software as default option in case the infiniband hardware does not exist. However, considering the stability for our NFVI system, we are unable to switch to the newest Kernel version, is there any way we can fix this problem?
kind regards,
Xun Huang
Bernard Metzler
Oct 27, 2017, 11:29:59 AM10/27/17
to 黄勋, zrlio-users, 明廷胜, 艾敬, 邓艳山
Hi Xun,
Maybe we could only fix 'by hand' the siw version for your target.
your kernel seem to include backports from newer kernels, so the
kernel interfaces changed compared to vanilla kernel.
We ran into similar issues when building for RedHat 7.2 a
whole wile ago. We defined a compile time flag 'IS_RH_7_2'
to cope with it. Maybe try adding that flag to your Kbuild
EXTRA_CFLAGS (add -DIS_RH_7_2 to it)?
It would be really good if you could install a newer
vanilla kernel. We do not have the resources to track all
distro kernels. Sorry about that...
Best,
Bernard.
-----zrlio...@googlegroups.com wrote: -----
To: "Bernard Metzler" <B...@zurich.ibm.com>
From: 黄勋
Sent by: zrlio...@googlegroups.com
Date: 10/27/2017 05:30AM
Cc: zrlio-users <zrlio...@googlegroups.com>, 明廷胜 <nwos...@fiberhome.com>, 艾敬 <aijin...@fiberhome.com>, 邓艳山 <ysd...@fiberhome.com>
To view this discussion on the web visit https://groups.google.com/d/msgid/zrlio-users/2017102711292808903441%40fiberhome.com.
Maybe we could only fix 'by hand' the siw version for your target.
your kernel seem to include backports from newer kernels, so the
kernel interfaces changed compared to vanilla kernel.
We ran into similar issues when building for RedHat 7.2 a
whole wile ago. We defined a compile time flag 'IS_RH_7_2'
to cope with it. Maybe try adding that flag to your Kbuild
EXTRA_CFLAGS (add -DIS_RH_7_2 to it)?
It would be really good if you could install a newer
vanilla kernel. We do not have the resources to track all
distro kernels. Sorry about that...
Best,
Bernard.
-----zrlio...@googlegroups.com wrote: -----
To: "Bernard Metzler" <B...@zurich.ibm.com>
From: 黄勋
Sent by: zrlio...@googlegroups.com
Date: 10/27/2017 05:30AM
Cc: zrlio-users <zrlio...@googlegroups.com>, 明廷胜 <nwos...@fiberhome.com>, 艾敬 <aijin...@fiberhome.com>, 邓艳山 <ysd...@fiberhome.com>
Subject: Re: Re: [zrlio-users] RDMA about qemu live migrate and softiwarp 3.10.0 version issue
黄勋
Nov 8, 2017, 2:16:09 AM11/8/17
to Bernard Metzler, zrlio-users, 明廷胜, 艾敬, 邓艳山, 易长路
Hi Bernard,
I spent a fews days trying again, and kernel 3.10.107 could support softiwarp-siw-pre4.4. The files under userlib directory can be compiled correctly with libibverbs-devel-1.1.8-4.el6.x86_64 and libibverbs-1.1.8-4.el6.x86_64, and I insmod siw.ko and other things,Then I went into '/home/libibverbs-1.1.8/examples', which contained some executable files, typing
[root@localhost examples]# ./ibv_devices
device node GUID
------ ----------------
siw_virbr0 52540085119e0000
siw_ens33 000c29f2f2cd0000
siw_lo 7369775f6c6f0000
device node GUID
------ ----------------
siw_virbr0 52540085119e0000
siw_ens33 000c29f2f2cd0000
siw_lo 7369775f6c6f0000
[root@localhost examples]# ./ibv_devinfo
hca_id: siw_virbr0
transport: iWARP (1)
fw_ver: 0.0.0
node_guid: 5254:0085:119e:0000
sys_image_guid: 5254:0085:119e:0000
vendor_id: 0x626d74
hca_id: siw_virbr0
transport: iWARP (1)
fw_ver: 0.0.0
node_guid: 5254:0085:119e:0000
sys_image_guid: 5254:0085:119e:0000
vendor_id: 0x626d74
......
which seems my work is correct.
After that, I yum install brdmacm-utils.x86_64 which provides 'rping' command, then I got stuck to it, and have not figured out how to use it correctly,
I tried like this,
[root@localhost home]# rping -s
rdma_create_event_channel: No such device
[root@localhost home]# rping -s siw_lo
rdma_create_event_channel: No such device
rdma_create_event_channel: No such device
[root@localhost home]# rping -s siw_lo
rdma_create_event_channel: No such device
wanted to build a server side and maybe a client side later, to rping the server side.
Haven't found useful references or webpages to show how to use rping, Do you have any ideas on it? Or are there any other utils instead of rping can test the connection of two hosts using softiwarp?
thanks
Kind Regards,
Xun
黄勋
Xun Huang
烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
网络产出线 研发部 集成测试部(Network Business Unit Product Testing Department Testing Department 1)
Email: huangx5749@fiberhome.com
湖北省武汉市东湖高新技术开发区高新四路6号
Address:（No.6,Gaoxinsilu,East Lake High-tech Development Zone,Wuhan,Hubei)
Bernard Metzler
Nov 8, 2017, 4:20:42 AM11/8/17
to 黄勋, zrlio-users, 明廷胜, 艾敬, 邓艳山, 易长路
Hi Xun,
you'd need to have user level RDMA connection management enabled. Can you
please check if the rdma_ucm kernel module is loaded? If not,
please 'modprobe' it.
If rping now works, please also check if your system allows applications
to register sufficient memory for RDMA communication. Edit the
/etc/security/limits.conf file accordingly. For example, you
may add
xun soft memlock unlimited
xun hard memlock unlimited
to the end of that file, if 'xun' is your login name on that
machine. This allows you to pin any amount of allocated
user memory within the kernel for RDMA access by siw.
Remember to use a new shell after editing the limits.conf to have
changes taking effect.
btw: you may restrict siw from using all available interfaces
on the system. For that, insert the siw module with a list
of interfaces siw shall use:
'sudo insmod ./siw.ko iface_list=lo,ens33'
...would not attach siw to virbr0 in your example.
rping should come with a man page. If not,
consider https://linux.die.net/man/1/rping
or other web resources.
Let us know how it goes.
Thanks,
Cc: zrlio-users <zrlio...@googlegroups.com>, 明廷胜 <nwos...@fiberhome.com>, 艾敬 <aijin...@fiberhome.com>, 邓艳山 <ysd...@fiberhome.com>, 易长路 <yichan...@fiberhome.com>
黄勋
Xun Huang烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
To view this discussion on the web visit https://groups.google.com/d/msgid/zrlio-users/2017110815145581869640%40fiberhome.com.
you'd need to have user level RDMA connection management enabled. Can you
please check if the rdma_ucm kernel module is loaded? If not,
please 'modprobe' it.
If rping now works, please also check if your system allows applications
to register sufficient memory for RDMA communication. Edit the
/etc/security/limits.conf file accordingly. For example, you
may add
xun soft memlock unlimited
xun hard memlock unlimited
to the end of that file, if 'xun' is your login name on that
machine. This allows you to pin any amount of allocated
user memory within the kernel for RDMA access by siw.
Remember to use a new shell after editing the limits.conf to have
changes taking effect.
btw: you may restrict siw from using all available interfaces
on the system. For that, insert the siw module with a list
of interfaces siw shall use:
'sudo insmod ./siw.ko iface_list=lo,ens33'
...would not attach siw to virbr0 in your example.
rping should come with a man page. If not,
consider https://linux.die.net/man/1/rping
or other web resources.
Let us know how it goes.
Thanks,
Bernard.
-----zrlio...@googlegroups.com wrote: -----
To: "Bernard Metzler" <B...@zurich.ibm.com>
From: 黄勋
Sent by: zrlio...@googlegroups.com
Date: 11/08/2017 08:16AM
-----zrlio...@googlegroups.com wrote: -----
To: "Bernard Metzler" <B...@zurich.ibm.com>
From: 黄勋
Sent by: zrlio...@googlegroups.com
Cc: zrlio-users <zrlio...@googlegroups.com>, 明廷胜 <nwos...@fiberhome.com>, 艾敬 <aijin...@fiberhome.com>, 邓艳山 <ysd...@fiberhome.com>, 易长路 <yichan...@fiberhome.com>
Subject: Re: Re: [zrlio-users] RDMA about qemu live migrate and softiwarp 3.10.0 version issue
Hi Bernard,
Xun Huang烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
网络产出线 研发部 集成测试部(Network Business Unit Product Testing Department Testing Department 1)
Email: huang...@fiberhome.com
黄勋
Nov 9, 2017, 3:50:49 AM11/9/17
to Bernard Metzler, zrlio-users, 明廷胜, 艾敬, 邓艳山, 易长路
Hi Bernard,
In my previous email, I said "yum install librdmacm-utils.x86_64" , that was one of the reason I failed,
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
libibverbs x86_64 13-7.el7 base 194 k
replacing libmthca.x86_64 1.0.6-4.el6
librdmacm-utils x86_64 13-7.el7 base 77 k
rdma-core-devel x86_64 13-7.el7 base 194 k
replacing libibverbs-devel.x86_64 1.1.8-4.el6
replacing librdmacm-devel.x86_64 1.0.21-0.el6
Installing for dependencies:
ibacm x86_64 13-7.el7 base 75 k
libibcm x86_64 13-7.el7 base 14 k
Updating for dependencies:
librdmacm x86_64 13-7.el7 base 59 k
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
libibverbs x86_64 13-7.el7 base 194 k
replacing libmthca.x86_64 1.0.6-4.el6
librdmacm-utils x86_64 13-7.el7 base 77 k
rdma-core-devel x86_64 13-7.el7 base 194 k
replacing libibverbs-devel.x86_64 1.1.8-4.el6
replacing librdmacm-devel.x86_64 1.0.21-0.el6
Installing for dependencies:
ibacm x86_64 13-7.el7 base 75 k
libibcm x86_64 13-7.el7 base 14 k
Updating for dependencies:
librdmacm x86_64 13-7.el7 base 59 k
if I confirm to install, many of my lib packages will be automatically replaced by newest virsion that yum provides by default, chaos happens.
So, I downloaded the packet manully from this website https://centos.pkgs.org/6/centos-x86_64
and installed all the things I needed using 'rpm -i', and compiled librdmacm-1.0.21.tar.gz by hand, then make & make install.
After that, I used rping command within this dir to test:
[root@localhost examples]# strace -f ./rping -s
it prints:
....
[pid 27337] futex(0x61f1a8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 27352] futex(0x61f1a8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 27337] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27352] <... futex resumed> ) = 0
[pid 27337] futex(0x61f1a8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 27352] futex(0x61f1a8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 27337] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27352] <... futex resumed> ) = 0
[pid 27337] futex(0x61f1a8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
and waiting here, after I typed rping -c as below,
[root@localhost examples]# ./rping -c -a 127.0.0.1 -v
the server terminal printed one more line:
[pid 27337] futex(0x61f1a8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 27352] futex(0x61f1a8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 27337] <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27352] <... futex resumed> ) = 0
[pid 27337] futex(0x61f1a8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 27352] write(3, "\f\0\0\0\10\0H\0010]\235\241\312\177\0\0", 16
and stuck here, empty after the number '16', nothing new happened afterwards, if ctrl+c the client and rping -c again, the client termital would print:
[root@localhost examples]# ./rping -c -a 127.0.0.1 -v
cma event RDMA_CM_EVENT_UNREACHABLE, error -110the server side prints:(secont attempt without using ftrace -f)
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
-----------------------------------
I have set /etc/security/limits.conf
and [root@localhost examples]# lsmod |grep rdma
xprtrdma 50626 0 rdma_ucm 22316 2
ib_uverbs 42495 10 ib_ucm,rdma_ucm
rdma_cm 43138 4 xprtrdma,ib_iser,rdma_ucm,ib_isert
ib_addr 14070 1 rdma_cm
ib_cm 42717 5 rdma_cm,ib_srp,ib_ucm,ib_srpt,ib_ipoib
ib_sa 29615 5 rdma_cm,ib_cm,ib_srp,rdma_ucm,ib_ipoib
iw_cm 18669 1 rdma_cm
ib_core 82647 17 siw,rdma_cm,ib_cm,ib_sa,iw_cm,xprtrdma,ib_mad,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert,ib_mthca
sunrpc 271941 8 nfsd,auth_rpcgss,lockd,xprtrdma,nfs_acl
[root@localhost home]# ibv_devices
device node GUID
------ ----------------
siw_virbr0 52540085119e0000
siw_ens33 000c29f2f2cd0000
siw_lo 7369775f6c6f0000
seems nothing about configuration goes wrong, however, my colleague did the similar operations as me, could successfully rping under ubuntu 16.04 with kernel 3.10.107.
btw. I make sure that firewalld and selinux have all been disabled.
thanks.
黄勋
Xun Huang
烽火通信科技股份有限公司(Fiberhome Telecommunication Technologies Co.,LTD)
网络产出线 研发部 集成测试部(Network Business Unit Product Testing Department Testing Department 1)
Email: huangx5749@fiberhome.com
Reply all
Reply to author
Forward
0 new messages