Compiling with infiniband support

68 views
Skip to first unread message

Aditya Bandla

unread,
May 28, 2019, 10:38:49 AM5/28/19
to upc-users
I'm trying to enable support for infiniband on a cluster running RHEL6. The configure script fails with the following error


configure error: User requested --enable-ibv but I don't know how to build ibv programs for your system

configure: error: See /home/users/nus/build/dbg_cupc2c/gasnet/config.log for details.

configure: error: sub-configure failed for gasnet

Failed during dbg_cupc2c configure, exit=1


I had a look at the config.log which had the following message related to ibv

conftest.c: In function 'main':

conftest.c:217:15: warning: initialization discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]

     char *s = ibv_get_device_name(&device);

               ^~~~~~~~~~~~~~~~~~~

/scratch/users/nus/miniconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: warning: libnl.so.1, needed by /usr/lib64/libibverbs.so, not found (try using -rpath or -rpath-link)

/scratch/users/nus/miniconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: /usr/lib64/libibverbs.so: undefined reference to `nl_connect'


I tried pointing ld to /lib64 where libnl.so.1 is found, but configure still fails

Any help would be much appreciated 



Paul Hargrove

unread,
May 28, 2019, 1:11:22 PM5/28/19
to Aditya Bandla, upc-users
Aditya,

Sorry to hear you are having problems.
Unfortunately, I don't believe any member of our team has seen the type of failure you report.

I should start by stating that the compiler warning is harmless and that you are correct to focus on the link failure.

I am able to build ibv-conduit without any problems on a system running Scientific Linux 6.10 (a de-branded RHEL6 much like Centos).

At the moment, the two theories we have are that the library may be damaged, or that there is something unusual about the ld you are using (not /usr/bin/ld).  Of these, the first is the easier to check on.
Below are a series of commands I ran to inspect libnl.  I suggest you try them as well and share your output.

First, I just wanted to confirm the location of libnl.so, which I did using ldd and one of the standard InfiniBand utilities:

$ ldd `which ibv_devices`
        linux-vdso.so.1 =>  (0x00007ffdd3fb0000)
        libibverbs.so.1 => /usr/lib64/libibverbs.so.1 (0x00007f04a400e000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f04a3df1000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f04a3bed000)
        libnl.so.1 => /lib64/libnl.so.1 (0x00007f04a399b000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f04a3607000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f04a4220000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f04a3383000)

Second, I checked with the package manager to determine the package containing the file highlighted above.

$ rpm -qf /lib64/libnl.so.1
libnl-1.1.4-2.el6.x86_64

Third, I asked the package manager to verify the library above is not modified (as confirmed by the presence of only dots in the first several columns).  The last argument to the command is the full package name highlighted in the output above.

$ rpm -qvV libnl-1.1.4-2.el6.x86_64
.........    /lib64/libnl.so.1
.........    /lib64/libnl.so.1.1.4
.........    /usr/share/doc/libnl-1.1.4
.........  d /usr/share/doc/libnl-1.1.4/COPYING


Hopefully the output of one of those commands will differ from mine in some way that will point us to the root of your problem.
If not, we can proceed to look at your linker and the environment variables which may influence its behavior.

-Paul

--
You received this message because you are subscribed to the Google Groups "upc-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upc-users+...@lbl.gov.
To post to this group, send email to upc-...@lbl.gov.
Visit this group at https://groups.google.com/a/lbl.gov/group/upc-users/.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/upc-users/9db8c44a-c12f-4e4b-b280-481de13c7c54%40lbl.gov.


--
Paul H. Hargrove <PHHar...@lbl.gov>
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department
Lawrence Berkeley National Laboratory

Aditya Bandla

unread,
May 28, 2019, 8:39:40 PM5/28/19
to upc-users, aditya...@gmail.com
Dear Paul

Thank you for the pointers, they were extremely useful. I was able to reproduce the commands and outputs you had given. It turned out that my linker syntax was wrong. 
I was able to successfully compile and install BUPC!

Following is the LDFLAGS I used, which worked
export LDFLAGS='-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath-link /lib64'

However, my next step was to build HIPMER and I run into the same exact error when HIPMER tries to link to the upcc executable. As HIPMER uses cmake, I am not sure how to pass the -rpath-link flag to ld

I have tried multiple flags, but thus far it keeps failing 

To unsubscribe from this group and stop receiving emails from it, send an email to upc-...@lbl.gov.

Paul Hargrove

unread,
May 29, 2019, 1:36:17 AM5/29/19
to Aditya Bandla, upc-users
Aditya,

I am glad that you have BUPC working.

I am not a user of cmake, but I think the following might contain the answers you need.

-Paul

To unsubscribe from this group and stop receiving emails from it, send an email to upc-users+...@lbl.gov.

To post to this group, send email to upc-...@lbl.gov.
Visit this group at https://groups.google.com/a/lbl.gov/group/upc-users/.
Reply all
Reply to author
Forward
0 new messages