building and linking with Intel libraries

[Florent Pruvost]

Hi,

I just started to play a little bit with Spack that I may use for the installation of complex scientific software stacks.

I tried to use the %intel in an installation process and I meet a problem with symbols coming from intel libraries that I don't encounter when I build the software outside Spack.

More specifically I want to write a package for the StarPU library (see a simplified version of the Spack package.py in attached) which is installed using autotools building system.
starpu homepage: http://starpu.gforge.inria.fr/

Spack founds compilers on my system (Ubuntu 13.10):

pruvost@hp9470m:~/work/gitwc/spack$ spack compilers
==> Available compilers
-- gcc ----------------------------------------------------------
gcc@4.8

-- intel --------------------------------------------------------
in...@14.0.2

pruvost@hp9470m:~/work/gitwc/spack$ spack compiler info intel
in...@14.0.2:
    cc  = /opt/intel/bin/icc
    cxx = /opt/intel/bin/icpc
    f77 = /opt/intel/bin/ifort
    fc  = /opt/intel/bin/ifort

With gcc "spack install starpu" works fine
but with intel (spack install starpu %intel), I get the following error:

usr/bin/ld.bfd.real: .libs/starpu_perfmodel_plot: hidden symbol `__intel_cpu_features_init_x' in /opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/libirc.a(cpu_feature_disp.o) is referenced by DSO
/usr/bin/ld.bfd.real: final link failed: Bad value
/usr/bin/ld.bfd.real: warning: libimf.so, needed by ../src/.libs/libstarpu-1.1.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld.bfd.real: warning: libsvml.so, needed by ../src/.libs/libstarpu-1.1.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld.bfd.real: warning: libirng.so, needed by ../src/.libs/libstarpu-1.1.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld.bfd.real: warning: libintlc.so.5, needed by ../src/.libs/libstarpu-1.1.so, not found (try using -rpath or -rpath-link)
make[2]: *** [starpu_perfmodel_plot] Erreur 1

If I try to compile starpu in my environement setting the following env. variables, compilation of starpu works:
    export CC=/opt/intel/bin/icc
    export CXX=/opt/intel/bin/icpc
    export FC=/opt/intel/bin/ifort
    export PATH=$PATH:/opt/intel/bin
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64

Do you have any idea of what is happening in Spack environement?

Thanks,

Florent.

[Todd Gamblin]

Florent,

I'm not sure what that error is.  I built your package without errors on an LLNL machine (TOSS / RHEL6.6) with both in...@14.0.2 and intel@15.x.

If you want to see what's different about the Spack environment, you can run this:

spack env starpu%in...@14.0.2

That will print out the environment it uses to build.  If you want to actually run bash in that environment, do this:

spack env starpu%in...@14.0.2 bash # or your favorite shell

Once in that environment, you can try running the compile as Spack would with "cc", "c++", etc. -- those are the compiler wrappers.

Finally if you want to see what compile lines are actually being used in the build you can do this:

spack -d install starpu%in...@14.0.2

The -d will give you two log files <spec>.in.log and <spec>.out.log, which show the input compiler parameters (the ones the build passed to the wrappers) and the ones that Spack used (the ones that were actually passed to the compiler).

I'd be interested to know what this problem is, if you figure it out, or if you want to send along more details I can see if I can make sense of things.

-Todd

--
You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.
Visit this group at http://groups.google.com/group/spack.
For more options, visit https://groups.google.com/d/optout.

[Florent Pruvost]

Hi Todd,

It seems that during the link of starpu executables the path to intel libraries is missing.

Indeed, adding "LDFLAGS=-L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64" to the configure() function in the package make it compile.

In my environment it links thanks to the LD_LIBRARY_PATH which is unset in Spack env. I understand that you need to have a more or less confined env. for the builds.

What I don't understand is, generally, how do you find the path to intel libs and how do you use it during the link of executables? Something with the rpath?

I attach my log files for the build which fails (without the modification LDFLAGS=...).

Thanks,

Florent

g...@4.8

[Todd Gamblin]

Ok I think I understand this issue now.

This is actually related to this issue:

https://github.com/scalability-llnl/spack/issues/6

In that issue, Scott Pakin points out that LANL's gcc compilers require some environment settings to find their deps (they're not RPATH'd).  Icc also requires some environment settings to find its deps, and intel ships a sourceme script with the compilers to do that.

I looked a little deeper, and LLNL has a rather nice environment in that the site installs intel compilers so they "just work" -- on our systems, /usr/local/bin/icc-14.0.174 is a wrapper that loads the environment and delegates to the compilers.  So I guess I've been lucky.

I said in the issue above that we should bundle some  environment settings with the compilers.  In the case above I can get what I need from ldd.  For icc it looks like I need more.  If it's just another rpath to the compiler lib directory that should be easy to encode per compiler type -- e.g. we could put this knowledge in intel.py in lib/spack/spack/compilers, and Spack's compiler wrappers would know to set the compiler's environment accordingly (and it wouldn't pollute the build b/c this would only be set for the compiler within the wrapper).

Is there anything needed besides a) LD_LIBRARY_PATH for finding compiler deps, and b) figuring out rpaths to add for compiler runtime libs?

-Todd

From: Florent Pruvost <florent...@inria.fr>
Date: Wednesday, April 22, 2015 at 10:11 AM
To: "sp...@googlegroups.com" <sp...@googlegroups.com>
Cc: "pruvost...@gmail.com" <pruvost...@gmail.com>
Subject: Re: [spack] building and linking with Intel libraries

Hi Todd,

It seems that during the link of starpu executables the path to intel libraries is missing.

Indeed, adding "LDFLAGS=-L/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64" to the configure() function in the package make it compile.

In my environment it links thanks to the LD_LIBRARY_PATH which is unset in Spack env. I understand that you need to have a more or less confined env. for the builds.

What I don't understand is, generally, how do you find the path to intel libs and how do you use it during the link of executables? Something with the rpath?

I attach my log files for the build which fails (without the modification LDFLAGS=...).

Thanks,

Florent

On Wednesday, April 22, 2015 at 1:29:27 AM UTC+2, Todd Gamblin wrote:

Florent,

I'm not sure what that error is.  I built your package without errors on an LLNL machine (TOSS / RHEL6.6) with both in...@14.0.2 and intel@15.x.

If you want to see what's different about the Spack environment, you can run this:

spack env starpu%in...@14.0.2

That will print out the environment it uses to build.  If you want to actually run bash in that environment, do this:

spack env starpu%in...@14.0.2 bash# or your favorite shell

[Lee, Greg]

FYI, with the Intel compilers at LLNL, we add icc.cfg, icpc.cfg, and ifort.cfg files to the bin directory of the Intel compiler that add the appropriate rpath (i.e., -Xlinker -rpath -Xlinker ${libdir}), so it’s actually not the /usr/local/bin wrappers doing the rpath magic here. For Spack, you may want to explicitly add rpaths for the Intel compilers, since the .cfg files are just a local trick that we use at LLNL.

 

                -Greg

[Todd Gamblin]

Thanks Greg.  Adding RPATHs should actually be easier than adding extra environment settings, so this is good.  Is that all that's in the cfg or is there other stuff required?

[Lee, Greg]

Todd,

 

We also add "-long_double", but I think this is irrelevant to Spack.

 

                -Greg

[ajstew...@gmail.com]

Hi,

I think I'm running into the same problem as Florent. When trying to build openmpi with the intel compilers, it crashes on the libtool dependency. The error message from spack-build.out looks like:

checking for gcc... /blues/gpfs/home/software/spack/lib/spack/env/cc
checking whether the C compiler works... no
configure: error: in `/tmp/spack-stage/spack-stage-wK0pkl/libtool-2.4.2':
configure: error: C compiler cannot create executables

I ran into this same problem before when I was trying to build with gcc and it was because I didn't set rpath properly. I tried to follow the above suggestions and added icc.cfg, icpc.cfg, and ifort.cfg files to my bin directory (/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/bin). All three files contained the following:

-Xlinker -rpath -Xlinker /soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/lib

Unfortunately, this didn't solve the issue. I noticed that icc, icpc, and ifort are symlinks, so I tried copying these .cfg files to that directory as well. No luck. I checked what we were using for our own LD_LIBRARY_PATH before switching to Spack. It came from sourcing the <install-dir>/bin/compilervars.sh script. The result looks like this:

-Xlinker -rpath -Xlinker /soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/mpi/mic/lib:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/ipp/lib/intel64:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/debugger_2016/libipt/intel64/lib:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64_lin

This didn't work either. Any other suggestions?

[Francois Bissey]

corresponding config.log please. Likely the C compiler doesn’t find one of its libraries or one of the
libraries passed to configure cannot be found. There are a few more possibilities but the right info
can be deduced from config.log.

François

[ajstew...@gmail.com]

config.log attached

[ajstew...@gmail.com]

Update: I just discovered that if I source <intel_dir>/bin/compilervars.sh -- adding all of those paths to my global variables -- I can get Spack to build libtool just fine with the intel compilers. But if I don't add those paths, then Spack cannot build. Since Spack wipes away LD_LIBRARY_PATH regardless, it must be some other global variable that it was missing.

[Todd Gamblin]

Thanks for the update!

It's on my list to add a config option in compilers.yaml so that you can add a file that needs to be sourced *inside* the compiler wrapper for a particular compiler.  LLNL builds its Intel compilers so that they get their runtime lib paths from a static configuration file -- I think by default they require sourcing compilervars. 

This actually wouldn't be that big of a change -- if you want to try adding another field to compilers.yaml, then propagate it to the lib/spack/env/cc wrapper through build_environment.py, let me know. Otherwise I can try to do this sooner rather than later.

-Todd

From: <sp...@googlegroups.com> on behalf of "ajstew...@gmail.com" <ajstew...@gmail.com>
Date: Tuesday, December 15, 2015 at 8:07 AM
To: Spack <sp...@googlegroups.com>
Subject: Re: [spack] building and linking with Intel libraries

--

You received this message because you are subscribed to the Google Groups "Spack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
To post to this group, send email to sp...@googlegroups.com.

Visit this group at https://groups.google.com/group/spack.

[Todd Gamblin]

Oh, and FYI, we're adding, as part of the Cray support for NERSC, the ability to search for compilers in module files.  This requires a similar capability -- recording in compilers.yaml the module that needs to be loaded for a particular compiler.

[ajstew...@gmail.com]

LLNL builds its Intel compilers so that they get their runtime lib paths from a static configuration file -- I think by default they require sourcing compilervars.

How exactly do you do this? Is there something more than just creating the icc.cfg, icpc.cfg, and ifort.cfg files? I just discovered that the missing environmental variable was LIBRARY_PATH, not LD_LIBRARY_PATH!

[Todd Gamblin]

From: "ajstew...@gmail.com" <ajstew...@gmail.com>
Date: Tuesday, December 15, 2015 at 8:35 AM
To: Spack <sp...@googlegroups.com>
Cc: Todd Gamblin <tgam...@llnl.gov>, "ajstew...@gmail.com" <ajstew...@gmail.com>

Subject: Re: [spack] building and linking with Intel libraries

LLNL builds its Intel compilers so that they get their runtime lib paths from a static configuration file -- I think by default they require sourcing compilervars.

How exactly do you do this? Is there something more than just creating the icc.cfg, icpc.cfg, and ifort.cfg files? I just discovered that the missing environmental variable was LIBRARY_PATH, not LD_LIBRARY_PATH!

I think it's just the .cfg files but could be wrong.  We deploy some compiler wrappers as well, but I'm not sure if those are required.  Greg Lee would know more about this.

Greg?

-Todd

[Lee, Greg]

Yes, we just create the .cfg files and add “-Xlinker -rpath -Xlinker <path>”.

 

                -Greg

 

From: sp...@googlegroups.com [mailto:sp...@googlegroups.com] On Behalf Of ajstew...@gmail.com

Sent: Tuesday, December 15, 2015 8:35 AM
To: Spack <sp...@googlegroups.com>

Cc: tgam...@llnl.gov; ajstew...@gmail.com
Subject: Re: [spack] building and linking with Intel libraries

 

LLNL builds its Intel compilers so that they get their runtime lib paths from a static configuration file -- I think by default they require sourcing compilervars.

How exactly do you do this? Is there something more than just creating the icc.cfg, icpc.cfg, and ifort.cfg files? I just discovered that the missing environmental variable was LIBRARY_PATH, not LD_LIBRARY_PATH!

--

[ajstew...@gmail.com]

Hmm. I'm not sure why that's not working for me. Can you post the exact contents and location of icc.cfg? My icc.cfg is in:

/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/bin

And contains:

-Xlinker -rpath -Xlinker /soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/lib

I have identical icpc.cfg and ifort.cfg files in the same directory. But it's not working for me for some reason.

[Lee, Greg]

I believe the .cfg files need to be in the actual bin directory where the mcpcom binary lives. For us this is in:

 

/usr/local/tools/ic-16.0.109/compilers_and_libraries_2016.0.109/linux/bin/intel64/icc.cfg

 

cab687@lee218:cat /usr/local/tools/ic-16.0.109/compilers_and_libraries_2016.0.109/linux/bin/intel64/icc.cfg

-Xlinker -rpath -Xlinker /usr/local/tools/ic-16.0.109/lib

 

You may notice we point it to a symlinked lib directory.

 

                -Greg

 

From: sp...@googlegroups.com [mailto:sp...@googlegroups.com] On Behalf Of ajstew...@gmail.com
Sent: Tuesday, December 15, 2015 8:50 AM
To: Spack <sp...@googlegroups.com>
Cc: ajstew...@gmail.com; tgam...@llnl.gov
Subject: Re: [spack] building and linking with Intel libraries

 

Hmm. I'm not sure why that's not working for me. Can you post the exact contents and location of icc.cfg? My icc.cfg is in:

--

[ajstew...@gmail.com]

I moved my icc.cfg, icpc.cfg, and ifort.cfg files into:

/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/bin/intel64

Unfortunately, still no luck. Any idea how to add LIBRARY_PATH to icc.cfg?

[Lee, Greg]

I’m not certain, but could you just do a -L in the .cfg file?

 

From: sp...@googlegroups.com [mailto:sp...@googlegroups.com] On Behalf Of ajstew...@gmail.com
Sent: Tuesday, December 15, 2015 9:05 AM
To: Spack <sp...@googlegroups.com>
Cc: ajstew...@gmail.com; tgam...@llnl.gov
Subject: Re: [spack] building and linking with Intel libraries

 

I moved my icc.cfg, icpc.cfg, and ifort.cfg files into:

/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/bin/intel64

Unfortunately, still no luck. Any idea how to add LIBRARY_PATH to icc.cfg?

--

[ajstew...@gmail.com]

Still no luck, unless I'm doing something wrong. I tried:

-L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/comp
ilers_and_libraries_2016.1.150/linux/ipp/lib/intel64:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin:/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/daal/../compiler/lib/intel64_lin

which is what LIBRARY_PATH is set to when sourcing bin/compilervars.sh. I also tried:

-L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/lib

But no luck

[Lee, Greg]

Does the -L option take a colon separated list of paths or do you need to enumerate each path as a -L option? Also, perhaps you can run icc -v to make sure the option is being passed appropriately.

 

I will also point out that the <prefix>/lib directory of our Intel compiler installations is a custom symlink that we add that points to the actual underlying directory that has the libraries (/usr/local/tools/ic-16.0.109/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64).

 

                -Greg

 

From: sp...@googlegroups.com [mailto:sp...@googlegroups.com] On Behalf Of ajstew...@gmail.com
Sent: Tuesday, December 15, 2015 9:31 AM
To: Spack <sp...@googlegroups.com>
Cc: ajstew...@gmail.com; tgam...@llnl.gov
Subject: Re: [spack] building and linking with Intel libraries

 

Still no luck, unless I'm doing something wrong. I tried:

--

[ajstew...@gmail.com]

It works!

For those of you following along, the following steps solved my issue:

1. Determine the LIBRARY_PATH needed by the Intel compilers. This involves sourcing bin/compilervars.sh. The newly added paths in LIBRARY_PATH are what you want. If you're like me and installed your pre-compiled binaries in a path like:
   
   /soft/spack/opt/spack/linux-x86_64/binaries
   
   then you will need to take a look at the bug I found here:
   
   https://software.intel.com/en-us/forums/intel-data-analytics-acceleration-library/topic/604617
   
   
2. Create icc.cfg, icpc.cfg, and ifort.cfg files in the following directory:
   
   /soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/bin/intel64
   
   It will not work if you put them in:
   
   /soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/bin
   
   Each of these files will contain the same thing; the newly added paths in LIBRARY_PATH, with " -L" separating them instead of colons. For me, the entire icc.cfg looks like this:
   
   -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/ipp/lib/intel64 -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64 -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4 -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/daal/../compiler/lib/intel64_lin

Note: it was not necessary to change the lib symlink, because lib points to compilers_and_libraries/linux/lib which points to .../../compilers_and_libraries_2016.1.150/linux/compiler/lib.

I would try the previous suggestions for setting rpath first. If that doesn't work, try my way.

Thanks for your help Greg!

[Todd Gamblin]

Nice!  I'm glad you got this working.  I hope this isn't necessary after we add some more support for source me files.

From: "ajstew...@gmail.com" <ajstew...@gmail.com>
Date: Tuesday, December 15, 2015 at 12:54 PM
To: Spack <sp...@googlegroups.com>
Cc: "ajstew...@gmail.com" <ajstew...@gmail.com>, Todd Gamblin <tgam...@llnl.gov>
Subject: Re: [spack] building and linking with Intel libraries

[ajstew...@gmail.com]

Yes, being able to source something like bin/compilervars.sh or even to be able to set environment variables (like the PGI license file in my other post) would be incredibly useful. compilevars.sh also sets dozens of other variables that may or may not be crucial and aren't set in Spack. Only time will tell whether or not I'll have to add more lines to icc.cfg in order to add the right paths.

[Massimiliano Culpo]

This post was quite useful to set up Intel compilers (version 15.0.2) with spack. I'll add a few notes on what I found skimming through the documentation:

1. The configuration file that correctly adds rpaths on my cluster reads:
   -Xlinker -rpath=/ssoft/intel/15.0.0/RH6/all/x86_E5v2/composer_xe_2015.2.164/ipp/lib/intel64 -Xlinker -rpath=/soft/intel/15.0.0/RH6/all/x86_E5v2/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4

   -Xlinker -rpath=/ssoft/intel/15.0.0/RH6/all/x86_E5v2/composer_xe_2015.2.164/mkl/lib/intel64 -Xlinker -rpath=/ssoft/intel/15.0.0/RH6/all/x86_E5v2/composer_xe_2015.2.164/compiler/lib/intel64
2. It seems there's no need to put icc.cfg and related files in a specific location. Instead one can just set the environment variables ICCCFG, ICPCCFG, IFORTCFG to the appropriate paths.

M.

[Pramod Kumbhar]

Hi Massimiliano and All,

I am following your instruction but I think missing some details. This is what I have done:

Here is compiler info:

$ spack compiler info intelintel@15.0.0:

       cc  = /gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/bin/intel64/icc

       cxx = /gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/bin/intel64/icpc

       f77 = /gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/bin/intel64/ifort

       fc  = /gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/bin/intel64/ifort

       modules  = []

       operating system  = redhat6

(the modules are installed by admins and I don't have permissions to change)

If I load intel module, LIBRARY_PATH is:

-bash-4.1$ echo $LIBRARY_PATH

/gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/compiler/lib/intel64:/gpfs/bbp.cscs.ch/apps/viz/intel/compiler/lib/intel64:/gpfs/bbp.cscs.ch/apps/viz/intel/impi/5.0.1.035/intel64/lib:/gpfs/bbp.cscs.ch/apps/viz/intel/ipp/lib/intel64:/gpfs/bbp.cscs.ch/apps/viz/intel/ipp/../compiler/lib/intel64:/gpfs/bbp.cscs.ch/apps/viz/intel/composerxe/mkl/lib/intel64:/gpfs/bbp.cscs.ch/apps/viz/intel/mkl/lib/intel64:/gpfs/bbp.cscs.ch/apps/viz/intel/tbb/lib/intel64

I created .cfg files with below content:

$ cat $ICCCFG

-Xlinker -rpath=/gpfs/bbp.cscs.ch/apps/viz/intel/ipp/lib/intel64 -Xlinker -rpath=/gpfs/bbp.cscs.ch/apps/viz/intel/tbb/lib/intel64/gcc4.4

-Xlinker -rpath=/gpfs/bbp.cscs.ch/apps/viz/intel/composerxe/mkl/lib/intel64 -Xlinker -rpath=/gpfs/bbp.cscs.ch/apps/viz/intel/compiler/lib/intel64

And set following env variables:

export IFORTCFG=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/cf/ifort.cfg

export ICPCCFG=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/cf/icpc.cfg

export ICPCCFG=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/cf/icpc.cfg

Now trying to install libxml2 with intel compiler end with an error:

$ spack install -v libxml2 %intel
....
ld: .libs/testAutomata: hidden symbol `__intel_cpu_features_init_x' in /gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/compiler/lib/intel64/libirc.a(cpu_feature_disp.o) is referenced by DSO

ld: final link failed: Nonrepresentable section on output

Any suggestion?

env info:

 spack env libxml2%intel

SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass

LOADEDMODULES=git/1.8.4.3:cmake/2.8.12:slurm/14.03.4.2

KDE_IS_PRELINKED=1

SLURM_TIME_FORMAT=relative

FC=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env/intel/ifort

IFORTCFG=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/cf/ifort.cfg

SPACK_FILE=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/bin/spack

CMAKE_VERSION_STRING=2.8.12

KDEDIRS=/usr

LESSOPEN=||/usr/bin/lesspipe.sh %s

SPACK_CC_RPATH_ARG=-Wl,-rpath,

SSH_CLIENT=128.179.253.248 65327 22

CVS_RSH=ssh

LOGNAME=kumbhar

USER=kumbhar

SPACK_PREFIX=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.0/libxml2-2.9.2-43kkhoikl2zeu2wczhqk2apt3rnvhs4y

PATH=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.0/xz-5.2.2-6tk6lowjdbkudq2a7f6wbjnfa5zdeptk/bin:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env/case-insensitive:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env/intel:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/bin:/gpfs/bbp.cscs.ch/apps/viz/tools/cmake/cmake-2.8.12/install/bin:/gpfs/bbp.cscs.ch/apps/viz/tools/git/git-1.8.4.3/install/bin:/usr/lib64/qt-3.3/bin:/gpfs/bbp.cscs.ch/home/kumbhar//.nix-profile/bin:/nix/var/nix/profiles/default/bin:/opt/cluster/admin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/c3-4:/usr/lpp/mmfs/bin/:/opt/ibutils/bin:/gpfs/bbp.cscs.ch/apps/common/tools/SystemScriptsCSCS:/gpfs/bbp.cscs.ch/home/kumbhar/bin

QTDIR=/usr/lib64/qt-3.3

HISTSIZE=1000

SPACK_ENV_PATH=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env/case-insensitive:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env/intel

SSH_CONNECTION=128.179.253.248 65327 148.187.84.11 22

LANG=en_US.UTF-8

QTLIB=/usr/lib64/qt-3.3/lib

TERM=xterm-256color

SHELL=/bin/bash

SPACK_SHORT_SPEC=lib...@2.9.2%in...@15.0.0~python arch=linux-redhat6-x86_64-43kkhoi

GIT_ROOT=/gpfs/bbp.cscs.ch/apps/viz/tools/git/git-1.8.4.3/install

QTINC=/usr/lib64/qt-3.3/include

DK_NODE=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/share/spack/dotkit/linux-redhat6-x86_64

SPACK_CXX_RPATH_ARG=-Wl,-rpath,

SQUEUE_FORMAT=%6i %8u %7a %.14j %.15P%.3t %.10r %.12S %.12e %.10L %.5D

APPS=/gpfs/bbp.cscs.ch/apps

SPACK_FC=/gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/bin/intel64/ifort

SINFO_FORMAT=%15P %5a %8s %.10l %.6c %.6z %.7D %10T %N

CCACHE_HASHDIR=

SQUEUE_SORT=t,-e,-p

SPACK_CXX=/gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/bin/intel64/icpc

MANPATH=/gpfs/bbp.cscs.ch/apps/viz/tools/cmake/cmake-2.8.12/install/man:/gpfs/bbp.cscs.ch/apps/viz/tools/git/git-1.8.4.3/install/man:/share/man::/opt/c3-4/man:

SPACK_F77=/gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/bin/intel64/ifort

GIT_LIBRARIES=-L$/gpfs/bbp.cscs.ch/apps/viz/tools/git/git-1.8.4.3/install/lib -lgsl -lgslcblas

G_BROKEN_FILENAMES=1

HOME=/gpfs/bbp.cscs.ch/home/kumbhar

MODULESHOME=/usr/share/Modules

GIT_INCLUDE_DIR=/gpfs/bbp.cscs.ch/apps/viz/tools/git/git-1.8.4.3/install/include

CMAKE_PREFIX_PATH=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.0/xz-5.2.2-6tk6lowjdbkudq2a7f6wbjnfa5zdeptk:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.0/zlib-1.2.8-jy2gq35ma7lfi234tb7quem75ixb3b5r

SPACK_DEPENDENCIES=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.0/xz-5.2.2-6tk6lowjdbkudq2a7f6wbjnfa5zdeptk:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.0/zlib-1.2.8-jy2gq35ma7lfi234tb7quem75ixb3b5r

ICCCFG=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/cf/icc.cfg

CC=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env/intel/icc

SPACK_INSTALL=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack

SPACK_COMPILER_SPEC=in...@15.0.0

SCRATCH=/gpfs/bbp.cscs.ch/scratch/gss/viz/kumbhar

CMAKE_ROOT=/gpfs/bbp.cscs.ch/apps/viz/tools/cmake/cmake-2.8.12/install

SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt

PKG_CONFIG_PATH=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.0/zlib-1.2.8-jy2gq35ma7lfi234tb7quem75ixb3b5r/lib/pkgconfig:/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.0/xz-5.2.2-6tk6lowjdbkudq2a7f6wbjnfa5zdeptk/lib/pkgconfig:/lib/pkgconfig

GIT_FOUND=1

GIT_HOME=/gpfs/bbp.cscs.ch/apps/viz/tools/git/git-1.8.4.3/install

F77=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env/intel/ifort

SPACK_CC=/gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/bin/intel64/icc

_=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/bin/spack

MODULEPATH=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/share/spack/modules/linux-redhat6-x86_64:/usr/share/Modules/modulefiles:/etc/modulefiles:/gpfs/bbp.cscs.ch/apps/viz/modulefiles:/gpfs/bbp.cscs.ch/apps/viz/modulefiles/BBP/viz:/gpfs/bbp.cscs.ch/apps/viz/modulefiles/BBP/platform:/gpfs/bbp.cscs.ch/apps/nix-releases/modules-lugano-viz-x86_64

ICPCCFG=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/cf/icpc.cfg

GIT_VERSION_STRING=1.8.4.3

SQUEUE_ALL=yes

_LMFILES_=/gpfs/bbp.cscs.ch/apps/viz/modulefiles/git/1.8.4.3:/gpfs/bbp.cscs.ch/apps/viz/modulefiles/cmake/2.8.12:/gpfs/bbp.cscs.ch/apps/viz/modulefiles/slurm/14.03.4.2

DISPLAY=bbpviz1.cscs.ch:11.0

CXX=/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/spack/env/intel/icpc

SSH_TTY=/dev/pts/2

HOSTNAME=bbpviz1.cscs.ch

NIX_REMOTE=daemon

HISTCONTROL=ignoredups

SHLVL=1

PWD=/gpfs/bbp.cscs.ch/home/kumbhar

CMAKE_FOUND=1

SPACK_DEBUG_LOG_DIR=/gpfs/bbp.cscs.ch/home/kumbhar

GIT_LIBRARY_DIRS=/gpfs/bbp.cscs.ch/apps/viz/tools/git/git-1.8.4.3/install/lib

SPACK_FC_RPATH_ARG=-Wl,-rpath,

MAIL=/var/spool/mail/kumbhar

CLUSTER=viz

SPACK_F77_RPATH_ARG=-Wl,-rpath,

BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`

}

[Kelly Thompson]

Florent,

I'm having the same issues that you report. Ultimately, I think we must wait for https://github.com/LLNL/spack/pull/943 to be finished and merged before compilers that depend on LD_LIBRARY_PATH (or other environment variables) will work properly in spack. However, I have had some limited suceess with 'spack install --dirty <spec>', so you might give that extra option a try.

-kt

On Thursday, April 16, 2015 at 11:18:47 AM UTC-6, Florent Pruvost wrote:

Hi,

I just started to play a little bit with Spack that I may use for the installation of complex scientific software stacks.

I tried to use the %intel in an installation process and I meet a problem with symbols coming from intel libraries that I don't encounter when I build the software outside Spack.

More specifically I want to write a package for the StarPU library (see a simplified version of the Spack package.py in attached) which is installed using autotools building system.
starpu homepage: http://starpu.gforge.inria.fr/

Spack founds compilers on my system (Ubuntu 13.10):

pruvost@hp9470m:~/work/gitwc/spack$ spack compilers
==> Available compilers
-- gcc ----------------------------------------------------------

gcc@4.8

[Massimiliano Culpo]

Hi Pramod,

I have run into a similar issue with Intel 16 on our RHEL 6 clusters. 

For me the fix was to have recent binutils in PATH, as it seems that the linker intel compilers are using is just `ld` and to the best of my knowledge I found no Intel specific environment variable to override this.

Hope it could help,

Massimiliano

SPACK_COMPILER_SPEC=intel@15.0.0

[Elizabeth F]

I've run across a number of packages that don't work with ancient `binutils` (this time on Ubuntu 12.04).  I added a `binutils` dependency to them, but was that maybe the wrong approach?  Is it better to `spack load binutils` before we build other stuff, as part of a bootstrapping procedure?

--

You received this message because you are subscribed to the Google Groups "Spack" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spack+unsubscribe@googlegroups.com.

[Pramod Kumbhar]

Hi Massimiliano and Elizabeth,

I tried following without any success: 

1. Tried pre-loading binutils (which was built with gcc) :

spack load binu...@2.25

which ld

~/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/gcc-5.4.0/binutils-2.25-p7waeodvjpj24usngrg2jdwtnblcrhpe/bin/ld

spack install -v libxml2 %intel

...end up with errors....

2. Adding binutils dependency to libxml2 doesn't help (as binutils build fails due to LD_LIBRARY_PATH)

3. Tried icc.cfg with path from LIBRARY_PATH:

-L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/ipp/lib/intel64 -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64 -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64 -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4 -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/daal/lib/intel64_lin -L/soft/spack/opt/spack/linux-x86_64/binaries/intel-16.0.1/compilers_and_libraries_2016.1.150/linux/daal/../compiler/lib/intel64_lin

When I set ICCCFG, ICPCCFG etc., how to verify they are actually being used? Where is relevant code section to handle these env variables?

If you have any other suggestions to enable use of intel compilers, I would be happy to try.

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.

To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.

[Elizabeth F]

Pramod,

 

1. Tried pre-loading binutils (which was built with gcc) :

spack load binu...@2.25

which ld

~/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhat6-x86_64/gcc-5.4.0/binutils-2.25-p7waeodvjpj24usngrg2jdwtnblcrhpe/bin/ld
spack install -v libxml2 %intel

...end up with errors....

In Spack, `build_environment.py` has the following code:

```

    # Stuff in here sanitizes the build environemnt to eliminate

    # anything the user has set that may interfere.

    if not dirty:

        # Remove these vars from the environment during build because they

        # can affect how some packages find libraries.  We want to make

        # sure that builds never pull in unintended external dependencies.

        # BUT: If things don't work, remove the env.unset() stuff here.

        env.unset('LD_LIBRARY_PATH')

        env.unset('LIBRARY_PATH')

        env.unset('CPATH')

        env.unset('LD_RUN_PATH')

        env.unset('DYLD_LIBRARY_PATH')

```

Unless you set dirty=True, this will unfortunately un-do any `spack load` you did previously.  I don't know how or when `dirty` gets set.  (And please, someone... avoid negative logic, call it `clean` instead.)  You could look through and find out, or ask on this forum --- or just comment out those lines in `build_environment.py` and see if things work for you.

This problem has been kicking around for a while.  And I'm beginning to wonder if the right way to handle compilers that need env vars set is to build wrappers that set the env var, then call the compiler.  For example, my `icc` wrapper would look something like:

```

!#/bin/sh

#

module load intel-compiler-16

icc $*

```

You could always give that a try, see how it goes.  One thing you could do to debug is to create a fake "compiler" that's really just a shell script that prints out the env vars.  Then you will be able to see for 100% sure what env vars Spack is passing to your compiler.

I've so far gotten away with only using GCC.  I'll have to use Intel soon, I've been hoping others come up with good solutions to this before I need it.

-- Elizabeth

[Pramod Kumbhar]

Thanks for detailed info! I think earlier I made mistake by adding newlines in icc.cfg with multiple Xlinker flags. With below single line in .cfg file, I am now able to compile libxml2 with Intel compiler:

-Xlinker -rpath=/gpfs/bbp.cscs.ch/apps/viz/intel/composer_xe_2015.0.090/compiler/lib/intel64

While building glib %intel (as part of qt dependency), I see following error:

configure: error: Package requirements (libffi >= 3.0.0) were not met:

No package 'libffi' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you

installed software in a non-standard prefix.

I see that libffi 3.2 is already installed by spack and is dependency of glib. I tried pre-loading it without any success:

-- linux-redhat6-x86_64 / in...@15.0.0 --------------------------

bz...@1.0.6       lib...@3.2.1
$ spack load lib...@3.2.1 %intel

I don't know how to adjust PKG_CONFIG_PATH.

-Pramod

[Pramod Kumbhar]

I don't know how to adjust PKG_CONFIG_PATH.

I mean, I see that libffi module set PKG_CONFIG_PATH properly and I thought I don't have to do anything. I just tried loading libffi with spack load and then installation of glib is working fine! (but couldn't reproduce this issue when I re-installed package after unloading module).

All looks good now! thanks!