Error compiling with --enable-shared --enable-xyce-shareable

[Sudsy]

Hi Guys,

I am trying to build Xyce for an environment where I cannot install dependencies. I need to be able to do an XCopy type deploy.

I am working with the serial version only and compiling inside a docker container based on amazonlinux. I have been able to get a standard build working.

The first question is whether the combination of --enable-shared --enable-xyce-shareable should support my scenario or should I be using a different set of options.

With these two options set, I receive errors in the build log:

make[4]: Entering directory `/Xyce-6.9/build/src/DeviceModelPKG/ADMS'
  CXX      N_DEV_RegisterADMSDevices.lo
  CXX      N_DEV_ADMSHBT_X.lo
  CXX      N_DEV_ADMSPSP103VA.lo
  CXX      N_DEV_ADMSPSP102VA.lo
  CXX      N_DEV_ADMSJUNCAP200.lo
  CXX      N_DEV_ADMSbsimcmg.lo
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
make[4]: *** [N_DEV_ADMSPSP103VA.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: Leaving directory `/Xyce-6.9/build/src/DeviceModelPKG/ADMS'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/Xyce-6.9/build/src/DeviceModelPKG'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/Xyce-6.9/build/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/Xyce-6.9/build/src'
make: *** [all-recursive] Error 1
Making install in src

I also tried just the --enable-xyce-shareable but receive errors about compiling with --fPIC 

Another option I tried was modifying the Xyce_STATIC_BINARY option inside defaultConfig.cmake. While this compiled successfully it didn't achieve the desired outcome. It threw an error when I moved it to another box looking for the fftw dependency.

I would really appreciate some pointers to get me moving in the right direction. 

Cheers,

Sudsy

[xyce-users]

The Xyce team does not support building Xyce under CMake at this time --- the CMakeLists.txt and "defaultConfig.cmake" files present are a relic of the obscure way we build for Xyce *only* on Windows.  Use of this build system is NOT maintained for linux or other Unix-like systems, and probably doesn't even work.  Please follow the guidance in the Xyce Building Guide (https://xyce.sandia.gov/documentation/BuildingGuide.html) for the correct way to build Xyce, which involves using the "configure" script, not CMake.  (The Building Guide says:  "NOTE:  Some open-source users have noticed that the Xyce source tree contains a set of "CMakeLists.txt" files and have tried to use CMake instead of autoconf to build Xyce.  These are used by the Xyce team for special builds of Xyce on Windows in a specific environment, and they are not maintained for use on other systems.   The Xyce team does not provide support for CMake; the autoconf build is our only supported process.")

We cannot help with the specifics of your build environment, deployment, or use of docker containers, but your report suggests a possible reason for the failure.  The fact that it is crashing the compiler during the build of the ADMS directory suggests that your compiler is running out of memory trying to optimize the code for analytic sensitivities.  This code is heavily templatized and can consume enormous amounts of RAM while building.  That is the only reason I can think of for your compiler to be crashing (with or without "--enable-shared") in this situation.  Perhaps it's related to how you've configured your docker container, or perhaps when you are running "make" you are using so many jobs (via "-j") that the combined memory demands of all the processes are overwhelming your machine.

The first I'd do to check that that's the problem is to follow the guidance in the "Special note for building on small-memory computer systems" section of the Building Guide (particularly the part about disabling analytic sensitivity computation in the ADMS directory).  You will get a version of Xyce with reduced functionality for sensitivity analysis, but it may be the only way to get past this problem, if the problem is indeed that the compiler is crashing because it's running out of memory.

If you use "--enable-xyce-shareable" you should also use "--enable-shared", so I'm not surprised that you had compilation errors with just the one.  "--enable-shared" causes Xyce to build shared libraries for libxyce as well as static libraries.  "--enable-xyce-shareable" makes Xyce *use* those shared libraries instead of statically linking to its static libraries.

[Sudsy]

Thanks for the tips.

I was able to build with --enable-shared --enable-xyce-shareable --disable-adms_sensitivities

Unfortunately when I xcopy the install folder to the same location on another machine without preinstalled dependencies I receive the following error.

error while loading shared libraries: libfftw3.so.3: cannot open shared object file: No such file or directory.

This is the same error I receive after moving the files when I compile without those options

So it seems that neither option is producing a true static build.

Interestingly, I have been able to install the rpm on this box successfully without any further dependencies. So, the build that generated the rpm package seems like what I need.

I thought there was a chance I might be able to move the rpm install location using rp. -i --prefix=/home and while it does install the files to the correct location I receive the following error when trying to execute /home/Xyce-Release-6.9.0-OPENSOURCE/bin/Xyce --help

/Xyce: error while loading shared libraries: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory

I checked and found that libmkl_intel_lp64.so does exist in /home/Xyce-Release-6.9.0-OPENSOURCE/lib

Do you have any clues about how I might be able to build in the same way as the rpm package but to a different directory like "/home" or some other ideas on how to get a truly static build?

I tried installing the rpm to a different folder using --prefix but unfortunately it failed to find 

[xyce-users]

You have a few choices here.

The first is not to use package-installed shared libraries for FFTW, but instead to build FFTW from source and install only the static libraries, then link Xyce against those.  The result should be a build of Xyce with static-linked FFTW, so when you move to the system that doesn't have it, the binary should work.

A second is simply to copy the libfftw shared libraries from the package install into the lib directory into which you've installed Xyce shared libraries.  The Xyce build you made will have had an RPATH set so it will look in that directory for any shared libraries it needs.  Any shared libraries that it needs, that aren't installed in the system lib directories, but *are* found in the directories listed in RPATH will be available to it.

The Xyce build that's in the RPMs has an RPATH that is set to the *intended* install directory, that is /usr/local/Xyce-<whatever>/lib, so will not find the MKL shared libraries properly if you use options to install elsewhere.  You will have to modify the RPATH in the binary manually using "chrpath" to make it look in the non-default directory you've installed it into.

RPATH is added to the build by libtool, and is generally set to search the directory $prefix/lib, where $prefix is what you passed to "--prefix=".  You could also add "-Wl,rpath=<path>" options to your LDFLAGS to add additional search paths for the binary to use at run time.

[xyce-users]

The configure options "--enable-shared" and "--enable-xyce-shareable" control *only* the creation of Xyce libraries (--enable-shared) and Xyce binaries (--enable-xyce-shareable).  Neither controls in any way at all how Xyce is linked to its third party libraries (TPLs).  The purpose of these options is to allow linking external applications to the Xyce libraries dynamically (--enable-shared) and to allow shared-library plugins to Xyce (--enable-xyce-shareable).  If neither of these things are what you're trying to do, you might as well not enable either option.  They won't help your deployment problem.

The best bets I can think of are the ideas I had in the previous email (either copying FFTW's shared libraries into the lib directory you created when you installed Xyce, or building FFTW from source and installing only the static library on the machine where you're building).  But you certainly have more choices.

One such option would be that you could try forcing your linker to link against the TPLs statically.  One way to do that might be to add "-Bstatic" to your LDFLAGS, which in some cases will cause the linker to choose static libraries instead of dynamic when possible. It does not guarantee that you'll get a truly static binary (many shared libraries such as libstdc++ won't be linked statically even with this option), but it might work well enough for you to prevent dynamic linking to FFTW, and maybe others you are having trouble with.

[xyce-users]

One final thought --- it occurred to me why your shared build puked during ADMS compilation when the static didn't.  When you build with "--enable-shared" every file is compiled twice, once for the static library and once for the shared.  It is very likely that while this was happening with a "make -j<some number>" this double compilation was consuming way more memory than the single did.

Since it sounds like really the problem you're trying to solve is Xyce's dependencies on third party libraries on a platform where you can't install the TPLs in system directories, I would recommend that you not bother enabling --enable-shared and --enable-xyce-shareable unless you need the shared Xyce libraries or shared-library plugin capability (both fairly esoteric use cases), and stick to the default build.  Then use one of the other techniques I suggested earlier (copy TPL shared libraries into Xyce's lib directory, use only static TPLs, adding rpaths, or adding "-Bstatic" to LDFLAGS) to try to get around the TPL issue.  That way you wouldn't have to use "--disable-adms_sensitivities", which makes sensitivity analysis on circuits involving Verilog-A models (VBIC, BSIMCMG, MEXTRAM, etc.) less accurate.

[Ben Sudbury]

Thanks so much for your help, I ended up going for the chrpath option using the rpm install. 

It's working very well for me. I currently have it set to ../lib which I have discovered is relative to where I am executing from. So with that rpath, it's completely portable but I have to make sure I set the working directory correctly.

Thanks again for your help.

--
You received this message because you are subscribed to the Google Groups "xyce-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xyce-users+...@googlegroups.com.
To post to this group, send email to xyce-...@googlegroups.com.
Visit this group at https://groups.google.com/group/xyce-users.
For more options, visit https://groups.google.com/d/optout.

--

Benjamin Sudbury

Director of Commercialisation

PV Lighthouse

+612 8038 6660

[Sudsy]

Just revisited this after almost a year. For various reasons, the rpm method was no linger suitable so I found myself compiling from source. Thought I should publish the solution.

The solution was to use LDFLAGS="-Wl,-rpath,../lib" \

And also copy the following shared libraries to the lib folder:

/usr/lib64/libfftw3.so.3 

/usr/lib64/libamd.so.2 

/usr/lib64/liblapack.so.3 

/usr/lib64/libblas.so.3 

/usr/lib64/libgfortran.so.3 

/usr/lib64/libquadmath.so.0

Thanks for the guidance, saved me twice now.

To unsubscribe from this group and stop receiving emails from it, send an email to xyce-users+unsubscribe@googlegroups.com.

To post to this group, send email to xyce-...@googlegroups.com.
Visit this group at https://groups.google.com/group/xyce-users.
For more options, visit https://groups.google.com/d/optout.

[xyce-users]

Yes, setting a relative RPATH is really the best thing here, because libtool will be putting an absolute rpath into the binary, which means you're not able to move the installed binary around.  "../lib" appears to work for you, so it may be good enough.

The Xyce team some time ago redid the way we build binaries for the team's RPMs so that they use relative RPATHS instead of absolute.  The actual RPATH we use for the builds in those RPMs is "$ORIGIN/../lib" which is a linux-specific RPATH that assures the binary finds the lib directory relative to where the binary is found, even respecting any symlinks or other weirdness that might be present.  Forcing "$ORIGIN" into the rpath is a tricky exercise in shell and Make escapes, though, and can take a little doing.  Our configure script, when it detects the build is one of our Intel compiler builds, uses:

    CXXFLAGS="'-Wl,-rpath,\$\$ORIGIN/../lib' $CXXFLAGS"

to prepend the desired RPATH setting into the existing CXXFLAGS given by the user and/or built up in the process of running configure.  It might be possible for you to get it done through the configure command line similarly, but the exact string you use to pass it to the configure command line might be different.  As I said, the shell and Make escapes necessary to get this passed through correctly are tricky, because both "\" and "$" are special characters in both shell and make --- so you have to escape them enough so that the right thing makes it to the right place.  If you try this, you have to make sure that the resulting binary actually got the RPATH you wanted after all the shells and makefiles have finished mangling things by using "readelf -d /path/to/installed/Xyce | grep RPATH" to examine the binary.

For more information on $ORIGIN and why it might be worth using, see https://enchildfone.wordpress.com/2010/03/23/a-description-of-rpath-origin-ld_library_path-and-portable-linux-binaries/ (he uses a kludgy way of forcing $ORIGIN into the rpath, which I don't recommend --- better to work out the crazy set of shell and Make escapes and do it right).