Installing python packages with virtualenv

[rad...@rci.rutgers.edu]

This question is a bit outside the direct scope of BigJob, but I only use
virtualenv for that purpose so I'm posting it here.

I'm trying to load additional python packages for use with my BigJob
script, specifically netCDF4. However, there is apparently a known bug
with using pip in this case:

=== begin snippet ===
> pip install netCDF4
...

HDF5_DIR environment variable not set, checking some standard locations ..

Traceback (most recent call last):

File "<string>", line 16, in <module>

File "/home/radakb/.bigjob/python/build/netCDF4/setup.py", line 114, in
<module>

raise ValueError('did not find HDF5 headers')

ValueError: did not find HDF5 headers
=== end snippet ===

I guess this is a problem with setup.py and compile time dependency on
numpy (I just googled "pip install netCDF4" and found a bunch of messages
boards about it).

I assume that I have to install this in some other way, but I have pretty
much zero experience with compiling libraries manually as I can usually
just use "module load" or what have you. Doesn't virtualenv add an extra
layer of complexity to this as well? What's the best way to go about this?

Brian


============================================ Current Address ============
Brian Radak : Rutgers University
PhD candidate - York Research Group : BioMaPS Institute
University of Minnesota - Twin Cities : CIPR 308
Graduate Program in Chemical Physics : 174 Frelinghuysen Road,
Department of Chemistry : Piscataway, NJ 08854
rada...@umn.edu : rad...@biomaps.rutgers.edu
=======================================================================
Sorry for the multiple e-mail addresses, just use the institute
appropriate address.

[Ashley Z]

Hello!

Quick off-the-cuff answers (did not fix the problem, but made some progress, and this may work on the machine you are using)

Virtualenv should encapsulate your Python session for both pip and setup.py.  That is to say, whether you run pip install or python setup.py, the packages should both install to your virtual environment.

I took a quick look at this for you and it appears that the installer looks for a HDF5_DIR environment variable, which you can see in the output you gave us.

On Stampede:

Make sure you have hdf5 loaded (you can see different available software with "module avail")

> module load hdf5

Then, find out where the hdf5 files are located.  I did an export | grep -i hdf5 for this, but there are probably more elegant ways.

> export | grep -i hdf5

...  (extra output removed)

declare -x TACC_HDF5_DIR="/opt/apps/intel13/hdf5/1.8.9"

and that is where HDF5 is living.

Export this to the environment variable that the netCDF4 installer wants:

> export HDF5_DIR=/opt/apps/intel13/hdf5/1.8.9/

Now when you run pip install netCDF4 , you will get a different error, this time about other headers.

Following the same procedure outlined above (module add netcdf, then look for the directory)

you can then do

export NETCDF4_DIR=/opt/apps/intel13/netcdf/4.1.1

...  at this point though, running pip install netCDF4 throws the following error:

/usr/bin/ld: /opt/apps/intel13/netcdf/4.1.1/lib/libnetcdf.a(nc4attr.o): relocation R_X86_64_32 against `.rodata.str1.4' can not be used when making a shared object; recompile with -fPIC

/opt/apps/intel13/netcdf/4.1.1/lib/libnetcdf.a: could not read symbols: Bad value

collect2: ld returned 1 exit status

/usr/bin/ld: /opt/apps/intel13/netcdf/4.1.1/lib/libnetcdf.a(nc4attr.o): relocation R_X86_64_32 against `.rodata.str1.4' can not be used when making a shared object; recompile with -fPIC

/opt/apps/intel13/netcdf/4.1.1/lib/libnetcdf.a: could not read symbols: Bad value

collect2: ld returned 1 exit status

And, I don't know how to fix this :)

I'm posting this though because this might work fine for you if you're not on Stampede, and hopefully what I outlined can help give you some ideas as to how to get this running on your system.

Also, maybe somebody else has some ideas/might be helped by this.

Hope this helps at least a little!

-Ashley Z

[Andre Merzky]

Hi Brian,

if you still need to compile/install HDF5 somewhere, feel free to
contact me on skype (andre_merzky) and I can walk you through the
process.

Cheers, Andre.

> --
> You received this message because you are subscribed to the Google Groups "bigjob-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bigjob-users...@googlegroups.com.
> To post to this group, send email to bigjob...@googlegroups.com.
> Visit this group at http://groups.google.com/group/bigjob-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Nothing is really difficult.