gurobi setup from container

[Tims]

I have the following in %post

        cd /share/apps/gurobi/gurobi810/linux64

        mv license/gurobi_client.lic license/gurobi.lic

        python setup.py install --user

and this in %environment

        export GUROBI_HOME="/share/apps/gurobi/gurobi810/linux64"

        export PATH=$PATH:${GUROBI_HOME}/bin

        export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${GUROBI_HOME}/lib"

        export GRB_LICENSE_FILE="$GUROBI_HOME/license/gurobi.lic"

The following errors out:

 srun -N1 -p slurm -t 00:20:00 singularity exec intelmpi-python2.7.simg bash -c  "unset PYTHONHOME && python -c 'from gurobipy import *'"

bash: line 1: 24736 Bus error               python -c 'from gurobipy import *'

srun: error: backend-0-0: task 0: Exited with exit code 135

A reluctant "sudo" works

srun -N1 -p slurm -t 00:20:00 sudo singularity exec intelmpi-python2.7.simg bash -c  "unset PYTHONHOME && python -c 'from gurobipy import *'"

Any clues? Alternatives?

THanks!

[Justin Cook]

Tims,

What happens when you try to run the container without SLURM?

Can you run with the `-d` option set for debugging and share the output? (`singularity -d exec ...`)

Thanks,

Justin

[v]

Re-forwarding to the list (see thread below) - the issue was solved - it was the .local folder on the path, which can be resolved by manual means (renaming / moving) or via the environment variables for the PYTHONNOUSERSITE.

Thanks for following up Justin!

---------- Forwarded message ---------
From: vanessa sochat <vso...@stanford.edu>
Date: Tue, Jan 29, 2019 at 2:24 PM
Subject: Re: [Singularity] gurobi setup from container
To: Tims Corbett <tims.c...@gmail.com>

Woohoo!

Just one more thing to try - there are environment variables for PYTHONNOUSERSITE and PYTHONUSERBASE, see

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONNOUSERSITE

It looks like if you set them the .local directory (which is the user site-packages) would be ignored. Also feel free to share our whole chain of discussion (or the final result) to the list in case someone in the future hits this problem! I've personally hit it at least 3 times on our cluster, and it always looks different but comes down to loading the wrong library against the container (and is fixed by nuking .local one way or another!)

On Tue, Jan 29, 2019 at 2:18 PM Tims Corbett <tims.c...@gmail.com> wrote:

That was it ! Host python seems to be working well even after renaming ~./local/lib/python2.7.

Thanks Vanessa ! 

On Tue, Jan 29, 2019 at 11:14 AM vanessa sochat <vso...@stanford.edu> wrote:

I've run into this issue before with the .local folder in the $HOME on my host - to fix it I temporarily renamed it to something else (probably not the best solution) or you could try to set a dummy home somewhere else with --home . Basically, you don't want python finding that .local directory.

On Tue, Jan 29, 2019 at 11:10 AM Tims Corbett <tims.c...@gmail.com> wrote:

Thanks Vanessa

Those 2 changes didn't quite help ( removing --user and expanding $GUROBI_HOME

I do need Gurobi on the host machine as well. Is that causing the container to fail? 

Why does this look for python libraries on the host?

 singularity exec --cleanenv intelmpi-python2.7-gurobi.img/  bash -c "unset PYTHONHOME && python -v -c 'from gurobipy import *'"

dlopen("/home/x/.local/lib/python2.7/site-packages/gurobipy/gurobipy.so", 2);

bash: line 1: 27580 Bus error               (core dumped) python -v -c 'from gurobipy import *'

Bootstrap: docker

From: python:2.7.8

%post

        mkdir -p /apps/gurobi

        cd /apps/gurobi

        wget https://packages.gurobi.com/8.1/gurobi8.1.0_linux64.tar.gz

        tar -xvf gurobi8.1.0_linux64.tar.gz

        unset PYTHONHOME

        cd /apps/gurobi/gurobi810/linux64

        #mv license/gurobi_client.lic license/gurobi.lic

        python setup.py install

%environment

        GUROBI_HOME=/apps/gurobi/gurobi810/linux64

        export PATH=$PATH:/apps/gurobi/gurobi810/linux64/bin

        export LD_LIBRARY_PATH=/apps/gurobi/gurobi810/linux64/lib:$LD_LIBRARY_PATH

        export GUROBI_HOME LD_LIBRARY_PATH

        export GRB_LICENSE_FILE=/apps/gurobi/gurobi810/linux64/license/gurobi.lic

On Tue, Jan 29, 2019 at 10:01 AM vanessa sochat <vso...@stanford.edu> wrote:

Hey Tims,

I think your issue is that you are calling gurobi from the container, but grabbing a library on the path in the host (via LD_LIBRARY_PATH). I *don't* have gurobi (or it's libraries) anywhere on my host, and I was able to get the container working with the following recipe:

Bootstrap: docker
From: python:2.7.8
%post
        mkdir -p /apps/gurobi
        cd /apps/gurobi
        wget https://packages.gurobi.com/8.1/gurobi8.1.0_linux64.tar.gz
        tar -xvf gurobi8.1.0_linux64.tar.gz
        unset PYTHONHOME
        cd /apps/gurobi/gurobi810/linux64
        #mv license/gurobi_client.lic license/gurobi.lic
        python setup.py install
%environment
        GUROBI_HOME=/apps/gurobi/gurobi810/linux64
        LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/apps/gurobi/gurobi810/linux64/lib
        export GUROBI_HOME LD_LIBRARY_PATH

Changes include:

 - installing not as a user (we don't need to, building the container we are root)

 - adding the correct LD_LIBRARY_PATH and exporting it

And then your command works as expected :)

Best,

Vanessa

On Tue, Jan 29, 2019 at 12:44 PM Tims Corbett <tims.c...@gmail.com> wrote:

All paths lead to core dump

singularity exec --cleanenv intelmpi-python2.7-gurobi.img/  bash -c "unset PYTHONHOME && python -v -c 'from gurobipy import *'"

import gurobipy # precompiled from /home/x/.local/lib/python2.7/site-packages/gurobipy/__init__.pyc

dlopen("/home/x/.local/lib/python2.7/site-packages/gurobipy/gurobipy.so", 2);

bash: line 1: 16403 Bus error               (core dumped) python -v -c 'from gurobipy import *'

singularity exec --cleanenv intelmpi-python2.7-gurobi.img/ which python

/usr/local/bin/python

singularity exec intelmpi-python2.7-gurobi.img/ which python

/usr/local/bin/python

$ singularity shell --cleanenv intelmpi-python2.7-gurobi.img/

Singularity: Invoking an interactive shell within container...

Singularity intelmpi-python2.7-gurobi.img:~> echo $PYTHONPATH

Singularity intelmpi-python2.7-gurobi.img:~> python

Python 2.7.8 (default, Nov 26 2014, 22:28:51)

[GCC 4.9.1] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>>

>>> from gurobipy import *

Bus error (core dumped)

python -c "import os"

OK

Here's the recipe:

Bootstrap: docker

From: python:2.7.8

%post

        mkdir -p /apps/gurobi

        cd /apps/gurobi

        wget https://packages.gurobi.com/8.1/gurobi8.1.0_linux64.tar.gz

        tar -xvf gurobi8.1.0_linux64.tar.gz

        unset PYTHONHOME

        cd /apps/gurobi/gurobi810/linux64

        #mv license/gurobi_client.lic license/gurobi.lic

        python setup.py install --user

%environment

        export GUROBI_HOME="/apps/gurobi/gurobi810/linux64"

        export PATH=$PATH:${GUROBI_HOME}/bin

        export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${GUROBI_HOME}/lib"

        export GRB_LICENSE_FILE="$GUROBI_HOME/license/gurobi.lic"

On Tue, Jan 29, 2019 at 9:04 AM vanessa sochat <vso...@stanford.edu> wrote:

Hey Tims,

Let's try two things. First, make sure to not include the environment from the host:

singularity exec --cleanenv gurobi.img bash -c "unset PYTHONHOME && python -v -c 'from gurobipy import *'"

Try doing this command with and without cleanenv to see what python you are hitting:

singularity exec --cleanenv gurobi.img which python
singularity exec gurobi.img which python

Then try looking at PYTHONPATH for both those cases:

singularity exec --cleanenv gurobi.img env | grep PYTHONPATH
singularity exec gurobi.img env | grep PYTHONPATH

Finally, try shelling into the container and trying to import gurobi:

singularity shell --cleanenv gurobi.img

echo $PYTHONPATH

python

from gutobipy import *

When you get something working (and please show me the output of the above) we would want to write the full set of commands into a script to execute via the runscript in the container.

On Tue, Jan 29, 2019 at 11:16 AM Tims Corbett <tims.c...@gmail.com> wrote:

Thanks Vanessa. I tried this trick.

singularity exec gurobi.img bash -c "unset PYTHONHOME && python -v -c 'from gurobipy import *'"

TRace shows that container is looking for library on the host machine. What to do?

import gurobipy # precompiled from /home/x/.local/lib/python2.7/site-packages/gurobipy/__init__.pyc

dlopen("/home/x/.local/lib/python2.7/site-packages/gurobipy/gurobipy.so", 2);

bash: line 1: 13345 Bus error               (core dumped) python -v -c 'from gurobipy import *'

On Mon, Jan 28, 2019, 4:51 PM vanessa sochat <vso...@stanford.edu wrote:

Try seeing if you can shell into the container and see the gurobi stuffs as your regular use, maybe you need to change permissions for the entire directory?

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University

--

Vanessa Villamia Sochat
Stanford University

--

Vanessa Villamia Sochat
Stanford University

--

Vanessa Villamia Sochat
Stanford University

--

Vanessa Villamia Sochat
Stanford University

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676