How to make a local image behave as a Docker Hub container?

[Pedro]

How can I get the execution of a local copy of a docker container to behave just like one hosted at Docker Hub?

I'm trying to keep a local copy of a Docker container, so I got one as follows:

singularity pull --name local_image.img docker://ubuntu

However, there are many differences between running this image instead of the Docker Hub container directly, such as system variables not set and many errors on permission.

So, the question is: how can one get a local file in which 

singularity exec local_file.img bash script.sh

behaves exactly as

singularity exec docker://ubuntu:latest bash script.sh

p.s.: I've also tried with singularity run, with no success.

Thanks!

[Dave Dykstra]

Pedro,

I think you need to be more specific and give exact error messages,
singularity version, host operating system, and instructions to
reproduce what you're seeing.

Dave

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Pedro]

Dave,

Thanks for pointing out. Here is some additional information:

Singularity version: 2.4.2

Host machine OS: Scientific Linux release 6.9

As previously stated, 

Method 1

singularity exec local_file.img bash script.sh

Method 2

singularity pull --name local_image.img docker://ubuntu

singularity exec local_file.img bash script.sh

The first difference between the two methods is that system variables are correctly assigned only in the first one.

The variables are assigned according to the following /environment file:

if test -z "$SINGULARITY_INIT"; then

    PATH=$PATH:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/util/opt/matlab/r2017a/bin

    PS1="Singularity.$SINGULARITY_CONTAINER> $PS1"

    SINGULARITY_INIT=1

    export PATH PS1 SINGULARITY_INIT

fi

. /docker_environment

I retrieve different results when displaying the path:

echo $PATH

Method 1 

/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/util/opt/matlab/r2017a/bin:/opt/conda/bin:/opt/conda/envs/test_env/bin

Method 2

/opt/conda/envs/rl/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Even after fixing system variables (by assigning it in run-time), the result of the two methods is distinct since I get permission errors in Method 2 that are absent in Method 1. 

e.g. this error when executing a python script:

File "simulation_interface.py", line 6, in <module>

    from gym.spaces import Box

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/gym/__init__.py", line 11, in <module>

    from gym.envs import make, spec

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/gym/envs/__init__.py", line 1, in <module>

    from gym.envs.registration import registry, register, make, spec

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/gym/envs/registration.py", line 1, in <module>

    import pkg_resources

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3074, in <module>

    @_call_aside

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3058, in _call_aside

    f(*args, **kwargs)

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3087, in _initialize_master_working_set

    working_set = WorkingSet._build_master()

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 655, in _build_master

    ws = cls()

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 648, in __init__

    self.add_entry(entry)

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 704, in add_entry

    for dist in find_distributions(entry, True):

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2059, in find_on_path

    path_item, entry, metadata, precedence=DEVELOP_DIST

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2488, in from_location

    py_version=py_version, platform=platform, **kw

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2828, in _reload_version

    md_version = _version_from_file(self._get_metadata(self.PKG_INFO))

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2453, in _version_from_file

    line = next(iter(version_lines), '')

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2621, in _get_metadata

    for line in self.get_metadata_lines(name):

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1895, in get_metadata_lines

    return yield_lines(self.get_metadata(name))

  File "/opt/conda/envs/rl/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1881, in get_metadata

    with io.open(self.path, encoding='utf-8', errors="replace") as f:

PermissionError: [Errno 13] Permission denied: '/opt/conda/envs/rl/lib/python3.5/site-packages/.wh.six-1.10.0-py3.5.egg-info'

Cheers

[Pedro]

My apologies for the bad formatting.

Also, method 1 is mistyped. The correct command is:

singularity exec docker://ubuntu:latest bash script.sh

I would appreciate any insights!

Cheers

[Dave Dykstra]

Pedro,

When I try running singularity exec docker://ubuntu:latest bash on
SL6.9 and singularity version 2.6.0, I get
FATAL: kernel too old
However if instead of "latest" I use "16.04" that starts the shell.

Since you are directly running the image from docker, how is that
/environment getting created? This is an area of singularity that I
don't know much about. When I run the above, inside the container
/environment is just one comment line.

Dave

[Pedro Lucas F. Albuquerque]

Dave,

I mentioned that image for the sake of the example.  I am actually running "pfrancaalb/rl".

Thanks for your help!

You received this message because you are subscribed to a topic in the Google Groups "singularity" group.
To unsubscribe from this topic, visit https://groups.google.com/a/lbl.gov/d/topic/singularity/qpmGfjzFlSc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to singularity...@lbl.gov.

--

Pedro Albuquerque

+55 (67) 8448-9298

[Dave Dykstra]

Pedro,

I was able to reproduce the $PATH problem with singularity-2.6.0 if I
use docker://pfrancaalb/rl. I think the problem is that /environment is
old style and not supported anymore, although I don't know why it still
works with execing directly from docker (maybe there's a backward
compatibility mode for docker). If I make a copy of /environment on my
host filesystem in /tmp/environment and exec local_file.img with an
option -B /tmp/environment:/.singularity.d/env/90-environment.sh then I
get the expected $PATH. Probably it should be fixed instead in the
original docker container.

As for the permission problem, you haven't said what to run in order to
reproduce that.

Dave

[Pedro]

Interesting. Thanks, Dave!

The permission issue can be observed with a python script importing a module:

source activate rl
python -c "import tensorflow"

Cheers.

[Dave Dykstra]

I don't get any errors if I either use that -B option I told you about
so the environment is set up ahead of time, or if I run a shell
and do "source /environment" before running those two commands.

WIth the latter it does give a weird prompt string ($PS1), but it
doesn't print any errors.

Dave

> > > > an email to singularity...@lbl.gov <javascript:>.

> > > >
> > > > --
> > > > You received this message because you are subscribed to a topic in the
> > > > Google Groups "singularity" group.
> > > > To unsubscribe from this topic, visit
> > > >
> > https://groups.google.com/a/lbl.gov/d/topic/singularity/qpmGfjzFlSc/unsubscribe
> > > > .
> > > > To unsubscribe from this group and all its topics, send an email to

> > > > singularity...@lbl.gov <javascript:>.

> > > >
> > > --
> > > Pedro Albuquerque
> > > +55 (67) 8448-9298
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "singularity" group.
> > > To unsubscribe from this group and stop receiving emails from it, send

> > an email to singularity...@lbl.gov <javascript:>.

[Pedro]

Dave,

I changed accordingly and got $PATH working, but still get the same error with the python script (maybe because of the singularity version?). 

I could get it to work by removing the virtual environment (conda) and installing the requirements via pip.

Thanks a lot for your assistance!