environment variable

[Luca Cozzuto]

Dear developers,

I'm trying to convert a docker image to a singularity one but I notice that in the final image I got both the environment variable I currently have and the one I have set in the docker image with the ENV tag.

If I use the -e parameter I strip any environment variable, also the one I would like to have it there since are required to some programs.

Can you help me in this, please?

Many thanks!

Luca

[vanessa s]

Hi Luca,

When you convert from Docker, Singularity will automatically grab the ENV variables that you've defined, so you shouldn't need to specify it twice. Which of the two are you wanting to not use? You can easily remove the extra specification from %environment. If you want to disable singularity adding, we don't have any kind of flag for that (but it could be easily added).

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Luca Cozzuto]

Hi Vanessa,

many thanks for your answer. 

When converting the image with "import" I would like to keep every environmental variables not specified in the docker image out of it. 

I'm also having problem with singularity shell that even with the option -e is still keeping the ENV variables. 

Many thanks.

Luca

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[vanessa s]

Hi Luca,

If you don't want "every environmental variables not specified in the docker image out of it." do you mean removing variables from the Docker image, or ones that you have defined in another singularity image? Could you please provide a toy example of your issue, with the current output and the desired? Thanks!

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Luca Cozzuto]

Hi Vanessa,

after several attempts I found that launching singularity exec -e IMAGE.img  env  is removing the unwanted environmental variables but with singularity shell -e IMAGE.img I can still see them.

I'm wondering if there is something I am doing wrong... 

Luca

[Luca Cozzuto]

My toy example is with a docker file like this:

FROM centos:latest

env FOO=hello BAR=world

I build the image and then convert it with docker2singularity. Then I see that with 

singularity exec -e test_latest-2017-07-26-ebe796a2e87e.img env|grep PYT 

I have no problem but when I launch singularity shell -e test_latest-2017-07-26-ebe796a2e87e.img

and type env I still have the environmental variables.

Best,

Luca

[Luca Cozzuto]

I also tried with singularity import from a docker export and I found that the -e option is able to remove the environmental variables this time but then variable set in the docker image are present in the final image (either using the -e option or not).

docker build -t test: .

singularity create --size 400 test.img

docker create test:latest

docker export 0145cc477510ad0a05fda9dacfaac95b2a097679b4d2511048ddfd548105c4a3 | singularity  import test.img

Thanks again,

L

[vanessa s]

Hey Luca,

Ah yes, I know docker2singularity well because I wrote most of the runtime script, that somehow was scraped of any credit via improper use of version control, lol. When you export the docker environment using that script,it writes directly to a file /environment, which mirrors how the very first singularity images were set up. Later images (made with singularity proper) store the docker environment variables in the image's metadata "env" folder, available via shell or exec with a simple source:

for script in /.singularity.d/env/*.sh; do
    if [ -f "$script" ]; thenHere is where
        . "$script"
    fi
done

so you don't have the environment variables because they aren't being sourced. The option with -e would clean the environment from your host, not what is in the container. Here is where some of this happens, if it helps.

I think if you want this to work as expected, you should put your docker image on docker hub, and then bootstrap appropriately:

singularity create image.img
sudo singularity bootstrap image.img Singularity

This will handle the Docker environment to coincide with the current implementation that sources it. Give that a go and see if it helps.

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Luca Cozzuto]

Many thanks!

the only thing is that I would like to avoid to use the dockerhub. I'm wondering if there is another way.

Thanks again.

Luca  

[Bennet Fauber]

I think I agree with Luca. The standard way of doing things maybe
shouldn't have such a heavy reliance on a commercial service? It
seems like that is the prevailing direction. Maybe I'm
misinterpreting recent developments, though.

[Luca Cozzuto]

It would be fantastic to have a way to preserve those ENV variable in the final image using docker export | singularity import without passing by docker hub or being forced to use sudo etc.

Luca 

[David Godlove]

Let me jump in here and say that you don't need to use Docker Hub itself to convert your Docker image to Singularity via bootstrap.  You can also set up registry on your local host and use that in the From: field when you bootstrap your container.  I'm not docker expert, but I've managed to do that a few times by looking around for help online.  It's not prohibitively difficult.  

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[vanessa s]

I advocate for Docker Hub because it is a means of getting images via layers. Even using docker2singularity requires the dependency of Docker installed, but Singularity --> Docker Hub does not, it uses a restful API. If you want to use layers without either of those things, then you can find them on your computer and import. But then you also need to issue commands to obtain the environment and other container metadata.

If you have a suggestion for a workflow that you would prefer other than building on Docker Hub, issuing singularity pull docker://ubuntu, (without needing Docker installed) and then having a working container, please share and we can chat about it.

[Luca Cozzuto]

Hi Vanessa,

I understand your point but I really don't like to send everything I need to be converted to a singularity image to dockerhub. I imagine that everyone who needs to convert a docker image to a singularity one has already installed on the system docker so it would be really nice to have a way to build the docker image in our computer and convert them in loco. I'm not an expert of layers, metadata etc and so I cannot help more on this side but I think that a proper image conversion should pass also the environmental variable present in that image. I don't know whether I am expressing myself correctly but I would like to use the singularity image in "shell" mode in the same way we use docker run -it... 

Many thanks again,

Luca

[vanessa s]

Hi Luca,

Ah I see where you are coming from. The reason for Docker Hub is because we can connect/use it without requiring the user to have Docker installed - imagine how funny it would be to require Docker to use Singularity? If you don't want to use the Docker Registry, your other option is to deploy your own and use your local images --> https://github.com/singularityware/singularity/issues/429#issuecomment-269085751 It basically is a local registry running on port 5000.

Other than using Docker / Docker Hub, how would you suggest going about adding some set of local Docker layers to an image?

Best,

Vanessa

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.