Container image repository

[Gregory M. Kurtzer]

Hi all,

I created a container image repository and I started working on the "rules" for uploading containers to this repository. Please have a look at it and send me feedback:

https://github.com/gmkurtzer/singularity-images

Thank you!

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

[vanessa]

I just started this script this morning and am about to jump on a plane, but wanted to share quickly, because (I think) it will be useful! It's a script to export a docker image via running to singularity:

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh

It would probably be annoying for people to have to generate Singularity equivalent Docker files for images, but just being able to export a Docker image seems like a solid way to start! The only test I've done is to create ubuntu images for ubuntu:14.04 and ubuntu:latest and the containers create and I can connect to them successfully. I'll be making the script take in proper arguments, and further:

- programatically determining the size

- some integration of setting to set up a runscript

- can we programatically get more meta data / etc about the images to also help making the DESCRIPTION and MAINTAINER files?

It's a bit alarming that more information isn't readily available about what is included in an image via docker inspect. Perhaps Singularity can do better by generating some data structure to live with the image that better summarizes this. Likely developers don't have this immediate need to just deploy some Dockerized app, but for researchers it's very important to be able to do things like computationally compare different images. Given that I'm looking for an image to perform some need, I'd want to be able to immediately generate some kind of unsupervised clustering of images based on these things.

Another detail about the singularity-images repo - this is a great idea, but depending on the size of the images (given we have OS here) it might go over the Github file size limit (50MB), and further each user only gets 1GB for Github LFS. If it doesn't work, perhaps Github can be used to submit PRs and discuss new images, but then when they are added, they can be transferred to some other equivalent. In the long term we would want that other equivalent to have functionality akin to Docker Hub, but a lot better, heh :)

I'd like to help out making these images - I'm new to singularity and I'm just figuring out the basics. I'm leaving for a trip today but will be back to working on this soon, bon voyage! :D

Best,

Vanessa

[Gregory M. Kurtzer]

Hi Vanessa!

Funnily enough, right when you sent this email I was in a discussion with Christian Kniep about just this! (rest of the comments inline)

On Mon, Jun 20, 2016 at 11:19 AM, vanessa <vso...@gmail.com> wrote:

I just started this script this morning and am about to jump on a plane, but wanted to share quickly, because (I think) it will be useful! It's a script to export a docker image via running to singularity:

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh

Looks very cool! Thank you!

 

It would probably be annoying for people to have to generate Singularity equivalent Docker files for images, but just being able to export a Docker image seems like a solid way to start! The only test I've done is to create ubuntu images for ubuntu:14.04 and ubuntu:latest and the containers create and I can connect to them successfully. I'll be making the script take in proper arguments, and further:

- programatically determining the size

- some integration of setting to set up a runscript

This came up specifically with my talk with Christian. He thinks that it may be possible to determine the Dockerfile CMD from within the Docker container root. I have not had a chance to investigate this, but it would indeed be very interesting if we could and then import it directly into a Singularity runscript (/singularity). 

 

- can we programatically get more meta data / etc about the images to also help making the DESCRIPTION and MAINTAINER files?

That would also be very helpful for the image repo! 

 

It's a bit alarming that more information isn't readily available about what is included in an image via docker inspect. Perhaps Singularity can do better by generating some data structure to live with the image that better summarizes this. Likely developers don't have this immediate need to just deploy some Dockerized app, but for researchers it's very important to be able to do things like computationally compare different images. Given that I'm looking for an image to perform some need, I'd want to be able to immediately generate some kind of unsupervised clustering of images based on these things.

That is an excellent idea. Can you create a GitHub issue/enhancement for this and include your ideas on what kind of information you want to store, and perhaps how you want to set and edit that info? Chances are it will go into 2.2 (instead of 2.1 which I am just finishing up on).

 

Another detail about the singularity-images repo - this is a great idea, but depending on the size of the images (given we have OS here) it might go over the Github file size limit (50MB), and further each user only gets 1GB for Github LFS. If it doesn't work, perhaps Github can be used to submit PRs and discuss new images, but then when they are added, they can be transferred to some other equivalent. In the long term we would want that other equivalent to have functionality akin to Docker Hub, but a lot better, heh :)

Oh, thank you for telling me about the GitHub limitations! I am open to suggestions on where and how to host the images.

 

I'd like to help out making these images - I'm new to singularity and I'm just figuring out the basics. I'm leaving for a trip today but will be back to working on this soon, bon voyage! :D

Thank you and have a safe trip!

Greg

 

Best,

Vanessa

On Tuesday, June 14, 2016 at 11:43:19 AM UTC-7, Gregory M. Kurtzer wrote:

Hi all,

I created a container image repository and I started working on the "rules" for uploading containers to this repository. Please have a look at it and send me feedback:

https://github.com/gmkurtzer/singularity-images

Thank you!

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Dave Love]

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> Hi Vanessa!
>
> Funnily enough, right when you sent this email I was in a discussion with
> Christian Kniep about just this! (rest of the comments inline)

Also on my list to ask for expertise around tooling after playing around
a bit, inspired by the FAQ entry. More thanks for the script.

Would this basically make Shifter obsolete? I could imagine doing it on
the fly in a resource manager prolog.

> This came up specifically with my talk with Christian. He thinks that it
> may be possible to determine the Dockerfile CMD from within the Docker
> container root.

Doesn't docker inspect provide it, amongst other things? I assumed that
was the Right Way.

> I have not had a chance to investigate this, but it would
> indeed be very interesting if we could and then import it directly into a
> Singularity runscript (/singularity).

> Oh, thank you for telling me about the GitHub limitations! I am open to
> suggestions on where and how to host the images.

I don't know about its limitations, but perhaps the new, reformed
Sourceforge, if national labs can't do it? (I spent a long time in a
national lab :-/.)

[Gregory M. Kurtzer]

On Mon, Jun 20, 2016 at 1:49 PM, Dave Love <d.l...@liverpool.ac.uk> wrote:

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> Hi Vanessa!
>
> Funnily enough, right when you sent this email I was in a discussion with
> Christian Kniep about just this! (rest of the comments inline)

Also on my list to ask for expertise around tooling after playing around
a bit, inspired by the FAQ entry.  More thanks for the script.

Would this basically make Shifter obsolete?  I could imagine doing it on
the fly in a resource manager prolog.

Only at the high level and superficially. Singularity is more of a general purpose mobility of compute solution that is very capable at HPC, Shifter's primary focus is targeting extreme scale HPC and integration with Cray and the resource manager. (I just added that to the FAQ)

> This came up specifically with my talk with Christian. He thinks that it
> may be possible to determine the Dockerfile CMD from within the Docker
> container root.

Doesn't docker inspect provide it, amongst other things?  I assumed that
was the Right Way.

> I have not had a chance to investigate this, but it would
> indeed be very interesting if we could and then import it directly into a
> Singularity runscript (/singularity).

> Oh, thank you for telling me about the GitHub limitations! I am open to
> suggestions on where and how to host the images.

I don't know about its limitations, but perhaps the new, reformed
Sourceforge, if national labs can't do it?  (I spent a long time in a
national lab :-/.)

In theory, I can host some resources to support it, but I don't have time to build and maintain a platform that would do this sort of hosting. I am open to ideas though!

[Dave Love]

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

>> I don't know about its limitations, but perhaps the new, reformed
>> Sourceforge, if national labs can't do it? (I spent a long time in a
>> national lab :-/.)
>>
>>
> In theory, I can host some resources to support it,

[I was implying it might not be easy...]

> but I don't have time
> to build and maintain a platform that would do this sort of hosting. I am
> open to ideas though!

What sort of a platform do you envisage?

Perhaps also, what sort of images might go there -- fairly plain
customizable bases for various distributions, random applications for
people to pull who don't have root, or what? (I ask because it's not
difficult to build them, and there isn't a mechanism for sharing on a
multi-access system.)

[Rémy Dernat]

This repository is a great idea and docker2singularity from Vanessa is very interesting.

Now, just a word about the 50MB github limitations. I think you would need your own hosted project using something like git-fat ( https://github.com/jedbrown/git-fat ) (inside gitlab forge ?) or even a more basic http/ftp storage (perhaps using some big files download process torrent/aspera (...) :  http://www.admin-magazine.com/HPC/Articles/Moving-Your-Data-It-s-Not-Always-Pleasant ). Now, you just have to find the storage server :)

Best regards,

Rémy

--

[Greg Keller]

I am hoping my compute nodes can grab the appropriate images on demand from a prioritized list of mirrors/hubs/neighbors one day.  So I can assume when a job launches, if the container isn't there it can try go get it from a fast/efficient source.  This way if there is a unique name + trusted md5 sum on a container in the mirror it will just make it pull it local as/when/if needed.

[Gregory M. Kurtzer]

On Tue, Jun 21, 2016 at 2:47 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

>> I don't know about its limitations, but perhaps the new, reformed
>> Sourceforge, if national labs can't do it?  (I spent a long time in a
>> national lab :-/.)
>>
>>
> In theory, I can host some resources to support it,

[I was implying it might not be easy...]

Ahh, yes. indeed! lol

 

> but I don't have time
> to build and maintain a platform that would do this sort of hosting. I am
> open to ideas though!

What sort of a platform do you envisage?

Well, in my opinion GitHub would have been fantastic in my mind. On the simple side, I was planning to obtain several base containers from some contributors and make them available for download, but there is no reason not to expand that and allow others to also post their work to be shared to their collegues, or even reference in scientific articles as reproducible workflows. It very quickly falls victim to feature creep! haha

 

Perhaps also, what sort of images might go there -- fairly plain
customizable bases for various distributions, random applications for
people to pull who don't have root, or what?  (I ask because it's not
difficult to build them, and there isn't a mechanism for sharing on a
multi-access system.)

It isn't difficult to build a base distribution and the use case you describe for simply getting a shell in a container, but some application stacks and workflows are quite difficult to integrate and replicate. That is where I think Singularity really shines.

[vanessa s]

See below - I originally sent from my Stanford email and it was rejected from the list :)

---------- Forwarded message ----------
From: vanessa sochat <vso...@stanford.edu>
Date: Wed, Jun 22, 2016 at 2:26 PM
Subject: Re: [Singularity] Re: Container image repository
To: singu...@lbl.gov

Hi Everyone!

Apologies for the late response - I am still traveling, but wanted to take a few minutes to join the discussion.

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh

Looks very cool! Thank you!

 

And @Remy submit a PR to get the size programatically as well, stop the container, and a check to ensure the user has sudo (given the need for docker).

 

It would probably be annoying for people to have to generate Singularity equivalent Docker files for images, but just being able to export a Docker image seems like a solid way to start! The only test I've done is to create ubuntu images for ubuntu:14.04 and ubuntu:latest and the containers create and I can connect to them successfully. I'll be making the script take in proper arguments, and further:

- programatically determining the size

- some integration of setting to set up a runscript

This came up specifically with my talk with Christian. He thinks that it may be possible to determine the Dockerfile CMD from within the Docker container root. I have not had a chance to investigate this, but it would indeed be very interesting if we could and then import it directly into a Singularity runscript (/singularity). 

+1. It looks like this is a pretty reasonable thing to do! https://github.com/CenturyLinkLabs/dockerfile-from-image 

 

- can we programatically get more meta data / etc about the images to also help making the DESCRIPTION and MAINTAINER files?

That would also be very helpful for the image repo! 

 

It's a bit alarming that more information isn't readily available about what is included in an image via docker inspect. Perhaps Singularity can do better by generating some data structure to live with the image that better summarizes this. Likely developers don't have this immediate need to just deploy some Dockerized app, but for researchers it's very important to be able to do things like computationally compare different images. Given that I'm looking for an image to perform some need, I'd want to be able to immediately generate some kind of unsupervised clustering of images based on these things.

That is an excellent idea. Can you create a GitHub issue/enhancement for this and include your ideas on what kind of information you want to store, and perhaps how you want to set and edit that info? Chances are it will go into 2.2 (instead of 2.1 which I am just finishing up on).

 

Will do - likely when I get back from my trip next week. I'll also think a bit about good places to put images. +1 what Greg mentioned about ease of grabbing an image.

--

Vanessa Villamia Sochat
Stanford University

(603) 321-0676

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Rémy Dernat]

Hi,

About the size calculation, it is quite weird. I do not know if docker compress its files inside its containers, but when you do an export and import it into singularity, the size is bigger. I added 50% to the docker size, just to be sure that it will be Ok for singularity (I think that added only 30% to the docker size would be enough). If it is too small, singularity complains that there is not enough space.

BTW, I think this thread should be splitted, because there are two different topic in it.

Cheers,
Rémy

[Dave Love]

[I thought I sent this earlier -- apologies if it's duplicated.]

vanessa s <vso...@gmail.com> writes:

> And @Remy submit a PR to get the size programatically as well, stop the
> container, and a check

> <https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh>

> to ensure the user has sudo (given the need for docker).

That looks wrong generally. It would fail for me -- I'm not in a "sudo"
or "root" group, but I can use sudo and docker (via "wheel" and
"docker"). I only needed sudo for the singularity part of the
conversion, not the docker one, when I tried it by hand. I think you
have to try executing commands and report if they don't work either bare
or with sudo.

Is it documented somewhere what "Size" actually means? I got the
impression you'd need to run the thing to get the actual size. I don't
know if you can rely on being able to run df /, but doing that in the
minimal "fedora" image provided by the EPEL6 docker packaging, I see
265945088 B used v. a Size from inspect of 214315878.

All else failing, obviously you'd at most double the time to do the
conversion if you just found the size of the export stream, which might
be faster than writing to a file.

>> This came up specifically with my talk with Christian. He thinks that it
>> may be possible to determine the Dockerfile CMD from within the Docker
>> container root. I have not had a chance to investigate this, but it would
>> indeed be very interesting if we could and then import it directly into a
>> Singularity runscript (/singularity).
>>
>
> +1. It looks like this is a pretty reasonable thing to do!
> https://github.com/CenturyLinkLabs/dockerfile-from-image

This might be a way to write a /singularity script for a converted
image, but I'm no docker expert and definitely don't understand the
format stuff:

Cmd=$(docker inspect --format='{{json .Config.Cmd}}' $image)
if [[ $Cmd != none ]]; then
echo '#!/bin/sh'
(IFS='[],'; echo $Cmd)
fi > singularity

However, there's ENTRYPOINT as well as CMD. You'd have to decide which
to use if they're both present.

[Something I notice experimenting is the lack of fabled fast startup
with docker with the installation I have. It seems similar to vagrant
with virtualbox, which I normally use, but which isn't nearly trendy
enough.]

[Gregory M. Kurtzer]

On Thursday, June 23, 2016, Dave Love <d.l...@liverpool.ac.uk> wrote:

[I thought I sent this earlier -- apologies if it's duplicated.]

vanessa s <vso...@gmail.com> writes:

> And @Remy submit a PR to get the size programatically as well, stop the
> container, and a check
> <https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh>
> to ensure the user has sudo (given the need for docker).

That looks wrong generally.  It would fail for me -- I'm not in a "sudo"
or "root" group, but I can use sudo and docker (via "wheel" and
"docker").  I only needed sudo for the singularity part of the
conversion, not the docker one, when I tried it by hand.  I think you
have to try executing commands and report if they don't work either bare
or with sudo.

That is how I'd recommend doing it as well. Perhaps do a:

if sudo true; then

...

if

As far as prefixing the docker commands with sudo, it seems reasonable and it should always work. 

 

Is it documented somewhere what "Size" actually means?  I got the
impression you'd need to run the thing to get the actual size.  I don't
know if you can rely on being able to run df /, but doing that in the
minimal "fedora" image provided by the EPEL6 docker packaging, I see
265945088 B used v. a Size from inspect of 214315878.

All else failing, obviously you'd at most double the time to do the
conversion if you just found the size of the export stream, which might
be faster than writing to a file.

Temporary files are generally bad especially when they are large, but you could in theory write the tar to disk, measure the size, and then create the singularity container and import. 

Hrmm, scratch that.... Horrible idea. Lol

 

>> This came up specifically with my talk with Christian. He thinks that it
>> may be possible to determine the Dockerfile CMD from within the Docker
>> container root. I have not had a chance to investigate this, but it would
>> indeed be very interesting if we could and then import it directly into a
>> Singularity runscript (/singularity).
>>
>
> +1. It looks like this is a pretty reasonable thing to do!
> https://github.com/CenturyLinkLabs/dockerfile-from-image

This might be a way to write a /singularity script for a converted
image, but I'm no docker expert and definitely don't understand the
format stuff:

Cmd=$(docker inspect --format='{{json .Config.Cmd}}' $image)
if [[ $Cmd != none ]]; then
  echo '#!/bin/sh'
  (IFS='[],'; echo $Cmd)
fi > singularity

I like this idea for the /singularity run script. 

 

However, there's ENTRYPOINT as well as CMD.  You'd have to decide which
to use if they're both present.

I am not totally familiar with the differences between entry point and cmd. Does entry point describe the shell to use within the container?

 

[Something I notice experimenting is the lack of fabled fast startup
with docker with the installation I have.  It seems similar to vagrant
with virtualbox, which I normally use, but which isn't nearly trendy
enough.]

 What kind of startup times are you seeing?

[Rémy Dernat]

Hi,

2016-06-24 6:39 GMT+02:00 Gregory M. Kurtzer <gmku...@lbl.gov>:

On Thursday, June 23, 2016, Dave Love <d.l...@liverpool.ac.uk> wrote:

[I thought I sent this earlier -- apologies if it's duplicated.]

vanessa s <vso...@gmail.com> writes:

> And @Remy submit a PR to get the size programatically as well, stop the
> container, and a check
> <https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh>
> to ensure the user has sudo (given the need for docker).

That looks wrong generally.  It would fail for me -- I'm not in a "sudo"
or "root" group, but I can use sudo and docker (via "wheel" and
"docker").  I only needed sudo for the singularity part of the
conversion, not the docker one, when I tried it by hand.  I think you
have to try executing commands and report if they don't work either bare
or with sudo.

That is how I'd recommend doing it as well. Perhaps do a:

if sudo true; then

...

if

As far as prefixing the docker commands with sudo, it seems reasonable and it should always work. 

I thought about this when I wrote this part (the 'docker' group). The fact that there is a docker group on your system does not prevent you from the need of sudo rights to create the singularity container. That is why I thought, "Ok, this is a script to convert docker to singularity, and you need at least sudo for the import stuff in singularity, why bother with docker special permission ?".

BTW, the 'docker'/'dockerroot' group is a special group with sudo rights on docker command.

If all you want to do is to export your docker container, why trying to convert it ? Ask your system administrator ;)

Perhaps we can also check if the user is a member of docker* and just warn him that he won't be able to do the full conversion.

 

 

Is it documented somewhere what "Size" actually means?  I got the
impression you'd need to run the thing to get the actual size.  I don't
know if you can rely on being able to run df /, but doing that in the
minimal "fedora" image provided by the EPEL6 docker packaging, I see
265945088 B used v. a Size from inspect of 214315878.

Hum, interesting. :)

As I said, it seems that there is already some kind of compression in the docker storage.

 

All else failing, obviously you'd at most double the time to do the
conversion if you just found the size of the export stream, which might
be faster than writing to a file.

Temporary files are generally bad especially when they are large, but you could in theory write the tar to disk, measure the size, and then create the singularity container and import. 

Hrmm, scratch that.... Horrible idea. Lol

+1

 

 

>> This came up specifically with my talk with Christian. He thinks that it
>> may be possible to determine the Dockerfile CMD from within the Docker
>> container root. I have not had a chance to investigate this, but it would
>> indeed be very interesting if we could and then import it directly into a
>> Singularity runscript (/singularity).
>>
>
> +1. It looks like this is a pretty reasonable thing to do!
> https://github.com/CenturyLinkLabs/dockerfile-from-image

This might be a way to write a /singularity script for a converted
image, but I'm no docker expert and definitely don't understand the
format stuff:

Cmd=$(docker inspect --format='{{json .Config.Cmd}}' $image)
if [[ $Cmd != none ]]; then
  echo '#!/bin/sh'
  (IFS='[],'; echo $Cmd)
fi > singularity

I like this idea for the /singularity run script. 

Yep, nice idea.

Best regards.

Rémy

 

 

However, there's ENTRYPOINT as well as CMD.  You'd have to decide which
to use if they're both present.

I am not totally familiar with the differences between entry point and cmd. Does entry point describe the shell to use within the container?

 

[Something I notice experimenting is the lack of fabled fast startup
with docker with the installation I have.  It seems similar to vagrant
with virtualbox, which I normally use, but which isn't nearly trendy
enough.]

 What kind of startup times are you seeing?

 

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

[Dave Love]

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> As far as prefixing the docker commands with sudo, it seems reasonable and
> it should always work.

Yes, I realized that was irrelevant, sorry.

>> However, there's ENTRYPOINT as well as CMD. You'd have to decide which
>> to use if they're both present.
>
>
> I am not totally familiar with the differences between entry point and cmd.
> Does entry point describe the shell to use within the container?

There's doc at
https://docs.docker.com/engine/reference/builder/#entrypoint

On looking again, I think you need to consider them both in constructing
/singularity but it needs a more careful reading.

>>
>> [Something I notice experimenting is the lack of fabled fast startup
>> with docker with the installation I have. It seems similar to vagrant
>> with virtualbox, which I normally use, but which isn't nearly trendy
>> enough.]
>>
>>
> What kind of startup times are you seeing?

30s or more.

[Gregory M. Kurtzer]

On Friday, June 24, 2016, Dave Love <d.l...@liverpool.ac.uk> wrote:

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

>> However, there's ENTRYPOINT as well as CMD.  You'd have to decide which
>> to use if they're both present.
>
>
> I am not totally familiar with the differences between entry point and cmd.
> Does entry point describe the shell to use within the container?

There's doc at
https://docs.docker.com/engine/reference/builder/#entrypoint

On looking again, I think you need to consider them both in constructing
/singularity but it needs a more careful reading.

Indeed, but there is also some confusing overlap with the Docker Vs Singularity nomenclature. 

In those examples it seems that Docker run uses ENTRYPOINT as the process is called like a shell. So in Singularity terms /bin/sh is the default ENTRYPOINT for the 'shell' command (in fact more pedantically it is the true entry point for all commands as it is used by system() and execv*()). But it seems as though an equivalent for Singularity would be to use /entrypoint when calling 'shell'. 

Singularity 'run' on the other hand (not being functionally equivalent to 'docker run') seems to be most closely related to the dockerfile CMD. 

If I am misreading can someone correct me please. 

>>
>> [Something I notice experimenting is the lack of fabled fast startup
>> with docker with the installation I have.  It seems similar to vagrant
>> with virtualbox, which I normally use, but which isn't nearly trendy
>> enough.]
>>
>>
>  What kind of startup times are you seeing?

30s or more.

Is that the time it takes to build and cache the image from the registry perhaps? Certainly it can't take that long to create and tear down the namespaces! ... I hope ...

[vanessa s]

I'm testing adding the CMD integration, and it seems to work fine, however it led me to discover that running a simple bash shell from the container does not:

This corresponds to lpadmin, sambashare, and docker (on my local machine) and these groups aren't present in the container:

I verified that /bin/bash works fine in the Docker container, and the error exists regardless of the run script (meaning I can create the image, do singularity run image and be taken to a shell, and tun /bin/bash to produce the same error). Any ideas?

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[vanessa s]

Here is a simple command to export the docker inspect json and import into the container as singularity.json (at the container base /)

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L94

$SUDOCMD docker inspect $container_id >> singularity.json
sudo singularity copy $new_container_name singularity.json /
rm singularity.json

 

I didn't change any of the fields / formatting of the json itself (I'd be happy to come up with something) but generally wanted to bring up the idea of having some singularity.json located at "/" in each container that would carry this same meta info as docker inspect. We could perhaps come up with a suggested standard, but allow the user to make any singularity.json that would be desired to expose container meta, options, variables, etc. You could add a command line util to give the user programmatic access, eg:

singularity info ubuntu:latest-2016-04-06.img

or even if we want to be consistent with docker:

singularity inspect ubuntu:latest-2016-04-06.img 

For the singularity.json - my suggestion would be a mixture of meta data about the container (akin to the current docker inspect) but then add fields that will capture programatically what the container can be used for (in terms of inputs and outputs) and variables the user can specify. If we want to do this right we probably should use some standard like the common workflow language, so these things can eventually be controlled by something to fit into an actual workflow. This way, in the long run a user could submit their images to our container hub, and they could be easily parsed for meta and allowable inputs and outputs, and then plug right into a nice workflow web interface. :)

[Gregory M. Kurtzer]

Are those groups you have on the host system? If so....

I suppose it would be possible to add groups that you have on the host and append that group information to the containers dynamic groupfile as a workaround. 

Perhaps you are not seeing those issues with Docker because Singularity is designed to preserve the users current working environment.... Thus the group's are maintained. 

Good work on the integration!

[Gregory M. Kurtzer]

On Saturday, June 25, 2016, vanessa s <vso...@gmail.com> wrote:

Here is a simple command to export the docker inspect json and import into the container as singularity.json (at the container base /)

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L94

$SUDOCMD docker inspect $container_id >> singularity.json
sudo singularity copy $new_container_name singularity.json /
rm singularity.json

I am curious if JSON is the best format for Singularity or the best location. 

An interesting idea would be to add this type of data to the container header. It doesn't have to be binary and it has the added benefit of not having to mount the container in order to get to the metadata. It will make manipulation of the data a bit trickier unless we have a defined data structure standard (which would include field lengths). 

 

 

I didn't change any of the fields / formatting of the json itself (I'd be happy to come up with something) but generally wanted to bring up the idea of having some singularity.json located at "/" in each container that would carry this same meta info as docker inspect. We could perhaps come up with a suggested standard, but allow the user to make any singularity.json that would be desired to expose container meta, options, variables, etc. You could add a command line util to give the user programmatic access, eg:

singularity info ubuntu:latest-2016-04-06.img

or even if we want to be consistent with docker:

singularity inspect ubuntu:latest-2016-04-06.img 

For the singularity.json - my suggestion would be a mixture of meta data about the container (akin to the current docker inspect) but then add fields that will capture programatically what the container can be used for (in terms of inputs and outputs) and variables the user can specify. If we want to do this right we probably should use some standard like the common workflow language, so these things can eventually be controlled by something to fit into an actual workflow. This way, in the long run a user could submit their images to our container hub, and they could be easily parsed for meta and allowable inputs and outputs, and then plug right into a nice workflow web interface. :)

What kinds of fields would be appropriate for this metadata? I've got a couple of ideas but it would be interesting to have a brainstorm of the types of data people would like to be able to obtain about a container. 

Great ideas, thanks!

[vanessa s]

The workaround worked!

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L97

The only issue would be if there is some kind of confidential stuffs there. See below for more comments:

On Sat, Jun 25, 2016 at 3:20 PM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

On Saturday, June 25, 2016, vanessa s <vso...@gmail.com> wrote:

Here is a simple command to export the docker inspect json and import into the container as singularity.json (at the container base /)

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L94

$SUDOCMD docker inspect $container_id >> singularity.json
sudo singularity copy $new_container_name singularity.json /
rm singularity.json

I am curious if JSON is the best format for Singularity or the best location. 

I don't have strong opinions about the location, but if we are to develop tools in python and for the web, then JSON is optimal. If you suggest something like yaml or RDF I'm going to run for the hills.

 

An interesting idea would be to add this type of data to the container header. It doesn't have to be binary and it has the added benefit of not having to mount the container in order to get to the metadata. It will make manipulation of the data a bit trickier unless we have a defined data structure standard (which would include field lengths). 

Oh, cool, I didn't know the containers had headers! How do I inspect / mess around with them? 

 

 

I didn't change any of the fields / formatting of the json itself (I'd be happy to come up with something) but generally wanted to bring up the idea of having some singularity.json located at "/" in each container that would carry this same meta info as docker inspect. We could perhaps come up with a suggested standard, but allow the user to make any singularity.json that would be desired to expose container meta, options, variables, etc. You could add a command line util to give the user programmatic access, eg:

singularity info ubuntu:latest-2016-04-06.img

or even if we want to be consistent with docker:

singularity inspect ubuntu:latest-2016-04-06.img 

For the singularity.json - my suggestion would be a mixture of meta data about the container (akin to the current docker inspect) but then add fields that will capture programatically what the container can be used for (in terms of inputs and outputs) and variables the user can specify. If we want to do this right we probably should use some standard like the common workflow language, so these things can eventually be controlled by something to fit into an actual workflow. This way, in the long run a user could submit their images to our container hub, and they could be easily parsed for meta and allowable inputs and outputs, and then plug right into a nice workflow web interface. :)

What kinds of fields would be appropriate for this metadata? I've got a couple of ideas but it would be interesting to have a brainstorm of the types of data people would like to be able to obtain about a container. 

If I wanted to use this in a workflow, I would need a list of inputs, outputs, along with acceptable values (file extensions, etc). It seems like in the case of a workflow there are two options - either to capture just inputs and outputs as file types, or as other containers that are acceptable. The first is completely open to connecting any two images given the input --matches--> output, the second is extremely limited but much less likely to lead to error in the workflow generation. Of those inputs and outputs, I would want to be able to specify variables for the container analysis (or purpose) like ports, certificates, and for meta data about the container I would likely want an author (someone to contact with questions or issues). If these are stored on a container hub then there would be a board for the container's issues. 

The entire spec for CWL is here, but my thinking is that we should take an extremely minimalist approach - basically having the minimal things listed above, and only expanding on that as we develop workflows / use cases and find that there is need. For so many of these standards big teams of ontologists come up with meta data things that are extremely detailed (and mostly useless) that do nothing but make the standard annoying and hard to use.

 

Great ideas, thanks!

:o) 

[Gregory M. Kurtzer]

On Sat, Jun 25, 2016 at 3:36 PM, vanessa s <vso...@gmail.com> wrote:

The workaround worked!

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L97

The only issue would be if there is some kind of confidential stuffs there. See below for more comments:

Fantastic, but that will only work if it is on the same system and imported as the same user. I think I can make it more portable...

At present, the Singularity process flow for the group file is:

1. mounts the container

2. copies the /etc/group file to the sessiondirectory

3. appends user info from getgrent() to the group file

4. binds the updated group file to the container's /etc/group

So I will look into adding a step 3.5 to also include all supplementary groups.

 

On Sat, Jun 25, 2016 at 3:20 PM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

On Saturday, June 25, 2016, vanessa s <vso...@gmail.com> wrote:

Here is a simple command to export the docker inspect json and import into the container as singularity.json (at the container base /)

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L94

$SUDOCMD docker inspect $container_id >> singularity.json
sudo singularity copy $new_container_name singularity.json /
rm singularity.json

I am curious if JSON is the best format for Singularity or the best location. 

I don't have strong opinions about the location, but if we are to develop tools in python and for the web, then JSON is optimal. If you suggest something like yaml or RDF I'm going to run for the hills.

Haha, I guess I am thinking more low level... But,... in theory I suppose we can print it in JSON (or YAML LOL) so it can be easily parseable by the calling process.

 

 

An interesting idea would be to add this type of data to the container header. It doesn't have to be binary and it has the added benefit of not having to mount the container in order to get to the metadata. It will make manipulation of the data a bit trickier unless we have a defined data structure standard (which would include field lengths). 

Oh, cool, I didn't know the containers had headers! How do I inspect / mess around with them? 

Well, at the moment, there isn't much in the header aside from the interpreter line which is what provides the magic for being able to execute a container file. There isn't much to see, but you can do:

$ head -n 1 container.img

But, much of the code is already in place that if we want to prepend more information there we can.

 

 

 

I didn't change any of the fields / formatting of the json itself (I'd be happy to come up with something) but generally wanted to bring up the idea of having some singularity.json located at "/" in each container that would carry this same meta info as docker inspect. We could perhaps come up with a suggested standard, but allow the user to make any singularity.json that would be desired to expose container meta, options, variables, etc. You could add a command line util to give the user programmatic access, eg:

singularity info ubuntu:latest-2016-04-06.img

or even if we want to be consistent with docker:

singularity inspect ubuntu:latest-2016-04-06.img 

For the singularity.json - my suggestion would be a mixture of meta data about the container (akin to the current docker inspect) but then add fields that will capture programatically what the container can be used for (in terms of inputs and outputs) and variables the user can specify. If we want to do this right we probably should use some standard like the common workflow language, so these things can eventually be controlled by something to fit into an actual workflow. This way, in the long run a user could submit their images to our container hub, and they could be easily parsed for meta and allowable inputs and outputs, and then plug right into a nice workflow web interface. :)

What kinds of fields would be appropriate for this metadata? I've got a couple of ideas but it would be interesting to have a brainstorm of the types of data people would like to be able to obtain about a container. 

If I wanted to use this in a workflow, I would need a list of inputs, outputs, along with acceptable values (file extensions, etc). It seems like in the case of a workflow there are two options - either to capture just inputs and outputs as file types, or as other containers that are acceptable. The first is completely open to connecting any two images given the input --matches--> output, the second is extremely limited but much less likely to lead to error in the workflow generation. Of those inputs and outputs, I would want to be able to specify variables for the container analysis (or purpose) like ports, certificates, and for meta data about the container I would likely want an author (someone to contact with questions or issues). If these are stored on a container hub then there would be a board for the container's issues. 

So would these items be specific to the runscript inside the container at /singularity or something more general about the contents inside the container and other metadata like build date, maintainer, contact, etc?

 

The entire spec for CWL is here, but my thinking is that we should take an extremely minimalist approach - basically having the minimal things listed above, and only expanding on that as we develop workflows / use cases and find that there is need. For so many of these standards big teams of ontologists come up with meta data things that are extremely detailed (and mostly useless) that do nothing but make the standard annoying and hard to use.

LOL, I agree... Keep it simple. :)

[vanessa s]

On Sat, Jun 25, 2016 at 3:49 PM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

On Sat, Jun 25, 2016 at 3:36 PM, vanessa s <vso...@gmail.com> wrote:

The workaround worked!

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L97

The only issue would be if there is some kind of confidential stuffs there. See below for more comments:

Fantastic, but that will only work if it is on the same system and imported as the same user. I think I can make it more portable...

At present, the Singularity process flow for the group file is:

1. mounts the container

2. copies the /etc/group file to the sessiondirectory

3. appends user info from getgrent() to the group file

4. binds the updated group file to the container's /etc/group

So I will look into adding a step 3.5 to also include all supplementary groups.

cool! I like this teamwork thing :O)

 

 

On Sat, Jun 25, 2016 at 3:20 PM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

On Saturday, June 25, 2016, vanessa s <vso...@gmail.com> wrote:

Here is a simple command to export the docker inspect json and import into the container as singularity.json (at the container base /)

https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L94

$SUDOCMD docker inspect $container_id >> singularity.json
sudo singularity copy $new_container_name singularity.json /
rm singularity.json

I am curious if JSON is the best format for Singularity or the best location. 

I don't have strong opinions about the location, but if we are to develop tools in python and for the web, then JSON is optimal. If you suggest something like yaml or RDF I'm going to run for the hills.

Haha, I guess I am thinking more low level... But,... in theory I suppose we can print it in JSON (or YAML LOL) so it can be easily parseable by the calling process.

 

Ah, understood.

 

 

An interesting idea would be to add this type of data to the container header. It doesn't have to be binary and it has the added benefit of not having to mount the container in order to get to the metadata. It will make manipulation of the data a bit trickier unless we have a defined data structure standard (which would include field lengths). 

Oh, cool, I didn't know the containers had headers! How do I inspect / mess around with them? 

Well, at the moment, there isn't much in the header aside from the interpreter line which is what provides the magic for being able to execute a container file. There isn't much to see, but you can do:

$ head -n 1 container.img

But, much of the code is already in place that if we want to prepend more information there we can.

 

ah yes, cool!

I changed the -n 1 to see the rest of the image, and saw some nice compiled goppelty gook.

 

 

 

I didn't change any of the fields / formatting of the json itself (I'd be happy to come up with something) but generally wanted to bring up the idea of having some singularity.json located at "/" in each container that would carry this same meta info as docker inspect. We could perhaps come up with a suggested standard, but allow the user to make any singularity.json that would be desired to expose container meta, options, variables, etc. You could add a command line util to give the user programmatic access, eg:

singularity info ubuntu:latest-2016-04-06.img

or even if we want to be consistent with docker:

singularity inspect ubuntu:latest-2016-04-06.img 

For the singularity.json - my suggestion would be a mixture of meta data about the container (akin to the current docker inspect) but then add fields that will capture programatically what the container can be used for (in terms of inputs and outputs) and variables the user can specify. If we want to do this right we probably should use some standard like the common workflow language, so these things can eventually be controlled by something to fit into an actual workflow. This way, in the long run a user could submit their images to our container hub, and they could be easily parsed for meta and allowable inputs and outputs, and then plug right into a nice workflow web interface. :)

What kinds of fields would be appropriate for this metadata? I've got a couple of ideas but it would be interesting to have a brainstorm of the types of data people would like to be able to obtain about a container. 

If I wanted to use this in a workflow, I would need a list of inputs, outputs, along with acceptable values (file extensions, etc). It seems like in the case of a workflow there are two options - either to capture just inputs and outputs as file types, or as other containers that are acceptable. The first is completely open to connecting any two images given the input --matches--> output, the second is extremely limited but much less likely to lead to error in the workflow generation. Of those inputs and outputs, I would want to be able to specify variables for the container analysis (or purpose) like ports, certificates, and for meta data about the container I would likely want an author (someone to contact with questions or issues). If these are stored on a container hub then there would be a board for the container's issues. 

So would these items be specific to the runscript inside the container at /singularity or something more general about the contents inside the container and other metadata like build date, maintainer, contact, etc?

 

This is a good question. Arguably, there are two (running) use cases here. One kind of container might be a modularized function to be run in a workflow, and in this case we would expect that the items be extremely specific to the runscript, to the extent that another program could parse the meta data and progmatically put together a workflow. The second use case is to capture a dump of something. If I'm a lazy researcher, I would want to just dump an entire execution environment and some files into a container, and just let someone else log in and do whatever. In this case, general information about the contents of the container and the meta data is most important. This said, how about some algorithm like:

1) if there is a runscript, add inputs/outputs, params etc to the datastructure

2) regardless of 1), translate all commands to copy files into the container into a data structure. A list seems like a reasonable start - in the future I can imagine an interactive interface for someone to generate a container, and in adding files they would be asked if the file is important to document for using the container, and if so, add more things about it.

3) Parse all libraries and programs into a meta data object

Ideally, I want to know exactly the software that is included in an image. The detail of versions is probably too hard, but (in the ideal world in my mind) we would be able to grab all commands exported from an image,eg:

commands=`$SUDOCMD docker export $container_id | $SUDOCMD singularity import $new_container_name`

and then do two things. First, store a stupid list of the base software (without paths) so a user can programatically compare the similarity of two images, and also this list becomes basic tags that are a searchable field for containers, and second, some kind of basic parsing and mapping onto a space of software, so if one of those tags is "bwa" I should know that's categorized as a genomic analysis tool. That makes it possible to group containers into another clustering based on what they might be used for, find containers for a specific domain, etc. What Docker hasn't done well is capturing the guts of containers and making them organized and searchable, and singularity can do much better I think.

[Gregory M. Kurtzer]

I will come back to your explanation of the metadata use-case later when I have my keyboard time, but I did just add a patch to also stage supplementary groups into the container's /etc/group file when running as non-root:

[gmk@centos7-x64 demo]$ id

uid=1000(gmk) gid=1000(gmk) groups=1000(gmk),10(wheel),2222(mookaka) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[gmk@centos7-x64 demo]$ singularity exec /tmp/Demo-1.img tail -n 5 /etc/group

utmp:x:22:

utempter:x:35:

gmk:x:1000:gmk

mookaka:x:2222:gmk

[gmk@centos7-x64 demo]$ 

Check out it in [master 0ad6052].

Thanks!

[Dave Love]

vanessa s <vso...@gmail.com> writes:

> I'm testing adding the CMD integration

> <https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L98>,

> and it seems to work fine, however it led me to discover that running a
> simple bash shell from the container does not:

I don't see such a problem with secondary groups which aren't defined in
the container. What is actually complaining?

> [image: Inline image 1]

[For what it's worth, it's easier for me, and I guess others, to deal
with cut-and-paste text rather than images, especially as you can quote
it.]

[Dave Love]

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> On Saturday, June 25, 2016, vanessa s <vso...@gmail.com> wrote:
>
>> Here is a simple command to export the docker inspect json and import into
>> the container as singularity.json (at the container base /)
>>
>>
>> https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L94
>>
>> $SUDOCMD docker inspect $container_id >> singularity.json
>>> sudo singularity copy $new_container_name singularity.json /
>>> rm singularity.json
>>
>>
> I am curious if JSON is the best format for Singularity or the best
> location.

Definitely not if you're in shell-land. Also very little of that
information is relevant to the conversion isn't it?

> An interesting idea would be to add this type of data to the container
> header. It doesn't have to be binary and it has the added benefit of not
> having to mount the container in order to get to the metadata.

Yes, but for anything like that in the filesystem, /etc seems the
right place, like /etc/redhat-release.

> What kinds of fields would be appropriate for this metadata? I've got a
> couple of ideas but it would be interesting to have a brainstorm of the
> types of data people would like to be able to obtain about a container.

The sort of things I was assuming would be architecture (which isn't
obvious, is it?), OS, information about its creation (including the
conversion in this case), some description of what it does (including
whether it's usefully runnable directly), requirements to run it
(e.g. which external MPI). I hadn't thought much about it, though.

[Dave Love]

vanessa s <vso...@gmail.com> writes:

> I don't have strong opinions about the location, but if we are to develop
> tools in python and for the web, then JSON is optimal. If you suggest
> something like yaml or RDF I'm going to run for the hills.

I don't understand how yaml is significantly different from JSON, but
they're both going to make processing in shell scripts much harder than
necessary. What's wrong with simple key-value data? (Otherwise I'll
propose s-expressions, which have been doing such jobs for decades.)

> Oh, cool, I didn't know the containers had headers! How do I inspect / mess
> around with them?

[Apart from head, see what file(1) says.]

>> If I wanted to use this in a workflow, I would need a list of inputs,
> outputs, along with acceptable values (file extensions, etc). It seems like
> in the case of a workflow there are two options - either to capture just
> inputs and outputs as file types, or as other containers that are
> acceptable.

I'm not sure what the latter means.

> The first is completely open to connecting any two images given
> the input --matches--> output, the second is extremely limited but much
> less likely to lead to error in the workflow generation. Of those inputs
> and outputs, I would want to be able to specify variables for the container
> analysis (or purpose) like ports, certificates, and for meta data about the
> container I would likely want an author (someone to contact with questions
> or issues). If these are stored on a container hub then there would be a
> board for the container's issues.
>
> The entire spec for CWL is here,

> <http://www.commonwl.org/draft-3/CommandLineTool.html> but my thinking is

> that we should take an extremely minimalist approach - basically having the
> minimal things listed above, and only expanding on that as we develop
> workflows / use cases and find that there is need. For so many of these
> standards big teams of ontologists come up with meta data things that are
> extremely detailed (and mostly useless) that do nothing but make the
> standard annoying and hard to use.

Yes, definitely don't make everyone to go down that sort of root! We're
talking Unix, and I want abstraction, modularity and composition. [I
say so from scripting scientific "workflows" since long before I heard
the term and before I used Unix, and I've suffered "e-science"...]

Do have documentation and useful metadata, which it seems to me it can
be done effectively in the traditional ways. [I'll try to provide that
for Singularity itself; there should be something like man pages
available locally, or more extensive help, that corresponds to the
version that you have, not just live web pages.] A container can have
help which is similar or identical to normal programs'.

This is the example I've been quoting as a useful demo of running a
Fedora 23 package (although the Debian one might have been better) that
I haven't ported to EPEL6, on which this is running. It's the sort of
use I want to be able to make of something like Singularity if not a
good example for modularity and composition, but the idea is that it's
no different from any other executable.

$ scilab -version
Scilab version "5.5.2.1427793548"
scilab-5.5.2

$ scilab --help|head -5
Usage:
scilab <arguments>
scilab-cli <arguments>
scilab-adv-cli <arguments>

In that case the container doesn't have internal man, for instance, just
the Scilab help, but it could.

There could be a singularity method to print doc and/or /singularity
could look after that as above.

[Dave Love]

For what it's worth, this is the conversion script I ended up with last
week. Note various fixmes.

Greg: feel free to include it in the distribution if appropriate, but
I'll try to improve it a bit from the quick running attempt.
Changes/ideas welcome of course.

By the way, you might want to consider copyright for contributions,
which obviously won't be (C) LBL unless you require assignments. Maybe
the lab has legalese for that. The little changes I've contributed
might now be copyright-significant in total.

This script is intended to be licence-compatible (or even
license-compatible) so that can be adjusted as necessary.

[vanessa s]

> On Jun 27, 2016, at 6:26 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:
>
> "Gregory M. Kurtzer" <gmku...@lbl.gov> writes:
>
>>> On Saturday, June 25, 2016, vanessa s <vso...@gmail.com> wrote:
>>>
>>> Here is a simple command to export the docker inspect json and import into
>>> the container as singularity.json (at the container base /)
>>>
>>>
>>> https://github.com/vsoch/singularity-tools/blob/master/docker/docker2singularity.sh#L94
>>>
>>> $SUDOCMD docker inspect $container_id >> singularity.json
>>>> sudo singularity copy $new_container_name singularity.json /
>>>> rm singularity.json
>> I am curious if JSON is the best format for Singularity or the best
>> location.
>
> Definitely not if you're in shell-land. Also very little of that
> information is relevant to the conversion isn't it?
>

Most use cases I can think of (web and desktop apps for workflow generation) are either in Python or JavaScript land. :)

>> An interesting idea would be to add this type of data to the container
>> header. It doesn't have to be binary and it has the added benefit of not
>> having to mount the container in order to get to the metadata.
>
> Yes, but for anything like that in the filesystem, /etc seems the
> right place, like /etc/redhat-release.
>

Any reason to not use both, perhaps for different things depending on the use case? For workflow generation, I wouldn't want to have to mount the container to learn what inputs it takes, etc.

>> What kinds of fields would be appropriate for this metadata? I've got a
>> couple of ideas but it would be interesting to have a brainstorm of the
>> types of data people would like to be able to obtain about a container.
>
> The sort of things I was assuming would be architecture (which isn't
> obvious, is it?), OS, information about its creation (including the
> conversion in this case), some description of what it does (including
> whether it's usefully runnable directly), requirements to run it
> (e.g. which external MPI). I hadn't thought much about it, though.

Good suggestions!

[vanessa s]

> On Jun 27, 2016, at 6:47 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:
>
> vanessa s <vso...@gmail.com> writes:
>
>> I don't have strong opinions about the location, but if we are to develop
>> tools in python and for the web, then JSON is optimal. If you suggest
>> something like yaml or RDF I'm going to run for the hills.
>
> I don't understand how yaml is significantly different from JSON, but
> they're both going to make processing in shell scripts much harder than
> necessary. What's wrong with simple key-value data? (Otherwise I'll
> propose s-expressions, which have been doing such jobs for decades.)
>

Json is key value, and it plugs seamlessly into Python as a dictionary and is obviously native to JavaScript (JavaScript object notation). We need to use modern data structures that are developer friendly so people find it easy to make applications.

+1

[vanessa s]

Frowny face - any reason to not do this open source? And with version control? I'd like us to work together, and this message basically shoves the script we've been collaborating on under a rug, and says "let's put it in a lbl cereal box with a stamp!"

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

> <singularity-from-docker>

[Gregory M. Kurtzer]

What I'd like to see is a collaboration working on the implementation details and then integrate those findings (citing the sources) into the existing Singularity command architecture (either directly into import or a new docker-import command). I would prefer to keep all Singularity functions within the shell based command syntax.

A point that Dave brought up which is worth reiterating is regarding licensing and copyright assertion. I won't accept a non-BSD license. MIT license may be acceptable if BSD is impossible for you, but no GPL, Apache or other OSI approved licenses will be accepted. Additionally, I leave copyright assertion up to the contributor. When you send a patch or pull request it is your choice to also include your copyright information (or not). I will not limit acceptance of a patch based on copyright. 

I will add that information to the website and documentation as soon as I can. 

[vanessa s]

On Mon, Jun 27, 2016 at 9:52 AM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

What I'd like to see is a collaboration working on the implementation details and then integrate those findings (citing the sources) into the existing Singularity command architecture (either directly into import or a new docker-import command). I would prefer to keep all Singularity functions within the shell based command syntax.

+1 on both these points - I agree that the core singularity should be packaged in a shell based command syntax, and that collaborations should feed into this in the way that best preserves documentation, etc. The workflow infrastructure that I am planning out is going to have a large web component, and command line, and so my suggestion is that the command line until reveals data structures that are seamlessly parsable into these technologies (and still command line friendly) :)

 

A point that Dave brought up which is worth reiterating is regarding licensing and copyright assertion. I won't accept a non-BSD license. MIT license may be acceptable if BSD is impossible for you, but no GPL, Apache or other OSI approved licenses will be accepted. Additionally, I leave copyright assertion up to the contributor. When you send a patch or pull request it is your choice to also include your copyright information (or not). I will not limit acceptance of a patch based on copyright. 

I am not hugely opinionated about licensing - I like MIT in that it is most permissive, and am ok with a well thought out choice. 

I will add that information to the website and documentation as soon as I can.

Thanks for putting your thoughts together so nicely! Same page! :O) 

--

[Gregory M. Kurtzer]

On Mon, Jun 27, 2016 at 10:23 AM, vanessa s <vso...@gmail.com> wrote:

On Mon, Jun 27, 2016 at 9:52 AM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

What I'd like to see is a collaboration working on the implementation details and then integrate those findings (citing the sources) into the existing Singularity command architecture (either directly into import or a new docker-import command). I would prefer to keep all Singularity functions within the shell based command syntax.

+1 on both these points - I agree that the core singularity should be packaged in a shell based command syntax, and that collaborations should feed into this in the way that best preserves documentation, etc. The workflow infrastructure that I am planning out is going to have a large web component, and command line, and so my suggestion is that the command line until reveals data structures that are seamlessly parsable into these technologies (and still command line friendly) :)

I am very interested in what you are thinking for the web component. I've had various ideas on this and I'd love for something like this to get some attention (I wish I could have been further, but time is slim so help and additional development initiatives are really appreciated!).

 

 

A point that Dave brought up which is worth reiterating is regarding licensing and copyright assertion. I won't accept a non-BSD license. MIT license may be acceptable if BSD is impossible for you, but no GPL, Apache or other OSI approved licenses will be accepted. Additionally, I leave copyright assertion up to the contributor. When you send a patch or pull request it is your choice to also include your copyright information (or not). I will not limit acceptance of a patch based on copyright. 

I am not hugely opinionated about licensing - I like MIT in that it is most permissive, and am ok with a well thought out choice. 

I too like the MIT license, but the US Department of Energy and UCOP (my employers) prefers the 3 clause BSD with some added verbiage on contributions... And I'm good with that. lol

 

I will add that information to the website and documentation as soon as I can.

Thanks for putting your thoughts together so nicely! Same page! :O) 

My keyboard time is limited today, so it will probably be tomorrow or soon after.

Thanks again!

Greg

[Dave Love]

vanessa s <vso...@gmail.com> writes:

> Most use cases I can think of (web and desktop apps for workflow
> generation) are either in Python or JavaScript land. :)

Well, that's not the land it comes from and some of us work in, and I
hope it's not going to emigrate more than necessary for educational
purposes.

I don't see how easily Unix-scriptable components are at odds with "web"
and specifically "desktop" usage. We had a principle that anything in
our original graphical "workflow" (if you like) system could be done
equivalently by typing at the command line, later extended to the
RESTful (if you like) interface to the server components. That system
was an important factor in successful competition with LBL, as it
happens.

>> Yes, but for anything like that in the filesystem, /etc seems the
>> right place, like /etc/redhat-release.
>>
>
> Any reason to not use both, perhaps for different things depending on
> the use case? For workflow generation, I wouldn't want to have to
> mount the container to learn what inputs it takes, etc.

A good reason for not having information duplicated is maintaining
consistency. The header is doubtless the best sole place for anything
that's not specifically relevant inside the container and I'm not sure
off-hand what might be necessary inside.

However, /runscript at least needs to be allowed to provide help, as in
my example (which just has /usr/bin/scilab linked to /singularity). It
takes 0.2-0.3s on my system, and is clearly simpler than invoking
singularity <something> directly, which would be perfectly fine for a
container that replied "unrecognized option".

[Dave Love]

vanessa s <vso...@gmail.com> writes:

>> I don't understand how yaml is significantly different from JSON, but
>> they're both going to make processing in shell scripts much harder than
>> necessary. What's wrong with simple key-value data? (Otherwise I'll
>> propose s-expressions, which have been doing such jobs for decades.)
>>
>
> Json is key value,

I know what JSON is, but I thought it was obvious that I meant flat
key-value like, for instance, the format for defining Docker images and
Singularity containers.

> and it plugs seamlessly into Python as a dictionary
> and is obviously native to JavaScript (JavaScript object notation). We
> need to use modern data structures that are developer friendly so
> people find it easy to make applications.

I've no idea how the functional equivalent of an association list, or
even a hash table, is "modern". Bash has associative arrays anyhow, but
that's not the point.

It's unreasonable to exclude the environments of people who are doing
development by insisting on language-du-jour with unnecessarily complex
input because it apparently can't parse simple input easily (which I
clearly in Python). For Lisp applications, I might want s-expressions,
or data frames (? -- I forget the terminology) for R, but would just
read well-defined simple input as necessary.

[Dave Love]

vanessa s <vso...@gmail.com> writes:

> Frowny face - any reason to not do this open source? And with version
> control? I'd like us to work together, and this message basically
> shoves the script we've been collaborating on under a rug, and says
> "let's put it in a lbl cereal box with a stamp!"

I'm no longer surprised by anything around the now-meaningless term
"open source". However, it's odd to be attacked for contributing to a
project under its permissive free software licence (which I wouldn't
choose) rather than balkanizing effort. Especially so in defence of
something without a licence to make it usable, despite containing some
of my code. What on earth is the objection to LBL branding, to the
extent there is any?

If Greg doesn't want a contribution, he doesn't have to put it into the
Singularity repo in an appropriate way (or get me to integrate it).
Anyhow what I posted is tolerably robust and, I think, a reasonable
basis for improvement. It actually works with Docker images I have,
including ones with Size 0, doesn't leak containers, and doesn't delete
anything in cwd. If it's not wanted for Singularity, I'll maintain it
separately. If someone has something better, please share it.

--
It's Time to Talk About Free Software Again — Bruce Perens

[Dave Love]

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> A point that Dave brought up which is worth reiterating is regarding
> licensing and copyright assertion. I won't accept a non-BSD license. MIT
> license may be acceptable if BSD is impossible for you, but no GPL, Apache
> or other OSI approved licenses will be accepted. Additionally, I leave
> copyright assertion up to the contributor. When you send a patch or pull
> request it is your choice to also include your copyright information (or
> not). I will not limit acceptance of a patch based on copyright.

One point is that the licence is LBNL-specific with the names
incorporated, and I'm not sure how that should be treated. Presumably
there are Labs rules/policy on all this, but I've never run into them.

I know you're not a beginner, and maybe Warewulf experience is
different, but I think it's worth tracking copyright holders. I try to
remember to add "Copyright <date> <holder> ..." for significant
contributions as well as noting them in log messages and noting when
they become significant. For instance, I originally gave up on
packaging for Debian because the ftpmaster insisted on a complete list
of copyright holders, which wasn't available (as for Linux, amongst many
other packages, but...). I've also had to re-write stuff whose
provenance wasn't available from revision history.

[vanessa s]

Hi Dave,

I can surely see,

We'll continue to disagree

because we have different roles,

and my preferences are aligned with my goals

No worries, my friend!

on this my work does not depend.

Forum arguing is not a game I want to play,

so have a nice day! :)

Best,

Vanessa

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

[Priedhorsky, Reid]

On Jun 28, 2016, at 7:32 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:

One point is that the licence is LBNL-specific with the names
incorporated, and I'm not sure how that should be treated.  Presumably
there are Labs rules/policy on all this, but I've never run into them.

I’ve expressed concern about this as well. The LBNL license is not an OSI-approved open source license and hasn’t to my knowledge been evaluated by anyone for compatibility with other OSS licenses.

If Singularity is going places, IMO the license situation should be clarified: either re-license to a standard OSS license or submit the LBNL license for community approval and get a third-party opinion on compatibility.

I did ask Greg about this offline a few months ago. IIRC, the LBNL license was the only license available to LBNL folks who wished to publish their source code, by institutional policy.

Reid

[Gregory M. Kurtzer]

On Tue, Jun 28, 2016 at 6:32 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:

"Gregory M. Kurtzer" <gmku...@lbl.gov> writes:

> A point that Dave brought up which is worth reiterating is regarding
> licensing and copyright assertion. I won't accept a non-BSD license. MIT
> license may be acceptable if BSD is impossible for you, but no GPL, Apache
> or other OSI approved licenses will be accepted. Additionally, I leave
> copyright assertion up to the contributor. When you send a patch or pull
> request it is your choice to also include your copyright information (or
> not). I will not limit acceptance of a patch based on copyright.

One point is that the licence is LBNL-specific with the names
incorporated, and I'm not sure how that should be treated.  Presumably
there are Labs rules/policy on all this, but I've never run into them.

When you say "names incorporated", I'm assuming that you are referring the the 3rd clause that references LBNL specifically. I can request that we use the standard 3 clause BSD wording but I would need some form of justification to make the case with legals.

 

I know you're not a beginner, and maybe Warewulf experience is
different, but I think it's worth tracking copyright holders.  I try to
remember to add "Copyright <date> <holder> ..." for significant
contributions as well as noting them in log messages and noting when
they become significant.  For instance, I originally gave up on
packaging for Debian because the ftpmaster insisted on a complete list
of copyright holders, which wasn't available (as for Linux, amongst many
other packages, but...).  I've also had to re-write stuff whose
provenance wasn't available from revision history.

As I mentioned, in my experience, I make this voluntary by the contributor and people would usually throw their names into the headers of the source code in question as they feel appropriate. Are you suggesting that this should be mandated and/or we should maintain a COPYRIGHT file that enumerates each copyright holder in addition to the relevant source files?

Thanks!

[Ralph Castain]

Not sure I understand what folks are talking about here. The LBNL license is essentially the standard 3-clause BSD and has never been questioned before, so why now the angst? We’ve worked with it for many years, and this is the first time anyone has expressed concern about it. It certainly has been accepted as compatible with anything that supports 3-clause BSD.

Whether or not Greg accepts 3rd party copyright “tags” is a separate question.

[Rémy Dernat]

@Vanessa and @Dave : why not having both files (one json and a simple csv/tsv file ?) ? I know how hard it is to parse complex JSON/YAML file from bash, even if there is some trick with awk (https://github.com/micha/jsawk) and how easy it is in pythonor JS. We are just talking about text file (not big at all !)...

About the licensing problem, I would prefer to see just the "BSD" or "MIT" too. IMO, see a copyright (c) with company name, and with anything else than standard OpenSource could be a problem for many of them.

[Gregory M. Kurtzer]

On Tue, Jun 28, 2016 at 8:05 AM, Priedhorsky, Reid <rei...@lanl.gov> wrote:

On Jun 28, 2016, at 7:32 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:

One point is that the licence is LBNL-specific with the names
incorporated, and I'm not sure how that should be treated.  Presumably
there are Labs rules/policy on all this, but I've never run into them.

I’ve expressed concern about this as well. The LBNL license is not an OSI-approved open source license and hasn’t to my knowledge been evaluated by anyone for compatibility with other OSS licenses.

Well, it is basically a standard 3 clause BSD license. The only difference specific to the license has to do with calling out the LBNL/DOE name can not be used for promotion.

Orrrr, are you referring to the bit of verbiage about contributions or the final NOTICE? If so, it should be mentioned that the license itself is standard 3 clause BSD.

 

If Singularity is going places, IMO the license situation should be clarified: either re-license to a standard OSS license or submit the LBNL license for community approval and get a third-party opinion on compatibility.

Would it help in your mind if I spoke with legals about changing the 3rd clause to be standard BSD wording, such that it is no longer modified?

 

I did ask Greg about this offline a few months ago. IIRC, the LBNL license was the only license available to LBNL folks who wished to publish their source code, by institutional policy.

It is not completely institutional policy, rather then collaboration with legals... Some people have gotten other licenses approved, but I will have to push back on them and make the case. 

Thoughts?

[Ralph Castain]

On Jun 28, 2016, at 8:21 AM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

On Tue, Jun 28, 2016 at 8:05 AM, Priedhorsky, Reid <rei...@lanl.gov> wrote:

On Jun 28, 2016, at 7:32 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:

One point is that the licence is LBNL-specific with the names
incorporated, and I'm not sure how that should be treated.  Presumably
there are Labs rules/policy on all this, but I've never run into them.

I’ve expressed concern about this as well. The LBNL license is not an OSI-approved open source license and hasn’t to my knowledge been evaluated by anyone for compatibility with other OSS licenses.

Well, it is basically a standard 3 clause BSD license. The only difference specific to the license has to do with calling out the LBNL/DOE name can not be used for promotion.

You can try, but I’d be surprised if the LBNL lawyers agreed to remove that clause. It doesn’t impact the usage part of the license, and it is certainly far from onerous.

Orrrr, are you referring to the bit of verbiage about contributions or the final NOTICE? If so, it should be mentioned that the license itself is standard 3 clause BSD.

 

If Singularity is going places, IMO the license situation should be clarified: either re-license to a standard OSS license or submit the LBNL license for community approval and get a third-party opinion on compatibility.

Would it help in your mind if I spoke with legals about changing the 3rd clause to be standard BSD wording, such that it is no longer modified?

 

I did ask Greg about this offline a few months ago. IIRC, the LBNL license was the only license available to LBNL folks who wished to publish their source code, by institutional policy.

It is not completely institutional policy, rather then collaboration with legals... Some people have gotten other licenses approved, but I will have to push back on them and make the case. 

Thoughts?

--

Gregory M. Kurtzer
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720

[Priedhorsky, Reid]

RC> The LBNL license is essentially the standard 3-clause BSD

“Essentially” isn’t what matters; what matters it that the difference is non-zero. Folks have to figure out if those differences matter, which is a pain. That’s the justification — it’s unusual. Not an OSI approved license; no community-trusted evaluation of compatibility.

http://www.catb.org/esr/Licensing-HOWTO.html#unoriginal

GK> The only difference specific to the license has to do with calling out the LBNL/DOE name can not be used for promotion.

No, it’s this:

You are under no obligation whatsoever to provide any bug fixes, patches, or

upgrades to the features, functionality or performance of the source code

("Enhancements") to anyone; however, if you choose to make your Enhancements

available either publicly, or directly to Lawrence Berkeley National

Laboratory, without imposing a separate written license agreement for such

Enhancements, then you hereby grant the following license: a non-exclusive,

royalty-free perpetual license to install, use, modify, prepare derivative

works, incorporate into other computer software, distribute, and sublicense

such enhancements or derivative works thereof, in binary and source code form.

The LBNL/DOE non-promotion is in standard clause 3.

RC> and has never been questioned before, so why now the angst? We’ve worked with it for many years, and this is the first time anyone has expressed concern about it.

Could be the right people weren’t paying attention? There are many reasons.

I’d encourage you address the concerns rather than dismissing them for timeliness. If they’re real, they’re real; if they’re not, they’re not.

RC> It certainly has been accepted as compatible with anything that supports 3-clause BSD.

I’d love to be shown wrong on this. Can you provide any citations to such acceptances? I couldn’t find any when I googled it.

Thanks,

Reid

[Gregory M. Kurtzer]

On Tue, Jun 28, 2016 at 8:32 AM, Priedhorsky, Reid <rei...@lanl.gov> wrote:

GK> The only difference specific to the license has to do with calling out the LBNL/DOE name can not be used for promotion.

No, it’s this:

You are under no obligation whatsoever to provide any bug fixes, patches, or

upgrades to the features, functionality or performance of the source code

("Enhancements") to anyone; however, if you choose to make your Enhancements

available either publicly, or directly to Lawrence Berkeley National

Laboratory, without imposing a separate written license agreement for such

Enhancements, then you hereby grant the following license: a non-exclusive,

royalty-free perpetual license to install, use, modify, prepare derivative

works, incorporate into other computer software, distribute, and sublicense

such enhancements or derivative works thereof, in binary and source code form.

Ahhh, that part I don't think will be able to be removed. But... It is technically not the license rather then a contribution agreement and (in simplistic summary) it only ensures that those enhancements or contributions will always be BSD'ish distributable (e.g. not restrict-able via patent's or in any other ways). What is the concern here? I am happy to discuss it with legals and ask for a layman's explanation of it to add to the FAQ. 

[Priedhorsky, Reid]

On Jun 28, 2016, at 9:40 AM, Gregory M. Kurtzer <gmku...@lbl.gov> wrote:

You are under no obligation whatsoever to provide any bug fixes, patches, or

upgrades to the features, functionality or performance of the source code

("Enhancements") to anyone; however, if you choose to make your Enhancements

available either publicly, or directly to Lawrence Berkeley National

Laboratory, without imposing a separate written license agreement for such

Enhancements, then you hereby grant the following license: a non-exclusive,

royalty-free perpetual license to install, use, modify, prepare derivative

works, incorporate into other computer software, distribute, and sublicense

such enhancements or derivative works thereof, in binary and source code form.

Ahhh, that part I don't think will be able to be removed. But... It is technically not the license rather then a contribution agreement

OK, this helps!

My confusion is that the paragraph above really looks like it’s part of the license. Why?

1. — It’s in the LICENSE file.

2. — There’s no caveat saying it’s not part of the license.

3. — It uses license-type language, e.g. “if you choose to make your Enhancements available... you hereby grant the following license”.

and (in simplistic summary) it only ensures that those enhancements or contributions will always be BSD'ish distributable (e.g. not restrict-able via patent's or in any other ways).

But this alters the terms beyond BSD 3-clause.

What is the concern here? I am happy to discuss it with legals and ask for a layman's explanation of it to add to the FAQ. 

I think situation would be clarified if a brief explanation of the circumstances under which one is bound by the above paragraph were added to LICENSE. IMO it needs to go there rather than the FAQ because the latter is not authoritative.

Thanks,

Reid

[Ralph Castain]

You might want to run this by your legal folks there at LANL as the LBNL licensing language has been accepted and used there for a long time (even predating my time there). I’m sure Greg isn't qualified to answer your concerns (nor does anyone on this mailing list have the requisite legal training to do so) ,and certainly he cannot comment on nor modify things that the lawyers wrote and UC approved.

That’s what lawyers are for - we should let them do their job :-)

[Gregory M. Kurtzer]

Hi Reid,

I had a talk with my legal council, and he mentioned that I was wrong. The wording is indeed part of the license, and now, as I understand, this wording is necessary for us allow contributions and ensure that no contributions limit our ability to use, update, or redistribute Singularity.

An example someone contributes code that they don't own (either via their employer or outright stolen). If that code gets noticed by the rightful owner, and we (LBNL) get challenged, we would have no recourse for protecting ourselves. This license does that. It also gives the contributor the freedom to assert a different license. So, if you are concerned, just contribute your work specifically under the standard non-modified BSD 3 Clause license.

In the end, we want to distribute Singularity as openly and freely as possible. This license is both open, and gives LBNL/DOE/UC legals the foundation they need to stand on to be protected from a perceived risk (e.g. tainted contribution).

Hopefully this addresses your concern, but if it does not I am probably not the best person to explain this. I am happy to put you or LANL legals in touch with LBNL legals.

Thanks,

Greg

--

You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Priedhorsky, Reid]

Greg,

Thanks for following up. This does address my concern, in the sense that I think everyone understands the situation better rather than we agree on the magnitude of the problem or whether it’s solved, which is helpful.

I should clarify that I’m not concerned about my ability to contribute but rather pitfalls for Singularity when growing beyond a LBNL project, which I believe is your goal. I’m speaking for myself as a software developer knowledgable about licensing issues. Specifically, that pitfall is that while LBNL lawyers are satisfied with your unusual license, the well-established best practice in the open source community is to use a standard license, and so not doing so is likely to cause difficulty in growing the Singularity community.

So my suggestion, which you can take or not take take obviously, since it’s your project, would be to either:

1. — Seek OSI approval for the LBNL license, or

2. — Ask your people for approval to use a different standard open source license, such as Apache or GPL, which may offer sufficient protections in the areas identified as important.

I’m sure we’ll meet in person before too long given our similar interests, and I’d be happy to discuss further over coffee or whatnot if you’re interested.

Thanks,

Reid

[Gregory M. Kurtzer]

Hi Reid,

Looking into OSI approval is a good suggestion, I will follow up on that.

Additionally, I can just hope that the user's interpretation and understanding of the license is inline with the intent of the license. So far there hasn't been a problem for mine or other open source LBNL projects that I know about, so I'll stay optimistic... <crosses fingers and knocks on wood>

Thanks again for the thoughts!