sandbox Singularity image is usable without Singularity?
192 views
Skip to first unread message
Igor Yakushin
May 17, 2018, 7:55:38 PM5/17/18
to singu...@lbl.gov
Hi All,
Can sandbox Singularity images be used in a standalone way on the systems where there is no Singularity installed if one just sets appropriate paths?
--
Igor Yakushin, Ph.D.
Computational Scientist
Kavli Institute for Cosmological Physics, ERC #413
Research Computing Center, room #2
The University of Chicago
Oliver Freyermuth
May 17, 2018, 8:06:24 PM5/17/18
to singu...@lbl.gov, Igor Yakushin
Hi Igor,
in principle, yes, if you also set LD_LIBRARY_PATH etc., but in the end this somehow defeats the container concept.
Also, you may run into ugly issues if stuff from the "real" OS takes over, for example to load dynamic libraries, and you get funny incompatibilities.
So I'd advice against it - but your mileage may vary.
However, you can also have Singularity run unprivileged, i.e. ship it on a shared filesystem, if you can live without the setuid root functionality (e.g. OverlayFS),
and your kernel is modern enough to support user namespaces and has them enabled. In that case, you can use Singularity without "installing" it.
That's not the recommended mode of operation, though.
If you really *need* to do it, and your kernel is modern enough, then you might also get away with alternatives such as runc or charliecloud, especially charliecloud is designed for this way of operation.
They can run a sandbox image which was build with Singularity, unprivileged, if the kernel has user namespace support enabled.
Without further treatment, however, you will lose some Singularity features, for example the RUN script, and Singularity's environment handling.
Still, I'd say this is less problematic than playing with paths alone.
Another alternative, if you're open to other Linux distros, is for example to use Gentoo Prefix. Gentoo Prefix ( https://wiki.gentoo.org/wiki/Project:Prefix ) can run unprivileged
in almost any Linux / Unix environment on almost any architecture without use of container technology.
Hope something is within those ideas for you,
Oliver
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity...@lbl.gov>.
in principle, yes, if you also set LD_LIBRARY_PATH etc., but in the end this somehow defeats the container concept.
Also, you may run into ugly issues if stuff from the "real" OS takes over, for example to load dynamic libraries, and you get funny incompatibilities.
So I'd advice against it - but your mileage may vary.
However, you can also have Singularity run unprivileged, i.e. ship it on a shared filesystem, if you can live without the setuid root functionality (e.g. OverlayFS),
and your kernel is modern enough to support user namespaces and has them enabled. In that case, you can use Singularity without "installing" it.
That's not the recommended mode of operation, though.
If you really *need* to do it, and your kernel is modern enough, then you might also get away with alternatives such as runc or charliecloud, especially charliecloud is designed for this way of operation.
They can run a sandbox image which was build with Singularity, unprivileged, if the kernel has user namespace support enabled.
Without further treatment, however, you will lose some Singularity features, for example the RUN script, and Singularity's environment handling.
Still, I'd say this is less problematic than playing with paths alone.
Another alternative, if you're open to other Linux distros, is for example to use Gentoo Prefix. Gentoo Prefix ( https://wiki.gentoo.org/wiki/Project:Prefix ) can run unprivileged
in almost any Linux / Unix environment on almost any architecture without use of container technology.
Hope something is within those ideas for you,
Oliver
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity...@lbl.gov>.
Igor Yakushin
May 17, 2018, 8:38:13 PM5/17/18
to Oliver Freyermuth, singu...@lbl.gov
Hi Oliver,
in principle, yes, if you also set LD_LIBRARY_PATH etc., but in the end this somehow defeats the container concept.
Also, you may run into ugly issues if stuff from the "real" OS takes over, for example to load dynamic libraries, and you get funny incompatibilities.
I am looking for a way to run singularity container like environment on systems where sysadmins are resisting installing singularity. In particular I need to run some deep learning code on OpenScience Grid and as far as I heard not all sites with GPUs there have singularity.
So I'd advice against it - but your mileage may vary.
However, you can also have Singularity run unprivileged, i.e. ship it on a shared filesystem, if you can live without the setuid root functionality (e.g. OverlayFS),
and your kernel is modern enough to support user namespaces and has them enabled. In that case, you can use Singularity without "installing" it.
How do I do it? Do I just build singularity and install it on a user account? Would --nv GPU feature work?
If you really *need* to do it, and your kernel is modern enough, then you might also get away with alternatives such as runc or charliecloud, especially charliecloud is designed for this way of operation.
They can run a sandbox image which was build with Singularity, unprivileged, if the kernel has user namespace support enabled.
How can I find out if the kernel has such a support?
Without further treatment, however, you will lose some Singularity features, for example the RUN script, and Singularity's environment handling.
The main feature I would hate to lose is --nv. Unpacking different versions of NVIDIA drivers into container the old way is so annoying...
Another alternative, if you're open to other Linux distros, is for example to use Gentoo Prefix. Gentoo Prefix ( https://wiki.gentoo.org/wiki/Project:Prefix ) can run unprivileged
in almost any Linux / Unix environment on almost any architecture without use of container technology.
Wow! With this, what is the advantage or disadvantage of containers? Will Gentoo installed on a user account under Linux be able to use GPU? OpenMP? MPI? Must one install NVIDIA drivers inside Prefix or it can use the ones from the host? Is Gentoo installed on one Linux portable to another linux by rsync or one has to install it from scratch on each system?
Thank you,
Igor
Gregory M. Kurtzer
May 17, 2018, 11:10:57 PM5/17/18
to singularity
User managed directory based containers are limited in terms of their functionality especially if you compare it to where we are taking the Singularity Image Format (SIF). You can read more about it here:
In a nutshell, Singularity is the only container system that uses a single file runtime image. This single file format was one of the primary design considerations for Singularity as it facilitates user freedom, mobility, ease of use, controls compliance, reproducibility, etc.. Moving forward we are devoting quite a bit of engineering resources to further enhancing the SIF format to include features like guaranteed reproducibility and trusted environments via cryptographically signed and validated runtime images (which no other container runtime can do).
The v3 release of Singularity that will include these features is targeted for this summer, so stay tuned! Also, we are planning on making a technology preview release available very very soon!
We are very excited about this and can't wait to have it released!
Greg
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
Gregory M. Kurtzer
CEO, Sylabs Inc.
Oliver Freyermuth
May 18, 2018, 4:00:56 AM5/18/18
to Igor Yakushin, singu...@lbl.gov
Hi Igor,
Am 18.05.2018 um 02:38 schrieb Igor Yakushin:
> Hi Oliver,
>
>
>
> in principle, yes, if you also set LD_LIBRARY_PATH etc., but in the end this somehow defeats the container concept.
> Also, you may run into ugly issues if stuff from the "real" OS takes over, for example to load dynamic libraries, and you get funny incompatibilities.
>
>
>
> I am looking for a way to run singularity container like environment on systems where sysadmins are resisting installing singularity. In particular I need to run some deep learning code on OpenScience Grid and as far as I heard not all sites with GPUs there have singularity.
ahhh, OSG! This should help ;-). See below.
>
>
>
>
> So I'd advice against it - but your mileage may vary.
>
> However, you can also have Singularity run unprivileged, i.e. ship it on a shared filesystem, if you can live without the setuid root functionality (e.g. OverlayFS),
> and your kernel is modern enough to support user namespaces and has them enabled. In that case, you can use Singularity without "installing" it.
>
>
>
> How do I do it? Do I just build singularity and install it on a user account? Would --nv GPU feature work?
I think you can even have it without installing in OSG.
See:
http://opensciencegrid.org/docs/worker-node/install-singularity/
However, there you also find the information that most sites are not yet ready for user namespace containers. When using CentOS, this means version 7.4 is needed (at least) and the features needs to be enabled explicitly.
As you can see on:
http://opensciencegrid.org/docs/worker-node/install-singularity/#unprivileged-singularity
this needs a kernel commandline flag and a sysctl setting. These you could check (OS version, /proc/cmdline, sysctl -a).
However, if a site has done so, your life is easy: You can use Singularity directly from OSG's CVMFS, i.e.:
/cvmfs/oasis.opensciencegrid.org/mis/singularity/el7-x86_64/bin/singularity
so it's already installed and maintained by them.
I sadly can not tell you whether --nv works - I don't know whether it was implemented for non-setuid root mode. But others on this list can surely answer that.
>
>
>
>
> If you really *need* to do it, and your kernel is modern enough, then you might also get away with alternatives such as runc or charliecloud, especially charliecloud is designed for this way of operation.
> They can run a sandbox image which was build with Singularity, unprivileged, if the kernel has user namespace support enabled.
>
>
> How can I find out if the kernel has such a support?
See above. Another approach is to just "try" if Singularity runs, it will tell you. That's probably easier.
Modern Kernels (I think starting from 4.3 or 4.4) have user namespace support included, unless explicitly disabled by the distribution.
>
>
>
> Without further treatment, however, you will lose some Singularity features, for example the RUN script, and Singularity's environment handling.
>
>
> The main feature I would hate to lose is --nv. Unpacking different versions of NVIDIA drivers into container the old way is so annoying...
>
>
>
>
> Another alternative, if you're open to other Linux distros, is for example to use Gentoo Prefix. Gentoo Prefix ( https://wiki.gentoo.org/wiki/Project:Prefix <https://wiki.gentoo.org/wiki/Project:Prefix> ) can run unprivileged
However, as "end user" you will probably not care ;-).
As such, you can use everything from the host, but there is no such thing as "--nv", so you need to implement that yourself (maybe hooking up https://github.com/NVIDIA/nvidia-container-runtime works, which was written for runc initially?). Apart from that, the good thing is that several MPI versions and nvidia driver versions are packaged for Gentoo.
Concerning portability, this depends - Gentoo is a *compiled* distribution. You can choose your compiler flags either such that the result is very portable and rsyncable, or in a way which is optimized for the CPU at hand.
Cheers and hope that helps,
Oliver
Am 18.05.2018 um 02:38 schrieb Igor Yakushin:
> Hi Oliver,
>
>
>
> in principle, yes, if you also set LD_LIBRARY_PATH etc., but in the end this somehow defeats the container concept.
> Also, you may run into ugly issues if stuff from the "real" OS takes over, for example to load dynamic libraries, and you get funny incompatibilities.
>
>
>
> I am looking for a way to run singularity container like environment on systems where sysadmins are resisting installing singularity. In particular I need to run some deep learning code on OpenScience Grid and as far as I heard not all sites with GPUs there have singularity.
>
>
>
>
> So I'd advice against it - but your mileage may vary.
>
> However, you can also have Singularity run unprivileged, i.e. ship it on a shared filesystem, if you can live without the setuid root functionality (e.g. OverlayFS),
> and your kernel is modern enough to support user namespaces and has them enabled. In that case, you can use Singularity without "installing" it.
>
>
>
> How do I do it? Do I just build singularity and install it on a user account? Would --nv GPU feature work?
See:
http://opensciencegrid.org/docs/worker-node/install-singularity/
However, there you also find the information that most sites are not yet ready for user namespace containers. When using CentOS, this means version 7.4 is needed (at least) and the features needs to be enabled explicitly.
As you can see on:
http://opensciencegrid.org/docs/worker-node/install-singularity/#unprivileged-singularity
this needs a kernel commandline flag and a sysctl setting. These you could check (OS version, /proc/cmdline, sysctl -a).
However, if a site has done so, your life is easy: You can use Singularity directly from OSG's CVMFS, i.e.:
/cvmfs/oasis.opensciencegrid.org/mis/singularity/el7-x86_64/bin/singularity
so it's already installed and maintained by them.
I sadly can not tell you whether --nv works - I don't know whether it was implemented for non-setuid root mode. But others on this list can surely answer that.
>
>
>
>
> If you really *need* to do it, and your kernel is modern enough, then you might also get away with alternatives such as runc or charliecloud, especially charliecloud is designed for this way of operation.
> They can run a sandbox image which was build with Singularity, unprivileged, if the kernel has user namespace support enabled.
>
>
> How can I find out if the kernel has such a support?
Modern Kernels (I think starting from 4.3 or 4.4) have user namespace support included, unless explicitly disabled by the distribution.
>
>
>
> Without further treatment, however, you will lose some Singularity features, for example the RUN script, and Singularity's environment handling.
>
>
> The main feature I would hate to lose is --nv. Unpacking different versions of NVIDIA drivers into container the old way is so annoying...
>
>
>
>
> in almost any Linux / Unix environment on almost any architecture without use of container technology.
>
>
> Wow! With this, what is the advantage or disadvantage of containers? Will Gentoo installed on a user account under Linux be able to use GPU? OpenMP? MPI? Must one install NVIDIA drivers inside Prefix or it can use the ones from the host? Is Gentoo installed on one Linux portable to another linux by rsync or one has to install it from scratch on each system?
The advantage and disadvantage is that there is no isolation, i.e. no PID namespace shielding out PIDs from other users.
>
>
> Wow! With this, what is the advantage or disadvantage of containers? Will Gentoo installed on a user account under Linux be able to use GPU? OpenMP? MPI? Must one install NVIDIA drivers inside Prefix or it can use the ones from the host? Is Gentoo installed on one Linux portable to another linux by rsync or one has to install it from scratch on each system?
However, as "end user" you will probably not care ;-).
As such, you can use everything from the host, but there is no such thing as "--nv", so you need to implement that yourself (maybe hooking up https://github.com/NVIDIA/nvidia-container-runtime works, which was written for runc initially?). Apart from that, the good thing is that several MPI versions and nvidia driver versions are packaged for Gentoo.
Concerning portability, this depends - Gentoo is a *compiled* distribution. You can choose your compiler flags either such that the result is very portable and rsyncable, or in a way which is optimized for the CPU at hand.
Cheers and hope that helps,
Oliver
Oliver Freyermuth
May 18, 2018, 4:54:10 AM5/18/18
to singu...@lbl.gov, Gregory M. Kurtzer, Thomas Hartmann, germ...@gmail.com
Hi Greg,
Am 18.05.2018 um 05:10 schrieb Gregory M. Kurtzer:
> User managed directory based containers are limited in terms of their functionality especially if you compare it to where we are taking the Singularity Image Format (SIF). You can read more about it here:
>
> https://www.sylabs.io/2018/03/sif-containing-your-containers/
>
> In a nutshell, Singularity is the only container system that uses a single file runtime image. This single file format was one of the primary design considerations for Singularity as it facilitates user freedom, mobility, ease of use, controls compliance, reproducibility, etc.. Moving forward we are devoting quite a bit of engineering resources to further enhancing the SIF format to include features like guaranteed reproducibility and trusted environments via cryptographically signed and validated runtime images (which no other container runtime can do).
>
> The v3 release of Singularity that will include these features is targeted for this summer, so stay tuned! Also, we are planning on making a technology preview release available very very soon!
>
> We are very excited about this and can't wait to have it released!
>
> Greg
thanks for the heads-up!
What's still unclear to me, though: Will Singularity still keep sandbox-support functional in the new series? There are many users of it, especially all CVMFS users.
Especially for them, SIF does not really increase mobility - if encryption is not required, and the FS is trusted and ensures the container is not modified, sandbox containers are more mobile.
Of course, SIF is ideal for many other usecases.
Also, with good encryption, things like casync will naturally stop working. Is there anything planned already on how to transfer deltas in the encrypted and compressed "SIF world"?
This would have to happen e.g. by reimplementing content-defined-chunking e.g. by leveraging casync / zsync at both ends.
Cheers,
Oliver
>
> On Thu, May 17, 2018 at 5:38 PM, Igor Yakushin <iv...@uchicago.edu <mailto:iv...@uchicago.edu>> wrote:
>
> Hi Oliver,
>
>
>
> in principle, yes, if you also set LD_LIBRARY_PATH etc., but in the end this somehow defeats the container concept.
> Also, you may run into ugly issues if stuff from the "real" OS takes over, for example to load dynamic libraries, and you get funny incompatibilities.
>
>
>
> I am looking for a way to run singularity container like environment on systems where sysadmins are resisting installing singularity. In particular I need to run some deep learning code on OpenScience Grid and as far as I heard not all sites with GPUs there have singularity.
>
>
>
>
> So I'd advice against it - but your mileage may vary.
>
> However, you can also have Singularity run unprivileged, i.e. ship it on a shared filesystem, if you can live without the setuid root functionality (e.g. OverlayFS),
> and your kernel is modern enough to support user namespaces and has them enabled. In that case, you can use Singularity without "installing" it.
>
>
>
> How do I do it? Do I just build singularity and install it on a user account? Would --nv GPU feature work?
>
>
>
>
> If you really *need* to do it, and your kernel is modern enough, then you might also get away with alternatives such as runc or charliecloud, especially charliecloud is designed for this way of operation.
> They can run a sandbox image which was build with Singularity, unprivileged, if the kernel has user namespace support enabled.
>
>
> How can I find out if the kernel has such a support?
>
>
>
> Without further treatment, however, you will lose some Singularity features, for example the RUN script, and Singularity's environment handling.
>
>
> The main feature I would hate to lose is --nv. Unpacking different versions of NVIDIA drivers into container the old way is so annoying...
>
>
>
>
> Another alternative, if you're open to other Linux distros, is for example to use Gentoo Prefix. Gentoo Prefix ( https://wiki.gentoo.org/wiki/Project:Prefix <https://wiki.gentoo.org/wiki/Project:Prefix> ) can run unprivileged
--
Oliver Freyermuth
Universität Bonn
Physikalisches Institut, Raum 1.047
Nußallee 12
53115 Bonn
--
Tel.: +49 228 73 2367
Fax: +49 228 73 7869
--
Am 18.05.2018 um 05:10 schrieb Gregory M. Kurtzer:
> User managed directory based containers are limited in terms of their functionality especially if you compare it to where we are taking the Singularity Image Format (SIF). You can read more about it here:
>
> https://www.sylabs.io/2018/03/sif-containing-your-containers/
>
> In a nutshell, Singularity is the only container system that uses a single file runtime image. This single file format was one of the primary design considerations for Singularity as it facilitates user freedom, mobility, ease of use, controls compliance, reproducibility, etc.. Moving forward we are devoting quite a bit of engineering resources to further enhancing the SIF format to include features like guaranteed reproducibility and trusted environments via cryptographically signed and validated runtime images (which no other container runtime can do).
>
> The v3 release of Singularity that will include these features is targeted for this summer, so stay tuned! Also, we are planning on making a technology preview release available very very soon!
>
> We are very excited about this and can't wait to have it released!
>
> Greg
What's still unclear to me, though: Will Singularity still keep sandbox-support functional in the new series? There are many users of it, especially all CVMFS users.
Especially for them, SIF does not really increase mobility - if encryption is not required, and the FS is trusted and ensures the container is not modified, sandbox containers are more mobile.
Of course, SIF is ideal for many other usecases.
Also, with good encryption, things like casync will naturally stop working. Is there anything planned already on how to transfer deltas in the encrypted and compressed "SIF world"?
This would have to happen e.g. by reimplementing content-defined-chunking e.g. by leveraging casync / zsync at both ends.
Cheers,
Oliver
>
> On Thu, May 17, 2018 at 5:38 PM, Igor Yakushin <iv...@uchicago.edu <mailto:iv...@uchicago.edu>> wrote:
>
> Hi Oliver,
>
>
>
> in principle, yes, if you also set LD_LIBRARY_PATH etc., but in the end this somehow defeats the container concept.
> Also, you may run into ugly issues if stuff from the "real" OS takes over, for example to load dynamic libraries, and you get funny incompatibilities.
>
>
>
> I am looking for a way to run singularity container like environment on systems where sysadmins are resisting installing singularity. In particular I need to run some deep learning code on OpenScience Grid and as far as I heard not all sites with GPUs there have singularity.
>
>
>
>
> So I'd advice against it - but your mileage may vary.
>
> However, you can also have Singularity run unprivileged, i.e. ship it on a shared filesystem, if you can live without the setuid root functionality (e.g. OverlayFS),
> and your kernel is modern enough to support user namespaces and has them enabled. In that case, you can use Singularity without "installing" it.
>
>
>
> How do I do it? Do I just build singularity and install it on a user account? Would --nv GPU feature work?
>
>
>
>
> If you really *need* to do it, and your kernel is modern enough, then you might also get away with alternatives such as runc or charliecloud, especially charliecloud is designed for this way of operation.
> They can run a sandbox image which was build with Singularity, unprivileged, if the kernel has user namespace support enabled.
>
>
> How can I find out if the kernel has such a support?
>
>
>
> Without further treatment, however, you will lose some Singularity features, for example the RUN script, and Singularity's environment handling.
>
>
> The main feature I would hate to lose is --nv. Unpacking different versions of NVIDIA drivers into container the old way is so annoying...
>
>
>
>
> in almost any Linux / Unix environment on almost any architecture without use of container technology.
>
>
> Wow! With this, what is the advantage or disadvantage of containers? Will Gentoo installed on a user account under Linux be able to use GPU? OpenMP? MPI? Must one install NVIDIA drivers inside Prefix or it can use the ones from the host? Is Gentoo installed on one Linux portable to another linux by rsync or one has to install it from scratch on each system?
>
> Thank you,
> Igor
> --
> Igor Yakushin, Ph.D.
> Computational Scientist
> Kavli Institute for Cosmological Physics, ERC #413
> Research Computing Center, room #2
> The University of Chicago
> 773-834-5313
> https://rcc.uchicago.edu
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity...@lbl.gov>.
>
>
> Wow! With this, what is the advantage or disadvantage of containers? Will Gentoo installed on a user account under Linux be able to use GPU? OpenMP? MPI? Must one install NVIDIA drivers inside Prefix or it can use the ones from the host? Is Gentoo installed on one Linux portable to another linux by rsync or one has to install it from scratch on each system?
>
> Thank you,
> Igor
> --
> Igor Yakushin, Ph.D.
> Computational Scientist
> Kavli Institute for Cosmological Physics, ERC #413
> Research Computing Center, room #2
> The University of Chicago
> 773-834-5313
> https://rcc.uchicago.edu
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
>
>
>
>
> --
> Gregory M. Kurtzer
> CEO, Sylabs Inc.
>
>
>
>
> --
> Gregory M. Kurtzer
> CEO, Sylabs Inc.
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity...@lbl.gov>.
> You received this message because you are subscribed to the Google Groups "singularity" group.
--
Oliver Freyermuth
Universität Bonn
Physikalisches Institut, Raum 1.047
Nußallee 12
53115 Bonn
--
Tel.: +49 228 73 2367
Fax: +49 228 73 7869
--
Thomas Hartmann
May 18, 2018, 6:23:22 AM5/18/18
to singu...@lbl.gov
Hi all,
concerning CVMFS I would second Oliver - worrying a bit about the
deduplicatibility of images with CVMFS. Will SIF be one single image or
have some kind of chunk(/block) structure? I suppose with a chunk format
and signed (and 'aligned') chunks, CVMFS inherent dedup should apply as
for sandbox dirs, or?
I guess, that encrypted images are naturally not dedup'table in any way
(assuming some salting for each images/key/...).
Cheers,
Thomas
concerning CVMFS I would second Oliver - worrying a bit about the
deduplicatibility of images with CVMFS. Will SIF be one single image or
have some kind of chunk(/block) structure? I suppose with a chunk format
and signed (and 'aligned') chunks, CVMFS inherent dedup should apply as
for sandbox dirs, or?
I guess, that encrypted images are naturally not dedup'table in any way
(assuming some salting for each images/key/...).
Cheers,
Thomas
David Trudgian
May 18, 2018, 10:38:42 AM5/18/18
to singularity, gmku...@gmail.com, thomas....@desy.de, germ...@gmail.com
Hi Oliver,
Sandbox support will remain. I'll let someone else address the SIF structural / sync concerns.
DT
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity+unsub...@lbl.gov>.
>
>
>
>
> --
> Gregory M. Kurtzer
> CEO, Sylabs Inc.
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity+unsub...@lbl.gov>.
Yannick Cote
May 18, 2018, 3:04:19 PM5/18/18
to singu...@lbl.gov, Gregory M. Kurtzer, thomas....@desy.de, germ...@gmail.com
Hi Oliver and Thomas,
SIF organizes various piece of data pertaining to a container inside a file. This file system image can be used directly by the SIF tools and libraries at runtime or by mapping the image to the file system if desired. It is also possible to explode the content of a SIF image into directories and files if necessary. We are currently expanding our SIF manipulation tool to that effect. Also like Dave T. already suggested, sandboxes can still be used where they make better sense. In other words, plenty of choice. We believe that the SIF container format will help with the majority of use cases where simplicity around mobility but also access to features like signing/verification and the storage of various data and meta-data related to the container is wanted.
For the case of encryption, this is of course an optional feature. I foresee encryption being used mostly in environments where users have direct control over how their data and files are managed. Once again, even though SIF's attempt is to simply be a file system designed for containers opening up a lot of possibilities, there may be cases where mapping its content via FUSE to the file system is more useful. Encryption is on our roadmap and there's still efforts required to get it out there. If you have any insights on how you would see it all working nicely, we'd be glad to start a thread on that :-)
Thomas, I am not sure what you mean by single image versus (chunks), but if it helps, a SIF file has a file system structure with a global header, data object descriptors (like inodes), and data blocks pointed to by those data object descriptors. Data objects inside a SIF file are completely independent and can be dumped individually by using the "sif dump <desc_id>" command. Note that you can also, at any time during the container lifetime add, remove, sign, verify list and get info for those data objects independently or by group. Yes data blocks are aligned to values that make sense. They may be page aligned or sector aligned but this is configurable.
Cheers,
- Yannick
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity+unsubscribe...@lbl.gov>.
>
>
>
>
> --
> Gregory M. Kurtzer
> CEO, Sylabs Inc.
>
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov <mailto:singularity+unsubscribe...@lbl.gov>.
--
Oliver Freyermuth
Universität Bonn
Physikalisches Institut, Raum 1.047
Nußallee 12
53115 Bonn
--
Tel.: +49 228 73 2367
Fax: +49 228 73 7869
--
--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.
Dave Dykstra
May 22, 2018, 5:45:51 PM5/22/18
to 'Oliver Freyermuth' via singularity, Igor Yakushin
If singularity is not installed on OSG resources with GPUs, it can
probably be installed if you ask. Note however that there's currently a
problem with the 2.5.x implementation of --nv:
https://github.com/singularityware/singularity/issues/1548
Dave
probably be installed if you ask. Note however that there's currently a
problem with the 2.5.x implementation of --nv:
https://github.com/singularityware/singularity/issues/1548
Dave
> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
> You received this message because you are subscribed to the Google Groups "singularity" group.
Igor Yakushin
May 23, 2018, 3:59:11 PM5/23/18
to Dave Dykstra, 'Oliver Freyermuth' via singularity
On Tue, May 22, 2018 at 4:45 PM, Dave Dykstra <d...@fnal.gov> wrote:
If singularity is not installed on OSG resources with GPUs, it can
probably be installed if you ask. Note however that there's currently a
problem with the 2.5.x implementation of --nv:
https://github.com/singularityware/singularity/issues/1548
It actually works fine for me in the new version with one peculiarity:
it does not work with 'shell' but works with 'exec'.
Igor Yakushin
May 23, 2018, 4:01:38 PM5/23/18
to Dave Dykstra, 'Oliver Freyermuth' via singularity
Another peculiarity I forgot to mention:
When I try to run a tensorflow docker container directly from docker repo, sometimes it works and sometimes it doesn't. However, if I build a local image based on that docker image, it works fine.
Not sure if it is simply some unreliability in loading docker container or it is related to --nv.
Reply all
Reply to author
Forward
0 new messages