'casync' as option to distribute images?

[Thomas Hartmann]

Hi all,

I just stumbled over 'casync' - and it's use case for delivering
containers by combining rsync deltas and git chuncking sounds imho
interesting

http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html
maybe it could serve as another way/tool to distribute Singularity images?

Cheers,
Thomas

[Michael Galloway]

Also, and just as an FYI, OSG uses cvmfs from CERN as a distribution method as well:

https://support.opensciencegrid.org/support/solutions/articles/12000024676-singularity-containers

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Thomas Hartmann]

yes, CVMFS is certainly the right way to distribute images on a larger
scale - but setting up a repo for each small group or granting too many
people access to one repo might get cumbersome (on the other hand,
Singularity hub might be the better solution for these cases)

> <mailto:singularity%2Bunsu...@lbl.gov>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to singularity...@lbl.gov

> <mailto:singularity...@lbl.gov>.

[vanessa s]

This doesn't sound so remarkably different than Docker, eg read this paragraph:

Encoding: Let's take a large linear data stream, split it into variable-sized chunks (the size of each being a function of the chunk's contents), and store these chunks in individual, compressed files in some directory, each file named after a strong hash value of its contents, so that the hash value may be used to as key for retrieving the full chunk data. Let's call this directory a "chunk store". At the same time, generate a "chunk index" file that lists these chunk hash values plus their respective chunk sizes in a simple linear array. The chunking algorithm is supposed to create variable, but similarly sized chunks from the data stream, and do so in a way that the same data results in the same chunks even if placed at varying offsets. For more information

• "chunk" --> layer
  
• "individual compressed files" --> .tar.gz files
  
• named after hash value of its contents --> this is what Docker does too :)
  
• "chunk store" --> cache folder
  
• "chunk index" --> image manifest
  

similar but not totally related, this looks pretty cool :)

https://github.com/ncw/rclone

On Fri, Aug 25, 2017 at 6:08 AM, Thomas Hartmann <thomas....@desy.de> wrote:

yes, CVMFS is certainly the right way to distribute images on a larger
scale - but setting up a repo for each small group or granting too many
people access to one repo might get cumbersome (on the other hand,
Singularity hub might be the better solution for these cases)

On 2017-08-25 13:56, Michael Galloway wrote:
> Also, and just as an FYI, OSG uses cvmfs from CERN as a distribution
> method as well:
>
> https://support.opensciencegrid.org/support/solutions/articles/12000024676-singularity-containers
>
>
> On Fri, Aug 25, 2017 at 4:32 AM Thomas Hartmann <thomas....@desy.de

> <mailto:thomas.hartmann@desy.de>> wrote:
>
>     Hi all,
>
>     I just stumbled over 'casync' - and it's use case for delivering
>     containers by combining rsync deltas and git chuncking sounds imho
>     interesting
>
>     http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html
>     maybe it could serve as another way/tool to distribute Singularity
>     images?
>
>     Cheers,
>       Thomas
>
>     --
>     You received this message because you are subscribed to the Google
>     Groups "singularity" group.
>     To unsubscribe from this group and stop receiving emails from it,

>     send an email to singularity+unsubscribe@lbl.gov
>     <mailto:singularity%2Bunsu...@lbl.gov>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to singularity+unsubscribe@lbl.gov
> <mailto:singularity+unsub...@lbl.gov>.

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676