Local cache for files downloaded during a build

7 views
Skip to first unread message

D.A. P.

unread,
Jun 17, 2021, 4:03:46 PMJun 17
to Singularity Community Edition
When a Singularity container definition file is boostrapped from a docker container, it appears that it makes use of locally cached blobs to prevent re-downloading the bootstrap image. However, during the build I may have a number of commands that collect files from the Internet. These can include apt / yum / dnf installations, conda installations, and others that are tedious to cache manually and then copy in via the %files section.

Are there any techniques for establishing a local cache of files collecting during a container build such that subsequent builds with minor edits to the build file don't trigger complete re-downloading of the same files from the Internet?

David Trudgian

unread,
Jun 17, 2021, 6:06:07 PMJun 17
to Singularity Community Edition
Hi,

I'm afraid there is no method by which SingularityCE can track the files that are downloaded by tools during a build. You could, using the new built time bind mount feature with Singularity 3.8, try to mount a directory on the host into the relevant cache location in the container during the build.

It would also be possible to fetch the files to the host, if they do not exist, in a `%setup` block... and then use them in the `%post` block. However this is not much different than caching manually and copying in via `%files`.

DT

D.A. P.

unread,
Jun 17, 2021, 9:11:05 PMJun 17
to Singularity Community Edition
Thanks for the ideas and confirmation of the current behavior. I suppose a cache server such as Squid might also help.
Reply all
Reply to author
Forward
0 new messages