Container format not recognised after upgrading Singularity from 2.6 to 3.1

[Will Furnass]

Hello all,

We upgraded from 2.6 to 3.1.1 on a Centos 7.x HPC system and have
found that a file is no longer a image format that Singularity
recognises:

$ singularity exec epic_analysis.img /bin/sh
FATAL: image format not recognized
$ singularity inspect epic_analysis.img
FATAL: exit status 255
$ file epic_analysis.img
epic_analysis.img: data
$ strings epic_analysis.img | head
#!/usr/bin/env run-singularity
#!/usr/bin/env run-singularity
/var/lib/singularity/mnt/final
.singularity.d/actions/exec
.singularity.d/actions/run
.singularity.d/actions/shell
.singularity.d/actions/test
.singularity.d/env/90-environment.sh
\$f)
5B)>


We are however able to pull/run images from Dockerhub and run other
locally-built images also worked with 2.6. For info, the problematic
image was built with 2.5.2. No pertinent messages in dmesg/syslog
after failed attempts to 'singularity exec'.

Any comments/suggestions for how to fault-find?

Regards,

Will

--
Dr Will Furnass | Acting Research Software Engineering team lead
Computer Science | University of Sheffield
+44 (0) 114 22 21936 | https://rse.shef.ac.uk
@willfurnass | https://learningpatterns.me
---
Book a code clinic session: https://rse.shef.ac.uk/support/code-clinic/

[Jason Stover]

Hi Will,

Can you attach debug output of a run here as a start? Was there
anything special about the image (that you know of?). Meaning, ext3
image, etc...

Thanks,
-J

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAHpBHzBE7U1dVtnN1JVxnvqobo98bcVC_c%3DLobxheMv1uyGv8A%40mail.gmail.com.

[Will Furnass]

Hi Jason,

Here's the debug output:

$ singularity --debug exec epic_analysis.img /bin/sh
VERBOSE [U=0,P=104546] print() Set messagelevel to: 5
DEBUG [U=0,P=104546] init() PIPE_EXEC_FD value: 8
VERBOSE [U=0,P=104546] init() Container runtime
VERBOSE [U=0,P=104546] is_suid() Check if we
are running as setuid
DEBUG [U=0,P=104546] init() Overlay seems
supported by kernel
DEBUG [U=0,P=104546] init() Drop privileges
DEBUG [U=131937,P=104546] init() Read json
configuration from pipe
DEBUG [U=131937,P=104546] init() Set child signal mask
DEBUG [U=131937,P=104546] init() Wait
completion of stage1
VERBOSE [U=131937,P=104559] priv_escalate() Get root privileges
DEBUG [U=0,P=104559] set_parent_death_signal() Set parent
death signal to 9
DEBUG [U=0,P=104559] prepare_stage() Entering in stage 1
DEBUG [U=131937,P=104559] set_parent_death_signal() Set parent
death signal to 9
VERBOSE [U=131937,P=104559] init() Spawn stage 1
VERBOSE [U=131937,P=104559]startup() Execute stage 1
DEBUG [U=131937,P=104559]Stage() Entering stage 1
DEBUG [U=131937,P=104559]prepareUserCaps() Ignoring
group 20013: group: unknown groupid 20013
DEBUG [U=131937,P=104559]prepareFd() Open file
descriptor for /scratch
DEBUG [U=131937,P=104559]prepareFd() Open file
descriptor for /fastdata
DEBUG [U=131937,P=104559]prepareFd() Open file
descriptor for /data
DEBUG [U=131937,P=104559]prepareFd() Open file
descriptor for /shared
DEBUG [U=131937,P=104559]Init() Entering
image format intializer
DEBUG [U=131937,P=104559]Init() Check for
image format sif
DEBUG [U=131937,P=104559]Init() sif format
initializer returns: SIF magic not found
DEBUG [U=131937,P=104559]Init() Check for
image format sandbox
DEBUG [U=131937,P=104559]Init() sandbox
format initializer returns: not a directory image
DEBUG [U=131937,P=104559]Init() Check for
image format squashfs
DEBUG [U=131937,P=104559]Init() squashfs
format initializer returns: not a valid squashfs image
DEBUG [U=131937,P=104559]Init() Check for
image format ext3
DEBUG [U=131937,P=104559]Init() ext3 format
initializer returns: file is not a valid ext3 image
FATAL [U=131937,P=104559]Stage() image format
not recognized
VERBOSE [U=131937,P=104546] init() stage 1
exited with status 255

The user created the problematic image from DockerHub
(markdunning/epic_analysis):

sudo singularity build --writable epic_analysis.img
docker://markdunning/epic_analysis

However, regenerating an image from DockerHub using 'pull' works fine:

$ SINGULARITY_LOCALCACHEDIR=$TMPDIR/ SINGULARITY_CACHEDIR=$TMPDIR/
singularity pull docker://markdunning/epic_analysis
$ singularity exec epic_analysis_latest.sif /bin/cat /etc/issue
Debian GNU/Linux 9 \n \l

Regards,

Will

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAGfAqt_pUVga2Sp20Feb5JZ%2B%2BCE-tbxu%2BbjBDXtCu7amRqNM8g%40mail.gmail.com.

[Jason Stover]

> The user created the problematic image from DockerHub (markdunning/epic_analysis):
>
> sudo singularity build --writable epic_analysis.img docker://markdunning/epic_analysis
>

Hrmm... that _should_ be an ext3 image then, which the Init() call
doesn't think is true.

What are the permissions on the image file? Does the user have read
access to that file? That is the only way I can replicate the issue:

$ sudo chmod 700 epic_analysis.img
$ /usr/local/singularity/3.1.1/bin/singularity exec epic_analysis.img
cat /etc/issue

FATAL: image format not recognized

If the permissions allow me to read the file, then I get:

$ /usr/local/singularity/3.1.1/bin/singularity exec epic_analysis.img

cat /etc/issue
Debian GNU/Linux 9 \n \l

-J

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAHpBHzBP05HgPzCZvs8u5%2BXvnHmzM_Ja2kFirTPTg1tq96v8aQ%40mail.gmail.com.

[Will Furnass]

Hi Jason

On Fri, 31 May 2019 at 16:29, Jason Stover <jason....@gmail.com> wrote:
>
> Hrmm... that _should_ be an ext3 image then, which the Init() call
> doesn't think is true.
>
> What are the permissions on the image file? Does the user have read
> access to that file?

Yes, the file is readable but not interpretable by Singularity 3.1:

[user@sharc-node002 public]$ ls -l epic_analysis.img
-rwx---r-x 1 otheruser md 5715787838 May 31 13:47 epic_analysis.img
[user@sharc-node002 public]$ strings epic_analysis.img | head -3
#!/usr/bin/env run-singularity
#!/usr/bin/env run-singularity
[user@sharc-node002 public]$ singularity exec epic_analysis.img /bin/sh

FATAL: image format not recognized

It's no big deal if we have to throw away the file; I just thought I'd
bring this up in case this relates to a corner case that might affect
others.

Cheers,

[Jason Stover]

Hrmm... The image looks corrupt...

> $ strings epic_analysis.img | head -3
> #!/usr/bin/env run-singularity
> #!/usr/bin/env run-singularity

That should only be showing up once. Here's what I get from the image
I just built:

$ strings epic_analysis.img | head -3
#!/usr/bin/env run-singularity

/usr/local/singularity/2.5.2/var/singularity/mnt/final # Hard coded path
.singularity.d/env/90-environment.sh

-J

> --
> You received this message because you are subscribed to the Google Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/singularity/CAHpBHzD5LSUzXqAo-U-g1yWY5oVSKG6-djrn1D9Rpx4DM0eBdw%40mail.gmail.com.

[Will Furnass]

On Fri, 31 May 2019 at 16:58, Jason Stover <jason....@gmail.com> wrote:
>
> Hrmm... The image looks corrupt...
>
> > $ strings epic_analysis.img | head -3
> > #!/usr/bin/env run-singularity
> > #!/usr/bin/env run-singularity
>
> That should only be showing up once. Here's what I get from the image
> I just built:
>
> $ strings epic_analysis.img | head -3
> #!/usr/bin/env run-singularity
> /usr/local/singularity/2.5.2/var/singularity/mnt/final # Hard coded path
> .singularity.d/env/90-environment.sh

Thanks for your thoughts; we'll mark this down as a fluke!

Cheers,

Will