Singularity pull/import of some images from docker hub suddenly stopped working

[Mike]

Hello,

It seems that for some images from dockerhub, singularity pull or import fails with an error message

ERROR Improperly formed manifest, layers or fsLayers must be present

Example:

 $ sudo singularity pull --size 1024 docker://ubuntu:latest
Initializing Singularity image subsystem
Opening image file: ubuntu-latest.img
Creating 1024MiB image
Binding image to loop
Creating file system within image
Image is done: ubuntu-latest.img
Docker image path: index.docker.io/library/ubuntu:latest
ERROR Improperly formed manifest, layers or fsLayers must be present

Same with import to existing .img file.
The problem also occurs with other images, e.g. for ubuntu:16.04, ubuntu:16.04.1, ubuntu:16.04.3, ubuntu:14.04, debian:stable

But, for ubuntu:14.04.1 and centos:latest, the images download and run fine. I have no problem pulling and running any of those images with docker.

The problem began less than 24 hours ago (now is 13 Sep. 2017, 13:00 UTC). I cross checked by rolling back my virtual test host system to a snapshot and by installing singularity on a clean physical host. My hosts are running Ubuntu 16.04.3 LTS.

The output from "singularity -vvvv import" is

 $ singularity create -s 1024 ubuntu.img
Initializing Singularity image subsystem
Opening image file: ubuntu.img
Creating 1024MiB image
Binding image to loop
Creating file system within image
Image is done: ubuntu.img

 $ sudo singularity -vvvv import ubuntu.img docker://ubuntu:latest
Increasing verbosity level (5)
Ending argument loop
Singularity version: 2.3.1-master.ga47422c
Exec'ing: /usr/local/libexec/singularity/cli/import.exec
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS not defined (None)
VERBOSE2 SINGULARITY_METADATA_FOLDER found as None/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /root/.singularity
VERBOSE2 SINGULARITY_ENVIRONMENT found as None/.singularity.d/environment
VERBOSE2 SINGULARITY_RUNSCRIPT found as None/singularity
VERBOSE2 SINGULARITY_TESTFILE found as None/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as None/.singularity.d/Singularity
VERBOSE2 SINGULARITY_ENVBASE found as None/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as None/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as /home/c102mf/junk
VERBOSE2 SHUB_NAMEBYHASH not defined (None)
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layers.ZYNCdB4H
VERBOSE2 SINGULARITY_PYTHREADS found as 9
VERBOSE2 SINGULARITY_CONTAINER found as docker://ubuntu:latest
DEBUG Found uri docker://
DEBUG
*** STARTING DOCKER IMPORT PYTHON  ****
DEBUG Docker layers and (env,labels,runscript) will be written to: /tmp/.singularity-layers.ZYNCdB4H
VERBOSE2 SINGULARITY_DOCKER_USERNAME not defined (None)
VERBOSE2 SINGULARITY_DOCKER_PASSWORD found
DEBUG Starting Docker IMPORT, includes environment, runscript, and metadata.
VERBOSE Docker image: ubuntu:latest
VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
DEBUG Headers found: Content-Type,Accept
VERBOSE Registry: index.docker.io
VERBOSE Namespace: library
VERBOSE Repo Name: ubuntu
VERBOSE Repo Tag: latest
VERBOSE Version: None
VERBOSE Obtaining tags: https://index.docker.io/v2/library/ubuntu/tags/list
DEBUG GET https://index.docker.io/v2/library/ubuntu/tags/list
DEBUG Http Error with code 401
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/ubuntu:pull
DEBUG Headers found: Content-Type,Authorization,Accept
VERBOSE3 Response on obtaining token is None.
Docker image path: index.docker.io/library/ubuntu:latest
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/ubuntu/manifests/latest
DEBUG GET https://index.docker.io/v2/library/ubuntu/manifests/latest
ERROR Improperly formed manifest, layers or fsLayers must be present

I see no substantial deviation for the cases that work

[...]
VERBOSE Obtaining tags: https://index.docker.io/v2/library/centos/tags/list
DEBUG GET https://index.docker.io/v2/library/centos/tags/list
DEBUG Http Error with code 401
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/centos:pull
DEBUG Headers found: Content-Type,Authorization,Accept
VERBOSE3 Response on obtaining token is None.
Docker image path: index.docker.io/library/centos:latest
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/centos/manifests/latest
DEBUG GET https://index.docker.io/v2/library/centos/manifests/latest
DEBUG Image manifest version 2.2 found.
DEBUG Adding digest sha256:74f0853ba93b37c8152648905c48965c774d5a7d2de1967aef86ef0144561f62
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/centos/manifests/latest
DEBUG GET https://index.docker.io/v2/library/centos/manifests/latest
Cache folder set to /root/.singularity/docker
DEBUG Using 9 workers for multiprocess.
VERBOSE3 Found Docker command (Entrypoint) None
VERBOSE3 Found Docker command (Cmd) /bin/bash
VERBOSE3 Adding Docker CMD as Singularity runscript...
DEBUG /bin/bash
[...]

even the 401 error is present.

I can only suppose that something in the way dockerhub are delivering their images has changed that is partially incompatible with Singularity. I'd appreciate any help - I am just in the process of convincing our users that Singularity is the way to go with complex software installations - too bad if it seems not to work.

Thanks! Mike

[David Godlove]

Hey this is fixed (last night by Vanessa) in development.  We probably need to get it out to master too.  

Docker changed their manifest last night and any new images are now broken.  

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Mike]

Thanks! This is good news (I was already doubting my sanity :-) )

[vanessa s]

working on a PR now!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Michael Galloway]

Thanks Vanessa!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

--

[Martin Cuma]

Hi everyone,

can you please post an update on when will this change be done in the master branch?

Or, provide what I assume may be a simple source code change so we can patch our existing installation?

We are stuck now with container building as most of our containers rely on base Docker images for bootstrap.

Thanks,
MC

[vanessa s]

Hey Martin,

If you want to patch the existing, you can install development branch. Greg can probably give more info about releases into master, it's been a busy week!

Best,

Vanessa

--

[Martin Cuma]

Well, it's not that easy, and desirable, to install the development branch on a production system where some of us don't have root to install applications, and where we want stable applications.

It would be good to deploy critical patches like this one quickly to the master.

Thanks,
MC

[Kandes, Martin]

Hi Martin,

If it's any help, I have a small repo [1] of non-Docker-based definition files for bootstrapping some of the Singularity containers our users have recently requested.

Marty

[1]

https://github.com/mkandes/naked-singularity

---

From: vanessa s [vso...@gmail.com]
Sent: Friday, September 15, 2017 10:01 AM
To: singu...@lbl.gov
Subject: Re: [Singularity] Re: Singularity pull/import of some images from docker hub suddenly stopped working

[vanessa s]

Greg is about to announce the patch I think - hang in there!

--

You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[David Godlove]

https://github.com/singularityware/singularity/releases/tag/2.3.2

On Sep 15, 2017 1:46 PM, "vanessa s" <vso...@gmail.com> wrote:

Greg is about to announce the patch I think - hang in there!

On Fri, Sep 15, 2017 at 10:17 AM, Martin Cuma <mart...@gmail.com> wrote:

Well, it's not that easy, and desirable, to install the development branch on a production system where some of us don't have root to install applications, and where we want stable applications.

It would be good to deploy critical patches like this one quickly to the master.

Thanks,
MC

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[vanessa s]

And a bit of prose, background, for those interested!

http://singularity.lbl.gov/release-2-3-2

https://github.com/singularityware/singularity/releases/tag/2.3.2

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.