Unable to pull docker images

[Shruti Shivakumar]

Hey,

After the docker maintenance update yesterday, I'm unable to pull images using singularity on my cluster. I'm using singularity 3.0, and the pulls fail with an exit status 1. Anyone else in the same boat? Also, is there any way I can get a more verbose error log? 

Any help is appreciated!

> singularity pull docker://godlovedc/lolcow

WARNING Authentication token file not found : Only pulls of public images will succeed

INFO:  Starting build...

Getting image source signatures

Skipping fetch of repeat blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118

Skipping fetch of repeat blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a

Skipping fetch of repeat blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2

Skipping fetch of repeat blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e

Skipping fetch of repeat blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9

Skipping fetch of repeat blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945

Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82

 0 B / 3.33 KiB [------------------------------------------]

 3.33 KiB / 3.33 KiB [==================================] 0s

Writing manifest to image destination

Storing signatures

INFO:   Creating SIF file...

FATAL:  Unable to pull docker://godlovedc/lolcow: exit status 1

Thanks

Shruti

[v]

Can you add --debug and some verbose flags to get more detail? I just tried (with same container and singularity version) and it worked okay for me, so it might be something about your environment (temporary storage, memory etc.). The debug will hopefully shed some more light on the exit status 1 so we know what's up!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Shruti Shivakumar]

Thanks for your quick response! Here's with the debug flags

WARNING [U=898858,P=25706] sylabsToken()                 Authentication token file not found : Only pulls of public images will succeed

DEBUG [U=898858,P=25706] NewBundle()                   Created temporary directory for bundle /tmp/sbuild-110230597

INFO [U=898858,P=25706] Full()                        Starting build...

DEBUG [U=898858,P=25706] updateCacheSubdir()           Caching directory set to /nethome/.singularity/cache/oci

Getting image source signatures

Skipping fetch of repeat blob sha256:9fb6c798fa41e509b58bccc5c29654c3ff4648b608f5daa67c1aab6a7d02c118

Skipping fetch of repeat blob sha256:3b61febd4aefe982e0cb9c696d415137384d1a01052b50a85aae46439e15e49a

Skipping fetch of repeat blob sha256:9d99b9777eb02b8943c0e72d7a7baec5c782f8fd976825c9d3fb48b3101aacc2

Skipping fetch of repeat blob sha256:d010c8cf75d7eb5d2504d5ffa0d19696e8d745a457dd8d28ec6dd41d3763617e

Skipping fetch of repeat blob sha256:7fac07fb303e0589b9c23e6f49d5dc1ff9d6f3c8c88cabe768b430bdb47f03a9

Skipping fetch of repeat blob sha256:8e860504ff1ee5dc7953672d128ce1e4aa4d8e3716eb39fe710b849c64b20945

Copying config sha256:73d5b1025fbfa138f2cacf45bbf3f61f7de891559fa25b28ab365c7d9c3cbd82

 0 B / 3.33 KiB [------------------------------------------]

 3.33 KiB / 3.33 KiB [==================================] 0s

Writing manifest to image destination

Storing signatures

DEBUG [U=898858,P=25706] Full()                        Inserting Metadata

DEBUG [U=898858,P=25706] Full()                        Calling assembler

INFO [U=898858,P=25706] Assemble()                    Creating SIF file...

FATAL [U=898858,P=25706] PullOciImage()                Unable to pull docker://godlovedc/lolcow: exit status 1

I tried clearing the user cache, but that does not solve the problem.

Thanks

Shruti

To unsubscribe from this group and stop receiving emails from it, send an email to singu...@lbl.gov.

[v]

I can't comment, but it looks like the function has issues, maybe the developer can comment if you leave a note here?

https://github.com/sylabs/singularity/issues/2943

(That's a super funny issue, btw! I like the visual of nuking something into oblivion) 

To trace (what I think is the call) the libexec pull is here, on error it reports that it can't pull the container (as in your log). It's hard to tell which error loop is being called because they print the same thing, but the first uses build from here. I would look into what is going on there and comparing with the messages that you see to try to understand what is going on (if one of the maintainers doesn't pipe in). It looks like we hit the Full() function given the first print line, and so I'd find where it is creating the SIF. This is a step that fails a lot for me because I just run out of memory for it to build, but could be something else.

I'm done for today but I can look more closely sometime this week when my brain is turned on again :)

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.