nexus 3.2.0-01 - Purge unused docker manifests and images - invalid manifest

[Morten Bjoernsvik]

Hi

When I try to run this task I get "invalid Manifest error"

There aren't many options to fill in other than docker registry,interval and email address, so not much to go wrong.

The only thing not default is my blob directory which is softlinked, But deleting separate manifests/images in the same docker registry work fine.
OS: Ubuntu 14.04LTS

Any help/hacks very appreciated

Thanks
--
Morten Bjoernsvik, Oslo, Norway

2017-01-03 20:22:30,523+0100 WARN  [quartz-2-thread-3] *TASK org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 1174f60a-2c44-46d6-9bf6-fe80e53f7dac : 'purge docker local' [repository.docker.gc] execution failure
org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task 'Purge unused docker manifests and images'; 1 failure
        at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95) [org.sonatype.goodies.common:2.2.3]
        at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:64) [na:na]
        at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:89) [org.sonatype.nexus.scheduling:3.2.0.01]
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:144) [org.sonatype.nexus.quartz:3.2.0.01]
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:105) [org.sonatype.nexus.quartz:3.2.0.01]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.quartz-scheduler.quartz:2.2.2]
        at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) [org.sonatype.nexus.thread:3.2.0.01]
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [org.apache.shiro.core:1.3.2]
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [org.apache.shiro.core:1.3.2]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_74]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_74]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_74]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
        Suppressed: org.sonatype.nexus.repository.docker.internal.V2Exception$ManifestInvalid: manifest invalid
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [na:1.8.0_74]
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [na:1.8.0_74]
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [na:1.8.0_74]
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [na:1.8.0_74]
                at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83) [na:na]
                at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77) [na:na]
                at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84) [na:na]
                at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247) [na:na]
                at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl.readManifest(V2ManifestUtilImpl.groovy:106) [na:na]
                at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.handleV2Assets(DockerGCFacetImpl.java:155) [na:na]
                at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.processRepository(DockerGCFacetImpl.java:90) [na:na]
                at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) [org.sonatype.nexus.transaction:3.2.0.01]
                at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54) [org.sonatype.nexus.transaction:3.2.0.01]
                at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.deleteUnusedManifestsAndImages(DockerGCFacetImpl.java:74) [na:na]
                at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [na:na]
                at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) [na:na]
                at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) [na:na]
                at org.sonatype.nexus.repository.docker.internal.DockerGCTask.execute(DockerGCTask.java:42) [na:na]
                at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:56) [na:na]
                ... 12 common frames omitted
2017-01-03 20:22:30,526+0100 INFO  [quartz-2-thread-3] *TASK org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'purge docker local' [repository.docker.gc] state change RUNNING -> WAITING (FAILED)

[Peter Lynch]

We are going to need a support zip to understand what is wrong in the manifest. Please file an issue at https://issues.sonatype.org under the NEXUS project and attach a support zip generated immediately following reproducing the issue. By default, the security on the issue will only be accessible by Sonatype and yourself.

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/1fb30dd3-40cc-47c7-8f25-d5750aa00f62%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Peter Lynch]

On Tuesday, January 3, 2017 at 5:01:41 PM UTC-4, Peter Lynch wrote:

We are going to need a support zip to understand what is wrong in the manifest. Please file an issue at https://issues.sonatype.org under the NEXUS project and attach a support zip generated immediately following reproducing the issue. By default, the security on the issue will only be accessible by Sonatype and yourself.

To see which manifest is causing the problem, set the logger org.sonatype.nexus.repository.docker to level DEBUG and repeat the docker purge task. Then provide a support zip in a Jira issue. Thanks!

[Alyssa H]

I have the same issue. I found the manifest that it's having problems with after setting debug on as suggested, but I can't tell where it comes from. It is listed as v2/-/blobs (but is type manifest).

2017-01-05 00:43:14,554+0000 INFO  [qtp131297425-23080] ahardy org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Purge unused docker manifests and images' [repository.docker.gc] runNow
2017-01-05 00:43:14,555+0000 INFO  [qtp131297425-23080] ahardy org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Purge unused docker manifests and images' [repository.docker.gc] state change WAITING -> RUNNING
2017-01-05 00:43:14,568+0000 INFO  [quartz-3-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection starting on repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=docker, name='registry-global'}
2017-01-05 00:43:14,699+0000 DEBUG [quartz-3-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl - Parsing manifest v2/-/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 of type unspecified
2017-01-05 00:43:14,701+0000 ERROR [quartz-3-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCTask - Failed to run task 'Purge unused docker manifests and images' on repository 'registry-global'
org.sonatype.nexus.repository.docker.internal.V2Exception$ManifestInvalid: manifest invalid

[Alyssa H]

I found that this sha is a common one in many of my images, so I pulled one of those images into a proxy nexus server. On that server, it lists the asset as a BLOB type instead of a manifest.

[Peter Lynch]

On Wed, Jan 4, 2017 at 9:26 PM, Alyssa H <alyssa....@gmail.com> wrote:

I found that this sha is a common one in many of my images, so I pulled one of those images into a proxy nexus server. On that server, it lists the asset as a BLOB type instead of a manifest.

You could find that asset in the asset browser (v2/-/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 is the full asset name) then you'd have a link to download the file.

Please create an issue at https://issues.sonatype.org in the NEXUS project and upload that file with a support zip.

 

On Wednesday, January 4, 2017 at 5:05:23 PM UTC-8, Alyssa H wrote:

I have the same issue. I found the manifest that it's having problems with after setting debug on as suggested, but I can't tell where it comes from. It is listed as v2/-/blobs (but is type manifest).

2017-01-05 00:43:14,554+0000 INFO  [qtp131297425-23080] ahardy org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Purge unused docker manifests and images' [repository.docker.gc] runNow
2017-01-05 00:43:14,555+0000 INFO  [qtp131297425-23080] ahardy org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Purge unused docker manifests and images' [repository.docker.gc] state change WAITING -> RUNNING
2017-01-05 00:43:14,568+0000 INFO  [quartz-3-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection starting on repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=docker, name='registry-global'}
2017-01-05 00:43:14,699+0000 DEBUG [quartz-3-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl - Parsing manifest v2/-/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 of type unspecified
2017-01-05 00:43:14,701+0000 ERROR [quartz-3-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCTask - Failed to run task 'Purge unused docker manifests and images' on repository 'registry-global'
org.sonatype.nexus.repository.docker.internal.V2Exception$ManifestInvalid: manifest invalid

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/c22924a9-9eef-41bf-87e5-0bc428028f00%40glists.sonatype.com.

[Dave H]

I am having this same issue - has anyone found a workaround?

[Peter Lynch]

On Thu, Jan 12, 2017 at 3:03 PM, Dave H <david....@gmail.com> wrote:

I am having this same issue - has anyone found a workaround?

The diagnosis is the same, find the problem asset or blob using the DEBUG log message and UI Browse, then file an issue at issues.sonatype.org in the NEXUS project with that file and a support zip attached - we will try to get the the bottom of it if filed in this way.

 

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/faf3b205-b2c0-4655-866d-c354f6e7639d%40glists.sonatype.com.

[Alyssa H]

I hadn't yet opened the Nexus for production use, so I deleted the repo and re-created it, re-pushing all of the images.

[Robert Contreras]

Was able to get past the 

Parsing manifest v2/-/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

error.

The blob is just an empty data file that the garbage collector has trouble with.

Anyway, what we did was DELETE that blob from the original artifact and then ran the Purge Unused manifests and images task again. The task completes without error, though you still need to run the Compact Blob Store task to clear up the disk space.

Example busybox container:

# Have to run this twice for some reason

curl -H 'Accept:application/json' -H 'Authorization:Basic base64-user:password' -X DELETE https://nexusserver.com/repository/name-of-repo/v2/busybox/manifests/sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912 -v

curl -H 'Accept:application/json' -H 'Authorization:Basic base64-user:password' -X DELETE https://mynexusserver.com/repository/name-of-repo/v2/busybox/manifests/sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912 -v

Run the Purge job and you'll get this error:

Parsing manifest v2/-/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 of type unspecified

Delete the empty data file blob:

curl -H 'Accept:application/json' -H 'Authorization:Basic base64-user:password' -X DELETE https://mynexusserver.com/repository/name-of-repo/v2/busybox/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -v

Ran the tasks again and have no more errors.

[Morten Bjoernsvik]

Hi I updated nexus-3.2.1-01, saw that it wanted a sonatype-work/nexus3/clean_cache directory, created it manually.
I then ran the 'purge docker local' and it ran without error:

2017-02-20 09:10:54,325+0100 INFO  [qtp2053206828-167] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'purge docker local' [repository.docker.gc] runNow
2017-02-20 09:10:54,326+0100 INFO  [qtp2053206828-167] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'purge docker local' [repository.docker.gc] state change WAITING -> RUNNING
2017-02-20 09:10:54,356+0100 INFO  [quartz-1-thread-3] *SYSTEM org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection starting on repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=docker, name='docker-local'}
2017-02-20 09:11:23,010+0100 INFO  [quartz-1-thread-3] *SYSTEM org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection completed on repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=docker, name='docker-local'}

[Peter Lynch]

On Mon, Feb 20, 2017 at 9:02 AM, Morten Bjoernsvik <morten.b...@gmail.com> wrote:

Hi I updated nexus-3.2.1-01, saw that it wanted a sonatype-work/nexus3/clean_cache directory, created it manually.

What does this mean? Did you mean clean_cache file ( not directory )? What made you think it wanted a clean_cache directory?

 

I then ran the 'purge docker local' and it ran without error:

2017-02-20 09:10:54,325+0100 INFO  [qtp2053206828-167] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'purge docker local' [repository.docker.gc] runNow
2017-02-20 09:10:54,326+0100 INFO  [qtp2053206828-167] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'purge docker local' [repository.docker.gc] state change WAITING -> RUNNING
2017-02-20 09:10:54,356+0100 INFO  [quartz-1-thread-3] *SYSTEM org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection starting on repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=docker, name='docker-local'}
2017-02-20 09:11:23,010+0100 INFO  [quartz-1-thread-3] *SYSTEM org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection completed on repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=docker, name='docker-local'}

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/bd12b32e-5164-46f0-92a2-1cff9eb3a778%40glists.sonatype.com.

[Dave H]

I can confirm this on our system as well.