We have recently migrated all our repositories from Nexus 2.14.11 (EC2 instance with EBS volume) to the most recent 3.14.0 version (Ubuntu18 based EC2 with Blobs store on S3).
A week later, we made a decision to move the instance to another subnet, therefore the content of sonatype-nexusfolder was copied to another instance without stopping the original one. The new instances started correctly and stopped right after without making a change.
Inspired by successful test, two days later, we tried to do the same again and finally migrate to another instance. However, this time we stopped original instance. After copying sonatype-instance content, neither new nor original one did not see any repositories. All our repositories are gone in Nexus UI.
There are numerous exceptions in logs of original nexus while stopping like:
2018-12-21 16:01:10,067+0000 ERROR [Thread-6] *SYSTEM com.sonatype.nexus.migration.repository.RepositoryPrepareStep - Failed transition: COMPLETED -> ABORTED
java.lang.RuntimeException: java.lang.RuntimeException: RESTEASY003940: Unable to instantiate MessageBodyReader
at org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(RegisterBuiltin.java:41)
and
2018-12-21 16:46:34,921+0000 ERROR [quartz-3-thread-2] *SYSTEM com.google.common.eventbus.EventBus.nexus.async - Could not dispatch event AssetDeletedEvent{metadata=Attac org.sonatype.nexus.common.stateguard.InvalidStateException: Invalid state: STOPPED; allowed: [STARTED] at org.sonatype.nexus.common.stateguard.StateGuard._ensure(StateGuard.java:115)
"Reconcile component database from blob store" task also fails with
2018-12-22 12:19:30,364+0000 INFO [qtp423295968-1512] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'reconciledatafromblob' [blobstore.rebuildComponentDB] runNow
2018-12-22 12:19:30,364+0000 INFO [qtp423295968-1512] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'reconciledatafromblob' [blobstore.rebuildComponentDB] state change WAITING -> RUNNING
2018-12-22 12:19:30,372+0000 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.blobstore.restore.RestoreMetadataTask - Task log: /opt/nexus/sonatype-work/nexus3/log/tasks/blobstore.rebuildComponentDB-20181222121930370.log
2018-12-22 12:19:30,372+0000 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.blobstore.restore.RestoreMetadataTask - ::DRY RUN:: Actions will be logged, but no changes will be made.
2018-12-22 12:19:30,639+0000 WARN [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access non-existent blob report.properties (S3PropertiesFile{bucket=nexus-repo, key=content/vol-28/chap-08/report-properties.properties})
.....
.....
2018-12-22 12:19:46,226+0000 WARN [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 3adca743-6dbf-4638-8ce5-56a921a7c71b (S3PropertiesFile{bucket=nexus-repo, key=content/vol-25/chap-16/3adca743-6dbf-4638-8ce5-56a921a7c71b.properties})
2018-12-22 12:19:46,226+0000 WARN [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.repository.storage.StorageTxImpl - Reached max retries: 8/8
2018-12-22 12:19:46,228+0000 WARN [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 4baf04dd-1b47-420a-b9ee-10586eb6e993 : 'reconciledatafromblob' [blobstore.rebuildComponentDB] execution failure
org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:226)
at sun.reflect.GeneratedMethodAccessor411.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
.....
Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob s3-nexus-repo@30B55E85-482EE75F-3AD6BEC6-D67AA14B-86EDA37A:3adca743-6dbf-4638-8ce5-56a921a7c71b exists in metadata, but is missing from the blobstore
at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:950)
at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source)
... 19 common frames omitted
2018-12-22 12:19:46,228+0000 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'reconciledatafromblob' [blobstore.rebuildComponentDB] state change RUNNING -> WAITING (FAILED)
Our build jobs are failing with (may be not related)
*UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /content/groups/public/com/ourproject/1.52-SNAPSHOT/ourproject-1.52-20181122.082121-9.pom
org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:226)
at sun.reflect.GeneratedMethodAccessor411.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
To conclude, Nexus did not survive a restart and our build pipelines are broken. Any inputs/help/ideas are highly appreciated.
I will gladly provide any missing information that can help.
BR,
Cyril
--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/8f94d312-8bee-498e-91b8-a3ce05267128%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.
aws s3 ls nexus-repo
PRE content/
2018-12-26 11:21:04 70 30B55E85-482EE75F-3AD6BEC6-D67AA14B-86EDA37A-metrics.properties
2018-12-13 10:56:08 40 metadata.propertiesWe have recently migrated all our repositories from Nexus 2.14.11 (EC2 instance with EBS volume) to the most recent 3.14.0 version (Ubuntu18 based EC2 with Blobs store on S3).
A week later, we made a decision to move the instance to another subnet, therefore the content of sonatype-nexusfolder was copied to another instance without stopping the original one. The new instances started correctly and stopped right after without making a change.
Inspired by successful test, two days later, we tried to do the same again and finally migrate to another instance. However, this time we stopped original instance. After copying sonatype-instance content, neither new nor original one did not see any repositories. All our repositories are gone in Nexus UI.
There are numerous exceptions in logs of original nexus while stopping like:
2018-12-21 16:01:10,067+0000 ERROR [Thread-6] *SYSTEM com.sonatype.nexus.migration.repository.RepositoryPrepareStep - Failed transition: COMPLETED -> ABORTED java.lang.RuntimeException: java.lang.RuntimeException: RESTEASY003940: Unable to instantiate MessageBodyReader at org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(RegisterBuiltin.java:41)
Thank you for the response.Yes, we are running Nexus OSS 3.14.0 version on Ubuntu18 based EC2 with Blobs store on S3.S3 is accessible from the instance and has about 326Gb of data.aws s3 ls nexus-repo
PRE content/
2018-12-26 11:21:04 70 30B55E85-482EE75F-3AD6BEC6-D67AA14B-86EDA37A-metrics.properties
2018-12-13 10:56:08 40 metadata.propertiesAs I mentioned above after migration from Nexus2 to Nexus3, the new service was fully functional about a week. Later we had planned to migrate to another instance in another subnet with the same Nexus version. Please, find attached a log file from the moment when I initiated stopping Nexus and many exceptions with RESTEASY003940 error.You may notice Disabling repository upgrade capability for 'apache-snapshots' at: http://localhost:8082/nexus/service/siesta/migrationagent in the log. This is our old Nexus2 instance with Upgrade capability enabled.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/657dbbd4-f12a-4c66-8616-9ad577e5c524%40glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/c2957ca4-e014-408c-aaba-e697805a96a0%40glists.sonatype.com.