Rebuild Metadata Job Taking Very Long Time (Nexus 3)
668 views
Skip to first unread message
jwa
Feb 26, 2018, 10:56:17 AM2/26/18
to Nexus Users
Hello,
I'm looking for some advice on my Nexus 3 installation. I'm finding certain operations are taking an exceptionally large amount of time.
For example, I have a weekly task to re-build the metadata for my artifacts, and this has been running for close to 36-hours. I have taken a thread-dump of the JVM which I have included below.
Looking at my sonatype-work directory the size of the `blobs` directory is 266GB and the `db` directory is 23GB; in case this hints at a high load.
Has anyone seen similar? It's getting to the point where I am considering ditching Nexus 3 and going back to Nexus 2.
-----
"quartz-3-thread-15 <command>sql.select from component where (group = :groupId and name = :artifactId and attributes.maven2.baseVersion = :baseVersion) and (bucket=#9:15)</command>" #299 prio=5 os_prio=0
tid=0x00007f5e240b5800 nid=0x333b runnable [0x00007f5ed8de6000]
java.lang.Thread.State: RUNNABLE
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.getClass(OSchemaShared.java:597)
at com.orientechnologies.orient.core.metadata.schema.OSchemaProxy.getClass(OSchemaProxy.java:168)
at com.orientechnologies.orient.core.record.impl.ODocument.getSchemaClass(ODocument.java:1979)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterItemField.getBinaryField(OSQLFilterItemField.java:152)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:373)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:384)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:384)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:97)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:384)
at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88)
at com.orientechnologies.orient.core.sql.filter.OSQLFilter.evaluate(OSQLFilter.java:105)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.evaluateRecord(OCommandExecutorSQLResultsetAbstract.java:422)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.filter(OCommandExecutorSQLResultsetAbstract.java:412)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:609)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:211)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3417)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3335)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:178)
at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:431)
at sun.reflect.GeneratedMethodAccessor796.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
at com.sun.proxy.$Proxy200.findComponents(Unknown Source)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.lambda$2(MetadataRebuilder.java:341)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker$$Lambda$433/1237393872.call(Unknown Source)
at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadataInner(MetadataRebuilder.java:340)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadata(MetadataRebuilder.java:307)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:115)
at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.rebuildMetadata(MavenHostedFacetImpl.java:109)
at org.sonatype.nexus.repository.maven.tasks.RebuildMaven2MetadataTask.execute(RebuildMaven2MetadataTask.java:60)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:73)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:93)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Rich Seddon
Feb 26, 2018, 11:05:53 AM2/26/18
to Nexus Users
What is your version? If it is prior to 3.4.0, you should upgrade:
Otherwise, please file a new issue in the "dev - nexus" project at https://issues.sonatype.org.
Peter Lynch
Feb 26, 2018, 11:06:27 AM2/26/18
to jwa, Nexus Users
On Mon, Feb 26, 2018 at 11:56 AM, jwa <jamie...@gmail.com> wrote:
Hello,I'm looking for some advice on my Nexus 3 installation. I'm finding certain operations are taking an exceptionally large amount of time.For example, I have a weekly task to re-build the metadata for my artifacts, and this has been running for close to 36-hours.
Why are you running this weekly? That task should only be run in special circumstances as a repair operation. If metadata is consistently wrong, requiring you to run this task, it implies files are not being deployed correctly in the first place.
--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/792b28d0-2947-4f17-9b1b-42e4507bbf46%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.
jwa
Feb 26, 2018, 11:40:18 AM2/26/18
to Nexus Users
Thanks Rich, I'll raise this; using the latest stable build (3.8.0).
jwa
Feb 26, 2018, 11:43:46 AM2/26/18
to Nexus Users, jamie...@gmail.com
Peter,
Upon occasion (once every ~3-4 weeks) we are seeing maven-metadata.xml files that are missing a single version that has been published. Re-running the metdata builder fixes it.
Unfortunately there are no exceptions or similar in the logs when this happens, so we cannot pin down the root cause. Given Nexus is a key part of our pipeline into production this is a serious cause for concern... hence considering ditching Nexus 3 and moving back to the stability of Nexus 2.
Jamie.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.
Peter Lynch
Feb 26, 2018, 1:52:49 PM2/26/18
to jwa, Nexus Users
On Mon, Feb 26, 2018 at 12:43 PM, jwa <jamie...@gmail.com> wrote:
Peter,Upon occasion (once every ~3-4 weeks) we are seeing maven-metadata.xml files that are missing a single version that has been published. Re-running the metdata builder fixes it.Unfortunately there are no exceptions or similar in the logs when this happens, so we cannot pin down the root cause. Given Nexus is a key part of our pipeline into production this is a serious cause for concern... hence considering ditching Nexus 3 and moving back to the stability of Nexus 2.
There are Apache Maven bugs and non Apache Maven clients that try to do mimic Maven deploys that do not write maven-metadata.xml correctly. It is the client responsibility to update the metadata files for Maven correctly. I suggest examining what user-agents/clients are updating metadata or uploading the artifacts in question using the user-agent header values in the Nexus request.log.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/0269852a-4891-4e83-81f3-bae7a52f774d%40glists.sonatype.com.
Reply all
Reply to author
Forward
0 new messages