Nexus 3 Yum Artifact Metadata regeneration -- Way too slow

276 views

Skip to first unread message

mme...@apixio.com

unread,

Mar 14, 2018, 6:02:55 PM3/14/18

to Nexus Users

We recently migrated completely from nexus2 to nexus3, now that it has full yum support (hosted). Recently, we noticed that when we were doing our rpm builds/uploads, that when we would deploy new instances, we were not seeing the expected artifacts on the machines. I dug into this a bit and observed the following:

[root@pkgsvcs-cdb3ca log]# grep 'event-10-thread-499' nexus.log
2018-03-13 00:11:53,744+0000 INFO [event-10-thread-499] jenkins org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoFacetImpl - Rebuilding yum metadata for repository snapshot-rpms
2018-03-13 00:11:53,744+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> ACTIVE
2018-03-13 00:11:53,744+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: ACTIVE
2018-03-13 00:11:53,745+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.AssetEntityAdapter - Counting assets with query: select count(*) from asset where (attributes.yum.asset_kind = :p0) and (bucket=#11:56), parameters: {p0=RPM}
2018-03-13 00:26:49,108+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.AssetEntityAdapter - Finding assets with query: select from asset where (attributes.yum.asset_kind = :p0) and (bucket=#11:56), parameters: {p0=COMPS}
2018-03-13 00:26:49,149+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.AssetEntityAdapter - Finding assets with query: select from asset where (attributes.yum.asset_kind = :p0) and (bucket=#11:56), parameters: {p0=REPODATA}
2018-03-13 00:26:49,161+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support detects repodata/b5b377a566e923835e71bde8e8c2a3c9dc39279bb4d49f2d1b5bbec48350a760-filelists.xml.gz as [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,161+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support assumes repodata/b5b377a566e923835e71bde8e8c2a3c9dc39279bb4d49f2d1b5bbec48350a760-filelists.xml.gz as [application/x-gzip, application/gzip, application/x-tgz, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,161+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - content/name types intersection [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,161+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Content repodata/b5b377a566e923835e71bde8e8c2a3c9dc39279bb4d49f2d1b5bbec48350a760-filelists.xml.gz declared as application/x-gzip, determined as application/x-gzip
2018-03-13 00:26:49,165+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support detects repodata/f6a3101a65ee327d76cfcd4a34a6343e7d9c7ff144a089f27b5b89b1b8dee5b5-primary.xml.gz as [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,165+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support assumes repodata/f6a3101a65ee327d76cfcd4a34a6343e7d9c7ff144a089f27b5b89b1b8dee5b5-primary.xml.gz as [application/x-gzip, application/gzip, application/x-tgz, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,165+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - content/name types intersection [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,165+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Content repodata/f6a3101a65ee327d76cfcd4a34a6343e7d9c7ff144a089f27b5b89b1b8dee5b5-primary.xml.gz declared as application/x-gzip, determined as application/x-gzip
2018-03-13 00:26:49,179+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support detects repodata/002e077e897e65b99b47e6705e1144b0b8e086dc4034308343b3b3fbc7f9eabd-other.xml.gz as [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,179+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support assumes repodata/002e077e897e65b99b47e6705e1144b0b8e086dc4034308343b3b3fbc7f9eabd-other.xml.gz as [application/x-gzip, application/gzip, application/x-tgz, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,179+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - content/name types intersection [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 00:26:49,179+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Content repodata/002e077e897e65b99b47e6705e1144b0b8e086dc4034308343b3b3fbc7f9eabd-other.xml.gz declared as application/x-gzip, determined as application/x-gzip
2018-03-13 00:26:49,196+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support detects repodata/repomd.xml as [application/xml, application/x-xml, text/xml]
2018-03-13 00:26:49,196+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support assumes repodata/repomd.xml as [application/xml, text/xml, application/x-xml]
2018-03-13 00:26:49,196+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - content/name types intersection [application/xml, application/x-xml, text/xml]
2018-03-13 00:26:49,196+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Content repodata/repomd.xml declared as application/xml, determined as application/xml
2018-03-13 00:26:49,198+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: ACTIVE -> OPEN
2018-03-13 00:26:49,236+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: OPEN
2018-03-13 00:26:49,236+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> CLOSED
2018-03-13 00:26:49,236+0000 DEBUG [event-10-thread-499] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: CLOSED

==========
[root@pkgsvcs-cdb3ca log]# grep 'event-10-thread-533' nexus.log
2018-03-13 01:21:44,924+0000 INFO [event-10-thread-533] jenkins org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoFacetImpl - Rebuilding yum metadata for repository snapshot-rpms
2018-03-13 01:21:44,924+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> ACTIVE
2018-03-13 01:21:44,924+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: ACTIVE
2018-03-13 01:21:44,925+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.AssetEntityAdapter - Counting assets with query: select count(*) from asset where (attributes.yum.asset_kind = :p0) and (bucket=#11:56), parameters: {p0=RPM}
2018-03-13 01:36:44,021+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.AssetEntityAdapter - Finding assets with query: select from asset where (attributes.yum.asset_kind = :p0) and (bucket=#11:56), parameters: {p0=COMPS}
2018-03-13 01:36:44,046+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.AssetEntityAdapter - Finding assets with query: select from asset where (attributes.yum.asset_kind = :p0) and (bucket=#11:56), parameters: {p0=REPODATA}
2018-03-13 01:36:44,051+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support detects repodata/e31c8020129321350fd322fb4c3d6ad0f6beca0bc540fb98921421483028170f-filelists.xml.gz as [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,051+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support assumes repodata/e31c8020129321350fd322fb4c3d6ad0f6beca0bc540fb98921421483028170f-filelists.xml.gz as [application/x-gzip, application/gzip, application/x-tgz, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,051+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - content/name types intersection [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,051+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Content repodata/e31c8020129321350fd322fb4c3d6ad0f6beca0bc540fb98921421483028170f-filelists.xml.gz declared as application/x-gzip, determined as application/x-gzip
2018-03-13 01:36:44,060+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support detects repodata/158a6f7ebfee803454689bca7aa05786ab92ef9843e5dae79e4d4e8d2b60c99e-primary.xml.gz as [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,060+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support assumes repodata/158a6f7ebfee803454689bca7aa05786ab92ef9843e5dae79e4d4e8d2b60c99e-primary.xml.gz as [application/x-gzip, application/gzip, application/x-tgz, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,060+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - content/name types intersection [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,060+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Content repodata/158a6f7ebfee803454689bca7aa05786ab92ef9843e5dae79e4d4e8d2b60c99e-primary.xml.gz declared as application/x-gzip, determined as application/x-gzip
2018-03-13 01:36:44,065+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support detects repodata/3de6dcd91fa36eed84cae64422afb99996ecda0f6c97dcd0c612d97d26f15b3e-other.xml.gz as [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,065+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support assumes repodata/3de6dcd91fa36eed84cae64422afb99996ecda0f6c97dcd0c612d97d26f15b3e-other.xml.gz as [application/x-gzip, application/gzip, application/x-tgz, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,065+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - content/name types intersection [application/gzip, application/gzip-compressed, application/gzipped, application/x-gunzip, application/x-gzip, application/x-gzip-compressed, gzip/document]
2018-03-13 01:36:44,065+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Content repodata/3de6dcd91fa36eed84cae64422afb99996ecda0f6c97dcd0c612d97d26f15b3e-other.xml.gz declared as application/x-gzip, determined as application/x-gzip
2018-03-13 01:36:44,069+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support detects repodata/repomd.xml as [application/xml, application/x-xml, text/xml]
2018-03-13 01:36:44,069+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Mime support assumes repodata/repomd.xml as [application/xml, text/xml, application/x-xml]
2018-03-13 01:36:44,069+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - content/name types intersection [application/xml, application/x-xml, text/xml]
2018-03-13 01:36:44,069+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.DefaultContentValidator - Content repodata/repomd.xml declared as application/xml, determined as application/xml
2018-03-13 01:36:44,074+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: ACTIVE -> OPEN
2018-03-13 01:36:44,100+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: OPEN
2018-03-13 01:36:44,100+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioning: OPEN -> CLOSED
2018-03-13 01:36:44,100+0000 DEBUG [event-10-thread-533] jenkins org.sonatype.nexus.repository.storage.StorageTxImpl - Transitioned: CLOSED

=========
In both of these cases, regenerating the metadata is taking 15 minutes. That is a major detriment and will cause inefficiencies for developers, slowing down a company's ability to handle CI/CD -- Has anyone else observed this? The only speculation I have is that I did not separate repos into multiple blob stores, so it has to traverse the full blob store to build the index data, which results in the slow timeframe. Curious what others have seen and have done to resolve this issue (if they have experienced it).

Rich Seddon

unread,

Mar 14, 2018, 6:19:01 PM3/14/18

to Nexus Users

Please file an issue for this in the "Dev - Nexus" project at https://issues.sonatypelorg so we can investigate.

Marc Mercer

unread,

Mar 14, 2018, 6:55:42 PM3/14/18

to Rich Seddon, Nexus Users

Ticket has been opened as directed:

https://issues.sonatype.org/browse/NEXUS-16545

I believe I had mixed up the other 'portal' with the dev portal, my apologies.

Marc Mercer

DevOps Architect

m: (408) 470-9256

e: mme...@apixio.com

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/8dc96280-43e2-4195-a593-c262b9408f03%40glists.sonatype.com.

For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

mme...@apixio.com

unread,

Mar 20, 2018, 6:07:02 PM3/20/18

to Nexus Users, rse...@sonatype.com

Unfortunately, I am going to have to find a way of reverting us to nexus 2.0 for now, because this major lag has literally broken our deployment process. Based on the lack of interaction I have seen on the dev portal, I have no way of knowing what to expect -- especially since we are consumers of the free version. Given that thought, I would hope or expect that we are not the only ones experiencing this, and that maybe if enough of us talk about the experience, it will get escalated and we will have some level of transparency and expectation to work with.

So, if anyone else has had similar issues with V3 since migrating the rpm repositories, please chime in. If you haven't, can you give a general idea how long your metadata content regeneration has taken? For us, it only gets worse if we do multiple rpm uploads because then it triggers repeated slow regenerations and backfills our deployment strategy.

Thanks,
- Marc

Reply all

Reply to author

Forward

0 new messages