Problem with hosted yum repodata depth

1,400 views
Skip to first unread message

Sverre Moe

unread,
Feb 7, 2018, 4:03:06 AM2/7/18
to Nexus Users
Finally Nexus now supports hosted yum repositories.

Have tried this out, but am finding a problem with Repodata depth.

dept=2: /testingrepo/master/opensuse42.3
dept=3: /testingrepo/user/work/opensuse42.3
dept=4: /testingrepo/user/featureA/work/opensuse42.3

Minimum depth is 2 for any Yum repository we have, but if I set to this value then the repo will be created in wrong place if I place the RPMS at a lower depth.
/testingrepo/master/opensuse42.3 CORRECT
/testingrepo/user/work/ WRONG, should be at /testingrepo/user/work/opensuse42.3
/testingrepo/user/featureA/ WRONG, should be at /testingrepo/user/featureA/work/opensuse42.3

The only way I see to avoid this would be to create several distinct hosted yum repositories for each BRANCH/OS, 
However then we will have a problem when a new branch is created, or a new OS added. I would have to manually create the nexus hosted yum for it before Jenkins can start building and publish the artifacts there. If I could automatically create a Yum repository if it does not exist through remote API in my Jenkins build pipeline script, then it could work.

Though we have hundreds of Yum repositories. For branch master we have 6 OS repositories, and there a lot of branches. It will be a massive list on Nexus.

Joseph Stephens

unread,
Feb 7, 2018, 7:05:49 AM2/7/18
to Sverre Moe, Nexus Users
Hi Sverre,

You are correct in that the Yum hosted implementation is not designed to generate metadata at multiple depths. I think to solve it you would only need distinct repositories per depth? i.e. in the above example a depth 2, a depth 3 and a depth 4 repository which would essentially be repository-master, repository-work, repository-feature. I know that changes the client setup but that will become easier once Yum Group is available.

Another workaround would be to artificially pad the repositories to all be at depth 4, i.e. make /testingrepo/master/opensuse42.3 into /testingrepo/1/2/master/opensuse42.3.

Let us know how you get on,

Joe

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/83e76d2c-fe19-4ee4-a306-3bbe41ee3bb7%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.



--
Joseph Stephens

Sverre Moe

unread,
Feb 14, 2018, 9:47:27 AM2/14/18
to Nexus Users, sverr...@gmail.com
For the time being we cannot move to use Nexus as our RPM repository manager.
This feature is still new and will hopefully improve in later releases.

Repository metadata at multiple depths is necessary. Otherwise it would be unmanageable.
For the workaround with padding: Cannot expect our users and systems engineers to know that they have to pad the repository URL.

Our builds of user branches in Jenkins are published to a testing repository. We have no control over the depth here on these branches.
The first time build, the repository does not exist. We thus create the folders necessary based on the branch, then transfer the files and run createrepo.
branch user/work => /testingrepo/user/work/opensuse42.3/
branch user/featureA/work => /testingrepo/user/featureA/work/opensuse42.3/

Both of these are each a separate RPM repository. The repository list in Nexus will become enormous and thus become unmanageable. Perhaps if this repositories list in Nexus could be filtered on type.

Anyway, the problem is more complicated than this. We could live with multiple Yum repositories, but if we have multiple repositories in Nexus with different depths we would need to create a new repository for each new branch we build automatically from Jenkins build pipeline script. As far as I know there is no way to do this outside of Nexus UI.

Also we need to archive old/previous releases in repository. I cannot find any mechanism to allow this other than to manually delete files in Nexus UI.
If we cannot archive then the repository might grow extremely large.


onsdag 7. februar 2018 13.05.49 UTC+1 skrev Joseph Stephens følgende:
Hi Sverre,

You are correct in that the Yum hosted implementation is not designed to generate metadata at multiple depths. I think to solve it you would only need distinct repositories per depth? i.e. in the above example a depth 2, a depth 3 and a depth 4 repository which would essentially be repository-master, repository-work, repository-feature. I know that changes the client setup but that will become easier once Yum Group is available.

Another workaround would be to artificially pad the repositories to all be at depth 4, i.e. make /testingrepo/master/opensuse42.3 into /testingrepo/1/2/master/opensuse42.3.

Let us know how you get on,

Joe
On Wed, Feb 7, 2018 at 5:03 AM, Sverre Moe <sverr...@gmail.com> wrote:
Finally Nexus now supports hosted yum repositories.

Have tried this out, but am finding a problem with Repodata depth.

dept=2: /testingrepo/master/opensuse42.3
dept=3: /testingrepo/user/work/opensuse42.3
dept=4: /testingrepo/user/featureA/work/opensuse42.3

Minimum depth is 2 for any Yum repository we have, but if I set to this value then the repo will be created in wrong place if I place the RPMS at a lower depth.
/testingrepo/master/opensuse42.3 CORRECT
/testingrepo/user/work/ WRONG, should be at /testingrepo/user/work/opensuse42.3
/testingrepo/user/featureA/ WRONG, should be at /testingrepo/user/featureA/work/opensuse42.3

The only way I see to avoid this would be to create several distinct hosted yum repositories for each BRANCH/OS, 
However then we will have a problem when a new branch is created, or a new OS added. I would have to manually create the nexus hosted yum for it before Jenkins can start building and publish the artifacts there. If I could automatically create a Yum repository if it does not exist through remote API in my Jenkins build pipeline script, then it could work.

Though we have hundreds of Yum repositories. For branch master we have 6 OS repositories, and there a lot of branches. It will be a massive list on Nexus.

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.



--
Joseph Stephens

Joseph Stephens

unread,
Feb 14, 2018, 10:21:16 AM2/14/18
to Sverre Moe, Nexus Users
(Accidentally mailed directly to Sverre, sending again to include the user list)

I'm unsure why you would need a new repository per branch, I think I'm missing part of the puzzle and I'd like to better understand your problem to see how we can improve NXRM. 

I've created a NXRM setup to demonstrate how I'm imagining your repository structure.

I've created three repositories: 

testingrepo-branch at depth 2

Inline image 3

testingrepo-work at depth 3

Inline image 2


testingrepo-feature at depth 4

Inline image 4
Testingrepo-feature, for example, could contain combinations of different users and branches and each would have their corresponding repodata generated. Is the problem here that you need the work RPMs and work/feature RPMs to live in the same NXRM repository?

Thanks for the feedback,

Joe

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/50dd3c52-71ce-4d31-8fc9-d2cbd9b56e3e%40glists.sonatype.com.



--
Joseph Stephens

Sverre Moe

unread,
Feb 14, 2018, 12:52:04 PM2/14/18
to Nexus Users, sverr...@gmail.com
I was way off with my original thinking.
Won't need a Nexus Yum repository for each branch. Your illustration was good and clarified a bit. It is not so much problem as I originally thought.

Would "only" need to create 3-4 repositories in Nexus. each with different depth.
Though this is only necessary in the testingrepo for continuous builds, as the releaserepo would only be at depth 2 always.

For user work branches, there is no limit actually for depth. To put to the extreme with user/path/to/feature/branch/A. It would never be an issue, but the point is its the users own choice what branch structure he sets up.

Example releaserepo:

Example: testingrepo

In my Jenkins pipeline build script I would need to check the depth from branch name to know which Nexus Yum Repository to move the file into.

Repository URL:
One downside though: The URL to repository.
This is what we are using today as our repository URL:

Would be much simpler if we could continue to use this URL prefix or something close to it.
The user should not need to know that he has to use either testingrepo-depth2 or testingrepo-dept3. That is Nexus configuration logic the user should not have to know about.

The user thinks: I have two levels on my work branch, then I need to point to the testingrepo-depth3 repository.

Third Party inclusion:
We also have third party RPMs, no repository, only raw file storage which is symlinked into each repository.
All repositories build on opensuse42.3 have this symlinked. To copy these to each repository whould be a waste of space and a tedious job each time any third party packages are added/updated.

Perhaps Yum Repository Group to combine two repositories
Not sure how as the documentation is not quite specific:
I have not fiddled with comps.xml before. Need to read up about it.

Archiving:
Archiving old RPMs when transferring new RPMs to repository. If there was possible to move files out of the Nexus repository and into an archive raw repository.


/Sverre



--
Joseph Stephens

Sverre Moe

unread,
Feb 15, 2018, 5:33:28 AM2/15/18
to Nexus Users, sverr...@gmail.com
Release yum repository in Nexus


Testing yum repository depth 2


Testing yum repository depth 3



I don't quite like that users need to know which depth testing repository they need to use.


Joseph Stephens

unread,
Feb 15, 2018, 6:30:12 AM2/15/18
to Sverre Moe, Nexus Users
Hi Sverre,

Perhaps Yum Repository Group to combine two repositories
Not sure how as the documentation is not quite specific:
I have not fiddled with comps.xml before. Need to read up about it.

This will not solve your problem. Comps.xml allows you to create "sets" of RPMs that you can install with a single command (e.g. yum groupinstall all-rpms-required-for-development).  It's unfortunate that yum uses the term group which clashes with NXRMs definition of group.

Archiving:
Archiving old RPMs when transferring new RPMs to repository. If there was possible to move files out of the Nexus repository and into an archive raw repository.

There is a PRO-only feature in development that would allow you to achieve this once it is released. I'm sure someone else would be willing to talk to you about this further if you are interested.

I don't quite like that users need to know which depth testing repository they need to use.

This problem will be partially solved, once Yum Group is released. It will allow reads to be done from a single repository and will look like the screenshot below

Inline image 1
This still won't solve the problem of your users having to know where to upload their RPMs and sadly I don't have a solution for that yet. Feel free to file an improvement ticket at https://issues.sonatype.org which will allow us to track interest.

Thanks,

Joe


--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/12446f3a-2022-4f77-9842-3e80f3438d62%40glists.sonatype.com.



--
Joseph Stephens

Sverre Moe

unread,
Feb 15, 2018, 6:41:03 AM2/15/18
to Nexus Users, sverr...@gmail.com
torsdag 15. februar 2018 12.30.12 UTC+1 skrev Joseph Stephens følgende:
Hi Sverre,

Perhaps Yum Repository Group to combine two repositories
Not sure how as the documentation is not quite specific:
I have not fiddled with comps.xml before. Need to read up about it.

This will not solve your problem. Comps.xml allows you to create "sets" of RPMs that you can install with a single command (e.g. yum groupinstall all-rpms-required-for-development).  It's unfortunate that yum uses the term group which clashes with NXRMs definition of group.

That's too bad. Only solution then would be to copy the third party RPMs into all release repositories.

Archiving:
Archiving old RPMs when transferring new RPMs to repository. If there was possible to move files out of the Nexus repository and into an archive raw repository.

There is a PRO-only feature in development that would allow you to achieve this once it is released. I'm sure someone else would be willing to talk to you about this further if you are interested.
 
Will this come in OSS later?
 
I don't quite like that users need to know which depth testing repository they need to use.

This problem will be partially solved, once Yum Group is released. It will allow reads to be done from a single repository and will look like the screenshot below
That is good. 
 
This still won't solve the problem of your users having to know where to upload their RPMs and sadly I don't have a solution for that yet. Feel free to file an improvement ticket at https://issues.sonatype.org which will allow us to track interest.
 
Since no users will upload directly it can be up to the build pipeline script to know which repository to upload into.

However, a yum repository without a depth configuration would solve this and the group problem.
If there could be a way to tell Nexus where the repodata should be generated. Perhaps by uploading a file nexus.conf.


/Sverre

Sverre Moe

unread,
Feb 15, 2018, 6:48:22 AM2/15/18
to Nexus Users, sverr...@gmail.com
torsdag 15. februar 2018 12.41.03 UTC+1 skrev Sverre Moe følgende:
torsdag 15. februar 2018 12.30.12 UTC+1 skrev Joseph Stephens følgende:
Hi Sverre,

Perhaps Yum Repository Group to combine two repositories
Not sure how as the documentation is not quite specific:
I have not fiddled with comps.xml before. Need to read up about it.

This will not solve your problem. Comps.xml allows you to create "sets" of RPMs that you can install with a single command (e.g. yum groupinstall all-rpms-required-for-development).  It's unfortunate that yum uses the term group which clashes with NXRMs definition of group.

That's too bad. Only solution then would be to copy the third party RPMs into all release repositories.
 
This could also be solved with Yum Group. Is it planned for Nexus 3.9.0 ?

Joseph Stephens

unread,
Feb 15, 2018, 6:55:17 AM2/15/18
to Sverre Moe, Nexus Users
This could also be solved with Yum Group. Is it planned for Nexus 3.9.0 

That's correct, once Yum Group is released you could create a third-party repository and group that together with the releases repository. It won't be in 3.9.0 and I don't have an exact date for you but we are making good progress.

Will this come in OSS later?

There are no plans to do that. There is also some work in the pipeline for tasks that will automatically "clean up" repositories but that will mean a "delete" rather than a "move and archive".

However, a yum repository without a depth configuration would solve this and the group problem.
If there could be a way to tell Nexus where the repodata should be generated. Perhaps by uploading a file nexus.conf.

I think Yum Group + the discussion above solves your problems, but if you think it would be helpful, I'd suggest filing this as an "Improvement" ticket.


--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/266eee18-a377-479a-a344-178a3a1bece8%40glists.sonatype.com.



--
Joseph Stephens

Sverre Moe

unread,
Feb 15, 2018, 7:05:22 AM2/15/18
to Nexus Users, sverr...@gmail.com
torsdag 15. februar 2018 12.55.17 UTC+1 skrev Joseph Stephens følgende:
This could also be solved with Yum Group. Is it planned for Nexus 3.9.0 

That's correct, once Yum Group is released you could create a third-party repository and group that together with the releases repository. It won't be in 3.9.0 and I don't have an exact date for you but we are making good progress.

Though, not quite sure how that will work.

Grouping the following nexus repositories:

This would be the URL to use on server adding the RPM repository.

How will grouping work. The third party repository might not have the branches structure, maybe just only distribution.

So the following RPM repository:
Would need to group

Or would it only work if the third party repository is of same structure as release repository?


/Sverre
Reply all
Reply to author
Forward
0 new messages