mergerepo and Nexus Professional staging repositories
210 views
Skip to first unread message
Alexander McPherson
Apr 23, 2012, 4:49:27 AM4/23/12
to Nexus Yum Plugin
Summary of discussion i have had to date with Sebastian
Sandy
On 4/20/12 5:05 PM, "Sandy McPherson" wrote:
The files are in the location that you have in your code.
I deleted the files by hand and re-ran mergerepo by hand, it still
does
not want to pick up multiple versions of the same RPM which are in
different repositories.
Your code is working fine for me, it is mergerepo that is letting me
down,
so I will press on with my script. All of the rest of the
functionality
provided by Nexus Pro and you plug-in fits our use case exactly, so
I'm
not going to let this beat meŠ Do you think I should raise a ticket
with
the yum guys?
Have a nice weekend
Sandy
-----------------------------------------------------------------------
On 4/20/12 3:41 PM, "Sebastian Herold"
Hi Sandy,
Mergerepo has a strange bug described here:
http://lists.baseurl.org/pipermail/rpm-metadata/2011-August/001370.html
It saves the repository information for a repository and generates the
same result over and over although you pass different parameters.
Therefore I implemented a method called cleanYumCacheDir, see:
http://code.google.com/p/nexus-yum-plugin/source/browse/trunk/src/main/jav
a/de/is24/nexus/yum/repository/task/
YumGroupRepositoryGenerationTask.java
which removes everything matching /var/tmp/yum-<user.name>-... .
Maybe,
this isn't working correctly in your case. Could you please find out,
under which location yum stores temporary file on your machine. Which
OS
are you using?
Regards,
Sebastian
--------------------------------------------------------------------------------------------------------------
Am 20.04.2012 um 11:06 schrieb Sandy McPherson:
Hi Sebastian,
I double checked that the plugin was at the version I built from
trunk.
The log is very short (320L) so I just included the complete file.
Looking through the logs and using some of the command manually I can
see that your plugin is doing everything that you would expect of it.
The
issue (or false interoperation my me) lies within mergerepo itself, as
it does not seem to handle merging different versions of the same RPM
from
different repos. The interesting part is where the staging repository
system-images-002
gets created by nexus.
I copied the mergrepo command out of the log and ran it manually. It
doesn't merge multiple versions of the same RPM from different
repositories, it seems to use the first in the list.
mergerepo --nogroups -d
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-001/
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-002/
-o /home/nexrep_d_nl_eng_nexs_01/work/storage/DEV
Loaded plugins: fastestmirror, presto, refresh-packagekit
Loading mirror speeds from cached hostfile
1/1 - corrie-masteruser-server-system-10.3547-20120419174847.noarch
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
Produces a single entry for my RPM which shows the version
10.3547-20120419174847 from 001.
Reversing the order of the repositories
mergerepo --nogroups -d
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-002/
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-001/
-o /home/nexrep_d_nl_eng_nexs_01/work/storage/DEV
Loaded plugins: fastestmirror, presto, refresh-packagekit
Loading mirror speeds from cached hostfile
Not using downloaded repomd.xml because it is older than what we have:
Current : Thu Apr 19 10:57:24 2012
Downloaded: Thu Apr 19 10:49:08 2012
1/1 - corrie-masteruser-server-system-10.3548-20120419175704.noarch
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
let's me see a single entry with the newer version
10.3548-20120419175704
from 002
Reversing the order again shows (strangely enough) still the newer RPM
mergerepo --nogroups -d
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-001/
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-002/
-o /home/nexrep_d_nl_eng_nexs_01/work/storage/DEV
Loaded plugins: fastestmirror, presto, refresh-packagekit
Loading mirror speeds from cached hostfile
Not using downloaded repomd.xml because it is older than what we have:
Current : Thu Apr 19 10:57:24 2012
Downloaded: Thu Apr 19 10:49:08 2012
1/1 - corrie-masteruser-server-system-10.3548-20120419175704.noarch
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
I am afraid I myself don't know who mergerepo is supposed to work with
multiple versions, however one of my colleagues is a genius with this
type of thing, so may be
-----------------------------------------------------------------------------
On 4/20/12 9:22 AM, "Sebastian Herold" wrote:
Hi Sandy,
please make sure your target group repository has maven2yum format. It
seems that mergerepo doesn't work well in your case. If this doesn't
help, please send me your nexus.xml, staging.xml and the part of your
nexus.log for the corresponding time period.
Good luck,
Sebastian
-----------------------------------------------------------------------------
Am 19.04.2012 um 18:07 schrieb Sandy McPherson:
Hi Sebastian,
Sorry to bother you again, but I have a further question/problem. I
have installed the new version of the plugin and when I deploy to
Nexus and close my staging repository I can see the RPM and install it
using YUM, the repodata directory is also generated in the storage
area of the staging profiles target group. I then deploy a newer
version which opens a new staging repository, but on closing this
second staging repository which gets added to the group correctly I
can only see the old version. I looked in the primary.xm.gz in the
group directory and it definitely only contains a single entry for the
old RPM.
Any ideas what I'm doing wrong?
-----------------------------------------------------------------------------
On 4/19/12 10:59 AM, "Sebastian Herold" wrote:
Hi Sandy,
in the trunk, the merging is already in place. Please, checkout the
trunk
and build the current snapshot or wait for the next release.
Regards,
Sebastian
-----------------------------------------------------------------------------
Am 19.04.2012 um 10:56 schrieb Sandy McPherson:
Hi Sebastian,
Thanks for the quick reply.
Just for clarification: we are using 2.0.2.1 of the plugin and 2.0.3
nexus-pro. If I use a Maven2Yum-group repository, is the merging
already in place? Or do we need to wait for your next release? If so
what
are your plans?
Thanks
Sandy
FYI I opened a ticket with Sonatype, because the maven2yum
repositoryTargets are not visible in the Webgui.
https://issues.sonatype.org/browse/SUPPORT-1515
There would also appear to be issues with the refreshing of some of
the GUI elements. I seem to have to logout and login again to gain
visibility of repositories I have created when configuring the system.
Don't know if you have visibility of this, but just to let you know.
----------------------------------------------------------------------------
On 4/19/12 10:23 AM, "Sebastian Herold"
<> wrote:
Hi Sandy,
your setup is similar to ours. Using staging repositories in group
repositories was a pain in the ass for a long time. I fixed the
problem in the trunk, but haven't released since then. The solution is
the newly introduced Maven2Yum-group repositories. It's possible to
assign these repositories as target repositories for staging profiles.
The Maven2Yum-groups merge the included repositories using mergerepo
(you need at least createrepo-0.9.1). But please use a Maven2yum-
repository as final repository, this avoids the copying of the
repodata/
directory from the staging repo to the final repository.
Best regards,
Sebastian
----------------------------------------------------------------------------
Am 18.04.2012 um 17:24 schrieb Sandy McPherson:
Hallo Sebastian,
Not sure if this is the correct way to contact you for questions
relating to the nexus yum plugin, but here goes. First of all the
plugin seems to work perfectly for my SNAPSHOT repositories, it was
also really easy to install and set up, so thanks for that.
My question is related to using the plugin in a nexus group. We use
Nexus Professional and the staging facilities, which mean we have
multiple staging repositories visible via a Nexus Group, and
potentially each of these staging repositories will have its own
repodata. Have you guys any experience in this type of setup? Does the
plugin already handle this?
Sandy
On 4/20/12 5:05 PM, "Sandy McPherson" wrote:
The files are in the location that you have in your code.
I deleted the files by hand and re-ran mergerepo by hand, it still
does
not want to pick up multiple versions of the same RPM which are in
different repositories.
Your code is working fine for me, it is mergerepo that is letting me
down,
so I will press on with my script. All of the rest of the
functionality
provided by Nexus Pro and you plug-in fits our use case exactly, so
I'm
not going to let this beat meŠ Do you think I should raise a ticket
with
the yum guys?
Have a nice weekend
Sandy
-----------------------------------------------------------------------
On 4/20/12 3:41 PM, "Sebastian Herold"
Hi Sandy,
Mergerepo has a strange bug described here:
http://lists.baseurl.org/pipermail/rpm-metadata/2011-August/001370.html
It saves the repository information for a repository and generates the
same result over and over although you pass different parameters.
Therefore I implemented a method called cleanYumCacheDir, see:
http://code.google.com/p/nexus-yum-plugin/source/browse/trunk/src/main/jav
a/de/is24/nexus/yum/repository/task/
YumGroupRepositoryGenerationTask.java
which removes everything matching /var/tmp/yum-<user.name>-... .
Maybe,
this isn't working correctly in your case. Could you please find out,
under which location yum stores temporary file on your machine. Which
OS
are you using?
Regards,
Sebastian
--------------------------------------------------------------------------------------------------------------
Am 20.04.2012 um 11:06 schrieb Sandy McPherson:
Hi Sebastian,
I double checked that the plugin was at the version I built from
trunk.
The log is very short (320L) so I just included the complete file.
Looking through the logs and using some of the command manually I can
see that your plugin is doing everything that you would expect of it.
The
issue (or false interoperation my me) lies within mergerepo itself, as
it does not seem to handle merging different versions of the same RPM
from
different repos. The interesting part is where the staging repository
system-images-002
gets created by nexus.
I copied the mergrepo command out of the log and ran it manually. It
doesn't merge multiple versions of the same RPM from different
repositories, it seems to use the first in the list.
mergerepo --nogroups -d
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-001/
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-002/
-o /home/nexrep_d_nl_eng_nexs_01/work/storage/DEV
Loaded plugins: fastestmirror, presto, refresh-packagekit
Loading mirror speeds from cached hostfile
1/1 - corrie-masteruser-server-system-10.3547-20120419174847.noarch
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
Produces a single entry for my RPM which shows the version
10.3547-20120419174847 from 001.
Reversing the order of the repositories
mergerepo --nogroups -d
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-002/
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-001/
-o /home/nexrep_d_nl_eng_nexs_01/work/storage/DEV
Loaded plugins: fastestmirror, presto, refresh-packagekit
Loading mirror speeds from cached hostfile
Not using downloaded repomd.xml because it is older than what we have:
Current : Thu Apr 19 10:57:24 2012
Downloaded: Thu Apr 19 10:49:08 2012
1/1 - corrie-masteruser-server-system-10.3548-20120419175704.noarch
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
let's me see a single entry with the newer version
10.3548-20120419175704
from 002
Reversing the order again shows (strangely enough) still the newer RPM
mergerepo --nogroups -d
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-001/
--repo=file:/home/nexrep_d_nl_eng_nexs_01/work/storage/system-
images-002/
-o /home/nexrep_d_nl_eng_nexs_01/work/storage/DEV
Loaded plugins: fastestmirror, presto, refresh-packagekit
Loading mirror speeds from cached hostfile
Not using downloaded repomd.xml because it is older than what we have:
Current : Thu Apr 19 10:57:24 2012
Downloaded: Thu Apr 19 10:49:08 2012
1/1 - corrie-masteruser-server-system-10.3548-20120419175704.noarch
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
I am afraid I myself don't know who mergerepo is supposed to work with
multiple versions, however one of my colleagues is a genius with this
type of thing, so may be
-----------------------------------------------------------------------------
On 4/20/12 9:22 AM, "Sebastian Herold" wrote:
Hi Sandy,
please make sure your target group repository has maven2yum format. It
seems that mergerepo doesn't work well in your case. If this doesn't
help, please send me your nexus.xml, staging.xml and the part of your
nexus.log for the corresponding time period.
Good luck,
Sebastian
-----------------------------------------------------------------------------
Am 19.04.2012 um 18:07 schrieb Sandy McPherson:
Hi Sebastian,
Sorry to bother you again, but I have a further question/problem. I
have installed the new version of the plugin and when I deploy to
Nexus and close my staging repository I can see the RPM and install it
using YUM, the repodata directory is also generated in the storage
area of the staging profiles target group. I then deploy a newer
version which opens a new staging repository, but on closing this
second staging repository which gets added to the group correctly I
can only see the old version. I looked in the primary.xm.gz in the
group directory and it definitely only contains a single entry for the
old RPM.
Any ideas what I'm doing wrong?
-----------------------------------------------------------------------------
On 4/19/12 10:59 AM, "Sebastian Herold" wrote:
Hi Sandy,
in the trunk, the merging is already in place. Please, checkout the
trunk
and build the current snapshot or wait for the next release.
Regards,
Sebastian
-----------------------------------------------------------------------------
Am 19.04.2012 um 10:56 schrieb Sandy McPherson:
Hi Sebastian,
Thanks for the quick reply.
Just for clarification: we are using 2.0.2.1 of the plugin and 2.0.3
nexus-pro. If I use a Maven2Yum-group repository, is the merging
already in place? Or do we need to wait for your next release? If so
what
are your plans?
Thanks
Sandy
FYI I opened a ticket with Sonatype, because the maven2yum
repositoryTargets are not visible in the Webgui.
https://issues.sonatype.org/browse/SUPPORT-1515
There would also appear to be issues with the refreshing of some of
the GUI elements. I seem to have to logout and login again to gain
visibility of repositories I have created when configuring the system.
Don't know if you have visibility of this, but just to let you know.
----------------------------------------------------------------------------
On 4/19/12 10:23 AM, "Sebastian Herold"
<> wrote:
Hi Sandy,
your setup is similar to ours. Using staging repositories in group
repositories was a pain in the ass for a long time. I fixed the
problem in the trunk, but haven't released since then. The solution is
the newly introduced Maven2Yum-group repositories. It's possible to
assign these repositories as target repositories for staging profiles.
The Maven2Yum-groups merge the included repositories using mergerepo
(you need at least createrepo-0.9.1). But please use a Maven2yum-
repository as final repository, this avoids the copying of the
repodata/
directory from the staging repo to the final repository.
Best regards,
Sebastian
----------------------------------------------------------------------------
Am 18.04.2012 um 17:24 schrieb Sandy McPherson:
Hallo Sebastian,
Not sure if this is the correct way to contact you for questions
relating to the nexus yum plugin, but here goes. First of all the
plugin seems to work perfectly for my SNAPSHOT repositories, it was
also really easy to install and set up, so thanks for that.
My question is related to using the plugin in a nexus group. We use
Nexus Professional and the staging facilities, which mean we have
multiple staging repositories visible via a Nexus Group, and
potentially each of these staging repositories will have its own
repodata. Have you guys any experience in this type of setup? Does the
plugin already handle this?
Reply all
Reply to author
Forward
0 new messages