Corrupted elasticsearch index Nexus 3

1,847 views
Skip to first unread message

Mr Dima

unread,
Dec 13, 2017, 5:03:12 AM12/13/17
to Nexus Users

We have an issue that the Maven Cental elasticsearch index (maven central repo) is getting corrupt very fast, this can happen multiple times per day. This leeds to exteremely, unworkably slow repository fetching. We need to delete corrupted index from elasticsearch (for which we had the API exposed), do a rebuilt in the Nexus 3 GUI of the repository and then sometimes bump the Nexus 3 container before all is "green" again.
There's also a docker registry running, but that index doesn't get corrupted. We're using Nexus 3.6.0. We are running on relatively slow storage (glusterfs) and maybe that has something todo with it. We don't have fast storage available, although we could map folders to the local (kubernetes) node which is quite fast (but don't know if we can map a specific elasticsearch folder to local storage and when the container is bumped / restarted and starts with an empty folder if things will be ok).
Does anyone know of related issues, a workaround or tweak so that we can prevent indexes from getting corrupt?


Thanks,

Dimitri

Rich Seddon

unread,
Dec 13, 2017, 9:23:50 AM12/13/17
to Nexus Users
Glusterfs absolutely is the problem, it is not supported:


You'll need to move this instance to a supported file system.

Rich

Rich Seddon

unread,
Dec 13, 2017, 12:14:54 PM12/13/17
to Nexus Users

Mr Dima

unread,
Dec 13, 2017, 3:01:43 PM12/13/17
to Nexus Users
Thanks, do you know if this is mostly related to the elasticsearch part within Nexus or also other components? I'm currently experimenting using /nexus-data on glusterfs and /nexus-data/elasticsearch as a direct volume mount...

Peter Lynch

unread,
Dec 13, 2017, 3:16:11 PM12/13/17
to Mr Dima, Nexus Users
On Wed, Dec 13, 2017 at 4:01 PM, Mr Dima <dimit...@gmail.com> wrote:
Thanks, do you know if this is mostly related to the elasticsearch part within Nexus or also other components? I'm currently experimenting using /nexus-data on glusterfs and /nexus-data/elasticsearch as a direct volume mount...

We are not testing at all with glusterfs and do not recommend using it for any file system Nexus Repository Manager relies on for any feature. Use at your own risk.
 


On Wednesday, December 13, 2017 at 6:14:54 PM UTC+1, Rich Seddon wrote:
Sorry about that, I gave the wrong link.  This should work:


On Wednesday, December 13, 2017 at 8:23:50 AM UTC-6, Rich Seddon wrote:
Glusterfs absolutely is the problem, it is not supported:


You'll need to move this instance to a supported file system.

Rich

On Wednesday, December 13, 2017 at 4:03:12 AM UTC-6, Mr Dima wrote:

We have an issue that the Maven Cental elasticsearch index (maven central repo) is getting corrupt very fast, this can happen multiple times per day. This leeds to exteremely, unworkably slow repository fetching. We need to delete corrupted index from elasticsearch (for which we had the API exposed), do a rebuilt in the Nexus 3 GUI of the repository and then sometimes bump the Nexus 3 container before all is "green" again.
There's also a docker registry running, but that index doesn't get corrupted. We're using Nexus 3.6.0. We are running on relatively slow storage (glusterfs) and maybe that has something todo with it. We don't have fast storage available, although we could map folders to the local (kubernetes) node which is quite fast (but don't know if we can map a specific elasticsearch folder to local storage and when the container is bumped / restarted and starts with an empty folder if things will be ok).
Does anyone know of related issues, a workaround or tweak so that we can prevent indexes from getting corrupt?


Thanks,

Dimitri

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
 To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/cbb48a92-38de-4475-8745-86b6ce0e33c9%40glists.sonatype.com.

Mr Dima

unread,
Dec 13, 2017, 3:29:06 PM12/13/17
to Nexus Users, dimit...@gmail.com
Thanks. For what it's worth, I've been using Nexus3 for almost a year on glusterfs and running as a docker container under kubernetes, mostly as a docker repo (with some nuget and maven) and it works well. There were plenty of small issues but those weren't filesystem related until we hit this one with a corrupt maven repo index. Even geo-replicated glusterfs and recovering the nexus-data container and persistent gluster volume has proven to work so far after they fixed an issue (https://bugzilla.redhat.com/show_bug.cgi?id=1510342)
Point about the risk taken!


On Wednesday, December 13, 2017 at 9:16:11 PM UTC+1, Peter Lynch wrote:
On Wed, Dec 13, 2017 at 4:01 PM, Mr Dima <dimit...@gmail.com> wrote:
Thanks, do you know if this is mostly related to the elasticsearch part within Nexus or also other components? I'm currently experimenting using /nexus-data on glusterfs and /nexus-data/elasticsearch as a direct volume mount...

We are not testing at all with glusterfs and do not recommend using it for any file system Nexus Repository Manager relies on for any feature. Use at your own risk.
 


On Wednesday, December 13, 2017 at 6:14:54 PM UTC+1, Rich Seddon wrote:
Sorry about that, I gave the wrong link.  This should work:


On Wednesday, December 13, 2017 at 8:23:50 AM UTC-6, Rich Seddon wrote:
Glusterfs absolutely is the problem, it is not supported:


You'll need to move this instance to a supported file system.

Rich

On Wednesday, December 13, 2017 at 4:03:12 AM UTC-6, Mr Dima wrote:

We have an issue that the Maven Cental elasticsearch index (maven central repo) is getting corrupt very fast, this can happen multiple times per day. This leeds to exteremely, unworkably slow repository fetching. We need to delete corrupted index from elasticsearch (for which we had the API exposed), do a rebuilt in the Nexus 3 GUI of the repository and then sometimes bump the Nexus 3 container before all is "green" again.
There's also a docker registry running, but that index doesn't get corrupted. We're using Nexus 3.6.0. We are running on relatively slow storage (glusterfs) and maybe that has something todo with it. We don't have fast storage available, although we could map folders to the local (kubernetes) node which is quite fast (but don't know if we can map a specific elasticsearch folder to local storage and when the container is bumped / restarted and starts with an empty folder if things will be ok).
Does anyone know of related issues, a workaround or tweak so that we can prevent indexes from getting corrupt?


Thanks,

Dimitri

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.

Brian Fox

unread,
Dec 13, 2017, 3:32:41 PM12/13/17
to Mr Dima, Nexus Users
We've seen issues in the past with indexes on high latency types of connections getting corrupted (happened sometimes with Lucene also). I'd guess this is due to the high churn of those particular files as compared to the blobs themselves. Without saying it should be better they way you're approaching it...it is logical to try and move the indexes off gluster if you've otherwise been stable.

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/acfaed5c-800d-4577-80e1-956cb27e8c1c%40glists.sonatype.com.
Reply all
Reply to author
Forward
0 new messages