Elasticsearch warnings "failure IndexShardRecoveryException"

[Neven Luetic]

We have a nexus setup actually not showing any problems at the surface, i.e. we can upload and download artifacts, search for existing components etc.

However when going through the logs after a recent uncontrolled disruption of the service, we noticed regular messages (dating from before the mentioned disruption) by elasicsearch concerning "failed shards" for some indexes.

Those messages occur quite frequently - probably whenever the index would have to be updated.

I have two questions here:
How can we determine, if and how this is an issue for us at all?
How can the indexes be rebuilt or restored?

Unfortunately documentation about the elasticsearch component of nexus is quite scarce.

  2017-06-12 00:00:00,206+0000 WARN  [elasticsearch[0830B166-40C32B3F-29D5B1EA-31CA71C6-25CC4CED][local_transport][T#1]] *S
YSTEM org.elasticsearch.cluster.action.shard - [0830B166-40C32B3F-29D5B1EA-31CA71C6-25CC4CED] [a52b0b100dbae53dfc54b6233c
c368c57e8dca35][0] received shard failed for [a52b0b100dbae53dfc54b6233cc368c57e8dca35][0], node[ThNZvHC4RCqRuP9g14s8HQ],
 [P], v[5], s[INITIALIZING], a[id=zQeMmOLyR6ui4XcTwDotWw], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-06-11T23:5
9:59.851Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: Engine
CreationFailureException[failed to create engine]; nested: EOFException; ]], indexUUID [jl80KDHmS-ijq14OsbwxHA], message
[master {0830B166-40C32B3F-29D5B1EA-31CA71C6-25CC4CED}{ThNZvHC4RCqRuP9g14s8HQ}{local}{local[1]}{local=true, master=true}
marked shard as initializing, but shard is marked as failed, resend shard failure], failure [Unknown]
2017-06-12 00:00:00,211+0000 WARN  [elasticsearch[0830B166-40C32B3F-29D5B1EA-31CA71C6-25CC4CED][generic][T#1479]] *SYSTEM
 org.elasticsearch.indices.cluster - [0830B166-40C32B3F-29D5B1EA-31CA71C6-25CC4CED] [[62d189601e138867db4d6cbfee3fc80fe18
7efe6][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recovery from gateway
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:254) [na:na]
    ... 5 common frames omitted
Caused by: java.io.EOFException: null
    at org.apache.lucene.store.InputStreamDataInput.readByte(InputStreamDataInput.java:37) [na:na]
    at org.apache.lucene.store.DataInput.readInt(DataInput.java:101) [na:na]
    at org.apache.lucene.store.DataInput.readLong(DataInput.java:157) [na:na]
    at org.elasticsearch.index.translog.Checkpoint.<init>(Checkpoint.java:54) [na:na]
    at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:83) [na:na]
    at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:337) [na:na]
    at org.elasticsearch.index.translog.Translog.<init>(Translog.java:179) [na:na]
    at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:209) [na:na]
    at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152) [na:na]

    ... 11 common frames omitted




 

[Wes Wannemacher]

Hello Neven,

Which version of Nexus Repository Manager are you using? 

I believe there will be a scheduled task you can create that rebuilds the indexes. The instructions for creating and executing the task differ based on the version you are using. At this point, rebuilding the indexes is your best bet.

-Wes

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.
To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/c478c2ee-676a-4dd0-91b1-5ac69f66370a%40glists.sonatype.com.
For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

[Neven Luetic]

Hi Wes,

thank you for your hint.

We have version 3.2.0-01 and there is a task called "Rebuild repository index", where I can select the repositories that should be rebuilt and an execution scheme for the task.

As I cannot tell which repositories the problematic indexes belong to (and I would expect some of them to belong to multiple ones), I would start this task once for all repositories.

I suppose recreating all indexes could be a bit risky as I could be left without any at all, if the rebuild fails?

And is there a way to tell how long this could take?

--
Neven

[Wes Wannemacher]

I wouldn't worry about the risk, the indexes are not the main system of record for nexus data. They are used in a few places, for performance reasons, but the rebuild process is meant to fix problems exactly like what you are encountering. 

Go ahead and rebuild the indexes for all repositories. There was a performance enhancement that was completed recently, however I do not remember if it was available in 3.2.x or if it was in 3.3.x. Anyhow, I still do not believe you will encounter any problems, even if the index rebuild takes a while. 

-Wes

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users+unsubscribe@glists.sonatype.com.
To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/bde80b41-3398-485a-bc2f-c93e637448bd%40glists.sonatype.com.

[Nicholas Blair]

The search index rebuild task enhancement Wes is referring to is scheduled to be included in the 3.3.2 release, which is expected in the immediate future:

https://issues.sonatype.org/browse/NEXUS-12520

If you do run into issues with index rebuild performance with 3.0 through 3.3.1, there are a few workarounds listed in the comments of that task to adjust the elasticsearch configuration.

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/bde80b41-3398-485a-bc2f-c93e637448bd%40glists.sonatype.com.

For more options, visit https://groups.google.com/a/glists.sonatype.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Nexus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nexus-users...@glists.sonatype.com.

To post to this group, send email to nexus...@glists.sonatype.com.

To view this discussion on the web visit https://groups.google.com/a/glists.sonatype.com/d/msgid/nexus-users/CALKYsFWXP-hbGe%2B9kWcFRook9NNUkE9m%2B%2B_KUGNssYbT6UJuEQ%40mail.gmail.com.