clustered indices inconsistent

[Bee RayDee]

We are receiving different search results for the same search when searching for content in the Admin panel. Also, when we attempt to re-index a single piece of content we get errors like this:

[29/05/14 13:54:45:128 CDT]  WARN engine.robin: [dotCMS_1] [live_20130809181151][1] failed to read latest segment infos on flush

java.io.FileNotFoundException: C:\dotcms\dotCMS\dotsecure\esdata\dotCMSContentIndexPROD\nodes\0\indices\live_20130809181151\1\index\segments_2gb (Access is denied)

at java.io.RandomAccessFile.open(Native Method)

at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)

at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:218)

at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)

at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:521)

at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:265)

at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:363)

at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)

at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:554)

at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:359)

at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:952)

at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:476)

at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:191)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:722)

[29/05/14 13:54:45:132 CDT]  WARN engine.robin: [dotCMS_1] [working_20130809181151][0] failed to read latest segment infos on flush

java.io.FileNotFoundException: C:\dotcms\dotCMS\dotsecure\esdata\dotCMSContentIndexPROD\nodes\0\indices\working_20130809181151\0\index\segments_2hk (Access is denied)

at java.io.RandomAccessFile.open(Native Method)

at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)

at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:218)

at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)

at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:521)

at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:265)

at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:363)

at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)

at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:554)

at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:359)

at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:952)

at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:476)

at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:191)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:722)

It appears the the indices are having trouble communicating and have become out of sync. Has anyone seen this before? I do not think that the file permissions have changed at all, so I'm not sure what other permissions to check.

[Bee RayDee]

we are also receiving errors like this one...not sure if these are related or not...

[29/05/14 14:01:50:126 CDT] ERROR image.ResizeImageServlet: Error creating thumbnail from ResizeImage servlet: null

ClientAbortException:  java.net.SocketException: Software caused connection abort: socket write error

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358)

at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)

at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)

at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:381)

at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:370)

at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)

at com.dotmarketing.servlets.image.ResizeImageServlet.service(ResizeImageServlet.java:486)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.dotmarketing.filters.CMSFilter.doFilter(CMSFilter.java:128)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.dotmarketing.filters.AutoLoginFilter.doFilter(AutoLoginFilter.java:61)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.mlh.dotcms.override.filters.URLMapFilter.doFilter(URLMapFilter.java:93)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.dotmarketing.filters.TimeMachineFilter.doFilter(TimeMachineFilter.java:36)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at com.dotmarketing.filters.CharsetEncodingFilter.doFilter(CharsetEncodingFilter.java:140)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)

at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)

at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)

at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)

at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)

at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)

at java.lang.Thread.run(Thread.java:722)

Caused by: java.net.SocketException: Software caused connection abort: socket write error

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)

at java.net.SocketOutputStream.write(SocketOutputStream.java:153)

at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:532)

at org.apache.jk.common.JkInputStream.doWrite(JkInputStream.java:162)

at org.apache.coyote.Response.doWrite(Response.java:560)

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)

... 38 more

[Bee RayDee]

and this is from the other server in the cluster: 

[29/05/14 13:38:21:791 CDT] ERROR content.ContentTool: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[qCnMLRZuT82n8_NB5WfJtw][live_20130809181151][0]: RemoteTransportException[[dotCMS_1][inet[/192.168.23.97:9301]][search/phase/query]]; nested: SearchParseException[[live_20130809181151][0]: from[-1],size[10]: Parse Failure [Failed to parse source [{"size":10,"query":{"query_string":{"query":"+structurename:blogentry +(conhost:ad337ec5-7d50-4bf6-9263-f07fe882dbc7 conhost:system_host) +live:true +(blogentry.tag:) +languageid:1 +deleted:false +((permissions:p654b0931-1027-41f7-ad4d-173115ed8ec1.1p*) )"}},"sort":[{"blogentry.posting_dotraw":{"order":"desc"}}]}]]]; nested: QueryParsingException[[live_20130809181151] Failed to parse query [+structurename:blogentry +(conhost:ad337ec5-7d50-4bf6-9263-f07fe882dbc7 conhost:system_host) +live:true +(blogentry.tag:) +languageid:1 +deleted:false +((permissions:p654b0931-1027-41f7-ad4d-173115ed8ec1.1p*) )]]; nested: ParseException[Cannot parse '+structurename:blogentry +(conhost:ad337ec5-7d50-4bf6-9263-f07fe882dbc7 conhost:system_host) +live:true +(blogentry.tag:) +languageid:1 +deleted:false +((permissions:p654b0931-1027-41f7-ad4d-173115ed8ec1.1p*) )': Encountered " ")" ") "" at line 1, column 120.

Was expecting one of:

    <BAREOPER> ...

    "(" ...

    "*" ...

    <QUOTED> ...

    <TERM> ...

    <PREFIXTERM> ...

    <WILDTERM> ...

    "[" ...

    "{" ...

    <NUMBER> ...

    ]; nested: ParseException[Encountered " ")" ") "" at line 1, column 120.

Was expecting one of:

    <BAREOPER> ...

    "(" ...

    "*" ...

    <QUOTED> ...

    <TERM> ...

    <PREFIXTERM> ...

    <WILDTERM> ...

    "[" ...

    "{" ...

    <NUMBER> ...

    ]; }{[H6h7Hp1FR_GndWINo5IIew][live_20130809181151][1]: SearchParseException[[live_20130809181151][1]: from[-1],size[10]: Parse Failure [Failed to parse source [{"size":10,"query":{"query_string":{"query":"+structurename:blogentry +(conhost:ad337ec5-7d50-4bf6-9263-f07fe882dbc7 conhost:system_host) +live:true +(blogentry.tag:) +languageid:1 +deleted:false +((permissions:p654b0931-1027-41f7-ad4d-173115ed8ec1.1p*) )"}},"sort":[{"blogentry.posting_dotraw":{"order":"desc"}}]}]]]; nested: QueryParsingException[[live_20130809181151] Failed to parse query [+structurename:blogentry +(conhost:ad337ec5-7d50-4bf6-9263-f07fe882dbc7 conhost:system_host) +live:true +(blogentry.tag:) +languageid:1 +deleted:false +((permissions:p654b0931-1027-41f7-ad4d-173115ed8ec1.1p*) )]]; nested: ParseException[Cannot parse '+structurename:blogentry +(conhost:ad337ec5-7d50-4bf6-9263-f07fe882dbc7 conhost:system_host) +live:true +(blogentry.tag:) +languageid:1 +deleted:false +((permissions:p654b0931-1027-41f7-ad4d-173115ed8ec1.1p*) )': Encountered " ")" ") "" at line 1, column 120.

Was expecting one of:

    <BAREOPER> ...

    "(" ...

    "*" ...

    <QUOTED> ...

    <TERM> ...

    <PREFIXTERM> ...

    <WILDTERM> ...

    "[" ...

    "{" ...

    <NUMBER> ...

    ]; nested: ParseException[Encountered " ")" ") "" at line 1, column 120.

Was expecting one of:

    <BAREOPER> ...

    "(" ...

    "*" ...

    <QUOTED> ...

    <TERM> ...

    <PREFIXTERM> ...

    <WILDTERM> ...

    "[" ...

    "{" ...

    <NUMBER> ...

    ]; }