EOF и Bad file descriptor

[Kirill]

Добрый день,

поток данных ежедневно составляет 78 000 000 документов объёмом 32ГБ.  Ежедневно создаётся новый индекс формата ДД-ММ-ГГ. 

Характеристики сервера: 

Debian

model name      : Intel(R) Xeon(R) CPU    X5550  @ 2.67GHz

cpu MHz         : 2666.807

cache size      : 8192 KB

cpu family      : 6

processor       : 15

cpu cores       : 4

ОЗУ: 48ГБ

Сеть 1ГБ

По сети подключено хранилище объёмом 10ТБ, на котором и лежат данные. 

Elasticksearch 1.4.3

На каждый индекс создаётся 5 шардов. 

indices.fielddata.cache.size:  50%

ES_HEAP_SIZE=24576m

refresh_interval=180s

Возникают проблемы с памятью (EOF) и ошибки Bad file descriptor.

[2015-02-09 17:24:55,082][DEBUG][action.search.type       ] [*.*.*.*] [06-02-2015][4], node[priU7u7aRHSQe0CdW9D38A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4f1f2632] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [06-02-2015][4]: query[ConstantScore(*:*)],from[0],size[0]: Query Failed [Failed to execute main query]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: Bad file descriptor: NIOFSIndexInput(path="/elasticdata/nodes/0/indices/06-02-2015/4/index/_18e_es090_0.doc")
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.facet.statistical.StatisticalFacetExecutor$Collector.setNextReader(StatisticalFacetExecutor.java:67)
at org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:67)
at org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)
at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:117)
... 7 more

[2015-02-12 11:21:16,800][DEBUG][action.search            ] [*.*.*.*] failed to optimize search type, continue as normal
org.elasticsearch.indices.IndexClosedException: [10-02-2015] closed
at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:705)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:85)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:45)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98)
at org.elasticsearch.client.FilterClient.execute(FilterClient.java:66)
at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.execute(BaseRestHandler.java:92)
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:334)
at org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:81)
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:53)
at org.elasticsearch.rest.RestController.executeHandler(RestController.java:197)
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:180)
at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:319)
at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
at org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Прошу посоветовать, как можно оптимизировать настройки ELK не изменяя железо. 

[Igor Motov]

А как выглядит эта EOF ошибка? Какие-нибудь другие ошибки в логах есть? Вы индексы периодически закрываете?

[Kirill]

Закрываю индексы старше месяца.  Логов с EOF ошибкой не сохранил. На данный момент, как альтернативное решение проблемы уменьшаю объём входных данных. 

вторник, 17 февраля 2015 г., 0:19:37 UTC+3 пользователь Igor Motov написал:

[Igor Motov]

Время появления ошибок совпадает со временем открытия/закрытия индексов? 

[Kirill]

Нет, не совпадает. Проблема появятся при активной работе через kiban'у.  

среда, 18 февраля 2015 г., 16:15:24 UTC+3 пользователь Igor Motov написал:

[Igor Motov]

Странно. А вы не могли бы в момент, когда эти ошибки появляются, запустить вот эти две команды и прислать мне результат

curl -XGET 'http://localhost:9200/_nodes/stats'

curl -XGET 'http://localhost:9200/_nodes'

[Kirill]

Добрый день,

подскажите как бороться с ошибкой Bad file descriptor ?

Лог ошибки ниже:

[2015-02-25 06:33:19,263][WARN ][index.merge.scheduler    ] [*.*.*.*] [2015-02-24][2] failed to merge

java.io.IOException: Bad file descriptor: NIOFSIndexInput(path="/elasticdata/nodes/0/indices/2015-02-24/2/index/_36.fdt")

at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:189)

at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:160)

at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)

at org.apache.lucene.store.BufferedChecksumIndexInput.readBytes(BufferedChecksumIndexInput.java:49)

at org.apache.lucene.store.DataInput.readBytes(DataInput.java:84)

at org.apache.lucene.store.DataInput.skipBytes(DataInput.java:298)

at org.apache.lucene.store.ChecksumIndexInput.seek(ChecksumIndexInput.java:51)

at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:267)

at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.checkIntegrity(CompressingStoredFieldsReader.java:535)

at org.apache.lucene.index.SegmentReader.checkIntegrity(SegmentReader.java:624)

at org.apache.lucene.index.SegmentMerger.<init>(SegmentMerger.java:61)

at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4165)

at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3775)

at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)

at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)

at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)

Caused by: java.io.IOException: Bad file descriptor

at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)

at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)

at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)

at sun.nio.ch.IOUtil.read(IOUtil.java:197)

at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:700)

at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:685)

at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)

... 15 more

[2015-02-25 06:33:19,315][WARN ][index.engine.internal    ] [*.*.*.*] [2015-02-24][2] failed engine [merge exception]

org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: Bad file descriptor: NIOFSIndexInput(path="/elasticdata/nodes/0/indices/2015-02-24/2/index/_36.fdt")

at org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:133)

at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:522)

Caused by: java.io.IOException: Bad file descriptor: NIOFSIndexInput(path="/elasticdata/nodes/0/indices/2015-02-24/2/index/_36.fdt")

at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:189)

at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:160)

at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)

at org.apache.lucene.store.BufferedChecksumIndexInput.readBytes(BufferedChecksumIndexInput.java:49)

at org.apache.lucene.store.DataInput.readBytes(DataInput.java:84)

at org.apache.lucene.store.DataInput.skipBytes(DataInput.java:298)

at org.apache.lucene.store.ChecksumIndexInput.seek(ChecksumIndexInput.java:51)

at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:267)

at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.checkIntegrity(CompressingStoredFieldsReader.java:535)

at org.apache.lucene.index.SegmentReader.checkIntegrity(SegmentReader.java:624)

at org.apache.lucene.index.SegmentMerger.<init>(SegmentMerger.java:61)

at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4165)

at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3775)

at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)

at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)

at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)

Caused by: java.io.IOException: Bad file descriptor

at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)

at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)

at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)

at sun.nio.ch.IOUtil.read(IOUtil.java:197)

at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:700)

at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:685)

at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)

... 15 more

[2015-02-25 06:33:19,757][WARN ][cluster.action.shard     ] [*.*.*.*] [2015-02-24][2] sending failed shard for [2015-02-24][2], node[7dPOKn1tS3yGgw3VB4mBrQ], [P], s[STARTED], indexUUID [bKZMgzTOSTqgybxPajaSWA], reason [engine failure, message [merge exception][MergeException[java.io.IOException: Bad file descriptor: NIOFSIndexInput(path="/elasticdata/nodes/0/indices/2015-02-24/2/index/_36.fdt")]; nested: IOException[Bad file descriptor: NIOFSIndexInput(path="/elasticdata/nodes/0/indices/2015-02-24/2/index/_36.fdt")]; nested: IOException[Bad file descriptor]; ]]

[2015-02-25 06:33:19,757][WARN ][cluster.action.shard     ] [*.*.*.*] [2015-02-24][2] received shard failed for [2015-02-24][2], node[7dPOKn1tS3yGgw3VB4mBrQ], [P], s[STARTED], indexUUID [bKZMgzTOSTqgybxPajaSWA], reason [engine failure, message [merge exception][MergeException[java.io.IOException: Bad file descriptor: NIOFSIndexInput(path="/elasticdata/nodes/0/indices/2015-02-24/2/index/_36.fdt")]; nested: IOException[Bad file descriptor: NIOFSIndexInput(path="/elasticdata/nodes/0/indices/2015-02-24/2/index/_36.fdt")]; nested: IOException[Bad file descriptor]; ]]

{"cluster_name":"elasticdata","nodes":{"7dPOKn1tS3yGgw3VB4mBrQ":{"name":"*.*.*.*","transport_address":"inet[/*.*.*.*:9300]","host":"*.*.*.*","ip":"*.*.*.*","version":"1.4.3","build":"36a29a7","http_address":"inet[/*.*.*.*:9200]","attributes":{"rack":"*.*.*.*","master":"true"},"settings":{"node":{"data":"true","rack":"*.*.*.*","master":"true","name":"*.*.*.*"},"index":{"number_of_replicas":"0","number_of_shards":"5"},"client":{"type":"node"},"http":{"port":"9200","cors":{"enabled":"true","allow-origin":"http://*.*.*.*"}},"name":"*.*.*.*","max-open-files":"true","path":{"data":"/es/*.*.*.*","work":"/es/*.*.*.*/temporary","home":"/opt/elasticsearch","logs":"/media/es/*.*.*.*/logs"},"indices":{"fielddata":{"cache":{"size":"50%"}}},"cluster":{"name":"elasticdata"},"discovery":{"zen":{"ping":{"multicast":{"enabled":"false"}}}}},"os":{"refresh_interval_in_millis":1000,"available_processors":16,"cpu":{"vendor":"Intel","model":"Xeon","mhz":2666,"total_cores":16,"total_sockets":1,"cores_per_socket":16,"cache_size_in_bytes":8192},"mem":{"total_in_bytes":50743226368},"swap":{"total_in_bytes":5986316288}},"process":{"refresh_interval_in_millis":1000,"id":15068,"max_file_descriptors":32000,"mlockall":false},"jvm":{"pid":15068,"version":"1.7.0_65","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"24.65-b04","vm_vendor":"Oracle Corporation","start_time_in_millis":1424350289731,"mem":{"heap_init_in_bytes":25769803776,"heap_max_in_bytes":25656426496,"non_heap_init_in_bytes":24313856,"non_heap_max_in_bytes":224395264,"direct_max_in_bytes":25656426496},"gc_collectors":["ParNew","ConcurrentMarkSweep"],"memory_pools":["Code Cache","Par Eden Space","Par Survivor Space","CMS Old Gen","CMS Perm Gen"]},"thread_pool":{"generic":{"type":"cached","keep_alive":"30s","queue_size":-1},"index":{"type":"fixed","min":16,"max":16,"queue_size":"200"},"bench":{"type":"scaling","min":1,"max":5,"keep_alive":"5m","queue_size":-1},"get":{"type":"fixed","min":16,"max":16,"queue_size":"1k"},"snapshot":{"type":"scaling","min":1,"max":5,"keep_alive":"5m","queue_size":-1},"merge":{"type":"scaling","min":1,"max":5,"keep_alive":"5m","queue_size":-1},"suggest":{"type":"fixed","min":16,"max":16,"queue_size":"1k"},"bulk":{"type":"fixed","min":16,"max":16,"queue_size":"50"},"optimize":{"type":"fixed","min":1,"max":1,"queue_size":-1},"warmer":{"type":"scaling","min":1,"max":5,"keep_alive":"5m","queue_size":-1},"flush":{"type":"scaling","min":1,"max":5,"keep_alive":"5m","queue_size":-1},"search":{"type":"fixed","min":48,"max":48,"queue_size":"1k"},"listener":{"type":"fixed","min":8,"max":8,"queue_size":-1},"percolate":{"type":"fixed","min":16,"max":16,"queue_size":"1k"},"management":{"type":"scaling","min":1,"max":5,"keep_alive":"5m","queue_size":-1},"refresh":{"type":"scaling","min":1,"max":8,"keep_alive":"5m","queue_size":-1}},"network":{"refresh_interval_in_millis":5000,"primary_interface":{"address":"*.*.*.*","name":"eth0","mac_address":"*.*.*.*"}},"transport":{"bound_address":"inet[/0:0:0:0:0:0:0:0:9300]","publish_address":"inet[/*.*.*.*:9300]"},"http":{"bound_address":"inet[/0:0:0:0:0:0:0:0:9200]","publish_address":"inet[/*.*.*.*:9200]","max_content_length_in_bytes":104857600},"plugins":[]}}}

Пришлось удалить проблемный индекс.

[2015-02-25 09:15:17,850][INFO ][cluster.metadata         ] [*.*.*.*] [2015-02-24] deleting index

{"cluster_name":"elasticdata","nodes":{"7dPOKn1tS3yGgw3VB4mBrQ":{"timestamp":1424845073300,"name":"*.*.*.*","transport_address":"inet[/*.*.*.*:9300]","host":"*.*.*.*","ip":["inet[/*.*.*.*:9300]","NONE"],"attributes":{"rack":"*.*.*.*","master":"true"},"indices":{"docs":{"count":234177838,"deleted":0},"store":{"size_in_bytes":99023288262,"throttle_time_in_millis":2433042},"indexing":{"index_total":185412136,"index_time_in_millis":49571949,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0,"noop_update_total":0,"is_throttled":false,"throttle_time_in_millis":0},"get":{"total":1521,"time_in_millis":348,"exists_total":1521,"exists_time_in_millis":348,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":504767,"query_time_in_millis":44287990,"query_current":0,"fetch_total":44093,"fetch_time_in_millis":294927,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":4169,"total_time_in_millis":21009471,"total_docs":344872075,"total_size_in_bytes":163986854800},"refresh":{"total":150385,"total_time_in_millis":13221286},"flush":{"total":3096,"total_time_in_millis":2008031},"warmer":{"current":0,"total":59511,"total_time_in_millis":5295},"filter_cache":{"memory_size_in_bytes":1916007880,"evictions":517880},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":3338146068,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":1059,"memory_in_bytes":357946582,"index_writer_memory_in_bytes":0,"index_writer_max_memory_in_bytes":2637834645,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":0},"translog":{"operations":1,"size_in_bytes":17},"suggest":{"total":0,"time_in_millis":0,"current":0},"query_cache":{"memory_size_in_bytes":0,"evictions":0,"hit_count":0,"miss_count":0}},"os":{"timestamp":1424845074429,"uptime_in_millis":4531968,"load_average":[0.27,0.28,0.28],"cpu":{"sys":0,"user":0,"idle":98,"usage":0,"stolen":0},"mem":{"free_in_bytes":8305721344,"used_in_bytes":42437505024,"free_percent":56,"used_percent":43,"actual_free_in_bytes":28607164416,"actual_used_in_bytes":22136061952},"swap":{"used_in_bytes":244482048,"free_in_bytes":5741834240}},"process":{"timestamp":1424845074429,"open_file_descriptors":2617,"cpu":{"percent":15,"sys_in_millis":4618570,"user_in_millis":77802710,"total_in_millis":82421280},"mem":{"resident_in_bytes":39342288896,"share_in_bytes":17747644416,"total_virtual_in_bytes":60921192448}},"jvm":{"timestamp":1424845074431,"uptime_in_millis":494784700,"mem":{"heap_used_in_bytes":14741006328,"heap_used_percent":57,"heap_committed_in_bytes":25656426496,"heap_max_in_bytes":25656426496,"non_heap_used_in_bytes":58665192,"non_heap_committed_in_bytes":87187456,"pools":{"young":{"used_in_bytes":14248464,"max_in_bytes":907345920,"peak_used_in_bytes":907345920,"peak_max_in_bytes":907345920},"survivor":{"used_in_bytes":28617480,"max_in_bytes":113377280,"peak_used_in_bytes":113377280,"peak_max_in_bytes":113377280},"old":{"used_in_bytes":14698140384,"max_in_bytes":24635703296,"peak_used_in_bytes":18699404128,"peak_max_in_bytes":24635703296}}},"threads":{"count":236,"peak_count":248},"gc":{"collectors":{"young":{"collection_count":7520,"collection_time_in_millis":340366},"old":{"collection_count":10,"collection_time_in_millis":1320}}},"buffer_pools":{"direct":{"count":1456,"used_in_bytes":46875144,"total_capacity_in_bytes":46875144},"mapped":{"count":659,"used_in_bytes":25649622115,"total_capacity_in_bytes":25649622115}}},"thread_pool":{"generic":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":5,"completed":54245},"index":{"threads":16,"queue":0,"active":0,"rejected":0,"largest":16,"completed":12604},"bench":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"get":{"threads":16,"queue":0,"active":0,"rejected":0,"largest":16,"completed":29},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"merge":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":5,"completed":233907},"suggest":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"bulk":{"threads":16,"queue":0,"active":0,"rejected":0,"largest":16,"completed":242791},"optimize":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"warmer":{"threads":2,"queue":0,"active":0,"rejected":0,"largest":3,"completed":43338},"flush":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":5,"completed":70001},"search":{"threads":48,"queue":0,"active":0,"rejected":0,"largest":48,"completed":592912},"listener":{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":28889},"percolate":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"management":{"threads":5,"queue":0,"active":1,"rejected":0,"largest":5,"completed":1427287},"refresh":{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":151001}},"network":{"tcp":{"active_opens":15580,"passive_opens":15474,"curr_estab":36,"in_segs":3248377045,"out_segs":10851775323,"retrans_segs":267117,"estab_resets":620,"attempt_fails":11677,"in_errs":0,"out_rsts":13217}},"fs":{"timestamp":1424845074432,"total":{"total_in_bytes":11614908575744,"free_in_bytes":6550996910080,"available_in_bytes":6550996910080},"data":[{"path":"/elasticdata/nodes/0","mount":"/media/es","dev":"//*.*.*.*/es/","total_in_bytes":11614908575744,"free_in_bytes":6550996910080,"available_in_bytes":6550996910080}]},"transport":{"server_open":13,"rx_count":6,"rx_size_in_bytes":1980,"tx_count":6,"tx_size_in_bytes":1980},"http":{"current_open":6,"total_opened":2811},"breakers":{"request":{"limit_size_in_bytes":10262570598,"limit_size":"9.5gb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0},"fielddata":{"limit_size_in_bytes":15393855897,"limit_size":"14.3gb","estimated_size_in_bytes":3338146068,"estimated_size":"3.1gb","overhead":1.03,"tripped":0},"parent":{"limit_size_in_bytes":17959498547,"limit_size":"16.7gb","estimated_size_in_bytes":3338146068,"estimated_size":"3.1gb","overhead":1.0,"tripped":0}}}}}

[Igor Motov]

Вы написали что у Вас по сети подключено хранилище данных. Вы не могли бы написать по какому протоколу и с какими параметрами? Проблема скорее всего в этом самом хранилище или в том, как оно подключено. То есть либо у вас там диск полетел, либо оно не дает открыть достаточное количество файлов одновременно, либо еще что-нибудь в этом роде. На стороне elasticsearch все вроде выглядит нормально. Вы системный лог не проверяли? Там система на это хранилище не ругается? 

[Kirill]

Тоже грешу на хранилище. Подключено по cifs.

#nano /etc/fstab

/*.*.*.*/es/ /media/es cifs noperm,rw,dir_mode=0775,file_mode=0775,uid=0,gid=100 0 0

среда, 25 февраля 2015 г., 22:01:49 UTC+3 пользователь Igor Motov написал:

[Igor Motov]

Поговорил со спецами по Lucene. Порекомендовали посмотреть, поддерживает ли ваше хранилище delete-on-last-close. Lucene использует эту фичу и если с ней неполадки, то они часто выражаются как bad file descriptor.  

[Kirill]

Добрый день,

подскажите пожалуйста, как поменять максимальное кол-во открытых файлов в windows storage 2008? В данный момент 16384 и ELK генерирует ошибку Too many open files.

четверг, 26 февраля 2015 г., 17:51:49 UTC+3 пользователь Igor Motov написал: