Logs form Filebeat do not reach Elasticsearh

202 views
Skip to first unread message

g...@recongate.com

unread,
Nov 27, 2018, 3:42:04 AM11/27/18
to elk-reasoner-discussion
Hi, I am running a local EFK stack on my kubernetes cluster and I can see my logs in Kibana only after restarting my Filebeat pods. Once restarted, the logs from the restart time reach their destination but a few minutes later they stop. 

This is a the Filebeat log:

2018-11-27T08:05:58.369Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2018-11-27T08:05:58.369Z INFO [publish] pipeline/retry.go:191   done
2018-11-27T08:06:05.486Z INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":3130,"time":{"ms":8}},"total":{"ticks":76680,"time":{"ms":8},"value":76680},"user":{"ticks":73550}},"info":{"ephemeral_id":"a3899fbb-9b46-4ea1-afb9-dfc5c3717390","uptime":{"ms":167520018}},"memstats":{"gc_next":47624032,"memory_alloc":24670472,"memory_total":1596393400,"rss":-16384}},"filebeat":{"harvester":{"open_files":9,"running":24}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":-4,"batches":1,"failed":9,"total":5},"read":{"bytes":1529},"write":{"bytes":4285}},"pipeline":{"clients":2,"events":{"active":4117,"retry":19}}},"registrar":{"states":{"current":13}},"system":{"load":{"1":0.21,"15":0.16,"5":0.13,"norm":{"1":0.0263,"15":0.02,"5":0.0163}}}}}}
2018-11-27T08:06:35.486Z INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":3140},"total":{"ticks":76720,"time":{"ms":36},"value":76720},"user":{"ticks":73580,"time":{"ms":36}}},"info":{"ephemeral_id":"a3899fbb-9b46-4ea1-afb9-dfc5c3717390","uptime":{"ms":167550018}},"memstats":{"gc_next":47626784,"memory_alloc":23821584,"memory_total":1596579352}},"filebeat":{"harvester":{"open_files":9,"running":24}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":4117}}},"registrar":{"states":{"current":13}},"system":{"load":{"1":0.12,"15":0.15,"5":0.12,"norm":{"1":0.015,"15":0.0188,"5":0.015}}}}}}
2018-11-27T08:06:58.372Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2018-11-27T08:06:58.372Z INFO [publish] pipeline/retry.go:168   done
2018-11-27T08:06:59.372Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure
2018-11-27T08:06:59.372Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://elasticsearch-local-svc:9200))
2018-11-27T08:06:59.372Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2018-11-27T08:06:59.372Z INFO [publish] pipeline/retry.go:191   done
2018-11-27T08:06:59.372Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2018-11-27T08:06:59.372Z INFO [publish] pipeline/retry.go:168   done
2018-11-27T08:06:59.373Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.3.2
2018-11-27T08:06:59.375Z INFO template/load.go:129 Template already exists and will not be overwritten.
2018-11-27T08:06:59.375Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://elasticsearch-local-svc:9200)) established
2018-11-27T08:06:59.375Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2018-11-27T08:06:59.375Z INFO [publish] pipeline/retry.go:191   done

In addition I am also running a similar stack with Amazon (Elasticsearch Service) with the same Filebeat configuration (except the destination) and there everything works fine. Does anyone know why this error is happening in my local EFK?
Reply all
Reply to author
Forward
0 new messages