Logs form Filebeat do not reach Elasticsearh

[g...@recongate.com]

Hi, I am running a local EFK stack on my kubernetes cluster and I can see my logs in Kibana only after restarting my Filebeat pods. Once restarted, the logs from the restart time reach their destination but a few minutes later they stop. 

This is a the Filebeat log:

2018-11-27T08:05:58.369Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer

2018-11-27T08:05:58.369Z INFO [publish] pipeline/retry.go:191   done

2018-11-27T08:06:05.486Z INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":3130,"time":{"ms":8}},"total":{"ticks":76680,"time":{"ms":8},"value":76680},"user":{"ticks":73550}},"info":{"ephemeral_id":"a3899fbb-9b46-4ea1-afb9-dfc5c3717390","uptime":{"ms":167520018}},"memstats":{"gc_next":47624032,"memory_alloc":24670472,"memory_total":1596393400,"rss":-16384}},"filebeat":{"harvester":{"open_files":9,"running":24}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":-4,"batches":1,"failed":9,"total":5},"read":{"bytes":1529},"write":{"bytes":4285}},"pipeline":{"clients":2,"events":{"active":4117,"retry":19}}},"registrar":{"states":{"current":13}},"system":{"load":{"1":0.21,"15":0.16,"5":0.13,"norm":{"1":0.0263,"15":0.02,"5":0.0163}}}}}}

2018-11-27T08:06:35.486Z INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":3140},"total":{"ticks":76720,"time":{"ms":36},"value":76720},"user":{"ticks":73580,"time":{"ms":36}}},"info":{"ephemeral_id":"a3899fbb-9b46-4ea1-afb9-dfc5c3717390","uptime":{"ms":167550018}},"memstats":{"gc_next":47626784,"memory_alloc":23821584,"memory_total":1596579352}},"filebeat":{"harvester":{"open_files":9,"running":24}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":2,"events":{"active":4117}}},"registrar":{"states":{"current":13}},"system":{"load":{"1":0.12,"15":0.15,"5":0.12,"norm":{"1":0.015,"15":0.0188,"5":0.015}}}}}}

2018-11-27T08:06:58.372Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer

2018-11-27T08:06:58.372Z INFO [publish] pipeline/retry.go:168   done

2018-11-27T08:06:59.372Z ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure

2018-11-27T08:06:59.372Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://elasticsearch-local-svc:9200))

2018-11-27T08:06:59.372Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer

2018-11-27T08:06:59.372Z INFO [publish] pipeline/retry.go:191   done

2018-11-27T08:06:59.372Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer

2018-11-27T08:06:59.372Z INFO [publish] pipeline/retry.go:168   done

2018-11-27T08:06:59.373Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.3.2

2018-11-27T08:06:59.375Z INFO template/load.go:129 Template already exists and will not be overwritten.

2018-11-27T08:06:59.375Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://elasticsearch-local-svc:9200)) established

2018-11-27T08:06:59.375Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer

2018-11-27T08:06:59.375Z INFO [publish] pipeline/retry.go:191   done

In addition I am also running a similar stack with Amazon (Elasticsearch Service) with the same Filebeat configuration (except the destination) and there everything works fine. Does anyone know why this error is happening in my local EFK?