Filebeat Issue

[david.p...@gmail.com]

Good afternoon,

I am having an issue with my Wazuh installation.  My graphs stop drawing after a few minutes.  If I restart Filebeat, they start building again.

I don't ever see any errors in the log, it just stops writing new lines.  The last time it stopped, the last few lines in the log look like this:

2021-02-18T23:32:20.271Z        DEBUG   [harvester]     log/log.go:102  End of file reached: /var/ossec/logs/alerts/alerts.json; Backoff now.
2021-02-18T23:32:21.269Z        DEBUG   [logstash]      logstash/async.go:159   16 events out of 16 events sent to logstash host localhost:5000. Continue sending
2021-02-18T23:32:21.271Z        DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [189: 0, 16]
2021-02-18T23:32:21.271Z        DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=16, start-seq=2996, end-seq=3011

2021-02-18T23:32:21.271Z        DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:16
2021-02-18T23:32:21.271Z        DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack
2021-02-18T23:32:21.271Z        DEBUG   [acker] beater/acker.go:64      stateful ack    {"count": 16}
2021-02-18T23:32:21.271Z        DEBUG   [registrar]     registrar/registrar.go:345      Processing 16 events
2021-02-18T23:32:21.271Z        DEBUG   [registrar]     registrar/registrar.go:315      Registrar state updates processed. Count: 16
2021-02-18T23:32:21.271Z        DEBUG   [registrar]     registrar/registrar.go:400      Write registry file: /var/lib/filebeat/registry

When I look at the ossec log I still see new alerts come in.

Does anyone have any suggestions?

[Jesus Linares]

Hi David,

That is weird behavior. Some questions:

1. Where are you running Filebeat (instance, Docker, Kubernetes, etc)? 
2. Is it possible that you are somehow changing the registry file (/var/lib/filebeat/registry)?

Please, share your Filebeat configuration.

[david.p...@gmail.com]

Filebeat is running on the same AWS instance as the rest of the stack is, except for Elasticsearch (I'm using the AWS provided service for that).

The registry does constantly have the current timestamp.  When I run lsof I don't see it in the output:

[a.dramage@wazuh ~]$ sudo lsof | grep /var/lib/filebeat/registry
[a.dramage@wazuh ~]$

My config is below.  I'm skipping comments in the name of brevity:

queue.mem:
  events: 8064
  flush.min_events: 64
  flush.timeout: 5s

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/ossec/logs/alerts/*.json

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:
  host: "localhost:5601"

output.logstash:
  hosts: ["localhost:5000"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logging.level: debug

I should mention that I have a cron job that's restarting Filebeat every five minutes.  It "works" but it just doesn't feel like the right way to handle the problem.