Troubleshooting events not forwarding

[Darren S.]

Greetings,

I have a two tier Fluentd setup, forwarding logs from an origin host to a central relay that logs to SaaS.

I'm having trouble getting one set of logs to forward for delivery, hoping for some tips on troubleshooting. At the origin, I have four in_tail sources set up to read files. 3 of them read a single file path each, and are successfully forwarding. The fourth reads a wildcard path, and doesn't appear to be forwarding events. It is reading events properly, as they can be output locally using stdout output.

Origin Fluentd:

- td-agent 4.3.0-1

Relevant config:

  <label @FORWARD>                                                                                                    
    <filter **>                                                                                                        
      @type record_transformer                                                                                        
      <record>                                                                                                        
        host sensor                                                                                                   
        tag ${tag}                                                                                                    
      </record>                                                                                                        
    </filter>                                                                                                          
    <match **>                                                                                                        
      @type forward                                                                                                    
      @id output_system_forward                                                                                        
      <server>                                                                                                        
        name "fluentd.example.com"                                                                              
        host "fluentd.example.com"                                                                              
        port 5144                                                                                                      
      </server>                                                                                                        
      <buffer>                                                                                                        
        @type "file"                                                                                                  
        path "/var/log/td-agent/forward.*.buffer"
        flush_mode interval
        flush_interval 10s
      </buffer>
    </match>
  </label>

  <source>

    @type tail                                                                                                        

    @id suricata_eve
    @label @FORWARD
    tag "suricata.log.eve"
    path "/var/log/suricata/eve.json"
    pos_file "/var/log/td-agent/suricata_json.pos"
    <parse>
      @type "json"
      unmatched_lines
    </parse>

  </source>

  <source>
    @type tail
    @id suricata_fast
    @label @FORWARD
    tag "suricata.log.fast"
    path "/var/log/suricata/fast.log"
    pos_file "/var/log/td-agent/suricata_fast.pos"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id suricata_log
    @label @FORWARD
    tag "suricata.log.main"
    path "/var/log/suricata/suricata.log"
    pos_file "/var/log/td-agent/suricata_log.pos"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>

  <source>

    # This source is not being forwarded

    @type tail
    @id zeek_json
    @label @FORWARD
    tag "zeek.*"
    path "/opt/zeek/logs/current/json_streaming_*.log"
    pos_file "/var/log/td-agent/zeek_json.pos"
    follow_inodes true
    enable_stat_watcher false
    <parse>
      @type "json"
      unmatched_lines
    </parse>
  </source>

When I run this configuration locally using a @type stdout label , matching events are output to standard output as expected, e.g.:

# td-agent  -c /etc/td-agent/config.d/zeek.conf

2021-12-23 15:29:00 -0700 [info]: parsing config file is succeeded path="/etc/td-agent/config.d/zeek.conf"            
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.4'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-kafka' version '0.17.3'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-s3' version '1.6.1'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2021-12-23 15:29:00 -0700 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2021-12-23 15:29:00 -0700 [info]: gem 'fluentd' version '1.14.3'

...

2021-12-23 15:29:00 -0700 [info]: starting fluentd-1.14.3 pid=51428 ruby="2.7.5"
2021-12-23 15:29:00 -0700 [info]: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "-c", "/etc/td-agent/config.d/zeek.conf", "--under-supervisor"]
2021-12-23 15:29:00 -0700 [info]: adding match in @STDOUT pattern="**" type="stdout"
2021-12-23 15:29:00 -0700 [info]: adding source type="tail"
2021-12-23 15:29:00 -0700 [info]: #0 starting fluentd worker pid=51433 ppid=51428 worker=0
2021-12-23 15:29:00 -0700 [info]: #0 [zeek_json] following tail of /opt/zeek/logs/current/json_streaming_tunnel.log

...

2021-12-23 15:29:00 -0700 [info]: #0 [zeek_json] following tail of /opt/zeek/logs/current/json_streaming_files.log
2021-12-23 15:29:00 -0700 [info]: #0 fluentd worker is now running worker=0
2021-12-23 15:29:01.853095530 -0700 zeek.opt.zeek.logs.current.json_streaming_syslog.log: {"_path":"syslog","_write_ts"
:"2021-12-23T22:29:00.033922Z","ts":"2021-12-23T22:29:00.033922Z","uid":"C27cbicftoYXVkzG1","id.orig_h":"10.0.1.2","id.orig_p":5140,"id.resp_h":"10.0.1.1","id.resp_p":514,"proto":"udp","facility":"CRON","severity":"INFO","message":"Dec 23 15:29:00 /usr/sbin/cron[7075]: (root) CMD (/usr/sbin/newsyslog)"}

...

Forwarder Fluentd (fluentd.example.com):

- fluentd-1.14.3

When I run this test configuration, the events I expect to see (that would match on "match **") are not output, leading me to believe that they're not being forwarded. The events forwarded from the other sources are masked out here with a @type null match, but without that they are output to standard output.

$ doas fluentd -c /etc/fluent/zeek-test.conf
2021-12-23 15:24:22 -0700 [info]: parsing config file is succeeded path="/etc/fluent/zeek-test.conf"
2021-12-23 15:24:22 -0700 [info]: gem 'fluentd' version '1.14.3'                          
2021-12-23 15:24:22 -0700 [info]: gem 'fluent-plugin-sumologic_output' version '1.7.3'
2021-12-23 15:24:22 -0700 [info]: gem 'fluent-plugin-sumologic_output' version '1.7.1'
2021-12-23 15:24:22 -0700 [info]: gem 'fluentd' version '1.11.1'
2021-12-23 15:24:22 -0700 [info]: using configuration file: <ROOT>        
  <source>
    @type forward
    @label @STDOUT
    port 5144
    bind "0.0.0.0"
  </source>
  <label @STDOUT>

    # These are output to stdout correctly when not null'd

    <match suricata.**>
      @type null
    </match>
    <match **>

      # No other events are output; expecting to see zeek.** tagged events

      @type stdout
    </match>
  </label>
</ROOT>

- Darren

[Darren S.]

On Thursday, December 23, 2021 at 4:11:17 PM UTC-7 Darren S. wrote:

Greetings,

I have a two tier Fluentd setup, forwarding logs from an origin host to a central relay that logs to SaaS.

I'm having trouble getting one set of logs to forward for delivery, hoping for some tips on troubleshooting.

This was a n00b problem on my side; td-agent user did not have access to read the log files. I don't seem to be able to find any indication of this in the td-agent log (/var/log/td-agent/td-agent.log, etc.). I think I only became aware of this when I attempted a test explicitly dropping privileges to the td-agent user/group using --user and --group (Ruby's error about file not found or similar). Does td-agent currently consider it a fatal error, or at least log, when a specified input log file cannot be opened/read?

- Darren