Docker Logging to fluentd via fluentd aggregator

[twelcome]

Hi List

I've got fluentd agents (td-agent) logging to fluentd aggregators and thereafter to elasticsearch (and kibana).

I'm struggling to add a docker logging to td-agent which will forward logs to fluentd aggregators and appear in Kibana.

I can get docker sending logs via td-agent directly to elasticsearch, however I want to use fluentd aggregators to scale better.

The basic bloc of configuration I'm using to send to aggregators is this, however nothing appears in Kibana, and doesn't appear to be sent to the aggregators:

----

<source>
  @type forward
  tag      docker.daemon
  @label   @mainstream
  port 12116
</source>

.

.

.

<label @mainstream>
  #log forwarding for docker only:
 <match docker.**>
    @type forward
    # primary host
    <server>
      host active-aggregator-fluentd.lol.net
      port 12114
    </server>
    # use secondary host
    <server>
      host backup-aggregator-fluentd.lol.net
      port 12114
      standby
    </server>
    expire_dns_cache 0
    <buffer tag>
      @type file
      flush_at_shutdown false
      flush_interval 15s
      path /var/spool/td-agent/docker/buffer/backlog
    </buffer>
    ignore_network_errors_at_startup true
    time_as_integer true
  </match>
  # Log Forwarding
  <match **>
    @type forward
    # primary host
    <server>
      host active-aggregator-fluentd.lol.net
      port 12114
    </server>
    # use secondary host
    <server>
      host backup-aggregator-fluentd.lol.net
      port 12114
      standby
    </server>
    expire_dns_cache 0
    <buffer tag>
      @type file
      flush_at_shutdown false
      flush_interval 15s
      path /var/spool/td-agent/buffer/backlog
    </buffer>
    ignore_network_errors_at_startup true
    time_as_integer true
  </match>
</label>

----

This block is added to the total td-agent configuration, which is as follows (apologies, it's rather long):

Note: the rest of this configuration for logging syslogs and audit logs is fine.

---

<system>
  log_level debug
</system>
<source>
  @type forward
  tag      docker.daemon
  @label   @mainstream
  port 12116
</source>
<source>
  @type    tail
  path     /data/log/kubernetes-slave.log
  pos_file /var/spool/td-agent/kubernetes-slave.log
  format /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?<host>[^ ]*) (@ (?<facility>[^ .]*)[.](?<priority>[^ ]*) |)(?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
  time_format %b %d %H:%M:%S
  tag      kubernetes.slave
  @label   @mainstream
</source>
<source>
  @type    tail
  path     /var/log/syslog
  pos_file /var/spool/td-agent/syslog
  format /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?<host>[^ ]*) (@ (?<facility>[^ .]*)[.](?<priority>[^ ]*) |)(?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
  time_format %b %d %H:%M:%S
  tag      system.syslog
  @label   @mainstream
</source>
<source>
  @type    tail
  path     /data/log/kubernetes-master.log
  pos_file /var/spool/td-agent/kubernetes-master.log
  format /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?<host>[^ ]*) (@ (?<facility>[^ .]*)[.](?<priority>[^ ]*) |)(?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
  time_format %b %d %H:%M:%S
  tag      kubernetes.master
  @label   @mainstream
</source>
<source>
  @type    tail
  path     /data/log/kubernetes.log
  pos_file /var/spool/td-agent/kubernetes.log
  format /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?<host>[^ ]*) (@ (?<facility>[^ .]*)[.](?<priority>[^ ]*) |)(?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
  time_format %b %d %H:%M:%S
  tag      kubernetes.service

  @label   @mainstream
</source>
<source>
  @type    tail
  path     /var/log/auth.log
  pos_file /var/spool/td-agent/auth.log
  format /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?<host>[^ ]*) (@ (?<facility>[^ .]*)[.](?<priority>[^ ]*) |)(?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
  time_format %b %d %H:%M:%S
  tag      system.auth
  @label   @mainstream
</source>
<match **>
  @type stdout
</match>
<label @mainstream>
  #log forwarding for docker only:
 <match docker.**>
    @type forward
    # primary host
    <server>
      host active-aggregator-fluentd.lol.net
      port 12114
    </server>
    # use secondary host
    <server>
      host backup-aggregator-fluentd.lol.net
      port 12114
      standby
    </server>
    expire_dns_cache 0
    <buffer tag>
      @type file
      flush_at_shutdown false
      flush_interval 15s
      path /var/spool/td-agent/docker/buffer/backlog
    </buffer>
    ignore_network_errors_at_startup true
    time_as_integer true
  </match>
  # Log Forwarding
  <match **>
    @type forward
    # primary host
    <server>
      host active-aggregator-fluentd.lol.net
      port 12114
    </server>
    # use secondary host
    <server>
      host backup-aggregator-fluentd.lol.net
      port 12114
      standby
    </server>
    expire_dns_cache 0
    <buffer tag>
      @type file
      flush_at_shutdown false
      flush_interval 15s
      path /var/spool/td-agent/buffer/backlog
    </buffer>
    ignore_network_errors_at_startup true
    time_as_integer true
  </match>
</label>

---

I'm testing by generating a docker message as follows, which appears to send without error:

---

for i in `seq 1 10`;do echo "$i " - $(docker run --log-driver=fluentd --log-opt tag="docker.{{.ID}}" --log-opt fluentd-address=10.10.0.1:12113 python:alpine echo "($i) `date` vibranium");done

---

Any suggestions on how to trace why this is not sending to the fluentd-aggregators would be helpful. Note, that when I use -vv or "@log_level debug" statements I get no visible information ion the td-agent.log relevant to this docker match or source bloc.

Thanks,

Traiano

[Mr. Fiber]

> however nothing appears in Kibana, and doesn't appear to be sent to the aggregators:

Does this mean buffer is growing but not flushed?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Traiano Welcome]

Hi

On Wed, May 30, 2018 at 1:55 AM, Mr. Fiber <repea...@gmail.com> wrote:

> however nothing appears in Kibana, and doesn't appear to be sent to the aggregators:

Does this mean buffer is growing but not flushed?

Turns out I just needed to have the correct matching configuration on the aggregator side :-( Works fine as derived from the fluentd documentation.