ignore specific files when tailing logs?

[Andrew Johnstone]

I'm wondering if someone can point me in the right direction.

With the in_tail plugin is there a way to exclude specific files?

I wish to ignore the docker container running fluentd from sending logs, the pattern in use is as follows.

If new files are created by creating new docker containers, I wish this to be logged too.

path /var/lib/docker/containers/*/*-json.log

The full configuration can be found below.

Thanks

Andy

<ROOT>

  <source>

    type tail

    path /var/lib/docker/containers/*/*-json.log

    pos_file /var/log/fluentd-docker.pos

    time_format %Y-%m-%dT%H:%M:%S

    tag docker.*

    format json

    refresh_interval 1m

  </source>

  <match docker.var.lib.docker.containers.*.*.log>

    type record_reformer

    container_id ${tag_parts[5]}

    tag docker.all

  </match>

  <match docker.all>

    type copy

    <store>

      type file

      path /var/log/docker/*.log

      format json

      include_time_key true

      log_level debug

    </store>

    <store>

      type s3

      s3_bucket photobox-services-logs

      s3_region eu-west-1

      path logs/docker

      s3_object_key_format %{path}/events/ts=%{time_slice}/%{container_id}/events_%{index}-%{hostname}.%{file_extension}

      buffer_path /var/log/fluent/s3

      time_slice_format %Y-%m-%d-%H

      time_slice_wait 10m

      utc 

      buffer_chunk_limit 256m

      flush_interval 1m

    </store>

  </match>

</ROOT>

[Andrew Johnstone]

Similar to http://logstash.net/docs/1.4.2/inputs/file#exclude

[Mr. Fiber]

Hi Andrew,

Hmm... * is not enough?

I think exclude seems useful but adhoc parameter.

What the pattern do you want to tail and ignore?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Johnstone]

Hi Masahiro,

As an example, this is what it would be tailing and any new files, however I would specifically want to ignore one file such as...

/mnt/docker/containers/a9090836e1b8efb44f3aaa067477ce46196d1c1e80a9230a6bdf010e9d870f43/a9090836e1b8efb44f3aaa067477ce46196d1c1e80a9230a6bdf010e9d870f43-json.log

Listing

sudo ls -al  /mnt/docker/containers/*/*-json.log

-rw------- 1 root root 878543517 Mar  9 09:34 /mnt/docker/containers/379cc31c1efd3b4df5c8b2df7cfc1a6f13c0aa5e58031b00948b44ce89ce6175/379cc31c1efd3b4df5c8b2df7cfc1a6f13c0aa5e58031b00948b44ce89ce6175-json.log

-rw------- 1 root root     14613 Feb 27 09:44 /mnt/docker/containers/643bbe4866ca5d6e2f3e81234e94a1f6394a8da00c3eb1e86d38c208a6931294/643bbe4866ca5d6e2f3e81234e94a1f6394a8da00c3eb1e86d38c208a6931294-json.log

-rw------- 1 root root      2997 Feb 25 17:07 /mnt/docker/containers/730b48e949f3aaf98326fc26b7d96f032187a00e087113136a159b31a7a6001e/730b48e949f3aaf98326fc26b7d96f032187a00e087113136a159b31a7a6001e-json.log

-rw------- 1 root root     28169 Mar  3 17:33 /mnt/docker/containers/a9090836e1b8efb44f3aaa067477ce46196d1c1e80a9230a6bdf010e9d870f43/a9090836e1b8efb44f3aaa067477ce46196d1c1e80a9230a6bdf010e9d870f43-json.log

-rw------- 1 root root 745112686 Mar  9 09:35 /mnt/docker/containers/c76e5773b44592d08d883a74070934b5e4bab34b286e8f3faf7ee93bd74698b7/c76e5773b44592d08d883a74070934b5e4bab34b286e8f3faf7ee93bd74698b7-json.log

I started to write up some code to add exclude, to the tail plugin.

Thanks

Andy

[Andrew Johnstone]

I've submitted PR (https://github.com/fluent/fluentd/pull/561) build  https://travis-ci.org/fluent/fluentd/builds/53650182.

Hopefully this will be accepted :)

Thanks

Andy

[Lance N.]

You can put the filename in the tags and then match that tag with a 'format null' block.