s3 - include tailed_path in s3 path or key?

[david goodine]

Hello,

I am using fluentd to forward Hashicorp nomad logs to an s3 bucket.  I have gotten logs to be written to the bucket, but I need to be able to separate the logs by service name.  Each service logs to a unique log name, plus extensions, ie. service-foo.stderr.123.  My goal is to have the service name by part of the s3 path:

<system>
  log_level info
</system>
<source>
  @type tail
  path "/mnt/nomad/alloc/*/alloc/logs/*"
  path_key "tailed_path"
  pos_file "/var/log/fluentd/nomad_log.pos"
  tag "s3.nomad"
  <parse>
    @type "none"
    unmatched_lines
  </parse>
</source>
<filter s3.nomad>
  type record_transformer
  enable_ruby
  <record>
    log_file ${record['tailed_path']}
    hostname ${hostname}
    tailed_path ${File.basename(record["tailed_path"]).sub(/(?<=.)\..*/, '')}
  </record>
</filter>
<match s3.nomad>
  @type s3
  aws_key_id xxxxxx
  aws_sec_key xxxxxx
  s3_bucket "enverus-ea-log-archive-dev"
  path "nomad-logs/%Y/%m/%d/%{tailed_path}/"
  time_slice_format %Y%m%d%H
  <buffer>
    @type "file"
    path "/var/log/fluentd"
    timekey 120
    timekey_wait 30s
    chunk_limit_size 256m
  </buffer>
</match>

This config does not work, though.  Instead of logs in nomad-logs/2021/04/30/service-foo.stderr.123/2021043002_9.gz, my s3 keys are like this:

s3://enverus-ea-log-archive-dev/nomad-logs/2021/04/30//2021043002_9.gz

Is it possible to use tailed_path in the s3 path?

Thanks for you time!

-Dave

[david goodine]

I ended up using another plugin to rewrite the tag (rewrite_tag_filter), and then used the tag:

<source>
  @type tail
  path /mnt/nomad/alloc/*/alloc/logs/*
  path_key tailed_path
  pos_file /var/log/fluentd/nomad_log.pos

  <parse>
    @type none
  </parse>
  tag s3.nomad

</source>

<filter s3.nomad>
 type record_transformer
 enable_ruby
 <record>
    log_file "${record['tailed_path']}"
    hostname "${hostname}"
    tailed_path ${File.basename(record["tailed_path"]).sub(/(?<=.)\..*/, '')}
 </record>
</filter>

<match s3.nomad>

  @type rewrite_tag_filter
  <rule>
    key     tailed_path
    pattern /(.*)/
    tag service.$1
  </rule>
</match>

<match service.*>
  @type s3
 
  aws_key_id foo
  aws_sec_key bar

  s3_bucket enverus-ea-log-archive-dev
  path "nomad-logs/%Y/%m/%d/${tag}/"

  <buffer tag,time>

    @type file
    path /var/log/fluentd
    timekey 120
    timekey_wait 30s
    chunk_limit_size 256m
  </buffer>

  time_slice_format %Y%m%d%H
</match>