loading logs in s3 with date partitions

[David Montgomery]

Hi,

I am using the s3 fluentd plugin.

I would like to create a bucket with paths like this:

mybucket/y=2012/m=11/d=06/h=08/myfile.gz

How to I dynamically create those buckets in that format?  Is there a plugin that will do that?  Can the fluentd s3 plugin do it?

If so then how?

Thanks

[Masahiro Nakagawa]

Hi David,

Currently, S3 plugin doesn't have such formatting feature.

Hmm... What about extracting the method which generates S3 path and

User overrides this method to customize the S3 path?

https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/out_s3.rb#L112

Extract points are from L112 to L125.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Justin Sermeno]

I believe you can hack this by putting a '/' into the time_slice configuration.

For example:

s3_object_key_format %{time_slice}_%{index}_%{hostname}.%{file_extension}

time_slice_format %Y-%m-%d/%Y-%m-%dT%H

https://github.com/campanja/fluent-output-router uses this in their readme, and we are also using the same trick in our deployment.