fluent-plugin-s3 csv format

[Mani kandan]

I am looking to write logs using a fixed format similar to Tomcat / IIS logs and store them in S3.

Expected format:

Date[tab]Value1[tab]Value2[tab]Value3

Normally, in file output plugins I would be able to do this using csv format (Ref: http://docs.fluentd.org/articles/out_file#format), but here it doesn't seem to be supported.

Can someone please help me out, if there is any workaround for this ? or is this something under development ?

[Mr. Fiber]

s3 plugin also has format parameter.

https://github.com/fluent/fluent-plugin-s3#output-configuration

This doesn't work on your environment?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mani kandan]

The format parameter works, am able to use ltsv, json and single_value, but I am saying that it doesn't support csv at the moment.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

> it doesn't support csv at the moment.

Could you write more details of this point?

http://docs.fluentd.org/articles/formatter-plugin-overview

These built-in formats should work.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Mani kandan]

Sure, here is the config that I tried to use,

<match app.S3>

  @type s3

  format csv

  delimiter \t

  aws_key_id XXXXXXXX

  aws_sec_key XXXXXXXX

  s3_bucket XXXXX

  s3_region us-east-1

  path Logs/

  buffer_path  /var/log/td-agent/buffer/s3

  time_slice_format %Y%m%d

  time_slice_wait 10s

  utc

  buffer_chunk_limit 256m

</match>

And all log messages started writing in blank / empty lines in the buffer file. But when I used ltsv / json it was working as expected.

[Mr. Fiber]

> And all log messages started writing in blank / empty lines in the buffer file.

Because you don't specify fields parameter. Here is format example:

JSON: {"f1":"a","f2":"b","f3":"c"}

LTSV: f1:a f2:b f3:c

CSV: a,b,c,

You can see CSV can't contain each field, e.g. f1, in the line unlike LTSV / JSON.

In general, CSV has a header for specifying fields in the head of file.

In streaming, applying header is hard because header is unclear.

So you need to specify fields parameter to point which fields are stored.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Mani kandan]

Ah Ok, Sorry for the confusion. Is there any other way to just take all fields instead of providing all the fields here ?

[Mr. Fiber]

> Is there any other way to just take all fields instead of providing all the fields here ?

No way because json doesn't guaratee the order of fields.

So automatic field listing is error prone.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Mani kandan]

Ok cool, Thanks for helping me out. I also realized that some of the fields are dynamic and so CSV might not make sense for me, will look into other options.