S3 Output Plugin - time_slice returning epoch time

[Christina Carkner]

I am trying to use copy to send a json formatted log file to both Logentries and S3.  I'd like one log file sent to S3 hourly.

Logentries is working fine, however when I try to send the file to S3, the file name that shows there always has the time_slice as epoch (19700101-00.json for example) and it is sending the log much more frequently than once an hour. Sometimes as much as once every 15 seconds. This is why I removed the _%{index} because otherwise I ended up with tons of ~20MB files in S3.

I tried multiple different iterations of settings, which didn't help. I also tried turning on debug logging but I'm not seeing anything.

This is the section of the config file around this.

<match logfile>

  @type copy

  <store>

    @type simple-logentries

    token foobar

    flush_interval 1s

  </store>

  <store>

    @type s3

    # This is the format the time stamp is in the json log

    format json

    #include_time_key true

    #time_key EventTime

    #time_format %Y-%m-%d %H:%M:%S

    store_as json

    buffer_type file

    buffer_path /var/log/td-agent/s3

    buffer_chunk_limit 1g

    buffer_queue_limit 2g

    flush_at_shutdown true

    time_slice_format %Y%m%d-%H

    time_slice_wait 10m

    aws_key_id foo

    aws_sec_key bar

    s3_region us-west-2

    s3_bucket backup.logs

    path WebLogs

    s3_object_key_format %{path}/%{hostname}/%{time_slice}.%{file_extension}

  </store>

</match>

[Mr. Fiber]

Please write your entire fluentd logs and log example here.

S3 plugin is widely used plugin so basic behaviour doesn't have a bug.

Hard to answer with only your configuration so we need more detailed information.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christina Carkner]

I've tried a couple of variations on my config file, I attached the current configuration to this reply.

I attached the fluentd logs that I have immediately after I restart. 

The input is a W3C IIS log that has been turned to JSON format by nxlog. 

Entries look like this after being changed by nxlog:

{

  "SourceModuleName": "xxxx_W3SVC1",

  "SourceModuleType": "im_file",

  "Hostname": "xxxx",

  "Datacenter": "xxxx",

  "log_type": "Azure Pack Website Log",

  "date": "2016-09-27",

  "time": "20:00:00",

  "s_ip": "xxxx",

  "cs_method": "GET",

  "cs_uri_stem": "xxxx",

  "cs_uri_query": "xxxx",

  "s_port": 443,

  "cs_username": "xxxx",

  "c_ip": "xxxx",

  "cs_User_Agent": null,

  "cs_host": "xxxx",

  "sc_status": 302,

  "sc_substatus": 0,

  "sc_win32_status": 64,

  "sc_bytes": 0,

  "cs_bytes": 136,

  "time_taken": 46,

  "EventTime": "2016-09-27 20:00:00",

  "tag": "azurePack"

}

In the buffer file output, the data looks like this:

1970-01-01 00:33:36     azurePack       {"SourceModuleName":"REDACTED_W3SVC1","SourceModuleType":"im_file","Hostname":"REDACTED","Datacenter":"xxxx","log_type":"Azure Pack Website Log","date":"2016-09-27","time":"22:03:19","s_ip":"x.x.x.x","cs_method":"GET","cs_uri_stem":"/404","cs_uri_query":"xxxx","s_port":443,"cs_username":"xxxx","c_ip":"xxxx","cs_User_Agent":"Mozilla/3.0+(compatible;+Indy+Library)","cs_host":"xxxx","sc_status":200,"sc_substatus":0,"sc_win32_status":0,"sc_bytes":30594,"cs_bytes":286,"time_taken":532,"EventTime":"2016-09-27 22:03:19"}

It does upload to S3, but it is doing so every minute rather than hour with a name of 1970010100_0.gz. The send to logentries works fine.

Essentially, it seems like for some reason time_slice_format isn't able to get the current time and is using the epoch?

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

You can check which input or output is wrong by inserting following filter.

<filter azurePack**>

  @type stdout

</filter>

If the time is no problem, output side setting is wrong.

If the time is wrong, input side setting is wrong.

I assume "EventReceivedTime" has wrong value.

For example, "foo".to_i is 0.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.