converting unix timestamp (secs since epoch) to datetime

[Jack Xue]

Hello:

Is there an existing plugin that can convert a unix timestmap JSON field to the format of 'YYYY-MM-DD HH:MM:SS'?

Thanks!

Jack

[Kiyoto Tamura]

Hi Jack,

One solution I can think of is, again, record_reformer. Time.at method should be callable like this:

<match foo.bar>

  my_time_field ${Time.at(my_time_field)}

</match>

should convert "my_time_field" from unix timestamp to YYYY-MM-DD HH:MM:SS Z, if you don't want the timezone field, you can use strftime like this:

<match foo.bar>
  my_time_field ${Time.at(my_time_field).strftime('%Y-%m-%d %H:%M:%S')}
</match>

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Check out Fluentd, the open source data collector to unify log management.

[Kellan Strong]

Hi Kiyoto,

I am trying to use this with a custom format. Is this possible? I am trying to use this against squid3 logs.

[Kiyoto Tamura]

Kellan-

I am assuming you are trying to parse a string that appears in squid3 logs? If so, you might want to look at the "time_format" parameter: http://docs.fluentd.org/articles/in_tail (search for "time_format")

[Kellan Strong]

Kiyoto,

Yes, the squid3 logs look something like this.

1419409823.123 USERNAME 200 https://www.google.com 456 TCP_ACCEPT 23

config would sorta look like this

<source>
  type tail
  path /var/log/squid3/access.log
  pos_file /var/log/td-agent//squid3.log.pos
  tag squid3
  format /^[^ ]* (?<username>[^ ]*) (?<code>[^ ]*) (?<url>[^ ]*) (?<size>[^ ]*) (?<tcp_code>[^ ]*) (?<other>[^ ]*)$/
</source>

Since I can't get the time I am ignore it for now. But from what you are saying is that it should look like this now? From what I am see is that your actually calling ruby inside the config.

something like t = Time.at(1419409823.123).strftime('%Y-%m-%d %H:%M:%S')
                     

> t = Time.­at(1419409­823.123).s­trftime('%­Y-%m-%d %H:%M­:%S')

=> "2014-12-24 08:30:23"

<source>
  type tail
  path /var/log/squid3/access.log
  pos_file /var/log/td-agent//squid3.log.pos
  tag squid3
  format /^(?<time>[^ ]*) (?<username>[^ ]*) (?<code>[^ ]*) (?<url>[^ ]*) (?<size>[^ ]*) (?<tcp_code>[^ ]*) (?<other>[^ ]*)$/
  time_format ${Time.at(time_format).strftime('%Y-%m-%d %H:%M:%S')}
</source>

[Kiyoto Tamura]

I see what you are trying to do now.

In Fluentd, time is internally held as Unix timestamp. So, in your case, you need to parse it like this:

<source>
  type tail
  path /var/log/squid3/access.log
  pos_file /var/log/td-agent//squid3.log.

pos

  tag squid3
  format /^(?<time>[^ ]*) (?<username>[^ ]*) (?<code>[^ ]*) (?<url>[^ ]*) (?<size>[^ ]*) (?<tcp_code>[^ ]*) (?<other>[^ ]*)$/

  time_format %S
</source>

The "%S" parses the time as Unix epoch. At this point, if you stdout the events with tag = squid3, it looks like this:

2015-01-07 05:59:14 +0000 squid3: {"username":"USERNAME","code":"200","url":"https://www.google.com","size":"456","tcp_code":"TCP_ACCEPT","other":"23"}

Then, if you want to add a time field with a particular format, you can use the record_transformer filter (for v0.12 and above). This is like record_transformer, but it's now a filter.

<source>
  type tail
  path /var/log/squid3/access.log
  pos_file /var/log/td-agent//squid3.log.

pos

  tag squid3
  format /^(?<time>[^ ]*) (?<username>[^ ]*) (?<code>[^ ]*) (?<url>[^ ]*) (?<size>[^ ]*) (?<tcp_code>[^ ]*) (?<other>[^ ]*)$/

  time_format %S
</source>

<filter squid3>
  type record_transformer
  enable_ruby
  <record>
    my_time ${Time.at(time).strftime('%Y-%m-%d %H:%M:%S')}
  </record>
</filter>

<match squid3>
  type stdout
</match>

Then, you get data like this:

2015-01-07 06:09:14 +0000 squid3: {"username":"USERNAME","code":"200","url":"https://www.google.com","size":"456","tcp_code":"TCP_ACCEPT","other":"23","my_time":"2015-01-07 06:09:14"}

Do you think this will work for you? If you are on v0.10.xx branch, then you need to replace the <filter> block with the record_reformer output like this

<match squid3>
  type record_transformer
  enable_ruby
  <record>
    my_time ${Time.at(time).strftime('%Y-%m-%d %H:%M:%S')}
  </record>

  tag squid3_transformed

</match>

<match squid3_transformed>
  ...

</match>

[Kellan Strong]

Awesome this makes this easy to understand.

Thank for this!

[Yogesh Srivastava]

Hello Jack,

In case of squid log, you can also change it by changing log format in squid.conf file. PFB the format.

logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %[un %Sh/%<a %mt

Thanks
Yogesh