Timestamp loses timezone data from rsyslog to fluentd

[Nitin Abhishek]

Hi,

Not sure if this is a bug, or an error in configuration on my part. Let me know if I should open an issue on github instead.

fluentd version: v0.12

Environment information: Docker FROM fluent/fluentd:v0.12-debian

Problem explanation:

Timestamp in the message at various levels:

 

rsyslog (using template RSYSLOG_ForwardFormat) in /var/log/syslog:

2017-02-15T10:36:47.257774+02:00 pi01 pi: SysTime is: Wed Feb 15 10:36:47 EET 2017

fluentd (out_stdout):

2017-02-15 10:36:47 +0000 syslog.user.notice: {"host":"pi01","ident":"pi","message":"SysTime is: Wed Feb 15 10:36:47 EET 2017"}

The timezone suffix is reset to +0000 when using the config below, and so when elasticsearch plugin sets the unix epoch time as the @timestamp field, it's +0200 hours ahead of actual time that the log was generated. 

For what it's worth, the machine running rsyslog and the docker host were on two different time zones, as the docker container was not explicitly set a timezone, and it defaulted to UTC.

# fluentd/conf/fluent.conf

<source>

  @type forward

  port 24224

  bind 0.0.0.0

  tag docker

</source>

<source>

  @type syslog

  port 5140

  bind 0.0.0.0

  tag syslog

</source>

<match **>

  @type copy

  <store>

    @type stdout

  </store>

  <store>

    @type elasticsearch

    host elasticsearch

    port 9200

    logstash_format true

    logstash_prefix fluentd

    logstash_dateformat %Y%m%d

    include_tag_key true

    type_name fluentd

    tag_key @log_name

    flush_interval 1s

  </store>

</match>

[Mr. Fiber]

Sorry for the delay reply.

> 2017-02-15T10:36:47.257774+02:00 pi01 pi: SysTime is: Wed Feb 15 10:36:47 EET 2017

Does rsyslog send same line to fluent's in_syslog?

fluentd's in_syslog now assumes rfc3164 format by default.

rfc3164's time_format is "%b %d %H:%M:%S", not "%Y-%m-%dT%H:%M:%S".

So with default in_syslog configuration, in_syslog seems to raise an error for invalid format.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nitin Abhishek]

No problem, I found a temprorary workaround, that syncs the timezones of the syslog sending server and the fluentd docker container. If the two systems are in the same timezone, the loss of timezone data doesn't seem to affect the final timestamp. But that would still break if the docker host running fluentd and syslog sending server were in different timezones.

I did try most default templates provided with rsyslog, all with similar result. I am currently using the line
    $ActionFileDefaultTemplate RSYSLOG_ForwardFormat

in rsyslog.conf, which is sending data in rfc3164 format according to this page. I had also tried "date-rfc3164" in the templates, with the same result.

I have attached rsyslog.conf, should it be needed.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/m2LFzZGsfP0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.