Question about syslog input and time format

[Samuel Mutel]

Hello,

I am using fluentd 0.14 and receive syslog events in rfc5424.
In the error log I have the message below:

2017-07-26 10:23:59 +0200 [error]: #0 invalid input data="<142>2017-07-26T10:23:59.369Z rcentweb02 tibco-jmsagent - - [] TEST1_VERSION=1.4.7;TEST1_JOBID=;TEST1_FULLMSG=JMSAGENT-001;JMSAgent root@RCENTWEB02 (re)started." error_class=Fluent::TimeParser::TimeParseError error="invalid time format: value = 2017-07-26T10:23:59.369Z rcentweb02 tibco-jmsagent, error_class = ArgumentError, error = string doesn't match"

What's wrong in the syslog event format?

Thanks in advance.
Regards.

[Mr. Fiber]

> <142>2017-07-26T10:23:59.369Z rcentweb02 tibco-jmsagent - - [] TEST1_VERSION=1.4.7;TEST1_JOBID=;TEST1_FULLMSG=JMSAGENT-001;JMSAgent root@RCENTWEB02 (re)started.

Because your format is not rfc5424.

rfc5424's header is "<pri>version timestampe ..." but your log starts with "<pri>timestamp".

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Samuel Mutel]

I use log4j version 1.2.
I change my configuration to be like:

log4j.appender.SYSLOG.layout.conversionPattern=1 %d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'} %X{ipAddr} %X{compName} - - [] TEST1_VERSION=%X{compVers};TEST1_JOBID=%X{session};TEST1_FULLMSG=%m%n

The result is:

2017-07-27 10:13:02 +0200 [error]: #0 invalid input data="<142>1 2017-07-27T10:13:02.700Z rcentweb02 tibco-jmsagent - - [] TEST1_VERSION=1.4.7;TEST1_JOBID=COLLECT:1501143180000:507679258;TEST1_FULLMSG=JMSAGENT-061;Command MYIT.myit_localSystem_collect.36 of MYIT\\myit ended: 0:0:2.635" error_class=Fluent::TimeParser::TimeParseError error="invalid time format: value = 1 2017-07-27T10:13:02.700Z rcentweb02, error_class = ArgumentError, error = string doesn't match"

The wrong format is located on "<142>1" ?

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

Do you set message_format rfc5424 in syslog configuration?

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Samuel Mutel]

Here is my config file:

<source>
  @type syslog
  message_format rfc5424
  tag graylog2
</source>
<match graylog2.*.*>
  @type copy
  <store>
    @type gelf
    host XX.XX.XX.XX
    port 12201
    flush_interval 5s
    use_record_host true
    add_msec_time true
    protocol udp
  </store>
  <store>
    @type file
    path /var/log/td-agent/failed_records
  </store>
</match>

[Mr. Fiber]

If you use v0.14, message_format should be in <parse>.

<parse>

  message_format rfc5424

</parse>

  

BTW, this is bug of parser compat parameters.

v0.12 config should be worked.

We will fix it soon.

Masahiro

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Samuel Mutel]

Hello,

Where should I put this configuration?
I tried in root and in source but it appears to not work properly...

Regards,
Samuel Mutel.

[Mr. Fiber]

> Where should I put this configuration?

See: https://docs.fluentd.org/v0.14/articles/in_syslog#ltparsegt-directive

- Conf

<source>
  @type syslog
  port 5170
  tag test.syslog
  <parse>
    message_format rfc5424
  </parse>
</source>

<match test.**>
  @type stdout
</match>

- log

2017-02-06 22:14:15.003000000 +0900 test.syslog.mail.emerg: {"host":"192.168.0.1","ident":"fluentd","pid":"-","msgid":"-","extradata":"-","message":"Hi, from Fluentd!"}

- Test Ruby script

require 'socket'
require 'json'

log = "<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd!"

us = UDPSocket.open
sa = Socket.pack_sockaddr_in(5170, '0.0.0.0')
us.send(log, 0, sa)
us.close

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.