issue after the rsyslog's template change

[alnat2]

Hi!
Yesterday I changed the rsyslog's template
$template CFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
to
$template CFormat,"%timegenerated% %HOSTNAME% %syslogseverity-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n".
After that, fluentd stopped taking the logs from this host. No mail notification, not sent any data to elasticsearch.
System: ubuntu 12.04.04, fluentd 0.10.50.

[Masahiro Nakagawa]

Do you use in_syslog?

Did you change in_syslog's format option too?

http://docs.fluentd.org/articles/in_syslog

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[alnat2]

Hi Masahiro !

Yes, I use in_syslog.
Here is my syslog's directive:
<source>
  type syslog
  port 42185
  tag syslog
</source>

What do I need to change ?

[Kiyoto Tamura]

Hi alnat2;

Thanks for the info. I could produced it. I suppose you have something like this in your /etc/rsyslog.conf:

*.* @127.0.0.1:42185;CFormat

Masa-

This is because we assume that syslog format is left untouched, or if it is altered, it still starts with <priority_number>. What do you think is the best way to proceed here?

--

Check out Fluentd, the open source data collector for high-volume data streams

[Kiyoto Tamura]

alnat2-

On second thought, I am still not convinced your change directly caused this issue: even with your previous format, in_syslog should have had a parse error (hence no data in Fluentd from rsyslog)

Can you check if you have a line like this in your /etc/rsyslog.conf?

*.* @127.0.0.1:42185;CFormat

And if so, please change that to

*.* @127.0.0.1:42185

, restart rsyslog and see if you see data in Fluentd.

Thanks,

[Masahiro Nakagawa]

Hmm... I don't understand this problem yet.

In in_syslog format, we can parse two cases:

- default case

<6>Sep 11 00:00:00 localhost logger: foo

https://github.com/fluent/fluentd/blob/7b60188a59ec3677e8766418c6b0b7347e855fdf/lib/fluent/plugin/in_syslog.rb#L23

- priority started format with TextParser

<6>foo.bar Sep 11 00:00:00: message

https://github.com/fluent/fluentd/blob/7b60188a59ec3677e8766418c6b0b7347e855fdf/lib/fluent/plugin/in_syslog.rb#L22

Does alnat2 custom format generate different format message?

[Kiyoto Tamura]

>Does alnat2 custom format generate different format message?

Yes. It produces something like this:

Jun 21 19:11:55 precise64 info sudo: pam_unix(sudo:session): session opened for user root by vagrant(uid=1000)

Note that the leading <N> is missing. I think the problem arises from overwriting the default rsyslog format.

Kiyoto

[alnat2]

Hi guys!
After your assumptions, I decided to try a workaround.
I tested on http://fluentular.herokuapp.com this regular expression:
^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<severity>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?[^\:]*\: *(?<message>.*)$
There it is working correctly.
After that I changed the directive in my /etc/td-agent/td-agent.conf to:

<source>
  type syslog
  port 42185

  format/^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<severity>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?$
  tag syslog
</source>
And restart td-agent. But it doesn't work. No parse errors in the /var/log/td-agent/td-agent.log .

[Masahiro Nakagawa]

Could you paste the your td-agent.log here?

[alnat2]

2014-06-22 11:35:32 +0400 [info]: shutting down fluentd
2014-06-22 11:35:42 +0400 [info]: process finished code=0
2014-06-22 11:35:43 +0400 [info]: starting fluentd-0.10.50
2014-06-22 11:35:43 +0400 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-mixin-config-placeholders' version '0.2.4'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-elasticsearch' version '0.3.0'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-flume' version '0.1.1'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-mail' version '0.0.5'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-mongo' version '0.7.3'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-notifier' version '0.2.2'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-s3' version '0.4.0'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-scribe' version '0.10.10'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-td' version '0.10.20'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-td-monitoring' version '0.1.2'
2014-06-22 11:35:43 +0400 [info]: gem 'fluent-plugin-webhdfs' version '0.2.2'
2014-06-22 11:35:43 +0400 [info]: gem 'fluentd' version '0.10.50'
2014-06-22 11:35:43 +0400 [info]: using configuration file: <ROOT>

  <source>
    type syslog
    port 42185

    format /^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?[^\:]*\: *(?<mes$
    tag syslog
  </source>
  <match syslog**>
    type copy
    <store>
      type elasticsearch
      host 192.168.7.52
      logstash_format true
      flush_interval 60s
    </store>
 <store>
      type notifier
      <def>
        pattern check_syslog
        check string_find
        warn_regexp sshd
        crit_regexp sshd
        target_key_pattern ident
      </def>
    </store>
  </match>
  <match notification**>
    type mail
    host smtp.gmail.com
    port 587
    user us...@gmail.com
    password pass
    enable_starttls_auto true
    from us...@gmail.com
    to us...@gmail.com
    subject [URGENT] SSH EVENT
    out_keys target_tag, pattern, value, message_time
  </match>
</ROOT>
2014-06-22 11:35:43 +0400 [info]: adding source type="syslog"
2014-06-22 11:35:43 +0400 [info]: adding match pattern="syslog**" type="copy"
2014-06-22 11:35:43 +0400 [info]: adding match pattern="notification**" type="mail"

I deleted the duplicate rows

[Kiyoto Tamura]

alnat2,

Can you run Fluentd with "-vv" option again? I am pretty sure in_syslog is failing to match the incoming data.

Also, if my guess is correct, unless you stick to rsyslog's default format (which starts with <severity_number>), in_syslog can't parse it no matter what regex you put into the "format" parameter.

You have two workarounds:

1. Make sure that you do not deviate from the default rsyslogd plugin: as far as I can tell, your template "CFormat" is not adding any new information. So, at least for the traffic you are redirecting to Fluentd (on port 41285), can you make sure the format is the default one?

2. Extend/modify in_syslog: since in_syslog expects the payload to start with the severity number and your log format doesn't, you would need to modify the code. I created an GitHub Issue on this, so the situation may change in the future.

Kiyoto

[Kiyoto Tamura]

alnat2-

Also, if possible, can you share your /etc/rsyslogd.conf? That would be hugely helpful.

[alnat2]

In td-agent -vv output:
2014-06-22 20:23:41 +0400 [debug]: plugin/in_syslog.rb:128:receive_data_parser: invalid syslog message: "Jun 22 20:23:41  ...

My /etc/rsyslog.conf:

#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support
#$ModLoad immark  # provides --MARK-- message capability
$ModLoad imfile # Load the imfile input module

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#

$template CFormat,"%timegenerated% %HOSTNAME% %syslogseverity-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

$ActionFileDefaultTemplate CFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup adm

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf


*.* @192.168.7.52:42185

[Kiyoto Tamura]

Ok, your "fluentd -vv" format confirms my hypothesis: the syslog message sent to Fluentd is missing the priority/severity number at the beginning of the payload.

What puzzles me is that your rsysloc.conf setup works locally for me, at least with a local instance of Fluentd (127.0.0.1:42185 instead of 192.168.52:42185).

The only possibility I can think of is that your global template is different than mine. Can you try modifying your config as follows? I highlighted the difference in bold.

#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support
#$ModLoad immark  # provides --MARK-- message capability
$ModLoad imfile # Load the imfile input module

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.

#
$template CFormat,"%timegenerated% %HOSTNAME% %syslogseverity-text% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

$template FluentdFormat,"<%pri%>%timegenerated% %HOSTNAME% %syslogseverity-text% %syslogtag%%msg:::sp-if-no-

1st-sp%%msg:::drop-last-lf%\n"

$ActionFileDefaultTemplate CFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup adm

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

*.* @192.168.7.52:42185;FluentdFormat

[alnat2]

Yes, it's works !
But the "ident" field contains severity's info, not a process' name.

[Masahiro Nakagawa]

> 2014-06-22 20:23:41 +0400 [debug]: plugin/in_syslog.rb:128:receive_data_parser: invalid syslog message: "Jun 22 20:23:41  ...

Ah, it should be warn just like in_tail.