Filter syslog Messages to add GeoIP

[asto...@redhat.com]

Hi Group,

I'm currently feeding my syslogs unfiltered into elasticsearch.

But now, I'd like to add GeoIPs to certain messages. Unfortunately, the IPs are somewhere inside the message field and have to be filtered out.

Examples:

Spammers try to get into my SMPT-Server, which lists in Kibana like that:

host:

mail

 

ident:

postfix/smtpd

 

pid:

xxxxx

 

message:

disconnect from unknown[

xxx.xxx.xxx.xxx]

 

@log_name:

system.mail.info

  ...

host:

mail

 

ident:

postfix/smtpd

 

pid:

xxxx

 

message:

warning: unknown[xxx.xxx.xxx.xxx]: SASL LOGIN authentication failed: authentication failure

 

@log_name:

system.mail.warn

host:

mail

 

ident:

postfix/smtpd

 

pid:

xxxxx

 

message:

connect from unknown[

xxx.xxx.xxx.xxx]

 

@log_name:

system.mail.info

How can I:

Step 1

- Filter out the IP-Adresses

[

xxx.xxx.xxx.xxx] from the message field (of a "host:mail ident:postfix" message only) and add them into a separate field

smpt_source_ip:xxx.xxx.xxx.xxx

while all other incoming syslog messages are still fed into elasticsearch without change

Step2:

- Add GeoIP Information to these smtp_source_ip Fields.

Which then leads to the final step3: displaying Attacker-Source on a map in Kibana. 

thanks for Tipps,

Andreas

[Mr. Fiber]

> Filter out the IP-Adresses 

IP address seems to be wrapped by [] so writing your parser or extracting it via regex format is better. 

<source>

  @type syslog

</source>

<filter syslog.**>

  @type parser

  format your_parser or /your regexp for IP/

</filter>

<filter syslog.**>

  @type geoip

</filter>

...

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[asto...@redhat.com]

Hi Masahiro

Maybe I'm to stupid for regex?

I currently try to parse the mail log on the mailserver directly using tail like this:

<source>

  @type tail

  path /var/log/maillog

  tag maillog

  read_from_head = true

  pos_file /root/fluentd/test.pos

  format /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)(?:\[(?<ipa>([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3}))\])?$/ 

</source>

Logs Entries come in like this:

Jun  2 13:22:29 mail postfix/smtpd[16604]: lost connection after AUTH from unknown[x.xxx.xxx.xxx]

Jun  2 13:22:29 mail postfix/smtpd[16604]: disconnect from unknown[x.xxxx.xxx.xxx]

Jun  2 13:22:29 mail postfix/smtpd[16604]: connect from unknown[x.xxxx.xxx.xxx]

But my regex doesn't work and I don't get an IPA: Field

.

On the other hand: how could I filter messages like:

Jun  2 13:22:29 mail postfix/smtpd[16604]: connect from unknown[x.xxxx.xxx.xxx]

*and*

Jun  2 13:22:48 mail postfix/smtpd[16604]: warning: unknown[x.xxxx.xxx.xxx]: SASL LOGIN authentication failed: authentication failure

Where ?<message> continues after the [?<ipa>] ?

Even if my regex worked, I'd cut the ?<message> at the IP and wouldn't continue.

thanks,

Andreas

[Mr. Fiber]

> But my regex doesn't work and I don't get an IPA: Field

It means you got "pattern not match" log or got event but record is wrong value?

We recommend to use fluentd-ui(td-agent-ui) for interactive regex check.

[Mr. Fiber]

Again, if you hard to extract values using regexp,

we recommend to write simple parser for specific format.

[asto...@redhat.com]

Got it now:

format /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*(\[(?<ip>([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})[\]])).*)$/

works. It filters the IP from within the <message> field.

But now I got a different problem:

This filter applies only to some of the entries within the Maillog. like postfix/smpt connect/disconnect. It does not work for entries from dovecot, postfix/qmgr, 

Is there a way to config: filter this, if failed then filter syslog

-or- 

filter syslog first, and then <match for whatsoever> and filter inside field <message> ?

Andreas

[Mr. Fiber]

It depends on your case.

Separete filter for each input or use multi-format-parser in syslog input or something.

I'm not sure which is the best.