Syslog relay - not showing true source device IP as 'host'

[Dan Woodruff]

Hi there,
My setup is as follows:

Many sending network devices (Cisco ASA/FWSM) --> syslog server running rsyslog --> ELSA

The syslog server is our central repository for all logs, then I filter out certain messages to forward on to ELSA.

The trouble is that all messages in ELSA show a sending host as the syslog server, so I lose where the message originally came from. I took a look at the patternsdb.xml file and it seems like the pattern for the Cisco Teardown messages doesn't start parsing until the "Teardown" word in a message such as this:

Nov 16 16:34:10 x.x.x.x %ASA-6-302014: Teardown TCP connection 2737237784 for outside:y.y.y.y/80 to inside:z.z.z.z/56755 duration 0:00:00 bytes 1046 TCP FINs

With a tcpdump though, I see the full original message coming across the wire with the correct source host:

16:35:06.456593 IP syslogserver.39446 > elsa.syslog: SYSLOG local4.info, length: 181E.....@.;.....eF..K.......gz<166>Nov 16 16:35:14 x.x.x.x %ASA-6-302014: Teardown TCP connection 2737247154 for outside:y.y.y.y/443 to inside:z.z.z.z/62727 duration 0:01:00 bytes 362 TCP FINs

I don't fully comprehend the patternsdb.xml file yet - need to spend some time reading the documentation and studying how it all fits together - but it seems like that source host is pulled from somewhere other than the log message itself. Is that correct? Is it possible to pull the source from the log message itself?

Thanks in advance!
Dan

[Martin Holste]

The host used, as configured in syslog-ng.conf, is the IP address observed on the wire.  There are a number of reasons for this (performance and guarantees when parsing), but as long as the hostname being sent is either an IP or an FQDN which will resolve to an IP, you can comment out the line in the syslog-ng.conf "rewrite(r_host);" which is what sets the host to be what was observed on the wire.  There is a good description of how this works with DNS in this archive from the syslog-ng mailing list: https://lists.balabit.hu/pipermail/syslog-ng/2009-November/013571.html .  This is untested in ELSA, but as long as the $HOST macro gets set to something that's an IP in syslog-ng.conf, the rest of ELSA won't care how it got there and everything should work fine.  A word of caution: relying on DNS when processing logs in high volumes is a very good way to start losing logs.

[Dan Woodruff]

Gotcha. We have no plans to do DNS lookups due to the performance hit, at least until we upgrade rsyslog which I read has a caching name server built into v7. Until then, I'll config ELSA to what works for now and worry about it later :)

I tried commenting out rewrite(r_host); and restarting syslog-ng, but I still do not get the true host in ELSA. Where does the $SOURCEIP value get set in this line:

rewrite r_host { set("$SOURCEIP", value("HOST")); };

Is it supposed to come from patterndb.xml? I don't see that the string $SOURCEIP in patterndb.xml, and not being very familiar with syslog-ng, I'm confused where that comes from? Is there also a change to patterndb.xml I would have to make?

Thanks!

[Martin Holste]

$SOURCEIP is a hard macro that syslog-ng populates which contains the IP sending the log as observed on the wire.  ELSA also adds a $pdb_extracted_sourceip macro for certain patterns which can extract a better IP to use from some log messages, and ELSA will use that if available.  Can you provide an example log and what it gets parsed into?

[Dan Woodruff]

Gotcha.

Here's an example.

Message contents as written to the log on the rsyslog intermediary:
Nov 26 15:55:40 x.x.251.13 %ASA-6-302014: Teardown TCP connection 4095820694 for outside:x.x.145.108/80 to inside:x.x.75.229/56235 duration 0:00:00 bytes 2698 TCP FINs

tcpdump -A output on the ELSA box:
E.....@.;.....eF..K.........<166>Nov 26 15:55:40 x.x.251.13 %ASA-6-302014: Teardown TCP connection 4095820694 for outside:x.x.145.108/80 to inside:x.x.75.229/56235 duration 0:00:00 bytes 2698 TCP FINs

What it is parsed into in ELSA web interface:



251.13 is the original sending device that I would love to have in the host field. 101.70 is the intermediate rsyslog box.

Thanks for the help on this! I really appreciate it, especially since it isn't strictly a supported configuration :)

Dan

[Dan Woodruff]

We'll try this again...

[Martin Holste]

If use_dns(no) and keep_hostname(yes) is in the config (which is how it ships with ELSA), then $HOST should be the host as observed in the text of the message.  What version of syslog-ng are you on?  I found a bug with 3.3.4 (which has since been fixed) which required setting flags(expect-hostname) in the source configuration to pickup the sending host name, so you might try changing:

udp();

to:

udp(flags(expect-hostname));

And see if that works.

[Dan Woodruff]

I was running 3.2.4, which is the version in the repo for Ubuntu 12.04.1. I tried flags(expect-hostname) to no avail so I installed 3.3.4, created some symlinks to the ELSA syslog-ng config file location and from the ELSA syslog-ng binary location to the new binary, then modified the /etc/init.d/syslog-ng file a bit to use the default PID file location, and was able to get it working. The host field now shows the source host instead of my intermediate syslog server. The only change I made to the ELSA syslog-ng config was to comment out rewrite(r_host); per your original suggestion.

The only odd thing I'm seeing now are these warning messages when I start syslog-ng:

WARNING: Configuration file format is too old, please update it to use the 3.3 format as some constructs might operate inefficiently;
WARNING: global: the default value of log_fifo_size() has changed to 10000 in version 3.3 to reflect log_iw_size() changes for tcp()/udp() window size changes;
WARNING: The default behaviour for injecting messages in db-parser() has changed in version 3.3 from internal to pass-through, use an explicit inject-mode(internal) option for old behaviour;

I'm sure next time I do an install.sh node update, I'll be looking back at this thread to figure out what I did to get it all working :) But for now, it seems to be good. Perhaps the Ubuntu repository will be updated with a newer version of syslog-ng eventually and I won't have to worry about it?

Thanks for all your help on this request and the hard work you've put into this tool, Martin! I really appreciate it. I think I sold my boss on ELSA last time we had a minor incident to investigate, so I'll be keeping up with this discussion board for sure :)

Dan

[Martin Holste]

Ah, very good!  I think that 3.3.4 is ready for primetime now (we've been using it for almost a year in production) and all of the ELSA-affecting bugs have been squashed, so it's probably time to update install.sh to build 3.3.4 instead of 3.2.5.  To get rid of the warnings, you can change syslog-ng.conf to have a header of

@version: 3.3

instead of

@version: 3.2

Glad to hear you're getting value out of ELSA!  We just used it this morning to troubleshoot a very complex load balancer problem in which we had to find which recurring log message was causing problems out of the over 200 million logs from the load balancers this morning, and we would have been out of luck without ELSA.

[Dan Woodruff]

Changing the version number got rid of the warnings. Thanks!

[Konrad W]

Hello,

I am experiencing the same issue here. 

Martin, I know you mentioned earlier that for that to work (comment out "rewrite(r_host);"), device sending logs would need to send either IP address or FQDN in the message. My device is sending a "hostname" but not FQDN. Is there a way around it to get that true device hostname reported in ELSA vs the IP address of the relay server?

Sample log message below:

Jan  9 00:00:00 [redacted-hostname non FQDN] %FWSM-6-302016: Teardown UDP connection 145581438601591105 for 80WLab:[redacted ip]/33054 to gwy:[redacted ip]/53 duration 0:00:00 bytes 162

Also, just to mention, when I commented out "rewrite(r_host);" ELSA was reporting the logs with "127.0.0.1" in the host field.

Thanks 

Konrad

[Jean Cot]

Hello Konrad,

Have you solved your issue ? Because I have the same issue with syslog-ng version @3.2.

Thanks
Jean Cot

[Konrad Weglowski]

Hey Jean,

Check if this resolves it for you...I couldn't do it as my sending syslog version was quite old:

https://groups.google.com/d/topic/enterprise-log-search-and-archive/CqON4TQyCH8/discussion

Thanks,

Konrad

--
You received this message because you are subscribed to a topic in the Google Groups "enterprise-log-search-and-archive" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/enterprise-log-search-and-archive/D7A05VKZzz0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to enterprise-log-search-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jean Cot]

Thanks for your answer. I remember that I tried it and not worked. I will investigate other solutions.

Regards
Jean

To unsubscribe from this group and all its topics, send an email to enterprise-log-search-and-archive+unsubscribe@googlegroups.com.