Using Splunkforwarder instead of syslog

[C. L. Martinez]

Hi all,

I am using a splunkforwarder instance to redirect raw syslog messages
to several splunk instances. I would like to integrate ELSA in this
infrastructure but, can I use my splukforwarders to redirect syslog
messages directly to ELSA instead to use rsyslog or syslog-ng??

Thanks.

[Martin Holste]

I haven't tried it, but from the looks of it, only a full "heavy"
Splunk instance can forward as true syslog. See the page here for
details: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwarddatatothird-partysystemsd.

A couple of other things to consider:

If you forwarded everything first to ELSA via syslog, and then to
Splunk conditionally, you could use ELSA for your more verbose,
"debug" logs, and Splunk for more important ones, letting you pay for
just the bare amount of data to index for your Splunk licensing.

You can have Splunk forward to ELSA as outlined in the link above.
However, ELSA uses the sender's IP address for the "host" field, so
the host field will be incorrect if received directly from Splunk.

[C. L. Martinez]

Thanks Martin. A "heavy forwarder" is a Splunk Universal Forwarder,
afaik. In principle, you don't need a full splunk version to redirect
data.

But, your option sounds really good ... Can I configure under ELSA (or
maybe syslog) some rewrite rules to forward data to splunk server with
the correct hostname?? The idea is:

Some syslog device -----> SplunkUniversalForwarder (or rsyslog) ---->
ELSA ----> Splunk

[Martin Holste]

Absolutely! That's what we do. We have ELSA (syslog-ng) receive all
Syslog from everything, then we replicate to things like CiscoWorks,
Splunk, etc., as needed, so all of our devices need only be configured
with one log host. I will also add that we use a regular Cisco CSM to
provide a VIP for syslog, then have two ELSA nodes receive round-robin
load-balanced syslog. That gives us redundancy to do maintenance on
our ELSA nodes and we never miss a log, nor do any devices that are
getting the replicated logs.

So, if your syslog-ng.conf for ELSA as a source "s_network" like this:

source s_network { udp(); tcp(); };

Add a Splunk destination:
destination d_splunk { udp("my.splunk.server" port(514)); };

# Add the log statement:
log {
source(s_network);
log { destination(d_splunk); }
log {
# ELSA specific rewrites, parser, etc.
destination(d_elsa);
};
};

This will forward everything on to Splunk completely unchanged, so it
should receive just fine.

[C. L. Martinez]

Good!! .. But instead to use syslog-ng I need to use rsyslog (all
servers platforms are RHEL based). Do you have some samples to use
rsyslog?

On the other hand, I will install ELSA in a RHEL cluster using RedHat
Cluster Suite (RHCS)... Do you know some type of problem that I need
to know using this type of infrastructure??

Thanks for your help Martin.

[Martin Holste]

> Good!! .. But instead to use syslog-ng I need to use rsyslog (all
> servers platforms are RHEL based). Do you have some samples to use
> rsyslog?

ELSA will not work with rsyslog as it uses syslog-ng's advanced
pattern-db fast pattern matcher to process logs. However, the
currently installed rsyslog will not be a problem as ELSA will run in
addition to it. The install.sh referred to in the quickstart doc will
download and compile syslog-ng for you on a RedHat system and it can
run alongside the stock rsyslog.

In addition, you can certainly use rsyslog to forward to ELSA or have
rsyslog be somewhere in the chain, you just have to have syslog-ng at
the end of the chain to give its logs to ELSA. I have an example of
how to use rsyslog to forward flat files on my blog here:
http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html
.

> On the other hand, I will install ELSA in a RHEL cluster using RedHat
> Cluster Suite (RHCS)... Do you know some type of problem that I need
> to know using this type of infrastructure??

There shouldn't be any services that require the cluster components,
but ELSA should install just fine on RedHat with the install.sh
script. Let me know if you run into any problems.