Clients don't work when OSSEC server is in High Availability?
Timothy Meader
some help on. To give a brief synopsis of the situation:
We originally had a single server setup running OSSEC. Last week, we
decided to combine this server with another two that were running as
a simple log server (in high availability fail-over mode using
heartbeat) to make better use of the existing systems. The log server
portion is running on the virtual IP xxx.xxx.xxx.7 on eth0:0, the
OSSEC server is setup to run on a secondary virtual IP,
xxx.xxx.xxx.29, on eth0:1. When running on a single server, OSSEC
worked fine. But now, the clients refuse to communicate properly with
the server.
Using tcpdump, I tracked this communications problem down to the fact
that the server response from OSSEC in the high availability setup is
going back to the client with the ACTUAL address of eth0
(xxx.xxx.xxx.17 or 18 depending on which of the two high-avail nodes
it's currently running on). What I need is for the server response to
come back to the client with the xxx.xxx.xxx.29 address as the
source. I've investigated the "IPsrcaddr" script that comes with
heartbeat, but unfortunately there are two issues that preclude me
from using it, so I'm looking to iptables for a means to handle this.
Basically, I need the responses to any traffic coming in on UDP port
1514 (or, alternatively, to the destination IP x.29) to go back out
with a src address of x.29 instead of x.17 or x.18. Is there a method
using iptables that can handle this problem? Or barring that, can
anyone think of any other means to accomplish this task if iptables
can't handle this on its own? Below, I've included the current
iptables data rules that the two cluster nodes are presently sharing.
Thank you in advance for any and all replies up-front.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 514 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 720 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport
1514 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport
5514 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport
5140 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport
8000:8001 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport
8089 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
---
Tim Meader
L-3 Communications, NASA EOS Security Operations
Timothy...@gsfc.nasa.gov
(301) 614-6371
David Williams
Hash: SHA1
Tim,
I think you need to add a SNAT rule to use iptables for this. I'm
not in a position to test this but I think something like this may
work for you:
- -t nat -A POSTROUTING -o eth0 -p udp --dport 1514 -j SNAT --to
xxx.xxx.xxx.29
The intent (as I said, I can't check this) is to add to the nat
table a postrouting rule for udp output on eth0 to port 1514 that
jumps to source network address translation setting the source
address to be xxx.xxx.xxx.29.
I hope that at least points you in the right direction.
-David
- --
_______________________________________________
GPG (http://www.gnupg.org/) key available from:
http://www.kayakero.net/per/david/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.6 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFHIK/JCzuSgviBh00RArKUAKCISTPuNn8HsBx194ViEHm1yjgXDwCbBfaB
Tjrzx6xrBw5Fvug4B8KHiRc=
=QxfP
-----END PGP SIGNATURE-----
Timothy Meader
iptables config (came up with a similar example after a web search),
but every time I do, I'm no longer able to startup IPtables due to an
error about "seems to have a -t table option" when I run
"/etc/init.d/iptables start". Admittedly, I don't know enough about
iptables syntax, but could you provide more explicit instruction on
WHERE to add that line? My actual /etc/sysconfig/iptables file is in
the message below (and my original message). Where in that file would
that line fit in?
Thanks in advance.
PS - I'd posted to the Linux-HA list as well for any possible help,
and one user stated that perhaps OSSEC isn't acting the way a program
should in order to run properly on a multi-homed system. They stated
that, in multi-homed cases, OSSEC should ideally be analyzing the
original dstip for packets it processes, and send all outgoing
responses with a matching srcip to avoid all this hassle. Is there
anyone that should be contacted to hopefully get OSSEC setup using
the proper behavior for HA or multi-homed systems? As it continues to
increase in popularity, I can see this only increasing as a problem.
At 11:01 AM 10/25/2007, you wrote:
>* PGP Signed by an unknown key: 10/25/07 at 11:01:29
>
>Tim,
> I think you need to add a SNAT rule to use iptables for this. I'm
>not in a position to test this but I think something like this may
>work for you:
>-t nat -A POSTROUTING -o eth0 -p udp --dport 1514 -j SNAT --to
>xxx.xxx.xxx.29
> The intent (as I said, I can't check this) is to add to the nat
>table a postrouting rule for udp output on eth0 to port 1514 that
>jumps to source network address translation setting the source
>address to be xxx.xxx.xxx.29.
> I hope that at least points you in the right direction.
> -David
>
>Timothy Meader wrote:
> > Hello, I'm having an issue that I'm hoping someone could provide me
> > some help on. To give a brief synopsis of the situation:
> >
> > We originally had a single server setup running OSSEC. Last week, we
> > decided to combine this server with another two that were running as
> > a simple log server (in high availability fail-over mode using
> > heartbeat) to make better use of the existing systems. The log server
> > portion is running on the virtual IP xxx.xxx.xxx.7 on eth0:0, the
> > OSSEC server is setup to run on a secondary virtual IP,
> > xxx.xxx.xxx.29, on eth0:1. When running on a single server, OSSEC
> > worked fine. But now, the clients refuse to communicate properly with
> > the server.
> > *filter
>--
>_______________________________________________
>GPG (http://www.gnupg.org/) key available from:
>http://www.kayakero.net/per/david/
>
>* Unknown Key
>* 0xF881874D (L)
David Williams
Hash: SHA1
Tim,
I'm guessing the issue is how the RedHat tool creates and stores
the rules. I'm a bit leery trying to troubleshoot iptables long
distance like this; however, if you have a good backup of the file
and you're OK with poking around, I'd try this first:
as root add the rule via the iptables command:
/sbin/iptables -A POSTROUTING -t nat -o eth0 -p udp --dport 1514 -j
SNAT --to xxx.xxx.xxx.29
That should insert the SNAT rule and it should start working from
that point. To save it you need to use something like:
/sbin/service/iptables save
If that does not work, you can use the Firewall/SELinux applet to
add a custom file to your firewall rules. I think this should be
about right for that custom file (you may want to save it in
/etc/sysconfig along with iptables (the main rule set file):
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
- -A POSTROUTING -o eth0 -p udp --dport 1514 -j SNAT --to-source
xxx.xxx.xxx.29
COMMIT
(Careful of the line breaks "-A POST....29" should be one line.)
That's the iptables-save format for a nat rule. I believe the main
problem you ran into is the nat table is specified at the top of the
file in that format. You may be able to put that block at the top
of your main rule set file; but, it does say not to edit by hand. I
would expect the custom file technique to be most likely to survive
updates to the iptables packages.
Again, I hope that's helpful,
-David
- --
_______________________________________________
GPG (http://www.gnupg.org/) key available from:
http://www.kayakero.net/per/david/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.6 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFHIQIhCzuSgviBh00RAh8zAJ0Yr+E2av1shsaz2BrND1atLlc0YgCgjA5E
MkbfQCmkPkmw57GLG6mPzg8=
=swoU
-----END PGP SIGNATURE-----
Timothy Meader
Again, thanks for the help. I've tried implementing the rules you
mentioned via the first method. I changed it slightly, since I'm
actually looking for packets that are sent from local port 1514 on
the server out to a high port on the client machine, therefore I
changed it to be:
-A POSTROUTING -t nat -o eth0 -p udp --sport 1514 -j SNAT --to xxx.xxx.xxx.29
That seemed to start catching the traffic I wanted, however, I've run
into an interesting issue.
Trying to SNAT the traffic to that virtual IP results in no packets
being sent back to the clients at all. As soon as I change it to a
different IP, then it works perfectly.
For example, this is what I'm seeing on one of the HA nodes where I
tried to implement the iptables rule (.39 and .40 are two of the
ossec clients):
(using "SNAT --to-source xxx.xxx.xxx.29")
"tcpdump udp port 1514" output:
--------------------------------------------
23:44:17.843321 IP xxx.xxx.xxx.39.32792 > xxx.xxx.xxx.29.1514: UDP, length 73
23:44:23.851493 IP xxx.xxx.xxx.39.32792 > xxx.xxx.xxx.29.1514: UDP, length 73
23:44:33.875446 IP xxx.xxx.xxx.39.32792 > xxx.xxx.xxx.29.1514: UDP, length 73
23:44:39.866540 IP xxx.xxx.xxx.40.32788 > xxx.xxx.xxx.29.1514: UDP, length 73
23:44:39.915789 IP xxx.xxx.xxx.39.32792 > xxx.xxx.xxx.29.1514: UDP, length 73
As you can see, there is no outbound traffic at all from the server
back to the client. Now, here is the same tcpdump after changing the
SNAT source-ip to something other than the virtual IP (eth0:0) on the
machine (in this case, I set it to "SNAT --to-source xxx.xxx.xxx.7"):
23:09:07.244932 IP xxx.xxx.xxx.40.32786 > xxx.xxx.xxx.29.1514: UDP, length 73
23:09:07.245869 IP xxx.xxx.xxx.7.1514 > xxx.xxx.xxx.40.32786: UDP, length 73
23:09:13.256676 IP xxx.xxx.xxx.40.32786 > xxx.xxx.xxx.29.1514: UDP, length 73
23:09:13.257084 IP xxx.xxx.xxx.7.1514 > xxx.xxx.xxx.40.32786: UDP, length 73
23:09:23.280362 IP xxx.xxx.xxx.40.32786 > xxx.xxx.xxx.29.1514: UDP, length 73
23:09:23.280813 IP xxx.xxx.xxx.7.1514 > xxx.xxx.xxx.40.32786: UDP, length 73
23:09:26.801841 IP xxx.xxx.xxx.39.32789 > xxx.xxx.xxx.29.1514: UDP, length 73
23:09:26.802341 IP xxx.xxx.xxx.7.1514 > xxx.xxx.xxx.39.32789: UDP, length 73
Here you can definitely see the traffic going out as intended by the
IPtables rule. Any idea why it would refuse to work when trying to
set the source-ip to be the original destination of the packets
coming from the clients? Is this a limitation of the POSTROUTING
chain in some way?
Thanks again for all your help in this matter. Hope to hear back soon.
At 04:52 PM 10/25/2007, you wrote:
>* PGP Signed by an unknown key: 10/25/07 at 16:52:49
>
>Tim,
> I'm guessing the issue is how the RedHat tool creates and stores
>the rules. I'm a bit leery trying to troubleshoot iptables long
>distance like this; however, if you have a good backup of the file
>and you're OK with poking around, I'd try this first:
>as root add the rule via the iptables command:
>
>/sbin/iptables -A POSTROUTING -t nat -o eth0 -p udp --dport 1514 -j
>SNAT --to xxx.xxx.xxx.29
>
> That should insert the SNAT rule and it should start working from
>that point. To save it you need to use something like:
>
>/sbin/service/iptables save
>
> If that does not work, you can use the Firewall/SELinux applet to
>add a custom file to your firewall rules. I think this should be
>about right for that custom file (you may want to save it in
>/etc/sysconfig along with iptables (the main rule set file):
>
>*nat
>:PREROUTING ACCEPT [0:0]
>:POSTROUTING ACCEPT [0:0]
>:OUTPUT ACCEPT [0:0]
>-A POSTROUTING -o eth0 -p udp --dport 1514 -j SNAT --to-source
>xxx.xxx.xxx.29
>COMMIT
>
> (Careful of the line breaks "-A POST....29" should be one line.)
>That's the iptables-save format for a nat rule. I believe the main
>problem you ran into is the nat table is specified at the top of the
>file in that format. You may be able to put that block at the top
>of your main rule set file; but, it does say not to edit by hand. I
>would expect the custom file technique to be most likely to survive
>updates to the iptables packages.
> Again, I hope that's helpful,
> -David
---
David Williams
Hash: SHA1
Tim,
I'll admit that I'm running low on ideas. I think you may get more
useful help posting this to the iptables list -- which seems to be
MIA at the moment: http://www.netfilter.org/mailinglists.html#ml-user
These are really straws to grasp at, sorry:
Because of some really weird things I saw a couple months ago I
would arping from .29 to .39 and .40 (or the upstream router,
anyway) maybe with tcpdump looking for the arp traffic to see if
that gets there:
/sbin/arping -c 3 -s xxx.xxx.xxx.29 -I eth0 xxx.xxx.xxx.39
Next, I would add some logging rules to look for the traffic from
.29 in iptables Before the -j REJECT line add:
- -A OUTPUT -s xxx.xxx.xxx.29 -j LOG --log-level INFO --log-prefix
"OUTPUT from .29"
That should log all outbound traffic from .29.
- -A RH-Firewall-1-INPUT -s xxx.xxx.xxx.29 -j LOG --log-level INFO
- --log-prefix "Rejecting INPUT/FORWARD from .29"
That should log all input and forwarded traffic just before it's
rejected. You'll have to restart iptables for those to get used.
Then try these two to get your SNAT rule back with some logging:
- -A POSTROUTING -t nat -o eth0 -p udp --sport 1514 -j LOG --log-level
INFO --log-prefix "POSTROUTING from port 1514"
- -A POSTROUTING -t nat -o eth0 -p udp --sport 1514 -j SNAT --to
xxx.xxx.xxx.29
Maybe you'll see something in the log (/var/log/messages by
default) that will shed some light on what's going on. I'm sorry
not to have any better ideas,
-David
- --
_______________________________________________
GPG (http://www.gnupg.org/) key available from:
http://www.kayakero.net/per/david/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.6 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFHIUJVCzuSgviBh00RAlhBAKCkfjZaRsE+HAuCGx1SrZvpcEzQAACgxKBo
aZvgpboO7oiXfvgRRuCA7MQ=
=jAwS
-----END PGP SIGNATURE-----
Daniel Cid
I can't help you much with the iptables rules, but you could try using
the "local_ip" option
in the server config to specify the IP address for OSSEC to use (in
your case the ip of eth0:1).
*example for ip 10.2.3.4:
<remote>
<local_ip>10.2.3.4</local_ip>
</remote>
http://www.ossec.net/main/manual/#remote_options
As for OSSEC analyzing the dst ip of the incoming packet and using
that for the reply,
I will take a look into implementing that (for v1.5)...
Thanks,
--
Daniel B. Cid
dcid ( at ) ossec.net