ossec-remoted not binding to 1514/udp?

[Patrick Melvin]

Hello,

I believe I'm having an issue with ossec-remoted binding to port 1514/udp.

** Basic Configuration/OS Information **

OS: Linux <server_name> 2.6.32-27-server #49-Ubuntu SMP Thu Dec 2
02:05:21 UTC 2010 x86_64 GNU/Linux
ossec: 2.5.1

make setmaxagents
Specify maximum number of agents: 4096
./install.sh

/etc/security/limits.conf have the following entries (and server was
restarted afterwards):
* soft nofile 4096
* hard nofile 4096

** Issue Description **

As stated, it appears that while ossec-remoted is running, it is not
binding to 1514/udp. I've been troubleshooting this and have not been
able to get any helpful information out of the logs, debug mode, or
stracing the process.

When I run a tcpdump, I see the agents trying to connect to the server
on 1514/udp, but the server responds back with the following:

ICMP <server_ip> udp port 1514 unreachable, length 109

Which indicates there's no process listening on the port. netstat does
not show 1514 in use.

I verified 1514/udp connectivity by utilizing netcat (nc) and
successfully connected to the server on 1514/udp.

strace show's the following for ossec-remoted:
recvfrom(4,

I have the following in ossec.conf with relation to remoted:
<remote>
<connection>syslog</connection>
</remote>

<remote>
<connection>secure</connection>
<allowed-ips>192.168.0.0/16</allowed-ips>
<port>1514</port>
<local_ip>(server_ip_address)</local_ip>
</remote>

If there's any addtional information that might be helpful, please let me know.

I've been researching using google but have found no resolutions to
this specific problem. Any ideas?

Thanks in advance,
Patrick

[Castle, Shane]

Um. "lsof -P -c ossec-remoted" ? There should be a line like this:

ossec-rem 3378 ossecr 4u IPv4 11436 UDP *:1514

which you should be able to get by being a bit more restrictive:

# lsof -P -a -i -c ossec-remoted
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ossec-rem 3378 ossecr 4u IPv4 11436 UDP *:1514

Also, you say you checked logs. This includes /var/ossec/logs/ossec.log
too? That's where OSSSEC puts all its info, including any failure
messages.

--
Shane Castle
Data Security Mgr, Boulder County IT
CISSP GSEC GCIH

[S.SEWAK]

Hi Patrick,

Thank you for your fast respond. I am not sure how to read the following message. Can you please explain what the messages below meant? Also can you tell me what IP address ( 198.168.0.0./16 ) is?  How do I know the IP address my computer is using?

thanks

S Ling.

[S.SEWAK]

Patrick,  

So there are hackers trying to hack into my computer? 

Can you give me a little more details ?

thanks

On Tue, Jan 4, 2011 at 4:01 PM, Patrick Melvin <patrick...@gmail.com> wrote:

[S.SEWAK]

Also, I have found these two IP addresses using my computer. can you tell me where there are from and whose are they?

108.14.253.249  and 71.190.201.177

thanks.

Ling

On Tue, Jan 4, 2011 at 4:01 PM, Patrick Melvin <patrick...@gmail.com> wrote:

[Patrick Melvin]

Hi Shane, thanks for the response.

Below is the output of the 2nd command you listed, unfortunatly, like
netstat, there are no 1514 entries:

$ sudo lsof -P -a -i -c ossec-remoted
lsof: WARNING: can't stat() fuse.gvfs-fuse-daemon file system
/home/<user_account>/.gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ossec-rem 18455 ossecr 4u IPv4 467430 0t0 UDP *:514

And you are correct, when I mentioned logs, it was
/var/ossec/logs/ossec.log and no failure messages are being reported.

When you ran the command, did you have a 514 entry? If not, do I have
something incorrect in my ossec.conf file that is causing remoted to
try to bind to 514/udp (syslog) instead of 1514/udp?

Thanks,
Patrick

[Castle, Shane]

Hmm. 514/udp is the syslog listener. All I can think is the install is wrong somehow, since as a server it should open 1514/udp, but it is not. It looks like a "local" install rather than a "server" install.

I think you should reinstall, making sure to reply "server" to the question "What kind of installation do you want (server, agent, local or help)?".

[Castle, Shane]

Also: check your ossec.conf <remote> section, and check it against the info here:
http://www.ossec.net/doc/syntax/head_ossec_config.remote.html

[Patrick Melvin]

Hi Shane, thanks for the help and sorry it's taken a bit to get back
to you. I've been running through ideas for the past few hours and
finally narrowed down the problem by trial and error. Unfortunately,
I'm not sure how to fix it.

The original client.keys file I was working with had 1300+ agents
listed, and as stated, remoted wasn't listening on 1514/udp. I
started over with 1 agent, and found that 2 processes of remoted would
start up and one of them was listening on 1514/udp (while the other
was listening on 514/udp).

I know that there's a file open limit default of 1024, so I guessed
that maybe that was still the issue. I put the client.keys file in
place with 1024 agents. ossec-remoted would not listen on 1514/udp.
Through some trial and error, found that 1015 (and sometimes 1016) was
my magic limit. Anything higher than this in the client.keys file,
would cause ossec-remoted to not listen on 1514/udp (i.e. that 2nd
instance of remoted would start up and end immediately).

I verified that root and all user accounts have a openfile limit of
4096. I haven't been able to adjust sysctl's variable kern.maxfiles,
because it doesn't exist (on Ubuntu or CentOS5).

With that said, I went through the same steps on CentOS5, and remoted
starts up and listens on 1514/udp with 1300+ agents in client.keys.
It appears the issue is isolated to Ubuntu. Is there another variable
or setting that is set to 1024 that is not allowing me to run ossec
with more than 1015 agents?

Thanks,
Patrick

[dan]

Hi S,
I'm sorry I let these messages through. You seem a bit confused. If you
have a question about OSSEC, but not this thread, please start a new
thread.
dan

[dan]

Hi Patrick,

Are you using OSSEC to collect syslog messages? If not, you don't need
this. If you are using it, put it after the secure connection type, and
add allowed-ips.

> <remote>
> <connection>secure</connection>
> <allowed-ips>192.168.0.0/16</allowed-ips>
> <port>1514</port>
> <local_ip>(server_ip_address)</local_ip>
> </remote>
>

You shouldn't need the allowed-ips in the secure section, I think it's
only for syslog. You also don't need to put the port in there if you're
using the default.

>
> If there's any addtional information that might be helpful, please let me know.
>
> I've been researching using google but have found no resolutions to
> this specific problem. Any ideas?
>
> Thanks in advance,
> Patrick

Look for entries for ossec-remoted in /var/ossec/logs/ossec.log. You may
need to enable debugging for that daemon (run it with the -d flag) to
get something more useful.

I'm currently using both syslog and secure on one of my servers, so it
is possible. I haven't tried it with so many clients though.

dan

[Patrick Melvin]

Hi Dan, thanks for the response. I was running into issues that
remoted would complain about not having an IP address in the remote
section (with that said, on my CentOS5 setup, the remote section is
left as default and it runs fine...another issue isolated to Ubuntu?)

ossec-remoted(1501): ERROR: No IP or network allowed in the access
list for syslog. No reason for running it. Exiting.

Once I added the IP, it went away. I'll get the port removed, was
grasping at straws when I added it ;)
After retesting what you stated, I see that the error was specifically
for the syslog section (even though it went away when it was added to
either the syslog or secure section). I'll remove syslog from the
configuration files going forward.

Anyways, on to the Ubuntu issue:

In the logs file, when debug is enabled, this is all I get when it
fails to start up and bind to 1514/udp:
(Note: This is what the output looks like when the remote syslog
section is still included in the ossec.conf. When working correctly,
it shows 2 remoted processes)

2011/01/05 14:45:12 ossec-remoted: DEBUG: Starting ...
2011/01/05 14:45:12 ossec-remoted: INFO: Started (pid: 4255).
2011/01/05 14:45:12 ossec-remoted: DEBUG: Forking remoted: '0'.
2011/01/05 14:45:12 ossec-remoted: Remote syslog allowed from: '192.168.0.0/16'
2011/01/05 14:45:12 ossec-remoted: DEBUG: Forking remoted: '1'.

$ ps -ef | grep remoted
ossecr 4256 1 0 14:45 ? 00:00:00 /var/ossec/bin/ossec-remoted -d

$ sudo lsof -P -a -i -c ossec-remoted

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

ossec-rem 4256 ossecr 4u IPv4 19255 0t0 UDP *:514

(This is the output when remote syslog section is removed from ossec.conf)

2011/01/05 14:56:22 ossec-remoted: DEBUG: Starting ...
2011/01/05 14:56:22 ossec-remoted: INFO: Started (pid: 4765).
2011/01/05 14:56:22 ossec-remoted: DEBUG: Forking remoted: '0'.

$ ps -ef | grep remoted
<blank>

$ sudo lsof -P -a -i -c ossec-remoted

<blank>

When I delete an agent from client.keys and (in this case) go back
down to 1015, these are the outputs:

2011/01/05 15:03:16 ossec-remoted: DEBUG: Starting ...
2011/01/05 15:03:16 ossec-remoted: INFO: Started (pid: 5350).
2011/01/05 15:03:16 ossec-remoted: DEBUG: Forking remoted: '0'.

$ ps -ef | grep remoted
ossecr 5351 1 0 15:03 ? 00:00:00 /var/ossec/bin/ossec-remoted -d

$ sudo lsof -P -a -i -c ossec-remoted

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

ossec-rem 5351 ossecr 4u IPv4 23036 0t0 UDP *:1514

Now I'm really confused. I went through the tests above, and put the
original client.keys file in place (1300+ agents), and it's working
now. Just replaced the original configuration file settings (as stated
in the start of the email) and it still works.

I'm really at a loss...

Thanks in advance,
Patrick

[Patrick Melvin]

One last weird update...it was at 1342 agents. Added 1 to bring the
total to 1343 and it was working then the process died. Now when I
restart ossec, I'm back to where I was before where ossec-remoted
isn't running:

2011/01/05 16:08:03 ossec-remoted: DEBUG: Starting ...
2011/01/05 16:08:03 ossec-remoted: INFO: Started (pid: 9045).
2011/01/05 16:08:03 ossec-remoted: DEBUG: Forking remoted: '0'.

$ ps -ef | grep remoted
<blank>

[Patrick Melvin]

Ok I think the below was caused by the owner/group being changed from
root:ossec to root:root. I changed client.keys back to root:ossec,
and ossec-remoted fired up and is staying up.

Hopefully this was the issue all along, unfortunately permissions was
not something I was checking during the troubleshooting process.