Re: [ossec-list] Clients authenticate, but don't connect (Corp env)
Santiago Bassett
Hi everyone,--I'm in a corporate environment, the environment we are deploying OSSEC to has around 1000 servers (I did the manual install and increased the agent limit). The firewall is allowing all UDP and TCP ports to pass through for our deployment. No traffic is being blocked to/from the OSSEC manager.We distributed OSSEC to an environment via Puppet and are able to get the agents to grab a client key over port 1515, but they are having issues connecting. A handful do eventually connect, but the majority don't, I don't see them come up in the OSSEC logs but they do appear as inactive agents.Any ideas as to why the majority of agents are not connecting, but do get their keys?
---
You received this message because you are subscribed to the Google Groups "ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
lostinthetubez
Looks like permissions or ownership are wrong on your client.keys file, which would certainly explain the agent not being able to connect. I assume you’ve checked that the client.keys file exists and contains the correct information for the agent you are using as an example here?
>> 2015/12/14 07:31:08 ossec-agentd(1103): ERROR: Unable to open file '/etc/client.keys'.
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Jamey B
Sent: Monday, December 14, 2015 12:55 PM
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] Clients authenticate, but don't connect (Corp env)
Thanks for that, I think this is a bigger issue than I believed judging by the read out below from one of the agents not connecting. Do you think the command you provided will fix it? It seems the install or CONF file went wonky during the install, but the agent has been reinstalled multiple times.
root@adr318 # cat /var/ossec/logs/ossec.log
2015/12/14 07:30:51 ossec-authd: INFO: Started (pid: 3787).
2015/12/14 07:30:58 ossec-execd(1314): INFO: Shutdown received. Deleting responses.
2015/12/14 07:30:58 ossec-execd(1225): INFO: SIGNAL Received. Exit Cleaning...
2015/12/14 07:31:08 ossec-execd: INFO: Started (pid: 3875).
2015/12/14 07:31:08 ossec-agentd: INFO: Using notify time: 600 and max time to reconnect: 1800
2015/12/14 07:31:08 ossec-agentd(1410): INFO: Reading authentication keys file.
2015/12/14 07:31:08 ossec-agentd(1103): ERROR: Unable to open file '/etc/client.keys'.
2015/12/14 07:31:08 ossec-agentd(1750): ERROR: No remote connection configured. Exiting.
2015/12/14 07:31:08 ossec-logcollector(1103): ERROR: Unable to open file '/queue/ossec/.agent_info'.
2015/12/14 07:31:08 ossec-config(1756): ERROR: Duplicated directory given: '/etc'.
2015/12/14 07:31:08 ossec-config(1756): ERROR: Duplicated directory given: '/bin'.
2015/12/14 07:31:08 ossec-syscheckd(1103): ERROR: Unable to open file '/queue/ossec/.agent_info'.
2015/12/14 07:31:08 ossec-syscheckd(1103): ERROR: Unable to open file '/queue/ossec/.agent_info'.
2015/12/14 07:31:11 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 07:31:11 ossec-rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 07:31:17 ossec-logcollector(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 07:31:17 ossec-logcollector(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
2015/12/14 07:31:19 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 07:31:19 ossec-rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 07:31:32 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 07:31:32 ossec-rootcheck(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
2015/12/14 09:50:10 ossec-execd(1314): INFO: Shutdown received. Deleting responses.
2015/12/14 09:50:10 ossec-execd(1225): INFO: SIGNAL Received. Exit Cleaning...
2015/12/14 09:50:20 ossec-execd: INFO: Started (pid: 15169).
2015/12/14 09:50:20 ossec-agentd: INFO: Using notify time: 600 and max time to reconnect: 1800
2015/12/14 09:50:20 ossec-agentd(1410): INFO: Reading authentication keys file.
2015/12/14 09:50:20 ossec-agentd(1103): ERROR: Unable to open file '/etc/client.keys'.
2015/12/14 09:50:20 ossec-agentd(1750): ERROR: No remote connection configured. Exiting.
2015/12/14 09:50:20 ossec-logcollector(1103): ERROR: Unable to open file '/queue/ossec/.agent_info'.
2015/12/14 09:50:20 ossec-config(1756): ERROR: Duplicated directory given: '/etc'.
2015/12/14 09:50:20 ossec-config(1756): ERROR: Duplicated directory given: '/bin'.
2015/12/14 09:50:20 ossec-syscheckd(1103): ERROR: Unable to open file '/queue/ossec/.agent_info'.
2015/12/14 09:50:20 ossec-syscheckd(1103): ERROR: Unable to open file '/queue/ossec/.agent_info'.
2015/12/14 09:50:23 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 09:50:23 ossec-rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 09:50:29 ossec-logcollector(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 09:50:29 ossec-logcollector(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
2015/12/14 09:50:31 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 09:50:31 ossec-rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 09:50:44 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2015/12/14 09:50:44 ossec-rootcheck(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
--
---
You received this message because you are subscribed to a topic in the Google Groups "ossec-list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ossec-list/7u88Yy5W7Rk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ossec-list+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Sincerely,
James Bearden III
lostinthetubez
Your commandline prompt indicates that this is not the same machine that you were talking about in the previous post. Please look at the situation on adr318, whatever that box is.
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Jamey B
Sent: Tuesday, December 15, 2015 7:06 AM
To: ossec...@googlegroups.com
Subject: Re: [ossec-list] Clients authenticate, but don't connect (Corp env)
Hi lostinthetubez,
Yes, the client.keys file exists on the server and the client has the correct key. The permissions are as follows for /var/ossec/etc/:
root@ccisprlx11 # ls -la ../etc/
total 136
dr-xr-x--- 3 root ossec 4096 Dec 14 17:23 .
dr-xr-x--- 13 root ossec 4096 Dec 14 16:59 ..
-r--r----- 1 root ossec 84 Dec 14 17:24 client.keys
-r--r----- 1 root ossec 97786 Jun 10 2015 decoder.xml
-r--r----- 1 root ossec 2842 Jun 10 2015 internal_options.conf
-r--r----- 1 root ossec 3519 May 4 2010 localtime
-r--r----- 1 root ossec 8360 Dec 14 16:59 ossec.conf
-rw-r----- 1 root root 88 Dec 14 16:59 ossec-init.conf
drwxrwx--- 2 root ossec 4096 Dec 14 16:59 shared
Do you see anything odd with the permissions?
lostinthetubez
Is selinux enabled? Long shot, I know. Regardless, OSSEC needs to be able to access the client.keys file, both on the agent and the manager, before it can communicate. If permissions and ownership aren’t the problem – which, they look fine btw – then I don’t honestly know why it would be complaining. You haven’t customized the users under which the services start, have you? Compare a client.keys from a working agent with a non-working agent. Perhaps there is a problem with the file format, encoding, or non-printable characters. Can’t really think of anything else at the moment.
From: ossec...@googlegroups.com [mailto:ossec...@googlegroups.com] On Behalf Of Jamey B
Sent: Tuesday, December 15, 2015 5:55 PM
To: ossec...@googlegroups.com
Subject: RE: [ossec-list] Clients authenticate, but don't connect (Corp env)
Sorry about that, that's my local VirtualBox image that I use for testing. OSSEC on the server with the client keys shows the same permissions as my local VM. Could it be a local OS issue that the server is on?
dan (ddp)
> Hi,
>
> SELINUX isn't enabled, we also looked at all the permissions and they appear
> fine.
>
> We manually added an agent on the server and manually imported a fresh
> client key, then restarted the agent. It successfully added itself without
> using authd that we had success with in a different environment (done via
> Puppet using command agent-auth -m <server ip> -p <port>). Should we use
> port 1515, then 1514 when using this?
>
> Perhaps we're not adding the agents correctly?
>
agent-auth connects to an authd process. So the power used there
should be the port authd is listening on.
What happens if you use manage_agents on the server to add an agent
and export the key. Then use manage_agents on the agent to import the
key?
dan (ddp)
> Hi Dan,
>
> When we use manage_agents and export the key to the agent, the agent works
> fine. We've had success this way, but obviously it's tedious for over 5000
> servers. Isn't this similar how authd works? I'm wondering if there's
> something we're not executing after the agent gets a key.
>
> I've regenerated the SSL key on the server (somehow it was missing), so
> agents no longer have issues connecting for their key -- this is what caused
> all the agent alerts a few posts ago. We are following the guide below, but
> the agents just don't connect after getting their key:
>
> http://dcid.me/blog/2011/01/automatically-creating-and-setting-up-the-agent-keys/
>
That was just part of the troubleshooting process. We now know that
agents CAN connect and work. So we have eliminated one issue. Only a
million more to go!
I might have missed it in the threat, but what version of OSSEC are you using?
When you run ossec-authd, what options are you using?
dan (ddp)
>
> /var/ossec/bin/agent-auth -m <IP> -p 1515
Ok, but I'd still like to know what options you're using with ossec-authd.
> /etc/init.d/ossec/ossec-hids restart
>
> I've confirmed via tcmpdump the agents are connecting over 1514. We also
> tried 'A <FQDN here>' at the end of the first command above, but have the
> same result.
>
>
> Here's what the agents are running:
>
> root@testlabex2 ./ossec-control status
>
> ossec-logcollector is running...
>
> ossec-syscheckd is running...
>
> ossec-agentd is running...
> ossec-execd is running...
>
>
> We are running version 2.8.2-49
>
What errors are in the ossec.log on the agents? What about the
server's ossec.log (possibly with debugging enabled)?