v5.1.1: WARN msg not allowed, Incorrectly formated, and Duplicate counters?

[Clint Alexander]

After a clean vanilla installation of v5.1.1 with 23 agents, I'm getting spammed in the server logs with:

 

ossec-remoted(1403): ERROR: Incorrectly formated message from 'ip.address.of.agent'.

 

--------------------------------

I'm also seeing a lot of:

 

ossec-remoted(1213): WARN: Message from ip.addr.of.agent not allowed.

 

--------------------------------

Every once in a while I see:

 

ossec-remoted(2202): ERROR: Error uncompressing string.

 

--------------------------------

Out of the 23 agents, 14 of them show as 'never connected' and in the logs of the agents we have:

 

ossec-agentd(1407): ERROR: Duplicated counter for 'HOSTNAME'

ossec-agentd(1214): WARN: Problem receiving message from 'ip.of.server'

ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: 'ip.of.server'

 

--------------------------------

 

Some of the agents that do connect end up disconnecting at some point and it requires a restart of the ossec server before I see them online again (and sometimes they don't come back online)

 

 

There are no firewalls between the agents and server and I'm running a mix of CentOS 5 and Redhat Ent 4 & 5 servers; primarily in x86_64 (64bit libs) but a few in i386 (32bit libs).

 

I've looked through the wiki category for errors:

http://www.ossec.net/wiki/index.php/Errors:1403

(there is no description page for 1213, 1214, or 4101)

http://www.ossec.net/wiki/index.php/Errors:AgentCommunication

 

None of the suggestions work.

 

 

I've reinstalled agents, the server, recreated/reassigned keys, restarted the services 100xs; stood on my left leg, then my right, faced north, then east, prayed to the Bit-God, did a raindance -- all to no avail.

 

Is there anyone that has had these problems and found a solution?

 

//Clint

 

[Daniel Cid]

Hi Clint,

These errors are related to one key being assigned to more than one
agent. When you do it,
you will have this duplicated counters, errors uncompressing (since it
wasn't able to decrypt
properly), etc.

I would suggest stopping ossec and re-creating the keys. One by one,
you go adding new
keys to the agents, making sure each key you create is only used once.

Thanks,

--
Daniel B. Cid
dcid ( at ) ossec.net

[Clint Alexander]

I confirmed that each key was unique; the agent even prompted the
information (name, ip, id) from the server to confirm and it was correct. So
this isn't likely to be the issue.

Could the order in which services are stopped and started be an issue?

I go and add the keys to each agent, restarting each agent as I finish it;
and then once all agents are completed, I restart the server. Should this be
done differently?

//Clint

[Clint Alexander]

I've figured out a few things but have not been led to a final conclusion,
yet.

I can "turn" these messages on and off by enabling and disabling database
logging. When I have $ossec/bin/ossec-control enable database - I get the
error log messages and all the agents go offline, but when I shut DB logging
off, they start to work.

This could be something special with just my setup or perhaps not many folks
use the MySQL database features... not sure, but more testing is needed...

[Daniel Cid]

Hi Clint,

That's so strange... The database output is a separated process and not related
to the manager/agents communication. Maybe your agents are getting blocked
via active response? That can happen if they are not white listed and you have
an invalid user/password in the config....

As far as the order to add the keys, they should be:

-Add keys on the manager
-Restart manager
-Import keys into the agents.
-Restart agents.

*btw, I added the command-line options to manage_agents on the latest snapshot:
http://ossec.net/files/snapshots/ossec-hids-090805.tar.gz

Thanks,

--
Daniel B. Cid
dcid ( at ) ossec.net

[Clint Alexander]

If I only had more time; I would have zero'd in on the issue by now.

I won't be surprised if they are unrelated in this situation as well. All I
can do at the moment is report the facts as they present themselves. But, I
don't want to throw an email to the list every time there is something
new -- once I finish migrating data into a new replicated sql cluster and
putting it into production, I'll have an small, but open buffer of time to
take advantage of.

So cool on that command-line option!! If we could get a command-line option
to do all the basic administration, that would be perfect (adding, removing
and updating agents, retrieving info, getting/assigning keys, etc). It would
make script wrapping much easier. ;)

Nice work, Dan!!