After upgrade to Wazuh, agents are listed, but as "never connected"

[Whit Blauvelt]

Okay, I've missed something. But with the former OSSEC server now Wazuh, at the same address, with the same list of agents recognized by it, they're all of status "never connected." These are generally OSSEC 2.9.x agents.

My experience before was to install 'em, key 'em, and they'd connect. Wazuh should have the keys since it has the list of agents; and rekeying one doesn't cause it to connect; so it's something other than a key problem. I'm planning to update all agents to Wazuh presently. Still, from what I read I expected them to work with it meanwhile.

[Jose Luis Ruiz]

Hi Whit,

The agents from OSSEC 2.9.X are compatible with Wazuh-Manager, and if the configuration is correct all should be connected.

Do you have any error in the logs?, can you verify the logs in the manager and the agent?

/var/ossec/logs/ossec.log.

The client.keys in the agent never connected exists in the client.keys in the manager??

cat /var/ossec/etc/client.keys | grep agent_name and the result should be the same IP, Agent-name IP and key than in your agent.

Regards

----------------

Jose Luis Ruiz

@jlruizmlg

www.wazuh.com

On December 18, 2017 at 11:38:20 PM, Whit Blauvelt (whit+...@transpect.com) wrote:

Okay, I've missed something. But with the former OSSEC server now Wazuh, at the same address, with the same list of agents recognized by it, they're all of status "never connected." These are generally OSSEC 2.9.x agents.

My experience before was to install 'em, key 'em, and they'd connect. Wazuh should have the keys since it has the list of agents; and rekeying one doesn't cause it to connect; so it's something other than a key problem. I'm planning to update all agents to Wazuh presently. Still, from what I read I expected them to work with it meanwhile.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/8b26554a-c45a-4d96-97eb-8998ee96f876%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Whit Blauvelt]

Hi Jose,

The agents from OSSEC 2.9.X are compatible with Wazuh-Manager, and if the configuration is correct all should be connected.

That's reassuring

 

Do you have any error in the logs?, can you verify the logs in the manager and the agent?

/var/ossec/logs/ossec.log

From the client I upgraded to Wazuh 3, it looks like the manager may not be listening at the encrypted port:

2017/12/19 11:49:07 ossec-agentd: WARNING: (4101): Waiting for server reply (not started). Tried: '172.16.14.30'.
2017/12/19 11:50:08 ossec-agentd: INFO: Trying to connect to server (172.16.14.30:1514).
2017/12/19 11:50:18 ossec-agentd: ERROR: (1218): Unable to send message to 'server': Connection refused

And from an OSSEC agent, the same complaint:

2017/12/19 01:09:18 ossec-agentd: INFO: Trying to connect to server 172.16.14.30, port 1514.
2017/12/19 01:09:18 INFO: Connected to 172.16.14.30 at address 172.16.14.30, port 1514
2017/12/19 01:09:28 ossec-agentd(1218): ERROR: Unable to send message to 'server'.
2017/12/19 01:09:40 ossec-agentd(1218): ERROR: Unable to send message to 'server'.
2017/12/19 01:09:41 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: '172.16.14.30'.

On the Wazuh server, lsof -i shows nothing on port 1514.

Okay, the client.keys file was owned by root. Fixed that.

The server is having other problem, but nothing pertinent to this:'

2017/12/19 10:45:59 ossec-analysisd: ERROR: Unable to open '/queue/syscheck/syscheck'
2017/12/19 10:45:59 ossec-analysisd: ERROR: Error handling integrity database.

2017/12/19 10:51:29 rootcheck: INFO: Starting rootcheck scan.
2017/12/19 10:51:29 ossec-analysisd: ERROR: (1103): Could not open file '/queue/rootcheck/rootcheck' due to [(13)-(Permis
2017/12/19 10:51:29 ossec-analysisd: ERROR: Error handling rootcheck database.

What's that about?

# ls -ld /var/ossec/queue
drwxr-x--- 13 root ossec 4096 Dec 13 14:03 /var/ossec/queue

# ls -ld /var/ossec/queue/syscheck
drwxr-x--- 2 ossec ossec 4096 Dec 14 14:12 /var/ossec/queue/syscheck

So it can't open locations it owns?

Also, from a fresh startup:

2017/12/19 12:10:27 ossec-remoted: CRITICAL: (1103): Could not open file '/queue/rids/sender_counter' due to [(13)-(Permission denied)].

Who is it trying to open that as? I just changed the ownership to ossec:ossec, and on restart again:

2017/12/19 12:14:19 ossec-remoted: CRITICAL: (1103): Could not open file '/queue/rids/sender_counter' due to [(13)-(Permission denied)].

The client.keys in the agent never connected exists in the client.keys in the manager??

cat /var/ossec/etc/client.keys | grep agent_name and the result should be the same IP, Agent-name IP and key than in your agent.

That has all 32 keys in it. But it was owned by root. So after changing the ownership of client.keys,there's some progress, in that for each agent there's this now in the log:

...
2017/12/19 12:14:19 ossec-remoted: INFO: No previous counter available for 'vmw-recon-dev2'.
2017/12/19 12:14:19 ossec-remoted: INFO: Assigning counter for agent vmw-recon-dev2: '0:0'.
2017/12/19 12:14:19 ossec-remoted: INFO: No previous counter available for 'vmw-recon-devstore'.
2017/12/19 12:14:19 ossec-remoted: INFO: Assigning counter for agent vmw-recon-devstore: '0:0'.
...

But then after those:

2017/12/19 12:14:19 ossec-remoted: ERROR: Unable to open agent file. errno: 13

What is the "agent file" in that context?

Still lsof -i shows nothing listening to port 1514, and the Kibana Wazuh Agents screen show all as "Never connected." Yet startup appears without complaint:

# ./ossec-control start
Starting Wazuh v3.0.1 (maintained by Wazuh Inc.)...
Started wazuh-clusterd...
Started wazuh-modulesd...
Started ossec-maild...
Started ossec-execd...
Started ossec-analysisd...
Started ossec-logcollector...
Started ossec-remoted...
Started ossec-syscheckd...
Started ossec-monitord...
Completed.

That's with the ossec.conf, which includes port 1514:

 <remote>
    <connection>syslog</connection>
    <port>514</port>
    <protocol>udp</protocol>
  </remote>

  <remote>
    <connection>secure</connection>
    <port>1514</port>
    <protocol>udp</protocol>
  </remote>

Whit

[Whit Blauvelt]

Possibly related:

https://github.com/ossec/ossec-hids/issues/1061

[Whit Blauvelt]

Ahah! send_counter needed to be owned by ossecr:ossec, not ossec:ossec (which I'd changed it to from root ownership which it initially had).

[Whit Blauvelt]

The syscheck stuff was about needing to be owned by ossecr:ossec too.

If there's not one already, this project could use an ownership sanity test script.

Best,
Whit

[Jose Luis Ruiz]

How did you upgrade from OSSEC 2.9.x to Wazuh 3.0? from sources or packages?

Which OS are you using CentOS, Ubuntu Debian?

I will try to reproduce, but wave many downloads and installations and is the first notice that this is happening.

Regards

----------------

Jose Luis Ruiz

@jlruizmlg

www.wazuh.com

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f4ff3f0d-7be1-4555-bc0e-001037616d60%40googlegroups.com.

[Jose Luis Ruiz]

Looking your mails seems like something went wrong in your upgrade

Did you followed our migrating guide from our documentation?

https://documentation.wazuh.com/current/migrating-from-ossec/ossec-packages-manager.html#up-ossec-manager

Please try to follow the following steps:

1. Backup your current configuration
Stop OSSEC:

# /var/ossec/bin/ossec-control stop

Check if all services are stopped:

# ps axu | grep ossec

Check if you have enough space to create a copy of /var/ossec:

# du -h /var/ossec | tail -n1
# df -h /var
Backup /var/ossec:

# cp -rp /var/ossec /var/ossec_backup
2. Remove your current installation
Debian and Ubuntu:

# apt-get remove ossec-hids wazuh-manager wazuh-api —purge (I am not sure if you still having ossec-hids installed in your system, this is why i add to the command line)
CentOS and Red Hat:

# yum remove ossec-hids wazuh-manager wazuh-api (I am not sure if you still having ossec-hids installed in your system, this is why i add to the command line)
Remove directory:

# rm -rf /var/ossec

Now install from scratch Wazuh following the next link:

https://documentation.wazuh.com/current/installation-guide/installing-wazuh-server/index.html

The copy the client.keys from your ossec_backup to the new folder

cp -p /var/ossec_backup/etc/client.keys /var/ossec/etc/
chmod 660 /var/ossec_backup/etc/client.keys
chown root:ossec /var/ossec_backup/etc/client.keys

Restart the manager with service wazuh-manager restart and verify if you have any error in the log.

Then if you want, you can enable Auth with the instructions that i gave you in a previous mail in the same thread.

Regards

----------------

Jose Luis Ruiz

@jlruizmlg

www.wazuh.com

[Whit Blauvelt]

I'm mostly out of the woods. The remaining error is in Kibana > Wazuh > Manager which complains "Manager - Status: Unexpected error located on controller. Error: Error doing a request to Kibana API. (code -2)."

I did follows the website's instructions for an upgrade. The old system was OSSEC from sources; the new is Wazuh from deb on Ubuntu.

The failures were related to file ownership. There's simply nothing in the OSSEC upgrade instructions that sets the file ownership for file restored from the old configuration. Perhaps that's not a problem if the old configuration was from debs; but it certainly is if the old configuration is from sources. At least it was in this case.

Is there documentation of what the ownership and perms should be of all the various files and directories? It would be easy to write a script to check or even reset those.

By the way, your support is superb. As is the obvious work that's gone into the Kibana Wazuh app.

[jua...@wazuh.com]

Hi Whit, and sorry for the late response.

The file ownership is different for some folders of the wazuh installation folder (usually /var/ossec). It's difficult to revise every file and for now I can't give you any documentation for that. But what we can do is try to check out why you're getting that error on the Wazuh App.

Can you give us the output of your web browser's console when you get that "Error doing a request to Kibana API" error? We'll try to find out what is causing that problem.

Thank you for your patience.

Best regards,

Juanjo