New linux agent fails to connect

[Ed Sandberg]

I deployed a new wazuh cluster v4.3.0 on ubuntu 20. I followed the agent installation instructions for linux on an ubuntu VM called cosmos and ran:

# curl -so wazuh-agent-4.3.0.deb https://packages.wazuh.com/4.x/apt/pool/main/w/wazuh-agent/wazuh-agent_4.3.0-1_amd64.deb && sudo WAZUH_MANAGER='192.168.220.41' dpkg -i ./wazuh-agent-4.3.0.deb

sudo systemctl daemon-reload
sudo systemctl enable wazuh-agent
sudo systemctl start wazuh-agent

This installs the agent but the service fails to start:

# sudo systemctl status wazuh-agent
â wazuh-agent.service - Wazuh agent
   Loaded: loaded (/usr/lib/systemd/system/wazuh-agent.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2022-05-16 12:59:00 CDT; 5s ago
  Process: 28905 ExecStart=/usr/bin/env /var/ossec/bin/wazuh-control start (code=exited, status=1/FAILURE)

May 16 12:59:00 cosmos systemd[1]: Starting Wazuh agent...
May 16 12:59:00 cosmos env[28905]: 2022/05/16 12:59:00 wazuh-agentd: ERROR: (4112): Invalid server address found: 'MANAGER_IP'
May 16 12:59:00 cosmos env[28905]: 2022/05/16 12:59:00 wazuh-agentd: CRITICAL: (1215): No client configured. Exiting.
May 16 12:59:00 cosmos env[28905]: wazuh-agentd: Configuration error. Exiting
May 16 12:59:00 cosmos systemd[1]: wazuh-agent.service: Control process exited, code=exited status=1
May 16 12:59:00 cosmos systemd[1]: Failed to start Wazuh agent.
May 16 12:59:00 cosmos systemd[1]: wazuh-agent.service: Unit entered failed state.
May 16 12:59:00 cosmos systemd[1]: wazuh-agent.service: Failed with result 'exit-code'.

So it looks like the env var is not setting the IP properly. I can verify this by editing /var/ossec/etc/ossec.conf where I see:

<address>MANAGER_IP</address>

If I insert the correct IP the service starts but the agent doesn't connect or register. I see this on the wazuh cluster master:

# /var/ossec/bin/agent_control -l

Wazuh agent_control. List of available agents:
   ID: 000, Name: wazuh1 (server), IP: 127.0.0.1, Active/Local
   ID: 001, Name: es-xps, IP: any, Active

so the new agent is not listed. If I go to the agent machine and run:

# /var/ossec/bin/agent-auth -m 192.168.220.41
2022/05/16 13:08:05 agent-auth: INFO: Started (pid: 31835).
2022/05/16 13:08:05 agent-auth: INFO: Requesting a key from server: 192.168.220.41
2022/05/16 13:08:05 agent-auth: INFO: No authentication password provided
2022/05/16 13:08:05 agent-auth: INFO: Using agent name as: cosmos
2022/05/16 13:08:05 agent-auth: INFO: Waiting for server reply
2022/05/16 13:08:05 agent-auth: INFO: Valid key received

Now the agent has registered but never connects. I see this on the cluster master:

# /var/ossec/bin/agent_control -l

Wazuh agent_control. List of available agents:
   ID: 000, Name: wazuh1 (server), IP: 127.0.0.1, Active/Local
   ID: 001, Name: es-xps, IP: any, Active
   ID: 006, Name: cosmos, IP: any, Never connected

No errors show up in the log:

# tail /var/ossec/logs/ossec.log
2022/05/16 18:08:05 wazuh-authd: INFO: New connection from 192.168.222.56
2022/05/16 18:08:05 wazuh-authd: INFO: Received request for a new agent (cosmos) from: 192.168.222.56
2022/05/16 18:08:05 wazuh-authd: INFO: Agent key generated for 'cosmos' (requested by any)
2022/05/16 18:08:09 wazuh-remoted: INFO: (1409): Authentication file changed. Updating.
2022/05/16 18:08:09 wazuh-remoted: INFO: (1410): Reading authentication keys file.

Restarting the service on the agent doesn't help. The firewall is wide open on the agent and the wazuh server:

# ufw status
Status: inactive
# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

[Federico Pacher]

Hi there,

Thank you for using Wazuh.

Could you please share the documentation you followed to install the Wazuh Agent in order to check it?.

The steps to  deploying Wazuh agents on Ubuntu Linux systems are the following:

1- Install the GPG key:

#  curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -

2- Add the repository

# echo "deb https://packages.wazuh.com/4.x/apt/ stable main" | tee -a /etc/apt/sources.list.d/wazuh.list

3- Update the package information:

# apt-get update

# WAZUH_MANAGER="192.168.220.41" apt-get install wazuh-agent

Be careful to use double quotes (") in the variable WAZUH_MANAGER.

Enable and start the Wazuh agent service.

# systemctl daemon-reload 

# systemctl enable wazuh-agent 

# systemctl start wazuh-agent

Recommended action - Disable Wazuh updates

Compatibility between the Wazuh agent and the Wazuh manager is guaranteed when the Wazuh manager version is later than or equal to that of the Wazuh agent. Therefore, we recommend disabling the Wazuh repository to prevent accidental upgrades. To do so, use the following command:

# sed -i "s/^deb/#deb/" /etc/apt/sources.list.d/wazuh.list 

# apt-get update

Here you have the official documentation of Wazuh to install an Agent.

I hope this information can help you.

Regards.

[Federico Pacher]

Hi there again,

I have been debugging what you have done to register your ubuntu Agent into your cluster and the registration using the command you used was successful. Could you mind sharing some information with me in order to go deeper into your problem?:

What I need is:
1- ossec.conf of your managers (master node and workers nodes), in case you cannot share these files please share how to configure the <cluster> tag of your master node and worker nodes, something like this:

  <cluster>
    <name>wazuh</name>
    <node_name>node02</node_name>
    <node_type>worker</node_type>
    <key>248d7d032ab400e6609308405fd8243f</key>
    <port>1516</port>
    <bind_addr>0.0.0.0</bind_addr>
    <nodes>
        <node>192.168.0.246</node>
    </nodes>
    <hidden>no</hidden>
    <disabled>no</disabled>
  </cluster>

2- the ossec.log file of your agent located in /var/ossec/log/ossec.log

Regards

[ed sandberg]

Responses inline.

On 5/16/22 13:44, Federico Pacher wrote:
> Could you please share the documentation you followed to install the
> Wazuh Agent in order to check it?.

In the Wazuh web interface navigate to Agents > Deploy new agent and
choose Ubuntu. Installation instructions are documented there that match
what I described in my previous email.

>
> The steps to  deploying Wazuh agents on Ubuntu Linux systems are the
> following:

> <https://documentation.wazuh.com/current/installation-guide/wazuh-agent/wazuh-agent-package-linux.html#deploying-wazuh-agents-on-linux-systems>

Those steps result in the exact same problems I saw with the
instructions given by the Wazuh web interface. The env var is still not
setting the IP properly, and correcting that still doesn't allow the VM
to connect.

[Federico Pacher]

Hi there,

Sorry for my late response. Would you mind sharing:

1- ossec.conf of your managers (master node and workers nodes), in case you cannot share these files please share how to configure the <cluster> tag of your master node and worker nodes, something like this:

  <cluster>
    <name>wazuh</name>
    <node_name>node02</node_name>
    <node_type>worker</node_type>
    <key>248d7d032ab400e6609308405fd8243f</key>
    <port>1516</port>
    <bind_addr>0.0.0.0</bind_addr>
    <nodes>
        <node>192.168.0.246</node>
    </nodes>
    <hidden>no</hidden>
    <disabled>no</disabled>
  </cluster>

2- the ossec.log file of your agent located in /var/ossec/log/ossec.log

Please check if your port 1515 and 1514 in your managers and your agents are open 
- 1514 Agent connection service
- 1515 Agent enrollment service

To check this:

Agent side

# nmap -p 1515 manager_ip

Manager side

# nmap -p 1515 agent_ip

The same with port 1514.

Regards

[ed sandberg]

Hey Federico,

I tried it again this morning and had slightly better luck. The env var
is still not setting the IP properly but the agent did connect after I
manually altered it and restarted the agent service. Not sure why it
connected this time and not before. I have lots of end users and it
would be better if they didn't have to manually edit config files to get
this to connect so I would still like a resolution of that problem if
possible.

I attached the ossec.conf files from both servers in the cluster.

Here are the nmap results:

From agent to server:

# nmap -p 1514,1515 192.168.220.41
PORT STATE SERVICE
1514/tcp open fujitsu-dtcns
1515/tcp open ifor-protocol

from server to agent:

# nmap -p 1514,1515 192.168.222.56

PORT STATE SERVICE
1514/tcp closed fujitsu-dtcns
1515/tcp closed ifor-protocol

Not sure why they show closed from the server to the agent. The
connection succeeded now and the firewall is disable on both machines:

# ufw status
Status: inactive

# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

but I can see that the server VM is listening but the agent VM is not
attempting to listen on those ports:

server listening ports:

# sudo netstat -tulpn | grep .*wazuh.*
tcp 0 0 0.0.0.0:1514 0.0.0.0:*
LISTEN 1940/wazuh-remoted
tcp 0 0 0.0.0.0:1515 0.0.0.0:*
LISTEN 1520/wazuh-authd

agent listening ports:

# netstat -tulpn | grep .*wazuh.*

On 5/19/22 07:15, Federico Pacher wrote:
> Hi there,
>
> Sorry for my late response. Would you mind sharing:
>
> 1- ossec.conf of your managers (master node and workers nodes), in case
> you cannot share these files please share how to configure the

> *<cluster> *tag of your master node and worker nodes, something like this:
> /*

>   <cluster>
>     <name>wazuh</name>
>     <node_name>node02</node_name>
>     <node_type>worker</node_type>
>     <key>248d7d032ab400e6609308405fd8243f</key>
>     <port>1516</port>
>     <bind_addr>0.0.0.0</bind_addr>
>     <nodes>
>         <node>192.168.0.246</node>
>     </nodes>
>     <hidden>no</hidden>
>     <disabled>no</disabled>

>   </cluster>*/
>
> 2- the ossec.log file of your agent located in /*/var/ossec/log/ossec.log*/

>
> Please check if your port 1515 and 1514 in your managers and your agents
> are open
> - 1514 Agent connection service
> - 1515 Agent enrollment service
>
> To check this:
>
> Agent side

> # /nmap -p 1515 manager_ip/
> Manager side
> # /nmap -p 1515 agent_ip/

>
> The same with port 1514.
>
> Regards
>
> On Tuesday, May 17, 2022 at 12:05:08 PM UTC-3 ed.sa...@adventiumlabs.com
> wrote:
>
> Responses inline.
>
> On 5/16/22 13:44, Federico Pacher wrote:
> > Could you please share the documentation you followed to install the
> > Wazuh Agent in order to check it?.
>
> In the Wazuh web interface navigate to Agents > Deploy new agent and
> choose Ubuntu. Installation instructions are documented there that
> match
> what I described in my previous email.
>
> >
> > The steps to  deploying Wazuh agents on Ubuntu Linux systems are the
> > following:
> >
> <https://documentation.wazuh.com/current/installation-guide/wazuh-agent/wazuh-agent-package-linux.html#deploying-wazuh-agents-on-linux-systems
> <https://documentation.wazuh.com/current/installation-guide/wazuh-agent/wazuh-agent-package-linux.html#deploying-wazuh-agents-on-linux-systems>>
>
>
> Those steps result in the exact same problems I saw with the
> instructions given by the Wazuh web interface. The env var is still not
> setting the IP properly, and correcting that still doesn't allow the VM
> to connect.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Wazuh mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wazuh+un...@googlegroups.com
> <mailto:wazuh+un...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/wazuh/a814a8ec-5039-451a-9ba3-f8b9f890ecaen%40googlegroups.com
> <https://groups.google.com/d/msgid/wazuh/a814a8ec-5039-451a-9ba3-f8b9f890ecaen%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Edward Sandberg
Adventium Labs
111 3rd Avenue S. Suite #100
Minneapolis, MN 55401
ed.sa...@adventiumlabs.com

[Federico Pacher]

Hi Edward,

First of all, you must open ports 1515 and 1514 in order to get the connection between agents and managers. 
You can do it by typing:

sudo ufw allow 1191/tcp

I have done many tests and I could not replicate your problem. I am using vagrant with these VM  'ubuntu/bionic64' and 'ubuntu/focal64'. Would you mind giving some references of which VM are you using in order to replicate exactly your environment.

Please share the ossec.log of the agent located in /var/ossec/logs to check for any ERROR or WARNING messages you are getting.

Regards