wazuh-kibana-app: all agents "disconnected"

[Michael McCarn]

Since updating to Wazuh 3.2.1, using wazuhapp-3.2.1_6.2.2.zip, all of my agents report "disconnected" within 2 - 4 hours of rebooting the wazuh server.

If I reboot the server, the agents reconnect -- again, for 2 - 4 hours, then become disconnected again.

[jesus.g...@wazuh.com]

Hi Michael, sorry you are having troubles with your upgrade. Please could you make a request directly to the Wazuh API

to discard if it's a Wazuh App related error?

# curl "wazuh_api_ip:55000/agents?select=status&pretty" -u user:pass

If you are seeing a disconnected agent on the Wazuh App, that appears as Active on the Wazuh API we have an App problem.

Otherwise give us a bit more information about your environment in order to give you a better assitance:

On the manager machine: 

# cat /var/ossec/logs/ossec.log | grep ERROR

# cat /var/ossec/logs/ossec.log | grep WARNING

- Which OS are you using on the manager machine?

- How you upgraded your manager and your agents?

- Are the from packages, repositories? 

- Which OS are using the wrong agents?

Thanks in advance, have a nice day.

Best regards,

Jesús

[Whit Blauvelt]

Still gathering data, but we're seeing a portion of our Windows agents disconnected over several days. Linux agents not having that problem. Restarting the (Linux) server, 2 of the 22 agents disconnect have reconnected. How fast are the reconnections happening in your case? Are they all Windows agents? We're on wazuhapp-3.2.0_6.2.1.zip

Whit

[Michael McCarn]

Jesús -

Thanks for your help.

Here are the results of the commands you listed:

# curl --insecure "https://redacted-IP:55000/agents?select=status&pretty" -u wapi:redacted-password

{

  "error": 0,

  "data": {

     "totalItems": 10,

     "items": [

        {

           "status": "Active",

           "id": "000"

        },

        {

           "status": "Disconnected",

           "id": "001"

        },

        {

           "status": "Disconnected",

           "id": "003"

        },

        {

           "status": "Disconnected",

           "id": "004"

        },

        {

           "status": "Disconnected",

           "id": "005"

        },

        {

           "status": "Disconnected",

           "id": "006"

        },

        {

           "status": "Disconnected",

           "id": "009"

        },

        {

           "status": "Disconnected",

           "id": "010"

        },

        {

           "status": "Disconnected",

           "id": "011"

        },

        {

           "status": "Disconnected",

           "id": "012"

        }

     ]

  }

}

/var/ossec/logs/ossec.log does not exist:

# ls /var/ossec/logs/ossec.log

ls: cannot access /var/ossec/logs/ossec.log: No such file or directory

I've attached 'ERROR.log' and 'WARNING.log' for Mar 4 created using the commands shown below (WARNING.log is empty)

# zgrep ERROR /var/ossec/logs/ossec/2018/Mar/ossec-04.log.gz >ERROR.log

# zgrep WARNING /var/ossec/logs/ossec/2018/Mar/ossec-04.log.gz >WARNING.log

Upgrade Notes (I have more details if you need them).  

• Jan 14
• Install the 3.1.0/6.1.1 virtual appliance
• Feb 14
• Upgrade elastic search,kibana,logstash, wazuh-api, wazuh-manager
• Feb 15
• wazuh app not loading; remove and reload app; clear indexes; remove and rebuild bundles
• Feb 19
• Update wazuh-server to 3.2.0-2
• Mar 1 
• upgrade elastic search, kibana and logstash using 'yum update'
• Upgrade wazuh-kibana-app
• kibana message "can't talk to logstash"
• systemctl stop kibana
• # curl -XDELETE localhost:9200/.kibana
• # curl -XDELETE localhost:9200/.wazuh
• # curl -XDELETE localhost:9200/.wazuh-version
• # rm -rf /usr/share/kibana/optimize/bundles
• # systemctl start kibana
• Default API not working
• create api user "wapi"
• enable self-signed SSL using /var/ossec/api/scripts/configure_api.sh
• Change API URL in kibana to use new url, https
• Mar 3 
• Update Wazuh using "yum update"
• Update windows, centos, ubuntu agents using 'var/ossec/bin/agent_upgrade -a <xxx>'
• Update OS X agents using downloaded pkg
• Update Proxmox agent from source
• System seemed OK
• Mar 4
• "Ups something went wrong"
• Cleared the indexes; seemed to help but failed again
• Removed and reinstalled wazuh-kibana-app

On Monday, March 5, 2018 at 8:23:31 AM UTC-5, Michael McCarn wrote:

[Michael McCarn]

Update:

I rebooted wazuh-server.  The ERROR entries in /var/ossec/logs/ossec.log after reboot started looked like possible permission errors:

# grep ERROR /var/ossec/logs/ossec.logs

2018/03/06 06:14:09 ossec-analysisd: ERROR: at sc_send_db(): received: 'err Cannot save HW information.'

2018/03/06 06:14:09 ossec-remoted: ERROR: socketerr (not available).

2018/03/06 06:14:09 ossec-remoted: ERROR: (1210): Queue '/queue/ossec/queue' not accessible: 'Connection refused'.

2018/03/06 06:14:09 ossec-logcollector: ERROR: socketerr (not available).

2018/03/06 06:14:09 ossec-logcollector: ERROR: (1224): Error sending message to queue.

2018/03/06 06:14:12 ossec-remoted: ERROR: (1210): Queue '/queue/ossec/queue' not accessible: 'Connection refused'.

2018/03/06 06:14:12 ossec-logcollector: ERROR: (1210): Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.

2018/03/06 06:14:51 wazuh-modulesd: ERROR: socketerr (not available).

I ran these commands:

# rpm --setperms wazuh-manager

# rpm --setugids wazuh-manager

# timedatectl set-timezone America/New_York

# reboot

Since rebooting:

• There are no new ERROR entries in /var/ossec/logs/ossec.log
  
• My agents have reconnected
• Data is accumulating again...

[jesus.g...@wazuh.com]

Hi Michael McCarn, 

Sorry for the late response, as I can see you have had a permission problem, the Wazuh Core team

already knows about these kind of errors and they are currently working to solve them for our new package

who is going to be published soon (I don't know exactly). Since your last message I understand that your problem 

is solved by yourself.

Any case let me know if you are facing any kind of new trouble. 

Hi Whit, regarding to your message we have new package since Wazuh 3.2.0 and I remember your environment is up

to date from other thread, also these kind of errors are solved by now or are going to be solved soon. Any case and as I said to Michael, 

feel free to tell us if you are facing errors or open a new thread if you think it needs.

All you guys have a nice day.

Best regards,

Jesús

[Michael McCarn]

Thanks for the followup.

My efforts have not helped.  If I reboot the Wazuh Manager, it stays online for about 30 minutes, then various services start going offline.

I'm waiting for 3.2.2 to see if that solves my problem.