WAZUH Stopped showing Agent Events

[zaffar abbas]

WAZUH was receiving Agent Events but all of the sudden it stopped fetching. Although I can see the event details over Wazuh Manager CLI but not on GUI Interface. 

Can someone please help?

[Mateo Cervilla]

Hello,

To give you some context, the flow of events is like this:

    Agent -> Manager (Server)-> Filebeat -> Indexer -> Dashboard

Somewhere in this sequence there may be some misconfiguration or error.

• Can see the alerts under /var/ossec/logs/alerts/alerts.json ?
  
  
• Can you try to generate an alert on the Manager and check if you can visulize it on the Dashboard?

You can re-check the settings and make sure everything is well configured. 

We can help you a bit more if you provide us with more information, like logs from the differents modules (Manager, Filebeat, Indexer, Dashboard) and then maybe we can find an error that is causing the problem.

Here is some information about Log data collection

Regards,

Mateo

[zaffar abbas]

Thank you for responding Mateo. The alert.json file you mentioned only show event logs till November 24th. However, There is a sub folder named 2022 containing log file of 10 days including today named ossec-alerts-14.log containing agent events information. 
Considering the event flow you highlighted, I guess the error occurs after Filebeat(only an assumption).

• Can you try to generate an alert on the Manager and check if you can visulize it on the Dashboard?

• • Could you please elaborate how can I check this?

[zaffar abbas]

[Mateo Cervilla]

Hi Zaffar,

So, in the file /var/ossec/logs/alerts/alerts.log (not alerts.json) the latest data is from November 24th too?

    >Can you try to generate an alert on the Manager and check if you can visualize it on the Dashboard?

        >Could you please elaborate how can I check this?

   
    Try to generate an alert on the machine running the manager, a ssh fail login for example. You should see on the alerts.log something like this:
       
        0.2.5,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,
        2022 Dec 14 06:57:27 localhost->/var/log/secure
        Rule: 5760 (level 5) -> 'sshd: authentication failed.'

    And then look on the dashboard if you can see that event.

The file you mention ossec-alerts-14.log should match the alerts.log, so that is unusual.

To be able to investigate your problem you can provide us with:

• Agents states and logs (ossec.log)
• Manager logs (ossec.log)
• Filebeat, Indexer and Dashboard logs (located in /var/log)

I will be looking for more information about your issue in the meantime.

Regards,

Mateo

[zaffar abbas]

 /var/ossec/logs/alerts/alerts.log
    This file is containing new events as well. This means ossec has these logs but they are not getting published over GUI of WAZUH-MANAGER or Kibana.

I have attached the OSSEC.LOG results below

[zaffar abbas]

Attached are the results for file Filebeat and Filebeat.1 inside (/var/log)

[zaffar abbas]

Hi Mateo, Have you had a chance to  look into these? 

[Mateo Cervilla]

Hi Zaffar,

From what I can see, you may have a problem with the alerts.json file. 

First of all, you should check if the latest alerts from it really don't match the ones from alerts.log.

If so, you should make a copy of it and create a new one to check if the problem continues. After this restart the manager and check again.

It could also be a bad configuration on the /var/ossec/etc/ossec.conf. You can check it and also copy and paste it here so I can take a look at it.

P.S. This is just my preference but I would appreciate if you can copy and paste the text of the logs instead of taking screenshots the next time, it is easier to read this way.

Kind regards!

Mateo

[zaffar abbas]

Hi Mateo, apologies for responding late. The weekend just hit harder than usual!

I had created a new blank file inside /var/ossec/logs/alerts named alerts.json and had moved the original one to the desktop. After that I had restarted wazuh manager along with other services (kibana/ filebeat/ elasticsearch) but issue persisted.

Further, here is the ossec.conf file result you had asked for.

<ossec_config>
  <global>
    <jsonout_output>no</jsonout_output>
    <alerts_log>yes</alerts_log>
    <logall>no</logall>
    <logall_json>no</logall_json>
    <email_notification>yes</email_notification>
    <smtp_server>10.14.1.10</smtp_server>
    <email_from>abc.com</email_from>
    <email_to>yyy.com</email_to>
    <email_to>zzz.com</email_to>
    <email_maxperhour>12</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>
    <agents_disconnection_time>10m</agents_disconnection_time>
    <agents_disconnection_alert_time>0</agents_disconnection_alert_time>
  </global>

<email_alerts>
    <email_to>yyy.com</email_to>
    <rule_id>60122</rule_id>
    <format>full</format>
    <level>5</level>
    <do_not_delay />
</email_alerts>

  <alerts>
    <log_alert_level>3</log_alert_level>
    <email_alert_level>5</email_alert_level>
  </alerts>

  <!-- Choose between "plain", "json", or "plain,json" for the format of internal logs -->
  <logging>
    <log_format>json</log_format>
  </logging>

  <remote>
    <connection>secure</connection>
    <port>1514</port>
    <protocol>tcp</protocol>
    <queue_size>131072</queue_size>
  </remote>

  <!-- Policy monitoring -->
  <rootcheck>
    <disabled>no</disabled>
    <check_files>yes</check_files>
    <check_trojans>yes</check_trojans>
    <check_dev>yes</check_dev>
    <check_sys>yes</check_sys>
    <check_pids>yes</check_pids>
    <check_ports>yes</check_ports>
    <check_if>yes</check_if>

    <!-- Frequency that rootcheck is executed - every 12 hours -->
    <frequency>43200</frequency>

    <rootkit_files>etc/rootcheck/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>etc/rootcheck/rootkit_trojans.txt</rootkit_trojans>

    <skip_nfs>yes</skip_nfs>
  </rootcheck>

  <wodle name="cis-cat">
    <disabled>yes</disabled>
    <timeout>1800</timeout>
    <interval>1d</interval>
    <scan-on-start>yes</scan-on-start>

    <java_path>wodles/java</java_path>
    <ciscat_path>wodles/ciscat</ciscat_path>
  </wodle>

  <!-- Osquery integration -->
  <wodle name="osquery">
    <disabled>yes</disabled>
    <run_daemon>yes</run_daemon>
    <log_path>/var/log/osquery/osqueryd.results.log</log_path>
    <config_path>/etc/osquery/osquery.conf</config_path>
    <add_labels>yes</add_labels>
  </wodle>

  <!-- System inventory -->
  <wodle name="syscollector">
    <disabled>no</disabled>
    <interval>1h</interval>
    <scan_on_start>yes</scan_on_start>
    <hardware>yes</hardware>
    <os>yes</os>
    <network>yes</network>
    <packages>yes</packages>
    <ports all="no">yes</ports>
    <processes>yes</processes>

    <!-- Database synchronization settings -->
    <synchronization>
      <max_eps>10</max_eps>
    </synchronization>
  </wodle>

  <sca>
    <enabled>yes</enabled>
    <scan_on_start>yes</scan_on_start>
    <interval>12h</interval>
    <skip_nfs>yes</skip_nfs>
  </sca>

  <vulnerability-detector>
    <enabled>yes</enabled>
    <interval>5m</interval>
    <min_full_scan_interval>5m</min_full_scan_interval>
    <run_on_start>yes</run_on_start>

    <!-- Ubuntu OS vulnerabilities -->
    <provider name="canonical">
      <enabled>no</enabled>
      <os>trusty</os>
      <os>xenial</os>
      <os>bionic</os>
      <os>focal</os>
      <os>jammy</os>
      <update_interval>1h</update_interval>
    </provider>

    <!-- Debian OS vulnerabilities -->
    <provider name="debian">
      <enabled>no</enabled>
      <os>stretch</os>
      <os>buster</os>
      <os>bullseye</os>
      <update_interval>1h</update_interval>
    </provider>

    <!-- RedHat OS vulnerabilities -->
    <provider name="redhat">
      <enabled>no</enabled>
      <os>5</os>
      <os>6</os>
      <os>7</os>
      <os>8</os>
      <os>9</os>
      <update_interval>1h</update_interval>
    </provider>

    <!-- Amazon Linux OS vulnerabilities -->
    <provider name="alas">
      <enabled>no</enabled>
      <os>amazon-linux</os>
      <os>amazon-linux-2</os>
      <update_interval>1h</update_interval>
    </provider>

    <!-- Arch OS vulnerabilities -->
    <provider name="arch">
      <enabled>no</enabled>
      <update_interval>1h</update_interval>
    </provider>

    <!-- Windows OS vulnerabilities -->
    <provider name="msu">
      <enabled>yes</enabled>
      <update_interval>1h</update_interval>
    </provider>

    <!-- Aggregate vulnerabilities -->
    <provider name="nvd">
      <enabled>yes</enabled>
      <update_from_year>2010</update_from_year>
      <update_interval>1h</update_interval>
    </provider>

  </vulnerability-detector>

  <!-- File integrity monitoring -->
  <syscheck>
    <disabled>no</disabled>

    <!-- Frequency that syscheck is executed default every 12 hours -->
    <frequency>43200</frequency>

    <scan_on_start>yes</scan_on_start>

    <!-- Generate alert when new file detected -->
    <alert_new_files>yes</alert_new_files>

    <!-- Don't ignore files that change more than 'frequency' times -->
    <auto_ignore frequency="10" timeframe="3600">no</auto_ignore>

    <!-- Directories to check  (perform all possible verifications) -->
    <directories>/etc,/usr/bin,/usr/sbin</directories>
    <directories>/bin,/sbin,/boot</directories>

    <!-- Files/directories to ignore -->
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
    <ignore>/etc/random-seed</ignore>
    <ignore>/etc/random.seed</ignore>
    <ignore>/etc/adjtime</ignore>
    <ignore>/etc/httpd/logs</ignore>
    <ignore>/etc/utmpx</ignore>
    <ignore>/etc/wtmpx</ignore>
    <ignore>/etc/cups/certs</ignore>
    <ignore>/etc/dumpdates</ignore>
    <ignore>/etc/svc/volatile</ignore>

    <!-- File types to ignore -->
    <ignore type="sregex">.log$|.swp$</ignore>

    <!-- Check the file, but never compute the diff -->
    <nodiff>/etc/ssl/private.key</nodiff>

    <skip_nfs>yes</skip_nfs>
    <skip_dev>yes</skip_dev>
    <skip_proc>yes</skip_proc>
    <skip_sys>yes</skip_sys>

    <!-- Nice value for Syscheck process -->
    <process_priority>10</process_priority>

    <!-- Maximum output throughput -->
    <max_eps>100</max_eps>

    <!-- Database synchronization settings -->
    <synchronization>
      <enabled>yes</enabled>
      <interval>5m</interval>
      <max_interval>1h</max_interval>
      <max_eps>10</max_eps>
    </synchronization>
  </syscheck>

  <!-- Active response -->
  <global>
    <white_list>127.0.0.1</white_list>
    <white_list>^localhost.localdomain$</white_list>
    <white_list>127.0.0.53</white_list>
  </global>

  <command>
    <name>disable-account</name>
    <executable>disable-account</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>restart-wazuh</name>
    <executable>restart-wazuh</executable>
  </command>

  <command>
    <name>firewall-drop</name>
    <executable>firewall-drop</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>host-deny</name>
    <executable>host-deny</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>route-null</name>
    <executable>route-null</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>win_route-null</name>
    <executable>route-null.exe</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <command>
    <name>netsh</name>
    <executable>netsh.exe</executable>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <!--
  <active-response>
    active-response options here
  </active-response>
  -->

  <!-- Log analysis -->
  <localfile>
    <log_format>command</log_format>
    <command>df -P</command>
    <frequency>360</frequency>
  </localfile>

  <localfile>
    <log_format>full_command</log_format>
    <command>netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d</command>
    <alias>netstat listening ports</alias>
    <frequency>360</frequency>
  </localfile>

  <localfile>
    <log_format>full_command</log_format>
    <command>last -n 20</command>
    <frequency>360</frequency>
  </localfile>

  <ruleset>
    <!-- Default ruleset -->
    <decoder_dir>ruleset/decoders</decoder_dir>
    <rule_dir>ruleset/rules</rule_dir>
    <rule_exclude>0215-policy_rules.xml</rule_exclude>
    <list>etc/lists/audit-keys</list>
    <list>etc/lists/amazon/aws-eventnames</list>
    <list>etc/lists/security-eventchannel</list>

    <!-- User-defined ruleset -->
    <decoder_dir>etc/decoders</decoder_dir>
    <rule_dir>etc/rules</rule_dir>
  </ruleset>

  <rule_test>
    <enabled>yes</enabled>
    <threads>1</threads>
    <max_sessions>64</max_sessions>
    <session_timeout>15m</session_timeout>
  </rule_test>

  <!-- Configuration for wazuh-authd -->
  <auth>
    <disabled>no</disabled>
    <port>1515</port>
    <use_source_ip>no</use_source_ip>
    <purge>yes</purge>
    <use_password>no</use_password>
    <ciphers>HIGH:!ADH:!EXP:!MD5:!RC4:!3DES:!CAMELLIA:@STRENGTH</ciphers>
    <!-- <ssl_agent_ca></ssl_agent_ca> -->
    <ssl_verify_host>no</ssl_verify_host>
    <ssl_manager_cert>etc/sslmanager.cert</ssl_manager_cert>
    <ssl_manager_key>etc/sslmanager.key</ssl_manager_key>
    <ssl_auto_negotiate>no</ssl_auto_negotiate>
  </auth>

  <cluster>
    <name>wazuh</name>
    <node_name>node01</node_name>
    <node_type>master</node_type>
    <key></key>
    <port>1516</port>
    <bind_addr>0.0.0.0</bind_addr>
    <nodes>
        <node>NODE_IP</node>
    </nodes>
    <hidden>no</hidden>
    <disabled>yes</disabled>
  </cluster>

</ossec_config>

<ossec_config>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/ossec/logs/active-responses.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/dpkg.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/kern.log</location>
  </localfile>

</ossec_config>
root@wazuh-VirtualBox:/var/ossec/etc#

[Mateo Cervilla]

Hi Zaffar,

Can you try changing the jsonout_output value to yes and check again?

It is located at the beginning of the ossec.conf file.

    <ossec_config>
        <global>
            <jsonout_output>yes</jsonout_output>

[zaffar abbas]

Aah! This finally worked Mat! A short human error resulted in a long diagnosis.

Thanks a lot for your support Mat. 

There is one more thing that I need support on. Currently when I open WAZHU interface, there is no validation or login page that can validate user. Could you please guide me on that as well? 

[Mateo Cervilla]

Hi Zaffar,

I'm glad it worked.

About the other issue, what you mean is that:

• Once you log in, the login page does not appear again
  
• The login page never appeared

You can try to log out, clear the cookies or enter with incognito mode.

If this is not the case, can you explain a bit more?

Regards,

Mateo

[Mateo Cervilla]

I'm pasting the private message here to continue the thread:

Hi Mateo,

Actually, the login page never appeared. When I access WAZUH interface over browser using its IP address and Port number it directly takes me to interface of Kibana and I can access everything over there. I even created another but still it doesnot ask me for any credentials and directly lends me to the interface.

I certainly need WAZUH to validate user first before granting access to interface. Could you please guide me on this?

It's strange behaviour. Can you try any of this to see if the login page appears?

• Try to log out
• Clear the cookies
• Enter with incognito mode

And check again.

In the meantime I'll be investigating the problem.

Regards,

Mateo

[Mateo Cervilla]

On Tue, Dec 20, 2022 at 2:24 PM zaffar abbas wrote:

I have tried using multiple browsers to access WAZUH interface but it never asks for any credentials.

I cannot find any option for logging in or logout either. Is there any possibility if the authentication or user validation is turned off by manger's side? How can we check it?

Hi Zaffar,

Please use Reply all  in Google Groups so we continue this conversation here.

I'm currently taking a look at your problem. I will reply as soon as I can.

Regards,

Mateo

[zaffar abbas]

Looking forward for your response Mateo!

[Mateo Cervilla]

Hi Zaffar,

Can you attach these files so I can take a look? (You can directly attach the files to the message instead of copy and paste)

• /etc/wazuh-dashboard/opensearch_dashboards.yml      (/etc/kibana/kibana.yml for kibana)
• /etc/wazuh-indexer/opensearch.yml       (/etc/elasticsearch/elasticsearch.yml for elastichsearch)

They may contain some possible configuration that enables anonymous access.

Regards,

Mateo

[zaffar abbas]

Apologies Mateo, somehow I was unable to attach the yml file. It was attaching the path instead of file. Here is the configuration you had asked for:

Please be known I have not configured SSL cert yet.

------Kibana------

server.host: 0.0.0.0
server.port: 8080
elasticsearch.hosts: http://10.14.6.13:9200
#elasticsearch.password: "ABC123"

#Elasticsearch from/to Kibana

#elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/ca/ca.crt
#elasticsearch.ssl.certificate: /etc/kibana/certs/kibana.crt
#elasticsearch.ssl.key: /etc/kibana/certs/kibana.key

#Browser from/to Kibana
#server.ssl.enabled: true
#server.ssl.certificate: /etc/kibana/certs/kibana.crt
#server.ssl.key: /etc/kibana/certs/kibana.key

#Elasticsearch authentication
#xpack.security.enabled: true
#elasticsearch.username: "elastic"
#uiSettings.overrides.defaultRoute: "/app/wazuh"
#elasticsearch.ssl.verificationMode: none
#telemetry.banner: false

-------ElasticSearch------

network.host: 10.14.6.13
node.name: elasticsearch
cluster.initial_master_nodes: elasticsearch

# Transport layer
#xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: none
#xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
#xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
#xpack.security.transport.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# HTTP layer
#xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.verification_mode: certificate
#xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
#xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
#xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

#Elasticsearch authentication
#xpack.security.enabled: true

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

[Mateo Cervilla]

Hi Zaffar,

I can see that most of the settings are default so the problem may not come from here. 

I'm going to consult with my team to see if we can provide you with a solution.

Regards,

Mateo

[zaffar abbas]

Sure Mateo, I will be waiting for your further response.

Thanks!

[Mateo Cervilla]

Hi Zaffar,

Can you try uncommenting ( removing the # ) the lines:

• xpack.security.enabled: true

in both elasticsearch.yml and kibana.yml, restart the services and check again?

I'll wait for your response.

Mateo

[zaffar abbas]

Hi Mateo, After enabling xpack security in elasticsearch. The following attached alert got generated and elasticsearch was unable to restart

[Mateo Cervilla]

Hello Zaffar,

As you can see, the log reports "Transport SSL must be enabled if security is enabled".

If you want to use any of the xpack security features by enabling (setting xpack.security.enabled = true), then you need to use TLS/SSL certificate.

I recommend you to follow the instructions for Installing Wazuh with Elastic Stack basic license. You can do it with the one that meets your requirements:

• All-in-one deployment
  
  • Where Wazuh server and Elastic Stack are installed on the same host.
• Distributed deployment
  
  • Where each component is installed in a separate host as a single-node or multi-node cluster.

You will see in this documentation that elasticsearch.yml and kibana.yml are downloaded and placed. If you read them, you will find that the configurations I gave you before are specified there.

If you need any more help, please let me know.

Regards,

Mateo

[zaffar abbas]

Hi Mateo, this worked! Thanks a lot for all of your support brother. Much appreciated!