Alerts not shown on dashboard

[rlin...@networkconfig.net]

Hi All,

Wazuh server unable to shown alerts on dashboard but alerts generates in logs. I am unable to understand what's the issue. I think index pattern have some problem. ?

** Alert 1536011683.74370: mail - ossec,pci_dss_10.6.1,gpg13_10.1,gdpr_IV_35.7.d,
2018 Sep 03 17:54:43 localhost->ossec-monitord
Rule: 502 (level 3) -> 'Ossec server started.'
ossec: Ossec started.


** Alert 1536011893.74570: - web,accesslog,attack,pci_dss_6.5,pci_dss_11.4,gdpr_IV_35.7.d,
2018 Sep 03 17:58:13 nagios-server->(nagios-server) 192.168.0.29->/var/log/httpd/access_log
Rule: 31101 (level 5) -> 'Web server 400 error code.'
Src IP: 192.168.0.10
192.168.0.10 - "" [03/Sep/2018:17:58:20 +0500] "GET /nagios HTTP/1.1" 401 479 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"

Regards,

rlinux57

[rlin...@networkconfig.net]

Hi,

Anyone please update this thread.

Regards,

rlinux57

[jesus.g...@wazuh.com]

Hi @rlinux57,

Let's check the agent "001":

- Open a CLI in the manager machine and paste the content of the next command:

cat /var/ossec/logs/alerts/alerts.json | grep "001"

- Open a CLI in the agent machine and paste the content of the next command:

cat /var/ossec/logs/ossec.log | grep -i -E "(warning|error|critical)"

What we are doing is looking for alerts from the agent in the manager (step 1) and looking for error messages in the agent machine (step 2).

Regards,

Jesús

[rlin...@networkconfig.net]

Hi Jesús,

Yes i got your point.

Log detail of wazuh server:

[root@localhost ~]# cat /var/ossec/logs/alerts/alerts.json | grep "001"
{"timestamp":"2018-09-04T10:31:24.800-0400","rule":{"level":6,"description":"sshd: insecure connection attempt (scan).","id":"5706","firedtimes":1,"mail":false,"groups":["syslog","sshd","recon"],"pci_dss":["11.4"],"gpg13":["4.12"],"gdpr":["IV_35.7.d"]},"agent":{"id":"001","ip":"192.168.0.29"},"manager":{"name":"localhost.localdomain"},"id":"1536071484.679","full_log":"Sep  4 10:31:32 Test-Usama sshd[9617]: Did not receive identification string from 127.0.0.1","predecoder":{"program_name":"sshd","timestamp":"Sep  4 10:31:32","hostname":"nagios-server"},"decoder":{"parent":"sshd","name":"sshd"},"data":{"srcip":"127.0.0.1"},"location":"/var/log/secure"}
{"timestamp":"2018-09-04T15:47:56.902-0400","rule":{"level":3,"description":"Ossec agent started.","id":"503","firedtimes":1,"mail":true,"groups":["ossec"],"pci_dss":["10.6.1","10.2.6"],"gpg13":["10.1"],"gdpr":["IV_35.7.d"]},"agent":{"id":"001","ip":"192.168.0.29"},"manager":{"name":"localhost.localdomain"},"id":"1536090476.5945","full_log":"ossec: Agent started: 'nagios-server->192.168.0.29'.","decoder":{"parent":"ossec","name":"ossec"},"data":{"data":"nagios-server->192.168.0.29"},"predecoder":{"hostname":"nagios-server"},"location":"ossec"}

Agent log output:

2018/09/04 15:48:12 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/audit/audit.log'.

2018/09/04 15:48:12 ossec-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'.
2018/09/04 15:48:12 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/messages'.
2018/09/04 15:48:12 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/secure'.
2018/09/04 15:48:12 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/maillog'.
2018/09/04 15:48:12 ossec-logcollector: INFO: Monitoring output of command(360): df -P
2018/09/04 15:48:12 ossec-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d
2018/09/04 15:48:12 ossec-logcollector: INFO: Monitoring full output of command(360): last -n 20
2018/09/04 15:48:12 ossec-logcollector: INFO: Started (pid: 23083).
2018/09/04 15:48:23 wazuh-modulesd:syscollector: INFO: Evaluation finished.

Regards,

rlinux57

[jesus.g...@wazuh.com]

Ok @rlinux57, 

Your agent is sending events and the manager is generating alerts. Now let's check if Elastic is indexing alerts:

curl -XGET "http://localhost:9200/wazuh-alerts-3.x-2018.09.04/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "term":{ "agent.id": "001" }
  }
}'

Replace localhost by your Elasticsearch host.

Regards,

Jesús

[rlin...@networkconfig.net]

Dear Jesús,

Please find below output:

[root@localhost ~]# curl -XGET "http://localhost:9200/wazuh-alerts-3.x-2018.09.04/_search" -H 'Content-Type: application/json' -d'

{

  "query": {

    "term":{ "agent.id": "001" }

  }

}'

Output:

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"wazuh-alerts-3.x-2018.09.04","index_uuid":"_na_","index":"wazuh-alerts-3.x-2018.09.04"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"wazuh-alerts-3.x-2018.09.04","index_uuid":"_na_","index":"wazuh-alerts-3.x-2018.09.04"},"status":404}

Regards,

rlinux57

[jesus.g...@wazuh.com]

Hello again @rlinux57,

It seems like your Elasticsearch is not creating new indices. Please let's verify some key points from your environment:

// Check if the alerts.json is being used by Logstash
lsof /var/ossec/logs/alerts/alerts.json

// Check if Elasticsearch has some error message
cat /var/log/elasticsearch/elasticsearch.log | grep -E "(ERROR|WARN)"

//Check if Logstash has some error message
cat /var/log/logstash/logstash-plain.log | grep -E "(ERROR|WARN)"

As a side note: if you are using Filebeat, the next command could be useful:

filebeat test output

Regards,

Jesús

[rlin...@networkconfig.net]

Hi again @Jesús,

Please check following output:

[root@localhost ~]# lsof /var/ossec/logs/alerts/alerts.json
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
ossec-ana 1564 ossec   10w   REG    8,1    14476 929780 /var/ossec/logs/alerts/alerts.json

[root@localhost ~]# cat /var/log/elasticsearch/elasticsearch.log | grep -E "(ERROR|WARN)"
[2018-09-05T10:59:11,223][WARN ][o.e.b.JNANatives         ] unable to install syscall filter:
[2018-09-05T11:00:03,082][WARN ][o.e.b.BootstrapChecks    ] [IOvmibY] max number of threads [1024] for user [elasticsearch] is too low, increase to at least [4096]
[2018-09-05T11:00:03,083][WARN ][o.e.b.BootstrapChecks    ] [IOvmibY] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2018-09-05T11:00:12,965][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [IOvmibY] Failed to clear cache for realms [[]]

[root@localhost ~]# cat /var/log/logstash/logstash-plain.log | grep -E "(ERROR|WARN)"
[2018-09-05T11:02:24,028][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:03:37,035][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:04:49,417][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:05:58,168][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:07:05,225][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:08:12,773][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:09:20,552][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I am not using filebeat.

Regards,

rlinux57

[jesus.g...@wazuh.com]

Ok @rlinux57,

Logstash is not working properly then you has no data being sent to Elasticsearch.

// Reassign ossec group
usermod -a -G ossec logstash

// Restart Logstash
systemctl restart logstash

Then check again the logs from Logstash

//Check if Logstash has some error message

cat /var/log/logstash/logstash-plain.log | grep -E "(ERROR|WARN)"

// Even better, use tailf or tail -f
tailf /var/log/logstash/logstash-plain.log | grep -E "(ERROR|WARN)"

Regards,

Jesús

[rlin...@networkconfig.net]

@Jesús

After reassign ossec group and restart logstash:

tailf /var/log/logstash/logstash-plain.log | grep -E "(ERROR|WARN)"

[2018-09-05T11:05:58,168][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:07:05,225][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:08:12,773][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-09-05T11:09:20,552][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

[2018-09-05T13:19:04,812][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Regards,

rlinux57

[jesus.g...@wazuh.com]

Hi @rlinux57,

If the group is fine, the most common cause for that error is permissions or X-Pack,

chown logstash:logstash -R /etc/logstash/
chmod 775 /etc/logstash/conf.d/01-wazuh.conf

Are you using X-Pack security features? If so, you must set credentials into the Logstash configuration file.

Regards,

Jesús

[rlin...@networkconfig.net]

Hi @Jesús

I am not using X-pack, Still getting same error after setting mentioned permissions.

Regards,

rlinux57

[jesus.g...@wazuh.com]

Hello again @rlinux57,

Let's increase permissions for a few other Logstash directories:

chown logstash:logstash -R /usr/share/logstash
chmod 755 -R /usr/share/logstash
chown logstash:logstash -R /var/lib/logstash
chmod 755 -R /var/lib/logstash
systemctl restart logstash

Now check one more time the logs from Logstash.

Regards,

Jesús

[rlin...@networkconfig.net]

Hi @Jesús,

After doing that, getting following warning:

[2018-09-05T17:58:53,240][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>[//localhost:9200], index=>"wazuh-alerts-3.x-%{+YYYY.MM.dd}", document_type=>"wazuh", id=>"47c5d5f74396f7bfe2bace5201e4f43d67ff3fa9072ea8ba13d2e62d2ff6bbea", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_0ab476d8-c589-46f8-81d9-dea0b6f569f5", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2018-09-05T17:58:55,640][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-09-05T17:58:55,930][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}

[2018-09-05T17:58:56,000][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}

[2018-09-05T17:58:56,048][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}

[2018-09-05T17:58:56,077][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}

[2018-09-05T17:58:56,117][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}

[2018-09-05T17:58:57,843][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x60bd4190 run>"}

[2018-09-05T17:58:58,080][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

[2018-09-05T17:58:59,206][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Regards,

rlinux57

[jesus.g...@wazuh.com]

Hello again @rlinux57,

Now your Logstash is just fine. That warning message is a known message regarding to types and Elastic 7.x. For now,

Wazuh works using types and we are planning fully remove them in Elastic 7.x. Now your Logstash logs look good to me.

Let us know if you are still getting trouble to get data into Elasticsearch.

Regards,

Jesús