New Wazuh installation no results in kibana

[Sardar]

Hello Folks, 

As I'm new to wazuh, please help me with this issue, 

I installed the wazuh + ELK as per the documentation (5.6.2) , ELK is up and running on CentOs 7 (single arch), 
I created filebeat-* index, but kibana shows no results found, 


in /var/ossec/logs/alerts/alerts.log  it shows the alerts, but there are no results in Kibana

even on other index (wazuh-alerts-* / wazuh-monitoring )  : No results

Once it was showing the logs with filebeat-* index, but again now it is shows "No results found", but that format was different, later what happened i'm not getting

Before filebeat-* index, I tried to create logstash-* index : it was showing this error: Unable to fetch mapping. Do you have indices matching the pattern? 

Even no results in Visualizations and Dashboards. 

I attached Filebeat index logs which I was receiving for few time, (but In the format of these logs in kibana there is no username or IP address showing ) 

I followed these steps also: 
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html 

Logstash config is like this : https://www.elastic.co/guide/en/beats/filebeat/5.3/logstash-output.html 

Thanks for your help in advance !

Regards,  

Sardar S. 

[Manuel Albarral]

Hi Sardar,

The manager and ELK are installed in the same server? If they are, you don't need Filebeat and you have to change the Logstash configuration file. That is explained here: https://documentation.wazuh.com/current/installation-guide/installing-elastic-stack/elastic_server_rpm.html#logstash, in the 4th step.

Best regards,

Manuel Albarral

[Sardar]

Thanks for the quick reply Dear Manuel Albarral, 

Yes manager and ELK are on same server, I tried this step also which you mentioned, 

Do you mean I can stop running filebeat ?? and keep only the logstash ? 

Please tell me how should be the configuration for indexing in kibana. 

[Manuel Albarral]

Yes, Filebeat is not necessary in a single host installation.

If you have installed Elasticsearch, Logstash and Kibana following our documentation, the Kibana indices should have been created correctly and you will see two index patterns: wazuh-alerts* and wazuh-monitoring-*. 

No additional configuration is necessary if you need a regular installation.

Best regards,

Manuel

[Sardar]

Yes Kibana indices have been created and index patterns: wazuh-alerts* and wazuh-monitoring-* also there ( but while discovering it show's no results found ) 

[Manuel Albarral]

Have you add Logstash user to OSSEC group as says the documentation? This is the command: usermod -a -G ossec logstash

[Sardar]

Yes I added this. 

[Manuel Albarral]

Great, to confirm that Logstash is reading your alert.json file, use this command: lsof /var/ossec/logs/alerts/alerts.json 

If Logstash is reading this file, it is possible that there are not new alerts. Restarting Wazuh will generate new alerts: systemctl restart wazuh-manager

Let me know if it works.

[Sardar]

I'm getting this output for lsof command :

[Manuel Albarral]

Type again the previous command to add the Logstash user and check again with lsof. You should have the user logstash reading this file. Restart Logstash after add the logstash user to the ossec group: systemctl restart logstash

[Sardar]

I added logstash user and restart logstash, still no results found in kibana

This is the tail of /var/ossec/logs/alerts/alerts.json 

[Manuel Albarral]

And what about the lsof command? Is the logstash user reading the file alerts.json?

To check if there are new alerts, type tail -f /var/ossec/logs/alerts/alerts.json and generate some new alert in another terminal with the command sudo su, for example. If Wazuh is working a new alert should appear in the terminal where you typed the tail command.

If you don't have any alert, check the status of Wazuh manager with /var/ossec/bin/ossec-control status

If the new alert was generated, it means that is an ELK issue, and everything points to Logstash is not reading the alerts.json file and for that reason it is not sending the alerts to Elasticsearch.

I'm waiting for the results. Best regards.

[Sardar]

Dear Manuel, 
Please look at my logstash config (01-wazuh.conf)  : 
are there any changes required ?? 

Tail of logstash-plain.log>> 

I stopped the filebeat already. 

Thanks for your help!

[Manuel Albarral]

There are the errors! Look at the closing curly brackets above the input line, they have to be commented out.

After that, restart logstash and let me know if it works.

[Sardar]

Still no results in kibana, but alerts are coming in /var/ossec/logs/alerts/alerts.json

one question, do I need to create logstash-* index for discovering alerts in kibana ??

when i'm trying to create it in kibana it shows: Unable to fetch mapping. Do you have indices matching the pattern?

Currently on Discover default is " wazuh-alerts-* " index pattern is there 

Please tell me what should I do ?

[Manuel Albarral]

No, you don't need to create logstash-* index. Check again with the lsof command if logstash is reading alerts.json. It is the first step, if it does not read the file, Elasticsearch will not have the alerts and Kibana will not show any results.

Please, type here the result of lsof /var/ossec/logs/alerts/alerts.json

[Sardar]

lsof /var/ossec/logs/alerts/alerts.json

 It shows below result, does it mean it is reading the file now ?
If yes then, again there are no alert's in kibana discover.

[Manuel Albarral]

Yes, now Logstash is sending alerts to Elasticsearch. Lets check if Elasticsearch is storing the alerts. Make this request and type here the result: curl localhost:9200/_cat/indices?v

[Sardar]

This is the output for the above curl : 

Thanks, 

[Manuel Albarral]

There is not an index of today for wazuh-alerts, but there is for wazuh-monitoring. Please, generate an alert by typing sudo su and make again the previous curl. Paste here the result, please.

If this creates the new index, restart wazuh-manager to get more alerts.

[Sardar]

Now it is showing the index of today, 
I again checked the tail of alerts.json new alerts are coming there, but still no results in kibana discover. 

Thanks !

[Manuel Albarral]

Could you paste here a screenshot of Kibana Discover tab?

[Sardar]

Please see this, these both screenshots are for both index pattern's wazuh-alerts-* and wazuh-monitoring-* 

Thanks!

[Manuel Albarral]

What do you get when you search in today index?  curl localhost:9200/wazuh-alerts-2017.10.06/_search?pretty

[Sardar]

I'm getting this reply.... 

Thanks!

[Manuel Albarral]

So, there is not the today index. This means that Elasticsearch is not storing the alerts. Today index should be named as "wazuh-alerts-2017.10.06". Please, paste here the tail of /var/log/logstash/logstash-plain.log.

Thank you.

[Sardar]

This is the output for *plain.log

(Elasticsearch is up and running)

Thanks!

[Manuel Albarral]

In your Logstash configuration file, chang the line hosts => ["localhost:9200"] by your Elasticsearch ip address: hosts => ["xxx.xxx.xxx.xxx:9200"]

Then, restart Logstash.

[Sardar]

Great!
Fantastic, I'm able to see the alerts in Kibana now. 

But the problem is i'm receiving the only alerts from Server / Manager IP only i.e agentid=000 
there are logs present in /var/ossec/logs/alerts/alerts.json from other agents also but these not displaying in kibana, 

and some of the recent alerts of agentid=000 are also not displaying there, 
is there any configuration I need to do for getting displayed all the alerts which are there in alerts.json

I tried refreshing the kibana and Auto-refresh also but new alerts are not coming, it is showing the old alerts only. 

Thanks a lot for your help, 

Best Regards, 
Sardar Shaikh

[Sardar]

Great! 

The logs from the other agents are also cooming now after I did restart of wazuh-manager. 

in the alert i'm getting one warning : No cached mapping for this field, 

Please find attached Screenshot. 

Thanks ! 

[Manuel Albarral]

Great! I'm happy it worked!

Run this command /var/ossec/bin/agent_control -l. You will see your connected agents. If you don't have any connected agent, follow the steps in our documentation https://documentation.wazuh.com/current/user-manual/agents/command-line/register.html#command-line-register

Best regards,

Manuel

[Manuel Albarral]

Go to Management > Index patterns and refresh the wazuh-alerts-* index pattern.

Now, this warning will disappear.

[Sardar]

Thanks for the help Dear Manuel, 

that warning is disappeared, 

But i'm not getting new alerts in kibana. 

Please help me with this, 

I changed the url in logstash config also, as you already 

tail of logstash-plain.log  attached, please have look at this 

[Sardar]

Dear Manuel Albarral, 

Can you please help me with this issue, 

I'm not getting alerts in Kibana, today it again show's no results found. 

but at /var/ossec/logs/alerts/alerts.json there are alerts showing.

Today I checked the indices by following cmd: lsof /var/ossec/logs/alerts/alerts.json  

then logstash user is not accessing the file /var/ossec/logs/alerts/alerts.json

and even not created today's index (cmd: curl localhost:9200/_cat/indices?pretty? ), 

I think When the day is changing it is not creating the index automatically, is there any configuration to do it ?

Please help me, As I'm new to Wazuh, I'm not understanding where the problem is coming, 

Thanks and Regards, 

Sardar S. 

[Manuel Albarral]

Hello Sardar,

The command lsof /var/ossec/logs/alerts/alerts.json checks who is reading a file, not the indices. If Logstash is not reading this file, data will not been sent and you will not see anything in Kibana.

Check the Logstash status with systemctl status logstash

Also, rerun the command usermod -a -G ossec logstash

Best regards,

Manuel Albarral

[Sardar]

Thanks for your reply Dear Manuel Albarral, 

logstash status is up and running, I did restart for logstash then it is reading the file *alerts.json, 

some alerts came for today in Kibana, but it is not showing the new alerts when I do refresh, 

and new alerts are coming in *alerts.json but not in Kibana,

Please help me with this, why it does not sync new alerts ? 

Did I miss any configuration ??

Thanks and Regards
Sardar S. 

[Manuel Albarral]

You are welcome, Sardar. 

The Logstash configuration file (/etc/logstash/conf.d/01-wazuh.conf) should be pointing to your Elasticsearch server.

Anyway, the alerts could take a few seconds to show up in Kibana.

Best regards,

Manuel

[Sardar]

yes logstash configuration file is pointed to my elasticsearch server, 
But still it is not showing recent alerts, 

when I do restart of the logstash, it show's some new alerts, and stops there only. not shows any realtime alerts whenever you do refresh in Kibana. 

what could be the possible error ? where I should check for this ? 

Please check i'm attaching Kibana screenshot, Config of logstash (01-wazuh.conf), /var/ossec/logs/ossec.log and  logstash-plain.log output. 

I tried looking for that ossec error and as per suggestions I stopped ossec, check for that diff file (it doesn't exists on that location) and restarted ossec, but still error comes in ossec.log 

Thanks & Regards, 

Sardar S. 

[Manuel Albarral]

In the Logstash log, you can see the request to localhost instead of the Elasticsearch IP. Please, run and paste here the results of the following commands:

grep -v '^#' /etc/elasticsearch/elasticsearch.yml

curl -XGET localhost:9200

curl -XGET [YOUR_ELASTICSEARCH_IP]:9200

Also, make sure that Logstash is properly restarted:

systemctl restart logstash

Best regards,

Manuel

[Sardar]

Hi Manuel, 

Please see the attached screenshot of above command output, 
and in logstash configuration I added same elasticsearch server url in output > elasticsearch > hosts. 


Thanks & Regards, 
Sardar. S

[Manuel Albarral]

Hi Sardar,

If you have a single-host installation, you don't need to add anything in elasticsearch.yml. I suggest to comment out the host and port lines in elasticsearch.yml and let "localhost:9200" in output > elasticsearch > hosts.

After that, restart Elasticsearch and Logstash. Let me know if it works.

[Manuel Albarral]

You have to comment out also the elasticsearch.url in /etc/kibana/config/kibana.yml. Remember to change also the Logstash configuration.

On Monday, October 9, 2017 at 3:32:03 PM UTC+2, Sardar wrote:

Hi Manuel, 

I commented network.host and port in elasticsearch.yml and hosts to localhost:9200

but the kibana status is Red now 

Please see attached screenshot. 

Thanks and Regards, 
Sardar S. 

[Sardar]

Dear Manuel, 

thanks for your reply I did as you said, but still kibana status is Red, Elasticsearch is trying to access other url rather localhost:9200 , when I curl -XGET 'localhost:9200' 
It shows the output : You know, for search, means elasticsearch is running on localhost fine, 
and commented elasticsearch.url in kibana.yml also and In logstash config also output of ES hosts to localhost:9200, 
But still status ir Red in Kibana 

Please see attached Kibana Screenshot 

Thanks & Regards, 
Sardar S. 

</blockquot

[Manuel Albarral]

Hello Sardar,

Did you restart Kibana after change the kibana.yml file?

systemctl restart kibana

Best regards,

Manuel

Yes I added this. 

<a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html" rel="nofollow" target="_blank" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Fbeats%2Ffilebeat%2Fcurrent%2Ffilebeat-template.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF72ZRjhpbBXU_MO9zWncri92pGfA';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Fbeats%2Ffilebeat%2Fcurrent%2Ffilebeat-template.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF72ZR

[Sardar]

Hi Manuel, 

yes I did restart of all ELK + wazuh-manager also

Thanks & Regards, 
Sardar S. 

[Manuel Albarral]

Hi Sardar,

The configuration could be cached in Kibana. Run the following command and restart Kibana: rm -rf /usr/share/kibana/optimize/bundles

Let me know if it works.

[Sardar]

Hi Manuel, 

I'm getting this error. 

[Manuel Albarral]

Please, paste here your kibana.yml, elasticsearch.yml and 01-wazuh.conf

Best regards,

Manuel

Hi Sardar,

Yes I added this. 

Hi Sardar,

The manager and ELK are installed in the same server? If they are, you don't need Filebeat and you have to change the Logstash configuration file. That is explained here: <a href="https://documentation.wazuh.com/current/installation-guide/installing-elastic-stack/elastic_server_rpm.html#logstash" rel="nofollow" target="_blank" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocumentation.wazuh.com%2Fcurrent%2Finstallation-guide%2Finstalling-elastic-stack%2Felastic_server_rpm.html%23logstash\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHOYp_z6YWibFgL_eNL7c4j9Mhq3Q';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocumentation.wazuh.com%2Fcurrent%2Finstallation-guide%2Finstalling-elastic-st

[Sardar]

Please check attached images, 
I did kibana restart and kibana is running now in browser but still no new alerts are coming, 
whenever I do restart of logstash server few more alerts will display and stops there only, no new alerts from alerts.json it shows. 

Thanks & Regards, 

Sardar S.