No Agent Events in Wazuh App for Kibana after upgrading to 3.6

[Robert H]

Hi All,

I just upgraded (most) of my lab setup to 3.6 and ELK 6.4.  I have a couple agents still with 3.5.  But what i notice is all the data and my dashboard in the Wazuh app is only showing alerts/data for the 2 wazuh managers and not any of the agents.  I also included the 3.5 agent and it's not showing up either, so it doesn't look to be related to the agent.  All agents are active and their logs do not show an error.  Alerts from the agents are showing in the /alerts/alert.log file and in Kibana (and the wazuh) in the Discover areas in Kibana have the alerts, but not in the dashboard or in the Wazuh app pages.  An exception is the vulnerability data and alert numbers do to be in the Wazuh app.  Did something change?  I did load the template.

# curl https://raw.githubusercontent.com/wazuh/wazuh/3.6/extensions/elasticsearch/wazuh-elastic6-template-alerts.json -o template.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19737  100 19737    0     0  48489      0 --:--:-- --:--:-- --:--:-- 48613

curl -XPUT 'http://localhost:9200/_template/wazuh' -H 'Content-Type: application/json' -d @template.json

{"acknowledged":true}

For example here is the end of the log from one of the Linux agents:

2018/08/31 12:50:06 ossec-agentd: INFO: Closing connection to server (wazuh2/192.168.1.241:1514/tcp).
2018/08/31 12:50:21 ossec-agentd: WARNING: Unable to reload hostname for 'wazuh2'. Using previous address.
2018/08/31 12:50:21 ossec-agentd: INFO: Trying to connect to server (wazuh2/192.168.1.241:1514/tcp).
2018/08/31 12:50:21 ossec-agentd: INFO: (4102): Connected to the server (wazuh2/192.168.1.241:1514/tcp).
2018/08/31 12:50:21 ossec-agentd: INFO: Server responded. Releasing lock.
2018/08/31 12:50:25 ossec-logcollector: INFO: Agent is now online. Process unlocked, continuing...
2018/08/31 13:31:16 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2018/08/31 13:31:21 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2018/08/31 14:31:16 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2018/08/31 14:31:23 wazuh-modulesd:syscollector: INFO: Evaluation finished.

I have wazuh2 defined in the /etc/hosts file, so not sure why that warning is there.

Any ideas or suggestions?

Regards,

Robert

[Robert H]

Okay, I fixed my dashboard.  I was using agent.name in the visualizations.  I had to change that to predecoder.hostname.  

But my Top 5 Agent and most/all of the similar visuals on the Overview pages all seem off.  Anything where agent names used to show up, now only shows my 2 managers or says no results, but there is alert data in the table views.  Top agents for vulnerabilities, show no agent names in the box, but there is data in the graph, etc.

I think something is mixed up in the app pages.

Regards,

Robert

[lu...@kuhlu.com]

I've noticed this as well. I'm particularly curious if the change from agent.name to precoder.hostname is intentional or a bug? If it's a bug, I don't want to change all my visualizations to use precoder.hostname.

As Robert mentioned, the only hosts to show up under agent.name are my manager names - no agents. 

Thanks,

[Juanjo Jiménez]

Hello Luke and Robert,
Apologies for the inconveniences. This is definitely not an intentional change but a bug. We’ve been able to reproduce it in our development environment, and we’ve created an issue in our GitHub repository to keep track on it.

We’ll fix it as soon as possible and include it on the next release.

In the meantime…

• On the Discover tab, you can still see a full list of alerts clicking on the Discover button.
• On the Agents tab, you can see visualizations tailored for each specific agents, because we use the agent.id field to filter them.

Regards,
Juanjo

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/7ea9d1f9-c106-4e4d-a7dd-88677c2edc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lu...@kuhlu.com]

Hi Juanjo,

Thanks for looking into this. Bugs happen. 

Are you thinking about a small point release to address this, or waiting for release 3.7? A point release would be my preference, but I want to know what I should plan for.

Sincerely,

On Monday, September 3, 2018 at 2:50:55 AM UTC-4, Juanjo Jiménez wrote:

Hello Luke and Robert,
Apologies for the inconveniences. This is definitely not an intentional change but a bug. We’ve been able to reproduce it in our development environment, and we’ve created an issue in our GitHub repository to keep track on it.

We’ll fix it as soon as possible and include it on the next release.

In the meantime…

• On the Discover tab, you can still see a full list of alerts clicking on the Discover button.
• On the Agents tab, you can see visualizations tailored for each specific agents, because we use the agent.id field to filter them.

Regards,
Juanjo

El dom., 2 sept. 2018 a las 14:21, Luke escribió:

I've noticed this as well. I'm particularly curious if the change from agent.name to precoder.hostname is intentional or a bug? If it's a bug, I don't want to change all my visualizations to use precoder.hostname.

As Robert mentioned, the only hosts to show up under agent.name are my manager names - no agents. 

Thanks,

On Friday, August 31, 2018 at 7:59:41 PM UTC-4, Robert H wrote:

Okay, I fixed my dashboard.  I was using agent.name in the visualizations.  I had to change that to predecoder.hostname.  

But my Top 5 Agent and most/all of the similar visuals on the Overview pages all seem off.  Anything where agent names used to show up, now only shows my 2 managers or says no results, but there is alert data in the table views.  Top agents for vulnerabilities, show no agent names in the box, but there is data in the graph, etc.

I think something is mixed up in the app pages.

Regards,

Robert

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[Juanjo Jiménez]

Hello again Luke,

Yes, we’re planning to launch a point release on the next days (no specific ETA at the moment), including several bugfixes such as this one related to the agent.name field.

Thank you so much for your patience and feedback. This helps us a lot to continue improving Wazuh and we really appreciate that. Don’t hesitate to open a new thread on this mailing list, or open a new issue at our repositories everytime you have a problem or questions related to Wazuh.

Regards,
Juanjo

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/7ea9d1f9-c106-4e4d-a7dd-88677c2edc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6ab3bed6-b3dc-4887-b288-c497c2f987bf%40googlegroups.com.

[Pedro Sánchez]

Hi everyone,

I would like to update this email thread with the latest progress we made.

The fix is already in place and prepared to be released: https://github.com/wazuh/wazuh/pull/1213

We are currently working on release 3.6.1, it will be ready by tomorrow.

Best regards,

Pedro.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CABywkcYUDWOqvX485Rb6upU4Gsm%3Dutko7bSVDdvM%2ByL-qaw_LA%40mail.gmail.com.

[Luke Salsich]

Wow, you guys are fast. Thank you!

--

Luke Salsich
Technology Director
Kuhlu LLC
lu...@kuhlu.com
(774) 224-0860

[Robert H]

Hi Juanjo and Pedro, and Luke.

Thanks for the update on the bug.  I thought this looked like a pretty big bug, maybe not technically, but it's effect.  Thanks for the fast resolution.  Could you let us know which components will need to be updated?

Thanks,

Robert

[Pedro Sánchez]

Hi Robert,

Since it will be a patch version you do not need to upgrade either Elastic Stack, Wazuh agents or Wazuh API, upgrading only Wazuh Manager to 6.3.1 will be enough.

Said so, I always recommend upgrading the full Wazuh environment to the latest (agents and API included).

Thanks for the consideration and feedback, best regards,

Pedro 'snaow' Sanchez.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/1d4cfdeb-e7dd-4797-96a5-f60003935731%40googlegroups.com.

[Pedro Sánchez]

Hi Luke, Robert,

We keep working on this, we are 1 day delayed but in good shape so far, I will update with you the progress.

If you take a look at our Github projects (wazuh/wazuh) you can see the progress we made so far with the testing.

Sorry again for the delay.

Regards,

Pedro.

[Luke Salsich]

No worries, Pedro. I know you are working on a lot.

Appreciate the update!

[Pedro Sánchez]

Luke, Robert, as you know, we did release version 3.6.1 end of last week, I hope you can give it a try and send us feedback.

I would like to thank you again for the continuous and very welcome feedback you send us on the daily basis.

Regards,

Pedro.

[Luke Salsich]

Hi Pedro,

I updated to 3.6.1 and it's working as expected. Thanks very much!

[Kat]

Hmm, with a brand new install on Ubuntu 16.4.5, and wazuh-manager 3.6.1-1 and all the requisite files for the install based on Ubuntu - I have now been lucky enough to see the same issue. And although the alerts.json file is quite full with lots of fun data, there are no alerts  in Kibana at all. 

I will go do more troubleshooting, but this is a first and I have been working with this for years, and because it is a virgin install of 3.6.1-1 I would not have expected this.. 

Let the investigation begin...

K

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/7ea9d1f9-c106-4e4d-a7dd-88677c2edc1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6ab3bed6-b3dc-4887-b288-c497c2f987bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CABywkcYUDWOqvX485Rb6upU4Gsm%3Dutko7bSVDdvM%2ByL-qaw_LA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Luke Salsich
Technology Director
Kuhlu LLC
lu...@kuhlu.com
(774) 224-0860

--

Luke Salsich
Technology Director
Kuhlu LLC
lu...@kuhlu.com
(774) 224-0860

[Robert H]

Yes, thanks guys for fixing this.  I changed my dashboards back and all the UI boxes/areas where the agent name were are back to showing good data.

@Kat

The issue was a missing agent.name field.  You should be able to verify if it's there or not by looking at any alerts in the Discovery tab.  Just expand one of the alerts and see if the agent.name field is in the table/list.  Since it's a fresh install, it might be something else.

Regards,

Robert

[Kat]

Nope - was not the problem - since this was a brand new - out of the box install of 3.6.1-1, so there should not have been any issues related to agent.name fields. But now for the freaky part. The install was last Saturday, and it is now Wednesday morning and alerts are showing now for the past 24 hours-only - no further back. WTF??!?!? I didn't touch it!! Ghosts in the machine!! Ahhhhhhhhhhh!!!!!

-Kat

[Pedro Sánchez]

Hi Kat!

Let us know if we can help you with the troubleshooting, I do not really believe in ghosts or poltergeists haha, I know you are an expert in Wazuh, something happened between last Saturday and today!

Maybe an automatic restart over the weekend? Maybe you can list manually the indexed data from the last days, the same thing for the rotated alets.json files.

We are already working on next release 3.7.0, it will be a really huge release, including between others: Analysisd multithread, Syscheck DB from plain text to SQLite (integrated with wazuh-db), agents current configuration on demand (the API will get the configuration the agent has loaded in memory), agents multi-group assignment..

I hope you guys can keep helping us with the feedback and contributions.

Best regards,

Pedro 'snaow' Sanchez de Castro.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6f71cf0c-0aee-4aab-be42-aa823969c0af%40googlegroups.com.

[Kat]

The system had been rebooted several times during the process Saturday and Sunday - but yes, I agree - something did change yesterday. And since Wazuh was running, just not reporting to the dashboard - I just have to go through all the logs of alerts and non-alerts to see if I can figure out what it was. Strangest thing.

[jesus.g...@wazuh.com]

Hi Kat,

Maybe Logstash/Filebeat is having troubles to read the alerts.json file? We can check it using lsof command:

lsof /var/ossec/logs/alerts/alerts.json

This way we can check which processes are reading the alerts.json file.

Regards,

Jesús