Geospatial fields not showing values

[ektadhu...@gmail.com]

Hi Team,

In kibana under maps while creating new maps I am trying to select geospatial field but it is showing only one default value i.e. data.aws.service.networkConnectionAction.remoteIpDetails.geoLocation

and not showing GeoLocation.location which was available before upgrade to  7.10(kibana/ elasticsearch, filebeat) and wazuh to 4.1.5

How to fix this?

Regards,

Ekta

[mayte...@wazuh.com]

Hi Ekta,

 

Sorry for the late response.

 

The GeoLocation.location field is still available in the Wazuh template for Elasticsearch 7.x: wazuh-template.json

 

 You can check it out on Kibana inside Stack Management > Index patterns > wazuh-alerts-*

 

 

If the field does not appear, maybe some steps were missing when performing the migration to 4.1.5: Upgrading Filebeat or maybe there are no wazuh-alerts indices created yet.

 

Keep us updated!

 

Best regards,

Mayte Ariza

[ektadhu...@gmail.com]

Hi Mayte,

I removed the filebeat and install it again. But after that indices are not creating in elasticsearch.

How to fix this?

Regards,

Ekta

[mayte...@wazuh.com]

Hi Ekta,

If the /var/ossec/logs/alerts/alerts.json file is being populated but no indices are created in Elasticsearch, check if the communication between Elasticsearch and Filebeat is working properly by running the command filebeat test output

If Filebeat cannot connect to Elasticsearch it may be because either Elasticsearch is down or Filebeat is misconfigured. (You can find the Filebeat configuration in the /etc/filebeat/filebeat.yml file and the Elasticsearch one in the /etc/elasticsearch/elasticsearch.yml file)

Other checks that may help us to find the root cause:

•  Check if Filebeat status is running. Use service filebeat status (or systemctl)
• Take a look at the Elasticsearch logs. They should be located in the /var/log/elasticsearch/<your_cluster_name>.log file.

Keep us updated!

 

Best regards,

Mayte Ariza

[ektadhu...@gmail.com]

Hi Mayte,

I have fixed the filebeat issue. Now index are creating in Elasticsearch and logs are coming in kibana.

But still GeoLocation.location field is not available for 4.x indices.

Kindly assist.

Thanks and Regards,

Ekta

[mayte...@wazuh.com]

Hi Ekta,

The Filebeat level GeoIP is enabled by default. It is implemented through a Filebeat pipeline and limited to public IP's on the data.srcip, data.win.eventdata.ipAddress, data.aws.sourceIPAddress and data.gcp.jsonPayload.sourceIP fields.

Could you check if any of these fields are indexed in Elasticsearch?

If none of these fields are present, the GeoIP info will not be obtained.

Keep us updated!

 

Best regards,

Mayte Ariza

[ektadhu...@gmail.com]

Hi Mayte,

I can see  data.srcip, data.win.eventdata.ipAddress fields are available and data is also coming for these field.

Regards,

Ekta

[mayte...@wazuh.com]

Hi Ekta,

Perform the following checks to help us to debug the issue:

• In the Wazuh server:

Check if the /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json file matches the one that appears in the Wazuh repository (Filebeat pipeline)

• In the Elasticsearch server:

- Get your current templates: curl -k -u <user>:<pass> https://localhost:9200/_cat/templates?pretty

Check if the Wazuh template is properly loaded and matches the desired indices (we use the Wazuh template to define the mapping for wazuh-alerts-4.x-* and wazuh-archives-4.x-* indices)

 If not maybe some steps where missing when performing the migration

- Check if the indices applied the Wazuh template properly: curl -k -u <user>:<pass> https://localhost:9200/wazuh-alerts*/_mapping/field/GeoLocation.location?pretty

- Get you current indices: curl -k -u <user>:<pass> -XGET https://localhost:9200/_cat/indices?s=index

- Count the occurrences of the GeoLocation.location field: 

curl -k -u <user>:<pass> -X GET "https://localhost:9200/wazuh-alerts*/_count?pretty" -H 'Content-Type: application/json' -d'

{

  "query": {

    "exists": {

      "field": "GeoLocation.location"

    }

  }

}'

Please, share the output with us. Keep us updated!

 

Best regards,

Mayte Ariza

[ektadhu...@gmail.com]

Hi Mayte,

I have attached the outputs. Kindly verify.

Regards,

Ekta

[mayte...@wazuh.com]

Hi Ekta,

It seems that the Wazuh indices created between the 2021.08.20 and 2021.08.25 (inclusive) do not have properly applied the mapping. Except for those, all of them seem to have applied it, so currently there should be no problem for the field to be indexed properly.

 

The wazuh template seems to be loaded properly and also the Filebeat pipeline.

 

There are 90517630 Geolocation.location indexed fields, so it seems there is no problem with this field. However, I do not know if they belong to wazuh-alerts-3.x indices or sample ones. They do not appear in any of the outputs, but due to the number of shards shown, perhaps they have been omitted.

 

Could you perform the following query that excludes the wazuh-alerts-3.x indices and the sample ones to verify if the Geolocation.location field is being indexed in the wazuh-alerts-4.x* indices?

 

curl -k -u <user>:<pass> -X GET "https://localhost:9200/wazuh-alerts-4.x*,-wazuh-alerts-4.x-sample*/_count?pretty" -H 'Content-Type: application/json' -d'

{

  "query": {

    "exists": {

      "field": "GeoLocation.location"

    }

  }

}'

 

Also, could you show us a screenshot of Kibana showing where the GeoLocation.location field is missing?

 

Please, share the output with us. Keep us updated!

 

Best regards,

Mayte Ariza

[ektadhu...@gmail.com]

Hi Mayte,

Please find the screenshots.

Regards,

Ekta

[mayte...@wazuh.com]

Hi Ekta,

Sorry for the late response.

There are 31379173 Geolocation.location indexed fields in the wazuh-alerts-4.x* indices, so they are being indexed properly.

Could you check if the Geolocation.location field appears in the wazuh-alerts-* index pattern? You should go to Stack Management > Index patterns > wazuh-alerts-*

If it does not appear, try refreshing the index pattern (It will refresh the field list)

Please keep us updated.

Best regards,

Mayte Ariza

[ektadhu...@gmail.com]

Hi Mayte,

As per your instruction I tried to refresh the list and one issue I found.

Please check the screenshot.

How we can fix it?

Regards,

Ekta

[mayte...@wazuh.com]

Hi Ekta!

 

That issue is related to the mapping. It means that some indices (that matches the wazuh-alerts-* index pattern) contain the Geolocation.location field with a type different from geo_point.

 

If you click on the edit button sometimes it provides us with new information about the conflict (which indexes have it and which types are in conflict)

 

You can also use the query I mentioned before to check the type of the Geolocation.location field in each of the indices : curl -k -u <user>:<pass> https://localhost:9200/wazuh-alerts-*/_mapping/field/GeoLocation.location?pretty

 

In order to fix the issue you need to reindex the indices with the wrong mapping to apply the correct one or eliminate those that do not correspond to the required one (when removing the indices you will lose all the data contained on those indices without the chance of recovering them if you do not have a backup)

 

Best regards,

Mayte Ariza