Geospatial fields not showing values

115 views
Skip to first unread message

ektadhu...@gmail.com

unread,
Sep 2, 2021, 10:17:39 AMSep 2
to Wazuh mailing list
Hi Team,

In kibana under maps while creating new maps I am trying to select geospatial field but it is showing only one default value i.e. data.aws.service.networkConnectionAction.remoteIpDetails.geoLocation

and not showing GeoLocation.location which was available before upgrade to  7.10(kibana/ elasticsearch, filebeat) and wazuh to 4.1.5

How to fix this?

Regards,
Ekta

mayte...@wazuh.com

unread,
Sep 6, 2021, 5:23:08 AMSep 6
to Wazuh mailing list
Hi Ekta,
 
Sorry for the late response.
 
The GeoLocation.location field is still available in the Wazuh template for Elasticsearch 7.x: wazuh-template.json
 
 You can check it out on Kibana inside Stack Management > Index patterns > wazuh-alerts-*
 
 geolocation.png
If the field does not appear, maybe some steps were missing when performing the migration to 4.1.5: Upgrading Filebeat or maybe there are no wazuh-alerts indices created yet.
 
Keep us updated!
 
Best regards,
Mayte Ariza

ektadhu...@gmail.com

unread,
Sep 7, 2021, 12:54:57 AMSep 7
to Wazuh mailing list
Hi Mayte,

I removed the filebeat and install it again. But after that indices are not creating in elasticsearch.
How to fix this?

Regards,
Ekta

mayte...@wazuh.com

unread,
Sep 7, 2021, 5:00:20 AMSep 7
to Wazuh mailing list
Hi Ekta,

If the /var/ossec/logs/alerts/alerts.json file is being populated but no indices are created in Elasticsearch, check if the communication between Elasticsearch and Filebeat is working properly by running the command filebeat test output

If Filebeat cannot connect to Elasticsearch it may be because either Elasticsearch is down or Filebeat is misconfigured. (You can find the Filebeat configuration in the /etc/filebeat/filebeat.yml file and the Elasticsearch one in the /etc/elasticsearch/elasticsearch.yml file)

Other checks that may help us to find the root cause:
  •  Check if Filebeat status is running. Use service filebeat status (or systemctl)
  • Take a look at the Elasticsearch logs. They should be located in the /var/log/elasticsearch/<your_cluster_name>.log file.
Keep us updated!
 
Best regards,
Mayte Ariza


ektadhu...@gmail.com

unread,
Sep 8, 2021, 12:01:13 AMSep 8
to Wazuh mailing list
Hi Mayte,

I have fixed the filebeat issue. Now index are creating in Elasticsearch and logs are coming in kibana.

But still GeoLocation.location field is not available for 4.x indices.

Kindly assist.

Thanks and Regards,
Ekta

mayte...@wazuh.com

unread,
Sep 8, 2021, 3:38:54 AMSep 8
to Wazuh mailing list
Hi Ekta,

The Filebeat level GeoIP is enabled by default. It is implemented through a Filebeat pipeline and limited to public IP's on the data.srcip, data.win.eventdata.ipAddress, data.aws.sourceIPAddress and data.gcp.jsonPayload.sourceIP fields.

Could you check if any of these fields are indexed in Elasticsearch?

If none of these fields are present, the GeoIP info will not be obtained.

Keep us updated!
 
Best regards,
Mayte Ariza


ektadhu...@gmail.com

unread,
Sep 8, 2021, 3:49:37 AMSep 8
to Wazuh mailing list
Hi Mayte,

I can see  data.srcipdata.win.eventdata.ipAddress fields are available and data is also coming for these field.

Regards,
Ekta

mayte...@wazuh.com

unread,
Sep 8, 2021, 7:46:13 AMSep 8
to Wazuh mailing list
Hi Ekta,

Perform the following checks to help us to debug the issue:
  • In the Wazuh server:
Check if the /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json file matches the one that appears in the Wazuh repository (Filebeat pipeline)

  • In the Elasticsearch server:
- Get your current templates: curl -k -u <user>:<pass> https://localhost:9200/_cat/templates?pretty
Check if the Wazuh template is properly loaded and matches the desired indices (we use the Wazuh template to define the mapping for wazuh-alerts-4.x-* and wazuh-archives-4.x-* indices)
 If not maybe some steps where missing when performing the migration

- Check if the indices applied the Wazuh template properly: curl -k -u <user>:<pass> https://localhost:9200/wazuh-alerts*/_mapping/field/GeoLocation.location?pretty

- Get you current indices: curl -k -u <user>:<pass> -XGET https://localhost:9200/_cat/indices?s=index

- Count the occurrences of the GeoLocation.location field: 

curl -k -u <user>:<pass> -X GET "https://localhost:9200/wazuh-alerts*/_count?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "exists": {
      "field": "GeoLocation.location"
    }
  }
}'


Please, share the output with us. Keep us updated!
 
Best regards,
Mayte Ariza

ektadhu...@gmail.com

unread,
Sep 9, 2021, 7:58:02 AMSep 9
to Wazuh mailing list
Hi Mayte,

I have attached the outputs. Kindly verify.

Regards,
Ekta

elasticsearch_output.png
template.txt
pipeline.txt
cmnd.txt

mayte...@wazuh.com

unread,
Sep 9, 2021, 10:45:05 AMSep 9
to Wazuh mailing list
Hi Ekta,

It seems that the Wazuh indices created between the 2021.08.20 and 2021.08.25 (inclusive) do not have properly applied the mapping. Except for those, all of them seem to have applied it, so currently there should be no problem for the field to be indexed properly.
 
The wazuh template seems to be loaded properly and also the Filebeat pipeline.
 
There are 90517630 Geolocation.location indexed fields, so it seems there is no problem with this field. However, I do not know if they belong to wazuh-alerts-3.x indices or sample ones. They do not appear in any of the outputs, but due to the number of shards shown, perhaps they have been omitted.
 
Could you perform the following query that excludes the wazuh-alerts-3.x indices and the sample ones to verify if the Geolocation.location field is being indexed in the wazuh-alerts-4.x* indices?
 
curl -k -u <user>:<pass> -X GET "https://localhost:9200/wazuh-alerts-4.x*,-wazuh-alerts-4.x-sample*/_count?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "exists": {
      "field": "GeoLocation.location"
    }
  }
}'
 
Also, could you show us a screenshot of Kibana showing where the GeoLocation.location field is missing?
 
Please, share the output with us. Keep us updated!
 
Best regards,
Mayte Ariza


ektadhu...@gmail.com

unread,
Sep 11, 2021, 11:31:31 AMSep 11
to Wazuh mailing list
Hi Mayte,

Please find the screenshots.

Regards,
Ekta

elasticsearch_opt.PNG
kibana_issue.PNG

mayte...@wazuh.com

unread,
Sep 15, 2021, 11:12:27 AMSep 15
to Wazuh mailing list
Hi Ekta,

Sorry for the late response.

There are 31379173 Geolocation.location indexed fields in the wazuh-alerts-4.x* indices, so they are being indexed properly.

Could you check if the Geolocation.location field appears in the wazuh-alerts-* index pattern? You should go to Stack Management > Index patterns > wazuh-alerts-*

If it does not appear, try refreshing the index pattern (It will refresh the field list)

Please keep us updated.

Best regards,
Mayte Ariza

ektadhu...@gmail.com

unread,
Sep 16, 2021, 4:42:47 AMSep 16
to Wazuh mailing list
Hi Mayte,

As per your instruction I tried to refresh the list and one issue I found.

Please check the screenshot.

How we can fix it?

Regards,
Ekta

index_error.png

mayte...@wazuh.com

unread,
Sep 20, 2021, 5:06:46 AMSep 20
to Wazuh mailing list
Hi Ekta!
 
That issue is related to the mapping. It means that some indices (that matches the wazuh-alerts-* index pattern) contain the Geolocation.location field with a type different from geo_point.
 
If you click on the edit button sometimes it provides us with new information about the conflict (which indexes have it and which types are in conflict)
 
You can also use the query I mentioned before to check the type of the Geolocation.location field in each of the indices : curl -k -u <user>:<pass> https://localhost:9200/wazuh-alerts-*/_mapping/field/GeoLocation.location?pretty
 
In order to fix the issue you need to reindex the indices with the wrong mapping to apply the correct one or eliminate those that do not correspond to the required one (when removing the indices you will lose all the data contained on those indices without the chance of recovering them if you do not have a backup)
 
Best regards,
Mayte Ariza
Reply all
Reply to author
Forward
0 new messages