Field type conflict The type of this field changes across indices.

525 views

Skip to first unread message

satish senapathi

unread,

Jul 8, 2024, 3:59:56 AM7/8/24

to Wazuh | Mailing List

hi,

Wazuh server

1. Add the custom fields in the Wazuh template. Find the data section in the /etc/filebeat/wazuh-template.json file, and add the highlighted custom fields to the data properties section:

{
 "order": 0,
 "index_patterns": [
    "wazuh-alerts-4.x-*",
    "wazuh-archives-4.x-*"
  ],
 "settings": {
   ...
  },
 "mappings": {
    "dynamic_templates": [
      {
   ...
 "data": {
   "properties": {
         "1min_loadAverage": {
                 "type": "double"
               },
         "5mins_loadAverage": { 
                 "type": "double"
               },
         "15mins_loadAverage": {
                 "type": "double"
               },
         "cpu_usage_%": { 
                 "type": "double"
               },
         "memory_usage_%": {
                 "type": "double"
               },
         "memory_available_bytes": { 
                 "type": "double"
               },
         "memory_used_bytes": {
                  "type":  "double"
               },
         "disk_used_bytes": {
                 "type": "double"
               },
         "disk_free_bytes": { 
                 "type": "double"
               },
         "disk_usage_%": {
                 "type": "double"
               },
     "audit": {
       "properties": {
                 "acct": {
                     "type": "keyword"

2. To apply the changes to the Wazuh template, run the command below:

$ sudo filebeat setup -index-management

An expected output is shown below:

Output
ILM policy and write alias loading not enabled. 
Index setup finished.

Note

Modifying existing fields is not permitted in the Wazuh indexer. Once an index is created with some data fields, any changes to the existing fields on the live index are not allowed. Regardless, there is a walkaround; re-indexing.

Re-indexing data

Re-indexing data is the process of moving data from one index to another while possibly transforming the data in the process. This is done when there are changes to the data structure or mappings that require re-organizing or updating the index.

Wazuh dashboard

1. Re-index your data. To use this technique, we will copy the data from the original index to a new index with updated schema definitions. Click on the upper-left menu icon and navigate to Management -> Dev Tools to access the console. We use this console to run the necessary queries to re-index the existing data.

2. Check existing indices:

GET _cat/indices

The output below shows the names of the existing indices and corresponding creation dates:

Output
green open wazuh-alerts-4.x-2024.07.02 TSj3p71XR5qTvwNR3JicwQ 3 0    3 0  53.5kb  53.5kb
green open wazuh-alerts-4.x-2023.04.23 VjfArkNxSOK8-DBht8YnSg 3 0  480 0   1.1mb   1.1mb
green open wazuh-alerts-4.x-2023.04.22 svIJ4pKvRk2fILuMNFCxtQ 1 0    0 0    208b    208b

3. Retrieve data about the index from the Wazuh indexer using a GET request. This confirms that the added custom fields are of the keyword type. In Wazuh, indices are created with the format wazuh-alerts-4.x-YYYY.MM.DD. Here, we will make use of the latest index wazuh-alerts-4.x-2024.07.02. Ensure you replace this index value with that of your latest index:

GET /wazuh-alerts-4.x-2024.07.02

An expected output is shown below:

Output
…
            },
            "id": {
              "type": "keyword"
            },
            "integration": {
              "type": "keyword"
            },
            "memory_available_bytes": {
              "type": "keyword"
            },
            "memory_usage_%": {
              "type": "keyword"
            },
            "memory_used_bytes": {
              "type": "keyword"
            },
            "netinfo": {
…

4. Extract the data from your latest source index to the new destination index named wazuh-alerts-4.x-backup with the re-index API. Replace the source index with your own latest index name value.

POST _reindex
{
  "source": {
    "index": "wazuh-alerts-4.x-2024.07.02"
  },
  "dest": {
    "index": "wazuh-alerts-4.x-backup"
  }
}

The new index with the new schema will be ready to use once the re-indexing is complete. Wazuh indexer returns a 200 – OK status code along with a JSON response that provides details about the operation. These details include the number of documents that were reindexed, the time it took to complete the operation, and any errors or warnings that occurred during the process.

5. Delete the old index. Replace the index name value with your own:

DELETE /wazuh-alerts-4.x-2024.07.02

6. Re-index the data from the backup index wazuh-alerts-4.x-backup to a new index with the original name wazuh-alerts-4.x-2024.07.02. This should be replaced with your own index name. This is done to apply our data type changes:

POST _reindex
{
  "source": {
    "index": "wazuh-alerts-4.x-backup"
  },
  "dest": {
    "index": "wazuh-alerts-4.x-2024.07.02"
  }
}

7. Delete the earlier created backup index:

DELETE /wazuh-alerts-4.x-backup

8. Verify the string data type for the custom fields has changed to the double data type in the updated index. Ensure you replace this index value with that of your own index::

GET /wazuh-alerts-4.x-2024.07.02

An expected output is shown below:

Output
…
            },
            "integration": {
              "type": "keyword"
            },
            "memory_available_bytes": {
              "type": "double"
            },
            "memory_usage_%": {
              "type": "double"
            },
            "memory_used_bytes": {
              "type": "double"
            },
            "netinfo": {
…

after these steps i got the mapping conflicts

Field type conflict

The type of this field changes across indices. It is unavailable for many analysis functions. The indices per type are as follows:

This table contains 2 rows.
Type
Index names

double

wazuh-alerts-4.x-2024.07.02, wazuh-alerts-4.x-2024.07.03

keyword

wazuh-alerts-4.x-2024.06.21, wazuh-alerts-4.x-2024.06.24, wazuh-alerts-4.x-2024.06.25, wazuh-alerts-4.x-2024.06.26, wazuh-alerts-4.x-2024.06.27, wazuh-alerts-4.x-2024.06.28, wazuh-alerts-4.x-2024.07.01

how to resolve this error

Lucas Esteban Pedrosa

unread,

Jul 8, 2024, 10:38:26 AM7/8/24

to Wazuh | Mailing List

Hello, Satish

The procedure you followed will give these fields the correct type for individual indices, but unless you apply this to every index, you may still have some older ones where a value was stored with these field names, but a different type. What you need to do now that you have these fields specified in filebeat is reindex every index beginning with the first one where these fields appeared. If you're in doubt about which would be the first, then you'd need to reindex every one of them since the earliest one you have.

I would also put attention to field names like "memory_usage_%", which contain a special character. If you only got the error you posted, then probably this is OK, but I'd suggest you watch for any errors in case it wouldn't be allowed.

Also, before reindexing, I would also recommend to completely restart filebeat with:

systemctl restart filebeat

The command you have there should be enough for it to take the changes, but completely restarting it will be more solid confirmation that, upon reindexing, you'll get the new fields saved properly.