JSON custom decoder numbers ingested as strings

[John Jacob]

Hi All,

Been using Wazuh for over a year now and recently tried to sort some of our custom JSON application logs by fields that should be numeric, and we noticed everything is stored as a string. We are using a very simple agent config that works great and pulls in our JSON logs, but we would like certain fields to be stored as a number, and not as a string so we can do things like sort by the number.

  <agent_config>
    <localfile>
      <log_format>json</log_format>
      <location>/data/logs/application.json</location>
    </localfile>
  </agent_config>

What do I need to do so when we decode this as JSON, our alertCount and elapsedMs fields are stored as numbers and not strings so we can perform numerical calculations on these?

[Kasim Mustapha]

Hello John,

Thanks for reaching out.

Apologies for the late response.

To store certain fields as numbers instead of strings, you can use the 'type' option in the mapping. 

By default, the Wazuh indexer analyzes values from existing alerts as string data types.  To change the default data type from string to double, do the following:

Wazuh server
1. Add the custom fields in the Wazuh template. 

Find the data section in the /etc/filebeat/wazuh-template.json file, and add the highlighted custom fields to the data properties section:

{
 "order": 0,
 "index_patterns": [
    "wazuh-alerts-4.x-*",
    "wazuh-archives-4.x-*"
  ],
 "settings": {
   ...
  },
 "mappings": {
    "dynamic_templates": [
      {
   ...
 "data": {
   "properties": {
         "alertCount": {
                 "type": "double"
               },
         "elapsedMs": {
                 "type": "double"
               },
     "audit": {
       "properties": {
                 "acct": {
                     "type": "keyword"
2. To apply the changes to the Wazuh template, run the command below:

sudo filebeat setup -index-management

An expected output is shown below:

Output
ILM policy and write alias loading not enabled.
Index setup finished.

Note that this change would only apply to any newly created index as modifying existing fields is not permitted in the Wazuh indexer. Once an index is created with some data fields, any changes to the existing fields on the live index are not allowed. Regardless, there is a walkaround; re-indexing.

To apply the re-index, kindly follow the indexing step used in this blog as a guide;

https://wazuh.com/blog/monitoring-linux-resource-usage-with-wazuh/#:~:text=Re%2Dindexing%20data,Security%20events. 

I hope this helps. Let me know if you have further questions.

Regards,

Kasim Mustapha

[John Jacob]

Kasim,

I applied the changes you suggested and it appears the field is being inserted as a double, however I still cannot sort by this field because of a "field type conflict".

Below is the error:

The type of this field changes across indices. It is unavailable for many analysis functions. The indices per type are as follows:

We cannot re-index data, and moving forward our indices will be set up properly but I'm wondering if there is a way to instruct the UI to allow us to sort on this field for newer indices moving forward?

[John Jacob]

We found a solution. We had to create the mapping above as a double as specified, allow the mapping to take place, and then store the data with our target fields. If the field is indexed before the mapping is created, it will be a string and not a double.

[satish senapathi]

hi Kasim,

iam facing the same issue, what are the solution steps