Custom Decoder/log not showing

[Fawwas Hamdi]

can someone help me with this issue as im trying to read a new log with a format like this 

{"timestamp": "Wed Mar 19", "status": "11:09:54", "session": "2025", "host_info": "Connected TO S1002/2913 host 1.1.1.1/3230 (1.1.1.1)"}

and already managed to create the decoder look like this 

<group name="saprouter">
    <rule id="100211" level="5">
            <decoded_as>json</decoded_as>
        <description>SAP Router log detected.</description>
    </rule>

    <rule id="100212" level="12">
        <decoded_as>json</decoded_as>
        <match>DENIED</match>
        <description>SAP Router connection denied.</description>
    </rule>
</group>

and after testing the log it is already processed

**Messages: WARNING: (7003): '64988eab' token expires INFO: (7202): Session initialized with token '1b72ef89' **Phase 1: Completed pre-decoding. full event: '{"timestamp": "Wed Mar 19", "status": "11:09:54", "session": "2025", "host_info": "Connected TO S1002/2913 host 1.1.1/3230 (1.1.1.1)"}' **Phase 2: Completed decoding. name: 'json' host_info: 'Connected TO S1002/2913 host 1.1.1.1/3230 (1.1.1.1)' session: '2025' status: '11:09:54' timestamp: 'Wed Mar 19' **Phase 3: Completed filtering (rules). id: '100211' level: '5' description: 'SAP Router log detected.' groups: '["saprouter"]' firedtimes: '1' mail: 'false' **Alert to be generated. but within the dashboard i cant seem to find the log and the agent file config looks like this since i cant seem to read the file i improvised
 <localfile>
   <log_format>json</log_format>
   <location>/usr/sap/saprouter/routerlog.json</location>
  </localfile>

   <localfile>
   <log_format>syslog</log_format>
   <location>/usr/sap/saprouter/routerlog.json</location>
  </localfile>

  <localfile>
   <log_format>syslog</log_format>
   <location>/usr/sap/saprouter/*</location>
  </localfile>

  <localfile>
   <log_format>json</log_format>
   <location>/usr/sap/saprouter/*.json</location>
  </localfile> the file that i wanted to monitor is under /usr/sap/saprouter/routerlog.json

[Fawwas Hamdi]

and for the file it is populated with the new log no problem whatsoever

[hasitha.u...@wazuh.com]

Hi  Fawwas,

I have found the issue, which is related to filebeat mapping.

2025-03-19T17:10:01.983+0530 WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc1eec92439a5d56c, ext:30048280841, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"2103c6d4-0d23-4893-934e-fa5d7b48fdef","hostname":"wazuh411.local","id":"9d7515ef-86eb-4b77-8fe2-423ebd4624fe","name":"wazuh411.local","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"wazuh411.local"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":2379543},"message":"{\"timestamp\":\"2025-03-19T17:09:58.763+0530\",\"rule\":{\"level\":5,\"description\":\"SAP Router log detected.\",\"id\":\"100211\",\"firedtimes\":2,\"mail\":false,\"groups\":[\"saprouter\"]},\"agent\":{\"id\":\"000\",\"name\":\"wazuh411.local\"},\"manager\":{\"name\":\"wazuh411.local\"},\"id\":\"1742384398.2418176\",\"full_log\":\"{\\\"timestamp\\\":\\\"Wed Mar 19\\\",\\\"status\\\":\\\"11:09:54\\\",\\\"session\\\":\\\"2025\\\",\\\"host_info\\\":\\\"Connected TO S1002/2913 host 1.1.1.1/3230 (1.1.1.1)\\\"}\",\"decoder\":{\"name\":\"json\"},\"data\":{\"status\":\"11:09:54\",\"timestamp\":\"Wed Mar 19\",\"session\":\"2025\",\"host_info\":\"Connected TO S1002/2913 host 1.1.1.1/3230 (1.1.1.1)\"},\"location\":\"/var/log/test.json\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::286685-2080", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00004e410), Source:"/var/ossec/logs/alerts/alerts.json", Offset:2380174, Timestamp:time.Time{wall:0xc1eec924399d8af7, ext:30047737519, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x45fdd, Device:0x820}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.timestamp] of type [date] in document with id 'Zyo0rpUBUzC5w5L-Dr49'. Preview of field's value: 'Wed Mar 19'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [Wed Mar 19] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}}

The error occurs because Wazuh indexer couldn’t process (or "index") an event due to a problem with parsing a field called data.timestamp. The value of this field is "Wed Mar 19", but Wazuh indexer can’t interpret it as a valid date. This results in a mapper_parsing_exception with a status code of 400 (Bad Request).

Please allow some time to test this issue on my end.

Regards,
Hasitha Upekshitha

[hasitha.u...@wazuh.com]

Hi  Fawwas,

By default JSON log not required to create decoder to extract details, however your logs timestamp is not in right format to indexed properly, Therefore we can add extra parameter for log, then while predecoding part we can ignore taking timestamp.

We can use  <out_format> tag in localfile to add extra parameter.
For example:
<out_format>saprouter: $(log)</out_format>
Once you configured log will receive to a manager like this.

saprouter: {"timestamp": "Wed Mar 19", "status": "11:09:54", "session": "2025", "host_info": "Connected TO S1002/2913 host 1.1.1.1/3230 (1.1.1.1)"}

1.   <localfile>
2.     <log_format>json</log_format>
3.     <location>/var/log/test.json</location>
4.     <out_format>saprouter: $(log)</out_format>
5.   </localfile>

Then you need to create custom decoders to extract these fields.
nano  /var/ossec/etc/decoders/ saprouter_custom_decoders.xml

1. <decoder name="saprouter">
2.          <prematch>saprouter</prematch>
3. </decoder>
4.  
5. <decoder name="saprouter-custom">
6.         <parent>saprouter</parent>
7.         <regex>\.+"timestamp":"(\.+)"</regex>
8.         <order>LogTime</order>
9. </decoder>
10.  
11. <decoder name="saprouter-custom">
12.         <parent>saprouter</parent>
13.         <regex>\.+"status":"(\.+)"</regex>
14.         <order>Status</order>
15. </decoder>
16.  
17. <decoder name="saprouter-custom">
18.         <parent>saprouter</parent>
19.         <regex>\.+"session":"(\.+)"</regex>
20.         <order>Session</order>
21. </decoder>
22.  
23. <decoder name="saprouter-custom">
24.         <parent>saprouter</parent>
25.         <regex>\.+"host_info":"(\.+)"</regex>
26.         <order>host_info</order>
27. </decoder>

Save and close the file, then set the file permissions.   chmod 660 /var/ossec/etc/decoders/ saprouter_custom_decoders.xml

chown wazuh:wazuh /var/ossec/etc/decoders/ saprouter_custom_rules.xml
Then you need to add these rules to the custom rule creation file. nano /var/ossec/etc/rules/saprouter_custom_rules.xml

1. <group name="saprouter">
2.     <rule id="100211" level="5">

1.      <decoded_as>saprouter</decoded_as>

1.      <description>SAP Router log detected.</description>
2.     </rule>
3.  
4.     <rule id="100212" level="12">

1.       <if_sid>100211</if_sid>

1.       <match>DENIED</match>
2.       <description>SAP Router connection denied.</description>
3.     </rule>
4. </group>

chmod 660 /var/ossec/etc/rules/saprouter_custom_rules.xml

chown wazuh:wazuh /var/ossec/etc/rules/saprouter_custom_rules.xml After adding make sure to restart Wazuh manager to apply changes. systemctl restart wazuh-manager

Further, you can learn more about how to create custom decoders and rules by following the documents.

https://documentation.wazuh.com/current/user-manual/ruleset/decoders/custom.html

https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/decoders.html

https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/rules.html

https://documentation.wazuh.com/current/user-manual/ruleset/rules/custom.html#custom-rules

https://wazuh.com/blog/creating-decoders-and-rules-from-scratch/

Let me know if you need further assistance on this. Regards, Hasitha Upekshitha