Event is not in GUI, but is in archives.log and lotgest works

[Miroslav M]

I modified decoder 0375 for Zimbra mail server. Since then such events are not in Wazuh GUI in wazuh-archives.

Event:
<code>196.1.1.2:44338 - - [05/Feb/2023:14:27:03 +0100]  "GET https://mail.example.cz/ HTTP/2.0" 200 4972 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0" "172.17.0.17:8443" "172.17.0.17:443"</code>

It is in /var/ossec/logs/archives/archives.log
<code>2023 Feb 05 14:27:04 (mail.example.cz) any->/opt/zimbra/log/nginx.access.log 196.1.1.2:44338 - - [05/Feb/2023:14:27:03 +0100]  "GET https://mail.example.cz/ HTTP/2.0" 200 4972 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0" "172.17.0.17:8443" "172.17.0.17:443"</code>

wazuh-logtest works:

<code>
**Phase 2: Completed decoding.
    name: 'zimbra-web-accesslog'
    parent: 'zimbra-web-accesslog'
    browser: 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0'
    http_version: '2.0'
    operation: 'GET'
    rcode: '200'
    route: 'https://mail.example.cz/'
    rsize: '4972'
    srcip: '196.1.1.2'
    srcport: '44338'
    timestamp: '05/Feb/2023:14:27:03'
    url: '-'
</code>

part of local_decoder.xml
<code>
<!-- Decoder for Zimbra NGINX access modified 0375-... -->
<decoder name="zimbra-web-accesslog">
    <type>web-log</type>
    <prematch>^\S+ \S+ \S+ \.*[\S+ \S\d+]  "\w+ \S+ HTTP\S+" </prematch>
</decoder>

<decoder name="zimbra-web-accesslog-glpi">
    <type>web-log</type>
    <parent>zimbra-web-accesslog</parent>
    <prematch>^\S+ - - [\d+/\w+/\d+:\d+:\d+:\d+ +\d+]  "\S+ \S+ HTTP/\.+"</prematch>
    <regex>^(\S+):(\d+) - - [(\d+/\w+/\d+:\d+:\d+:\d+) +\d+]  "(\S+) (\S+) HTTP/(\.+)" (\d+) (\S+) "(\.+)" "(\.+)"</regex>
    <order>srcip,srcport, timestamp,operation, route, http_version, rcode, rsize, url, browser</order>
</decoder>

</code>

What is wrong? Thanks

[Miroslav M]

Double spaces are lost, so also here: https://pastebin.com/P8tXyH3r

[Julian Bustamante Narvaez]

Hi, I will be solving your request, as soon as I replicate the problem and have the solution, I will let you know.

did you set any rules when matching this decoder?

Regards

[Julian Bustamante Narvaez]

I used your local decoder and try with logtest but it doesn't generate any alert after phase 3, you must configure a rule so that the alert is displayed in alert.json and therefore in the dashboard.

https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/rules.html

Let me know if this helps you, if not please ask me again.

Regards

[Miroslav M]

No specific rules for these events.

**Phase 3: Completed filtering (rules).
    id: '31100'
    level: '0'
    description: 'Access log messages grouped.'
    groups: '['web', 'accesslog']'
    firedtimes: '1'
    mail: 'False'

Thank you

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/n80h8pDmoxU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/fad26c69-3262-4ecb-baeb-f9e6c11fdcc0n%40googlegroups.com.

[Julian Bustamante Narvaez]

So you can create a rule in /var/ossec/etc/rules/local_rules.xml file.

https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax

for example:

  <rule id="110001" level="12">
    <if_sid>31100</if_sid>
      <description>You description</description>
  </rule>

 

**Phase 3: Completed filtering (rules).

    id: '110001'
    level: '12'
    description: 'You description'
    groups: '['local', 'syslog', 'sshd']'
    firedtimes: '1'
    mail: 'True'
**Alert to be generated.

Regards

[Miroslav M]

It is misunderstanding. I do not need any alert. I need the event to be visible in the GUI, which is not. Opensearch does not list it after I created the decoder. But it is the /var/ossec/logs/archives/archives.log

[Julian Bustamante Narvaez]

To view events not alerted in Wazuh. These steps are to be carried out on the Wazuh manager

1. Set <logall_json>yes</logall_json> in /var/ossec/etc/ossec.conf
2. Set archives: enabled to true in /etc/filebeat/filebeat.yml

archives: enabled: true

3. Restart filebeat

systemctl restart filebeat

4. Restart wazuh manager

systemctl restart wazuh-manager

The following steps are to be carried out on the Wazuh dashboard
5. On the  Wazuh Dashboard, under Opensearch Plugins, -> index management -> indices, verify wazuh-archives-x.x-xxxx.xx.xx is present

6. On the Wazuh Dashboard -> stack management -> index pattern and select Create index pattern. Use wazuh-archives-* as index pattern name

7. On the Wazuh Dashboard, under OpenSearch Dashboards -> discover you'll find these events there

Note:

Please note that enabling the log_all setting will increase your storage usage as all events that arrives on the Wazuh manager will get saved.

I hope this helps

[Miroslav M]

Thanks, but I have it all for a few months on this installation and it works file. I receive logs and I work with them regularly in the Discover / wazuch-archives-* pattern.

The issue appeared after enabling mentioned decoder. Since then events like the one mentioned in the first post are not displayed in Discover. 

If I disable decoder again, it is not parsed (decoded), but is on the Discover screen:

Dne 06. 02. 23 v 20:54 'Julian Bustamante Narvaez' via Wazuh mailing list napsal(a):

[Julian Bustamante Narvaez]

I understand what you want.
I already replicated the problem and the same thing happens to me, let me analyze why this is happening and as soon as I have the solution I will let you know

[Julian Bustamante Narvaez]

Hi, The problem is in the timestamp variable that you use in your decoder, since timestamp is reserved as its own.

a workaround is to change timestamp to logtimestamp.

You can see the warning  -> cat /var/log/filebeat/filebeat | grep WARN

WARN    [elasticsearch]    elasticsearch/client.go:408    Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc0f0c70fe6b206ac, ext:71418538034535, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-archives-pipeline"}, Fields:{"agent":{"ephemeral_id":"779a7be1-2926-41ae-bbc8-56f4dadc80b6","hostname":"julian-A15-FA506QM","id":"859f3452-3db4-4a9e-a0bb-9e861914a48b","name":"julian-A15-FA506QM","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.archives","module":"wazuh"},"fields":{"index_prefix":"wazuh-archives-4.x-"},"fileset":{"name":"archives"},"host":{"name":"julian-A15-FA506QM"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/archives/archives.json"},"offset":4195761},"message":"{\"timestamp\":\"2023-02-07T14:21:30.054-0500\",\"agent\":{\"id\":\"000\",\"name\":\"julian-A15-FA506QM\"},\"manager\":{\"name\":\"julian-A15-FA506QM\"},\"id\":\"1675797690.227084\",\"full_log\":\"196.1.1.2:44338 - - [05/Feb/2023:14:27:03 +0100]  \\\"GET https://mail.example.cz/ HTTP/2.0\\\" 200 4972 \\\"-\\\" \\\"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0\\\" \\\"172.17.0.17:8443\\\" \\\"172.17.0.17:443\\\"\",\"decoder\":{\"parent\":\"zimbra-web-accesslog\",\"name\":\"zimbra-web-accesslog\"},\"data\":{\"srcip\":\"196.1.1.2\",\"srcport\":\"44338\",\"url\":\"-\",\"timestamp\":\"05/Feb/2023:14:27:03\",\"operation\":\"GET\",\"route\":\"https://mail.example.cz/\",\"http_version\":\"2.0\",\"rcode\":\"200\",\"rsize\":\"4972\",\"browser\":\"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0\"},\"location\":\"/home/thejbte/Documents/test.txt\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::3326419-66310", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc0003a15f0), Source:"/var/ossec/logs/archives/archives.json", Offset:4196554, Timestamp:time.Time{wall:0xc0f09495f9c433c2, ext:19730857992849, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x32c1d3, Device:0x10306}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [data.timestamp] of type [date] in document with id 'qSBTLYYBv72PO4WjkBuK'. Preview of field's value: '05/Feb/2023:14:27:03'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [05/Feb/2023:14:27:03] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}}

Decoder:

<!-- Decoder for Zimbra NGINX access modified 0375-... -->
<decoder name="zimbra-web-accesslog">
    <type>web-log</type>
    <prematch>^\S+ \S+ \S+ \.*[\S+ \S\d+]  "\w+ \S+ HTTP\S+" </prematch>
</decoder>

<decoder name="zimbra-web-accesslog-glpi">
    <type>web-log</type>
    <parent>zimbra-web-accesslog</parent>
    <prematch>^\S+ - - [\d+/\w+/\d+:\d+:\d+:\d+ +\d+]  "\S+ \S+ HTTP/\.+"</prematch>
    <regex>^(\S+):(\d+) - - [(\d+/\w+/\d+:\d+:\d+:\d+) +\d+]  "(\S+) (\S+) HTTP/(\.+)" (\d+) (\S+) "(\.+)" "(\.+)"</regex>

    <order>srcip,srcport, logtimestamp,operation, route, http_version, rcode, rsize, url, browser</order>
</decoder>

Dashboard:

Feb 7, 2023 @ 14:37:16.642

agent.name:julian-A15-FA506QM agent.id:000 manager.name:julian-A15-FA506QM data.srcip:196.1.1.2 data.logtimestamp:05/Feb/2023:14:27:03 data.route:https://mail.example.cz/ data.browser:Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0 data.srcport:44338 data.http_version:2.0 data.rcode:200 data.operation:GET data.url:- data.rsize:4972 decoder.parent:zimbra-web-accesslog decoder.name:zimbra-web-accesslog full_log:196.1.1.2:44338 - - [05/Feb/2023:14:27:03 +0100] "GET https://mail.example.cz/ HTTP/2.0" 200 4972 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0" "172.17.0.17:8443" "172.17.0.17:443" input.type:log @timestamp:Feb 7, 2023 @ 14:37:16.642 location:/home/thejbte/Documents/test.txt id:1675798636.252657 GeoLocation.country_name:India GeoLocation.location:{ "lon": 77, "lat": 20 } timestamp:Feb 7, 2023 @ 14:37:16.642 _index:wazuh-archives-4.x-2023.02.07

attached screenshots

Regards

[Miroslav M]

Many thanks, I am happy to confirm it works.

Dne 07. 02. 23 v 21:06 'Julian Bustamante Narvaez' via Wazuh mailing list napsal(a):

--

You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/n80h8pDmoxU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f578d823-b9ce-45cd-ba57-11e7b99fb86fn%40googlegroups.com.