Problem Ingesting JSON Log

[Bill Justesen]

I am unable to get Wazuh 4.3.8 to ingest a JSON log from a Windows client. The Wazuh agent ossec.log file on Windows will show that the file matches the pattern when it is created.

For the fun of it, I created a three-line test log on the Wauzh server and then used the following command on it:

cat testlog.log >> /var/log/syslog

Sure enough, Wazuh ingested it and those three lines were saved to the archives.log and archives.json files. Yet, the logs from Windows still don't appear.

I've been trying to figure this out for a while. Any ideas?

[Damian Alfredo Mangold]

Hello, thanks for using Wazuh!

My name is Damián and I will try to help you with your question, but first, could you give me more information to understand your question correctly?

I understand that you want to see some logs that arrive at the server from a windows agent, but you don't see them. Is this correct ?

[Mauricio Ruben Santillan]

Hello!

Could you share more information about the issue?

Specifically, what type of data are you trying to ingest? a JSON file o Windows events? Could you share a sample of it?

Can you share the configuration you added for this in your Agent's configuration

• C:\Program Files (x86)\ossec-agent\ossec.conf
• C:\Program Files (x86)\ossec-agent\shared\agent.conf if using centralized configuration.

The agent's log file (C:\Program Files (x86)\ossec-agent\ossec.log) should also mention it "reads" the file or if there's an issue with it (You can enable logging to debug mode in the agent's internal_options.conf).

Also, most of the times you'll need some custom rule/s for these events. Have you added any? Could you share it?

In case you wanted to ingest a JSON file with log collector, you would need to add a module similar to next one:

<localfile>
  <log_format>json</log_format>
  <location>C:\Path\to\file.json</location>
</localfile>

Now if you need to ingest Windows events (configuration set by default on the Windows Wazuh agent), you need to have at least next modules:

    <!-- Log analysis -->
    <localfile>
        <location>Application</location>
        <log_format>eventchannel</log_format>
    </localfile>
    <localfile>
        <location>Microsoft-Windows-Windows Defender/Operational</location>
        <log_format>eventchannel</log_format>
    </localfile>
    <localfile>
        <location>Security</location>
        <log_format>eventchannel</log_format>
        <query>Event/System[EventID != 5145 and EventID != 5156 and EventID != 5447 and EventID != 4656 and EventID != 4658 and EventID != 4663 and EventID != 4660 and EventID != 4670 and EventID != 4690 and EventID != 4703 and EventID != 4907 and EventID != 5152 and EventID != 5157]</query>
    </localfile>
    <localfile>
        <location>System</location>
        <log_format>eventchannel</log_format>
    </localfile>

You can add other Windows Events channels by setting the Windows event channel of your choice in location and eventchannel   in log_format. You can also set a query to filter Windows events.

Now, once this is set, you'll need to add some custom rules for these events you're ingesting. Specially if you're ingesting a custom JSON file.

In case of Windows events, although Wazuh includes many rules for them, you might need to add rules for some specific Windows channels. In some case your event might end up triggering a level = 0 rule.

There's related information next:

• https://documentation.wazuh.com/current/user-manual/capabilities/log-data-collection/how-it-works.html
• https://documentation.wazuh.com/current/user-manual/capabilities/log-data-collection/how-to-collect-wlogs.html
• https://documentation.wazuh.com/current/user-manual/capabilities/log-data-collection/log-data-faq.html

I hope this helps! Let us know!

On Tuesday, December 6, 2022 at 6:17:43 PM UTC-3 bjus...@osawatomieks.org wrote:

[Bill Justesen]

To answer questions for both of you: the original logs are Windows DNS events. I have a script that converts them into JSON format. These new files are the logs I'm trying to ingest.

I'm wondering if I need to create a custom rule, as Mauricio indicated, although when I test each line using /var/ossec/bin/wazuh-logtest then it seems to understand the JSON properly. (See below.)

Here are the relevant sections of configuration files and other information:

Windows Server

C:\Program Files (x86)\ossec-agent\ossec.conf (section appended at end of default file)

<ossec_config>
  <localfile>
    <location>C:\Logs\*.json</location>
    <log_format>json</log_format>
  </localfile>
</ossec_config>

Wazuh Server

/var/ossec/etc/ossec.conf (selected snipped of global directive)

  <global>
    <jsonout_output>yes</jsonout_output>
    <alerts_log>yes</alerts_log>
    <logall>yes</logall>
    <logall_json>yes</logall_json>
  <global>

/etc/filebeat/filebeat.yml (selection snippet of file)

filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: true

When a new JSON file is created in Windows, the ossec.log file has an entry that shows:

2022/12/07 06:27:24 wazuh-agent: INFO: (1957): New file that matches the 'C:\Logs\*.json' pattern: 'C:\Logs\Microsoft-Windows-DNSServer-Analytical-20221207_062630.json'.

Even with logcollector.debug set to verbose (and restarting the Wazuh service), there isn't any more information than that in the ossec.log.

Three lines from that corresponding JSON file above are:

{"timestamp":"2022-12-07T12:25:35.812Z","Message":"QUERY_RECEIVED","Source":"172.16.98.138","QTYPE":1,"TCP":0,"QNAME":"docs.google.com","InterfaceIP":"172.16.98.20","Flags":256,"Port":59154,"RD":1,"XID":45112,"Id":256,"Version":0,"Qualifiers":null,"Level":4,"Task":1,"Opcode":0,"Keywords":"-9223372036854775807","RecordId":27,"ProviderName":"Microsoft-Windows-DNSServer","ProviderId":"eb79061a-a566-4698-9119-3ed2807060e7","LogName":null,"ProcessId":1836,"ThreadId":3804,"MachineName":"dc.domain.local","UserId":{"BinaryLength":12,"AccountDomainSid":null,"Value":"S-1-5-18"},"ActivityId":null,"RelatedActivityId":null,"LevelDisplayName":"Information","OpcodeDisplayName":"Info","TaskDisplayName":"LOOK_UP"}
{"timestamp":"2022-12-07T12:25:35.949Z","Message":"RECURSE_RESPONSE_IN","Flags":33152,"AD":0,"QTYPE":1,"TCP":0,"CacheScope":"Default","XID":37867,"ServerScope":".","AA":0,"InterfaceIP":"0.0.0.0","QNAME":"docs.google.com","Source":"172.16.98.1","Port":0,"Id":261,"Version":0,"Qualifiers":null,"Level":4,"Task":2,"Opcode":0,"Keywords":"-9223372036854775776","RecordId":29,"ProviderName":"Microsoft-Windows-DNSServer","ProviderId":"eb79061a-a566-4698-9119-3ed2807060e7","LogName":null,"ProcessId":1836,"ThreadId":3804,"MachineName":"dc.domain.local","UserId":{"BinaryLength":12,"AccountDomainSid":null,"Value":"S-1-5-18"},"ActivityId":null,"RelatedActivityId":null,"LevelDisplayName":"Information","OpcodeDisplayName":"Info","TaskDisplayName":"RECURSE_QUERY"}
{"timestamp":"2022-12-07T12:25:35.949Z","Message":"RESPONSE_SUCCESS","Zone":"Cache","Flags":33152,"AD":0,"QTYPE":1,"Destination":"172.16.98.138","RCODE":"0","XID":45112,"AA":0,"TCP":0,"Scope":"Default","InterfaceIP":"172.16.98.20","QNAME":"docs.google.com","DNSSEC":"0","Port":59154,"Id":257,"Version":0,"Qualifiers":null,"Level":4,"Task":1,"Opcode":0,"Keywords":"-9223372036854775806","RecordId":30,"ProviderName":"Microsoft-Windows-DNSServer","ProviderId":"eb79061a-a566-4698-9119-3ed2807060e7","LogName":null,"ProcessId":1836,"ThreadId":3804,"MachineName":"dc.domain.local","UserId":{"BinaryLength":12,"AccountDomainSid":null,"Value":"S-1-5-18"},"ActivityId":null,"RelatedActivityId":null,"LevelDisplayName":"Information","OpcodeDisplayName":"Info","TaskDisplayName":"LOOK_UP"}

Results of first line run through wazuh-logtest.

**Phase 2: Completed decoding.
        name: 'json'
        ActivityId: 'null'
        Flags: '256'
        Id: '256'
        InterfaceIP: '172.16.98.20'
        Keywords: '-9223372036854775807'
        Level: '4'
        LevelDisplayName: 'Information'
        LogName: 'null'
        MachineName: 'dc.domain.local'
        Message: 'QUERY_RECEIVED'
        Opcode: '0'
        OpcodeDisplayName: 'Info'
        Port: '59154'
        ProcessId: '1836'
        ProviderId: 'eb79061a-a566-4698-9119-3ed2807060e7'
        ProviderName: 'Microsoft-Windows-DNSServer'
        QNAME: 'docs.google.com'
        QTYPE: '1'
        Qualifiers: 'null'
        RD: '1'
        RecordId: '27'
        RelatedActivityId: 'null'
        Source: '172.16.98.138'
        TCP: '0'
        Task: '1'
        TaskDisplayName: 'LOOK_UP'
        ThreadId: '3804'
        UserId.AccountDomainSid: 'null'
        UserId.BinaryLength: '12'
        UserId.Value: 'S-1-5-18'
        Version: '0'
        XID: '45112'
        timestamp: '2022-12-07T12:25:35.812Z'

Would I still need to write a custom rule/decoder for this? It seems to decode it just fine.

[Bill Justesen]

I didn't need to make any decoders, but I did create a rule in the /var/ossec/ruleset/rules/local_rules.xml file that should log the files.

  <rule id="100010" level="0">
    <decoded_as>json</decoded_as>
    <field name="timestamp">\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d.\d\d\dZ</field>
    <field name="ProviderName">Microsoft-Windows-DNSServer</field>
    <description>Windows DNS event logged</description>
  </rule>

Now when testing from wazuh-logtest, I get a completed phase 3. So there's some success!

**Phase 3: Completed filtering (rules).
        id: '100010'
        level: '0'
        description: 'Windows DNS event logged'
        groups: '['local', 'syslog', 'sshd']'
        firedtimes: '2'
        mail: 'False'

Yet for some reason, the logs still don't show up in Wazuh. That seems to be the only thing lacking now...

[Mauricio Ruben Santillan]

Hello Bill,

Since you're creating a custom JSON file with Windows events, then you will need a custom rule for your events.

But I also noticed you already created a rule for them and still it is not matching your events. Have in mind that you set the rule with level=0 (zero) which by default, won't generate any Wazuh alert. Still, your rule is not matching and most probably because of the timestamp criteria you added to it. Also make sure to add your rule inside a <group></group>.

Try using your rule like this:

<group name="windows-dns-custom">

  <rule id="100010" level="5">
    <decoded_as>json</decoded_as>

    <field name="ProviderName">Microsoft-Windows-DNSServer</field>
    <description>Windows DNS event logged</description>
  </rule>

</group>

And I got a correct match with it:

**Phase 3: Completed filtering (rules).
        id: '100010'

        level: '5'

        description: 'Windows DNS event logged'

        groups: '['windows-dns-custom']'
        firedtimes: '1'
        mail: 'False'
**Alert to be generated.

Let me know how it goes.

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/-tMpU6YT0T8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/07f5b72e-c4ee-4900-945e-ec1a52205f4fn%40googlegroups.com.

--

Mauricio Santillan IT Security Engineer - Support DRI The Open Source Security Platform

[Bill Justesen]

Thank you for the help.

I still wasn't getting any logs, however, and decided to look at one other thing. My script was saving the log files as UTF-16. When I converted them to UTF-8, the floodgates opened!

Sheesh! It might be good to add UTF-16 support in the future. Ha ha. What an adventure!

One more mystery solved...