Parsing /var/ossec/logs/archive.json with filebeat

[Sanjay Rajak]

hello all,

i am using wazuh 3.9.1 with elasticsearch 7.1 and filebeat 7.1 on centos7. i have enabled log all in ossec.conf "<logall_json>yes</logall_json>" for recording all logs that are received. I would like to parse all the decoded logs in to the existing wazuh  index. Looking for pointers.

Thanks

[Jose Miguel Garcia Ramos]

Hi Sanjay,

To parse all the decoded logs, the following steps are needed

Set <logall_json>yes</logall_json> under the <global> tag in the /var/ossec/etc/ossec.conf file

Restart the Wazuh-manager:

/var/ossec/bin/ossec-control restart

All the log data should be now in /var/ossec/logs/archives/archives.json which you can check with:

tail -f /var/ossec/logs/archives/archives.json

Now it’s necessary to indicate Filebeat that he must monitor our archives.json file.

So add the previous path under the ‘paths’ parameter in /etc/filebeat/filebeat.yml like the following:

After that, restart the service as usual with:

systemctl restart filebeat or service filebeat restart

Then, Kibana should be receiving our full data log and parsing it :

As you can see in the image, Kibana is receiving the full data logs.

It’s important to note that these alerts don’t have the parametersrule.* in the alert because they didn’t trigger a rule.

Let me paste you a screenshot of a classic rule so you can check the difference easily.

Please let me know if you have any questions or concerns.

Best Regards,

Jose

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/96817a84-a72d-4f79-a8e8-31a9ba5a9034%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sanjay Rajak]

Will do, thanks Jose.

[Jose Miguel Garcia Ramos]

Glad to help,

Feel free to ask us anytime!

Best Regards,

Jose

[Sanjay Rajak]

Thanks. 

[Alberto Rodriguez]

Hello Sanjay, Jose Miguel

  The solution proposed here will send all data (alerts or archives) to the same index. There is an alternative: sending alerts to index wazuh-alerts-3.x* and sending archives to wazuh-archives-3.x*. For that, you will need to modify the file /etc/filebeat/wazuh-template.json replacing the line

"index_patterns": ["wazuh-alerts-3.x-*"]

by

"index_patterns": ["wazuh-alerts-3.x-*", "wazuh-archives-3.x-*"]

Then, you'll need to modify the filebeat.yml file configuration as follow:

# Wazuh - Filebeat configuration file

filebeat.inputs:
  - type: log
    paths:
      - '/var/ossec/logs/alerts/alerts.json'
    fields:
      origin: "wazuh-alerts-3.x"
  - type: log
    paths:
      - '/var/ossec/logs/archives/archives.json'
    fields:
      origin: "wazuh-archives-3.x"



setup.template.json.enabled: true
setup.template.json.path: "/etc/filebeat/wazuh-template.json"
setup.template.json.name: "wazuh"
setup.template.overwrite: true

processors:
  - decode_json_fields:
      fields: ['message']
      process_array: true
      max_depth: 200
      target: ''
      overwrite_keys: true
  - drop_fields:
      fields: ['message', 'ecs', 'beat', 'input_type', 'tags', 'count', '@version', 'log', 'offset', 'type', 'host']
  - rename:
      fields:
        - from: "data.aws.sourceIPAddress"
          to: "@src_ip"
      ignore_missing: true
      fail_on_error: false
      when:
        regexp:
          data.aws.sourceIPAddress: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b
  - rename:
      fields:
        - from: "data.srcip"
          to: "@src_ip"
      ignore_missing: true
      fail_on_error: false
      when:
        regexp:
          data.srcip: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b
  - rename:
      fields:
        - from: "data.win.eventdata.ipAddress"
          to: "@src_ip"
      ignore_missing: true
      fail_on_error: false
      when:
        regexp:
          data.win.eventdata.ipAddress: \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b


output.elasticsearch:
  hosts: ['http://localhost:9200']
  pipeline: geoip
  indices:
    - index: '%{[fields.origin]}-%{+yyyy.MM.dd}'

Then, restart filebeat: systemctl restart filebeat

Additionally, you have to define an index pattern in Kibana: Management -> Index Patterns -> Create Index Pattern -> Type "wazuh-archives-3.x*" in Index pattern box and Nex step -> Select @timestamp in "Time Filter field name" and click on Create index pattern. Now you will be able to select the index in discover and see all events.

Best regards,

Alberto R.

PS: I removed my previous message in order to clarfy all said on those messages in only one, sorry for the inconveniences. 

Thanks. 

Will do, thanks Jose.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[Sanjay Rajak]

Dear Alberto,

I am using wazuh 3.9.3 vm_image and have enabled log all in ossec.conf "<logall_json>yes</logall_json>" for recording all logs that are received. Filebeat.yml input looks like this

filebeat.inputs:

  - type: log

    paths:

      - '/var/ossec/logs/alerts/alerts.json'

      - '/var/ossec/logs/archives/archives.json'.

However, i am not receiving any sysmon log in archives.json, with 3.9.2 no such issue occurred. But when i set level="0" value to 1 in /var/ossec/ruleset/rules/0595-win-sysmon_rules.xml for sysmon, i get sysmon logs in archives.json

Am i missing something.

Thank You

[Jose Miguel Garcia Ramos]

Hi Sanjay,

If I understood you correctly, you have our Wazuh OVA which contains ES, Kibana, and Wazuh-Manager in a virtual machine and you connected a Windows Agent right?

I reviewed and tested your configuration and found no errors.

The behavior you describe is really weird since changing `level` in the rules only affects when a rule is triggered, which doesn't apply to our case since we are getting all events even if they don't trigger a rule.

Agents don't report Sysmon events by default, in order to get Sysmon events from Windows Agent, you need to properly configure it in that machine. You might find useful this link https://wazuh.com/blog/how-to-collect-windows-events-with-wazuh/. After configuring it you should be able to find the logs in `archives.json`.

Please let me know if you have any problems

Best regards,

Jose

On Mon, Jul 22, 2019 at 12:30 PM Jose Miguel Garcia Ramos <jm.g...@wazuh.com> wrote:

Hi Sanjay,

If I understood you correctly, you have our Wazuh OVA which contains ES, Kibana, and Wazuh-Manager in a virtual machine and you connected a Windows Agent right?

I reviewed and tested your configuration and found no errors.

Agents don't report Sysmon events by default, in order to get Sysmon events from Windows Agent, you need to properly configure it in that machine. You might find useful this link https://wazuh.com/blog/how-to-collect-windows-events-with-wazuh/. After configuring it you should be able to find the logs in `archives.json`.

The behavior you describe is really weird since changing `level` in the rules only affects when a rule is triggered, which doesn't apply to our case since we are getting all events even if they don't trigger a rule.

Please let me know if you have any other questions,

Best regards,

Jose

Thanks. 

Will do, thanks Jose.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/96817a84-a72d-4f79-a8e8-31a9ba5a9034%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/d4dae9fd-62c0-47bc-8fe6-bd7cb61acce1%40googlegroups.com.