Send log files with Wazuh Agents

[MajorFudge]

Hello team,

I'm facing a problem with sending separate log files from the fluentd server.

What I want to achieve:

I have a Fluentd server which querys the vCloud Cell PostgreSQL database and save logs to the local file.

It looks like that:

 sudo ls -alth /var/log/td-agent/ | grep fluent
drwxr-xr-x  2 td-agent td-agent 4.0K Sep 20 01:30 005-fluentd_vcloud
-rw-r--r--  1 td-agent td-agent 4.7M Sep 20 01:30 005-fluentd_vcloud.202309192215.log
-rw-r--r--  1 td-agent td-agent 2.7M Sep 20 01:25 005-fluentd_vcloud.202309192210.log
-rw-r--r--  1 td-agent td-agent 472K Sep 20 01:20 005-fluentd_vcloud.202309192205.log
-rw-r--r--  1 td-agent td-agent 896K Sep 20 01:15 005-fluentd_vcloud.202309192200.log
-rw-r--r--  1 td-agent td-agent 4.5M Sep 20 01:10 005-fluentd_vcloud.202309192155.log
-rw-r--r--  1 td-agent td-agent 4.4M Sep 20 01:05 005-fluentd_vcloud.202309192150.log
-rw-r--r--  1 td-agent td-agent 3.6M Sep 20 01:00 005-fluentd_vcloud.202309192145.log
-rw-r--r--  1 td-agent td-agent 257K Sep 20 00:55 005-fluentd_vcloud.202309192140.log
-rw-r--r--  1 td-agent td-agent 563K Sep 20 00:50 005-fluentd_vcloud.202309192135.log
-rw-r--r--  1 td-agent td-agent  22M Sep 20 00:34 005-fluentd_vcloud.202309190000.log

So fluentd save the buffer chunks to file, these files rotate and etc.

Here is the example of a log from one of the file:

sudo tail -n 1 /var/log/td-agent/005-fluentd_vcloud.202309192145.log
2023-09-20T00:49:56+03:00 vcloud.audit_trail {"id":2906075,"event_type":"com/vmware/vcloud/event/oAuthToken/create","event_time":"2023-09-19 21:49:55.847000+0000","org_member_id":"07e76aa-1835-3f8-962a-717e352f5c6a","tenant_id":"a93c9db9-7471-312-8d09-a8f7eeda85f9","event":{"cellId":{"id":"22d9b76c-86a8-4d9e-ae89-c2e127fae4b5"},"status":"FAILURE","taskId":null,"actorId":null,"eventId":{"id":"5485b7e3-298e-47c9-a128-d95575384a6"},"orgUser":{"id":"07e7aa-1835-39f8-962a-717e352f5c6a","name":"system","type":"com.vmware.vcloud.entity.user"},"actorName":null,"entityRef":{"id":"","name":null,"type":"com.vmware.vcloud.entity.token"},"eventTime":1695160195.847,"eventType":"com/vmware/vcloud/event/oAuthToken/create","internalId":null,"taskCellId":null,"description":"OAuth token creation for client '${oauthToken.clientName}(${oauthToken.clientId})' failed","serviceName":"com.vmware.vcloud","operatingOrg":{"id":"a93c9db9-7471-3192-8d09-a8f7eeda85f9","name":"System","type":"com.vmware.vcloud.entity.org"},"internalUserId":{"id":"808cc07b-7092-3666-bcc5-89910188d5e7"},"hasExternalObject":false,"additionalProperties":{"oauthToken.clientId":null,"currentContext.user.id":{"id":"808cc07b-7092-3666-bcc5-89910188d5e7","name":"system","type":"com.vmware.vcloud.entity.user"},"currentContext.login.org.id":{"id":"a93c9db9-7471-3192-8d09-a8f7eeda85f9","name":null,"type":"com.vmware.vcloud.entity.org"},"currentContext.login.user.id":{"id":"808cc07b-7092-3666-bcc5-89910188d5e7","name":"system","type":"com.vmware.vcloud.entity.user"},"currentContext.user.proxyAddress":"","currentContext.user.clientIpAddress":""}}}

I have installed the Wazuh agent on that server and set up the location config:

  <localfile>
    <log_format>postgresql_log</log_format>
    <location>/var/log/td-agent/005-fluentd_vcloud*.log</location>
    <out_format>vcloud: $(log)</out_format>
  </localfile>

I expect that agent will add this tag vcloud: $(log) to the event and I will be able to build a custom decoder.

On the Wazuh server I have created a custom decoder:

<decoder name="vcloud_cell">
    <prematch>^vcloud: </prematch>
    <plugin_decoder>JSON_Decoder</plugin_decoder>
</decoder>

And a simple rule:

  <rule id="100012" level="5">
        <decoded_as>vcloud_cell</decoded_as>
        <description>Logs from the vCloud Cell</description>
        <options>no_full_log</options>
  </rule>

I have turned on logall_json, but I don't see anything in the archives.json.

Please assist me with that little issue.

Kind regards,

Kirill

[MajorFudge]

Hello, team!

Any update regarding this topic?

среда, 20 сентября 2023 г. в 00:45:08 UTC+2, MajorFudge:

[Mateo Cervilla]

Hi, sorry for the delay

I'm going to try to replicate your issue and check if I see something wrong.

Your configuration looks good.

Meanwhile, can you share me one of the log files so I can do the test more accurate?

I'll let you know as soon as I can.

Regards

[Mateo Cervilla]

Hi, thanks for the logs.

I've been testing and fixed the issues:

You should configure the localfile like this:

<localfile>
    <log_format>syslog</log_format>

    <location>/var/log/td-agent/005-fluentd_vcloud*.log</location>
    <out_format>vcloud: $(log)</out_format>
</localfile>

And the decoder:

<decoder name="vcloud_cell">
    <prematch>^vcloud: </prematch>

</decoder>

Apparently, the postgresql_log was preventing localfile to work properly and <plugin_decoder>JSON_Decoder</plugin_decoder> did the same with the decoder.

Try with this configurations and let me know how it goes.

Regards!

[MajorFudge]

Hi, thank you for your reply.

Unfortunately this didn't help.

I still don't see anything in the archives.

And here I have a question:

Fluentd save logs to buffer file and some period of time buffer flushes log chunks to the file 005-fluentd_vcloud_202309*.log.

These files have following owners and permissions:

-rw-r--r--  1 td-agent td-agent 5.7M Sep 22 10:40 005-fluentd_vcloud.202309220725.log
-rw-r--r--  1 td-agent td-agent 6.0M Sep 22 10:35 005-fluentd_vcloud.202309220720.log

So the owner always a td-agent (fluentd). Could this cause the problem?

четверг, 21 сентября 2023 г. в 19:00:51 UTC+2, Mateo Cervilla:

[Mateo Cervilla]

Hi,

-rw-r--r-- means that it has read permissions for everyone, so I don't think that's the reason.

It may be an issue with the connection between the manager and the agent.

Have you configured the localfile on the agent side right?

Do you receive any other alerts from that agent?

Can you check if the agent is online?

• /var/ossec/bin/agent_control -l
  

You can also check the debug logs for the logcollector on the agent and check if there is some error:

On the Agent:

• Enable logcollector.debug=2 in /var/ossec/etc/internal_options.conf
• Restart the agent systemctl restart wazuh-agent
• Check or share me the logs /var/ossec/logs/ossec.log

Regards

[Kirill Golubenko]

Hi,

Here is my ossec.conf location config on the agent side:

<ossec_config>
  <localfile>
    <log_format>syslog</log_format>
    <location>/var/ossec/logs/active-responses.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/dpkg.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/kern.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/td-agent/005-fluentd_vcloud*.log</location>
    <out_format>vcloud: $(log)</out_format>
  </localfile>

</ossec_config>

Here is the output of the sudo /var/ossec/bin/agent_control -l
Wazuh agent_control. List of available agents:
   ID: 017, Name: wfluentd01-srv-infra, IP: any, Active

By the way, when I go to Wazuh dashboard > Agent > wfluentd01-srv-infra > Log collection, I see these files:

Also I see that some alerts generates:

In the attachment you will find the debug output:

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/PpQGT1PIu-w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/5806d33e-cd8b-44ce-9709-b971e06902a6n%40googlegroups.com.

[Mateo Cervilla]

Everything looks good,

I can see in the logs that the files are being monitored and don't see any errors or warnings:

2023/09/22 21:14:42 wazuh-logcollector[3185319] logcollector.c:1371 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 7/1000.
2023/09/22 21:14:42 wazuh-logcollector[3185319] logcollector.c:1356 at check_pattern_expand(): DEBUG: (1957): New file that matches the '/var/log/td-agent/005-fluentd_vcloud*.log' pattern: '/var/log/td-agent/005-fluentd_vcloud.202309220720.log'.
2023/09/22 21:14:42 wazuh-logcollector[3185319] logcollector.c:1371 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 8/1000.
2023/09/22 21:14:42 wazuh-logcollector[3185319] logcollector.c:1356 at check_pattern_expand(): DEBUG: (1957): New file that matches the '/var/log/td-agent/005-fluentd_vcloud*.log' pattern: '/var/log/td-agent/005-fluentd_vcloud.202309220725.log'.
2023/09/22 21:14:42 wazuh-logcollector[3185319] logcollector.c:1371 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 9/1000.
2023/09/22 21:14:42 wazuh-logcollector[3185319] logcollector.c:1356 at check_pattern_expand(): DEBUG: (1957): New file that matches the '/var/log/td-agent/005-fluentd_vcloud*.log' pattern: '/var/log/td-agent/

I believe the events should be recorded in the archives.

Can you share me your /var/ossec/logs/archives/archives.json from your manager?

[MajorFudge]

Sure, here it is: https://drive.google.com/file/d/1RBksSpy0rLPL6UMjHP8YW0ehYTGF4SMa/view?usp=drive_link

File is bigger then 100 Mb, so I can't put it in the attachments.

пятница, 22 сентября 2023 г. в 22:23:37 UTC+2, Mateo Cervilla:

[MajorFudge]

It looks like the archives.json does not include my logs from that agent.

суббота, 23 сентября 2023 г. в 09:43:48 UTC+2, MajorFudge:

[Mateo Cervilla]

Hi, sorry for the delay.

I don't have access to that file https://drive.google.com/file/d/1RBksSpy0rLPL6UMjHP8YW0ehYTGF4SMa/view?usp=drive_link

Can you give me?

Regards

[MajorFudge]

Hi Mateo,

https://drive.google.com/file/d/1RBksSpy0rLPL6UMjHP8YW0ehYTGF4SMa/view?usp=sharing

Try it one more time.

понедельник, 25 сентября 2023 г. в 22:46:06 UTC+2, Mateo Cervilla:

[MajorFudge]

Hello Mateo,

Sorry for pushing

Could you please check this log file. I still don't see anything in the archives.json file.

вторник, 26 сентября 2023 г. в 08:33:36 UTC+2, MajorFudge:

[MajorFudge]

I have checked Wazuh dashboard: agent >> configuration >> Log collection

I see that Wazuh receives these file, and my tag is in place: vcloud: $(log)

And that is what I see:

{
  "logcollector-localfile": {
    "localfile": [
      {
        "logformat": "command",
        "command": "df -P",
        "alias": "df -P",
        "ignore_binaries": "no",
        "target": [
          "agent"
        ],
        "frequency": 360
      },
      {
        "logformat": "full_command",
        "command": "netstat -tulpn | sed 's/\\([[:alnum:]]\\+\\)\\ \\+[[:digit:]]\\+\\ \\+[[:digit:]]\\+\\ \\+\\(.*\\):\\([[:digit:]]*\\)\\ \\+\\([0-9\\.\\:\\*]\\+\\).\\+\\ \\([[:digit:]]*\\/[[:alnum:]\\-]*\\).*/\\1 \\2 == \\3 == \\4 \\5/' | sort -k 4 -g | sed 's/ == \\(.*\\) ==/:\\1/' | sed 1,2d",
        "alias": "netstat listening ports",
        "ignore_binaries": "no",
        "target": [
          "agent"
        ],
        "frequency": 360
      },
      {
        "logformat": "full_command",
        "command": "last -n 20",
        "alias": "last -n 20",
        "ignore_binaries": "no",
        "target": [
          "agent"
        ],
        "frequency": 360
      },
      {
        "file": "/var/ossec/logs/active-responses.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "target": [
          "agent"
        ]
      },
      {
        "file": "/var/log/auth.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "target": [
          "agent"
        ]
      },
      {
        "file": "/var/log/dpkg.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "target": [
          "agent"
        ]
      },
      {
        "file": "/var/log/kern.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "target": [
          "agent"
        ]
      },
      {
        "file": "/var/log/td-agent/005-fluentd_vcloud.202309271955.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "out_format": [
          {
            "target": "all",
            "format": "vcloud: $(log)"
          }
        ]
      },
      {
        "file": "/var/log/td-agent/005-fluentd_vcloud.202309272000.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "out_format": [
          {
            "target": "all",
            "format": "vcloud: $(log)"
          }
        ]
      },
      {
        "file": "/var/log/td-agent/005-fluentd_vcloud.202309272005.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "out_format": [
          {
            "target": "all",
            "format": "vcloud: $(log)"
          }
        ]
      },
      {
        "file": "/var/log/td-agent/005-fluentd_vcloud.202309272010.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "out_format": [
          {
            "target": "all",
            "format": "vcloud: $(log)"
          }
        ]
      }
    ],
    "localfile-logs": [
      {
        "file": "/var/ossec/logs/active-responses.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "target": [
          "agent"
        ]
      },
      {
        "file": "/var/log/auth.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "target": [
          "agent"
        ]
      },
      {
        "file": "/var/log/dpkg.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "target": [
          "agent"
        ]
      },
      {
        "file": "/var/log/kern.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "target": [
          "agent"
        ]
      },
      {
        "file": "/var/log/td-agent/005-fluentd_vcloud.202309271955.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "out_format": [
          {
            "target": "all",
            "format": "vcloud: $(log)"
          }
        ]
      },
      {
        "file": "/var/log/td-agent/005-fluentd_vcloud.202309272000.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "out_format": [
          {
            "target": "all",
            "format": "vcloud: $(log)"
          }
        ]
      },
      {
        "file": "/var/log/td-agent/005-fluentd_vcloud.202309272005.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "out_format": [
          {
            "target": "all",
            "format": "vcloud: $(log)"
          }
        ]
      },
      {
        "file": "/var/log/td-agent/005-fluentd_vcloud.202309272010.log",
        "logformat": "syslog",
        "ignore_binaries": "no",
        "only-future-events": "yes",
        "out_format": [
          {
            "target": "all",
            "format": "vcloud: $(log)"
          }
        ]
      }
    ],
    "localfile-windowsevent": [],
    "localfile-commands": [
      {
        "logformat": "command",
        "command": "df -P",
        "alias": "df -P",
        "ignore_binaries": "no",
        "target": [
          "agent"
        ],
        "frequency": 360
      },
      {
        "logformat": "full_command",
        "command": "netstat -tulpn | sed 's/\\([[:alnum:]]\\+\\)\\ \\+[[:digit:]]\\+\\ \\+[[:digit:]]\\+\\ \\+\\(.*\\):\\([[:digit:]]*\\)\\ \\+\\([0-9\\.\\:\\*]\\+\\).\\+\\ \\([[:digit:]]*\\/[[:alnum:]\\-]*\\).*/\\1 \\2 == \\3 == \\4 \\5/' | sort -k 4 -g | sed 's/ == \\(.*\\) ==/:\\1/' | sed 1,2d",
        "alias": "netstat listening ports",
        "ignore_binaries": "no",
        "target": [
          "agent"
        ],
        "frequency": 360
      },
      {
        "logformat": "full_command",
        "command": "last -n 20",
        "alias": "last -n 20",
        "ignore_binaries": "no",
        "target": [
          "agent"
        ],
        "frequency": 360
      }
    ]
  },
  "logcollector-socket": {}
}

среда, 27 сентября 2023 г. в 22:23:01 UTC+2, MajorFudge:

[Mateo Cervilla]

Hi, 

I'm currently checking it with my team.

Thanks for giving more information, it is always appreciated.

It look like the logcollector is working property, but something is still messing with the system.

We are discussing your issue and as soon I have an answer for you I'll let you know.

Sorry for keep you waiting.

Regards

[MajorFudge]

I don't know if that can help, but here you can find my architecture scheme.

четверг, 28 сентября 2023 г. в 05:16:02 UTC+2, Mateo Cervilla:

[MajorFudge]

Hi Mateo,

I have finally fix it by removing the start of the string symbol from the prematch in the decoder.

<decoder name="vcloud_cell">
    <prematch>vcloud: </prematch>
</decoder>

Of course it is better to make this tag more specific.

Thank you very much for your time!

четверг, 28 сентября 2023 г. в 19:12:34 UTC+2, MajorFudge:

[MajorFudge]

No, I was wrong.

I have made a local test and create a file on the Wazuh server with logs from vcloud.

I configured a localfile and restarted the wazuh-server, but nothing has happened.

After that I tried to change the decoder and removed the ^ symbol from the prematch. And it actually fixed the problem.

But logs from the remote agent still didn't come up.

After that on the fluentd server I have created a static file /var/log/td-agent/005-fluentd_vcloud.log and put few lines there.

I have also changed the paramert logcollector.open_attempts=0.

In the ossec.conf I set to monitor this static file.

  <localfile>
    <log_format>json</log_format>
    <location>/var/log/td-agent/005-fluentd_vcloud.log</location>
    <out_format>vcloud_remote: $(log)</out_format>
  </localfile>

After restart I was able to see these logs in the archives.

And here is the main part, on the agent side in the log I see that agent reads the log:

2023/09/29 11:54:50 wazuh-logcollector[3725875] read_json.c:158 at read_json(): DEBUG: Read 5 lines from /var/log/td-agent/005-fluentd_vcloud.log
2023/09/29 11:56:15 wazuh-logcollector[3725875] read_json.c:158 at read_json(): DEBUG: Read 6 lines from /var/log/td-agent/005-fluentd_vcloud.log

After that I tried to change back the localfile config with masked location 005-fluentd_vcloud.*.log:

  <localfile>
    <log_format>json</log_format>
    <location>/var/log/td-agent/005-fluentd_vcloud.*.log</location>
    <out_format>vcloud_remote: $(log)</out_format>
  </localfile>

In the logs I see that agent is able to discover these files, but it doen't read them:

2023/09/29 12:20:22 wazuh-logcollector[3727640] logcollector.c:1356 at check_pattern_expand(): DEBUG: (1957): New file that matches the '/var/log/td-agent/005-fluentd_vcloud.*.log' pattern: '/var/log/td-agent/005-fluentd_vcloud.202309290905.log'.

I don't see an option to  write all logs to one file on the fluentd level. So it creates a log file every 5 minutes. So now I need to solve the problem and make Wazuh agent read these files.

пятница, 29 сентября 2023 г. в 10:38:29 UTC+2, MajorFudge:

[MajorFudge]

I made one more test.

I have added new line to one of the file that has been discovered but not read.

After that an alert was created in the archives.

That means, that if I have a static file, wazuh agent will discover it, but will not read it.

But if you append a new line there, in that case it will read it.

As a workaround I put to the localfile a new path that points to the buffer files of the fluentd.

пятница, 29 сентября 2023 г. в 11:31:08 UTC+2, MajorFudge:

[Mateo Cervilla]

Hello again,

So did you manage to solve your problem?
If so, I'm glad you got it.

If not, or if you still need more help, don't hesitate to contact me.

Kind regards!