Retrieving custom logs

[Dunter]

Hey all I've been having some trouble with retrieving custom logs. I have decoders and rules that work.

The logs are located in : /var/www/html/storage/logs and are in the format of device-log-yyy-mm-dd.log

In the shared group agent.conf on on the monitor i have this: 

  <localfile>
    <location>/var/www/html/storage/logs/device-log-%Y-%m-%d.log</location>
    <log_format>regex</log_format>
    <regex type="pcre2">\[202\d-\d\d\-\d\d\s\d\d\:\d\d\:\d\d\]\sproduction.></regex>
  </localfile>

My log files look like this and the regex should grab the info up until INFO:

[2023-03-16 09:32:54] production.INFO: API=start-meeting UUID=8F6B0C79-46D0-4878-822D-3ADC8853EEE9 USERAGENT="Company/4 CFNetwork/1404.0.5 Darwin/22.3.0" AUTH=PERMITTED REQUEST={"site_id":66641,"zoom_url":"https:\/\/company.zoom.us\/j\/93429125467","device_id":"8F655555-444-4878-822D-3ADC8853EEE9"} RESPONSE={"status":200,"message":"Log added Successfully","auth":"PERMITTED"}  

It appears that the monitor is not even receiving the raw data from the location. 

[Francisco Tuduri]

Hello Dunter!

The <localfile> configuration that you are using is invalid. You can use the verify-agent-conf tool to check if the configuration is correct.

When I add that configuration to my agent.conf and run this tool, I get:

verify-agent-conf: Verifying [etc/shared/default/agent.conf]
2023/05/03 13:09:48 verify-agent-conf: ERROR: (1235): Invalid value for element 'log_format': regex.
2023/05/03 13:09:48 verify-agent-conf: ERROR: (1202): Configuration error at 'etc/shared/default/agent.conf'.
2023/05/03 13:09:48 verify-agent-conf: ERROR: (1207): Localfile remote configuration in 'etc/shared/default/agent.conf' is corrupted.

As you can see here, regex is not a valid value for log_format. And the tag regex does not belong to a localfile configuration

The format syslog should work for your log:

<localfile>
    <location>/var/www/html/storage/logs/device-log-%Y-%m-%d.log</location>
    <log_format>syslog</log_format>
  </localfile>

Then you should tune your custom decoders to grab the required data.

Let me know if you need any more help with this.

Regards!

[Dunter]

Thank you so much Francisco. I seem to still be running into issues. 

To start, here is the result of the log-test function: (just to see that it does successfully grab the log)

Type one log per line

[2023-03-16 09:32:54] production.INFO: API=start-meeting UUID=8F6B0C79-46D0-4878-822D-3ADC8853EEE9 USERAGENT="Company/4 CFNetwork/1404.0.5 Darwin/22.3.0" AUTH=PERMITTED REQUEST={"site_id":66641,"zoom_url":"https:\/\/company.zoom.us\/j\/93429125467","device_id":"8F655555-444-4878-822D-3ADC8853EEE9"} RESPONSE={"status":200,"message":"Log added Successfully","auth":"PERMITTED"}

**Phase 1: Completed pre-decoding.
        full event: '[2023-03-16 09:32:54] production.INFO: API=start-meeting UUID=8F6B0C79-46D0-4878-822D-3ADC8853EEE9 USERAGENT="Company/4 CFNetwork/1404.0.5 Darwin/22.3.0" AUTH=PERMITTED REQUEST={"site_id":66641,"zoom_url":"https:\/\/company.zoom.us\/j\/93429125467","device_id":"8F655555-444-4878-822D-3ADC8853EEE9"} RESPONSE={"status":200,"message":"Log added Successfully","auth":"PERMITTED"}'

**Phase 2: Completed decoding.
        name: 'app-server'
        api: 'start-meeting'
        date: '2023-03-16'
        level: 'INFO'
        request_device_id: '8F655555-444-4878-822D-3ADC8853EEE9'
        response_auth: 'PERMITTED'
        response_message: 'Log added Successfully'
        site_id: '66641'
        time: '09:32:54'
        user-agent: 'Company/4 CFNetwork/1404.0.5 Darwin/22.3.0'
        uuid: '8F6B0C79-46D0-4878-822D-3ADC8853EEE9'

**Phase 3: Completed filtering (rules).
        id: '100102'
        level: '2'
        description: 'Informational app-server log'
        groups: '['app-server']'
        firedtimes: '1'
        mail: 'False'

Here is my agent.conf: 

<agent_config>
  <!-- Shared agent configuration here -->

  <localfile>
    <location>/var/www/html/storage/logs/device-log-%Y-%m-%d.log</location>
    <log_format>syslog</log_format>
  </localfile>

</agent_config>

 

& response from verify-agent-log:

verify-agent-conf: Verifying [etc/shared/app-production/agent.conf]

verify-agent-conf: OK 

Here is a screenshot of the log file that should be captured: 

I'm stuck

[Francisco Tuduri]

Ok, one thing to keep in mind is that Wazuh is designed to process current events. Take a look at this other thread.

Basically, when it finds a new file to monitor it processes the new data that is appended to that file since that moment. It does not read the already existing data on that file. (There is a workaround as mentioned on that other thread, and similarly here. However, that is not the intended use case but in some situations might work)

I think that is what could be happening here, all those logs will not be processed because they already existed when the agent started monitoring them.

If new data is added to any of those files, most likely device-log-2023-05-03.log, the new data should be processed. 

Could you check that?

One way to do that is to enable the logall_json option on the manager's ossec.conf, and restart the manager.

Then, when new log entries are added to device-log-2023-05-03.log they should be sent to the manager and stored in /var/ossec/logs/archives/archives.json even if they don't trip a rule.

After doing that, please check the file /var/ossec/logs/archives/archives.json to see if it has any event coming from those logs.

Let me know how it goes.

Regards!

[Dunter]

Thank you so much, its working now!

[Francisco Tuduri]

Great to hear that!

Regards!

[Dunter]

Hey Francisco, I have run into another problem with this - it was working for about 8 hours and then stopped. Can't figure out why. AFAIK nothing changed in the system and there are no errors in the manager log file. I don't have access to the machine that the desired logs are on - is there an API way to pull ossec.log from the agent? 

[Francisco Tuduri]

Hello Dunter!

A few questions to try to understand the situation:

> it was working for about 8 hours and then stopped

Do you mean it worked only 8 hours since your previous message on May 4? Or did you set up a new configuration recently and this is the one that stopped working?

I understand that you are no longer seeing events coming from this agent's /var/www/html/storage/logs/device-log-%Y-%m-%d.log files. But, besides that, the agent is working correctly, right? it is active and receiving other types of events?

How are you seeing that it stopped? are you using the dashboard? alerts.json? archives.json?

Can we be certain that the agent has new events to read from those files?

> I don't have access to the machine that the desired logs are on - is there an API way to pull ossec.log from the agent?

So no way to access the machine remotely?
Unfortunately, we can't pull the agent's log through the API.
However, we can use the API to pull some other information from the agent.
The module logcollector has a limit on the number of files that it can monitor. Once this number is reached it does not add new files to its list. This limit is one of the module's internal configurations. Maybe the problem could be related to this.

We can pull the following info from the agent and see if it gives us any clue on what is going on:

• Get the active configuration for logcollector->localfile and logcollector->internal:
  • API reference: https://documentation.wazuh.com/current/user-manual/api/reference.html#operation/api.controllers.agent_controller.get_agent_config
  • Example calls (for agent id 022):
    • localfile: curl -k -X GET "https://localhost:55000/agents/022/config/logcollector/localfile?pretty=true" -H  "Authorization: Bearer $TOKEN"
    • internal: curl -k -X GET "https://localhost:55000/agents/022/config/logcollector/internal?pretty=true" -H  "Authorization: Bearer $TOKEN"

• Get the logcollector stats:
  • API reference: https://documentation.wazuh.com/current/user-manual/api/reference.html#operation/api.controllers.agent_controller.get_component_stats
  • Example call: curl -k -X GET "https://localhost:55000/agents/022/stats/logcollector?pretty=true" -H  "Authorization: Bearer $TOKEN"

Please, try to gather this information and send it back. Of course, if you think there is any sensitive information in that data please obfuscate it before sharing it.

Regards!

[Dunter]

Thank you! Sorry, to clarify - my  /var/www/html/storage/logs/device-log-%Y-%m-%d.log was only being collected once and then new events would stop. I solved the problem by changing the localfile configuration to this: 

  <localfile>
    <location>/var/www/html/storage/logs/device-log-*.log</location> (previously the * was %Y-%m-%d)
    <log_format>syslog</log_format>
    <age>1d</age> (new addition)
  </localfile>

with my previous localfile that used the sdrt format it seems like wazuh was trying to collect the file at midnight everyday and if it didn't exist, it wouldn't retry to collect it. The way our logs work is that only after the event has been generated does the log file get created for the day. If i'm correct I think this stipulation should be added to the documentation

[Francisco Tuduri]

Oh, I see!

You were on the right track.

Logcollector attempts to read the file to be monitored. If the file does not exist it waits some time and then retries to open the file. If the file still does not exist after so many attempts it just ignores it from that point forward.

The number of attempts and the time between attempts are configurable, but are part of the internal configurations:

• logcollector.open_attempts: Number of attempts to open a log file. The value 0 means that the number of attempts is infinite. Default: 8
• logcollector.vcheck_files: File checking interval, in seconds. Default: 64

I run some tests locally with a non-existing file and default values, and this is the log (only relevant lines, and debug enabled):

2023/05/12 15:30:09 wazuh-logcollector[167496] logcollector.c:940 at update_fname(): INFO: (1952): Monitoring variable log file: '/tmp/log_test-2023-05-12.log'.
2023/05/12 15:30:09 wazuh-logcollector[167496] logcollector.c:974 at handle_file(): ERROR: (1103): Could not open file '/tmp/log_test-2023-05-12.log' due to [(2)-(No such file or directory)].
2023/05/12 15:30:09 wazuh-logcollector[167496] logcollector.c:1070 at handle_file(): DEBUG: (1962): Unable to open file '/tmp/log_test-2023-05-12.log'. Remaining attempts: 7
2023/05/12 15:30:09 wazuh-logcollector[167496] logcollector.c:1228 at set_read(): DEBUG: Socket target for '/tmp/log_test-2023-05-12.log' -> agent
2023/05/12 15:30:09 wazuh-logcollector[167496] logcollector.c:379 at LogCollectorStart(): INFO: (1950): Analyzing file: '/tmp/log_test-2023-05-12.log'.
2023/05/12 15:31:13 wazuh-logcollector[167496] logcollector.c:1070 at handle_file(): DEBUG: (1962): Unable to open file '/tmp/log_test-2023-05-12.log'. Remaining attempts: 6
2023/05/12 15:32:17 wazuh-logcollector[167496] logcollector.c:1070 at handle_file(): DEBUG: (1962): Unable to open file '/tmp/log_test-2023-05-12.log'. Remaining attempts: 5
2023/05/12 15:33:21 wazuh-logcollector[167496] logcollector.c:1070 at handle_file(): DEBUG: (1962): Unable to open file '/tmp/log_test-2023-05-12.log'. Remaining attempts: 4
2023/05/12 15:34:25 wazuh-logcollector[167496] logcollector.c:1070 at handle_file(): DEBUG: (1962): Unable to open file '/tmp/log_test-2023-05-12.log'. Remaining attempts: 3
2023/05/12 15:35:29 wazuh-logcollector[167496] logcollector.c:1070 at handle_file(): DEBUG: (1962): Unable to open file '/tmp/log_test-2023-05-12.log'. Remaining attempts: 2
2023/05/12 15:36:33 wazuh-logcollector[167496] logcollector.c:1070 at handle_file(): DEBUG: (1962): Unable to open file '/tmp/log_test-2023-05-12.log'. Remaining attempts: 1
2023/05/12 15:37:37 wazuh-logcollector[167496] logcollector.c:1070 at handle_file(): DEBUG: (1962): Unable to open file '/tmp/log_test-2023-05-12.log'. Remaining attempts: 0
2023/05/12 15:38:41 wazuh-logcollector[167496] logcollector.c:811 at LogCollectorStart(): INFO: (1904): File not available, ignoring it: '/tmp/log_test-2023-05-12.log'.

So, with the default settings it has to find the target file in less than 9 minutes approx. (8*64 seconds), otherwise, it ignores it.

If your current configuration (using age and the * wildcard) is working fine you can keep using it.

If you'd rather go back to the one with the strftime format string, you should set logcollector.open_attempts to 0 (infinite attempts). This has to be done locally on each desired agent, internal configurations cannot be set remotely. And if you decide to do this remember to do it in local_internal_options.conf (instead of internal_options.conf) so it does not get overwritten during upgrades: https://documentation.wazuh.com/current/user-manual/reference/internal-options.html#internal-configuration

Regards!