ossec-logcollector holding deleted files

[Daniil Sobolev]

Hi Wazuh team! 

We've noticed that ossec-logcollector keeps holding deleted files preventing them from rotation, which leading to additional disk space  utilization.

Is it expected behaviour? Could it be fixed somehow if not?

We're currently on wazuh 3.9.5.

Thanks! 

lsof -p 16546

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME

ossec-log 16546 root  cwd    DIR              253,0      244       64 /

ossec-log 16546 root  rtd    DIR              253,0      244       64 /

ossec-log 16546 root  txt    REG              253,0  1496182  6384623 /var/ossec/bin/ossec-logcollector

ossec-log 16546 root  mem    REG              253,0    61560  2167061 /usr/lib64/libnss_files-2.17.so

ossec-log 16546 root  mem    REG              253,0  2156272  2167043 /usr/lib64/libc-2.17.so

ossec-log 16546 root  mem    REG              253,0   142144  2167069 /usr/lib64/libpthread-2.17.so

ossec-log 16546 root  mem    REG              253,0  6807736  4673103 /var/ossec/lib/libwazuhext.so

ossec-log 16546 root  mem    REG              253,0    19248  2167049 /usr/lib64/libdl-2.17.so

ossec-log 16546 root  mem    REG              253,0    43712  2167073 /usr/lib64/librt-2.17.so

ossec-log 16546 root  mem    REG              253,0   163312  2167036 /usr/lib64/ld-2.17.so

ossec-log 16546 root    0u   CHR                1,3      0t0     1028 /dev/null

ossec-log 16546 root    1u   CHR                1,3      0t0     1028 /dev/null

ossec-log 16546 root    2u   CHR                1,3      0t0     1028 /dev/null

ossec-log 16546 root    3r   CHR                1,9      0t0     1033 /dev/urandom

ossec-log 16546 root    4u  unix 0xffff99885eb57b40      0t0 23564518 socket

ossec-log 16546 root    5r   REG              253,2 53227812       72 /var/log/audit/audit.log

ossec-log 16546 root    6r   REG              253,0        0  6384664 /var/ossec/logs/active-responses.log

ossec-log 16546 root    7r   REG              253,1    14742     4464 /var/log/messages

ossec-log 16546 root    8r   REG              253,1     6863     4483 /var/log/secure-20210329 (deleted)

ossec-log 16546 root    9r   REG              253,1        0      107 /var/log/maillog

ossec-log 16546 root   10r   REG              253,1  1271009       94 /var/log/kern.log

ossec-log 16546 root   11u  unix 0xffff9988cd796a40      0t0 23562017 /var/ossec/queue/ossec/logcollector

[Victor M Fernandez-Castro]

Hi Daniil,

Logcollector holds the log files, but it checks every 64 seconds (internal option logcollector.vcheck_files) whether the files were truncated or rotated:

• If the actual file size is less than the currently open cursor, the file was truncated.
• If the actual file inode is different than the currently open file, the file was rotated.

In both cases, Logcollector will reload the file.

For instance, if we are monitoring a sample file like /root/test/secure.log, and we rotate it:

rm secure.log && touch secure.log

lsof -p `pidof wazuh-logcollector`

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME

(...))
wazuh-log 12639 root    7r   REG                8,1        0     530788 /root/test/secure.log (deleted)

ls -i secure.log
517279 secure.log

Indeed, the inode number has changed from 530788 to 517279. After a while, Logcollector detects that rotation:

2021/03/29 12:24:39 wazuh-logcollector[12639] logcollector.c:444 at LogCollectorStart(): DEBUG: Performing file check.
2021/03/29 12:24:39 wazuh-logcollector[12639] logcollector.c:661 at LogCollectorStart(): DEBUG: File inode changed. /root/test/secure.log

Then Logcollector produces an alert:

** Alert 1617013479.5967: - ossec,pci_dss_10.5.2,pci_dss_10.5.5,gpg13_10.1,gdpr_II_5.1.f,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.9,tsc_CC6.1,tsc_CC7.2,tsc_CC7.3,tsc_PI1.4,tsc_PI1.5,tsc_CC7.1,tsc_CC8.1,
2021 Mar 29 12:24:39 groovy->wazuh-logcollector
Rule: 591 (level 3) -> 'Log file rotated.'
ossec: File rotated (inode changed): '/root/test/secure.log'.

If we check lsof again, Logcollector is now following the new file. That allowed the OS remove the old file.

lsof -p `pidof wazuh-logcollector`

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME

(...)
wazuh-log 12639 root    7r   REG                8,1        0     517279 /root/test/secure.log

You can let Logcollector chack the files more often, overwriting logcollector.vcheck_files in etc/local_internal_options.conf:

logcollector.vcheck_files=30

On the other hand, if your system has any particular condition that prevents Logcollector from detecting file rotation (like some file system) you can make Logcollector reload the files no matter if they have been rotated or not:

logcollector.force_reload=1

Hope that helps.

Best regards,

Victor M. Fernandez-Castro 
Director of engineering | vic...@wazuh.com

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/dd6fbf02-f937-4017-b44d-db479cd5569fn%40googlegroups.com.

[Daniil Sobolev]

Hi Viktor,

Thank you very much for your answer! 

I wonder how much time it could take for logcollector to release the file? I can see it holding it for hours.

Also, if the agent cannot connect to wazuh server will it affect logcollector's behaviour that you've described above? 

 

понедельник, 29 марта 2021 г. в 13:44:03 UTC+3, Victor Fernandez:

[Victor M Fernandez-Castro]

Hi Daniil,

Logcollector should check for file truncation (and therefore reload it) every 64 seconds with the default configuration. However, we detected that Logcollector would stop checking files while the agent is disconnected. In fact, the agent connector requests Logcollector to "freeze" while it's disconnected. That behavior was fixed in 3.11.0 (PR #4222). From 3.11, Logcollector should continue checking files every 64 seconds even if the agent is disconnected.

Best regards,

Victor M. Fernandez-Castro 
Director of engineering | vic...@wazuh.com

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f9fb37c9-ba52-49d9-a460-0176b9911753n%40googlegroups.com.

[Daniil Sobolev]

Thanks Victor, that explains our issue! We'll consider upgrading.

понедельник, 29 марта 2021 г. в 14:11:32 UTC+3, Victor Fernandez: