File limit reached for local files

[Bibek Chaudhary]

Hi,

If a local file is configured to read a folder which contains more than 200 files, we get an error.

wazuh-agent: WARNING: (1960): File limit has been reached (200).

Wazuh version: 4.3.5

Config:

  <localfile>

     <location>C:\Users\test_folder\*</location>

     <log_format>syslog</log_format>    

  </localfile>

Looking at the wazuh code, it seems to be hard coded for the local files,

https://github.com/wazuh/wazuh/blob/a2b13fb67bc737a34102b911f3721b95ccec6ba4/src/logcollector/logcollector.h#L21

Do we have any configuration to change this ?

[Md. Nazmur Sakib]

Hi Bibek Chaudhary,

Hope you are doing well. Thank you for using Wazuh.

You are getting the error "wazuh-agent: WARNING: (1960): File limit has been reached (200)." for <syscheck> file limit.

By default, agents stop adding files to the database once 100k files have been scanned.

To change this limit, you need to change the subsection <file_limit> (inside section <syscheck>) in your agents' configuration file.

You can increase the file limit number or even disable this limit.

<file_limit>

  <enabled>yes</enabled>

  <entries>100000</entries>

</file_limit>

Default value: 100000

Allowed values: Integer number between 1 and 2147483647.

Please take a look at this link: https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/syscheck.html#file-limit

Keep in mind that if the number of files scanned is increased, then the time for a complete file scan is also increased. If you increase the limit, you may also need to increase the scan frequency time to set it to an according rate. So that the agent is not constantly scanning the files.

I would recommend ignoring the folders where files are generated and modified continuously by the system.

You can do that using the ignore tag.

Ex:

<!-- Files/directories to ignore -->

<ignore>/var/log</ignore>

Note that this configuration needs to change inside the ossec.conf file of agent or agent.conf of the group.

Make sure to restart the agent after changing the configuration in the ossec.conf

Please let me know if this helps or if you need any further information regarding this.

Regards

Md. Nazmur Sakib

[Bibek Chaudhary]

Hi Nazmur Sakib,

Thank you for the reply.

I increased the value from 100k to 200k, but the error persists. Additionally, I'd like to point out that when I include the localfile configuration I mentioned in my question, which monitors a directory containing 201 files, that's when the "File limit has been reached (200)" error occurs.

<localfile>
      <location>%SystemDrive%\multiple\clone\\*</location>
      <log_format>syslog</log_format>
</localfile>

Regards

[Bibek Chaudhary]

Hi,

Also, you can replicate it by using the following attached dns sample log file. I have made 201 copies of it at "%SystemDrive%\multiple\clone\\".

Regards

[Md. Nazmur Sakib]

Hi Bibek Chaudhary,

Sorry for the late response. 

As you can see there is another warning agent message queue is full. This is because of this configuration you are pointing to and the Wazuh Anti-flooding mechanism.

<localfile>

      <location>%SystemDrive%\multiple\clone\\*</location>

      <log_format>syslog</log_format>

</localfile>

WARNING: (1960): File limit has been reached (200) is because of the <syscheck> file limit and your <syscheck>  configuration.

Check this document to learn more about Anti-flooding mechanism:

https://documentation.wazuh.com/current/user-manual/agents/antiflooding.html

queue_size: Sets the capacity of the agent buffer in number of events.

Default value: 5000

Allowed values: Any number between 1 and 100000.

events_per_second:

Specifies the number of events that can be sent to the manager per second.

Default value: 500

Allowed values: Any number between 1 and 1000.

The default configuration is:

<client_buffer>

  <!-- Agent buffer options -->

  <disabled>no</disabled>

  <queue_size>5000</queue_size>

  <events_per_second>500</events_per_second>

</client_buffer>

Let's say in the default configuration if your agent is sending about 700EPS buffer will be filled in 25 seconds.

Change this configuration inside your ossec.conf of agent based on your need.

https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/client-buffer.html

Also, check Anti-flooding in agent modules:

https://documentation.wazuh.com/current/user-manual/agents/antiflooding.html#anti-flooding-in-agent-modules

The variables defined for this purpose are called logcollector.max_lines, wazuh_modules.max_eps, and much care should be given when changing these values. These are advanced configurations located at Internal configuration.

I hope this answers your questions. Please let me know if you need any further information or assistance.

Regards

[Bibek Chaudhary]

Hi Nazmur Sakib,

I've made adjustments by increasing the queue_size and eps, as well as modifying the values for logcollector.max_lines and wazuh_modules.max_eps, as shown below:

<client_buffer>
      <disabled>no</disabled>
      <events_per_second>700</events_per_second>
      <queue_size>10000</queue_size>
</client_buffer>

# Logcollector - Maximum number of lines to read from the same file [100..1000000]
# 0. Disable line burst limitation
logcollector.max_lines=100000

# Wazuh modules - maximum number of events per second sent by each module [1..1000]
wazuh_modules.max_eps=500

However, the same error message persists. 

I'm curious about why the error message is somewhat misleading, stating "File limit has been reached (200)" instead of providing a more direct error message related to the queue size or buffer.

Regards

[Md. Nazmur Sakib]

Hi Bibek Chaudhary,

The above configurations are to tackle the error you were getting “agent message queue is full” as I saw another error in the screenshot you shared.

To tackle the WARNING: (1960): File limit has been reached (200) error consider the following:

In Internal configuration increase the value of the following configuration:

# Logcollector - Maximum number of files to be monitored [1..100000]

logcollector.max_files=

# Maximum number of file descriptor that Logcollector can open [1024..1048576]

# This value must be higher than logcollector.max_files

logcollector.rlimit_nofile=

Change the subsection <file_limit> (inside section <syscheck>) in your agent’s configuration file.

You can increase the file limit number or even disable this limit.

Allowed values: Integer number between 1 and 2147483647.

Ignore the folders where files are generated and modified continuously by the system.

You can do that using the ignore tag. inside the configuration of the agent.

Ex:

<!-- Files/directories to ignore -->

<ignore>/var/log</ignore>

Restart the manager and agent after making changes in their configuration.

If you still face this issue. It might be because your agent is sending more files than the limit.

Let me know if this solves your issue.

Regards 

Md. Nazmur Sakib

[Bibek Chaudhary]

Hi Nazmur Sakib,

I have configured my setup quite simplistically, so it's unlikely that the number of files has surpassed the limit. I kindly request you to examine my configurations alongside the accompanying log files. From my perspective, if the file limit had been exceeded, it should have generated a warning message indicating the limit I've configured, rather than referencing an arbitrary 200-file limit, which doesn't appear to be documented anywhere.

Thanks,

Bibek Chaudhary

[Bibek Chaudhary]

Hi,

I looked deeper in the code and found the following:

https://github.com/wazuh/wazuh/blob/2a2b88bfb2ea30903728372471b33540a3b3d976/src/config/localfile-config.c#L21-L70

    if (maximum_files && current_files >= maximum_files) {
        mwarn(FILE_LIMIT, maximum_files);
        return 0;
    }

I am guessing this part of the code is responsible for the warning I am receiving. But I am not sure, where this maximum_files value is initialized.

Is it from here: https://github.com/wazuh/wazuh/blob/2a2b88bfb2ea30903728372471b33540a3b3d976/src/logcollector/config.c#L45-L46 ?

maximum_lines = getDefine_Int("logcollector", "max_lines", 0, 1000000);
maximum_files = getDefine_Int("logcollector", "max_files", 1, 100000);

If so, then I should be getting another error "(1960): File limit has been reached (%d). Please reduce the number of files or increase \"logcollector.max_files\"." and not 

"(1960): File limit has been reached (%d).".

The error message is defined in here: https://github.com/wazuh/wazuh/blob/2a2b88bfb2ea30903728372471b33540a3b3d976/src/error_messages/error_messages.h#L219-L222

I do get the error "(1960): File limit has been reached (%d). Please reduce the number of files or increase \"logcollector.max_files\"." ,  if I have more files in a folder than defined in "logcollector.max_files", which I find valid.

So, what is the config that needs to be changed to avoid the other error ?

Regards,

Bibek Chaudhary

[Bibek Chaudhary]

Hi,

In logcollector.h file WIN32_MAX_FILES has been initialized to 200, which is then used in the config.c file, which is causing the error.

logcollector.h: https://github.com/wazuh/wazuh/blob/v4.3.5/src/logcollector/logcollector.h#L21

config.c: https://github.com/wazuh/wazuh/blob/v4.3.5/src/logcollector/config.c#L78-L82

New log, after I ran the client in debug mode.

2023/09/20 21:41:32 wazuh-agent[1996] win_utils.c:116 at local_start(): DEBUG: Reading logcollector configuration.
2023/09/20 21:41:32 wazuh-agent[1996] config.c:81 at LogCollectorConfig(): DEBUG: The maximum number of files to monitor cannot exceed 200 in Windows, so it will be limited.

2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1628 at check_pattern_expand(): INFO: (1957): New file that matches the 'C:\multiple\clone\\*' pattern: 'C:\multiple\clone\\Clone_94.txt'.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1643 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 196/200.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1628 at check_pattern_expand(): INFO: (1957): New file that matches the 'C:\multiple\clone\\*' pattern: 'C:\multiple\clone\\Clone_95.txt'.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1643 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 197/200.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1628 at check_pattern_expand(): INFO: (1957): New file that matches the 'C:\multiple\clone\\*' pattern: 'C:\multiple\clone\\Clone_96.txt'.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1643 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 198/200.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1628 at check_pattern_expand(): INFO: (1957): New file that matches the 'C:\multiple\clone\\*' pattern: 'C:\multiple\clone\\Clone_97.txt'.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1643 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 199/200.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1628 at check_pattern_expand(): INFO: (1957): New file that matches the 'C:\multiple\clone\\*' pattern: 'C:\multiple\clone\\Clone_98.txt'.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1643 at check_pattern_expand(): DEBUG: (1961): Files being monitored: 200/200.
2023/09/20 21:41:35 wazuh-agent[1996] logcollector.c:1549 at check_pattern_expand(): WARNING: (1960): File limit has been reached (200).

Also, I found an issue created in Wazuh, which looks similar to me.

https://github.com/wazuh/wazuh/issues/14144

Regards,

Bibek Chaudhary

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/HxQk_rWE2Dw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f0361a28-cf2b-42b2-8c02-0e43b4894ea9n%40googlegroups.com.