We are running Wazuh version 4.2.5 with around 700 custom rules and 10 custom decoder files. When we try to create/update any decoder/rule files, we get error 3013 - Timeout executing API request. The same issue discussed under thread API Timeout (google.com) . From api.log, I can see that the request failed with /manager/configuration/validation timeout 500. This was supposed to be fixed as per bug Fix GET /manager/configuration/validation timeout issue by CarlosRS9 · Pull Request #6635 · wazuh/wazuh (github.com)
We have found the same issue with multiple Wazuh clusters.
Request timeout is set to 400000 from Wazuh API Configuration
I was not able to find the option api_socket.settimeout(10) in framework/wazuh/core/manager.py file somehow.
Can you please have a look at this and suggest a fix:
Screenshot of the error:
Wazuh API Config:
Extract from api.log:
2022/01/19 11:18:58 INFO: wazuh-wui 127.0.0.1 "PUT /rules/files/0600-SentinelOne-decoder.xml" with parameters {"overwrite": "false"} and body {} done in 0.008s: 200
2022/01/19 11:18:58 INFO: wazuh-wui 127.0.0.1 "GET /cluster/status" with parameters {} and body {} done in 0.010s: 200
2022/01/19 11:19:09 INFO: wazuh-wui 127.0.0.1 "GET /cluster/configuration/validation" with parameters {} and body {} done in 11.204s: 500
2022/01/19 11:19:10 INFO: wazuh-wui 127.0.0.1 "GET /security/user/authenticate" with parameters {} and body {} done in 0.267s: 200
2022/01/19 11:19:10 INFO: wazuh-wui 127.0.0.1 "GET /manager/info" with parameters {} and body {} done in 0.027s: 200
2022/01/19 11:19:10 INFO: wazuh-wui 127.0.0.1 "GET /agents" with parameters {"agents_list": "000"} and body {} done in 0.008s: 200
2022/01/19 11:19:10 INFO: wazuh-wui 127.0.0.1 "GET /cluster/status" with parameters {} and body {} done in 0.009s: 200
2022/01/19 11:19:10 INFO: wazuh-wui 127.0.0.1 "GET /cluster/local/info" with parameters {} and body {} done in 0.007s: 200
2022/01/19 11:19:10 INFO: wazuh-wui 127.0.0.1 "DELETE /rules/files/0600-SentinelOne-decoder.xml" with parameters {} and body {} done in 0.008s: 200
Extract from ossec.log:
2022/01/19 11:56:29 sca: INFO: Loaded policy '/var/ossec/ruleset/sca/cis_debian10.yml'
2022/01/19 11:56:29 sca: INFO: Starting Security Configuration Assessment scan.
2022/01/19 11:56:29 wazuh-modulesd:database: INFO: Module started.
2022/01/19 11:56:29 wazuh-modulesd:download: INFO: Module started.
2022/01/19 11:56:29 wazuh-modulesd:control: INFO: Starting control thread.
2022/01/19 11:56:29 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...
2022/01/19 11:56:29 wazuh-modulesd:task-manager: INFO: (8200): Module Task Manager started.
2022/01/19 11:56:29 wazuh-modulesd:syscollector: INFO: Module started.
2022/01/19 11:56:29 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2022/01/19 11:56:29 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/cis_debian10.yml'
2022/01/19 11:56:29 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2022/01/19 11:56:29 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.
2022/01/19 11:56:30 wazuh-maild: CRITICAL: (1501): Invalid SMTP Server: smtp.example.wazuh.com
2022/01/19 11:56:55 rootcheck: INFO: Ending rootcheck scan.
2022/01/19 11:57:03 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_debian10.yml'
2022/01/19 11:57:03 sca: INFO: Security Configuration Assessment scan finished. Duration: 34 seconds.
2022/01/19 11:57:18 wazuh-analysisd: ERROR: (1231): Invalid NULL element in the configuration.
2022/01/19 11:57:18 wazuh-analysisd: CRITICAL: (1202): Configuration error at 'etc/decoders/0600-SentinelOne-decoder.xml'.
2022/01/19 11:57:52 wazuh-analysisd: ERROR: (1231): Invalid NULL element in the configuration.
2022/01/19 11:57:52 wazuh-analysisd: CRITICAL: (1202): Configuration error at 'etc/decoders/0600-SentinelOne-decoder.xml'.
2022/01/19 12:45:03 wazuh-analysisd[590785] debug_op.c:70 at _log(): DEBUG: Logging module auto-initialized
2022/01/19 12:45:03 wazuh-analysisd[590785] analysisd.c:375 at main(): DEBUG: Wazuh home directory: /var/ossec
2022/01/19 12:45:03 wazuh-analysisd[590785] analysisd.c:390 at main(): DEBUG: Found user/group ...
2022/01/19 12:45:03 wazuh-analysisd[590785] analysisd.c:397 at main(): DEBUG: Active response initialized ...
Hi! hope you’re doing well.
There’re a few things to review here:
0600-SentinelOne-decoder.xml file is not properly formatted as we can see in the ossec.logs:2022/01/19 11:57:18 wazuh-analysisd: ERROR: (1231): Invalid NULL element in the configuration.
2022/01/19 11:57:18 wazuh-analysisd: CRITICAL: (1202): Configuration error at 'etc/decoders/0600-SentinelOne-decoder.xml'.
2022/01/19 11:57:52 wazuh-analysisd: ERROR: (1231): Invalid NULL element in the configuration.
2022/01/19 11:57:52 wazuh-analysisd: CRITICAL: (1202): Configuration error at 'etc/decoders/0600-SentinelOne-decoder.xml'.
This file should be reviewed.
0600-SentinelOne-decoder.xml that you’re uploading to etc/rules via the API is a group of rules for the decoder previously mentioned.2022/01/19 11:18:58 INFO: wazuh-wui 127.0.0.1 "PUT /rules/files/0600-SentinelOne-decoder.xml" with parameters {"overwrite": "false"} and body {} done in 0.008s: 200
timeout_api_exe from /var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.2.5-py3.9.egg/wazuh/core/cluster/cluster.json.Please notice you can find it with:
# find -name 'cluster.json'
./framework/wazuh/core/cluster/cluster.json
./framework/python/lib/python3.9/site-packages/wazuh-4.2.5-py3.9.egg/wazuh/core/cluster/cluster.json
Regards,
Alexis
Hi again,
Does adding the custom rules to ruleset/rules make any difference when it comes to validation and performance ?
Yes, indeed. All custom rules and decoders will be validated in the cluster validation process.
The issue is fixed after setting the timeout values on all the nodes.
Glad to hear that!
Regards,
and thanks for using Wazuh!
