Too Many Requests

637 views
Skip to first unread message

Juan Ferdinan

unread,
Dec 25, 2022, 11:58:56 PM12/25/22
to Wazuh mailing list
Hi Wazuh Teams

This morning I had trouble opening my wazuh dashboard and when I could open it, I got an error message like the one below
Screen Shot 2022-12-26 at 11.54.48.png

 Why is this happening, is there a solution to the problem?


Thanks & Regrads
Juan

Gustavo Choquevilca

unread,
Dec 26, 2022, 8:04:39 AM12/26/22
to Wazuh mailing list
Hello, thank you very much for using Wazuh!
To know more information about this problem, you can share with me some logs like the ones described below:

Indexer/elasticsearch
  • cat /var/log/wazuh-indexer/<elasticsearch-cluster-name>.log | grep -iE "error|warn"
Wazuh manager
  • grep -iE 'ERROR' /var/ossec/logs/api.log
  • grep -iE 'ERROR' /var/ossec/logs/cluster.log
  • grep -iE 'ERROR' /var/ossec/logs/ossec.log
 Kibana/dashboard
  • grep -iE 'WARN|ERROR' /var/log/kibana/kibana.log
        or
  • journalctl -u wazuh-dashboard
  • cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

Juan Ferdinan

unread,
Dec 28, 2022, 9:40:49 PM12/28/22
to Wazuh mailing list
Hi Gustavo

Thanks for helping me, here are the results

1. # cat /var/log/wazuh-indexer/<elasticsearch-cluster-name>.log | grep -iE "error|warn"
bash: elasticsearch-cluster-name: No such file or directory

2. # grep -iE 'ERROR' /var/ossec/logs/api.log
No result

3. # grep -iE 'ERROR' /var/ossec/logs/cluster.log
No result

4. # grep -iE 'ERROR' /var/ossec/logs/ossec.log
2022/12/29 00:00:10 wazuh-integratord: ERROR: Unable to run integration for custom-discord -> integrations
2022/12/29 00:00:10 wazuh-integratord: ERROR: While running custom-discord -> integrations. Output: KeyError: 'ip'
2022/12/29 00:00:10 wazuh-integratord: ERROR: Exit status was: 1

5. # grep -iE 'WARN|ERROR' /var/log/kibana/kibana.log
grep: /var/log/kibana/kibana.log: No such file or directory

6. # journalctl -u wazuh-dashboard
-- Logs begin at Mon 2022-11-28 17:48:58 WIB, end at Thu 2022-12-29 09:37:07 WIB. --
Nov 29 09:34:25 xxx opensearch-dashboards[5210]: {"type":"log","@timestamp":"2022-11-29T02:34:25Z","tags":["info","branding"],"pid":5210,"messa>
Nov 29 09:34:25 xxx opensearch-dashboards[5210]: {"type":"log","@timestamp":"2022-11-29T02:34:25Z","tags":["info","branding"],"pid":5210,"messa>

7. # cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"
{"date":"2022-08-12T03:18:12.750Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 401"}
{"date":"2022-08-15T10:02:30.487Z","level":"error","location":"queue:delayApiRequest","message":"An error ocurred in the delayed request: \"DELETE /security/user/authenticate\": Request failed with status code 401"}
{"date":"2022-08-30T03:50:00.688Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/remoted?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-08-30T03:50:00.715Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/analysisd?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-08-30T06:54:30.777Z","level":"error","location":"queue:delayApiRequest","message":"An error ocurred in the delayed request: \"DELETE /security/user/authenticate\": Request failed with status code 401"}
{"date":"2022-08-30T09:51:32.048Z","level":"error","location":"wazuh-api:makeRequest","data":{"title":"Wazuh Internal Error","detail":"Timeout executing API request","dapi_errors":{"node01":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"error":3021}}
{"date":"2022-09-01T00:08:00.976Z","level":"error","location":"queue:delayApiRequest","message":"An error ocurred in the delayed request: \"DELETE /security/user/authenticate\": Request failed with status code 401"}
{"date":"2022-09-01T06:41:19.760Z","level":"error","location":"wazuh-api:makeRequest","data":{"title":"Wazuh Internal Error","detail":"Timeout executing API request","dapi_errors":{"node01":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"error":3021}}
{"date":"2022-09-01T06:41:25.244Z","level":"error","location":"wazuh-api:makeRequest","data":{"title":"Wazuh Internal Error","detail":"Timeout executing API request","dapi_errors":{"node01":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"error":3021}}
{"date":"2022-09-01T06:41:25.252Z","level":"error","location":"wazuh-api:makeRequest","data":{"title":"Wazuh Internal Error","detail":"Timeout executing API request","dapi_errors":{"node01":{"error":"Timeout executing API request","logfile":"WAZUH_HOME/logs/api.log"}},"error":3021}}
{"date":"2022-09-01T06:56:27.998Z","level":"error","location":"reporting:extendedInformation","message":"Request failed with status code 401"}
{"date":"2022-09-05T08:29:58.393Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 127.0.0.1:55000"}
{"date":"2022-09-05T08:30:00.748Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/remoted?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T08:30:00.750Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/analysisd?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T08:30:00.751Z","level":"error","location":"monitoring:getApiInfo","message":"connect ECONNREFUSED 127.0.0.1:55000"}
{"date":"2022-09-05T08:35:00.365Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/remoted?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T08:35:00.367Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/analysisd?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T08:40:00.620Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/remoted?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T08:40:00.622Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/analysisd?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T08:45:00.885Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/remoted?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T08:45:00.887Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/analysisd?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T08:45:00.888Z","level":"error","location":"monitoring:getApiInfo","message":"connect ECONNREFUSED 127.0.0.1:55000"}
{"date":"2022-09-05T11:20:00.953Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/remoted?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-05T11:20:00.954Z","level":"info","location":"Cron-scheduler","data":{"message":"connect ECONNREFUSED 127.0.0.1:55000","stack":"Error: connect ECONNREFUSED 127.0.0.1:55000\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)","config":{"url":"https://localhost:55000/manager/stats/analysisd?pretty","method":"get","data":"{}","params":{}}}}
{"date":"2022-09-06T04:07:15.970Z","level":"error","location":"queue:delayApiRequest","message":"An error ocurred in the delayed request: \"DELETE /security/user/authenticate\": Request failed with status code 401"}

Thanks & Regards
Juan
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted

Juan Ferdinan

unread,
Jan 1, 2023, 11:01:54 PM1/1/23
to Gustavo Choquevilca, Wazuh mailing list
Hi Gustavo

Here are the results :

#systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-12-26 17:14:31 WIB; 6 days ago
       Docs: https://documentation.wazuh.com
   Main PID: 7415 (java)
      Tasks: 188 (limit: 9443)
     Memory: 2.8G
     CGroup: /system.slice/wazuh-indexer.service
             └─7415 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.hea>

Dec 26 17:13:53 xxx systemd[1]: wazuh-indexer.service: Succeeded.
Dec 26 17:13:53 xxx systemd[1]: Stopped Wazuh-indexer.
Dec 26 17:13:53 xxx systemd[1]: Starting Wazuh-indexer...
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: An illegal reflective access operation has occurred
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-detectio>
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: All illegal access operations will be denied in a future release
Dec 26 17:14:31 xxx systemd[1]: Started Wazuh-indexer.

# service wazuh-indexer status
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-12-26 17:14:31 WIB; 6 days ago
       Docs: https://documentation.wazuh.com
   Main PID: 7415 (java)
      Tasks: 188 (limit: 9443)
     Memory: 2.8G
     CGroup: /system.slice/wazuh-indexer.service
             └─7415 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.hea>

Dec 26 17:13:53 xxx systemd[1]: wazuh-indexer.service: Succeeded.
Dec 26 17:13:53 xxx systemd[1]: Stopped Wazuh-indexer.
Dec 26 17:13:53 xxx systemd[1]: Starting Wazuh-indexer...
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: An illegal reflective access operation has occurred
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-detectio>
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Dec 26 17:14:22 xxx systemd-entrypoint[7415]: WARNING: All illegal access operations will be denied in a future release
Dec 26 17:14:31 xxx systemd[1]: Started Wazuh-indexer.

# cat /var/log/wazuh-indexer/wazuh-indexer-cluster.log | grep -iE "error|warn"
cat: /var/log/wazuh-indexer/wazuh-indexer-cluster.log: No such file or directory

# systemctl status elasticarch
Unit elasticarch.service could not be found.

# service elasticsearch status
Unit elasticsearch.service could not be found.


Thanks & Regards
Randy


On Fri, Dec 30, 2022 at 3:55 PM 'Gustavo Choquevilca' via Wazuh mailing list <wa...@googlegroups.com> wrote:
This error may be caused by a problem with the wazuh-indexer/elascticsearch connection.
To check the status of the service, you can execute the following:
  • systemctl status elasticarch
  • service elasticsearch status
or,
  • systemctl status wazuh-indexer
  • service wazuh-indexer status

On the other hand, you can see more details about the error in the wazuh-indexer logs in the path /var/log/wazuh-indexer/:

cat /var/log/wazuh-indexer/wazuh-indexer-cluster.log | grep -iE "error|warn"
--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/xtrbhotFZ08/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/8a88c31d-3149-4448-9000-7de8e977dbe0n%40googlegroups.com.
Message has been deleted
Message has been deleted
Message has been deleted

Juan Ferdinan

unread,
Jan 4, 2023, 4:12:35 PM1/4/23
to Gustavo Choquevilca, Wazuh mailing list
Hi Gustavo

Why is every reply message from you always deleted?
Here are the results that I have done based on what you told me to do

1. curl https://xxx:9200/_cat/indices/wazuh-alerts-* -u xxx:xxx -k
curl: (7) Failed to connect to xxx port 9200: Connection refused

2. # filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 7.10.2

3. from one of the lists below, which one is included in the wazuh-indexer logs?
Screen Shot 2023-01-04 at 14.52.55.png



Thanks & Regards
Randy


On Wed, Jan 4, 2023 at 2:31 AM 'Gustavo Choquevilca' via Wazuh mailing list <wa...@googlegroups.com> wrote:
Yesterday I sent an answer for you but I think my answer has been deleted, here I send it again:
Checking the wazuh-dashboard logs this error may be related to a problem with the wazuh-indexer connection.
You can check the status as mentioned here.

I also see that you couldn't find the wazuh-indexer logs, they are located in this path: /var/log/wazuh-indexer

On Wednesday, December 28, 2022 at 11:40:49 PM UTC-3 juan.f...@ottodigital.id wrote:

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/xtrbhotFZ08/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/01af9d15-5cf7-4ed4-b6b8-db8eaa7ea501n%40googlegroups.com.
Message has been deleted
Message has been deleted

Gustavo Choquevilca

unread,
Jan 5, 2023, 8:28:25 AM1/5/23
to Wazuh mailing list
My messages are being deleted by the spam filter apparently, it's very rare!

The wazuh-indexer log is: wazuh-cluster.log
Reply all
Reply to author
Forward
0 new messages