Status Code: 500, Error: Internal Server Error -- Received error after reboot

[SirNerd]

Hello,

I need some help with figuring out why I received this error when I restarted the group of Linux servers for our cluster server.  I was working with a security computer and they set up everything and didn't give me any documentation.  I have 3 Index servers, 3 Manager servers, and Dashboard. There is a Load Balancer Server from the looks of it I don't think that is being used.  I patched all these servers and restarted them. After they came back up I received this error. 

{
    "statusCode": 500,
    "error": "Internal Server Error",
    "message": "An internal server error occurred."
}

Server OS: Ubuntu 22.04.3

Is there a specific reboot sequence that needs to be followed?

[Ian Yenien Serrano]

Hi SirNerd, I understand that you had an error: "Internal Server Error", that error was just after lifting reboot the system and then it worked?

If so, when all the components are restarted it takes a while to finish running everything so that it is in normal working order.

[MatV]

Hello Ian,

We were down for multiple days.  It looks like you have start the index and manager servers first. Then start the load balancer and dashboard after. 

[Ian Yenien Serrano]

Wazuh dashboard needs Wazuh indexer and the manager to be running correctly. If Wazuh indexer is not working or not finished starting Wazuh dashboard usually gives the error you mention.

[MatV]

Hi Ian,

Is there a way to test and make sure the indexers are working correctly?

[Ian Yenien Serrano]

Hi MatV,

Hi if you want to test that the wazuh indexer is working you can try these 2 commands if they are running

filebeat test output

systemctl status wazuh-indexer

[MatV]

Hi Ian,

The command filebeat comes back with command not found.  The services are running fine on all 3 of my index servers. Now, I can't get the dashboard service to start again.  Are there logs that I can get you to help with troubleshooting?  If so how do I get those logs.

Thank you,

[Ian Yenien Serrano]

About the command not found, did you install filebeat as the documentation says?

https://documentation.wazuh.com/current/installation-guide/wazuh-server/step-by-step.html#installing-filebeat

I am sharing the wazuh dashboard troubleshooting documentation for you to see

https://documentation.wazuh.com/current/user-manual/wazuh-dashboard/troubleshooting.html

[MatV]

Hi Ian, 

I found were the filebeat was installed.  I'm not sure if it's failing on the second indexer server, or if it trying to communicate to the third indexer. 

Here are the results:

elasticsearch: https://<indexerServer1>.pueblochc.org:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: x.x.x.203
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
elasticsearch: https:// <indexerServer2>  .pueblochc.org:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: x.x.x.153
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 429 Too Many Requests: {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1024580208/977.1mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1024580208/977.1mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]","bytes_wanted":1024580208,"bytes_limit":1020054732,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1024580208/977.1mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1024580208/977.1mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]","bytes_wanted":1024580208,"bytes_limit":1020054732,"durability":"TRANSIENT"},"status":429}

[Ian Yenien Serrano]

It seems like you are hitting a circuit breaking exception due to the data size limit being exceeded. You can try increasing the limit by modifying the OpenSearch configuration. What are the main specs of the servers?

You can change the total heap space via /etc/wazuh-indexer/jvm.options file. Here is an example of an increase from ~ 2GB to 4GB:
# -Xms1931m
# -Xmx1931m
-Xms4g
-Xmx4g
You can check if that works. Also there is a nice guide about how to tune your environment to avoid these exceptions here.

[MatV]

Hello Ian,

I did increase those and that seem to help with the issue and now I am about to get the Dashboard to startup again. 

Thank you again for all your help.  I really appreciate all your help. 

Thank care,

Mat