internal server error {statusCode 500}

[siddha...@gmail.com]

Hi Team,

I'm using Distributed architecture for wazuh on Ubuntu 18.04.

while browse kibana I'm getting internal server error message.

i don't have enough knowledge to resolve this issue.

so could someone help me?
Thanks a lot! Really appreciate your help!

[Victor Moreno Jimenez]

Hi Siddharth, Seems like your Elasticsearch is running out of memory. Please check that the host where Elasticsearch installed has enough RAM to run Elasticsearch. If that doesn't solve your issue please share with us the following information to reproduce your issue:

• Wazuh version.

• Steps you followed to install Wazuh.

• Hardware specs of Wazuh, Elasticsearch and Kibana nodes.

• Kibana/Elasticsearch logs. Check your system logs.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/422ff9c2-b969-4620-96d2-4305fb53fd86n%40googlegroups.com.

[siddharth jha]

I have checked and increased RAM and same error i can see and i refresh the browser after that i can see normal interface but few after moments i received same error again refresh its work again normally and after that when i check my agent events where i can see new error "elastic search internal server error".i have attached error's screenshot.

• Wazuh version.---i'm using wazuh 3.13.

• Steps you followed to install Wazuh--- i followed below link install and i used distributed architecture.

https://documentation.wazuh.com/3.13/installation-guide/index.html

• Hardware specs of Wazuh, Elasticsearch and Kibana node

wazuh server- 4vcpu,8GB RAM,300 HDD and ubuntu 18.04 and elastic server configuration is the same as it is only i increased RAM now its 16 GB.

• Kibana/Elasticsearch logs. Check your system logs. --could you please help me to check this.

[Victor Moreno Jimenez]

Hi siddharth jha,

First of all, I would recommend using the latest Wazuh version https://documentation.wazuh.com/4.0/installation-guide/index.html instead of `3.13`.

If you don't have any special interest in using 3.13 you could follow the distributed guide for Wazuh `4.x` which works like a charm.

Anyways, if you still want to test Wazuh `3.13` I would need the following information to debug your issue:

- Kibana logs: Kibana logs to system logs, in your case Syslog: `cat /var/log/syslog | grep -i kibana`

- Elasticsearch logs: you could find Elasticserach logs in Syslog as well: `cat /var/log/syslog | grep -i elastic`

- Wazuh logs: in your manager `cat /var/ossec/logs/ossec.log`.

I'm waiting for your answer.

Hope it helps! :)

Regards,

Víctor.

[siddharth jha]

Hi Victor,

Thanks for your reply and as you suggested I have also checked the latest version of wazuh but i thought i should first try to resolve this issue which i'm facing currently.

I don't have any personal interest in this version. I just love to learn and apply new things :)

and i try to check logs from your given cmd but i'm unable to do that because when i run that command there are so many logs which i can't read.

please suggest.

[victor...@wazuh.com]

Hi siddharth jha,
In order to help you, please share with us the content of the following commands:

- In your manager node:  `journalctl -u wazuh-manager.service -xe`. If you have cluster setup, in your master node.

- In your Elasticsearch node: `journalctl -u elasticsearch.service -xe`

- In your Kibana node: `journalctl -u kibana.service -xe`

- In your manager node `cat /var/ossec/logs/ossec.log`. If you have cluster setup, in your master node.

I attach example of Kibana output.

I'm waiting for your response.

[victor...@wazuh.com]

Hi siddharth jha,
Last message was unformatted and the attached file is empty. Sorry about that, I attach an example file again and repeat the message:

Hi siddharth jha,
In order to help you, please share with us the content of the following commands:

• In your manager node: journalctl -u wazuh-manager.service -xe. If you have cluster setup, in your master node.

• In your Elasticsearch node: journalctl -u elasticsearch.service -xe

• In your Kibana node: journalctl -u kibana.service -xe

• In your manager node cat /var/ossec/logs/ossec.log. If you have cluster setup, in your master node.

I attach example of Kibana output.

I’m waiting for your response.

​

[siddha...@gmail.com]

Hi victor,

Thank you very much for your time and support.

please find attached required logs.

kindly check and suggest.

[siddha...@gmail.com]

Hi victor,

could you suggest something on this.

[victor...@wazuh.com]

Hi siddharth jha,
First of all sorry for the late reply.
I’m afraid that I tried to replicate your issue without success.
Service logs that you shared with us do not give a clear clue about the error. I’ll suggest you re-install your distributed environment with the new
Wazuh 4.1.0 that we just deployed! If you still have any error in the deployment process, don’t hesitate to ask again.

Here is the distributed installation guide for 4.1.0 https://documentation.wazuh.com/current/installation-guide/open-distro/distributed-deployment/index.html#distributed-index

Regards,
Víctor.

​

[siddha...@gmail.com]

Hi Victor,

Thank you for your kind support.

i have some query's please suggest.

1. do i need to re-install on other vm( server) .

2. i want to install fresh wazuh 4.1.0 as all in one deployment method on other vm and on that vm can i get all the information and alerts which is stored in current  distributed environment ?

 please suggest.

I also checked wazuh manager , file beat , Elasticsearch and kibana services everything showing up and running.

but when I try to check alerts in Elasticsearch by below cmd I'm getting error 

curl 10.12.186.71:9200/_cat/indices/wazuh-alerts-3.x-*

{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1022432624/975mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1022432624/975mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, accounting=82932920/79mb]","bytes_wanted":1022432624,"bytes_limit":1020054732,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1022432624/975mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1022432624/975mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b, model_inference=0/0b, accounting=82932920/79mb]","bytes_wanted":1022432624,"bytes_limit":1020054732,"durability":"PERMANENT"},"status":429}root@wazuhElastic:~# Restart-Service -Name wazuh

  

please suggest 

Thank you.

[victor...@wazuh.com]

Hi siddharth,

You could backup your data via snapshot and then restore it in your new environment, I’ll show you an example:

You can create and restore snapshots both by using the Elasticsearch API or directly from the Kibana web interface (not in OpenDistro). For this example, we will use the API because it will work with all installations. First, we start creating a repository for the snapshots, for simplicity, we will use a local repository. Start by creating the directory where you want to store the snapshots, for example, /mount/elasticsearch backup:

mkdir -p /mount/elasticsearch backup
chown elasticsearch: /mount/elasticsearch_backup/

And add this line to /etc/elasticsearch/elasticsearch.yml:

path.repo: ["/mount/elasticsearch_backup"]

And restart Elasticsearch. Now, add the repository to Elasticsearch by using the following API call, I am using the Kibana Dev Tools console but you can also use curl for the API calls:

PUT /_snapshot/elasticsearch_backup
{
  "type": "fs",
  "settings": {
    "delegate_type": "fs",
    "location": "/mount/elasticsearch_backup",
    "compress": true
  }
}

Then you can take a snapshot with the following call:

PUT /_snapshot/elasticsearch_backup/new_snapshot

Then you can take a snapshot with the following call:

PUT /_snapshot/elasticsearch_backup/new_snapshot2
{
  "indices": "wazuh-alerts*",
}

You can see the snapshots in your repository by running:

GET _snapshot/elasticsearch_backup/_all

Finally, to restore the snapshot in your new environment, you'll have to export the snapshot and the content in /mount/elasticsearch_backup and import it to your new environment.

POST _snapshot/my-repository/2/new_snapshot2

Here you have a couple of documents with more information about Elasticsearch snapshots.
Wazuh blog about snapshot management, here you can find how to configure Elasticsearch to periodically store snapshots and how to set your repository in different cloud services: https://wazuh.com/blog/index-backup-management/ OpenDistro documentation page about snapshots: https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/snapshot-restore/

About your issue with Elasticsearch, seems like your Elasticsearch is running out of ram. I’ll suggest you increase the heap size as Elasticsearch forums suggest:

• https://discuss.elastic.co/t/cause-and-how-to-avoid-data-too-large-http-request-exception/156279
• https://stackoverflow.com/questions/61870751/circuit-breaking-exception-parent-data-too-large-data-for-http-request

Hope it helps

​

[siddha...@gmail.com]

Hi victor,

Thanks for sharing these details,

About  issue with Elasticsearch  

i have check your shared articles but couldn't implement right way.

i have added 6 GB to this path -- etc\elasticsearch\jvm.option  please suggest is it right or not.

[Victor Moreno Jimenez]

Hi Siddharth,

If you want to increase the JVM heap, remember that the min and max value should be the same value.

In order to do that add the following lines to your /etc/elasticsearch/jvm.options:

-Xms6g
-Xmx6g

After that restart Elasticsearch:

systemctl restart elasticsearch

Monitor your node to check if the issue still happening.

Hope it helps!

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/8b444b09-db6f-49ae-abb4-6074ed27ce3cn%40googlegroups.com.