I'm running a setup consisting of a load balancer, dashboard server, master, 2 workers and 3 data nodes on ubuntu 22.04 hosts. Everything has been running fine until this morning!
There were a lot of updates queued for each server so I ran the updates. I noticed it changed the opensearch.yml and jvm.options on the data nodes during the update and had to tweak things to get the data cluster back working.
Once the data cluster was back operating, running filebeat test output show on the master gives ok results across all servers
Restarting the dashboard, I get the server not ready message in the web interface.
checking the logs shows me the following after restarting the dashboard service
Mar 29 10:59:36 wazdash opensearch-dashboards[2176]: {"type":"log","@timestamp":"2023-03-29T07:59:36Z","tags":["info","plugins-service"],"pid":2176,"message":"Plugin \"dataSourceManagement\" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"}
Mar 29 10:59:36 wazdash opensearch-dashboards[2176]: {"type":"log","@timestamp":"2023-03-29T07:59:36Z","tags":["info","plugins-service"],"pid":2176,"message":"Plugin \"dataSource\" is disabled."}
Mar 29 10:59:36 wazdash opensearch-dashboards[2176]: {"type":"log","@timestamp":"2023-03-29T07:59:36Z","tags":["info","plugins-service"],"pid":2176,"message":"Plugin \"visTypeXy\" is disabled."}
Mar 29 10:59:36 wazdash opensearch-dashboards[2176]: {"type":"log","@timestamp":"2023-03-29T07:59:36Z","tags":["info","plugins-system"],"pid":2176,"message":"Setting up [45] plugins: [**redacted**]"}
Mar 29 10:59:37 wazdash opensearch-dashboards[2176]: {"type":"log","@timestamp":"2023-03-29T07:59:37Z","tags":["info","savedobjects-service"],"pid":2176,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
Mar 29 10:59:37 wazdash opensearch-dashboards[2176]: {"type":"log","@timestamp":"2023-03-29T07:59:37Z","tags":["error","opensearch","data"],"pid":2176,"message":"[ConnectionError]: connect ECONNREFUSED
127.0.0.1:9200"}
Mar 29 10:59:37 wazdash opensearch-dashboards[2176]: {"type":"log","@timestamp":"2023-03-29T07:59:37Z","tags":["error","savedobjects-service"],"pid":2176,"message":"Unable to retrieve version information from OpenSearch nodes."}
Mar 29 10:59:40 wazdash opensearch-dashboards[2176]: {"type":"log","@timestamp":"2023-03-29T07:59:40Z","tags":["error","opensearch","data"],"pid":2176,"message":"[ConnectionError]: connect ECONNREFUSED
127.0.0.1:9200"}
I'm a bit baffled now as to how to resolve this!
Also as a side issue, when I check the status of the data node cluster, the ram.percent of one of the nodes is at 98%. Where do I alter the available RAM?
Thanks in advance