[AtoM 2.5.1] Elasticsearch Down

[Yoo Young Lee]

Hi,

We successfully upgraded our AtoM instance to the latest version, but the Elasticsearch is down quite often.

I found your documentation and followed your instruction here - https://www.accesstomemory.org/en/docs/2.5/admin-manual/maintenance/elasticsearch/#maintenance-elasticsearch

Below is the result of the curl -XGET 'http://localhost:9200/_cluster/health/?level=shards&pretty=true'

I see the status is yellow and the number_of_nodes (1) = number_of_replicas (1) which shouldn't be according to the document. If we modify the text (as suggested in the documentation) to set the number_of_replicas to 0, this would work? I just wanted to verify with you first.

Thank you so much for your help,

Yoo Young

{

  "cluster_name" : "elasticsearch",

  "status" : "yellow",

  "timed_out" : false,

  "number_of_nodes" : 1,

  "number_of_data_nodes" : 1,

  "active_primary_shards" : 4,

  "active_shards" : 4,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 4,

  "delayed_unassigned_shards" : 0,

  "number_of_pending_tasks" : 0,

  "number_of_in_flight_fetch" : 0,

  "task_max_waiting_in_queue_millis" : 0,

  "active_shards_percent_as_number" : 50.0,

  "indices" : {

    "atom" : {

      "status" : "yellow",

      "number_of_shards" : 4,

      "number_of_replicas" : 1,

      "active_primary_shards" : 4,

      "active_shards" : 4,

      "relocating_shards" : 0,

      "initializing_shards" : 0,

      "unassigned_shards" : 4,

      "shards" : {

        "0" : {

          "status" : "yellow",

          "primary_active" : true,

          "active_shards" : 1,

          "relocating_shards" : 0,

          "initializing_shards" : 0,

          "unassigned_shards" : 1

        },

        "1" : {

          "status" : "yellow",

          "primary_active" : true,

          "active_shards" : 1,

          "relocating_shards" : 0,

          "initializing_shards" : 0,

          "unassigned_shards" : 1

        },

        "2" : {

          "status" : "yellow",

          "primary_active" : true,

          "active_shards" : 1,

          "relocating_shards" : 0,

          "initializing_shards" : 0,

          "unassigned_shards" : 1

        },

        "3" : {

          "status" : "yellow",

          "primary_active" : true,

          "active_shards" : 1,

          "relocating_shards" : 0,

          "initializing_shards" : 0,

          "unassigned_shards" : 1

        }

      }

    }

  }

}

[Dan Gillean]

Hi there, 

In the vast majority of cases, ES crashing tends to be a memory issue. If you are able to allocate more memory, this would likely help! The top answer on this StackExchange thread may offer some further insight: 

• https://stackoverflow.com/questions/15213764/elasticsearch-disappearing-crashing-cannot-find-log-files-or-any-other-info  
  

You can also use a resource monitor such as htop to view system resources  and processes in real-time while you try to replicate the issue. We have some suggestions on this in our Troubleshooting page, here: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/troubleshooting/#monitoring-active-processes-and-checking-execution-limits  
  

Additionally i should add about the yellow health status: I think it's primarily because we're only using one node in our default AtoM Elasticsearch deployment, and ES is warning you that this could involve data loss if the node goes down. According to the ES docs, "yellow means that the primary shard is allocated but replicas are not." Since ES is not our primary data store and it's easy to repopulate, I don't think this is critical.  See also:

• https://discuss.elastic.co/t/elasticsearch-health-status-is-yellow-how-to-up-green/81765
• http://chrissimpson.co.uk/elasticsearch-yellow-cluster-status-explained.html  

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6eb2753e-b1c7-4adf-9407-953bd429dfba%40googlegroups.com.

[Yoo Young Lee]

Hi Dan,

Thank you for your response! I requested to increase our VM RAM from 2GB to 7GB and just restarted. 

Best,

Yoo Young

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZKo_hSOBKSYid11ys6BL-D9OdxEq8n-2za-kE4k_Au2bw%40mail.gmail.com.

[Yoo Young Lee]

Hi Dan,

Since we've increased the RAM memory, there was no issue.

Thank you for your help!

Yoo Young

[Dan Gillean]

Thanks for updating the thread and letting us know, Yoo! Glad to hear things are working now! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAM7%2B6mSQHsMTpsutAtVyLu6WRnYD6Nu0eKdJ__LH%3DCKiszbYjQ%40mail.gmail.com.