Rolling Restart an Elasticsearch Cluster with Ansible

[Lance A. Brown]

I've come up with what I think is a safe way to rolling restart an
Elasticsearch cluster using Ansible handlers.

Why is this needed?

Even if you use a serial setting to limit the number of nodes processed
at one time, Ansible will restart elasticsearch nodes and continue
processing as soon as the elasticsearch service restart reports itself
complete. This pushes the cluster into a red state due to muliple data
nodes being restarted at once, and can cause performance problems.

Solution:

Perform a rolling restart of each elasticsearch node and wait for the
cluster to stabilize before continuing processing nodes. This set of
chained handlers restarts each node while keeping the cluster from
thrashing on reallocating shards during the process.

A gist of the handlers/main.yml file for my Elasticsearch role is at
https://gist.github.com/labrown/5341ebec47bfba6dd7d4

I welcome any comments and/or suggestions.

--[Lance]

[Brian Coca]

check the wait_for module, it was created for these situations.

[Lance A. Brown]

Brian Coca said the following on 9/17/2014 10:22 PM:

> check the wait_for module, it was created for these situations.

How do you make wait_for wait on an URL? I know you can have it wait on
a socket to come up and look for data on the port connection, but I need
more than that to ensure the cluster is in the correct state before
proceeding.

--[Lance]

--
GPG Fingerprint: 409B A409 A38D 92BF 15D9 6EEE 9A82 F2AC 69AC 07B9
CACert.org Assurer

[Brian Coca]

well, in that case I would use wait_for to let the server come up and
then get_url inside a do-until loop to get the 'green light'.


--
Brian Coca
Stultorum infinitus est numerus
0110000101110010011001010110111000100111011101000010000001111001011011110111010100100000011100110110110101100001011100100111010000100001
Pedo mellon a minno

[Ludovic Petetin]

Hi,

I'm operating a Solr Cluster of 50 servers and I have been confronted to the same question than you. The warmup of the Solr index can take up to 30 seconds and wait_for on the port was not enough.
At first, I wrote a new module to launch a query every X seconds and expect a result (wait_for_result command='wget -O - "http://{{ ansible_fqdn }}:{{ active_country_port[country].port }}/searchsolrnode{{ country }}/{{ country}}/select?q=*:*&rows=0"' result="numFound" delay=10 timeout=600) until Ansible came up with it's do...until loop which is perfect in this case :
    - shell: wget -O - "http://{{ ansible_fqdn }}:{{ active_country_port[country].port }}/searchsolrnode{{ country }}/{{ country}}/select?q=*:*&rows=0"
      register: result
      until: result.stdout.find("numFound") != -1
      retries: 20
      delay: 5

[Lance A. Brown]

On 2014-09-17 11:06 pm, Brian Coca wrote:
> well, in that case I would use wait_for to let the server come up and
> then get_url inside a do-until loop to get the 'green light'.

I had that configuration initially after I realized wait_for would not
do what I needed. Then I realized setting a timeout on the curl call
(-m 2) gets me effectively the same thing. If the node isn't back up
yet, the curl call times out and the grep fails. No need to run two
tasks.

--[Lance]

- name: wait node {{ansible_hostname}}-{{service}}
local_action: "shell curl -s -m 2 'localhost:9200/_cat/nodes?h=name' |
tr -d ' ' | grep -E '^{{ansible_hostname}}-{{service}}$' "
register: result
until: result.rc == 0
retries: 200
delay: 3
notify:
- cluster routing all {{ansible_hostname}}-{{service}}

[James Martin]

I think a more "ansiblish" approach would be:

  - name: wait node {{ansible_hostname}}-{{service}} 

    uri: url=http://localhost:9200/_cat/nodes?h=name

         timeout=120

         return_content=yes

    register: result

    until: (ansible_hostname + '-' + service) in result.content

    retries: 200 

    delay: 3

[Lance A. Brown]

On 2014-09-18 10:05 am, James Martin wrote:
> I think a more "ansiblish" approach would be:
>
> - name: wait node {{ansible_hostname}}-{{service}}
> uri: url=http://localhost:9200/_cat/nodes?h=name
> timeout=120
> return_content=yes
> register: result
> until: (ansible_hostname + '-' + service) in result.content
> retries: 200
> delay: 3

Oh! I like that. Thanks!

--[Lance]

[Sam Doran]

I've written two rolling Elasticsearch playbooks: one for restarting the cluster, the other for updating the cluster. I use the feature of Ansible that will intelligently take JSON output in a register variable and make it addressable (not sure if I'm using the right terminology). Because Elasticsearch returns JSON when you query the API, you can use the registered output of the uri module and a do-until loop to wait for a certain condition before continuing. So instead of using result.stdout or result.content with a search, you can use result.json.[field] to be very precise.

    - name: Wait for cluster health to return to yellow
      uri: url=http://localhost:{{ es_http_port }}/_cluster/health method=GET
      register: response
      until: "response.json.status == 'yellow'"
      retries: 5
      delay: 30

The contents of response.json are:

ok: [node01.acme.com] => {
    "response": {
        "changed": false,
        "content_length": "227",
        "content_location": "http://localhost:9500/_cluster/health",
        "content_type": "application/json; charset=UTF-8",
        "invocation": {
            "module_args": "url=http://localhost:9500/_cluster/health method=GET",
            "module_name": "uri"
        },
        "json": {
            "active_primary_shards": 405,
            "active_shards": 810,
            "cluster_name": "elasticsearch",
            "initializing_shards": 0,
            "number_of_data_nodes": 3,
            "number_of_nodes": 4,
            "relocating_shards": 0,
            "status": "green",
            "timed_out": false,
            "unassigned_shards": 0
        },
        "redirected": false,
        "status": 200
    }
}

I'm not sure of the value of using pre and post tasks with a serial of one in my playbooks. It could all most likely be one list of tasks.

I based the playbooks on the Elasticsearch documentation for updating 1.0 and later and rolling restart.