Problem with curator to elastic search connection after Search Guard installation

388 views

Skip to first unread message

LS

unread,

Jan 18, 2018, 6:33:00 AM1/18/18

to Search Guard Community Forum

Hi,

I use curator to remove old indexes and after Search Guard installation it's not working. It seems that there is a problem with SSL connection. I'm pretty sure that I configured something incorrectly but I wasn't able to find what so I will appreciate any help.

My system info:

- system Ubuntu 16.04.3 LTS

- JVM - 1.8.0_151

- Search Guard - 5.4.1

- Elasticsearch 6.1.1

- I don't use any enterprise modules or any additional plugins

My curator.yml file looks like this:

client:

hosts:

- 127.0.0.1

port: 9200

url_prefix:

use_ssl: True

certificate: /home/ls/.curator/root-ca.pem

client_cert:

client_key:

ssl_no_validate: False

http_auth: admin:admin

timeout: 30

master_only: False

logging:

loglevel: DEBUG

logfile: /home/ls/.curator/debug.log

logformat: default

blacklist: ['elasticsearch', 'urllib3']

Config file:

---
actions:
  1:
    action: delete_indices
    description: 
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: a-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:
  2:
    action: delete_indices
    description: 
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: b
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

When running curator I get the following output:

ls@elastic:~/.curator$ curator /home/ls/.curator/delete_index.yml --config /home/ls/.curator/curator.yml
Traceback (most recent call last):
  File "/usr/local/bin/curator", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/curator/cli.py", line 211, in cli
    run(config, action_file, dry_run)
  File "/usr/local/lib/python2.7/dist-packages/curator/cli.py", line 158, in run
    client = get_client(**client_args)
  File "/usr/local/lib/python2.7/dist-packages/curator/utils.py", line 903, in get_client
    'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: ConnectionError(error return without exception set) caused by: SystemError(error return without exception set)

and output in the log file:

2018-01-18 10:01:19,811 DEBUG                curator.cli                    run:108  Client and logging options validated.
2018-01-18 10:01:19,811 DEBUG                curator.cli                    run:112  default_timeout = 30
2018-01-18 10:01:19,822 DEBUG                curator.cli                    run:116  action_file: /home/ls/.curator/delete_index.yml
2018-01-18 10:01:19,848 DEBUG                curator.cli                    run:118  action_config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}, 2: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}}}
2018-01-18 10:01:19,849 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <type 'dict'>}
2018-01-18 10:01:19,850 DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}, 2: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}}}
2018-01-18 10:01:19,850 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])])}
2018-01-18 10:01:19,851 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}
2018-01-18 10:01:19,852 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])]), 'description': Any([<type 'str'>, <type 'unicode'>]), 'filters': <type 'list'>, 'options': <type 'dict'>}
2018-01-18 10:01:19,852 DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}
2018-01-18 10:01:19,866 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'continue_if_exception': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c27429230>, msg=None)]), 'timeout_override': Any([Coerce(int, msg=None), None]), 'ignore_empty_list': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c27429aa0>, msg=None)]), 'disable_action': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c27429668>, msg=None)])}
2018-01-18 10:01:19,866 DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'continue_if_exception': False, 'ignore_empty_list': True, 'disable_action': False}
2018-01-18 10:01:19,867 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function f at 0x7f0c2741bb18>
2018-01-18 10:01:19,867 DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}]
2018-01-18 10:01:19,868 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c2741b488>, msg=None)]), 'kind': Any(['prefix', 'suffix', 'timestring', 'regex']), 'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'value': Any([<type 'str'>, <type 'unicode'>])}
2018-01-18 10:01:19,868 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}
2018-01-18 10:01:19,869 DEBUG     curator.validators.filters                      f:48   Filter #0: {'exclude': False, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}
2018-01-18 10:01:19,870 DEBUG     curator.defaults.filtertypes                    age:56   AGE FILTER = [{'direction': Any(['older', 'younger'])}, {'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years'])}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any([<type 'str'>, <type 'unicode'>])}, {'epoch': Any([Coerce(int, msg=None), None])}, {'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c2743f0c8>, msg=None)])}, {'source': Any(['name', 'creation_date', 'field_stats'])}, {'stats_result': Any(['min_value', 'max_value'])}, {'timestring': Any([<type 'str'>, <type 'unicode'>])}]
2018-01-18 10:01:19,871 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'source': Any(['name', 'creation_date', 'field_stats']), 'direction': Any(['older', 'younger']), 'stats_result': Any(['min_value', 'max_value']), 'timestring': Any([<type 'str'>, <type 'unicode'>]), 'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c2743f0c8>, msg=None)]), 'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'unit_count': Coerce(int, msg=None), 'epoch': Any([Coerce(int, msg=None), None]), 'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years']), 'unit_count_pattern': Any([<type 'str'>, <type 'unicode'>])}
2018-01-18 10:01:19,871 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'direction': 'older', 'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit_count': 7, 'unit': 'days'}
2018-01-18 10:01:19,872 DEBUG     curator.validators.filters                      f:48   Filter #1: {'direction': 'older', 'stats_result': 'min_value', 'filtertype': 'age', 'source': 'name', 'epoch': None, 'timestring': '%Y.%m.%d', 'exclude': False, 'unit_count': 7, 'unit': 'days'}
2018-01-18 10:01:19,872 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])])}
2018-01-18 10:01:19,872 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}
2018-01-18 10:01:19,873 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])]), 'description': Any([<type 'str'>, <type 'unicode'>]), 'filters': <type 'list'>, 'options': <type 'dict'>}
2018-01-18 10:01:19,874 DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}
2018-01-18 10:01:19,887 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'continue_if_exception': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c273d0050>, msg=None)]), 'timeout_override': Any([Coerce(int, msg=None), None]), 'ignore_empty_list': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c273d08c0>, msg=None)]), 'disable_action': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c273d0488>, msg=None)])}
2018-01-18 10:01:19,887 DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'continue_if_exception': False, 'ignore_empty_list': True, 'disable_action': False}
2018-01-18 10:01:19,888 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function f at 0x7f0c27444b18>
2018-01-18 10:01:19,888 DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}]
2018-01-18 10:01:19,889 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c27444488>, msg=None)]), 'kind': Any(['prefix', 'suffix', 'timestring', 'regex']), 'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'value': Any([<type 'str'>, <type 'unicode'>])}
2018-01-18 10:01:19,889 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}
2018-01-18 10:01:19,889 DEBUG     curator.validators.filters                      f:48   Filter #0: {'exclude': False, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}
2018-01-18 10:01:19,890 DEBUG     curator.defaults.filtertypes                    age:56   AGE FILTER = [{'direction': Any(['older', 'younger'])}, {'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years'])}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any([<type 'str'>, <type 'unicode'>])}, {'epoch': Any([Coerce(int, msg=None), None])}, {'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c2741bde8>, msg=None)])}, {'source': Any(['name', 'creation_date', 'field_stats'])}, {'stats_result': Any(['min_value', 'max_value'])}, {'timestring': Any([<type 'str'>, <type 'unicode'>])}]
2018-01-18 10:01:19,891 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'source': Any(['name', 'creation_date', 'field_stats']), 'direction': Any(['older', 'younger']), 'stats_result': Any(['min_value', 'max_value']), 'timestring': Any([<type 'str'>, <type 'unicode'>]), 'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f0c2741bde8>, msg=None)]), 'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'unit_count': Coerce(int, msg=None), 'epoch': Any([Coerce(int, msg=None), None]), 'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years']), 'unit_count_pattern': Any([<type 'str'>, <type 'unicode'>])}
2018-01-18 10:01:19,891 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'direction': 'older', 'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit_count': 7, 'unit': 'days'}
2018-01-18 10:01:19,892 DEBUG     curator.validators.filters                      f:48   Filter #1: {'direction': 'older', 'stats_result': 'min_value', 'filtertype': 'age', 'source': 'name', 'epoch': None, 'timestring': '%Y.%m.%d', 'exclude': False, 'unit_count': 7, 'unit': 'days'}
2018-01-18 10:01:19,892 DEBUG                curator.cli                    run:121  Full list of actions: {1: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': False, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'direction': 'older', 'stats_result': 'min_value', 'filtertype': 'age', 'source': 'name', 'epoch': None, 'timestring': '%Y.%m.%d', 'exclude': False, 'unit_count': 7, 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}, 2: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': False, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'direction': 'older', 'stats_result': 'min_value', 'filtertype': 'age', 'source': 'name', 'epoch': None, 'timestring': '%Y.%m.%d', 'exclude': False, 'unit_count': 7, 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}}
2018-01-18 10:01:19,892 DEBUG                curator.cli                    run:126  action_disabled = False
2018-01-18 10:01:19,893 DEBUG                curator.cli                    run:130  continue_if_exception = False
2018-01-18 10:01:19,893 DEBUG                curator.cli                    run:132  timeout_override = None
2018-01-18 10:01:19,893 DEBUG                curator.cli                    run:134  ignore_empty_list = True
2018-01-18 10:01:19,893 INFO                 curator.cli                    run:144  Preparing Action ID: 1, "delete_indices"
2018-01-18 10:01:19,893 DEBUG              curator.utils             get_client:803  kwargs = {'aws_secret_key': None, 'url_prefix': '', 'http_auth': 'admin:admin', 'certificate': '/home/ls/.curator/root-ca.pem', 'aws_key': None, 'aws_sign_request': False, 'port': 9200, 'hosts': ['127.0.0.1'], 'timeout': 30, 'aws_token': None, 'use_ssl': True, 'master_only': False, 'client_cert': None, 'ssl_no_validate': False, 'client_key': None}
2018-01-18 10:01:19,894 DEBUG              curator.utils             get_client:809  Attempting to verify SSL certificate.
2018-01-18 10:01:19,894 DEBUG              curator.utils             get_client:880  Not using "requests_aws4auth" python module to connect.
2018-01-18 12:05:13,800 DEBUG                curator.cli                    run:108  Client and logging options validated.
2018-01-18 12:05:13,801 DEBUG                curator.cli                    run:112  default_timeout = 30
2018-01-18 12:05:13,801 DEBUG                curator.cli                    run:116  action_file: /home/ls/.curator/delete_index.yml
2018-01-18 12:05:13,821 DEBUG                curator.cli                    run:118  action_config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}, 2: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}}}
2018-01-18 12:05:13,822 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <type 'dict'>}
2018-01-18 12:05:13,822 DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}, 2: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}}}
2018-01-18 12:05:13,823 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}
2018-01-18 12:05:13,824 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])]), 'description': Any([<type 'str'>, <type 'unicode'>]), 'filters': <type 'list'>, 'options': <type 'dict'>}
2018-01-18 12:05:13,824 DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}
2018-01-18 12:05:13,837 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'continue_if_exception': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04ba0230>, msg=None)]), 'timeout_override': Any([Coerce(int, msg=None), None]), 'ignore_empty_list': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04ba0aa0>, msg=None)]), 'disable_action': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04ba0668>, msg=None)])}
2018-01-18 12:05:13,837 DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'continue_if_exception': False, 'ignore_empty_list': True, 'disable_action': False}
2018-01-18 12:05:13,838 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function f at 0x7f5c04b92b18>
2018-01-18 12:05:13,838 DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}]
2018-01-18 12:05:13,838 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04b92488>, msg=None)]), 'kind': Any(['prefix', 'suffix', 'timestring', 'regex']), 'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'value': Any([<type 'str'>, <type 'unicode'>])}
2018-01-18 12:05:13,839 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}
2018-01-18 12:05:13,839 DEBUG     curator.validators.filters                      f:48   Filter #0: {'exclude': False, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}
2018-01-18 12:05:13,840 DEBUG     curator.defaults.filtertypes                    age:56   AGE FILTER = [{'direction': Any(['older', 'younger'])}, {'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years'])}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any([<type 'str'>, <type 'unicode'>])}, {'epoch': Any([Coerce(int, msg=None), None])}, {'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04bb60c8>, msg=None)])}, {'source': Any(['name', 'creation_date', 'field_stats'])}, {'stats_result': Any(['min_value', 'max_value'])}, {'timestring': Any([<type 'str'>, <type 'unicode'>])}]
2018-01-18 12:05:13,841 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'source': Any(['name', 'creation_date', 'field_stats']), 'direction': Any(['older', 'younger']), 'stats_result': Any(['min_value', 'max_value']), 'timestring': Any([<type 'str'>, <type 'unicode'>]), 'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04bb60c8>, msg=None)]), 'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'unit_count': Coerce(int, msg=None), 'epoch': Any([Coerce(int, msg=None), None]), 'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years']), 'unit_count_pattern': Any([<type 'str'>, <type 'unicode'>])}
2018-01-18 12:05:13,841 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'direction': 'older', 'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit_count': 7, 'unit': 'days'}
2018-01-18 12:05:13,841 DEBUG     curator.validators.filters                      f:48   Filter #1: {'direction': 'older', 'stats_result': 'min_value', 'filtertype': 'age', 'source': 'name', 'epoch': None, 'timestring': '%Y.%m.%d', 'exclude': False, 'unit_count': 7, 'unit': 'days'}
2018-01-18 12:05:13,842 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])])}
2018-01-18 12:05:13,842 DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}
2018-01-18 12:05:13,843 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any([In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot'])]), 'description': Any([<type 'str'>, <type 'unicode'>]), 'filters': <type 'list'>, 'options': <type 'dict'>}
2018-01-18 12:05:13,843 DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}
2018-01-18 12:05:13,851 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'continue_if_exception': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04b47050>, msg=None)]), 'timeout_override': Any([Coerce(int, msg=None), None]), 'ignore_empty_list': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04b478c0>, msg=None)]), 'disable_action': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04b47488>, msg=None)])}
2018-01-18 12:05:13,851 DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'continue_if_exception': False, 'ignore_empty_list': True, 'disable_action': False}
2018-01-18 12:05:13,851 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function f at 0x7f5c04bbbb18>
2018-01-18 12:05:13,851 DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'exclude': None, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'source': 'name', 'direction': 'older', 'unit_count': 7, 'timestring': '%Y.%m.%d', 'exclude': None, 'filtertype': 'age', 'unit': 'days'}]
2018-01-18 12:05:13,852 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04bbb488>, msg=None)]), 'kind': Any(['prefix', 'suffix', 'timestring', 'regex']), 'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'value': Any([<type 'str'>, <type 'unicode'>])}
2018-01-18 12:05:13,852 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}
2018-01-18 12:05:13,852 DEBUG     curator.validators.filters                      f:48   Filter #0: {'exclude': False, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}
2018-01-18 12:05:13,853 DEBUG     curator.defaults.filtertypes                    age:56   AGE FILTER = [{'direction': Any(['older', 'younger'])}, {'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years'])}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any([<type 'str'>, <type 'unicode'>])}, {'epoch': Any([Coerce(int, msg=None), None])}, {'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04b92de8>, msg=None)])}, {'source': Any(['name', 'creation_date', 'field_stats'])}, {'stats_result': Any(['min_value', 'max_value'])}, {'timestring': Any([<type 'str'>, <type 'unicode'>])}]
2018-01-18 12:05:13,853 DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'source': Any(['name', 'creation_date', 'field_stats']), 'direction': Any(['older', 'younger']), 'stats_result': Any(['min_value', 'max_value']), 'timestring': Any([<type 'str'>, <type 'unicode'>]), 'exclude': Any([<type 'bool'>, All(Any([<type 'str'>, <type 'unicode'>]), <function Boolean at 0x7f5c04b92de8>, msg=None)]), 'filtertype': Any([In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state'])]), 'unit_count': Coerce(int, msg=None), 'epoch': Any([Coerce(int, msg=None), None]), 'unit': Any(['seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years']), 'unit_count_pattern': Any([<type 'str'>, <type 'unicode'>])}
2018-01-18 12:05:13,853 DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'direction': 'older', 'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit_count': 7, 'unit': 'days'}
2018-01-18 12:05:13,853 DEBUG     curator.validators.filters                      f:48   Filter #1: {'direction': 'older', 'stats_result': 'min_value', 'filtertype': 'age', 'source': 'name', 'epoch': None, 'timestring': '%Y.%m.%d', 'exclude': False, 'unit_count': 7, 'unit': 'days'}
2018-01-18 12:05:13,854 DEBUG                curator.cli                    run:121  Full list of actions: {1: {'action': 'delete_indices', 'description': 'Delete UMP indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': False, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'ump-'}, {'direction': 'older', 'stats_result': 'min_value', 'filtertype': 'age', 'source': 'name', 'epoch': None, 'timestring': '%Y.%m.%d', 'exclude': False, 'unit_count': 7, 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}, 2: {'action': 'delete_indices', 'description': 'Delete Linkyfi indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'filters': [{'exclude': False, 'kind': 'prefix', 'filtertype': 'pattern', 'value': 'linkyfi'}, {'direction': 'older', 'stats_result': 'min_value', 'filtertype': 'age', 'source': 'name', 'epoch': None, 'timestring': '%Y.%m.%d', 'exclude': False, 'unit_count': 7, 'unit': 'days'}], 'options': {'continue_if_exception': False, 'timeout_override': None, 'ignore_empty_list': True, 'disable_action': False}}}
2018-01-18 12:05:13,854 DEBUG                curator.cli                    run:126  action_disabled = False
2018-01-18 12:05:13,854 DEBUG                curator.cli                    run:130  continue_if_exception = False
2018-01-18 12:05:13,854 DEBUG                curator.cli                    run:132  timeout_override = None
2018-01-18 12:05:13,854 DEBUG                curator.cli                    run:134  ignore_empty_list = True
2018-01-18 12:05:13,854 INFO                 curator.cli                    run:144  Preparing Action ID: 1, "delete_indices"
2018-01-18 12:05:13,854 DEBUG              curator.utils             get_client:803  kwargs = {'aws_secret_key': None, 'url_prefix': '', 'http_auth': 'admin:admin', 'certificate': '/home/ls/.curator/root-ca.pem', 'aws_key': None, 'aws_sign_request': False, 'port': 9200, 'hosts': ['127.0.0.1'], 'timeout': 30, 'aws_token': None, 'use_ssl': True, 'master_only': False, 'client_cert': None, 'ssl_no_validate': False, 'client_key': None}
2018-01-18 12:05:13,854 DEBUG              curator.utils             get_client:809  Attempting to verify SSL certificate.
2018-01-18 12:05:13,855 DEBUG              curator.utils             get_client:880  Not using "requests_aws4auth" python module to connect.

SG

unread,

Jan 18, 2018, 6:50:55 AM1/18/18

to search...@googlegroups.com

Its recommended to use a recent python 3 version because python 2.7 has a lot of ssl issues.

https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/python-security.html
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/security-errors.html

Best is to install the most recent version of curator as a single binary (V 5.4 as of now):
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/installation.html
https://packages.elastic.co/curator/5/debian/pool/main/e/elasticsearch-curator/elasticsearch-curator_5.4.1_amd64.deb
https://packages.elastic.co/curator/5/centos/7/Packages/elasticsearch-curator-5.4.1-1.x86_64.rpm

See also
https://github.com/elastic/curator/pull/783
https://github.com/floragunncom/search-guard/issues/401
https://github.com/bobrik/docker-curator/pull/22

Programmatic python connection is working with:
apt-get install python3 python3-pip
pip3 install elasticsearch requests
pip3 install cryptography
pip3 install pyopenssl ndg-httpsclient pyasn1
https://gist.github.com/floragunncom/9319a994ae09df64b2a173128f745ed2

If its not working with the above mentioned recommendations please open a issue here:
https://github.com/elastic/curator

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/80ccb278-89e0-4698-b04f-bc2efd7ecc80%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Lucas Schill

unread,

Jan 25, 2018, 11:39:25 AM1/25/18

to Search Guard Community Forum

Hi,
I'm seeing the same issue and am unable to find a resolution. Were you able to get this fixes?

Search Guard

unread,

Feb 1, 2018, 4:09:56 PM2/1/18

to Search Guard Community Forum

...

Jannik Z

unread,

Dec 21, 2018, 8:19:22 AM12/21/18

to Search Guard Community Forum

I'm using python 3.6 in this docker container: https://hub.docker.com/r/bobrik/curator

Without success! Could you make a quick test with this container?

`elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch. Error: ConnectionError(("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')],)",)) caused by: SSLError(("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')],)",))`