Unable to get the elasticsearch returner working.

[Andy Rudeseal]

I've read all the docs I can find on this.  I'm trying to get this working from a minion before I apply to a production salt master/syndics.   I hope someone here can help....I know I've probably missed something or did something wrong in the process.

Here is what I have:

Salt Version:
          Salt: 3004.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4
 
System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.11.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

I installed the elasticsearch client like so:

python3 -m pip install elasticsearch

Running this again confirms it's installed.

python3 -m pip install elasticsearch
Requirement already satisfied: elasticsearch in /usr/local/lib/python3.6/site-packages
Requirement already satisfied: elastic-transport<9,>=8 in /usr/local/lib/python3.6/site-packages (from elasticsearch)
Requirement already satisfied: certifi in /usr/local/lib/python3.6/site-packages (from elastic-transport<9,>=8->elasticsearch)
Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/site-packages (from elastic-transport<9,>=8->elasticsearch)
Requirement already satisfied: urllib3<2,>=1.26.2 in /usr/local/lib/python3.6/site-packages (from elastic-transport<9,>=8->elasticsearch)

This is my minion config file, now I have no idea how to setup http_auth username and password here, I'm just guessing (I am no developer so looking at the code doesn't help).  The docs don't even go into how .... <sarcasim>I guess everyone uses an unsecured elasticsearch</sarcasim>

master: salt
log_level: debug
external_job_cache: elasticsearch

elasticsearch:
    host: 'elastic.blah.com:443'
    use_ssl: true
    username: user
    password: pass
    index: 'salt-returner'
    index_date: True
    number_of_shards: 1
    number_of_replicas: 1
    debug_returner_payload: True
    states_count: True
    states_order_output: True
    states_single_index: True
    functions_blacklist:
      - test.ping
      - saltutil.find_job

This is the output:

salt-call --local grains.items --return elasticsearch
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Missing configuration file: /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/qad.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/qad.conf
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/usr/lib/python3.6/site-packages/salt/grains/zfs.py'>
[DEBUG   ] /etc/resolv.conf: The domain and search keywords are mutually exclusive.
[DEBUG   ] Elapsed time getting FQDNs: 0.11004424095153809 seconds
[DEBUG   ] Loading static grains from /etc/salt/grains
[DEBUG   ] LazyLoaded zfs.is_supported
[DEBUG   ] Determining pillar cache
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.module.grains' from '/usr/lib/python3.6/site-packages/salt/modules/grains.py'>
[DEBUG   ] LazyLoaded grains.items
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] LazyLoaded path.which
[DEBUG   ] Override  __salt__: <module 'salt.loaded.int.module.dracr' from '/usr/lib/python3.6/site-packages/salt/modules/dracr.py'>
[DEBUG   ] Missing configuration file: /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Including configuration from '/etc/salt/minion.d/qad.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/qad.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: db3dfe52-5a10-4bbc-bd11-402609bbc91e
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Could not LazyLoad idem.hub: 'idem' __virtual__ returned False: No module named 'pop'
[DEBUG   ] Override  __salt__: <module 'salt.loaded.int.module.kubeadm' from '/usr/lib/python3.6/site-packages/salt/modules/kubeadm.py'>
[INFO    ] Executing command 'pdbedit' in directory '/root'
[DEBUG   ] stdout: Version 4.10.16
[DEBUG   ] output: Version 4.10.16
[DEBUG   ] Override  __salt__: <module 'salt.loaded.int.module.udev' from '/usr/lib/python3.6/site-packages/salt/modules/udev.py'>
[DEBUG   ] Override  __pillar__: <module 'salt.loaded.int.module.virtualenv_mod' from '/usr/lib/python3.6/site-packages/salt/modules/virtualenv_mod.py'>
[DEBUG   ] DSC: Only available on Windows systems
[DEBUG   ] Module PSGet: Only available on Windows systems
[DEBUG   ] Could not LazyLoad elasticsearch.index_exists: 'elasticsearch' __virtual__ returned False: Cannot load module elasticsearch: elasticsearch libraries not found
[DEBUG   ] LazyLoaded nagios.list_plugins
[DEBUG   ] Could not LazyLoad elasticsearch.returner: 'elasticsearch.returner' is not available.
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded nested.output

Any help on getting this working is greatly appreciated.

[Andy Rudeseal]

This seems to be the problem, reverting to 7.17 of the library resolved the issue

https://github.com/saltstack/salt/issues/61973

That bug was reported over a year ago, so not sure if it is going to be resolved anytime soon, and elastic is already now on 8.8.  This workaround may not last long.