I have about 500 salt-minions running and i keep seeing intermittent issues where they randomly crash after some time with the following backtrace:
[DEBUG ] Minion of '172.16.0.1' is handling event tag '_minion_mine'
[DEBUG ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'C01-B07-VM6', 'tcp://
172.16.0.1:4506')
[DEBUG ] Minion return retry timer set to 7 seconds (randomized)
[DEBUG ] Closing AsyncReqChannel instance
[DEBUG ] Closing IPCMessageClient instance
[DEBUG ] schedule.handle_func: Removing /var/cache/salt/minion/proc/20221020223936633621
[DEBUG ] Subprocess Schedule(name=__mine_interval, jid=20221020223936633621) cleaned up
Process MinionKeepAlive:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/virt-py3/lib/python3.8/site-packages/salt/scripts.py", line 142, in minion_process
minion.start()
File "/root/virt-py3/lib/python3.8/site-packages/salt/cli/daemons.py", line 325, in start
self._real_start()
File "/root/virt-py3/lib/python3.8/site-packages/salt/cli/daemons.py", line 337, in _real_start
self.minion.tune_in()
File "/root/virt-py3/lib/python3.8/site-packages/salt/minion.py", line 1182, in tune_in
self.io_loop.start()
File "/root/virt-py3/lib/python3.8/site-packages/salt/ext/tornado/ioloop.py", line 865, in start
event_pairs = self._impl.poll(poll_timeout)
File "/root/virt-py3/lib/python3.8/site-packages/salt/scripts.py", line 109, in handle_hup
manager.minion.reload()
AttributeError: 'MinionManager' object has no attribute 'reload'
(virt-py3) [root@C01-B07-VM6 ~]#