BUG? v3007.1 - mine.get not work when scheduled

[Dave Macias]

Good Day Everyone,

Ive been holding off upgrading to v3007.1 for lack of time.

I tackled it this week and was testing some jobs we have. Noticed that a job that relies on getting data from another minion (mine) was failing.

After some testing noticed that it's `mine.get` function that is having an issue when under v3007.1 and scheduled. That same mine.get function works perfectly fine when running directly from the master.

This is the schedule:

minion-1:
    schedule:
      enabled: true
      test_job:
        args:
        - minion-2
        - get_mine_data
        enabled: true
        function: mine.get
        jid_include: true
        maxrunning: 1
        name: test_job
        saved: true
        seconds: 30

The above schedule produces the below logs on my minion (pypi v3007.1)

2024-08-21 19:05:02,483 [salt.utils.schedule][INFO    ]  Running scheduled job: test_job with jid 20240821190502483289
2024-08-21 19:05:02,487 [salt.utils.schedule][INFO    ]  Running scheduled job: __master_alive_salt-dev-vm.domain.net with jid 20240821190502487178
2024-08-21 19:05:02,808 [salt.utils.schedule][ERROR   ]  Unhandled exception running mine.get
Traceback (most recent call last):
  File "/root/.venv/lib64/python3.11/site-packages/salt/utils/schedule.py", line 880, in handle_func
    ret["return"] = self.functions[func](*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib64/python3.11/site-packages/salt/loader/lazy.py", line 160, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib64/python3.11/site-packages/salt/loader/lazy.py", line 1269, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib64/python3.11/site-packages/salt/loader/lazy.py", line 1284, in _run_as
    return _func_or_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.11/site-packages/salt/modules/mine.py", line 362, in get
    ret = _mine_get(load, __opts__)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.11/site-packages/salt/modules/mine.py", line 78, in _mine_get
    with salt.channel.client.ReqChannel.factory(opts) as channel:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib64/python3.11/site-packages/salt/channel/client.py", line 56, in factory
    return SyncWrapper(
           ^^^^^^^^^^^^
  File "/root/.venv/lib64/python3.11/site-packages/salt/utils/asynchronous.py", line 76, in __init__
    self.obj = cls(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib64/python3.11/site-packages/salt/channel/client.py", line 134, in factory
    auth = salt.crypt.AsyncAuth(opts, io_loop=io_loop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib64/python3.11/site-packages/salt/crypt.py", line 695, in __new__
    auth.__singleton_init__(opts, io_loop=io_loop)
  File "/root/.venv/lib64/python3.11/site-packages/salt/crypt.py", line 742, in __singleton_init__
    self._authenticate_future = tornado.concurrent.Future()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/asyncio/events.py", line 677, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Schedule(name=test_job, jid=20240821190502483289)'.
2024-08-21 19:05:02,821 [salt.minion      ][INFO    ]  Returning information for job: req

This is the `Exception` message the above produces: (yes...i know jids are diff but thats the msg)

RuntimeError: There is no current event loop in thread 'Schedule(name=test_job, jid=20240821192002481926)'.

Running the below on the salt master (docker build with pypi v3007.1) works perfectly fine.

> salt minon-1 mine.get minion-2 get_mine_data

If i move my salt-minion to (pypi) v3006.8, the schedule `test_job` works perfectly fine.

salt minion and master run on python3.11.

Is this a bug?

Any input is appreciated!

Best,
Dave

[Dave Macias]

For giggles i removed all cache data and moved to v3007.1 for the minion just to see but no change :/

```

systemctl stop salt-minion.service
rm /var/log/salt/* /var/cache/salt/* /var/run/salt/* -rf
rm -rf .venv

cp -r venv3.11_v3007.1 .venv

systemctl restart salt-minion.service

```

Moving back to v3006.8 is fine.
It's looking like a bug :(

[Dave Macias]

Update:

Posted ticket on github

https://github.com/saltstack/salt/issues/66839

[Dave Macias]

Update:

In the meantime, i moved our master/minions to v3006.9 and all is well again in the world.