Minion stuck believen highstate is still running.

[Estevo U. C. Castro]

I have this minion in a stuck state, where it will refuse to execute state.highstate because it (or the master?) believes it is still executing some highstate job I launched yesterday.

# salt-call state.highstate
local:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 1019 and was started at 2014, Apr 16 18:56:58.369395 with jid 20140416185658369395

There is no process with PID 1019 in this minion.  Problem persist even if I reboot the master and the minion.  How do I reset this state so this machine can run highstate again?

[Estevo U. C. Castro]

Just in case this may help anyone else:

I found a /var/cache/salt/minion/proc/20140416185658369395 in the minion.  Deleting it fixed the problem.

[Seth House]

The saltutil module has various job-related functions you can use to
find running jobs, signal jobs, kill jobs, etc.

http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.saltutil.html#salt.modules.saltutil.kill_job

> --
> You received this message because you are subscribed to the Google Groups
> "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dirk Warnich]

Having the same problem. Tried to kill job, but it doesn't work. See console output: 

[root@kdev salt]# salt kdev0016 state.highstate

kdev0016:

    Data failed to compile:

----------

    The function "state.highstate" is running as PID 1242 and was started at 2015, Feb 25 13:35:12.748637 with jid 20150225133512748637

[root@kdev salt]# salt 'kdev0016' saltutil.kill_job 1242

kdev0016:

[root@kdev salt]# salt-run jobs.active

20150225133512748637:

    ----------

    Arguments:

    Function:

        state.highstate

    Returned:

    Running:

        |_

          ----------

          kdev0016:

              1242

    Target:

        kdev0016

    Target-type:

        glob

    User:

        root

[tim.h...@gmail.com]

Hi Dirk,

I think saltutil.kill_job takes the jobID not the processID as
its argument.

cheers,
Tim

[Dmitry Golubenko]

В Срд, 25/02/2015 в 12:10 -0800, tim.h...@gmail.com пишет:

> Hi Dirk,
>
> I think saltutil.kill_job takes the jobID not the processID as
> its argument.

also check for duplicate minion processes, this leads to same error for
any highstate jobs: one process take highstate job first, then second
responds with "highstate is running"

>
> cheers,
> Tim
>

[Dirk Warnich]

Ah of course.

Thanks Tim