Reactor system and error log : The function "state.apply" is running as PID XXXX and was started at 2016, Nov 30 12:18:22.432381 with jid xxxxxxxxxxxxxxxxxx

[Samir Amara]

Hello,

I configured a reactor event a following :

reactor:                            
  - 'salt/minion/*/start':          
    - salt://reactor/event.sls

....event.sls:

clear_pillar:
  local.state.apply:
    - tgt: {{ data['id'] }}
    - arg:
      - saltutil.refresh_pillar

config_network:
  local.state.apply:
    - tgt: {{ data['id'] }}
    - arg:
      - reactor.utils.minion_network

Sometimes that works well, but most of the time it does not work. And I see this logs:

 The function "state.apply" is running as PID 1978 and was started at 2016, Nov 30 12:18:22.411602 with jid 20161130121822411602

2016-11-30 12:18:22,415 [salt.utils.event ][DEBUG   ][15885] Sending
 event: tag = 20161130121822411602; data = {'_stamp':
'2016-11-30T11:18:22.414638', 'minions': ['sama-li126v.revops.local']}
2016-11-30
 12:18:22,417 [salt.utils.event ][DEBUG   ][15885] Sending event: tag =
salt/job/20161130121822411602/new; data = {'tgt_type': 'glob', 'jid':
'20161130121822411602', 'tgt': 'sama-li126v.revops.local', '_stamp':
'2016-11-30T11:18:22.416453', 'user': 'root', 'arg':
['saltutil.refresh_pillar'], 'fun': 'state.apply', 'minions':
['sama-li126v.revops.local']}
2016-11-30 12:18:22,419
[salt.loaded.int.returner.local_cache][DEBUG   ][15885] Adding minions
for job 20161130121822411602: ['sama-li126v.revops.local']
2016-11-30
 12:18:22,420 [salt.master      ][INFO    ][15885] User root Published
command state.apply with jid 20161130121822411602
2016-11-30
12:18:22,421 [salt.master      ][DEBUG   ][15885] Published command
details {'tgt_type': 'glob', 'jid': '20161130121822411602', 'tgt':
'sama-li126v.revops.local', 'ret': '', 'to': 20, 'user': 'root', 'arg':
['saltutil.refresh_pillar'], 'fun': 'state.apply'}
2016-11-30
12:18:22,427 [salt.transport.zeromq][DEBUG   ][15862] Initializing new
AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'master_master',
'tcp://127.0.0.1:4506', 'clear')
2016-11-30 12:18:22,433
[salt.utils.event ][DEBUG   ][15883] Sending event: tag =
20161130121822432381; data = {'_stamp': '2016-11-30T11:18:22.433306',
'minions': ['sama-li126v.revops.local']}
2016-11-30 12:18:22,435
[salt.utils.event ][DEBUG   ][15883] Sending event: tag =
salt/job/20161130121822432381/new; data = {'tgt_type': 'glob', 'jid':
'20161130121822432381', 'tgt': 'sama-li126v.revops.local', '_stamp':
'2016-11-30T11:18:22.434808', 'user': 'root', 'arg':
['reactor.utils.minion_network'], 'fun': 'state.apply', 'minions':
['sama-li126v.revops.local']}
2016-11-30 12:18:22,437
[salt.loaded.int.returner.local_cache][DEBUG   ][15883] Adding minions
for job 20161130121822432381: ['sama-li126v.revops.local']
2016-11-30
 12:18:22,438 [salt.master      ][INFO    ][15883] User root Published
command state.apply with jid 20161130121822432381
2016-11-30
12:18:22,438 [salt.master      ][DEBUG   ][15883] Published command
details {'tgt_type': 'glob', 'jid': '20161130121822432381', 'tgt':
'sama-li126v.revops.local', 'ret': '', 'to': 20, 'user': 'root', 'arg':
['reactor.utils.minion_network'], 'fun': 'state.apply'}
2016-11-30 12:18:22,444 [salt.utils.reactor][DEBUG   ][15862] Gathering reactors for tag 20161130121822411602
2016-11-30 12:18:22,444 [salt.utils.reactor][DEBUG   ][15862] Gathering reactors for tag salt/job/20161130121822411602/new
2016-11-30 12:18:22,445 [salt.utils.reactor][DEBUG   ][15862] Gathering reactors for tag 20161130121822432381
2016-11-30 12:18:22,445 [salt.utils.reactor][DEBUG   ][15862] Gathering reactors for tag salt/job/20161130121822432381/new
2016-11-30
 12:18:22,462 [salt.utils.process][DEBUG   ][15862] ThreadPool executing
 func: <bound method RunnerClient.low of <salt.runner.RunnerClient
 object at 0x1132150>> with args:('state.orchestrate', {'mods':
'reactor.utils.change_portgroup', 'name':
u'invoke_orchestrate_update_potgroup', 'pillar':
OrderedDict([('event_data', 'sama-li126v.revops.local')]), 'state':
'runner', '__id__': 'invoke_orchestrate_update_potgroup', '__sls__':
u'/data/salt/cache/master/files/base/reactor/event.sls', 'order': 1,
'__user__': 'Reactor'}) kwargs{}
2016-11-30 12:18:22,464 [salt.config      ][DEBUG   ][15862] Reading configuration from /etc/salt/master
2016-11-30 12:18:22,692 [salt.utils.lazy  ][DEBUG   ][15882] LazyLoaded local_cache.prep_jid
2016-11-30
 12:18:22,693 [salt.utils.job   ][INFO    ][15882] Got return from
sama-li126v.revops.local for job 20161130121822432381
2016-11-30
12:18:22,694 [salt.transport.ipc][DEBUG   ][15882] Initializing new
IPCClient for path: /var/run/salt/master/master_event_pull.ipc
2016-11-30 12:18:22,694 [salt.utils.lazy  ][DEBUG   ][15884] LazyLoaded local_cache.prep_jid
2016-11-30
 12:18:22,697 [salt.utils.event ][DEBUG   ][15882] Sending event: tag =
salt/job/20161130121822432381/ret/sama-li126v.revops.local; data =
{'fun_args': ['reactor.utils.minion_network'], 'jid':
'20161130121822432381', 'return': ['The function "state.apply" is running as PID 1978 and was started at 2016, Nov 30 12:18:22.411602 with jid 20161130121822411602'],
 'retcode': 1, 'success': True, 'cmd': '_return', '_stamp':
'2016-11-30T11:18:22.696348', 'fun': 'state.apply', 'id':
'sama-li126v.revops.local', 'out': 'highstate'}
2016-11-30
12:18:22,697 [salt.utils.job   ][INFO    ][15884] Got return from
sama-li126v.revops.local for job 20161130121822411602
2016-11-30
12:18:22,698 [salt.transport.ipc][DEBUG   ][15884] Initializing new
IPCClient for path: /var/run/salt/master/master_event_pull.ipc
2016-11-30
 12:18:22,701 [salt.utils.event ][DEBUG   ][15884] Sending event: tag =
salt/job/20161130121822411602/ret/sama-li126v.revops.local; data =
{'fun_args': ['saltutil.refresh_pillar'], 'jid': '20161130121822411602',
 'return': ['The function "state.apply" is running as PID 1981 and was started at 2016, Nov 30 12:18:22.432381 with jid 20161130121822432381'],
 'retcode': 1, 'success': True, 'cmd': '_return', '_stamp':
'2016-11-30T11:18:22.700158', 'fun': 'state.apply', 'id':
'sama-li126v.revops.local', 'out': 'highstate'}

I need help please.

[Seth House]

By default only one state run can happen on a minion at a time. This
is because state runs often do heavy things to a system and the
likelyhood that one state run will trip on another is high. For
example: two state runs trying to call out to the system package
manager at the same time.

If you want to change that behavior take a look at the `queue` and
`concurrent` keyword arguments to `state.apply`:

https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.state.html#salt.modules.state.apply

On Wed, Nov 30, 2016 at 5:55 AM, Sam Amara <samir...@gmail.com> wrote:

>
>
> Le mercredi 30 novembre 2016 13:48:26 UTC+1, Samir Amara a écrit :
>>
>> Hello,
>>
>> I configured a reactor event a following :
>>
>> reactor:
>> - 'salt/minion/*/start':
>> - salt://reactor/event.sls
>>
>>
>> ....event.sls:
>>
>> clear_pillar:
>> local.state.apply:
>> - tgt: {{ data['id'] }}
>> - arg:
>> - saltutil.refresh_pillar
>>
>> config_network:
>> local.state.apply:
>> - tgt: {{ data['id'] }}
>> - arg:
>> - reactor.utils.minion_network
>>
>>
>> Sometimes that works well, but most of the time it does not work. And I
>> see this logs:
>> The function "state.apply" is running as PID 1978 and was started at
>> 2016, Nov 30 12:18:22.411602 with jid 20161130121822411602
>>

> --
> You received this message because you are subscribed to the Google Groups
> "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.