[kosta]

on master salt '*' test.ping

on one of the minions always get :

Execution is still running on x.y.com

trace log on minion:

2013-05-19 09:15:44,666 [salt.minion                                 ][INFO    ] User root Executing command saltutil.find_job with jid 20130519051544649576
2013-05-19 09:15:44,667 [salt.minion                                 ][DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20130519051544649576', 'tgt': '*', 'ret': '', 'user': 'root', 'arg': ['20130519051505748366'], 'fun': 'saltutil.find_job'}
2013-05-19 09:15:44,672 [salt.loaded.int.module.cmdmod               ][INFO    ] Executing command 'vzps -E 0 -efH|cut -b 6-' in directory '/root'
2013-05-19 09:15:44,677 [salt.loaded.int.module.cmdmod               ][DEBUG   ] output:
2013-05-19 09:15:44,677 [salt.minion                                 ][WARNING ] The minion function caused an exception: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 428, in _thread_return
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/salt/modules/saltutil.py", line 389, in find_job
    for data in running():
  File "/usr/lib/python2.6/site-packages/salt/modules/saltutil.py", line 356, in running
    procs = __salt__['status.procs']()
  File "/usr/lib/python2.6/site-packages/salt/modules/status.py", line 55, in procs
    guide = plines.pop(0).split()
IndexError: pop from empty list

what do you say?

thanx

tryed to reinstall... worked for a while and came back again

minion 15.1

centos 6.4

[Colton Myers]

Are all your minions 0.15.1?

Is your master 0.15.1?

Additionally, how is the minion that is failing different from the other minions which always succeed?  Different OS, different versions, etc?

--

Colton Myers

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[kosta]

all 0.15.1

nothing is different

CentOS release 6.4 (Final)

its one of openvz hosts

all guests inside are ok

i have more installs like that, all working

[Colton Myers]

So the only command you've run is test.ping?  Or did you run others previously?  The stacktrace in the minion log does not seem like it could have been caused by test.ping, as it doesn't call into the status.py module.....

I'm guessing that the minion errored out on a previous run of state.highstate or a call into the status module, and then when you try to ping it, it's unreachable.

Any idea what you ran right *before* the behavior showed up?

--

Colton Myers

[kosta]

ok... now after running top a saw alots of minion proccesses running... hundreds..no idea why

will try to find out

[kosta]

ok 

when i choose 

'*'

there is a problem, but

'name*'

no problem

013-05-20 19:10:38,629 [salt.minion                                 ][DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20130520151038566264', 'tgt': '*', 'ret': '', 'user': 'root', 'arg': [], 'fun': 'test.version'}

2013-05-20 19:10:38,634 [salt.minion                                 ][INFO    ] Returning information for job: 20130520151038566264

2013-05-20 19:10:52,254 [salt.minion                                 ][INFO    ] User root Executing command saltutil.find_job with jid 20130520151052198826

2013-05-20 19:10:52,254 [salt.minion                                 ][DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20130520151052198826', 'tgt': '*', 'ret': '', 'user': 'root', 'arg': ['20130520151038566264'], 'fun': 'saltutil.find_job'}

2013-05-20 19:10:52,259 [salt.loaded.int.module.cmdmod               ][INFO    ] Executing command 'vzps -E 0 -efH|cut -b 6-' in directory '/root'

2013-05-20 19:10:52,263 [salt.loaded.int.module.cmdmod               ][DEBUG   ] output:

2013-05-20 19:10:52,264 [salt.minion                                 ][WARNING ] The minion function caused an exception: Traceback (most recent call last):

  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 428, in _thread_return

    ret['return'] = func(*args, **kwargs)

  File "/usr/lib/python2.6/site-packages/salt/modules/saltutil.py", line 389, in find_job

    for data in running():

  File "/usr/lib/python2.6/site-packages/salt/modules/saltutil.py", line 356, in running

    procs = __salt__['status.procs']()

  File "/usr/lib/python2.6/site-packages/salt/modules/status.py", line 55, in procs

    guide = plines.pop(0).split()

IndexError: pop from empty list

[kosta]

ok what elsei did find out:

2013-05-20 19:41:12,604 [salt.loaded.int.module.cmdmod               ][INFO    ] Executing command 'vzps -E 0 -efH|cut -b 6-' in directory '/root'

2013-05-20 19:41:12,608 [salt.loaded.int.module.cmdmod               ][DEBUG   ] output: /bin/sh: vzps: command not found

in this server this "vzps" was installed and quit with segmentation falt...

why do we need to run it ?

[Colton Myers]

It looks like `vzps` is used in lieu of the `ps` command if the OS reports that it's being virtualized with OpenVZ:  https://github.com/saltstack/salt/blob/develop/salt/grains/core.py#L510-L511

This means that many status and service commands will fail if you are using OpenVZ but vzps is not available.  (Basically anywhere Salt uses the `ps` command to inspect processes will fail)

Are you using OpenVZ?  And if so, do you know why the vzps command is not present?

--

Colton Myers

[kosta]

because it didnt work for me causing segmentation fault

and it doesnt install from epel

[Colton Myers]

You didn't answer my question:  are you using OpenVZ for your virtualization?

--

Colton Myers

[kosta]

ah, sorry

yes

[Colton Myers]

I remember hearing something about an architectural problem with OpenVZ -- let me ask around and see if I can track it down.

--

Colton Myers

[kosta]

though all my servers are openvz...

only one had a problem...

[Colton Myers]

Very strange.  Is it the same one that fails every time?

--

Colton Myers

[Thomas S Hatch]

Sorry I have been slow on this one, it looks like the the pscommand is failing out that system. Can you print out the return from:

salt-call grains.item ps

and then run the ps command that is returned and give us the output? I think that somehow the ps command is failing to return data

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[kosta]

i am sorry i have already removed vzps package

and now it works ok so i cant reproduce...

anyway this is not package i would relay on

[Colton Myers]

So it's working now?  Awesome, keep us posted if you have further problems.

--

Colton Myers

[JP Espinosa]

https://github.com/saltstack/salt/pull/6137

This fix is related to this, vzps is not an official OpenVZ tool anymore

On Tuesday, May 21, 2013 11:55:32 AM UTC-5, basepi wrote:

So it's working now?  Awesome, keep us posted if you have further problems.

--

Colton Myers

On Tue, May 21, 2013 at 2:36 AM, kosta <kost...@gmail.com> wrote:

i am sorry i have already removed vzps package

and now it works ok so i cant reproduce...

anyway this is not package i would relay on

On Tue, May 21, 2013 at 11:23 AM, Thomas S Hatch <that...@gmail.com> wrote:

Sorry I have been slow on this one, it looks like the the pscommand is failing out that system. Can you print out the return from:

salt-call grains.item ps

and then run the ps command that is returned and give us the output? I think that somehow the ps command is failing to return data

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

saltstack.com | www.saltstack.com