salt-api+master+syndic broken?

[Volker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

i am experiencing a problem with minion-discovery when going through
the salt-api. My current setup is:

user@somehost -> saltmaster+salt-api -> syndic1 -> minion1

On the saltmaster i only have the key of syndic1.
On the syndic1 i only have the key of minions.

When publishing commands on _syndic1_ for minion1, everything works as
expected.

When publishing commands on _saltmaster_ for minion1 via commandline
it also works.

What does not work, is publishing commands through salt-api on saltmaster.

I always get an empty reply back. Running all parties in debug mode,
shows, that the communication itself works fine. The publish reaches
syndic1 and minion1, the return then reaches syndic1 and saltmaster.
Its just saltapi that does not get anything back.

It seems like the saltmaster receives the return of minion1 and then
discards it, because minion1 is not present on saltmaster.

The only way i can get it to work is, by putting the minion1 key onto
saltmaster.

Has anyone tried this or experienced this kind of behaviour?

Im currently running

Salt: 0.16.0-1735-g537af34
Python: 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
Jinja2: 2.5.5
M2Crypto: 0.20.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.1.0
PyYAML: 3.09
PyZMQ: 13.1.0
ZMQ: 3.2.3



Im aware of 0.17.2, but updating is my last resort :-)

- - felskrone

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKPXV8ACgkQHaTGAGocg2J7VACePe5SffNslUmP9a/ECtqgimu1
tPoAn1MkOr/uZe8oKGHMV4fPuqZoLQ/w
=rzSb
-----END PGP SIGNATURE-----

[Seth House]

I will test this today. What version of salt-api are you running?

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Volker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Seth,

On 11/22/13 3:10 PM, Seth House wrote:
> I will test this today. What version of salt-api are you running?
>

Im running

ii salt-api 0.8.2
Generic, modular network access system

If you want me to test anything, im also on irc in #salt and #salt-dev.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKPa+0ACgkQHaTGAGocg2KW3gCfQ8dvIGOxLX6egTFcx3xqtpUJ
4kAAoJ83LmfRRbOHZmnR2jCGMgsoWpsr
=nnUm
-----END PGP SIGNATURE-----

[Volker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/22/13 3:36 PM, Volker wrote:
> Hi Seth,
>
> On 11/22/13 3:10 PM, Seth House wrote:
>> I will test this today. What version of salt-api are you
>> running?
>

Hey Seth,

did you happen to find anything yet?

- - felskrone

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKUjhgACgkQHaTGAGocg2LxGwCfYc88+hz6W7N6+SBl47zB5Pot
oJQAnRF5skVIUPRUdDnPQkJuP2TaiEpH
=EbT8
-----END PGP SIGNATURE-----

[Seth House]

On Tue, Nov 26, 2013 at 5:03 AM, Volker <g...@schwicking.de> wrote:
> did you happen to find anything yet?

felskrone, we're setting up a test tonight. I should know more tomorrow.

[Seth House]

felskrone, I was finally able to test this today -- although not with
your exact Salt and salt-api version pairing. I wasn't able to
reproduce the issue but that's not too surprising since there have
been quite a few syndic changes between 0.16.0 and 0.17.3.

I should have thought to ask this question before now: what is the
result of running the command on the _saltmaster_ CLI...but using
eauth (the -a flag)? Does it still work as before or fail like it does
when going through salt-api on that machine?

[Volker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

On 11/28/13 12:36 AM, Seth House wrote:>>
>> felskrone, we're setting up a test tonight. I should know more
tomorrow.
>

I updated to 0.17.* latest git to just to make sure, its not related
to that. So we're on:

salt-master:
salt02:~/api-examples $ salt * --versions-report
Salt: 0.17.0-5148-g1ab7fd7

Python: 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
Jinja2: 2.5.5
M2Crypto: 0.20.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.1.0
PyYAML: 3.09
PyZMQ: 13.1.0
ZMQ: 3.2.3

syndic:
syndic01:~ $ salt * --versions-report
Salt: 0.17.0-5148-g1ab7fd7

Python: 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
Jinja2: 2.5.5
M2Crypto: 0.20.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.1.0
PyYAML: 3.09
PyZMQ: 13.1.0
ZMQ: 3.2.3

minion:
server062:~# salt-call --versions-report
Salt: 0.17.0-5148-g1ab7fd7
Python: 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)

Jinja2: 2.5.5
M2Crypto: 0.20.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.1.0
PyYAML: 3.09
PyZMQ: 13.1.0
ZMQ: 3.2.3

I tried using eauth with the salt-binary today and it looks pretty good.

salt02:~ $ salt server062* abuse.check_joomla pkg_id=123456789 --auth=pam
username: swapi
password: <password>
server062.mydomain.com:
----------
data:
/var/www/<pkg_i>_<rand_hash>/...
msg:
----------
success:
True

I see the command on the master, the syndic and the minion, etc. So
that works. Doing the same over the salt-api still only works, if i
put the minions key on the saltmaster AND the syndic.

I posted some debug-info here:

http://pastebin.com/AqwujGT6

I'm using rest_wsgi.py

Last thing i can do is update to the latest available salt-api which
is i believe 0.8.3.

- - felskrone




-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKcllkACgkQHaTGAGocg2KcswCaAguXc39mgmnyrxwhyraVE4FY
8JMAn0370mXuC2VBBxZidpcFaatU6Ou+
=W08f
-----END PGP SIGNATURE-----

[Volker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/2/13 3:16 PM, Volker wrote:
> Hi,
>
> On 11/28/13 12:36 AM, Seth House wrote:>>
>>> felskrone, we're setting up a test tonight. I should know more
> tomorrow.

I just rebuild the salt-api-debian package, installed it and when i
publish a job i get this in the apache error-log:

###
[Mon Dec 02 15:31:22 2013] [error] [client 10.30.8.9] mod_wsgi
(pid=8018): Exception occurred processing WSGI script
'/usr/share/pyshared/saltapi/netapi/rest_wsgi.py'.
[Mon Dec 02 15:31:22 2013] [error] [client 10.30.8.9] Traceback (most
recent call last):
[Mon Dec 02 15:31:22 2013] [error] [client 10.30.8.9] File
"/usr/share/pyshared/saltapi/netapi/rest_wsgi.py", line 255, in
application
[Mon Dec 02 15:31:22 2013] [error] [client 10.30.8.9]
saltenviron(environ)
[Mon Dec 02 15:31:22 2013] [error] [client 10.30.8.9] File
"/usr/share/pyshared/saltapi/netapi/rest_wsgi.py", line 246, in
saltenviron
[Mon Dec 02 15:31:22 2013] [error] [client 10.30.8.9]
environ['SALT_OPTS'] = __opts__
[Mon Dec 02 15:31:22 2013] [error] [client 10.30.8.9] NameError:
global name '__opts__' is not defined
###

The above also happens when i try to start the wsgi-app on the cli:

###
Traceback (most recent call last):
File "rest_wsgi.py", line 302, in <module>
start()
File "rest_wsgi.py", line 291, in start
mod_opts = __opts__.get(short_name, {})
NameError: global name '__opts__' is not defined
###

Does importing salt+saltapi already populate __opts__?

Besides that i dont see anything, that would make __opts__ available
to the app.

- - felskrone








-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKcnEkACgkQHaTGAGocg2IZCACfVod3rpqhEEpU7wOHTE+QF9cv
/0IAoIeJDb8GoxawbnzkD3+0m2FVwaN/
=ma27
-----END PGP SIGNATURE-----

[Volker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

HI,

On 12/2/13 3:16 PM, Volker wrote:
>

> I see the command on the master, the syndic and the minion, etc.
> So that works. Doing the same over the salt-api still only works,
> if i put the minions key on the saltmaster AND the syndic.
>

i kept digging and finally found the solution/bug that is causing this
behaviour. It has nothing to do with salt-api, its a bug in
utils/minions.py combined with a new_job announcement on the eventbus
which is relevant for the localclient.

Here is how it happens:

The publisher receives a new job from the localclient (instantiated by
the rest_wsgi-application) with certain parameters. The publisher
takes the given parameters and tries to figure out, which minions are
expected to return something for this job.

The publisher does that, by using salt.utils.minions.check_minions()
with 'tgt' and 'expr_form' as parameters. In my case that is
'minion_id' (fqdn) and 'glob'.

Now utils.minions internally calls _check_glob_minions() which is
rather lazy and just does glob.glob() on the pki/master/minions
directory. Because my salt-api runs on a commanding saltmaster
(order_masters=true), there are no matching minion keys except the
syndic ones. That causes an empty list to be returned to the publisher
which then announces a new job on the eventbus with an empty minions list:

###
{'tag': '20131205231610337853', 'data': {'_stamp':
'2013-12-05_23:16:10.338119', 'minions': []}}
###

This does not yet break anything, because the publisher can handle an
empty minions list just fine.

Now we go back to the localclient which sits there and waits for the
published job to return something. After publishing a job, the
localclient tells itself to go to self.get_returns to wait for events
on a specific jid. The parameters for get_returns are the jid we
received from the publisher and wait for it... yes, the minion list we
also received from the publisher which is empty!

That makes the while-loop in get-returns break immediately and return
an empty result to the client because this always evaluates to true:

###
841 if len(found.intersection(minions)) >= len(minions):
842 # All minions have returned, break out of the loop
843 break
###

This can be easily tested by just touching the required minion-id in
the masters pki-directory. This also makes sense considering my
earlier findings where i put the minions key on the master and the
syndic to make it work.

@Seth: i have no idea why your test-setup works fine. It shouldnt have
if it looked like mine :-)

Fixing this:
The one thing i dont yet understand is, why this happens when i go
through the salt-api and not on the command line with eauth=pam.

I'll be happy to supply a patch. The thing is, im not sure how to
approach this until i figured out, whats the difference between the
localclient from cli and and the localclient from salt-api.

If anyone of the developers could enlighten me here on the last two
things, it would be highely appreciated.

- - volker

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKhCOEACgkQHaTGAGocg2JCwACeMpZNYnL0YH/pg/e271Z9KJ1T
hscAnRSiPtUgkaZgFcUSaYOtPfSUigN1
=e1sE
-----END PGP SIGNATURE-----

[Colton Myers]

Wow, thanks for the great detective work!  I'm not familiar enough with the differences in the API to know why it's only failing there, but I'll make sure Seth gets eyes on this.

--

Colton Myers

-----END PGP SIGNATURE-----

[Seth House]

On Thu, Dec 5, 2013 at 3:14 PM, Volker <g...@schwicking.de> wrote:
> The one thing i dont yet understand is, why this happens when i go
> through the salt-api and not on the command line with eauth=pam.

When you're trying this without going through salt-api are you using
``LocalClient()`` in a Python shell or calling ``salt`` at the CLI?
And are you always testing with eauth credentials?

salt-api wraps LocalClient().cmd():

https://github.com/saltstack/salt-api/blob/develop/saltapi/__init__.py#L84

https://github.com/saltstack/salt/blob/develop/salt/client/__init__.py#L325

Salt's CLI wraps LocalClient().cmd_cli():

https://github.com/saltstack/salt/blob/develop/salt/client/__init__.py#L450

[Volker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I setup a simplistic test-script to test wether its related to
LocalClient.cmd() and LocalClient.cmd_cli()

###
1 #!/usr/bin/python
2 import salt.config
3 import salt.client
4
5 ##################################
6 def run():
7
8 client = salt.client.LocalClient('/etc/salt/master')
9
10 for host in client.cmd_cli(tgt='server01.mydomain.com',
11 fun='test.ping',
12 timeout=10,
13 expr_form='glob'):
14 print str(host)
15
16 if __name__ == '__main__':
17 run()
###

Just switching between client.cmd() and client.cmd_cli() in line 10
shows the described problem.

On 12/6/13 11:34 PM, Seth House wrote:
>
> salt-api wraps LocalClient().cmd():
>
Here the Script exits immediately.

salt02:~/tests $ time python cmd-test.py

real 0m0.275s
user 0m0.244s
sys 0m0.028s

>
> Salt's CLI wraps LocalClient().cmd_cli():
>

Here i get the expected return.

salt02:~/tests $ time python cmd-test.py
{'server01.mydomain.com': {'ret': True}}

real 0m29.265s
user 0m0.264s
sys 0m0.008s

I have not yet dug any deeper.

- - felskrone

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKk1jQACgkQHaTGAGocg2KakACglkTuyey5v7ra1P9w+XEfP8N6
9VgAoLE9bJD0TDX+vrr1V6YpFYhkcTHl
=+Srf
-----END PGP SIGNATURE-----

[Seth House]

Nice work! felskrone, will you please file an issue with this
information on Salt's bug tracker?

[Volker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/9/13 6:06 PM, Seth House wrote:
> Nice work! felskrone, will you please file an issue with this
> information on Salt's bug tracker?
>

https://github.com/saltstack/salt/issues/9141

Done :-)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKnShgACgkQHaTGAGocg2LiRwCgu3Vym3RLXn/AcgQrKRXtto+v
WDkAnRQk0gBY9MNz6+yRJZP5N0p6La34
=eD7F
-----END PGP SIGNATURE-----