salt-api behaves asynchronously?

[JanetB]

I've run into a problem with salt-api executing state.sls. The first time I execute it the result is empty, the next time (if done soon enough) returns the job_id but not the result of the execution, and the third time returns that the package is already installed.

Here's the state file and the output:

mysql-server:

  pkg:

    - installed

    - name: MySQL-server

    - version: 5.5.34

[ec2-user@ip-10-72-49-10 ~]$ sudo curl -k -sS https://10.72.49.10:8081 -H "Accept:application/json" -H "Content-type:application/x-www-form-urlencoded" -H "X-Auth-Token:18199ae1ab4014f61cf2fb5861193c88237fb919" -d client='local' -d fun='state.sls' -d arg='mysql-server' -d tgt='10.110.5.184'

{"return": [{}]}

[ec2-user@ip-10-72-49-10 ~]$ sudo curl -k -sS https://10.72.49.10:8081 -H "Accept:application/json" -H "Content-type:application/x-www-form-urlencoded" -H "X-Auth-Token:18199ae1ab4014f61cf2fb5861193c88237fb919" -d client='local' -d fun='state.sls' -d arg='mysql-server' -d tgt='10.110.5.184'

{"return": [{"10.110.5.184": ["The function \"state.sls\" is running as PID 18942 and was started at 2014, Jan 29 22:57:18.667028 with jid 20140129225718667028"]}]}

[ec2-user@ip-10-72-49-10 ~]$ sudo curl -k -sS https://10.72.49.10:8081 -H "Accept:application/json" -H "Content-type:application/x-www-form-urlencoded" -H "X-Auth-Token:18199ae1ab4014f61cf2fb5861193c88237fb919" -d client='local' -d fun='state.sls' -d arg='mysql-server' -d tgt='10.110.5.184'

{"return": [{"10.110.5.184": {"pkg_|-mysql-server_|-MySQL-server_|-installed": {"comment": "Package MySQL-server is already installed", "__run_num__": 0, "changes": {}, "name": "MySQL-server", "result": true}}}]}

Any ideas as to what I'm doing wrong? Thanks!

[Seth House]

What you're seeing is expected behavior -- but not friendly behavior. :)

Salt executions are always asynchronous. The master does a
fire-and-forget publish and the minion(s) return results back to the
master in the background. Salt's CLI performs various tricks to give
the appearance of synchronous behavior. Read on...

On Wed, Jan 29, 2014 at 5:50 PM, JanetB <jbors...@gmail.com> wrote:
> The first time I
> execute it the result is empty

This is a timeout issue. The minion starts executing the job but
Salt's Python API doesn't wait long enough for the job to finish, thus
the empty return. The job is still running on the minion, however. You
can increase the timeout that Salt will wait by passing the "timeout"
kwarg. The curl syntax for that though salt-api is: ``-d
timeout="300"``.

If you're hitting timeout issues consistently you might want to
embrace the asynchronous behavior by starting the execution against
the /minions URL which will return a JID. You can then look up the
results with the /jobs/<JID HERE> URL. As minions return data that URL
will start to populate with the returns.

> the next time (if done soon enough) returns
> the job_id but not the result of the execution

Salt blocks multiple state runs from being executed at the same time
to avoid one stepping on the other.

> and the third time returns
> that the package is already installed.

At this point the original execution has finished successfully and
returned data to the master. When you run the state a second time the
package has been installed. You can see the result of the first job by
looking up the job ID:

# List all jobs and find the last state run. (Or just grab the JID
from the error message you posted above.)
salt-run jobs.list_jobs

# Look up the result of the job
salt-run jobs.lookup_jid <JID HERE>

[Jesusaurus]

On Wed, Jan 29, 2014 at 8:44 PM, Seth House <se...@eseth.com> wrote:

This is a timeout issue. The minion starts executing the job but
Salt's Python API doesn't wait long enough for the job to finish, thus
the empty return. The job is still running on the minion, however. You
can increase the timeout that Salt will wait by passing the "timeout"
kwarg. The curl syntax for that though salt-api is: ``-d
timeout="300"``.

It would probably be useful if, when the timeout is hit, we return the job id and a message indicating that the job is still running instead of an empty return. 

-- 

Jesusaurus

[JanetB]

A non-empty result set would definitely be useful. :) 

[JanetB]

Thank you Seth! This will get me up and running!

[Seth House]

On Thu, Jan 30, 2014 at 9:23 AM, Jesusaurus <Jesus...@inbox.com> wrote:
> It would probably be useful if, when the timeout is hit, we return the job
> id and a message indicating that the job is still running instead of an
> empty return.

Agreed. We could return a 303 response if salt-api gets empty data
from Salt. The redirect could point to the URL for the job (e.g.,
/jobs/<JID>) which contains more information.

So 200 for success, 303 for a timeout. Thoughts?

[JanetB]

That seems like it would work. Then, with 200 we'd have the result, with 303 we'd get the jid and could then get the results. Otherwise, it is difficult to determine the outcome. And re-executing a state can lead to other errors. For example, with package installation, the first time the state is executed it succeeds and installs the package. The next time that state is executed I've seen an error returned because there is an update to the package available.