Minion never returns from state and/or master never sees state completion

[Hugues Demers]

The gist of my problem: I'm trying to install CouchDB on a minion, but calling state.sls hangs and never returns even though CouchDB has been installed. 

My question: how can I further diagnose the problem? Currently, I don't have a handle on it because I don't know what's keeping the minion from finishing. 

The command:

salt "ip-10-10-205-127" state.sls couchdb dev -v

The state file:

couchdb:

  pkgrepo:

    - managed

    - ppa: nilya/couchdb-1.3

    - name: deb http://ppa.launchpad.net/nilya/couchdb-1.3/ubuntu quantal main

  pkg:

    - installed

    - require:

        - pkgrepo: couchdb

Versions:

salt-minion --version: 0.15.1

salt-master --version: 0.15.1

Both running on Ubuntu.

The output from the command:

Executing job with jid 20130510103516666102

-------------------------------------------

Execution is still running on ip-10-10-205-127

Execution is still running on ip-10-10-205-127

Execution is still running on ip-10-10-205-127

Execution is still running on ip-10-10-205-127

Execution is still running on ip-10-10-205-127

Execution is still running on ip-10-10-205-127

^CExiting on Ctrl-C

This job's jid is:

20130510103516666102

The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later run:

salt-run jobs.lookup_jid 20130510103516666102

At this point, CouchDB is installed on the minion. I can query it: couchdb -V returns couchdb - Apache CouchDB 1.3.0. But something is preventing the master and/or minion from finishing. I've tried waiting a long time, but the job never finishes.

This:

salt-run jobs.lookup_jid 20130510103516666102

returns nothing. And this:

salt-run jobs.active

returns:

'20130510103516666102':

  Arguments:

  - couchdb

  - dev

  Function: state.sls

  Returned:

  - jid

  Running: []

  Target: ip-10-10-205-127

  Target-type: glob

Re-running state.sls gives:

ip-10-10-205-127:

    Data failed to compile:

----------

    The function "state.sls" is running as PID 3456 and was started at 2013, May 10 10:35:16.666102 with jid 20130510103516666102

I've tried installing other packages and it works fine. It does not seem related to a particular minion as I've spun up a lot of them and it's always the same story. I've suspected a CouchDB init.d script bug where trying to start/stop the daemons just after package installation did not work (but did if you rebooted), cf. https://bugs.launchpad.net/ubuntu/+source/couchdb/+bug/448682. It looks like it's been solved in 1.3.0, I've tried manually stopping after package installation and it works.

Any ideas on how to further diagnose this?

Thanks!

[Colton Myers]

Have you tried removing the PPA part of the state to make sure that's not where the problem lies?  (I can't imagine that would be the problem, but just want to make sure)

Otherwise, if you could run both the master and minion in debug mode and then post the relevant pieces of the logs, that might help.

--

Colton Myers

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Hugues Demers]

I have run the state as it is (with the PPA part) in debug mode, both the master and the minion. Whenever the minion is in debug mode everything works fine. When not in debug, same thing, it never returns.

I've tried what you suggested, removing the PPA part, and it's the same behavior: when in debug mode it works. 

What's different in debug mode? Aside from the debug messages and the daemon... Could this be related to timing issues?  

Thanks.

[Colton Myers]

Whoa, weird.

Is the minion daemonized in both cases?  Are you starting it manually from the command line or using the init scripts?

--

Colton Myers

[Hugues Demers]

Yes very weird. 

The minion is started from the command line: 

sudo salt-minion -d debug

It is not daemonized when started in debug mode, but is when started in non-debug mode.

Trying now to start it in non-debug non-daemonized mode. Will report soon.

[Hugues Demers]

I ran the minion in non-debug, non-daemonized mode, started from the command-line and it works. 

Is it the init.d script? I tried running the minion from the command line in non-debug, daemon mode and... it works. 

At this point, I had to re-test with the minion started from it's init script (really an upstart job). It does not work, as previously reported. 

So... Running from an init script breaks something. But what.

On Friday, May 10, 2013 1:54:10 PM UTC-4, basepi wrote:

[Colton Myers]

Well, that is a little more plausible that "running with debug log level fixes it", so it's a start.

I would post a bug with all the information you have discovered.  We should do some testing with the init scripts on other platforms, see if we can reproduce it, etc.  Just have to find the time.... =P

--

Colton Myers

[Hugues Demers]

Done: https://github.com/saltstack/salt/issues/4974

[Hugues Demers]

Disregard previous issue, was logged to GitHub with wrong credentials.

That's the one: https://github.com/saltstack/salt/issues/4975

On Friday, May 10, 2013 2:35:38 PM UTC-4, basepi wrote: