Supervisor Job is not starting

[Sheethal PN]

I am trying to install edx and this particular job fails. 

TASK: [supervisor | start supervisor] *****************************************

failed: [localhost] => {"failed": true, "item": ""}

msg: start: Job failed to start

While running the command service supervisor restart, i get the following output.

root@awsvlpa01:/var/tmp/configuration/playbooks# service supervisor restart                                             stop: Unknown instance:

start: Job failed to start

I have supervisor installed and is in the latest version.

[Ned Batchelder]

Hi Sheetal,

Installing Open edX is complicated.  We'll need more information in order to help.  What steps did you take? What instructions were you following? 

The logs are in /edx/var/logs, can you poke around in there to find a clue for us? Can you pastebin the complete output of the session?

--Ned.

--
You received this message because you are subscribed to the Google Groups "Open edX operations" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openedx-ops...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openedx-ops/b1968c39-6aa5-448a-baad-4957fe0e54f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sheethal PN]

The command i ran was 

/var/tmp/configuration/playbooks# ansible-playbook --extra-vars=@"./Dev_custom_vars.yml" -c local ./edx_sandbox.yml -i "localhost,"

PLAY [Configure instance(s)] **************************************************

GATHERING FACTS ***************************************************************

ok: [localhost]

TASK: [security | Check if we are vulnerable] *********************************

failed: [localhost] => {"changed": true, "cmd": " foo='() { echo vulnerable; }' bash -c foo ", "delta": "0:00:00.015581", "end": "2016-03-17 00:00:59.969442", "item": "", "rc": 127, "start": "2016-03-17 00:00:59.953861"}

stderr: bash: foo: command not found

...ignoring

TASK: [security | Apply bash security update if we are vulnerable] ************

skipping: [localhost]

TASK: [security | Check again and fail if we are still vulnerable] ************

skipping: [localhost]

TASK: [user | debug var=user_info] ********************************************

ok: [localhost] => {

    "item": "",

    "user_info": []

}

TASK: [user | create the edxadmin group] **************************************

ok: [localhost]

TASK: [user | ensure sudoers.d is read] ***************************************

ok: [localhost]

TASK: [user | grant full sudo access to the edxadmin group] *******************

ok: [localhost]

TASK: [user | create the users] ***********************************************

skipping: [localhost]

TASK: [user | create .ssh directory] ******************************************

skipping: [localhost]

TASK: [user | assign admin role to admin users] *******************************

skipping: [localhost]

TASK: [user | copy github key[s] to .ssh/authorized_keys2] ********************

skipping: [localhost]

TASK: [user | copy additional authorized keys] ********************************

skipping: [localhost]

TASK: [user | create bashrc file for normal users] ****************************

skipping: [localhost]

TASK: [user | create .profile for all users] **********************************

skipping: [localhost]

TASK: [user | modify shell for restricted users] ******************************

skipping: [localhost]

TASK: [user | create bashrc file for restricted users] ************************

skipping: [localhost]

TASK: [user | create sudoers file from template] ******************************

ok: [localhost]

TASK: [user | change home directory ownership to root for restricted users] ***

skipping: [localhost]

TASK: [user | create ~/bin directory] *****************************************

skipping: [localhost]

TASK: [user | create allowed command links] ***********************************

skipping: [localhost]

TASK: [common | Add user www-data] ********************************************

ok: [localhost]

TASK: [common | Create common directories] ************************************

ok: [localhost] => (item=/edx/var)

ok: [localhost] => (item=/edx/app)

ok: [localhost] => (item=/edx/bin)

ok: [localhost] => (item=/edx/etc)

TASK: [common | Install python-pycurl] ****************************************

ok: [localhost]

TASK: [common | Add git apt repository] ***************************************

ok: [localhost]

TASK: [common | Install role-independent useful system packages] **************

ok: [localhost]

TASK: [common | Create common log directory] **********************************

ok: [localhost]

TASK: [common | upload sudo config for key forwarding as root] ****************

ok: [localhost]

TASK: [common | pip install virtualenv] ***************************************

ok: [localhost] => (item=pip==1.5.4)

ok: [localhost] => (item=virtualenv==1.11.4)

ok: [localhost] => (item=virtualenvwrapper)

TASK: [common | Install rsyslog configuration for edX] ************************

ok: [localhost]

TASK: [common | Install logrotate configuration for edX] **********************

ok: [localhost]

TASK: [common | Create hourly subdirectory in logrotate.d] ********************

ok: [localhost]

TASK: [common | Install logrotate configuration for tracking file] ************

ok: [localhost]

TASK: [common | Add logrotate for tracking.log to cron.hourly] ****************

ok: [localhost]

TASK: [common | Remove old tracking.log config from /etc/logrotate.d] *********

ok: [localhost]

TASK: [common | update /etc/hosts] ********************************************

skipping: [localhost]

TASK: [common | update /etc/hostname] *****************************************

skipping: [localhost]

TASK: [common | run hostname] *************************************************

skipping: [localhost]

TASK: [common | update /etc/dhcp/dhclient.conf] *******************************

skipping: [localhost]

TASK: [common | update the ssh motd on Ubuntu] ********************************

ok: [localhost] => (item=/etc/update-motd.d/10-help-text)

ok: [localhost] => (item=/usr/share/landscape/50-landscape-sysinfo)

ok: [localhost] => (item=/etc/update-motd.d/51-cloudguest)

ok: [localhost] => (item=/etc/update-motd.d/91-release-upgrade)

TASK: [common | add ssh-warning banner motd] **********************************

ok: [localhost]

TASK: [common | update ssh config] ********************************************

ok: [localhost]

TASK: [nginx | create nginx app dirs] *****************************************

ok: [localhost] => (item=/edx/app/nginx)

ok: [localhost] => (item=/edx/app/nginx/sites-available)

ok: [localhost] => (item=/edx/app/nginx/sites-enabled)

TASK: [nginx | create nginx data dirs] ****************************************

ok: [localhost] => (item=/edx/var/nginx)

changed: [localhost] => (item=/edx/var/log/nginx)

TASK: [nginx | Install nginx packages] ****************************************

ok: [localhost]

TASK: [nginx | Server configuration file] *************************************

ok: [localhost]

TASK: [nginx | Creating common nginx configuration] ***************************

ok: [localhost]

TASK: [nginx | Create robot rules] ********************************************

skipping: [localhost]

TASK: [nginx | Creating link for common nginx configuration] ******************

ok: [localhost]

TASK: [nginx | Copying nginx configs for ['cms', 'lms', 'forum', 'xqueue']] ***

ok: [localhost] => (item=cms)

ok: [localhost] => (item=lms)

ok: [localhost] => (item=forum)

ok: [localhost] => (item=xqueue)

TASK: [nginx | Creating nginx config links for ['cms', 'lms', 'forum', 'xqueue']] ***

ok: [localhost] => (item=cms)

ok: [localhost] => (item=lms)

ok: [localhost] => (item=forum)

ok: [localhost] => (item=xqueue)

TASK: [nginx | Write out htpasswd file] ***************************************

skipping: [localhost]

TASK: [nginx | Create nginx log file location (just in case)] *****************

changed: [localhost]

TASK: [nginx | stat] **********************************************************

ok: [localhost]

TASK: [nginx | stat] **********************************************************

ok: [localhost]

TASK: [nginx | copy ssl cert] *************************************************

skipping: [localhost]

TASK: [nginx | copy ssl key] **************************************************

skipping: [localhost]

TASK: [nginx | Removing default nginx config and restart (enabled)] ***********

ok: [localhost]

TASK: [nginx | Set up nginx access log rotation] ******************************

ok: [localhost]

TASK: [nginx | Set up nginx access log rotation] ******************************

ok: [localhost]

TASK: [nginx | make sure nginx has started] ***********************************

ok: [localhost]

TASK: [edxlocal | install packages needed for single server] ******************

ok: [localhost]

TASK: [edxlocal | setup the migration db user] ********************************

ok: [localhost]

TASK: [edxlocal | setup the edxapp db user] ***********************************

ok: [localhost]

TASK: [edxlocal | create a database for edxapp] *******************************

ok: [localhost]

TASK: [edxlocal | setup the xqueue db user] ***********************************

ok: [localhost]

TASK: [edxlocal | create a database for xqueue] *******************************

ok: [localhost]

TASK: [edxlocal | setup the ora db user] **************************************

ok: [localhost]

TASK: [edxlocal | create a database for ora] **********************************

ok: [localhost]

TASK: [edxlocal | setup the discern db user] **********************************

skipping: [localhost]

TASK: [edxlocal | create a database for discern] ******************************

skipping: [localhost]

TASK: [edxlocal | install memcached] ******************************************

ok: [localhost]

TASK: [mongo | install python pymongo for mongo_user ansible module] **********

ok: [localhost]

TASK: [mongo | add the mongodb signing key] ***********************************

ok: [localhost]

TASK: [mongo | add the mongodb repo to the sources list] **********************

ok: [localhost]

TASK: [mongo | install mongo server and recommends] ***************************

ok: [localhost]

TASK: [mongo | create mongo dirs] *********************************************

ok: [localhost] => (item=/edx/var/mongo)

ok: [localhost] => (item=/edx/var/mongo/mongodb)

ok: [localhost] => (item=/edx/var/log/mongo)

TASK: [mongo | stop mongo service] ********************************************

changed: [localhost]

TASK: [mongo | move mongodb to {{mongo_data_dir}}] ****************************

skipping: [localhost]

TASK: [mongo | copy mongodb key file] *****************************************

skipping: [localhost]

TASK: [mongo | copy configuration template] ***********************************

ok: [localhost]

TASK: [mongo | start mongo service] *******************************************

changed: [localhost]

TASK: [mongo | wait for mongo server to start] ********************************

ok: [localhost]

TASK: [mongo | Create the file to initialize the mongod replica set] **********

skipping: [localhost]

TASK: [mongo | Initialize the replication set] ********************************

skipping: [localhost]

TASK: [mongo | create a mongodb user] *****************************************

skipping: [localhost] => (item={'password': 'password', 'user': 'cs_comments_service', 'database': 'cs_comments_service'})

skipping: [localhost] => (item={'password': 'password', 'user': 'exdapp', 'database': 'edxapp'})

TASK: [supervisor | create application user] **********************************

ok: [localhost]

TASK: [supervisor | create supervisor service user] ***************************

ok: [localhost]

TASK: [supervisor | create supervisor directories] ****************************

ok: [localhost] => (item=/edx/app/supervisor)

ok: [localhost] => (item=/edx/app/supervisor/venvs/supervisor)

changed: [localhost] => (item=/edx/app/supervisor/conf.d)

changed: [localhost] => (item=/edx/app/supervisor/conf.available.d)

TASK: [supervisor | create service user accessible dirs] **********************

changed: [localhost] => (item=/edx/app/supervisor/conf.d)

changed: [localhost] => (item=/edx/app/supervisor/conf.available.d)

TASK: [supervisor | create supervisor directories] ****************************

ok: [localhost] => (item=/edx/var/supervisor)

ok: [localhost] => (item=/edx/var/log/supervisor)

TASK: [supervisor | install supervisor in its venv] ***************************

ok: [localhost]

TASK: [supervisor | install supervisor in its venv] ***************************

skipping: [localhost]

TASK: [supervisor | create supervisor upstart job] ****************************

ok: [localhost]

TASK: [supervisor | create pre_supervisor upstart job] ************************

skipping: [localhost]

TASK: [supervisor | write the pre_suprevisor python script] *******************

skipping: [localhost]

TASK: [supervisor | create supervisor master config] **************************

ok: [localhost]

TASK: [supervisor | create a symlink for supervisortctl] **********************

ok: [localhost]

TASK: [supervisor | create a symlink for supervisor cfg] **********************

ok: [localhost] => (item=/edx/app/supervisor/supervisord.conf)

ok: [localhost] => (item=/edx/app/supervisor/conf.d)

TASK: [supervisor | start supervisor] *****************************************

failed: [localhost] => {"failed": true, "item": ""}

msg: start: Job failed to start

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************

           to retry, use: --limit @/root/edx_sandbox.retry

localhost                  : ok=68   changed=7    unreachable=0    failed=1

The log file supervisord.log 

016-03-16 22:53:22,217 CRIT Supervisor running as root (no user in config file)

2016-03-16 22:53:22,218 WARN No file matches via include "/edx/app/supervisor/conf.d/*.conf"

2016-03-16 22:53:22,234 INFO RPC interface 'supervisor' initialized

2016-03-16 22:53:22,234 CRIT Server 'inet_http_server' running without any HTTP authentication checking

2016-03-16 22:53:22,234 INFO RPC interface 'supervisor' initialized

2016-03-16 22:53:22,234 CRIT Server 'unix_http_server' running without any HTTP authentication checking

2016-03-16 22:53:22,247 INFO daemonizing the supervisord process

2016-03-16 22:53:22,248 INFO supervisord started with pid 8269

2016-03-16 22:54:12,680 CRIT Supervisor running as root (no user in config file)

2016-03-16 22:54:12,681 WARN No file matches via include "/edx/app/supervisor/conf.d/*.conf"

2016-03-17 02:55:47,841 CRIT Supervisor running as root (no user in config file)

2016-03-17 02:55:47,841 WARN No file matches via include "/edx/app/supervisor/conf.d/*.conf"

2016-03-17 02:58:42,529 CRIT Supervisor running as root (no user in config file)

2016-03-17 02:58:42,529 WARN No file matches via include "/edx/app/supervisor/conf.d/*.conf"

2016-03-17 03:01:52,498 CRIT Supervisor running as root (no user in config file)

2016-03-17 03:01:52,498 WARN No file matches via include "/edx/app/supervisor/conf.d/*.conf"

[Sheethal PN]

Hi Ned, 

The content in /etc/supervisor/supervisord.conf is 

; supervisor config file

[unix_http_server]

file=/var/run/supervisor.sock   ; (the path to the socket file)

chmod=0700                       ; sockef file mode (default 0700)

[supervisord]

logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)

pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)

childlogdir=/var/log/supervisor            ; ('AUTO' child log dir, default $TEMP)

; the below section must remain in the config file for RPC

; (supervisorctl/web interface) to work, additional interfaces may be

; added by defining them in separate rpcinterface: sections

[rpcinterface:supervisor]

supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]

serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket

; The [include] section can just contain the "files" setting.  This

; setting can list multiple files (separated by whitespace or

; newlines).  It can also contain wildcards.  The filenames are

; interpreted as relative to this file.  Included files *cannot*

; include files themselves.

[include]

files = /etc/supervisor/conf.d/*.conf