Supervisor service service-msgsender was unable to be started ERROR

[ronald....@seagate.com]

I modified the scalr-server.rb file to change the user session timeout and then ran /opt/scalr-server/bin/scalr-server-ctl reconfigure to restart with the new changes and got the following error:

[2017-01-12T13:37:41-06:00] ERROR: Running exception handlers

  - Raven::Chef::SentryHandler

Running handlers complete

[2017-01-12T13:37:42-06:00] ERROR: Exception handlers complete

[2017-01-12T13:37:42-06:00] FATAL: Stacktrace dumped to /opt/scalr-server/embedded/cookbooks/cache/chef-stacktrace.out

Chef Client failed. 53 resources updated in 36.9977899 seconds

[2017-01-12T13:37:42-06:00] ERROR: supervisor_service[service-msgsender] (scalr-server::group_service_always_post line 5) had an error: RuntimeError: Supervisor service service-msgsender was unable to be started

[2017-01-12T13:37:42-06:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Here is the stacktrace dump that it mentioned above:

cat /opt/scalr-server/embedded/cookbooks/cache/chef-stacktrace.out

Generated at 2017-01-12 13:37:42 -0600

RuntimeError: supervisor_service[service-msgsender] (scalr-server::group_service_always_post line 5) had an error: RuntimeError: Supervisor service service-msgsender was unable to be started

/opt/scalr-server/embedded/cookbooks/supervisor/providers/service.rb:49:in `block (2 levels) in class_from_file'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/mixin/why_run.rb:52:in `call'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/mixin/why_run.rb:52:in `add_action'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/provider.rb:165:in `converge_by'

/opt/scalr-server/embedded/cookbooks/supervisor/providers/service.rb:47:in `block in class_from_file'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/provider/lwrp_base.rb:138:in `instance_eval'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/provider/lwrp_base.rb:138:in `block in action'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/provider.rb:134:in `run_action'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/resource.rb:562:in `run_action'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/runner.rb:49:in `run_action'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/runner.rb:81:in `block (2 levels) in converge'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/runner.rb:81:in `each'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/runner.rb:81:in `block in converge'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/resource_collection/resource_list.rb:83:in `block in execute_each_resource'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/resource_collection/resource_list.rb:81:in `execute_each_resource'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/runner.rb:80:in `converge'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/client.rb:339:in `block in converge'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/client.rb:334:in `catch'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/client.rb:334:in `converge'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/client.rb:353:in `converge_and_save'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/client.rb:457:in `run'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application.rb:271:in `block in fork_chef_client'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application.rb:259:in `fork'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application.rb:259:in `fork_chef_client'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application.rb:225:in `block in run_chef_client'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/local_mode.rb:39:in `with_server_connectivity'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application.rb:213:in `run_chef_client'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application/solo.rb:280:in `block in interval_run_chef_client'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application/solo.rb:269:in `loop'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application/solo.rb:269:in `interval_run_chef_client'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application/solo.rb:247:in `run_application'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/lib/chef/application.rb:60:in `run'

/opt/scalr-server/embedded/lib/ruby/gems/2.1.0/gems/chef-12.3.0/bin/chef-solo:25:in `<top (required)>'

/opt/scalr-server/embedded/bin/chef-solo:23:in `load'

/opt/scalr-server/embedded/bin/chef-solo:23:in `<main>'

I then changed the scalr-server.rb file back to the way it was before and ran the reconfigure again but got the same result. Any ideas or suggestions what might be causing this?

Thanks,

Ron 

[Marc O'Brien]

Hi Ron,

This error should not actually be related to the scalr-server.rb changes but rather a failure to start a core service (service-msgsender) which is a part of the “reconfigure” command. Usually this is because there is a stuck or zombie process preventing the new service from starting.  For troubleshooting, first steps are:

1. Verify Scalr service status with: "scalr-server-manage status" .  We expect to see msgsender in FAILED status.

2. Stop all services with: "scalr-server-manage stop all"

3. Verify no running zombie Scalr processes with : "ps -aux | grep msgsender”  Modify grep command if necessary to check for other services.  If there is a stuck/zombie process, kill the zombie process by pid.

4. Reconfigure with: “scalr-server-ctl reconfigure" again as a Reconfigure should start all processes on successful execution.

Once the stuck process has been resolved we would expect to see reconfigure complete correctly once again.  If troubles persist the next step will be to look at your logs in /opt/scalr-server/var/log.  Let us know how it goes or if you have any questions.

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[ronald....@seagate.com]

Thanks for the reply. I did exactly as you said, there were a couple of other process stuck so I killed those, and then I ran reconfigure. It completed without a problem. Then I went back and changed the scalr-server.rb file back to adding in the user session timeout value, stopped it all and ran reconfigure and got different errors. I changed it back to how it was originally and get the same errors. Here is the output:

[2017-01-12T15:43:41-06:00] ERROR: Running exception handlers

[2017-01-12T15:43:42-06:00] WARN: ** [Raven] Error from Sentry server (400): {"error":"Bad data reconstructing object (JSONDecodeError, Invalid control character u'\\x1b' at: line 1 column 363 (char 362))"}

  - Raven::Chef::SentryHandler

Running handlers complete

[2017-01-12T15:43:42-06:00] ERROR: Exception handlers complete

[2017-01-12T15:43:42-06:00] FATAL: Stacktrace dumped to /opt/scalr-server/embedded/cookbooks/cache/chef-stacktrace.out

Chef Client failed. 41 resources updated in 25.581378827 seconds

[2017-01-12T15:43:42-06:00] ERROR: execute[Upgrade Scalr Database] (scalr-server::group_app_enabled_post line 42) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'

---- Begin output of /opt/scalr-server/embedded/bin/php upgrade.php ----

STDOUT: 43:41 - Cannot start a new process because another one has already been started.

STDERR: 

---- End output of /opt/scalr-server/embedded/bin/php upgrade.php ----

Ran /opt/scalr-server/embedded/bin/php upgrade.php returned 1

[2017-01-12T15:43:42-06:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

I'll send this just for now and will send the logs if you need them. Thanks.

[Marc O'Brien]

Hi Ron,

This is indeed different behavior as the error occurs on the upgrade DB step, but the root cause appears to be the same " STDOUT: 43:41 - Cannot start a new process because another one has already been started."

I am wondering if the same situation occurred following your successful reconfigure.  (Note: No need to stop all services between each reconfigure.  Typically only required when troubleshooting or upgrading Scalr) If you have the desired scalr-server.rb settings in place now, are you able to repeat the same resolution steps from before and reconfigure?

Many thanks,
Wm. Marc O'Brien
Scalr Technical Support

[ronald....@seagate.com]

It didn't occur the very first time I tried, but I had removed my changes to the scalr-server.rb file before that. I added the changes back and started to get this new error, so I backed those changes back out again but still get the error. I seem to get it no matter what.