Watchdog doesn't restarts service after reboot

[blinov.s...@gmail.com]

Greetings,

I have ubic-watchdog in crontab jobs, but if I reboot the server, my service doesn't starts automatically. I need to manually invoke ubic restart ... every time.

How can I make watchdog to run service at startup?

[Vyacheslav Matyukhin]

Hi!

 

It means your environment when you start the service manually doesn't correspond with the environment of cron daemon.

 

The most popular cause of this issue is that when you install ubic from CPAN, 'ubic' and 'ubic-watchdog' scripts are installed in /usr/local/bin/, but cron's default PATH is /bin:/usr/bin.

 

If that's the case, upgrading to the latest Ubic release might help (you might need to reconfigure your crontab with 'ubic-admin setup --reconfigure').

Starting from 1.48, setup process puts full path to ubic-watchdog script in cron job, and starting from 1.52, ubic-watchdog always prepends PATH with /usr/local/bin.

There were also some improvements with perlbrew support.

 

Note that these improvements still don't guarantee to fix all environment-related issues. For example, if your service depends on the environment variable FOO to be set, and if you set FOO in your ~/.bash_profile and always start the service with 'sudo ubic start my_daemon', then watchdog won't be able to start your service, because it never loads your bash_profile.

This is something you should track yourself, and if your service depends on such configuration, you should provide it in your service config. (E.g., if you're using SimpleDaemon, pass it in 'env' option.)

 

Hope this helps.

PS: If the problem still persists, you'll have to reveal more details, e.g. content of your /var/log/ubic/watchdog.log.

 

09.10.2013, 18:07, "blinov.s...@gmail.com" <blinov.s...@gmail.com>:

Greetings,

I have ubic-watchdog in crontab jobs, but if I reboot the server, my service doesn't starts automatically. I need to manually invoke ubic restart ... every time.

How can I make watchdog to run service at startup?

 

--
You received this message because you are subscribed to the Google Groups "Ubic" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ubic-perl+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[blinov.s...@gmail.com]

Hi!

I have installed latest Ubic from CPAN and reconfigured, but the problem still the same.

After reboot /var/log/ubic/watchdog.log contains only one line:

[Thu Oct 10 07:47:03 2013] ubic.watchdog status is 'not running', restarting

If I run 'ubic status' I got:
site off
ubic
ubic.ping off
ubic.update off
ubic.watchdog running (pid 519)

So I don't understand why if service 'site' is off, watchdog doesn't restarts it? If I manually run 'ubic-watchdog' it still doesn't restarts service.

среда, 9 октября 2013 г., 19:01:38 UTC+4 пользователь Vyacheslav Matyukhin написал:

[Иван Бессарабов]

Status "off" mean that is has been correctly stopped. In this state the  servis will not be automaticly restarted. Try staring that servis and then killing it. Then the servic will be in the state "Not running" which means that it has died unexpecticly. And the services in "Not running" will be restarted by watchdog.

[blinov.s...@gmail.com]

Ok, but how can I make a service to start automatically after reboot?

I have created a script in /etc/init.d

where I wrote ubic restart site

but it doesn't works.

четверг, 10 октября 2013 г., 12:20:00 UTC+4 пользователь Иван Бессарабов написал:

[Vyacheslav Matyukhin]

Read https://metacpan.org/module/Ubic::Manual::FAQ#How-is-ubic-compatible-with-SysV-style-etc-rcX.d-symlinks.

You don't have to write init scripts with ubic. Start your service once, and ubic will always try to keep it running. Stop your service once, and it will stay down after the reboot.

 

Think of this start/stop status as a desired state of the service. This state is saved persistently (in /var/lib/ubic/status/ dir), and whenever ubic-watchdog notices that the service is down when is should be up, it brings it up. This happens both when your daemon dies *and* after reboots.

 

10.10.2013, 16:18, "blinov.s...@gmail.com" <blinov.s...@gmail.com>:

[blinov.s...@gmail.com]

I still cannot get it to work.

If I run manually 'ubic-watchdog' after reboot, my service starts, but automatically it doesn't.

I have following line in crontab: 

* * * * * bash -c 'source /root/perl5/perlbrew/etc/bashrc && /root/perl5/perlbrew/perls/perl-5.16.2/bin/ubic-watchdog ubic.watchdog' >>/var/log/ubic/watchdog.log 2>>/var/log/ubic/watchdog.err.log

Maybe it is not configured correctly?

Cron it self is working OK.

четверг, 10 октября 2013 г., 16:44:36 UTC+4 пользователь Vyacheslav Matyukhin написал:

[Vyacheslav Matyukhin]

You're using perlbrew with system-wide installation, interesting. (Not necessarily wrong, but I never tried it.)

 

What do you see in /var/log/ubic/watchdog.log and /var/log/ubic/watchdog.err.log after reboot?

 

If you don't run 'ubic-watchdog' manually, what do you see in 'ubic status'?

Something like this?

site   not running

ubic.watchdog    not running        <--- I'm trying to figure out if watchdog service fails to start, or if it starts but then fails to start 'site' because it inherits the wrong environment

 

10.10.2013, 19:49, "blinov.s...@gmail.com" <blinov.s...@gmail.com>:

[blinov.s...@gmail.com]

Ok, this what I have done:

root@vps-0022:~# ubic status
site running

ubic
      ubic.ping off
      ubic.update off
      ubic.watchdog running (pid 519)

root@vps-0022:~# reboot

root@vps-0022:~# cat /var/log/ubic/watchdog.log 

[Thu Oct 10 07:47:03 2013] ubic.watchdog status is 'not running', restarting

[Thu Oct 10 12:17:03 2013] ubic.watchdog status is 'not running', restarting
[Thu Oct 10 12:17:04 2013] site status is 'not running', restarting
[Thu Oct 10 12:17:04 2013] Failed to revive site: not running
[Thu Oct 10 15:27:04 2013] ubic.watchdog status is 'not running', restarting
[Thu Oct 10 15:27:04 2013] site status is 'not running', restarting
[Thu Oct 10 15:27:04 2013] Failed to revive site: not running
[Thu Oct 10 15:28:05 2013] site status is 'not running', restarting
[Thu Oct 10 15:28:05 2013] Failed to revive site: not running
[Thu Oct 10 15:29:05 2013] site status is 'not running', restarting
[Thu Oct 10 15:29:05 2013] Failed to revive site: not running
[Thu Oct 10 15:31:01 2013] ubic.watchdog status is 'not running', restarting
[Thu Oct 10 15:32:02 2013] ubic.watchdog status is 'not running', restarting
[Thu Oct 10 15:33:06 2013] ubic.watchdog status is 'not running', restarting
[Thu Oct 10 15:33:06 2013] site status is 'not running', restarting
[Thu Oct 10 15:33:07 2013] Failed to revive site: not running
[Thu Oct 10 15:39:04 2013] ubic.watchdog status is 'not running', restarting
[Thu Oct 10 15:39:04 2013] site status is 'not running', restarting
[Thu Oct 10 15:39:04 2013] Failed to revive site: not running
[Thu Oct 10 15:40:05 2013] site status is 'not running', restarting
[Thu Oct 10 15:40:05 2013] Failed to revive site: not running
[Thu Oct 10 15:41:04 2013] site status is 'not running', restarting
[Thu Oct 10 15:41:04 2013] Failed to revive site: not running
[Fri Oct 11 06:35:04 2013] ubic.watchdog status is 'not running', restarting
[Fri Oct 11 06:35:04 2013] site status is 'not running', restarting
[Fri Oct 11 06:35:04 2013] Failed to revive site: not running
[Fri Oct 11 06:36:04 2013] site status is 'not running', restarting
[Fri Oct 11 06:36:04 2013] Failed to revive site: not running

(So the error is constantly the same, but totally unclear what is happening)

root@vps-0022:~# cat /var/log/ubic/watchdog.err.log

(nothing)

root@vps-0022:~# ubic start site
Starting site... started
(!!!)

root@vps-0022:~# cat /etc/ubic/service/site
use Ubic::Service::Plack;
return Ubic::Service::Plack->new({
server => 'Starman',

server_args => {workers =>1, env => 'production'},
app => "/var/www/script/simple_c_m_s",
app_name => 'site',
port => 3000,
ubic_log => '/var/www/logs/ubic.log',
stdout => '/var/www/logs/stdout.log',
stderr => '/var/www/logs/stderr.log',
user => 'www',
});

(Maybe it can help)

I haven't clearly understood what is "perlbrew with system-wide installation", I have just installed perlbrew according to manual. I can try to reinstall it another way if it will help to get Ubic to work.

четверг, 10 октября 2013 г., 20:16:27 UTC+4 пользователь Vyacheslav Matyukhin написал:

[blinov.s...@gmail.com]

If I run 'ubic status' just after reboot, I see

root@vps-0022:~# ubic status
site not running

ubic
      ubic.ping off
      ubic.update off

      ubic.watchdog running (pid 518)

[Vyacheslav Matyukhin]

Hello again, sorry for the late answer.

 

You should check your service logs:

ubic_log => '/var/www/logs/ubic.log',

stdout => '/var/www/logs/stdout.log',

stderr => '/var/www/logs/stderr.log',

 

It should help you figure out what's wrong.

 

PS:

When I said "perlbrew with system-wide installation", I didn't mean "system-wide installation of perlbrew", I meant system-wide installation of Ubic.

It was surprising to me, because I thought most people:

1) either install perlbrew and Ubic locally (there's a question prompting you to make a choice when you run 'ubic-admin setup' for the first time)

2) or use system perl and install Ubic into /etc

This was a wrong assumption, as I see now. But I don't see any issues with using it this way, so it's probably irrelevant.

 

11.10.2013, 10:53, "blinov.s...@gmail.com" <blinov.s...@gmail.com>:

--

[blinov.s...@gmail.com]

Hello,

I have finally found the error in stderr.log:

Can't exec "plackup": Permission denied at /root/perl5/perlbrew/perls/perl-5.16.2/lib/site_perl/5.16.2/Ubic/Daemon.pm line 387

Do you have idea on how to quickly resolve it?

понедельник, 14 октября 2013 г., 21:32:07 UTC+4 пользователь Vyacheslav Matyukhin написал:

[blinov.s...@gmail.com]

I have finally discovered that Ubic works after reboot only if run the service as root. I wonder if it can lead to any security problems? I wanted to run it as 'www', but I got 'Permission denied'.

I tried to reproduce the error switching to 'www' in terminal and running:

perl

exec ('plackup --workers 1 --env development --server Starman --port 3000 /var/www/script/simple_c_m_s');

But this way all is working ok.

Any idea?

пятница, 25 октября 2013 г., 13:09:48 UTC+4 пользователь blinov.s...@gmail.com написал:

[Vyacheslav Matyukhin]

Maybe 'www' user doesn't have reading permissions for the dir where plackup script is installed?

 

Your terminal attempt might have worked because PATH was different from what's configured for Ubic. Did you source perlbrew configs before trying that exec?

/root dir usually have chmod 0700, so if your perl modules and scripts are installed in /root/perl5/..., then it's not surprising.

 

29.10.2013, 18:16, "blinov.s...@gmail.com" <blinov.s...@gmail.com>: