hypnotoad systemd start problem

[davi...@gmail.com]

Dear sir;

     I meet a problem when systemd start hypnotoad, detail follow:

systemctl status mojo_test.service

● mojo_test.service - My Mojolicious application

   Loaded: loaded (/usr/lib/systemd/system/mojo_test.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Thu 2017-11-23 14:57:05 CST; 7s ago

  Process: 11559 ExecStart=/usr/local/bin/hypnotoad /home/mojo_test.pl (code=exited, status=2)

Nov 23 14:57:05 localhost.localdomain hypnotoad[11559]: Compilation failed in require at /usr/local/share/perl5/Mojo/File.pm line 20.

Nov 23 14:57:05 localhost.localdomain hypnotoad[11559]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/File.pm line 20.

Nov 23 14:57:05 localhost.localdomain hypnotoad[11559]: Compilation failed in require at /usr/local/share/perl5/Mojo/Server/Hypnotoad.pm 

line 7.

Nov 23 14:57:05 localhost.localdomain hypnotoad[11559]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/Server/Hypnotoad.pm 

line 7.

Nov 23 14:57:05 localhost.localdomain hypnotoad[11559]: Compilation failed in require at /usr/local/bin/hypnotoad line 7.

Nov 23 14:57:05 localhost.localdomain hypnotoad[11559]: BEGIN failed--compilation aborted at /usr/local/bin/hypnotoad line 7.

Nov 23 14:57:05 localhost.localdomain systemd[1]: mojo_test.service: control process exited, code=exited status=2

Nov 23 14:57:05 localhost.localdomain systemd[1]: Failed to start My Mojolicious application.

Nov 23 14:57:05 localhost.localdomain systemd[1]: Unit mojo_test.service entered failed state.

Nov 23 14:57:05 localhost.localdomain systemd[1]: mojo_test.service failed.

mojo_test.service

[Unit]

Description=My Mojolicious application

Requires=network.target

After=network.target

[Service]

Type=forking

PIDFile=/run/mojo_test.pid

ExecStart=/usr/local/bin/hypnotoad /home/mojo_test.pl

ExecStop=/usr/local/bin/hypnotoad -s /home/mojo_test.pl

ExecReload=/usr/local/bin/hypnotoad /home/mojo_test.pl

KillMode=process

[Install]

WantedBy=multi-user.target

use /home/hypnotoad mojo_test.pl start to run is ok. 

the running enviorments:

centos 7

perl v5.16

mojolicious new update

please give some guides, thank you!

David

[b.du...@keele.ac.uk]

Hi David,

I'm no expert with systemd, but this is my mojo.service file which runs the application in /usr/local/manager-application and works for me.   My differences are in the Service tag.

mojo.service

[Unit]

Description=My Mojolicious application

Requires=network.target

After=network.target

[Service]

Type=simple

RemainAfterExit=yes

SyslogIdentifier=mojolicious

WorkingDirectory=/usr/local/manager-application

ExecStart=/usr/local/bin/hypnotoad script/manager_app -f

ExecStop=/usr/local/bin/hypnotoad --stop script/manager_app

ExecReload=/usr/local/bin/hypnotoad script/manager_app

Restart=on-abort

RestartSec=90s

[Install]

WantedBy=multi-user.target

running
CentOS 7
perl 5.16.3
mojolcious 6.58

(file transcribed by hand, so watch for speling mistakes)  HTH

Boyd

On Thursday, 23 November 2017 09:24:40 UTC, davi...@gmail.com wrote:

Dear sir;

     I meet a problem when systemd start hypnotoad, detail follow:

[Dan Book]

There is probably more relevant information higher in the log; the status command is only showing you the end of the log. Try running: journalctl -u mojo_test.service

-Dan

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscribe@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[Dan Book]

I would not recommend running hypnotoad as a Type=simple service with -f. This will not allow you to use hot-restarts via systemctl reload.

-Dan

[Boyd Duffee]

On 23/11/17 16:14, Dan Book wrote:
> I would not recommend running hypnotoad as a Type=simple service with -f.
> This will not allow you to use hot-restarts via systemctl reload.

Naively, I _can_ hot-restart with systemctl reload servicename, but now
you've got me worried.
Any idea why it's not advisable and is Type=forking the proper solution?

--
Boyd Duffee Keele University (01782) 734225
DevOps - Student Facing Systems

Some people are born on third base and go through life
thinking they hit a triple - Barry Switzer

[Daniel Mantovani]

Hi,

Type=forking should work just fine, I am not sure that with Type=simple you will have all the functionallity you can get from systemd.

A couple of advices though:

1) to get more confortable with systemd, you probably want to replicate as much as possible the example in Mojolicious::Guides::Cookbook. It just works very well (https://metacpan.org/pod/distribution/Mojolicious/lib/Mojolicious/Guides/Cookbook.pod#Hypnotoad)

2) But if you need to have pidfile and log files outside your working directory (like in /run/your_app/hypnotoad.pid and /var/log/your_app/production.log etc...), this simple approach will not work, because you probably don't want to run your application with permissions so high it will be able to write in /run or /var/log directories. Instead you normally use a user with minimal permissions.

Besides as far as I know hypnotoad requires to have permission rights in an already created directory for the pid file and also for the log file.

If that is the case, what you can do is modify your original service file as follows:

(I am supossing here that "/home/mojo_test.pl" is a typo, I guess if your user is "mojo_user" for instance, the path to the app will be something like "/home/mojo_user/mojo_test.pl")

mojo_test.service

[Unit]

Description=My Mojolicious application

Requires=network.target

After=network.target

[Service]

Type=forking

PIDFile=/run/mojo_test/hypnotoad.pid

User=mojo_user

Group=mojo_user

PermissionsStartOnly=True

ExecStartPre=/usr/bin/install -o mojo_user -g mojo_user -d /run/mojo_test

ExecStartPre=/usr/bin/install -o mojo_user -g mojo_user -d /var/log/mojo_test

ExecStart=/usr/local/bin/hypnotoad /home/mojo_user/mojo_test.pl

ExecReload=/usr/local/bin/hypnotoad /home/mojo_user/mojo_test.pl

KillMode=process

[Install]

WantedBy=multi-user.target

Please check that in your configuration file you have the pid set as /run/mojo_test/hypnotoad.pid, and that when

you define the path for the log file it goes inside /var/log/mojo_test directory, something like

sub startup {

    ...

    $self->app->log->path('/var/log/mojo_test/production.log');

    ...

}

About the zero downtime hypnotoad feature, with reload it should work without any problems (you can check that in the log file). If you restart instead of course it will kill the process and then start again, and you can check that also in the log file.

One last thing, if you happen to have a recent version of systemd or you are allowed to install that (not my case unfortunatelly), you should check RuntimeDirectory= and LogsDirectory= directives. You can check if those are supported in your system with 

$ man systemd.directives

If you have those, you will probably not need the ExecStartPre directives above, because the only thing these do is to create those directories.

Hope it helps.

BR,

Daniel

[Dan Book]

The best practice IMO in regards to permissions for the run and log directories is to make subdirectories of /var/run and /var/log owned by the user that will be running hypnotoad, and put the pidfile/logfiles in there.

The other issue though is if you are running hypnotoad to listen on a privileged port like 80 or 443, then it needs to start as root or under a user with the appropriate capabilities assigned (I'm not sure on the details for this option). You can use https://metacpan.org/pod/Mojolicious::Plugin::SetUserGroup to drop permissions after starting as root.

-Dan

--

[Dan Book]

The way that hot-restart works is that the USR2 signal is sent to the currently running manager, which itself will spawn a new manager and then wait for it to be ready, then exit. I don't think that Type=simple will correctly track the new process when this happens since the new process was not spawned by systemd, but I could be wrong, I haven't tested this since hypnotoad's pidfile behavior was fixed to work better for systemd. But in any case unless there's some other benefit to running it in foreground mode, I still think it's better to use its "default" forking mode.

-Dan

[davi...@gmail.com]

Hi Daniel,

Thank you deltail adivces.

I test follow your guide, but it's error same as I report before, it's in Mojolicious 7.57.

I once change Mojolicious to 6.58 to test, it's still error, error output:

[root@localhost ~]# systemctl status mojo_test.service

● mojo_test.service - My Mojolicious application

   Loaded: loaded (/usr/lib/systemd/system/mojo_test.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Fri 2017-11-24 18:23:20 CST; 6s ago

  Process: 2822 ExecStart=/root/perl5/bin/hypnotoad /home/mojo_user/mojo_test.pl (code=exited, status=2)

  Process: 2820 ExecStartPre=/usr/bin/install -o mojo_user -g mojo_user -d /var/log/mojo_test (code=exited, status=0/SUCCESS)

  Process: 2819 ExecStartPre=/usr/bin/install -o mojo_user -g mojo_user -d /run/mojo_test (code=exited, status=0/SUCCESS)

Nov 24 18:23:20 localhost.localdomain hypnotoad[2822]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/Server/Daemon.pm line 2.

Nov 24 18:23:20 localhost.localdomain hypnotoad[2822]: Compilation failed in require at /usr/local/share/perl5/Mojo/Base.pm line 77.

Nov 24 18:23:20 localhost.localdomain hypnotoad[2822]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/Server/Prefork.pm line 2.

Nov 24 18:23:20 localhost.localdomain hypnotoad[2822]: Compilation failed in require at /usr/local/share/perl5/Mojo/Server/Hypnotoad.pm line 9.

Nov 24 18:23:20 localhost.localdomain hypnotoad[2822]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/Server/Hypnotoad.pm line 9.

Nov 24 18:23:20 localhost.localdomain hypnotoad[2822]: Compilation failed in require at /root/perl5/bin/hypnotoad line 20.

Nov 24 18:23:20 localhost.localdomain systemd[1]: mojo_test.service: control process exited, code=exited status=2

Nov 24 18:23:20 localhost.localdomain systemd[1]: Failed to start My Mojolicious application.

Nov 24 18:23:20 localhost.localdomain systemd[1]: Unit mojo_test.service entered failed state.

Nov 24 18:23:20 localhost.localdomain systemd[1]: mojo_test.service failed.

David

在 2017年11月24日星期五 UTC+8上午8:45:57，Daniel Mantovani写道：

[davi...@gmail.com]

Hi Boyd, thank you suggestion.

I test that use your service unit configuration, result same error (mojolicious 7.57).

I also change Mojolicious to 6.58, it's still error, but it's different, error output is:

[root@localhost ~]# systemctl status mojo

● mojo.service - My Mojolicious application

   Loaded: loaded (/usr/lib/systemd/system/mojo.service; enabled; vendor preset: disabled)

   Active: active (exited) (Result: exit-code) since Fri 2017-11-24 17:26:52 CST; 1min 29s ago

  Process: 1104 ExecStart=/root/perl5/bin/hypnotoad /usr/local/manager-application/script/manager_app -f (code=exited, status=2)

 Main PID: 1104 (code=exited, status=2)

   CGroup: /system.slice/mojo.service

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: Compilation failed in require at /usr/local/share/perl5/Mojo/Server.pm line 6.

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/Server.pm line 6.

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: Compilation failed in require at /usr/local/share/perl5/Mojo/Base.pm line 77.

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/Server/Daemon.pm line 2.

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: Compilation failed in require at /usr/local/share/perl5/Mojo/Base.pm line 77.

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/Server/Prefork.pm line 2.

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: Compilation failed in require at /usr/local/share/perl5/Mojo/Server/Hypnotoad.pm line 9.

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: BEGIN failed--compilation aborted at /usr/local/share/perl5/Mojo/Server/Hypnotoad.pm line 9.

Nov 24 17:26:52 localhost.localdomain mojolicious[1104]: Compilation failed in require at /root/perl5/bin/hypnotoad line 20.

Nov 24 17:26:52 localhost.localdomain systemd[1]: mojo.service: main process exited, code=exited, status=2/INVALIDARGUMENT

I surprise that why your Hypnotoad of Mojolicious 6.58 is in usr/local/bin/, mine is in /root/perl5/bin, we are all use centos 7 and perl v5.16.3, it's same.

David

在 2017年11月23日星期四 UTC+8下午11:13:37，b.du...@keele.ac.uk写道：

[Daniel Mantovani]

Hi David, those 'Compilation failed in require at ... ' normally mean that perl is not able to find some modules at an "use" or "require" function call. You probably have to review those files at the lines mentioned, also which perl are using in each case (whether you are using system perl, or better installed your own user perl with perlbrew, or with plenv), and probably what directories are in your @INC array in each case. 

There are several ways to install modules in perl, and it is possible that when you run the script from inside the Unit file there are different versions of @INC leading to different directories an missing libraries in one of the cases.

[Daniel Mantovani]

Hi Dan, thanks for your comments. The problem I run into when tried to create the /run/myapp or /var/run/myapp directory manually or by a separate mechanism is that depending on the host system it will be deleted during a reboot. For instance in RHEL 7.x /var/run (or /run, it is a symlink to it actually) by design will delete all subdirectories during reboot.

BR,

Daniel

-Dan

To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.

[Dan Book]

Please check journalctl to find the real error that occurs before the lines you pasted. journalctl -u mojo_test.service

-Dan

--

[davi...@gmail.com]

Hi Daniel,

     Thank you gudie, but it's running ok when manually  start.

     I try to check the perl.

David

在 2017年11月24日星期五 UTC+8下午9:58:54，Daniel Mantovani写道：

[davi...@gmail.com]

Hi Dan,

     Thank you help, the error output that is result of run  journalctl -u mojo_test.

David

在 2017年11月25日星期六 UTC+8上午12:14:30，Dan Book写道：

To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.

[Daniel Mantovani]

Hi David,

Exactly, thats why you should check what are the differences among both cases (manual start vs systemd start). That means checking perl that end up called in each case (probably looking at the shebang line at invoqued hypnotoad), relevant environment variables, running directory, users permissions to access needed files, and as I said before, what is the content of the @INC array in each case.

Or you could install your own perl version (newer that the 5.16 you mention in the original entry) and all perl modules related to your code, and assure that your manual setup and the sytemd uses this controlled environment (for instance, make sure that running "which hypnotoad" from where you run the manual start gives you the same path you use in your service file). To go this way, you should use perlbrew or plenv. Here is a link with some explanations on using the latter http://www.dagolden.com/index.php/2390/setting-up-a-perl-development-environment-with-plenv/, that I found usefull.

Hope it helps,

BR,

Daniel

[davi...@gmail.com]

Hi Daniel,

 

Thank you very much.

David

在 2017年11月26日星期日 UTC+8下午10:02:43，Daniel Mantovani写道：

[Dan Book]

I just came across another solution for the /var/run directory for RHEL 7.x or any other system using systemd in this way:

Create a file in /etc/tmpfiles.d like /etc/tmpfiles.d/myapp.conf containing this line:

d /var/run/myapp 775 my_user my_user

Replacing myapp with the /var/run directory you want to create, and my_user with the user and group that will be using the directory. This directory should be automatically created whenever /run is recreated, and you can do it manually by running:

sudo systemd-tmpfiles --create /etc/tmpfiles.d/myapp.conf

To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscribe@googlegroups.com.