Problem with hypnotoad

[Joseph Fridy]

I have recently transitioned from solo development to beta testing for a Mojolicious::Lite application.  Upon this transition, I switched from my casual use of morbo to hypnotoad.  In order to not change my nginx configuration, I had hypnotoad listen on port 3000.  The application appeared and I made my beta testers aware of its existence and went to bed.  Some time the next day, I got word of failures.  When I tried to connect to the application, I could connect but I was unable to access database objects correctly.  The Mojolicious::Lite script is named setupTransfer.pl.  There were many tens of processes named setupTransfer.pl running on the server.  I attempted hypnotoad --stop setupTransfer.pl, but it failed with an error on Time::Piece.  I rather inelegantly killed all the setupTransfer.pl processes and hypnotoad.  I checked my script for errors, and found nothing salient.  I restarted hypnotoad, and all appeared to be working, but within a few hours setupTransfer.pl processes started to proliferate, apparently without bound.  I have stopped and restarted hypnotoad several times since.  For fear that this is a consequence of some confusion because of making hypnotoad listen on port 3000, I have changed my nginx configuration and switched to port 8080.  Clearly I am ignorant, and doing something stupid, but cursory documentation checks have not pointed out my error.  hypnotoad starts with an initial inventory of setupTransfer.pl processes of 9, and they appear to increase monotonically with activity.  The server (a fairly lightweight ec2 instance) starts to exhibit problems somewhere in the 60 or 70 range. 

What stupid thing am I doing?

Regards,

Joe Fridy

[Scott H]

I don't know if this helps, I have this in my config file.

hypnotoad => {

listen => ['http://*:3000'],

pid_file => 'hypnotoad.pid',

workers => 10,

spare => 5,

proxy => 1

},

here is my systemd file to start it:

[Unit]

Description=MyMojoApp

After=network.target

User=scott

Group=scott

[Service]

User=scott

Group=scott

Type=forking

PIDFile=hypnotoad.pid

ExecStart=/usr/local/bin/carton exec hypnotoad /home/scott/mojoapp/script/myapp

ExecReload=/usr/local/bin/carton exec hypnotoad /home/scott/mojoapp/script/myapp

ExecStop=/usr/local/bin/carton exec hypnotoad /home/scott/mojoapp/script/myapp

KillMode=process

WorkingDirectory=/home/scott/mojoapp

[Install]

WantedBy=multi-user.target

--
You received this message because you are subscribed to the Google Groups "Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/CA%2Bj3PDg%3DTdMhCWnh6aNHenRX-HJcJTCgeQL5rK9riRHq1gHoUQ%40mail.gmail.com.

[Dan Book]

Make sure your actions aren't going to hang up for excessive amounts of time, or the manager process will attempt to replace it (possibly causing your excessive forking). See https://metacpan.org/pod/Mojo::Server::Hypnotoad#heartbeat_timeout (defaults to 50 seconds)

Make sure you are invoking hypnotoad using its full path, and that it is the hypnotoad installed for the same Perl you've installed all your dependencies to.

Make sure you are using a fork safe database connection manager like DBIx::Connector or Mojo::Pg, and not attempting to share any live database connections across forks (e.g. the body of the Lite script will run in the manager process before workers are forked from it - request handlers should retrieve a new connection from your connection manager so they can connect anew if needed).

-Dan

[Joseph Fridy]

I am assuming that the first caveat suggests I should increase the timeouts?  This seems unlikely to be the cause, because the monotonic increase begins immediately - every contact appears to spawn a process that does not go away.  Increases in process count only happen when someone visits the site.  My database connections are accomplished by scripting the aws command line interface - I have no persistent database connections.  This is behaving like every connection to hypnotoad leaves a process sitting around forever.  

I have increased the timeouts, without much hope or expectation that that will help.  Preliminary observations suggest that it does not.

Regards,

Joe Fridy

To view this discussion on the web visit https://groups.google.com/d/msgid/mojolicious/CABMkAVUFWqd_So2wiY-M4rkzWbTX4U%2BZb_pywNY33rFWzQRDGw%40mail.gmail.com.

[Joseph Fridy]

This target:

any '/publicKey' => sub {
  my $c = shift;

  my $publicKey = `gpg --armor --export $FromAddress`;

  $c->render(text => "<pre>$publicKey</pre>");
  censorLogs();

};

When visited will increase the count of processes by one.  Apparently forever.

regards,

Joe Fridy

[Joseph Fridy]

The problem was in censorLogs - it forked a sub-process and failed to wait for its conclusion.  I eliminated the fork and the proliferation of processes seems to have stopped.

Thank you for your attention.  As expected, I was doing something stupid.

Regards,

Joe Fridy