Hypnotoad worker hard stop issues

[sean.q...@endurance.com]

We have a collection of 'micro services' running with hypnotoad, which all share the same framework code. We've been running this for about two years now, periodically adding new services. Outside of local development all our services are behind an nginx proxy, which routes to the appropriate service in the collection. This has worked fantastically well so far.

But in our newest service, while testing in our beta environment, we started noticing an odd behavior. We would occasionally get a 405 error response on calls, including calls from shared code that should never fail unexpectedly from test code. Watching the log we observed the following entries always corresponded with the error response:

[Thu Jun 30 12:21:47 2016] [debug] Worker 34303 stopped

[Thu Jun 30 12:21:47 2016] [debug] Worker 34442 started 

Playing the with configuration value for 'accepts' increased or decreased the frequency with which we see this error. Using hypnotoad locally and a low accepts value I am able to easily replicate this, and doing tests with curl it appears the worker is hanging up unexpectedly after accepting the connection:

Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)

> GET /account HTTP/1.1

> Host: 127.0.0.1:3000

> User-Agent: curl/7.43.0

>

* Empty reply from server

* Connection #0 to host 127.0.0.1 left intact

curl: (52) Empty reply from server

We haven't done anything obviously different with this service than with any of the others where we do not appear to be having this issue.

1. Why does hypnotoad seem to accept the connection, but then kill the Worker handling the request immediately (un-gracefully) before it finishes responding?
2. Is there any additional debugging I could or should do to gain more information?
1. We thought at first out service was dieing in some new and unusual way, but we see nothing in the application log to indicate it ever even started handling the request. 
2. If something is happening to generate STDERR output, would that end up in the mojo log where we see the Worker start/stop etc? I notice the STDERR output we see in the terminal using morbo (prefork), as we usually do for local development, appears to be missing from the mojo service log; where does it go?
3. Tailing the service log for other instances I do see the Worker stopped/start entries, but no empty replies from the service (restart appears graceful as expected); what might influence this behavior difference in our new service?
1. FYI when replicating locally all my services are symlinked to the same moniker.conf file They also all inherit from the same base class for their startup code.

Any suggestions for how to resolve this issue would be most welcome! While I can replicate it I can not see anything different in this one service that might cause this behavior difference?!?!

-Regards,

Sean

[sri]

Always try the latest Mojolicious release first, this has already been fixed.

--

sebastian

[sri]

Always try the latest Mojolicious release first, this has already been fixed.

It's really crazy how often this bug comes up.

    https://github.com/kraih/mojo/commit/80a4f3fd98f673abe65b223e401f5e3d5de554e7

For future reference.

--

sebastian 

[Sean Quinlan]

Excellent, that appears to have fixed the issue! And thank you for the link to the bug.

While I will try to remember to upgrade mojo on my dev box when replicating before posting (we use perlbrew and try to keep fairly recent, but do have a controlled upgrade flow), what more could I have done to track down the error? Where does the STDERR output go under hypnotoad, in case that would be helpful? Is there a howto I've missed somewhere with more debugging guidance?

-Thanks,

Sean

--
You received this message because you are subscribed to a topic in the Google Groups "Mojolicious" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mojolicious/MG8GYSjEB9Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mojolicious...@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.