problem with nginx and puma - runs ok and then stops hours later

[Allen Maxwell]

I’m having a consistent problem with a server I set up and hope someone can point me in the right direction.  Haven’t been able to find anything helpful on Google so far.

Server is ubuntu VM…  rails, nginx and puma.  It’s a redmine app if that makes a difference…

When I start things up it seems to work great.  no problems, no errors in logs.  All good.

It runs for a few hours (not sure exactly how long at this point but a while…)

Then, I get a few errors like the following right in a row…

2016/02/26 20:30:02 [crit] 1401#0: *520 connect() to unix:/home/deploy/sites/mmre-redmine/shared/sockets/puma.sock failed (2: No such file or directory) while connecting to upstream, client: 67.2.45.216, server: localhost, request: "GET /favicon.ico HTTP/1.1", upstream: "http://unix:/home/deploy/sites/mmre-redmine/shared/sockets/puma.sock:/500.html", host: "mmre-projects.com", referrer: "http://mmre-projects.com/projects/re-vision-tablet-project/boards"

and all of a sudden I start getting 502 Gateway errors.  

If I restart puma (which is still running) it is all good again for a few hours.

I’ve set up a number of servers on DigitalOcean the same as this one and haven’t had any problems like this.  Any hints out there?

Thanks,

Max

[Michael Ries]

Allen, not sure if this will turn out to be your issue, but I saw a similar symptom a while ago.

In my case it turned out to be a case where if a few big requests came in around the same time the puma server would run out of heap space and start thrashing its GC. That made it unresponsive when nginx tried to open a connection.

Perhaps tracking some collectd stats or adding some stats checking inside the ruby VM could help identify if that is the same cause I saw.

--
http://utruby.org
http://groups.google.com/group/urug
 
please prefix the subject with [JOB] when regarding job opportunities
---
You received this message because you are subscribed to the Google Groups "Utah Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to urug+uns...@googlegroups.com.
To post to this group, send email to ur...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Allen Maxwell]

I suppose it’s possible but I have my doubts.  This server has a single app on it which is being accessed by a total of 7 people right now - and I know of only one who has actually accessed it other than me.

I’ll look through the logs a bit deeper and see what I can find.  There may be a pattern.  I suppose I could set up a cron job and have it restart puma when the server doesn’t answer a request… but that just seems silly (and a bandaid over an unidentified problem).

Thanks Michael

[Charles Wood]

Also use htop or similar to check memory usage before you restart puma. If I remember correctly, when the Ruby VM runs GC, it doesn't release allocated memory. (It does use its currently allocated memory for its heap before asking the OS for more.) 

If it's hit a memory limit, it may be swapping to disk and slowing down. 

I've also seen people use their process monitoring or cron to restart puma or similar after a couple hours just to avoid this sort of thing.

ᐧ

[sellis]

I second what Charles said. If this is mission-critical to stay alive at all times while you are tracking down the source of the problem, installing something like Monit will at least get the application immediately restarted so that you don't have long spells of outage.

It might also be a good idea to install new relic because it can give you transactional tracking, memory problem analysis, and more. It gives great inspection to see what's going on when sometimes logs do not.

[Michael Utley]

What version of ruby are you running? I know certain ruby 2.1 versions have had memory issues. As noted here. http://stackoverflow.com/questions/25756491/rails-app-running-on-puma-and-nginx-keeps-dying-every-few-hours-with-bad-gateway

On Feb 28, 2016, at 8:41 AM, sellis <stephen...@gmail.com> wrote:

I second what Charles said. If this is mission-critical to stay alive at all times while you are tracking down the source of the problem, installing something like Monit will at least get the application immediately restarted so that you don't have long spells of outage.  

It might also be a good idea to install new relic because it can give you transactional tracking, memory problem analysis, and more. It gives great inspection to see what's going on when sometimes logs do not.

[Allen Maxwell]

thanks Mike - I’m running 2.1.7 which may be the problem - although that’s a few releases after the one mentioned in that SO issue.

I cheated and added a cron job to restart puma every 3 hours… bandaid and not a good choice but I’m gonna see if it’ll work for a bit.  I’m running this version of ruby on a couple other servers and not having the same problems but they are digital ocean servers and not a VM in my colo….  Maybe I’ll get my sysadmin to bump up the memory.

thanks