Proxy error with Apache and Thin probably dying periodically
Gaveen Prabhasara
been using mongrel_cluster behind Apache for this sever. But after I
gave thin a spin couple of weeks ago, I was quite impressed and thought
maybe I could go live with Thin, and I hope to. :)
I've been trying to solve a couple of things so far. Actually I've only
come across two significant issues yet.
1.) Once in a while when a new request is made, like a link click,
I get this error page from Apache.
"Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /user/user_details.
Reason: Error reading from remote server
Apache/2.2.3 (CentOS) Server at xxx.xxx.xxx.xxx Port 80"
The error is not specific to any page or part of the app. It sometimes
occur in the home page too. I've manually checked and verified all the
thin instances are running and responding. And if I reload the page it
gets processed correctly.
2.) Every morning when I try to access the site, I get an application
error page, which will not go away no matter how many times I reload
the page. :) Then I have to restart Thin cluster to bring the app back
to life. I have yet to manually check thin instances in this case.
I remember also having same things with Mongrel too. I'm somewhat new
to Rails world, and am a sys admin. I'd be grateful if anybody can give
me an insight to these issues.
Cheers,
Gav
PS: Here's my config options it it helps:
pid: tmp/pids/thin.pid
user: mongrel
log: log/thin.log
timeout: 60
port: 8000
chdir: /var/www/app/my_app/current
environment: production
servers: 5
group: mongrel
address: 127.0.0.1
macournoyer
if one of the thins crashed, the error was probably logged in the
logfiles under /var/www/app/my_app/current/log/thin.*.log
Let me know if you see anything particular in one of those
On Feb 25, 12:44 pm, Gaveen Prabhasara <gaveen.sky...@gmail.com>
wrote:
macournoyer
if one of the thins crashed the error was probably logged in the
logfile under /var/www/app/my_app/current/log/thin.*.log
let me know if you find anything special in one of those
mbr...@gmail.com
related to proxy errors.
We had similar symptoms to your second issue as well. Our problem was
related to the database connection being closed from inactivity on
some of our application servers. Restarting the app server worked
because a new DB connection was made at startup. We saw this problem
on both mongrel and thin though...
What version of rails/database are you running? What does your rails
error log show?
Dinooz
to start many instances, many virtual domains and the price of RAM is
very expensive. I had to tweak the child processes of apache and limit
the number of processes of the rails cluster with thin.
Check the memory usage before start any web server, after start Apache
the after start thin. and see how much you have available, if you
notice due to your web activity will reach the limit, tweak your
setting to make sure your server will have always some space in RAM
available to work.
Dinooz
PS -> Regarding the mod_proxy, I don't think is the issue, since you
could use load_balancer in a similar way.
Gaveen Prabhasara
First thank you all for quick support, then sorry for my late feedback.
I'm not well, that's why the delay.
On Mon, 2008-02-25 at 12:07 -0800, mbr...@alum.mit.edu wrote:
> We had similar symptoms to your second issue as well. Our problem was
> related to the database connection being closed from inactivity on
> some of our application servers. Restarting the app server worked
> because a new DB connection was made at startup. We saw this problem
> on both mongrel and thin though...
I think this might be the case. Our application is still in development,
so one a handful of people access it. That can explain why it goes down
in the night. For a test I had the app accessed during night and surely
it was working in the morning. I tried mongrel also and so that during
the night app goes down. So I guess I might be experiencing the same
issue you had. Can I know what you did with it?
> What version of rails/database are you running? What does your rails
> error log show?
I'm running Rails 1.2.6 with Ruby 1.8.6 patch 5000. DB is MySQL 5.0.22
While we have a possible root cause for one issue, I'm not sure about
the other case where I get Proxy Error.
On Mon, 2008-02-25 at 10:03 -0800, macournoyer wrote:
> hey Gaveen,
>
> if one of the thins crashed, the error was probably logged in the
> logfiles under /var/www/app/my_app/current/log/thin.*.log
>
> Let me know if you see anything particular in one of those
>
In the case where I get that proxy error page once in a while, I've
look more into it. Well, as I said I manually confirmed that all the
thin instances were running. I checked the logs, but there was nothing
relevant. So I guess this in not a dead thin issue.
On Mon, 2008-02-25 at 12:20 -0800, Dinooz wrote:
> I'll suggest to look at the memory on the server, It's very very easy
> to start many instances, many virtual domains and the price of RAM is
> very expensive. I had to tweak the child processes of apache and limit
> the number of processes of the rails cluster with thin.
Although I'm not particularly happy with the server memory status, I
think there's enough memory. I've run httperf a couple of times before
to see about memory and resources, and I don't think memory is the
issue with this case.
Side note: although my httperf sessions were not aimed to be benchmarks
I was very happy to see the rough figures. Thin handled as much as twice
the number of request/second, and consumed less memory. :) And again, it
wasn't any benchmark, just some grunt work.
> PS -> Regarding the mod_proxy, I don't think is the issue, since you
> could use load_balancer in a similar way.
I'm not entirely sure what you mean. I'm using mod_proxy_balancer, just
as in the mongrel docs. Anyway, I'm also thinking of migrating to Nginx,
but that has to wait.
Looking forward to hear your thoughts. Thanks.
Cheers,
Gav
macournoyer
On Feb 27, 9:48 am, Gaveen Prabhasara <gaveen.sky...@gmail.com> wrote:
> Hi,
>
> First thank you all for quick support, then sorry for my late feedback.
> I'm not well, that's why the delay.
>
mbr...@gmail.com
around the issue. Ezra had originally suggested something similar, but
calling verify_active_connections! alone did not work for me on
postgrseql (neither did setting
ActiveRecord::Base.verification_timeout) - I had to issue a db command
to keep the connection alive. We added this code to the end of
environment.rb:
## See http://www.ruby-forum.com/topic/94759
Thread.new do
loop do
sleep(30*60)
logger.fatal("Running
ActiveRecord::Base.verify_active_connections!")
# Verify active connections and run a simple query to connect
# through to the database
ActiveRecord::Base.verify_active_connections!
ActiveRecord::Base.connection.select_value('select 1')
end
end
>
> First thank you all for quick support, then sorry for my late feedback.
> I'm not well, that's why the delay.
>
Gaveen Prabhasara
> have a look at http://mongrel.rubyforge.org/wiki/FAQ#Q:Mongrelstopsworkingifitsleftaloneforalongtime.
>
what Michael suggested. I'd like to try this one before I go ahead and
create a sleeper thread.
Any thought on the proxy error page issue, anyone? Could this have
anything to do with the Service Providers bandwidth/load? I've checked
the log files and so far there was nothing I could find save a few
application errors. I'm working with the developers on those.
Cheers,
Gav
Gaveen Prabhasara
Both the problems have been solved by now. First one about the MySQL connection was solved by removing Ruby/MySQL driver and installing MySQL/Ruby driver. We required the driver, by the way. The second thing about the proxy error was a blunder by me. :) So I got it sorted out. It was a busy time for me. So, I'm sorry if I had initially not provided all the information.
I wrote a blog post completely detailing the case and the solutions at http://gaveen.owain.org/2008/04/errnoepipe-broken-pipe-mysql-error-in.html
Thank you all, for the ideas.