I'm working on a Rails project which is going live soon. At first I've been using mongrel_cluster behind Apache for this sever. But after I gave thin a spin couple of weeks ago, I was quite impressed and thought maybe I could go live with Thin, and I hope to. :)
I've been trying to solve a couple of things so far. Actually I've only come across two significant issues yet.
1.) Once in a while when a new request is made, like a link click, I get this error page from Apache. "Proxy Error
The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /user/user_details.
Reason: Error reading from remote server
Apache/2.2.3 (CentOS) Server at xxx.xxx.xxx.xxx Port 80"
The error is not specific to any page or part of the app. It sometimes occur in the home page too. I've manually checked and verified all the thin instances are running and responding. And if I reload the page it gets processed correctly.
2.) Every morning when I try to access the site, I get an application error page, which will not go away no matter how many times I reload the page. :) Then I have to restart Thin cluster to bring the app back to life. I have yet to manually check thin instances in this case.
I remember also having same things with Mongrel too. I'm somewhat new to Rails world, and am a sys admin. I'd be grateful if anybody can give me an insight to these issues.
> I'm working on a Rails project which is going live soon. At first I've
> been using mongrel_cluster behind Apache for this sever. But after I
> gave thin a spin couple of weeks ago, I was quite impressed and thought
> maybe I could go live with Thin, and I hope to. :)
> I've been trying to solve a couple of things so far. Actually I've only
> come across two significant issues yet.
> 1.) Once in a while when a new request is made, like a link click,
> I get this error page from Apache.
> "Proxy Error
> The proxy server received an invalid response from an upstream server.
> The proxy server could not handle the request GET /user/user_details.
> Reason: Error reading from remote server
> Apache/2.2.3 (CentOS) Server at xxx.xxx.xxx.xxx Port 80"
> The error is not specific to any page or part of the app. It sometimes
> occur in the home page too. I've manually checked and verified all the
> thin instances are running and responding. And if I reload the page it
> gets processed correctly.
> 2.) Every morning when I try to access the site, I get an application
> error page, which will not go away no matter how many times I reload
> the page. :) Then I have to restart Thin cluster to bring the app back
> to life. I have yet to manually check thin instances in this case.
> I remember also having same things with Mongrel too. I'm somewhat new
> to Rails world, and am a sys admin. I'd be grateful if anybody can give
> me an insight to these issues.
> I'm working on a Rails project which is going live soon. At first I've
> been using mongrel_cluster behind Apache for this sever. But after I
> gave thin a spin couple of weeks ago, I was quite impressed and thought
> maybe I could go live with Thin, and I hope to. :)
> I've been trying to solve a couple of things so far. Actually I've only
> come across two significant issues yet.
> 1.) Once in a while when a new request is made, like a link click,
> I get this error page from Apache.
> "Proxy Error
> The proxy server received an invalid response from an upstream server.
> The proxy server could not handle the request GET /user/user_details.
> Reason: Error reading from remote server
> Apache/2.2.3 (CentOS) Server at xxx.xxx.xxx.xxx Port 80"
> The error is not specific to any page or part of the app. It sometimes
> occur in the home page too. I've manually checked and verified all the
> thin instances are running and responding. And if I reload the page it
> gets processed correctly.
> 2.) Every morning when I try to access the site, I get an application
> error page, which will not go away no matter how many times I reload
> the page. :) Then I have to restart Thin cluster to bring the app back
> to life. I have yet to manually check thin instances in this case.
> I remember also having same things with Mongrel too. I'm somewhat new
> to Rails world, and am a sys admin. I'd be grateful if anybody can give
> me an insight to these issues.
Try apache 2.2.8 - there are several patches in mod_proxy specifically
related to proxy errors.
We had similar symptoms to your second issue as well. Our problem was
related to the database connection being closed from inactivity on
some of our application servers. Restarting the app server worked
because a new DB connection was made at startup. We saw this problem
on both mongrel and thin though...
What version of rails/database are you running? What does your rails
error log show?
On Feb 25, 12:44 pm, Gaveen Prabhasara <gaveen.sky...@gmail.com>
wrote:
> I'm working on a Rails project which is going live soon. At first I've
> been using mongrel_cluster behind Apache for this sever. But after I
> gave thin a spin couple of weeks ago, I was quite impressed and thought
> maybe I could go live with Thin, and I hope to. :)
> I've been trying to solve a couple of things so far. Actually I've only
> come across two significant issues yet.
> 1.) Once in a while when a new request is made, like a link click,
> I get this error page from Apache.
> "Proxy Error
> The proxy server received an invalid response from an upstream server.
> The proxy server could not handle the request GET /user/user_details.
> Reason: Error reading from remote server
> Apache/2.2.3 (CentOS) Server at xxx.xxx.xxx.xxx Port 80"
> The error is not specific to any page or part of the app. It sometimes
> occur in the home page too. I've manually checked and verified all the
> thin instances are running and responding. And if I reload the page it
> gets processed correctly.
> 2.) Every morning when I try to access the site, I get an application
> error page, which will not go away no matter how many times I reload
> the page. :) Then I have to restart Thin cluster to bring the app back
> to life. I have yet to manually check thin instances in this case.
> I remember also having same things with Mongrel too. I'm somewhat new
> to Rails world, and am a sys admin. I'd be grateful if anybody can give
> me an insight to these issues.
I'll suggest to look at the memory on the server, It's very very easy
to start many instances, many virtual domains and the price of RAM is
very expensive. I had to tweak the child processes of apache and limit
the number of processes of the rails cluster with thin.
Check the memory usage before start any web server, after start Apache
the after start thin. and see how much you have available, if you
notice due to your web activity will reach the limit, tweak your
setting to make sure your server will have always some space in RAM
available to work.
Dinooz
PS -> Regarding the mod_proxy, I don't think is the issue, since you
could use load_balancer in a similar way.
First thank you all for quick support, then sorry for my late feedback. I'm not well, that's why the delay.
On Mon, 2008-02-25 at 12:07 -0800, mbry...@alum.mit.edu wrote: > We had similar symptoms to your second issue as well. Our problem was > related to the database connection being closed from inactivity on > some of our application servers. Restarting the app server worked > because a new DB connection was made at startup. We saw this problem > on both mongrel and thin though...
I think this might be the case. Our application is still in development, so one a handful of people access it. That can explain why it goes down in the night. For a test I had the app accessed during night and surely it was working in the morning. I tried mongrel also and so that during the night app goes down. So I guess I might be experiencing the same issue you had. Can I know what you did with it?
> What version of rails/database are you running? What does your rails > error log show?
I'm running Rails 1.2.6 with Ruby 1.8.6 patch 5000. DB is MySQL 5.0.22
While we have a possible root cause for one issue, I'm not sure about the other case where I get Proxy Error.
On Mon, 2008-02-25 at 10:03 -0800, macournoyer wrote: > hey Gaveen,
> if one of the thins crashed, the error was probably logged in the > logfiles under /var/www/app/my_app/current/log/thin.*.log
> Let me know if you see anything particular in one of those
In the case where I get that proxy error page once in a while, I've look more into it. Well, as I said I manually confirmed that all the thin instances were running. I checked the logs, but there was nothing relevant. So I guess this in not a dead thin issue.
On Mon, 2008-02-25 at 12:20 -0800, Dinooz wrote: > I'll suggest to look at the memory on the server, It's very very easy > to start many instances, many virtual domains and the price of RAM is > very expensive. I had to tweak the child processes of apache and limit > the number of processes of the rails cluster with thin.
Although I'm not particularly happy with the server memory status, I think there's enough memory. I've run httperf a couple of times before to see about memory and resources, and I don't think memory is the issue with this case.
Side note: although my httperf sessions were not aimed to be benchmarks I was very happy to see the rough figures. Thin handled as much as twice the number of request/second, and consumed less memory. :) And again, it wasn't any benchmark, just some grunt work.
> PS -> Regarding the mod_proxy, I don't think is the issue, since you > could use load_balancer in a similar way.
I'm not entirely sure what you mean. I'm using mod_proxy_balancer, just as in the mongrel docs. Anyway, I'm also thinking of migrating to Nginx, but that has to wait.
> First thank you all for quick support, then sorry for my late feedback.
> I'm not well, that's why the delay.
> On Mon, 2008-02-25 at 12:07 -0800, mbry...@alum.mit.edu wrote:
> > We had similar symptoms to your second issue as well. Our problem was
> > related to the database connection being closed from inactivity on
> > some of our application servers. Restarting the app server worked
> > because a new DB connection was made at startup. We saw this problem
> > on both mongrel and thin though...
> I think this might be the case. Our application is still in development,
> so one a handful of people access it. That can explain why it goes down
> in the night. For a test I had the app accessed during night and surely
> it was working in the morning. I tried mongrel also and so that during
> the night app goes down. So I guess I might be experiencing the same
> issue you had. Can I know what you did with it?
> > What version of rails/database are you running? What does your rails
> > error log show?
> I'm running Rails 1.2.6 with Ruby 1.8.6 patch 5000. DB is MySQL 5.0.22
> While we have a possible root cause for one issue, I'm not sure about
> the other case where I get Proxy Error.
> On Mon, 2008-02-25 at 10:03 -0800, macournoyer wrote:
> > hey Gaveen,
> > if one of the thins crashed, the error was probably logged in the
> > logfiles under /var/www/app/my_app/current/log/thin.*.log
> > Let me know if you see anything particular in one of those
> In the case where I get that proxy error page once in a while, I've
> look more into it. Well, as I said I manually confirmed that all the
> thin instances were running. I checked the logs, but there was nothing
> relevant. So I guess this in not a dead thin issue.
> On Mon, 2008-02-25 at 12:20 -0800, Dinooz wrote:
> > I'll suggest to look at the memory on the server, It's very very easy
> > to start many instances, many virtual domains and the price of RAM is
> > very expensive. I had to tweak the child processes of apache and limit
> > the number of processes of the rails cluster with thin.
> Although I'm not particularly happy with the server memory status, I
> think there's enough memory. I've run httperf a couple of times before
> to see about memory and resources, and I don't think memory is the
> issue with this case.
> Side note: although my httperf sessions were not aimed to be benchmarks
> I was very happy to see the rough figures. Thin handled as much as twice
> the number of request/second, and consumed less memory. :) And again, it
> wasn't any benchmark, just some grunt work.
> > PS -> Regarding the mod_proxy, I don't think is the issue, since you
> > could use load_balancer in a similar way.
> I'm not entirely sure what you mean. I'm using mod_proxy_balancer, just
> as in the mongrel docs. Anyway, I'm also thinking of migrating to Nginx,
> but that has to wait.
To keep database connections alive, I added the following which worked
around the issue. Ezra had originally suggested something similar, but
calling verify_active_connections! alone did not work for me on
postgrseql (neither did setting
ActiveRecord::Base.verification_timeout) - I had to issue a db command
to keep the connection alive. We added this code to the end of
environment.rb:
## See http://www.ruby-forum.com/topic/94759 Thread.new do
loop do
sleep(30*60)
logger.fatal("Running
ActiveRecord::Base.verify_active_connections!")
# Verify active connections and run a simple query to connect
# through to the database
ActiveRecord::Base.verify_active_connections!
ActiveRecord::Base.connection.select_value('select 1')
end
end
On Feb 27, 9:48 am, Gaveen Prabhasara <gaveen.sky...@gmail.com> wrote:
> First thank you all for quick support, then sorry for my late feedback.
> I'm not well, that's why the delay.
> On Mon, 2008-02-25 at 12:07 -0800, mbry...@alum.mit.edu wrote:
> > We had similar symptoms to your second issue as well. Our problem was
> > related to the database connection being closed from inactivity on
> > some of our application servers. Restarting the app server worked
> > because a new DB connection was made at startup. We saw this problem
> > on both mongrel and thin though...
> I think this might be the case. Our application is still in development,
> so one a handful of people access it. That can explain why it goes down
> in the night. For a test I had the app accessed during night and surely
> it was working in the morning. I tried mongrel also and so that during
> the night app goes down. So I guess I might be experiencing the same
> issue you had. Can I know what you did with it?
> > What version of rails/database are you running? What does your rails
> > error log show?
> I'm running Rails 1.2.6 with Ruby 1.8.6 patch 5000. DB is MySQL 5.0.22
> While we have a possible root cause for one issue, I'm not sure about
> the other case where I get Proxy Error.
> On Mon, 2008-02-25 at 10:03 -0800, macournoyer wrote:
> > hey Gaveen,
> > if one of the thins crashed, the error was probably logged in the
> > logfiles under /var/www/app/my_app/current/log/thin.*.log
> > Let me know if you see anything particular in one of those
> In the case where I get that proxy error page once in a while, I've
> look more into it. Well, as I said I manually confirmed that all the
> thin instances were running. I checked the logs, but there was nothing
> relevant. So I guess this in not a dead thin issue.
> On Mon, 2008-02-25 at 12:20 -0800, Dinooz wrote:
> > I'll suggest to look at the memory on the server, It's very very easy
> > to start many instances, many virtual domains and the price of RAM is
> > very expensive. I had to tweak the child processes of apache and limit
> > the number of processes of the rails cluster with thin.
> Although I'm not particularly happy with the server memory status, I
> think there's enough memory. I've run httperf a couple of times before
> to see about memory and resources, and I don't think memory is the
> issue with this case.
> Side note: although my httperf sessions were not aimed to be benchmarks
> I was very happy to see the rough figures. Thin handled as much as twice
> the number of request/second, and consumed less memory. :) And again, it
> wasn't any benchmark, just some grunt work.
> > PS -> Regarding the mod_proxy, I don't think is the issue, since you
> > could use load_balancer in a similar way.
> I'm not entirely sure what you mean. I'm using mod_proxy_balancer, just
> as in the mongrel docs. Anyway, I'm also thinking of migrating to Nginx,
> but that has to wait.
Thanks, I'll first try this one. If it still persists I think I can do what Michael suggested. I'd like to try this one before I go ahead and create a sleeper thread.
Any thought on the proxy error page issue, anyone? Could this have anything to do with the Service Providers bandwidth/load? I've checked the log files and so far there was nothing I could find save a few application errors. I'm working with the developers on those.
On Mon, 2008-02-25 at 23:14 +0530, Gaveen Prabhasara wrote: > 1.) Once in a while when a new request is made, like a link click, > I get this error page from Apache. > "Proxy Error
> The proxy server received an invalid response from an upstream server. > The proxy server could not handle the request GET /user/user_details.
> Reason: Error reading from remote server
> Apache/2.2.3 (CentOS) Server at xxx.xxx.xxx.xxx Port 80"
> The error is not specific to any page or part of the app. It sometimes > occur in the home page too. I've manually checked and verified all the > thin instances are running and responding. And if I reload the page it > gets processed correctly.
Sorry to bring up this old post. But I thought of keeping the list updated about the current status and solutions.
Both the problems have been solved by now. First one about the MySQL connection was solved by removing Ruby/MySQL driver and installing MySQL/Ruby driver. We required the driver, by the way. The second thing about the proxy error was a blunder by me. :) So I got it sorted out. It was a busy time for me. So, I'm sorry if I had initially not provided all the information.
