503 blues

[Brandon Casci]

Hello

I keep getting flurries of 503's. It's really hard for me to pin down where the issue is, but I think it may be somewhere around haproxy. The 503's seem to occur mostly after a redirect. For example, any request not to www gets redirected by apache to www.domain. It's really hard for me to diagnose this..I'm just guess at this point.

Has anyone else had this happen?

--
=========================================
I'm having a bout of spondylosis, so I can't hammer away at the keyboard too long. Please don't mistake short e-mails for rude e-mails..
=========================================

[Matthew Conway]

Hard to say - try turning up log level on haproxy?

Can you duplicate it easily, or does it just happen intermittently?

Using mongrels or passenger?  Does it only happen after a deploy (i.e. passenger takes long on first request which it initializes rails stack).

Matt

--

You received this message because you are subscribed to the Google Groups "rubber" group.
To post to this group, send email to rubbe...@googlegroups.com.
To unsubscribe from this group, send email to rubber-ec2+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubber-ec2?hl=en.

[Brandon Casci]

I'm using passenger. It's an intermittent problem and doesn't happen around a deployment. I tried to trigger it with apache benchmark, but it didn't fail for any of the tests. I kept an eye on the passenger status during the tests and it did not run out of workers. The problem seems to happen way more often on requests to domain.com, rather than www.domain.com though. In fact, that's how it was noticed, when people were requesting the non www url. I'm not familiar with haproxy, but will poke around and explore the logging options.

[Brandon Casci]

I bumped up haproxy log level to debug. It doesn't show anything sure the fits of 503's, and passenger shows plenty of inactive workers. There are some broken pipee errors in the apache log, between apache and the passenger app pool, but the times don't correspond to the 503's.Thanks for the input though, I know this isn't a rubber problem. It's a bit of a wtf moment.

[Brandon Casci]

I disabled haproxy to take it out of the picture but still getting 503's. Here is a header and response from one of the failed requests. You'll notice the response has no server information. I'm starting to think this may be a problem at EC2, but not on my instance.

http://www.loudcaster.com/images/btn_signup_for_loudcaster.gif?1259788768
 
GET /images/btn_signup_for_loudcaster.gif?1259788768 HTTP/1.1
Host: www.loudcaster.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102815 Ubuntu/9.04 (jaunty) Firefox/3.0.15
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.loudcaster.com/?fqdvgfedddddddddwdewq
Cookie: __unam=cf3a7e5-121eedbb15f-76a89091-625; _csoot=1259776244158; _csuid=4945a72636ef825f; _loudcaster_session=BAh7BzoQX2NzcmZfdG9rZW4iMW41Z3h2ajdsZ3IzMVdVc0NhaTJ5ZGQvckZnRG1va2ROeXhXUGh3V2xRQ1k9Og9zZXNzaW9uX2lkIiVlNmQyNTAxMDA0N2E1YjM5N2Q3YWY5MDhkZjQ4OGY4Yg%3D%3D--7fee5ee2da8fb2e91b972f3c576ca8d542f05b5a
If-Modified-Since: Wed, 02 Dec 2009 21:19:28 GMT
If-None-Match: "11b52f1-bee-479c56e023800"
Cache-Control: max-age=0
 
HTTP/1.x 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

[Brandon Casci]

I scrapped the instance for a brand new one.....so far so good! I guess you just get a bad instance now and then. At least it didn't take long to swap one out. I've lived with hardware related issues at other hosts for a lot longer.

[Matthew Conway]

We had two instance become unreachable tonight too (both in availability zone us-east-b) - must be something up at EC2.

[Brandon Casci]

Ya....it's starting to happen on my new instance too. Although....new thought. I'm still getting used to passenger. I wonder if the 503's are from haproxy timing out while waiting for an available worker.....but then again it happens with staic content too....so this would only make sense if passenger is serving static content. 

Sent from my iPhone

[Brian Del Vecchio]

I don't think it's a timeout issue because you're getting the 503s fairly close to instantly.

Also, you're correct--the assets are not going through Passenger.

Brian Del Vecchio  |  b...@hybernaut.com  |  617-899-0798  |  @hybernaut

[Brandon Casci]

Ya, I hear ya. It's just sad for me. :(

Sent from my iPhone