port 80 **** Hit EOF while fetching headers

[zhenwu...@gmail.com]

Hi, Experts:

We built our system using apache + django and we have lots of clients sending updates to our backend server at around 1min interval. But we do have 9000 around devices. From time to time, we are seeing port 80 **** Hit EOF while fetching headers in apache log or 500 error, could anybody tell me what does this mean and what maybe the possible reason to cause this? or any suggestion what I could do to isolate what cause this issue.

thanks in advance.

[Antonis Christofides]

Hi,

With an average of 150 requests per second, it is likely that your server is overloaded. You need to examine the available memory and CPU usage.

Regards,

A.

Antonis Christofides
http://djangodeployment.com

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5072b5af-5f78-4b75-83af-70da26bff125%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[zhenwu...@gmail.com]

Thanks for your response. 

Our server has 24 Cores and 192G memory, should be big enough to handle this kind of load? BTW, one more thing I want to mention is that we are using host based docker, possible any config or limitation there?

thanks again.

[Antonis Christofides]

Hi,

Your server most probably can handle the load. It depends of course on how intensive the processing is. The kind of error you are getting (port 80 **** Hit EOF while fetching headers) suggests that the problem is not in Django but in Apache, or maybe you are hitting some operating system limit. However nothing is certain without more information.

If you haven't already done so, you should configure Apache and syslog to use a timestamp that is accurate to the microsecond. You can then check to see other log files such as /var/log/syslog (assuming Linux) to see whether they have more information than the apache log at the exact time the error occurs.

Given that this is most probably not a Django issue but an Apache issue or an operating system issue, you may be able to get better help elsewhere, for example in serverfault.com.

Regards,

A.

Antonis Christofides
http://djangodeployment.com

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4d62efaf-654b-40ff-a895-efcb1bf2a7e8%40googlegroups.com.

[James Schneider]

Our server has 24 Cores and 192G memory, should be big enough to handle this kind of load? BTW, one more thing I want to mention is that we are using host based docker, possible any config or limitation there?

While I'm not familiar with your particular error, it's possible that your problem is in one of these areas:

1. Your web server is threaded, but your code is not thread safe, and you're hitting a race condition. The object you are expecting to exist has been unexpectedly modified or deleted by another thread. Can't comment further without knowledge of your underlying setup, code, etc. Alternatively, a file being read/written is modified/removed by a different process/thread.

2. Your underlying web server is not tuned to utilize the hardware you've mentioned. A typical Apache install will not spawn enough workers or threads to use that much horsepower. If your processor, disk, and RAM are not pegged (or close to), you aren't getting the bang for your buck that you thought with a large server (or you may not have enough load to tax it).

3. Your database may be congested, causing the Django calls to run slowly, possibly resulting in timeouts. Check the DB logs for over-subscription.

4. There may also be errant clients who are sending incomplete requests, or potentially port scanning where no headers are sent, and you are actually seeing a false-positive log.

Otherwise, you'll need to determine whether the error is coming from Apache receiving headers from the client, or if it is having issues with Django sending the needed headers back to Apache. 

-James

[zhenwu...@gmail.com]

Thanks James for your help.

Yeah, you are right, Python is not multi-thread supported, but, I felt it should be thread safe in this scenario.

I did further dig and found error message in apache error log showing [mpm_event:error] [pid 36398:tid 139973512394496] AH00485: scoreboard is full, not at MaxRequestWorkers

What does this mean? I googled and people saying that this is apache crash and I did check my wsgi log and noticed that each time I saw 500 error, I did see crashing happening in apache, but one thing I could not understand is that even I saw at certain time all are returning 200, I still see this error message in apache log. then only possible explanation is that maybe all those 200 returned are the threads not crashed. 

No matter what, I upgraded apache to 2.4.25, I was using 2.4.17, now, I did not see crash anymore, and neither 500 response, will monitor the system for more time to see it is really fixed.

thanks again for all your help