httplib.BadStatusLine and 500 errors on deploy

[Aral]

I keep getting deployment errors while running ./manage.py update (or
appcfg update).

I've always been getting random 500 errors like:

Initiating update.
Cloning 212 static files.
ERROR:root:An unexpected error occurred. Aborting.
Rolling back the update.
Error 500: --- begin server output ---

Server Error (500)
A server error has occured.
--- end server output ---

(Issue: http://code.google.com/p/googleappengine/issues/detail?id=590)

Today, I also started seeing httplib.BadStatusLine errors (full
traceback at end, issue: http://code.google.com/p/googleappengine/issues/detail?id=610)

Is anyone else getting these?

I get a successful deploy roughly every 1 in 5 times, with the others
giving me errors. Needless to say, it's slowing down my development
process.

Thanks,
Aral

Full traceback of the BadStatusLine error:

Initiating update.
Cloning 224 static files.
Cloned 100 files.
Cloned 200 files.
Cloning 763 application files.
ERROR:root:An unexpected error occurred. Aborting.
Rolling back the update.
Traceback (most recent call last):
File "./manage.py", line 30, in <module>
execute_manager(settings)
File "/Users/aral/singularity/__init__.py", line 272, in
execute_manager

File "/Users/aral/singularity/__init__.py", line 219, in execute

File "/Users/aral/singularity/appengine_django/management/commands/
update.py", line 54, in run_from_argv
run_appcfg()
File "/Users/aral/singularity/appengine_django/management/commands/
update.py", line 43, in run_appcfg
appcfg.main(new_args)
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py",
line 1936, in main
AppCfgApp(argv).Run()
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py",
line 1521, in Run
self.action.function(self)
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py",
line 1733, in Update
lambda path: open(os.path.join(basepath, path), "rb"))
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py",
line 1313, in DoUpload
missing_files = self.Begin()
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py",
line 1207, in Begin
CloneFiles("/api/appversion/clonefiles", files_to_clone,
"application")
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py",
line 1201, in CloneFiles
payload=BuildClonePostBody(chunk))
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py",
line 296, in Send
f = self.opener.open(req)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/urllib2.py", line 374, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/urllib2.py", line 392, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/urllib2.py", line 353, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/urllib2.py", line 1101, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/urllib2.py", line 1074, in do_open
r = h.getresponse()
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/httplib.py", line 924, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/httplib.py", line 385, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/
python2.5/httplib.py", line 349, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine

[Brian Clapper]

On Aug 3, 1:15 pm, Aral <a...@aralbalkan.com> wrote:
> I keep getting deployment errors while running ./manage.py update (or
> appcfg update).
>
> I've always been getting random 500 errors like:
>
> Initiating update.
> Cloning 212 static files.
> ERROR:root:An unexpected error occurred. Aborting.
> Rolling back the update.
> Error 500: --- begin server output ---
>
> Server Error (500)
> A server error has occured.
> --- end server output ---

I've seen quite a few of these failures today. I want to say it's
correlated
with lots of files in the upload (i.e., more likely to fail when the
update
is large), but I don't have enough data to confirm that hunch.

-Brian Clapper

[Brian Clapper]

Bad hunch on my part, as often happens when data is sparse. :) I just had
an "appcfg.py update" fail five times in a row, before succeeding on the
sixth try. Only one file needed to be uploaded. The "unexpected error"
occurred consistently during the cloning of the static files:

Loaded authentication cookies from /home/bmc/.appcfg_cookies
Scanning files on local disk.
Scanned 500 files.
Initiating update.
Cloning 50 static files.
2008-08-04 17:50:20,632 ERROR appcfg.py:1334 An unexpected error

occurred. Aborting.
Rolling back the update.
Error 500: --- begin server output ---

Server Error (500)
A server error has occured.
--- end server output ---

Regards,

Brian Clapper, http://www.clapper.org/bmc/

[Brian Clapper]

On Aug 3, 1:15 pm, Aral <a...@aralbalkan.com> wrote:
> I keep getting deployment errors while running ./manage.py update (or
> appcfg update).
>
> I've always been getting random 500 errors like:
>
> Initiating update.
> Cloning 212 static files.
> ERROR:root:An unexpected error occurred. Aborting.
> Rolling back the update.
> Error 500: --- begin server output ---
>
> Server Error (500)
> A server error has occured.
> --- end server output ---

I've been seeing this error a lot today.

FWIW, I hacked google.appengine.tools.appcfg to dump the exception
when it encounters a server problem. The exception trace consistently
occurs in the same area of code, something called "cloneblobs".

If that helps at all.

Here's the trace.

...

Cloning 64 static files.
2008-08-09 14:21:09,281 ERROR appcfg.py:1334 An unexpected error
occurred. Aborting.
2008-08-09 14:21:09,282 ERROR appcfg.py:1335 Exception

Traceback (most recent call last):

File "/home/bmc/google_appengine/google/appengine/tools/appcfg.py",

line 1313, in DoUpload
missing_files = self.Begin()

File "/home/bmc/google_appengine/google/appengine/tools/appcfg.py",
line 1206, in Begin
CloneFiles("/api/appversion/cloneblobs", blobs_to_clone, "static")
File "/home/bmc/google_appengine/google/appengine/tools/appcfg.py",

line 1201, in CloneFiles
payload=BuildClonePostBody(chunk))

File "/home/bmc/google_appengine/google/appengine/tools/appcfg.py",

line 296, in Send
f = self.opener.open(req)

File "/usr/lib/python2.5/urllib2.py", line 387, in open
response = meth(req, response)
File "/usr/lib/python2.5/urllib2.py", line 498, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.5/urllib2.py", line 425, in error
return self._call_chain(*args)
File "/usr/lib/python2.5/urllib2.py", line 360, in _call_chain
result = func(*args)
File "/usr/lib/python2.5/urllib2.py", line 506, in
http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
Rolling back the update.

...

Regards,

Brian Clapper,http://www.clapper.org/bmc/
Trying to be happy is like trying to build a machine for which the
only
specification is that it should run noiselessly.