Recursion Limit
Jeff Anderson
I've recently had something strange happen.
At least four different URLs have produced this same error on my site. I
haven't updated smug to the most recent master yet, but I thought I'd
share this problem anyway.
Here's the traceback:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 86, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/httpd/www.programmerq.net/smug/smug/views/show.py", line 39, in show
basepath=basepath, request=request)
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 100, in get_content
if not dest_file.exists():
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 179, in exists
self.lookup()
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 209, in lookup
self.retrieve()
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 292, in retrieve
self.repo)
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 67, in get_source
dotsmug = load_dotsmug(path, repo)
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 52, in load_dotsmug
dotsmugs = [DotSmug(raw.content) for raw in rawfiles if raw.exists()]
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 179, in exists
self.lookup()
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 209, in lookup
self.retrieve()
File "/home/httpd/www.programmerq.net/smug/smug/load.py", line 271, in retrieve
self.mimetype, encoding = guess_type(self.path, strict=False)
File "/usr/lib/python2.6/mimetypes.py", line 242, in guess_type
return guess_type(url, strict)
File "/usr/lib/python2.6/mimetypes.py", line 242, in guess_type
return guess_type(url, strict)
File "/usr/lib/python2.6/mimetypes.py", line 242, in guess_type
return guess_type(url, strict)
File "/usr/lib/python2.6/mimetypes.py", line 242, in guess_type
return guess_type(url, strict)
File "/usr/lib/python2.6/mimetypes.py", line 242, in guess_type
return guess_type(url, strict)
File "/usr/lib/python2.6/mimetypes.py", line 242, in guess_type
return guess_type(url, strict)
...
...
...
RuntimeError: maximum recursion depth exceeded
Simply visiting the URLs that experienced don't reproduce the problem. Both POST and GET were empty each time this error happened. The error has happened with different user agents and requesting IP addresses. This error has happened on two directories (where an index.html would have been used), a file that exists, and a file that doesn't exist.
Honestly, I'm at a loss as to how to start diagnosing this one. I'm not even sure if the bug would be in smug, or in Python's mimetypes.py
While I'm at it, I've also had subprocess fail a few times throwing "OSError: [Errno 12] Cannot allocate memory", which is understandable since I have my VPS loaded quite heavily. I've corrected the misconfiguration that resulted in apache eating memory, but this type of error still happens from time to time. I'm just mentioning this other error on the off chance that it may be a piece in the puzzle to the maximum recursion limit problem.
Andrew McNabb
> Hello,
>
> I've recently had something strange happen.
>
> At least four different URLs have produced this same error on my site. I
> haven't updated smug to the most recent master yet, but I thought I'd
> share this problem anyway.
> File "/usr/lib/python2.6/mimetypes.py", line 242, in guess_type
> return guess_type(url, strict)
>
> ...
> ...
> ...
>
> RuntimeError: maximum recursion depth exceeded
Wow. I just looked at the code for guess_type (in Python's mimetypes
module). It's pretty ridiculous. The init() function actually
overwrites mimetypes.guess_type. It looks like somehow line 307 in the
init() function isn't being run. This is definitely a Python bug, but
I'm not seeing what could cause this. It seems to me that anything that
could be causing this should raise some other exception.
> While I'm at it, I've also had subprocess fail a few times throwing "OSError: [Errno 12] Cannot allocate memory", which is understandable since I have my VPS loaded quite heavily. I've corrected the misconfiguration that resulted in apache eating memory, but this type of error still happens from time to time. I'm just mentioning this other error on the off chance that it may be a piece in the puzzle to the maximum recursion limit problem.
I don't think this is related to the maximum recursion limit problem,
but it really looks bad. If it can't allocate memory, that means that
your machine is really overloaded.
--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
sste...@gmail.com
On Jul 13, 2009, at 2:58 PM, Andrew McNabb wrote:
>
> On Fri, Jul 10, 2009 at 06:56:07PM -0600, Jeff Anderson wrote:
>> Hello,
>>
>> I've recently had something strange happen.
>>
>> At least four different URLs have produced this same error on my
>> site. I
>> haven't updated smug to the most recent master yet, but I thought I'd
>> share this problem anyway.
>
>
>> File "/usr/lib/python2.6/mimetypes.py", line 242, in guess_type
>> return guess_type(url, strict)
>>
>> ...
>> ...
>> ...
>>
>> RuntimeError: maximum recursion depth exceeded
>
> Wow. I just looked at the code for guess_type (in Python's mimetypes
> module). It's pretty ridiculous. The init() function actually
> overwrites mimetypes.guess_type. It looks like somehow line 307 in
> the
> init() function isn't being run.
or, db.guess_type is equal to mimetypes.guess_type which would lead to
infinite recursion which is what is happening.
Not executing a line of code just seems bloody unlikely.
> This is definitely a Python bug, but I'm not seeing what could cause
> this. It seems to me that anything that
> could be causing this should raise some other exception.
See above. Not likely a Python bug.
S
Andrew McNabb
>
> > Wow. I just looked at the code for guess_type (in Python's mimetypes
> > module). It's pretty ridiculous. The init() function actually
> > overwrites mimetypes.guess_type. It looks like somehow line 307 in
> > the
> > init() function isn't being run.
>
> or, db.guess_type is equal to mimetypes.guess_type which would lead to
> infinite recursion which is what is happening.
That seems very likely as the direct cause of the problem. However, it
is still a very odd situation to be in.
> > This is definitely a Python bug, but I'm not seeing what could cause
> > this. It seems to me that anything that could be causing this
> > should raise some other exception.
>
> See above. Not likely a Python bug.
Yes, but what would set db.guess_type equal to mimetypes.guess_type? If
it's anything in mimetypes.py doing this, then it's a Python bug.
Andrew McNabb
I looked it up, and this is indeed a known Python bug:
http://bugs.python.org/issue5853
Andrew McNabb
>
> On Mon, Jul 13, 2009 at 04:20:50PM -0400, sste...@gmail.com wrote:
> >
> > See above. Not likely a Python bug.
>
> I looked it up, and this is indeed a known Python bug:
>
> http://bugs.python.org/issue5853
By the way, it looks like this bug was introduced in Python 2.6.2 (and
does not exist in Python 2.6).
sste...@gmail.com
On Jul 13, 2009, at 5:15 PM, Andrew McNabb wrote:
>> See above. Not likely a Python bug.
>
> Yes, but what would set db.guess_type equal to
> mimetypes.guess_type? If
> it's anything in mimetypes.py doing this, then it's a Python bug.
Maybe a matter of terminology.
I'd consider it a library bug.
If it didn't execute one particular line of code as suggested earlier,
I would call that a Python bug i.e. something in the interpreter
itself being broken.
S
sste...@gmail.com
On Jul 13, 2009, at 5:32 PM, Andrew McNabb wrote:
>
> On Mon, Jul 13, 2009 at 04:20:50PM -0400, sste...@gmail.com wrote:
>>
>> See above. Not likely a Python bug.
>
> I looked it up, and this is indeed a known Python bug:
>
> http://bugs.python.org/issue5853
Cool. When I saw that bizarre init() code replacing a module level
function with the method inside the class, that was clearly going to
cause trouble in a multi-threaded app.
Glad it's been fixed in the trunk.
So, is the workaround seems to be to call the mimetypes.init()
manually before threading anything out.
S
sste...@gmail.com
> does not exist in Python 2.6).
Interesting.
I wonder why the breaking change was made and, more importantly, why
there isn't a test case to catch the breakage. Hopefully there is one
now!
S
Andrew McNabb
Yeah, it's just a difference in terminology. I think of there being
"Python stdlib bugs" and "Python interpreter bugs", both of which are
"Python bugs". My justification for this terminology is that I consider
a "Python bug" to be anything that you would report on bugs.python.org.
I've found that terminology is the source of half of all disagreements.
:)
Andrew McNabb
>
> Interesting.
>
> I wonder why the breaking change was made and, more importantly, why
> there isn't a test case to catch the breakage. Hopefully there is one
> now!
The change was made when someone found that the idiotic monkeypatching
didn't always work. They made a quick workaround, which worked except
for the race condition. :)
Anyway, it's really hard to do a test case for race conditions.
sste...@gmail.com
On Jul 13, 2009, at 8:09 PM, Andrew McNabb wrote:
> I've found that terminology is the source of half of all
> disagreements.
No it's not! ;-)
S
sste...@gmail.com
> On Mon, Jul 13, 2009 at 06:42:37PM -0400, sste...@gmail.com wrote:
>>
>> Interesting.
>>
>> I wonder why the breaking change was made and, more importantly, why
>> there isn't a test case to catch the breakage. Hopefully there is
>> one
>> now!
>
> The change was made when someone found that the idiotic monkeypatching
> didn't always work. They made a quick workaround, which worked except
> for the race condition. :)
>
> Anyway, it's really hard to do a test case for race conditions.
Yes, but you can see idiotic monkeypatching from miles away if you've
been around the block a couple of times ;-).
Let's just say, in this particular case, the way this module is
written is just, um, less than elegant?
I'd be less nice about it if I had a patch, or time to fix it but
right now I don't so I have no right to criticize.
Hopefully it'll get fixed next go-round but in the meantime, just call
the init() method before you do anything "tricky" like, for example,
writing a program using the module. ;-)
S
Andrew McNabb
>
> Yes, but you can see idiotic monkeypatching from miles away if you've
> been around the block a couple of times ;-).
>
> Let's just say, in this particular case, the way this module is
> written is just, um, less than elegant?
Yeah, it made me pretty ill when I first looked at it. :)
> I'd be less nice about it if I had a patch, or time to fix it but
> right now I don't so I have no right to criticize.
Fortunately, when they fixed the race condition, they did it by getting
rid of the monkeypatching. If you look at the file in trunk, it will
make you very happy.
> Hopefully it'll get fixed next go-round but in the meantime, just call
> the init() method before you do anything "tricky" like, for example,
> writing a program using the module. ;-)
Python 2.6.2 is the first and last version affected by this, so I'm not
going to get too worried about it.
sste...@gmail.com
On Jul 13, 2009, at 11:29 PM, Andrew McNabb wrote:
>> Python 2.6.2 is the first and last version affected by this, so I'm
>> not
> going to get too worried about it.
Ok, glad to hear they fixed it right. Any idea when's 2.6.3 due?
S
Andrew McNabb
>
> Ok, glad to hear they fixed it right. Any idea when's 2.6.3 due?
Apparently 2.6 was released on October 1st, 2.6.1 was released on
December 4th, and 2.6.2 was released on April 14th. I would guess that
we're due for an update soon, but I don't know any details.