Exceptions when posting, viewing diffs after upgrading to 1.7.2

[Colin Caughie]

Since upgrading to 1.7.2 we're getting exceptions when posting and
viewing diffs against a Mercurial repository; this is currently
preventing us from posting any reviews.

Stack traces below. Any ideas on how to fix this would be much appreciated.

Post operation:

2013-01-22 10:05:22,899 - ERROR - Internal Server Error:
/api/review-requests/12
21/diffs/
Traceback (most recent call last):
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Django-1.4.3-py2.6
.egg/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Django-1.4.3-py2.6.egg/django/views/decorators/cache.py",
line 89, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Django-1.4.3-py2.6.egg/django/views/decorators/vary.py",
line 19, in inner_func
response = func(*args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Djblets-0.7.8-py2.6.egg/djblets/webapi/resources.py",
line 461, in __call__
result = view(request, api_format=api_format, *args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Djblets-0.7.8-py2.6.egg/djblets/webapi/resources.py",
line 662, in post
return self.create(*args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/webapi/decorators.py",
line 127, in _check
return view_func(*args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Djblets-0.7.8-py2.6.egg/djblets/webapi/decorators.py",
line 88, in _checklogin
return view_func(*args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Djblets-0.7.8-py2.6.egg/djblets/webapi/decorators.py",
line 62, in _call
return view_func(*args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Djblets-0.7.8-py2.6.egg/djblets/webapi/decorators.py",
line 235, in _validate
return view_func(*args, **new_kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/webapi/resources.py",
line 1668, in create
form = UploadDiffForm(review_request, form_data, request.FILES)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/reviews/forms.py",
line 276, in __init__
data, *args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/diffviewer/forms.py",
line 51, in __init__
if self.repository.get_scmtool().get_diffs_use_absolute_paths():
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/scmtools/models.py",
line 134, in get_scmtool
return cls(self)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/scmtools/hg.py",
line 31, in __init__
self.client = HgClient(repository.path, repository.local_site)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/scmtools/hg.py",
line 215, in __init__
if parse_version(version) <= parse_version("1.2"):
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/pkg_resources.py",
line 1913, in parse_version
for part in _parse_version_parts(s.lower()):
TypeError: 'NoneType' object is not callable

View diff operation:

2013-01-22 10:17:17,014 - ERROR - Internal Server Error: /r/1218/
Traceback (most recent call last):
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/Django-1.4.3-py2.6.egg/django/core/handlers/base.py",
line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/accounts/decorators.py",
line 22, in _check
return view_func(*args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/reviews/views.py",
line 728, in review_detail
'screenshots': screenshots,
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/reviews/views.py",
line 124, in _make_review_request_context
upload_diff_form = UploadDiffForm(review_request)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/reviews/forms.py",
line 276, in __init__
data, *args, **kwargs)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/diffviewer/forms.py",
line 51, in __init__
if self.repository.get_scmtool().get_diffs_use_absolute_paths():
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/scmtools/models.py",
line 134, in get_scmtool
return cls(self)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/scmtools/hg.py",
line 31, in __init__
self.client = HgClient(repository.path, repository.local_site)
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/ReviewBoard-1.7.2-py2.6.egg/reviewboard/scmtools/hg.py",
line 215, in __init__
if parse_version(version) <= parse_version("1.2"):
File
"/home/hg/reviewboard-venv/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/pkg_resources.py",
line 1913, in parse_version
for part in _parse_version_parts(s.lower()):
TypeError: 'NoneType' object is not callable

[Christian Hammond]

Hi Colin,

There was another report of this recently, but I don't have it handy right now. Basically, for some reason, mercurial's version is being reported to us as None, which is… strange. Not to mention that the only difference in 1.7.x in that code is that instead of parsing the version we get from Mercurial in that code, we're using another function to parse it.

Can you tell me what version of mercurial you have installed? And though I expect the result will be the same as what I saw before, can you try:

$ python
>>> from mercurial.__version__ import version
>>> print version

Christian

--
Christian Hammond - chi...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com

[Colin Caughie]

Update: I just created a completely new virtualenv and installed RB and
Mercurial into it from scratch, and now it seems to be working.

I still can't explain why it was happening before though. I looked in
the code for HgClient and couldn't see anything that could cause this to
happen. I tried running all of the relevant things interactively and
they all behaved as expected:

(reviewboard-venv)hg@athena:~$ python
Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from mercurial.__version__ import version
>>> version
'2.5-rc'
>>> from pkg_resources import parse_version
>>> parse_version(version)
('00000002', '00000005', '*final-', '*c', '*final')
>>>

(I had upgraded to the latest Mercurial release to see if it would fix
the problem, but it didn't).

On 1/22/2013 10:42 AM, Christian Hammond wrote:
> Hi Colin,
>

> There was another report of this recently, but I don't have it handy right now. Basically, for some reason, mercurial's version is being reported to us as None, which is� strange. Not to mention that the only difference in 1.7.x in that code is that instead of parsing the version we get from Mercurial in that code, we're using another function to parse it.

[Colin Caughie]

Update #2: It's still happening. It seems to be strangely sporadic
though; sometimes a diff will fail to display and the exception will be
thrown, but then we refresh the browser and it works fine. This is
strange -- I'd have thought something as simple as reading and parsing a
version constant would be a bit more deterministic.

[Christian Hammond]

You would think so.

Can you modify the code to log the version we get, and to log mercurial.__file__?

Christian

On Jan 22, 2013, at 11:08, Colin Caughie <c.ca...@gmail.com> wrote:

> Update #2: It's still happening. It seems to be strangely sporadic though; sometimes a diff will fail to display and the exception will be thrown, but then we refresh the browser and it works fine. This is strange -- I'd have thought something as simple as reading and parsing a version constant would be a bit more deterministic.
>
> On 1/22/2013 10:51 AM, Colin Caughie wrote:
>> Update: I just created a completely new virtualenv and installed RB and Mercurial into it from scratch, and now it seems to be working.
>>
>> I still can't explain why it was happening before though. I looked in the code for HgClient and couldn't see anything that could cause this to happen. I tried running all of the relevant things interactively and they all behaved as expected:
>>
>> (reviewboard-venv)hg@athena:~$ python
>> Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
>> [GCC 4.4.5] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> from mercurial.__version__ import version
>> >>> version
>> '2.5-rc'
>> >>> from pkg_resources import parse_version
>> >>> parse_version(version)
>> ('00000002', '00000005', '*final-', '*c', '*final')
>> >>>
>>
>> (I had upgraded to the latest Mercurial release to see if it would fix the problem, but it didn't).
>>
>>
>> On 1/22/2013 10:42 AM, Christian Hammond wrote:
>>> Hi Colin,
>>>

>>> There was another report of this recently, but I don't have it handy right now. Basically, for some reason, mercurial's version is being reported to us as None, which is… strange. Not to mention that the only difference in 1.7.x in that code is that instead of parsing the version we get from Mercurial in that code, we're using another function to parse it.

[Colin Caughie]

In order to change the code I set up a new virtualenv for development
and set up a dev environment as per
http://www.reviewboard.org/docs/codebase/dev/getting-started/. I added
the extra logging, pointed my web server at it, and of course now I
can't get the problem to occur. I then switched back to my production
install and it still isn't happening.

This seems to be a somewhat elusive problem. If I can get it to happen
again reliably I'll investigate further. Let me know if there's anything
else you want me to try.

Incidentally I tried logging mercurial.__file__ but couldn't figure out
how to import it. What code do I need to do this?

On 1/22/2013 11:09 AM, Christian Hammond wrote:
> You would think so.
>
> Can you modify the code to log the version we get, and to log mercurial.__file__?
>
> Christian
>
>
> On Jan 22, 2013, at 11:08, Colin Caughie <c.ca...@gmail.com> wrote:
>
>> Update #2: It's still happening. It seems to be strangely sporadic though; sometimes a diff will fail to display and the exception will be thrown, but then we refresh the browser and it works fine. This is strange -- I'd have thought something as simple as reading and parsing a version constant would be a bit more deterministic.
>>
>> On 1/22/2013 10:51 AM, Colin Caughie wrote:
>>> Update: I just created a completely new virtualenv and installed RB and Mercurial into it from scratch, and now it seems to be working.
>>>
>>> I still can't explain why it was happening before though. I looked in the code for HgClient and couldn't see anything that could cause this to happen. I tried running all of the relevant things interactively and they all behaved as expected:
>>>
>>> (reviewboard-venv)hg@athena:~$ python
>>> Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
>>> [GCC 4.4.5] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> from mercurial.__version__ import version
>>>>>> version
>>> '2.5-rc'
>>>>>> from pkg_resources import parse_version
>>>>>> parse_version(version)
>>> ('00000002', '00000005', '*final-', '*c', '*final')
>>> (I had upgraded to the latest Mercurial release to see if it would fix the problem, but it didn't).
>>>
>>>
>>> On 1/22/2013 10:42 AM, Christian Hammond wrote:
>>>> Hi Colin,
>>>>

>>>> There was another report of this recently, but I don't have it handy right now. Basically, for some reason, mercurial's version is being reported to us as None, which is� strange. Not to mention that the only difference in 1.7.x in that code is that instead of parsing the version we get from Mercurial in that code, we're using another function to parse it.

[Christian Hammond]

Today seems to be the day for weird bugs. I'd be very interested in seeing if you hit it again and if logging turns up anything.

To get to mercurial.__file__, you can do:

import mercurial
mercurial.__file__

Christian

>>>>> There was another report of this recently, but I don't have it handy right now. Basically, for some reason, mercurial's version is being reported to us as None, which is… strange. Not to mention that the only difference in 1.7.x in that code is that instead of parsing the version we get from Mercurial in that code, we're using another function to parse it.

[adam.c...@gmail.com]

Hi,

We're seeing this (intermittently) too - on RB 1.7.4 with Mercurial 2.3.2

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/core/handlers/base.py", line 111, in get_response

    response = callback(request, *callback_args, **callback_kwargs)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/accounts/decorators.py", line 22, in _check
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/views.py", line 728, in review_detail
    'screenshots': screenshots,
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/views.py", line 124, in _make_review_request_context
    upload_diff_form = UploadDiffForm(review_request)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/forms.py", line 276, in __init__
    data, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/diffviewer/forms.py", line 51, in __init__
    if self.repository.get_scmtool().get_diffs_use_absolute_paths():
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/models.py", line 134, in get_scmtool
    return cls(self)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/hg.py", line 31, in __init__
    self.client = HgClient(repository.path, repository.local_site)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/hg.py", line 239, in __init__
    if parse_version(hg_version) <= parse_version("1.2"):
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1956, in parse_version

    for part in _parse_version_parts(s.lower()):

TypeError: 'NoneType' object is not callable

I've modified the code to add a log statement of what "hg_version" is. Will report back with anything interesting.

Adam

[adam.c...@gmail.com]

Hi again,

So. With more and more logging after I was getting more and more paranoid, I've finally hit a wall trying to figure out why this sometimes fails.

I have confirmed through logging that
* hg_version is never None (unsurprisingly it's always the string "2.3.2") immediately before the call to parse_version()
* parse_version is as expected (function in pkg_resources), and is at the same memory address for successive working and then failing calls.

* hg_version itself always comes from the mercurial.util.version case

* mercurial.__file__ is always the same: /usr/lib/python2.7/dist-packages/mercurial/__init__.pyc

* Getting a shallow copy through slicing of hg_version doesn't change anything (i added [:] at the end of the call to mercurial.util.version)

* Once it starts going wrong, it tends to stay wrong until Apache (through mod_wsgi) switches requests to a different thread (or is this thread being killed?)

* Parsing the hard-coded "1.2" string never fails

Since I wasn't getting anywhere with fixing it, I've just hard-coded the > 1.2 case.

My only thoughts as to how this could be "None"d between the log statement and the call to parse_version would be if the thread was being killed and Python starts setting variables to None. I have no evidence to suggest this is happening, just a guess.

I can provide log statements which show the above conclusions if anyone's interested (contains PID, Python thread's ident and name, repr(parse_version), repr(hg_version), mercurial.__file__)

Any insights on what's going on here would be gratefully accepted! Although I've worked around it, this one has piqued my curiosity so I'm keen to understand what's happening.

Adam

[Raja Venkataraman]

I dont think None is passed into the parse_version because if that is the case, then an AttributeError should be thrown.(AttributeError: 'NoneType' object has no attribute 'lower') but the error message here is different. 

It seems pkg_resources's _parse_version_parts is yielding the values back to the line 1956 (where the error happens). If it were to not yield anything then the line 

for part in _parse_version_parts(s.lower()):

would throw an error as the NoneType that its yielding cannot be called and hence the error. But Im not clear how a string with value 2.3.2 will not yield any value .

Regards

Raja

[Adam Collard]

Hi,

Good point Raja. I was so hung up on it being hg_version that was None I didn't even read the tracback carefully. I knew sharing this would help, thank you!

Now reading it more carefully i think it's one of two situations:

1. _parse_version_parts is None.

2. s.lower is None.

I don't think it's possible to say which at this stage. I don't think it's the result from the call that is None otherwise you'd get a different exception again (NoneType object is not iterable).

I'll add some debugging tomorrow inside pkg_resources to help identify which of the two it is, but I have my suspicions :)

Adam

--
 
---
You received this message because you are subscribed to the Google Groups "reviewboard-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Colin Caughie]

Hi,

Adam did you manage to get any further with this? We're still having this problem on an intermittent basis with RB 1.7.5. I added the same type of logging that you did and came to much the same conclusions you did. I've established that I can call hg_version.lower() before the call to parse_version() and get the correct result, but I haven't yet figured out how to put logging into the pkg_resources module in order to debug further.

Thanks,
Colin

On 2/6/2013 12:12 PM, Adam Collard wrote:

Hi,

Good point Raja. I was so hung up on it being hg_version that was None I didn't even read the tracback carefully. I knew sharing this would help, thank you!

Now reading it more carefully i think it's one of two situations:

1. _parse_version_parts is None.

2. s.lower is None.

I don't think it's possible to say which at this stage. I don't think it's the result from the call that is None otherwise you'd get a different exception again (NoneType object is not iterable).

I'll add some debugging tomorrow inside pkg_resources to help identify which of the two it is, but I have my suspicions :)

Adam

On 6 Feb 2013, at 21:48, Raja Venkataraman <raja...@gmail.com> wrote:

I dont think None is passed into the parse_version because if that is the case, then an AttributeError should be thrown.(AttributeError: 'NoneType' object has no attribute 'lower') but the error message here is different.�

It seems pkg_resources's _parse_version_parts is yielding the values back to the line 1956 (where the error happens). If it were to not yield anything then the line�

for part in _parse_version_parts(s.lower()):

would throw an error as the NoneType that its yielding cannot be called and hence the error. But Im not clear how a string with value 2.3.2 will not yield any value .

Regards

Raja

On Wednesday, February 6, 2013 6:33:06 PM UTC+5:30, Adam wrote:

Hi again,

So. With more and more logging after I was getting more and more paranoid, I've finally hit a wall trying to figure out why this sometimes fails.

I have confirmed through logging that
* hg_version is never None (unsurprisingly it's always the string "2.3.2") immediately before the call to parse_version()
* parse_version is as expected (function in pkg_resources), and is at the same memory address for successive working and then failing calls.

* hg_version itself always comes from the mercurial.util.version case

* mercurial.__file__ is always the same: /usr/lib/python2.7/dist-packages/mercurial/__init__.pyc

* Getting a shallow copy through slicing of hg_version doesn't change anything (i added [:] at the end of the call to mercurial.util.version)

* Once it starts going wrong, it tends to stay wrong until Apache (through mod_wsgi) switches requests to a different thread (or is this thread being killed?)

* Parsing the hard-coded "1.2" string never fails

Since I wasn't getting anywhere with fixing it, I've just hard-coded the > 1.2 case.

My only thoughts as to how this could be "None"d between the log statement and the call to parse_version would be if the thread was being killed and Python starts setting variables to None. I have no evidence to suggest this is happening, just a guess.

I can provide log statements which show the above conclusions if anyone's interested (contains PID, Python thread's ident and name, repr(parse_version), repr(hg_version), mercurial.__file__)

Any insights on what's going on here would be gratefully accepted! Although I've worked around it, this one has piqued my curiosity so I'm keen to understand what's happening.

Adam

On 4 February 2013 11:56, adam.c...@gmail.com <adam.c...@gmail.com> wrote:

Hi,

We're seeing this (intermittently) too - on RB 1.7.4 with Mercurial 2.3.2

Traceback (most recent call last):

� File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/core/handlers/base.py", line 111, in get_response

��� response = callback(request, *callback_args, **callback_kwargs)

� File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/accounts/decorators.py", line 22, in _check
��� return view_func(*args, **kwargs)
� File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/views.py", line 728, in review_detail
��� 'screenshots': screenshots,
� File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/views.py", line 124, in _make_review_request_context
��� upload_diff_form = UploadDiffForm(review_request)
� File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/forms.py", line 276, in __init__
��� data, *args, **kwargs)
� File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/diffviewer/forms.py", line 51, in __init__
��� if self.repository.get_scmtool().get_diffs_use_absolute_paths():
� File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/models.py", line 134, in get_scmtool
��� return cls(self)
� File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/hg.py", line 31, in __init__
��� self.client = HgClient(repository.path, repository.local_site)
� File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/hg.py", line 239, in __init__
��� if parse_version(hg_version) <= parse_version("1.2"):
� File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1956, in parse_version

��� for part in _parse_version_parts(s.lower()):

TypeError: 'NoneType' object is not callable

I've modified the code to add a log statement of what "hg_version" is. Will report back with anything interesting.

Adam

On 22 January 2013 22:13, Christian Hammond <chi...@gmail.com> wrote:

Today seems to be the day for weird bugs. I'd be very interested in seeing if you hit it again and if logging turns up anything.

To get to mercurial.__file__, you can do:

� � import mercurial
� � mercurial.__file__

Christian

>>>>> There was another report of this recently, but I don't have it handy right now. Basically, for some reason, mercurial's version is being reported to us as None, which is� strange. Not to mention that the only difference in 1.7.x in that code is that instead of parsing the version we get from Mercurial in that code, we're using another function to parse it.

>>>>>
>>>>> Can you tell me what version of mercurial you have installed? And though I expect the result will be the same as what I saw before, can you try:
>>>>>

>>>>> � � $ python
>>>>> � � >>> from mercurial.__version__ import version
>>>>> � � >>> print version
>>>>>
>>>>> Christian
>

--
�

---
You received this message because you are subscribed to the Google Groups "reviewboard-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

�
�

--
�

---
You received this message because you are subscribed to the Google Groups "reviewboard-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

�
�

[Colin Caughie]

I've made a couple more observations on this:

1. This is going to sound really weird, but it always happens on Monday mornings. Sometimes it carries on through Tuesday morning but for the past few weeks I haven't seen it from Tuesday afternoon onwards. Maybe this is just pure fluke but I've observed this pattern for several weeks now.

2. I imported the "logging" module into pkg_resources.py itself and traced the values being passed into parse_version() using logging.warn(). I did this last Monday afternoon and the problem didn't repeat again until this morning; until that point I saw expected values being traced. This morning the failure occurred, but this time the exception came from the logging.warn() statement and was:

��� logging.warn('parse_version(%s), lower is %s)', s, s.lower())
AttributeError: 'NoneType' object has no attribute 'warn'

So it's kind of looking like at some point the whole of pkg_resources.py just gets somehow messed up and a bunch of its globals somehow get turned into None. I don't know enough about the innards of Python to be able to even guess what might be causing this.

Unless anyone can figure out what's going on here I'm thinking of submitting a patch that reverts hg.py to doing its own version parsing, even if it means copying code from pkg_resources.py. Would such a patch be accepted?

Thanks,
Colin

[Christian Hammond]

I suspect this is somehow related to some of the code in our extension support.

Have you upgraded to 1.7.7.1? I'm very curious if the problem goes away, as we changed the logic there.

Christian

-- 
Christian Hammond - chi...@chipx86.com
Review Board - http://www.reviewboard.org

Beanbag, Inc. - http://www.beanbaginc.com

On May 6, 2013, at 8:52 AM, Colin Caughie <c.ca...@gmail.com> wrote:

I've made a couple more observations on this:

1. This is going to sound really weird, but it always happens on Monday mornings. Sometimes it carries on through Tuesday morning but for the past few weeks I haven't seen it from Tuesday afternoon onwards. Maybe this is just pure fluke but I've observed this pattern for several weeks now.

2. I imported the "logging" module into pkg_resources.py itself and traced the values being passed into parse_version() using logging.warn(). I did this last Monday afternoon and the problem didn't repeat again until this morning; until that point I saw expected values being traced. This morning the failure occurred, but this time the exception came from the logging.warn() statement and was:

    logging.warn('parse_version(%s), lower is %s)', s, s.lower())
AttributeError: 'NoneType' object has no attribute 'warn'

So it's kind of looking like at some point the whole of pkg_resources.py just gets somehow messed up and a bunch of its globals somehow get turned into None. I don't know enough about the innards of Python to be able to even guess what might be causing this.

Unless anyone can figure out what's going on here I'm thinking of submitting a patch that reverts hg.py to doing its own version parsing, even if it means copying code from pkg_resources.py. Would such a patch be accepted?

Thanks,
Colin

On 4/29/2013 9:40 AM, Colin Caughie wrote:

Hi,

Adam did you manage to get any further with this? We're still having this problem on an intermittent basis with RB 1.7.5. I added the same type of logging that you did and came to much the same conclusions you did. I've established that I can call hg_version.lower() before the call to parse_version() and get the correct result, but I haven't yet figured out how to put logging into the pkg_resources module in order to debug further.

Thanks,
Colin

On 2/6/2013 12:12 PM, Adam Collard wrote:

Hi,

Good point Raja. I was so hung up on it being hg_version that was None I didn't even read the tracback carefully. I knew sharing this would help, thank you!

Now reading it more carefully i think it's one of two situations:

1. _parse_version_parts is None.

2. s.lower is None.

I don't think it's possible to say which at this stage. I don't think it's the result from the call that is None otherwise you'd get a different exception again (NoneType object is not iterable).

I'll add some debugging tomorrow inside pkg_resources to help identify which of the two it is, but I have my suspicions :)

Adam

On 6 Feb 2013, at 21:48, Raja Venkataraman <raja...@gmail.com> wrote:

I dont think None is passed into the parse_version because if that is the case, then an AttributeError should be thrown.(AttributeError: 'NoneType' object has no attribute 'lower') but the error message here is different. 

It seems pkg_resources's _parse_version_parts is yielding the values back to the line 1956 (where the error happens). If it were to not yield anything then the line 

for part in _parse_version_parts(s.lower()):

would throw an error as the NoneType that its yielding cannot be called and hence the error. But Im not clear how a string with value 2.3.2 will not yield any value .

Regards

Raja

On Wednesday, February 6, 2013 6:33:06 PM UTC+5:30, Adam wrote:

Hi again,

So. With more and more logging after I was getting more and more paranoid, I've finally hit a wall trying to figure out why this sometimes fails.

I have confirmed through logging that
* hg_version is never None (unsurprisingly it's always the string "2.3.2") immediately before the call to parse_version()
* parse_version is as expected (function in pkg_resources), and is at the same memory address for successive working and then failing calls.

* hg_version itself always comes from the mercurial.util.version case

* mercurial.__file__ is always the same: /usr/lib/python2.7/dist-packages/mercurial/__init__.pyc

* Getting a shallow copy through slicing of hg_version doesn't change anything (i added [:] at the end of the call to mercurial.util.version)

* Once it starts going wrong, it tends to stay wrong until Apache (through mod_wsgi) switches requests to a different thread (or is this thread being killed?)

* Parsing the hard-coded "1.2" string never fails

Since I wasn't getting anywhere with fixing it, I've just hard-coded the > 1.2 case.

My only thoughts as to how this could be "None"d between the log statement and the call to parse_version would be if the thread was being killed and Python starts setting variables to None. I have no evidence to suggest this is happening, just a guess.

I can provide log statements which show the above conclusions if anyone's interested (contains PID, Python thread's ident and name, repr(parse_version), repr(hg_version), mercurial.__file__)

Any insights on what's going on here would be gratefully accepted! Although I've worked around it, this one has piqued my curiosity so I'm keen to understand what's happening.

Adam

On 4 February 2013 11:56, adam.c...@gmail.com <adam.c...@gmail.com> wrote:

Hi,

We're seeing this (intermittently) too - on RB 1.7.4 with Mercurial 2.3.2

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/Django-1.4.3-py2.7.egg/django/core/handlers/base.py", line 111, in get_response

    response = callback(request, *callback_args, **callback_kwargs)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/accounts/decorators.py", line 22, in _check

    return view_func(*args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/views.py", line 728, in review_detail

    'screenshots': screenshots,

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/views.py", line 124, in _make_review_request_context

    upload_diff_form = UploadDiffForm(review_request)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/reviews/forms.py", line 276, in __init__

    data, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/diffviewer/forms.py", line 51, in __init__

    if self.repository.get_scmtool().get_diffs_use_absolute_paths():

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/models.py", line 134, in get_scmtool

    return cls(self)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/hg.py", line 31, in __init__

    self.client = HgClient(repository.path, repository.local_site)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.4-py2.7.egg/reviewboard/scmtools/hg.py", line 239, in __init__

    if parse_version(hg_version) <= parse_version("1.2"):

  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1956, in parse_version

    for part in _parse_version_parts(s.lower()):

TypeError: 'NoneType' object is not callable

I've modified the code to add a log statement of what "hg_version" is. Will report back with anything interesting.

Adam

On 22 January 2013 22:13, Christian Hammond <chi...@gmail.com> wrote:

Today seems to be the day for weird bugs. I'd be very interested in seeing if you hit it again and if logging turns up anything.

To get to mercurial.__file__, you can do:

    import mercurial
    mercurial.__file__

Christian

>>>>> There was another report of this recently, but I don't have it handy right now. Basically, for some reason, mercurial's version is being reported to us as None, which is… strange. Not to mention that the only difference in 1.7.x in that code is that instead of parsing the version we get from Mercurial in that code, we're using another function to parse it.

>>>>>
>>>>> Can you tell me what version of mercurial you have installed? And though I expect the result will be the same as what I saw before, can you try:
>>>>>

>>>>>     $ python
>>>>>     >>> from mercurial.__version__ import version
>>>>>     >>> print version
>>>>>
>>>>> Christian
>

--
 

---
You received this message because you are subscribed to the Google Groups "reviewboard-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

 
---
You received this message because you are subscribed to the Google Groups "reviewboard-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard-d...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

[Colin Caughie]

No I haven't - I'll try that before I go any further.

Thanks,
Colin

On 5/6/2013 10:43 AM, Christian Hammond wrote:

I suspect this is somehow related to some of the code in our extension support.

Have you upgraded to 1.7.7.1? I'm very curious if the problem goes away, as we changed the logic there.

Christian

--�
Christian Hammond -�chi...@chipx86.com
Review Board -�http://www.reviewboard.org

Beanbag, Inc. - http://www.beanbaginc.com

On May 6, 2013, at 8:52 AM, Colin Caughie <c.ca...@gmail.com> wrote:

I've made a couple more observations on this:

1. This is going to sound really weird, but it always happens on Monday mornings. Sometimes it carries on through Tuesday morning but for the past few weeks I haven't seen it from Tuesday afternoon onwards. Maybe this is just pure fluke but I've observed this pattern for several weeks now.

2. I imported the "logging" module into pkg_resources.py itself and traced the values being passed into parse_version() using logging.warn(). I did this last Monday afternoon and the problem didn't repeat again until this morning; until that point I saw expected values being traced. This morning the failure occurred, but this time the exception came from the logging.warn() statement and was:

��� logging.warn('parse_version(%s), lower is %s)', s, s.lower())

[Colin Caughie]

That did the trick - Monday has been and gone and the problem did not occur. Thanks!

Unfortunately 1.7.7.1 did introduce the arguably more serious issue 2971 (https://code.google.com/p/reviewboard/issues/detail?id=2971). If my fix (http://reviews.reviewboard.org/r/4121/) could make it into a release soon it would be much appreciated. :)

Thanks,
Colin

[Christian Hammond]

Hi Colin,

I'll take a look shortly. We should be getting a release out this week or next. Sorry for the trouble, and thanks for the patch :)

Christian

--
Christian Hammond - chi...@chipx86.com

Review Board - http://www.reviewboard.org

Beanbag, Inc. - http://www.beanbaginc.com

On Tue, May 14, 2013 at 8:06 AM, Colin Caughie <c.ca...@gmail.com> wrote:

That did the trick - Monday has been and gone and the problem did not occur. Thanks!

Unfortunately 1.7.7.1 did introduce the arguably more serious issue 2971 (https://code.google.com/p/reviewboard/issues/detail?id=2971). If my fix (http://reviews.reviewboard.org/r/4121/) could make it into a release soon it would be much appreciated. :)

Thanks,
Colin

On 5/6/2013 11:10 AM, Colin Caughie wrote:

No I haven't - I'll try that before I go any further.

Thanks,
Colin

On 5/6/2013 10:43 AM, Christian Hammond wrote:

I suspect this is somehow related to some of the code in our extension support.

Have you upgraded to 1.7.7.1? I'm very curious if the problem goes away, as we changed the logic there.

Christian

-- 
Christian Hammond - chi...@chipx86.com

Review Board - http://www.reviewboard.org

Beanbag, Inc. - http://www.beanbaginc.com

On May 6, 2013, at 8:52 AM, Colin Caughie <c.ca...@gmail.com> wrote:

I've made a couple more observations on this:

1. This is going to sound really weird, but it always happens on Monday mornings. Sometimes it carries on through Tuesday morning but for the past few weeks I haven't seen it from Tuesday afternoon onwards. Maybe this is just pure fluke but I've observed this pattern for several weeks now.

2. I imported the "logging" module into pkg_resources.py itself and traced the values being passed into parse_version() using logging.warn(). I did this last Monday afternoon and the problem didn't repeat again until this morning; until that point I saw expected values being traced. This morning the failure occurred, but this time the exception came from the logging.warn() statement and was:

    logging.warn('parse_version(%s), lower is %s)', s, s.lower())
AttributeError: 'NoneType' object has no attribute 'warn'

So it's kind of looking like at some point the whole of pkg_resources.py just gets somehow messed up and a bunch of its globals somehow get turned into None. I don't know enough about the innards of Python to be able to even guess what might be causing this.

Unless anyone can figure out what's going on here I'm thinking of submitting a patch that reverts hg.py to doing its own version parsing, even if it means copying code from pkg_resources.py. Would such a patch be accepted?

Thanks,
Colin

[Ali Saidi]

FYI, this bug is related to an issue we started having when we upgraded to 1.7.9 related to posting diffs. For some diffs reviewboard would fail to cleanly apply the diffs and inspecting the tmp directory it seems like hgweb was returning a standard web page instead of the file that reviewboard was requesting from the mercurial server. Applying the patch (#4121) solve this problem for us as well.

Thanks,

Ali