ReviewBoard 1.5 under Apache: IOError when posting review from command line post-review tool

[Rob]

Starting in ReviewBoard 1.5 (Python 2.5.1 linux64 compiled
mod_python.so) the changes to use Paramiko result in this exception
when posting a review changelist (from Perforce) via the commandline
post-review tool.

HTTP POSTing to http://sw-web3.altera.com/review/api/json/accounts/login/:
{'username': 'rromano', 'password': '**************'}

Traceback (most recent call last):

File "/tools/python/2.5.1/linux64/lib/python2.5/site-packages/
mod_python/importer.py", line 1537, in HandlerDispatch
default=default_handler, arg=req, silent=hlist.silent)

File "/tools/python/2.5.1/linux64/lib/python2.5/site-packages/
mod_python/importer.py", line 1229, in _process_target
result = _execute_target(config, req, object, arg)

File "/tools/python/2.5.1/linux64/lib/python2.5/site-packages/
mod_python/importer.py", line 1128, in _execute_target
result = object(arg)

File "/tools/django/trunk/1.2.3/django/core/handlers/modpython.py",
line 228, in handler
return ModPythonHandler()(req)

File "/tools/django/trunk/1.2.3/django/core/handlers/modpython.py",
line 191, in __call__
self.load_middleware()

File "/tools/django/trunk/1.2.3/django/core/handlers/base.py", line
40, in load_middleware
mod = import_module(mw_module)

File "/tools/django/trunk/1.2.3/django/utils/importlib.py", line 35,
in import_module
__import__(name)

File "/usr/local/altera/django/reviewboard/admin/middleware.py",
line 13, in <module>
from reviewboard.webapi.json import service_not_configured

File "/usr/local/altera/django/reviewboard/webapi/json.py", line 29,
in <module>
from reviewboard.diffviewer.forms import EmptyDiffError

File "/usr/local/altera/django/reviewboard/diffviewer/forms.py",
line 7, in <module>
from reviewboard.diffviewer.diffutils import
DEFAULT_DIFF_COMPAT_VERSION

File "/usr/local/altera/django/reviewboard/diffviewer/diffutils.py",
line 29, in <module>
from reviewboard.scmtools.core import PRE_CREATION, HEAD

File "/usr/local/altera/django/reviewboard/scmtools/core.py", line
5, in <module>
from reviewboard.scmtools import sshutils

File "/usr/local/altera/django/reviewboard/scmtools/sshutils.py",
line 5, in <module>
import paramiko

File "/usr/local/altera/django/reviewboard/paramiko/__init__.py",
line 69, in <module>
from transport import randpool, SecurityOptions, Transport

File "/usr/local/altera/django/reviewboard/paramiko/transport.py",
line 32, in <module>
from paramiko import util

File "/usr/local/altera/django/reviewboard/paramiko/util.py", line
32, in <module>
from paramiko.common import *

File "/usr/local/altera/django/reviewboard/paramiko/common.py", line
98, in <module>
from rng import StrongLockingRandomPool

File "/usr/local/altera/django/reviewboard/paramiko/rng.py", line
44, in <module>
if ((platform is not None and platform.system().lower() ==
'windows') or

File "/tools/python/2.5.1/linux64/lib/python2.5/platform.py", line
1042, in system
return uname()[0]

File "/tools/python/2.5.1/linux64/lib/python2.5/platform.py", line
1014, in uname
processor = _syscmd_uname('-p','')

File "/tools/python/2.5.1/linux64/lib/python2.5/platform.py", line
801, in _syscmd_uname
rc = f.close()

IOError: [Errno 10] No child processes


This looks very fundamental. Python's platform.uname() fn does a
subprocess call to `uname` and it's not liked.

Incidentally, under ReviewBoard 1.0.x, I had similar issues in
ReviewBoard's own code using subprocess.POpen to extract diffs from
`p4`. I simply changed the code to use os.system() and it worked fine
thereafter.

Are there known issues with the subprocess.POpen module when running
under Apache?

[Christian Hammond]

I'm not really sure. This is the first I've heard of this issue. The
call actually comes from paramiko, a Python module we use for the SSH
support. Might be worth seeing if other people outside Review Board
have hit that issue.

Christian

> --
> Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/
> Happy user? Let us know at http://www.reviewboard.org/users/
> -~----------~----~----~----~------~----~------~--~---
> To unsubscribe from this group, send email to reviewboard...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en

--
--
Christian Hammond - chi...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com

[Dennis]

Rob, have you seen this?

http://bugs.python.org/issue1731717

Implementing the proposed workaround in the latest entry of that
thread fixed a lot of the [Errno 10] problems for me--all of the ones
from the diff viewer, in fact.

I may have fixed the remaining ones from that 'uname -p' function call
in platform.py that you're seeing, by inserting the return string from
'uname -o' on my system to the output value of syscmd_uname.

It looks to me like the intent was to use 'uname -o' in the first
place, since that returns the operating system name, while uname -p
returns the processor type. Somehow, this function ends up calling
uname -p.

I assigned 'GNU/Linux' to the output of syscmd_uname in platform.py,
and so far I've not seen the [Errno 10] in 3 consecutive post-review
submissions.

I'm not certain that it's fixed completely, but it looks good so far.

Dennis

On Oct 13, 12:30 pm, Rob <rrom...@gmail.com> wrote:
> Starting in ReviewBoard 1.5 (Python 2.5.1 linux64 compiled
> mod_python.so)  the changes to use Paramiko result in this exception
> when posting a review changelist (from Perforce) via the commandline
> post-review tool.
>

> HTTP POSTing tohttp://sw-web3.altera.com/review/api/json/accounts/login/:

[Rob]

Thank you very much for the reference and tips. I'll give that a
try.

[Dennis]

You're welcome. I hope that helps.

In the interest of accuracy, I need to modify my post earlier
regarding the problems with platform.py.

After more thorough testing, I do not believe that these exceptions
are directly caused by any of the functions defined in platform.py.
Rather, I think they are all instances of race conditions between
subprocess creation and attempts to use the resulting subprocesses.

Note that platform.py (at least the version I have), uses os.popen()
to create subprocesses to make the system calls, not
subprocess.Popen().

It seems that in certain circumstances, with certain processors, etc.,
these same race condition issues can affect os.popen() calls as well
as subprocess.Popen().

Stubbing out the _cleanup() function to be a no-op, as recommended in
the bug thread to which I sent the link, dramatically reduced the
number of exceptions during diff viewing. It didn't totally eliminate
them. So, as also suggested in that thread, I've added a
time.sleep(0.1) line in perforce.py, between the call to open the
perforce cmdline subprocess and the line to communicate with it. I'm
hoping 100ms will provide enough time for Popen to win the race every
time.

A similar small wait may need to be added between platform.py module's
calls to os.popen and calls to read().

Good luck,
Dennis