Issue 2686 in reviewboard: 1.6.10 breaks svn+ssh for Subversion

[revie...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

What version are you running?
1.6.10

What's the URL of the page containing the problem?
Any page that requests a diff and the respository edit pages.

What steps will reproduce the problem?
1. Take a working 1.6.9 install using Subversion with svn+ssh.
2. Update to 1.6.10 using rb-site script.
3. Attempt to access a review that doesn't have a cached diff which is
hosted by a Subversion svn+ssh respository.

What is the expected output? What do you see instead?
The diffs should load.

Rather, the logs display the following:

13:25:32 DEBUG
SVNTool: Attempting ssh connection with host: ********, username:
reviewboard
13:25:32 DEBUG
starting thread (client mode): 0x92628bd0L
13:25:32 INFO
Connected (version 2.0, client OpenSSH_5.3p1)
13:25:32 DEBUG
kex
algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1']
server key:['ssh-rsa', 'ssh-dss'] client
encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijnda...@lysator.liu.se']
server
encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijnda...@lysator.liu.se']
client
mac:['hmac-md5', 'hmac-sha1', 'uma...@openssh.com', 'hmac-ripemd160', 'hmac-ri...@openssh.com', 'hmac-sha1-96', 'hmac-md5-96']
server
mac:['hmac-md5', 'hmac-sha1', 'uma...@openssh.com', 'hmac-ripemd160', 'hmac-ri...@openssh.com', 'hmac-sha1-96', 'hmac-md5-96']
client compress:['none', 'zl...@openssh.com'] server
compress:['none', 'zl...@openssh.com'] client lang:[''] server lang:[''] kex
follows?False
13:25:32 DEBUG
Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
13:25:32 DEBUG
using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher:
local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote
hmac-sha1; compression: local none, remote none
13:25:32 DEBUG
Switch to new keys ...
13:25:32 DEBUG
Trying SSH key 6d41f76410303cb216b7b8f7c687dab6
13:25:32 DEBUG
userauth is OK
13:25:32 INFO
Authentication (publickey) successful!
13:25:32 DEBUG
EOF in transport thread
13:25:33 ERROR
SVN: Failed to get repository information for
svn+ssh://reviewboard@********/home/svn/repo: Network connection closed
unexpectedly
13:25:33 DEBUG
EOF in transport thread

What operating system are you using? What browser?
Windows, Chrome

Please provide any additional information below.
I downgraded to 1.6.9 by using easy_install to install 1.6.9 and ran the
rb-site script against the install. I realize that any database schema
changes between 1.6.9 and 1.6.10 will cause other breakage, but when I did
this, the svn+ssh connection began working again.

[revie...@googlecode.com]

Updates:
Status: NeedInfo

Comment #1 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Are you sure it wasn't coincidental timing? We don't have any changes to
either rbssh or svn in this release.

[revie...@googlecode.com]

Comment #2 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I don't know how it could be since it seems to work correctly with version
1.6.9. Is there something I can do to get more information about the
failure? Everything online seems to indicate that it is an SSH issue, but I
have tested and retested those settings. Could this be due to a change in
the version of a dependency in this version of ReviewBoard?

[revie...@googlecode.com]

Comment #3 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

No dependency changes were made. The only SCM code that changed was some
file parsing for CVS and some hosting support for Mercurial, neither of
which comes anywhere near the Subversion code.

Certainly nothing changed to impact SSH/rb-ssh code.

I believe it's working in 1.6.9, but that traceback is showing some
communication error between rb-site and the repository. That's why I'm
wondering if there happened to be some downtime as you were trying 1.6.10
that resolved itself (or someone resolved it) after. Nothing else to me is
making sense right now.

I don't think there's anything more to get out of that failure. As it said,
the connection was closed unexpectedly, presumably because of a server
issue or it not liking something that was sent across.

Would you be able to set up a 1.6.10 on a developer box/VM/virtualenv and
see if it still happens?

[revie...@googlecode.com]

Comment #4 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I moved the /var/lib/reviewboard to /var/lib/reviewboard-backup and did a
fresh install of 1.6.10 to /var/lib/reviewboard using sqlite as the
database. This should ensure that there wasn't anything in my existing
database confusing things. I then restarted memcached and apache2.

After going to the install the first time, it informed me that the data
directory had the wrong permissions. I ran the command to fix them.

I added a repository and was asked to trust the host. The following
error: "Unable to authenticate against this repository using one of the
supported authentication types." I copied the old id_rsa over using the
following command. I have tested this id_rsa with ssh -i and connected
successfully to the svn server.

sudo cp -a /var/lib/reviewboard-backup/data/.ssh/id_rsa
/var/lib/reviewboard/data/.ssh/id_rsa

ReviewBoard didn't pick this up, so I downloaded the key from the server
and used the ReviewBoard UI to upload the SSH key. This picked up the key.
I then tried to add the repository again and saw the following error: "A
repository was not found at the specified path."

I went ahead and did the same process described above using ReviewBoard
1.6.9 using "sudo easy_install ReviewBoard==1.6.9" and the rest of the
procedure is the same. The result was that no errors were thrown and the
repository is added without problems.

[revie...@googlecode.com]

Comment #5 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

By the way, the server I am running on is Ubuntu 10.04.4 LTS -- "Linux
reviewboard 2.6.32-30-generic #59-Ubuntu SMP Tue Mar 1 21:30:46 UTC 2011
x86_64 GNU/Linux" with Python 2.6.5

[revie...@googlecode.com]

Comment #6 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Very odd. Do you have a log file from that attempt? It should cover more
what was happening there.

[revie...@googlecode.com]

Comment #7 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I am currently setting up a VM with CentOS 5 to see if that makes a
difference. The log from the previous attempt follows:

13:45:09 DEBUG

SVNTool: Attempting ssh connection with host: ********, username:
reviewboard

13:45:09 DEBUG

starting thread (client mode): 0x71514790L

13:45:09 INFO

Connected (version 2.0, client OpenSSH_5.3p1)

13:45:09 DEBUG

kex
algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1']
server key:['ssh-rsa', 'ssh-dss'] client
encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijnda...@lysator.liu.se']
server
encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijnda...@lysator.liu.se']
client
mac:['hmac-md5', 'hmac-sha1', 'uma...@openssh.com', 'hmac-ripemd160', 'hmac-ri...@openssh.com', 'hmac-sha1-96', 'hmac-md5-96']
server
mac:['hmac-md5', 'hmac-sha1', 'uma...@openssh.com', 'hmac-ripemd160', 'hmac-ri...@openssh.com', 'hmac-sha1-96', 'hmac-md5-96']
client compress:['none', 'zl...@openssh.com'] server
compress:['none', 'zl...@openssh.com'] client lang:[''] server lang:[''] kex
follows?False

13:45:09 DEBUG

Ciphers agreed: local=aes128-ctr, remote=aes128-ctr

13:45:09 DEBUG

using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher:
local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote
hmac-sha1; compression: local none, remote none

13:45:09 DEBUG

Switch to new keys ...

13:45:09 DEBUG

Trying SSH key 6d41f76410303cb216b7b8f7c687dab6

13:45:09 DEBUG

userauth is OK

13:45:09 INFO

Authentication (publickey) successful!

13:45:09 DEBUG

EOF in transport thread

13:45:09 ERROR

SVN: Failed to get repository information for

svn+ssh://reviewboard@********/home/svn/********: Network connection closed
unexpectedly

13:45:09 DEBUG

EOF in transport thread

[revie...@googlecode.com]

Comment #8 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Try setting the DEBUG_RBSSH to 1 in the environment for Apache (SetEnv),
and set DEBUG_LOGDIR to the directory you're using for reviewboard.log.
There may be some more info there.

[revie...@googlecode.com]

Comment #9 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I added both of those to my apache configuration in the VirtualHost
associated with ReviewBoard at the beginning as follows, but I didn't see
any new information appear in the log and I don't see any new log files.

<VirtualHost *:80>
SetEnv DEBUG_RBSSH 1
SetEnv DEBUG_LOGDIR /var/lib/reviewboard/logs/

ServerName reviewboard
DocumentRoot "/var/lib/reviewboard/htdocs"
.
.
.

Also, I have run into issues using CentOS as a base since the default yum
repos don't have all the needed packages. It appears from the documentation
that the Debian-based distros are the preferred bases anyhow so I am
probably giving up on that avenue.

[revie...@googlecode.com]

Comment #10 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Debian-based distros are definitely a lot easier. Fedora's not bad either,
since they maintain up-to-date RB packages.

Sorry, I was in a rush and gave you the wrong environment variable for the
log directory. It should be RBSSH_LOG_DIR.

[revie...@googlecode.com]

Comment #11 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I made that change, but it still isn't giving me the additional output. I
made sure to restart apache2 and memcached. I tried moving the SetEnv
options into the Location tag with the other ones in case that made a
difference too. It didn't. I noticed in the apache config, there is also a
reference to PythonDebug Off. Is that something I can turn on to get more
data or is that more for gdb-style stepping?

<VirtualHost *:80>
ServerName reviewboard
DocumentRoot "/var/lib/reviewboard/htdocs"

# Error handlers
ErrorDocument 500 /errordocs/500.html

# Serve django pages
<Location "/">
PythonPath "['/var/lib/reviewboard/conf'] + sys.path"
SetEnv DEBUG_RBSSH 1
SetEnv RBSSH_LOG_DIR "/var/lib/reviewboard/logs/"
SetEnv DJANGO_SETTINGS_MODULE reviewboard.settings
SetEnv PYTHON_EGG_CACHE "/var/lib/reviewboard/tmp/egg_cache"
SetEnv HOME "/var/lib/reviewboard/data"
SetHandler mod_python
PythonHandler django.core.handlers.modpython
PythonAutoReload Off
PythonDebug Off
# Used to run multiple mod_python sites in the same apache
PythonInterpreter reviewboard_reviewboard
</Location>
.
.
.

(I am heading out for the day, but I will check back tomorrow morning to
see if you have any other suggestions for how to get more debug output and
help solve this problem. I really appreciate your help so far, thanks!)

[revie...@googlecode.com]

Comment #12 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I have reverted back to 1.6.9 on Ubuntu and that is working just fine. If
others are running into the same issue described in this bug report, please
post here and maybe we can gather enough information to see what is causing
this problem. Thanks!

[revie...@googlecode.com]

Comment #13 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

PythonDebug Off shouldn't be related.

I'm surprised the SetEnv calls didn't work.

I'm trying to reproduce this bug here, but haven't succeeded yet. We run
unit tests nightly that cover SSH-backed SVN repositories, and haven't hit
anything, and as I said, there's nothing in 1.6.10 that relates to this in
any form. So, I'm really unsure of what's going on.

[revie...@googlecode.com]

Comment #14 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Just to be completely sure, the log files should be rbssh-<pid>.log in that
directory.

[revie...@googlecode.com]

Comment #15 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

One thing you could try (I know you reverted back now) is to run this
manually, as Apache.


$ sudo -s
$ sudo -u apache2 -s
$ DEBUG_RBSSH=1 rbssh <url from the log file>

And just see what's outputted to the terminal.

[revie...@googlecode.com]

Comment #16 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Forgot a step. You'll need to set HOME to be /var/lib/reviewboard/data as
well.

[revie...@googlecode.com]

Comment #17 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I still have the virtual machine that is running 1.6.10 and is broken so I
was able to generate the following output using the method you described.

/usr/local/lib/python2.6/dist-packages/pycrypto-2.3-py2.6-linux-x86_64.egg/Crypto/Util/randpool.py:40:
RandomPool_DeprecationWarning:
This application uses RandomPool, which is
BROKEN in older releases. See http://www.pycrypto.org/randpool-broken
RandomPool_DeprecationWarning)
Traceback (most recent call last):
File "/usr/local/bin/rbssh", line 9, in <module>
load_entry_point('ReviewBoard==1.6.10', 'console_scripts', 'rbssh')()
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 299, in
load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2229, in
load_entry_point
return ep.load()
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1948, in
load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])

File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/cmdline/rbssh.py",
line 42, in <module>
from reviewboard.scmtools import sshutils

File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/scmtools/sshutils.py",
line 8, in <module>
from reviewboard.scmtools.errors import AuthenticationError, \

File "/usr/local/lib/python2.6/dist-packages/ReviewBoard-1.6.10-py2.6.egg/reviewboard/scmtools/errors.py",
line 4, in <module>
from djblets.util.templatetags.djblets_utils import humanize_list

File "/usr/local/lib/python2.6/dist-packages/Djblets-0.6.21-py2.6.egg/djblets/util/templatetags/djblets_utils.py",
line 31, in <module>
from django.contrib.auth.models import AnonymousUser, User

File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/contrib/auth/models.py",
line 7, in <module>
from django.db import models

File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/db/__init__.py",
line 14, in <module>
if not settings.DATABASES:

File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/utils/functional.py",
line 276, in __getattr__
self._setup()

File "/usr/local/lib/python2.6/dist-packages/Django-1.3.1-py2.6.egg/django/conf/__init__.py",
line 40, in _setup
raise ImportError("Settings cannot be imported, because environment
variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable
DJANGO_SETTINGS_MODULE is undefined.

[revie...@googlecode.com]

Updates:
Status: Confirmed
Owner: chip...@gmail.com
Labels: Component-SCMTools

Comment #18 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Now that is interesting, and gets us much further. While I don't know why
I'm not hitting it, I think I can come up with a repro case for 1.6.11 now.

[revie...@googlecode.com]

Updates:
Status: Fixed
Labels: Djblets

Comment #19 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

This has been fixed on the release-0.6.x branch of Djblets (e940ca0). You
can manually download and install this, and hopefully it will fix the
problem. RB 1.6.11 (coming this week) will require this version, so it'll
be fixed along with that release.

Thanks for all the debugging info and being patient with this. Hopefully
this fixes the problem for you! I'd definitely be interested in knowing if
a custom-built Djblets from this branch fixes the problem for you.

[revie...@googlecode.com]

Comment #20 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

That sounds great! I would be happy to test this ahead of the 1.6.11
release. I'm not familiar with the process of the manual install you
mentioned. If you could direct me to some docs for that, I will go ahead
and test. Thanks!

[revie...@googlecode.com]

Comment #21 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

You would have to check out Djblets from git
(http://github.com/djblets/djblets/), check out the release-1.6.x branch,
and then do:

$ sudo ./setup.py install

Then just restart/reload your web server.

[revie...@googlecode.com]

Comment #22 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I did the following:

$ git clone -b release-0.6.x https://github.com/djblets/djblets.git
$ cd djblets/
$ sudo ./setup.py install
$ sudo service memcached restart
$ sudo service apache2 restart

Unfortunately, I got the same error through the ReviewBoard UI. I ran the
same rbssh command you had me run before as the www-data user and got the
following output:

[revie...@googlecode.com]

Comment #23 on issue 2686 by chip...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Upgrade to 1.6.11 and see if this is fixed for you.

[revie...@googlecode.com]

Comment #24 on issue 2686 by cea...@gmail.com: 1.6.10 breaks svn+ssh for
Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

It looks like ReviewBoard 1.6.11 has resolved the issue for me. :) Thanks
so much for working with me on this!

[revie...@googlecode.com]

Comment #25 on issue 2686 by alin.pan...@gmail.com: 1.6.10 breaks svn+ssh
for Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

I just upgraded from 1.6.9 to 1.6.14. My setup also used svn+ssh, and had
some diff files uploaded. After the upgrade, I am facing the same problem.
The reviewboard server log says:
DEBUG userauth is OK
INFO Authentication (publickey) successful!

DEBUG EOF in transport thread

ERROR SVN: Failed to get repository information for

svn+ssh://reviewboard@***: To better debug SSH connection problems, remove
the -q option from 'ssh' in the [tunnels] section of your Subversion
configuration file. Network connection closed unexpectedly

As suggested above, I ran:

$ sudo -s
$ sudo -u apache2 -s
$ DEBUG_RBSSH=1 rbssh <url from the log file>

It got me the following output:
/usr/lib/python2.4/site-packages/paramiko-1.7.7.2-py2.4.egg/paramiko/client.py:94:
UserWarning:
Unknown ssh-rsa host key for ***
Error connecting to server: No authentication methods available

Additional "pip freeze" info:
Django==1.3.4
Djblets==0.6.27
MySQL-python==1.2.3
PIL==1.1.6
Pygments==1.5
RBTools==0.4.1
ReviewBoard==1.6.14
django-evolution==0.6.7
feedparser==5.1.2
flup==1.0.3.dev-20110405
paramiko==1.7.7.2
pycrypto==2.6
python-dateutil==1.5
python-memcached==1.48
pytz==2012c
recaptcha-client==1.0.6
simplejson==2.0.9

Any ideas as to why this is happening? I should point out that running "ssh
-i /var/www/review-site/data/.ssh/id_rsa reviewboard@***" is establishing
the connection just fine.

[revie...@googlecode.com]

Comment #26 on issue 2686 by andreasf...@gmail.com: 1.6.10 breaks svn+ssh
for Subversion
http://code.google.com/p/reviewboard/issues/detail?id=2686

Same problem with newest version 1.7.5
After upgrading from 1.6.1 the repository access is no more possible


rbssh gives the following:

SSH: Unknown error accessing user key: Private key file is encrypted

Traceback (most recent call last):

File "/usr/local/bin/rbssh", line 8, in <module>
load_entry_point('ReviewBoard==1.7.5', 'console_scripts', 'rbssh')()

File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-1.7.5-py2.7.egg/reviewboard/cmdline/rbssh.py",
line 330, in main
channel.get_pty()
File "build/bdist.linux-x86_64/egg/paramiko/channel.py", line 158, in
get_pty
File "build/bdist.linux-x86_64/egg/paramiko/channel.py", line 1114, in
_wait_for_event
paramiko.SSHException: Channel closed.

After downgrading to 1.6.9 everything works fine again