RB, SVN, and AD

[Cathy Mullican]

I have RB 2.5.6.1 set up and working on Ubuntu 14.04, with AD authentication -- I can log in to RB as admin or as my domain user.

Now I'm trying to add a repo.  Our SVN repo runs under VisualSVN (Pro), with AD authentication.  I can set up the server to join the group, and run svn info from the command line successfully, but I get errors trying to create the repo. Not really sure where to go with it from here; any suggestions?

[Christian Hammond]

Hi Cathy,

Can you show me what errors you're seeing, along with the configuration set for the repository? Also, is there anything in the Review Board log files?

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

On Thu, Jun 23, 2016 at 5:06 PM, Cathy Mullican <cmul...@gmail.com> wrote:

I have RB 2.5.6.1 set up and working on Ubuntu 14.04, with AD authentication -- I can log in to RB as admin or as my domain user.

Now I'm trying to add a repo.  Our SVN repo runs under VisualSVN (Pro), with AD authentication.  I can set up the server to join the group, and run svn info from the command line successfully, but I get errors trying to create the repo. Not really sure where to go with it from here; any suggestions?

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cathy Mullican]

There's not much to show -- see the attached screenshot. That is the correct URL, as shown by svn info, and googling tells me that's the message I'd see with an authentication error.

There are no files in /var/www/rb.revshare.int/logs/ ; is there another location I should be checking?  

I've used RB before, but this is the first time I've set it up.

[Christian Hammond]

Hi Cathy,

You may need to enable logging in Admin UI -> Logging Settings.

For the SVN repository path, you'll need to point it to the root of the SVN repository, rather than a subdirectory within it. Basically, the "Repository Root" value from "svn info".

See if that fixes it. If not, I'll help with some additional commands you can try on the server to better diagnose this.

One last thing: Make sure to enter your credentials in the Username/Password fields again once you've hit an error like this. The browser has a nasty tendency to overwrite the values you've provided with what's in the password manager. We've worked around this several times in the past, but some browsers (ahem, Chrome) have been working tirelessly to override what webapp developers want in this regard.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

Same error with the repository root path -- I actually started with that. 

I enabled logging, and now have this error message:

2016-06-24 20:29:24,091 - ERROR -  - SVN: Failed to get repository information for https://az-fs1.revshare.int/svn/rad: ra_serf was compiled for serf 1.3.8 but loaded an incompatible 32676.1946284232.32676 library

Which I thought told me what I needed; Ubuntu 14.0.4 ships with serf 1.3.3 -- but I've upgraded to 1.3.8 (built from source, confirmed GSSAPI support included), and I'm still getting the same error. 

I can successfully run svn info on the commend line; the Ubuntu box is joined to the AD domain.

[Christian Hammond]

Hi Cathy,

This might be an incompatibility between libsvn and pysvn. You may need to now recompile pysvn and replace the copies on the filesystem. That or go back to purely system libs for svn, libsvn, pysvn, serf, etc.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

The stock libserf in Ubuntu 14.04 lacks GSSAPI support, which is why I have to build that from source.  ( https://bugs.launchpad.net/ubuntu/+source/serf/+bug/1303167 -- why they never released a fixed version, when the patch is right there in the ticket, I can't say.) Everything except libserf is stock.

Trying to reinstall pysvn with apt tells me it's already up to date; trying to build from source is...not working well, but that's probably going off into the weeds.

[Christian Hammond]

Hi Cathy,

How are you trying to build from source? It is a bit of a pain to do from the upstream source, but perhaps you can rebuild the deb.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

I did

apt-get source python-svn

then started trying to follow the directions in INSTALL.html, but they didn't work at all...although lookin gback now, some of that may have been because it was Friday afternoon and I missed something; I'm trying again now. 

[Cathy Mullican]

OK, it definitely works better when you don't accidentally skip a step!

I can now successfully do svn info from the command line, but I still can't create the repo in RB. The error message in the log is now:

2016-06-27 17:00:00,253 - ERROR -  - SVN: Failed to get repository information for https://az-fs1.revshare.int/svn/rad: Unable to connect to a repository at URL 'https://az-fs1.revshare.int/svn/rad'

Error running context: An error occurred during authentication

No more ra_serf error, so that's progress, at least! But also nothing very informative, at least to my eye.

[Christian Hammond]

Hi Cathy,

Progress is good! I think I'll need more info on your setup at this point though.

Can you tell me more about how authentication works on your Subversion setup? From the client's end, is it a standard username/password, or is more involved?

What does the server setup look like?

The error message shown there ("Error running context: An error occurred during authentication") is coming from Subversion itself.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

The server is VisualSVN (paid edition), with Windows AD authentication.  The server where RB is running is joined to the domain, and I can authenticate from the command line. Most users connect via TortoiseSVN on their Windows systems; I also have one other Ubuntu box configured so that i can join the domain, authenticate with kinit, and run svn from the command line.

[Christian Hammond]

Okay. So it's probably one of two things:

1) Something is still messed up somewhere with the recompilation. I don't know what, and can't really debug that from here.

2) The standard way of authenticating that we do doesn't support your setup.

It could easily be #2. We must be able to authenticate to the Subversion server using a username and password (or anonymously). If this is going through some alternative method for authentication, then it may require additional support in Review Board.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

Since it is working on the command line at this point, my money would be on #2 rather than #1.

http://serverfault.com/questions/183231/how-to-configure-review-board-running-under-linux-to-use-a-ldap-user

is the most relevant-seeming info I've found so far, but enough has changed in the 5+ years since it was posted that applying the info there is not entirely straightforward. (Recreating a .subversion/auth tree is relatively straightforward; figuring out the LDAP auth configuration, less so.)

[Cathy Mullican]

It looks like RB isn't using (doesn't support?) kerberos authentication, and that seems to be necessary for the set up we have. 

Sadly, this may mean we can't use RB at this time. :(

[Christian Hammond]

Hi Cathy,

Would you or someone on your end who has a familiarity with Python and Kerberos be willing to work with us on adding support? Review Board is open source, and I'd be willing to take a patch and assist with any work toward it.

Christian

--

[Stephen Gallagher]

I don't have time to work on such a patch directly, but I'd be happy to lend my Kerberos experience towards reviewing any such patch. I think that would be a very useful feature.

I'd recommend working on full SPNEGO support rather than a Kerberos-specific solution. Take a look at the python-gssapi package; it should do most of what is needed.

[Cathy Mullican]

Sadly, I don't think anyone here has the necessary expertise -- my primary language is Perl, not Python, and I'm pretty new to kerberos/AD/LDAP, too -- but I'll ask around my network and see if anyone I know might be up for it.

[Cathy Mullican]

Asking around my network got a few potentially-useful resources, but at this point, I think the change actually needs to happen in the SVN library. 

Since the person who wrote SubvertPy also wrote PyKerberos, I wonder if it might work better than PySVN, and I'm not actually sure why I had it using PySVN instead of SubvertPy anyway,  but I'm finding the SubvertPy documentation pretty opaque so far. Maybe someone in the subvertpy IRC channel can help...

[Christian Hammond]

Hi Cathy,

SubvertPy support was added as an alternative to PySVN, given how difficult PySVN is to install. However, we've found it to be more buggy in other ways (many random problems have been solved by moving from SubvertPy to PySVN). That said, if it works for you, and there's any ability to do PyKerberos integration, great! :)

I admit I have very little knowledge of how Kerberos and SVN work together (or really much at all involving Kerberos). If I knew more, I might be able to recommend a course of action.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

It wasn't entirely clear to me from the documentation which library was preferred, and neither seems to have a terribly active support community.  I've switched to subvertpy for the moment, but switching back is easily done if it doesn't work out.

https://pythonhosted.org/python-gssapi/examples.html#real-world-use is one of the more useful-seeming resources I've been pointed at, but I can't quite see where that logic needs to happen.

https://pypi.python.org/pypi/kerberos is seriously lacking in documentation; I can't tell if it's useful or not.

[Cathy Mullican]

I followed the directions at https://www.reviewboard.org/docs/codebase/dev/getting-started/ to set up a dev environment.  I made a change based on some info from #subvertpy, though I'm not particularly confident in it, but I'd like to test it.  I tried to use the upgrade directions to upgrade my current site, since it's basically a test site at this point anyway. The first step (easy_install -U) seemed to work, but when I tried to upgrade the site with rb_site upgrade (using the rb_site from my dev tree), it fails; the long chain of errors ends with 

 

django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: No module named MySQLdb

Now, it was working with MySQL just fine previously, so I'm not quite sure what's wrong here.  I've tried a couple of solutions found online, but no luck so far.

[Christian Hammond]

Hi Cathy,

Sounds like the dev environment doesn't have the appropriate module for it.

I'd suggest, rather than trying to use MySQL with your current database, that you set up a completely separate SQLite-based database to test with. Having two different environments that point to the same database will cause problems, and may break your main server.

SQLite support is built-in, and while it's not at all recommended for production use, you'll have a better time with it for development.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

Well, I was hoping to have one site with one DB, and just test the modified code with my existing RB site, which is configured with mysql, but I can try again tomorrow with a second site.  

I tried installing the python-mysql module from within the dev environment, but I still get the same error.  Getting kind of frustrated with this, though the payoff if I can get it working will be nice.

[Christian Hammond]

Hi Cathy,

Can you show me how you tried installing the package, and how the dev environment is set up?

As for the site and DB, Review Board expects to know where on the file system its associated site lives (in order to access local data, media files, logs, configuration, etc.), and if you set up a second site in a different environment but with the same exact DB, Review Board will go "Huh, I've been moved" and try to update some settings. This will break your production install in ways.

That, along with annoyances around the MySQL Python module, are why I'd recommend a separate SQLite setup.

Christian

[Cathy Mullican]

As I said, I followed the directions to set up a virtual environment:

pip install virtualenv
virtualenv ~/envs/rb_krb
source ~/envs/rb_krb/bin/activate

install any missing dependencies (just gettext, IIRC, but it's been a few days now, and I don't see it in my command history)

git clone git://github.com/djblets/djblets.git

cd djblets

python setup.py develop

pip install -r dev-requirements.txt

git clone git://github.com/reviewboard/reviewboard.git

cd reviewboard

python setup.py develop

python ./contrib/internal/prepare-dev.py

git clone git://github.com/reviewboard/rbtools.git

python setup.py develop

python setup.py install

After making the change I'm trying to test, I tried to upgrade RB from within the rb_krb environment with

easy_install -U ReviewBoard 

which did not give an error, then tried to upgrade the existing site with

 /root/envs/rb_krb/bin/rb-site upgrade /var/www/rb.revshare.int

and saw the error for the first time.  I then, after googling, ran 

pip install mysql-python

within the virtual environment

but it fails with

 EnvironmentError: mysql_config not found

There does not, in fact, seem to be a mysql_config file anywhere on the system, but the site is working with mysql just fine before I started trying to dive into the code. :)

I also tried 

apt-get install python-mysqldb

which tells me python-mysqldb is already the newest version.

While I would like this to be a production system, unless I can get the repo added, it's useless, so it's functionally a test / dev system for now.

[Christian Hammond]

For the mysql_config issue, what you'll need is the development package for MySQL.

Try:

    apt-get build-dep python-mysqldb

This will install all dependencies needed to build that package: Python development headers, MySQL development headers, etc.

Then the pip install should work.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

Woot; that got me fun new errors in the RB error log that I may actually be able to do something about, but probably not until late today or tomorrow (meetings....). But the install and upgrade processes succeeded this time. Progress!

You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/iUIsa-VP5-0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.

[Cathy Mullican]

Well, I got through the first few errors (updated some symlinks and permissions), but it's still giving me a 500 when I try to load the site. Nothing in the RB log file except " - INFO -  - Reloading logging settings"; nothing in the apache error or access logs, either., Not sure where to go from here.

[Christian Hammond]

Hi Cathy,

This is against the main MySQL database? If so, it might be because of the mixed installs (production servers are meant to run off of a Review Board site directory, which must be consistent in settings across all servers using the DB, and installed packages rather than development trees).

For Review Board development, you really do want to use SQLite and the built-in development server. I know you're trying to keep things the same as the production server, but it's not designed to be run against two very different sorts of installs.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

I'll try it, but there's only one site and one database, on one server, so I don't really see how consistency is an issue here.  This was supposed to be a nice simple tool to introduce code reviews for the team, and instead, it's become something of a nightmare.  The only thing is, I don't see any other tools that look like they'd work better, so I'm continuing to try to make this work; I really would like to add a more formal review step to the process.

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.

[Cathy Mullican]

I'll note that it would be helpful if there were any documentation about setting RB up with sqlite, beyond "Typically on development setups, SQLite is used for the database, as this allows for quick and easy database creation, backups, multiple versions, and deletions."

Running  ./contrib/internal/devserver.py informs me I need a settings file and points me at the template, which tells me I have to give an absolute path if I'm using SQLite...but nothing about how I'm supposed to set up that file, or whether it will be created automatically, or what. 

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.

[Christian Hammond]

I'm sorry it's proving to be a complicated tool for your usage... That's definitely not our goal, though this is the first time we've had a request to support Kerberos with Subversion. Just not a common setup. Most use SSH keys or username/password for authentication, and have no problem getting set up with that.

Production servers shouldn't use SQLite, just development servers. So you'd essentially be standing up a separate database/instance for the development, and once there's support working there, the patched build can be deployed to production.

The reason you wouldn't share the same database between the two is that the production database is expecting to be a production setup, and not a development environment setup. The two are not readily compatible for a few important reasons. (You can make it work, but it requires some stuff that I don't want to put you through.)

So, maybe it's not worth you getting a full-on dev environment going for now. Since this is proving to be problematic, and what you really want is to be testing against your production server (which I gather nobody else is really using), I think what you should do is keep the MySQL setup but edit the files for the installed Python package for Review Board.

If you were to continue with the development environment, you'd want to run ./contrib/internal/prepare-dev.py to generate the settings_local.py file. (See https://www.reviewboard.org/docs/codebase/dev/getting-started/#review-board). The file would set up the SQLite database and other information as well.

You would also need to be working off of the release-2.5.x branch for Review Board, and release-0.9.x for Djblets.

I know this is all getting more complicated than you like at this point, which is why I think it might be worth stepping back and editing the production files to get started, if the production server's not being used by people yet.

I've been trying to research what's needed for Kerberos support, and nothing's been all that helpful. A handful of people have discussed Subversion and Kerberos without talking about how it's integrated or how you authenticate against it. If I could find something informative about this, I'd be happy to help get a patch going for your production setup.

Just to check, is there any way you can have a standard username/password for your Subversion server that Review Board can talk to instead? Or is Kerberos a strict requirement?

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---

You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/iUIsa-VP5-0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Christian Hammond]

Hey Stephen, Cathy,

Trying to educate myself on this... I'd love to get some sanity-checking and additional details on how this works.

I dug through the Subversion and serf source code. It looks like libsvn handles doing SPNEGO on our behalf, so long as the user has run kinit at some point. What I don't know (still investigating the code, but it's 4AM here so I'm done for today) is whether we can use a keytab, or really how that works (still trying to learn this stuff).

I *think* libsvn/serf will do the right thing in this case. From what I read, it should be possible for a keytab to be uploaded to the Review Board server and, somehow, establish a session that clients can use to request tickets. I need to figure this out, and would appreciate insights here.

I think you also need to explicitly tell Subversion to use serf, but this might be outdated information. If so, you'd want to edit the Subversion config in /path/to/rbsitedir/data/.subversion/servers to have:

    [global]

    http-library = serf

What I don't know yet is how kinit/keytabs relate to the user. Does this have to be run within the same login session as the apps needing tickets? I'm wondering if there's a way to have, say, an Apache crontab refresh a session periodically, so that svn can continue to get the right tickets when communicating.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

So here's what I know:

Our SVN server is running VisualSVN Enterprise Edition rather than svnserve

Integrated Windows Authentication is a feature of that product, and currently, if it's enabled, it's the only authentication method allowed.

I've run kinit on the server where I'm trying to install and configure RB, and I can successfully use the command-line svn client.

The guy that does subvertpy is quite a few timezones away from me, so conversation is slow going, but here's the exchange:

[01 Jul 16 00:43] * jelmer: cmullican: subvertpy does have authentication support

[01 Jul 16 00:43] * jelmer: but it needs to be explicitly enabled

[01 Jul 16 10:50] * cmullican: Great!  How do I do that?  I'm finding the documentation a little opaque.

[03 Jul 16 05:24] * jelmer: cmullican: see examples/ra_commit.py

I didn't find the example terribly enlightening, but was trying to do something based on it to see if I could figure out what was going on.

My background is mostly Perl / *nix / SQL, so I'm pretty new to this mixed *nix / Windows / LDAP / krb stuff myself.  

I also have an email in to VisualSVN to see if they know of any tools that support their authentication, with the secondary hope that they might be inspired to allow a service user to authenticate through a different method even when using Windows auth for everyone else.

The config file in /path/to/rbsitedir/data/.subversion/servers is identical to the one in ~/.subversion/servers, which is working.

[Stephen Gallagher]

I'm still away from my computer, so I can't give a lot of specifics, so here is some general help:

A keytab is basically just a Kerberos principal's password in a file format (for all intents and purposes it is plaintext).

Any user can run:
kinit -k -t /path/to/keytab principal@REALM
and get a ticket for that principal.

Normally this would be run as the user in the login session for the service (ideally the service would use libkrb5 directly instead of the kinit CLI). However, you *can* hack around it by setting the env var KRB5CCNAME to something like "FILE:/tmp/krv5cc_UID" and then set that same env var in the service session. So you could run a cron job to keep the ticket in that credential cache updated by running kinit against the keytab.

[Cathy Mullican]

Well, after all that fuss...it turns out that you CAN combine Integrated Windows Authentication and Basic Windows Authentication with VisualSVN Server, it's just that the user still has to be in your Active Directory. (Docs were unclear, and I'm not admin on the SVN server, so there was a minor grapevine issue, too.) 

So, after some clean-up from my attempt at setting up a dev environment, it's working.

[Christian Hammond]

Hi Cathy,

Glad that's been solved! Been working on figuring out the Kerberos stuff, but I haven't gotten as far as I'd like. The way you have it now is probably the ideal way, for performance and maintenance reasons.

Would you be able to share what you did to set this up?

Thanks,

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Cathy Mullican]

Just had to add
http-auth-types=basic
to the [global] section of the .subversion/servers file and enable basic auth in addition to the integrated Windows auth on the server.

Our initial confusion was because you can't have a non-AD user in addition to the Windows users, and because I wasn't seeing the VisualSVN UI myself, didn't realize that we were looking at two slightly different things.

It would be nice from a security perspective if the kerberos auth worked, but it's not urgent.

[Cathy Mullican]

One last related question -- is there a way for RB to automatically import a group of AD users so I can configure the accounts and maybe do some more testing, or do I need to have each person go to the site and log in in order for the account to be created?

[Christian Hammond]

For now, they'll need to log in. We're working on AD/LDAP user/group sync as part of our Power Pack product.

Christian