Issue with mod_wsgi using trac and subversion 1.7

[Adrian Gschwend]

Hi group,

I have an interesting problem on my server with mod_wsgi:

I run trac and subversion in the same configuration, see
http://ktk.netlabs.org/misc/svn.netlabs.conf for my configuration.

The configuration starts with the SVN repositories residing on /repos
and then configures trac with mod_wsgi. So far this used to work just
fine. As you can see on http://svn.netlabs.org I run a lot of TRAC
repositories for several OSS projects.

Last week I upgraded subversion on the server to version 1.7, since
then I can still checkout and commit to svn repositories with clients
which are below version 1.7 (I tested 1.6.x), everything works as
expected. However, any subversion client >= 1.7 cannot commit anymore,
checkout works as before though.

After dumping with tcpdump I could see the following packet (decoded
TCP stream via wireshark):

--
POST /repos/kbuild/!svn/me HTTP/1.1
User-Agent: SVN/1.7.3 neon/0.29.6
Connection: TE
TE: trailers
Host: svn.netlabs.org
Content-Type: application/vnd.svn-skel
DAV: http://subversion.tigris.org/xmlns/dav/svn/depth
DAV: http://subversion.tigris.org/xmlns/dav/svn/mergeinfo
DAV: http://subversion.tigris.org/xmlns/dav/svn/log-revprops
Content-Length: 14
Accept-Encoding: gzip
Authorization: Basic 1lkdfjwel3fldkj==

( create-txn )HTTP/1.1 404 Not Found
Date: Wed, 29 Feb 2012 13:50:42 GMT
Server: Apache
Content-Length: 21
Content-Type: text/plain

Environment not found
--

This is definitely an error message from TRAC in case there is no trac
environment found. So for some reason mod_wsgi hits in at this point
instead of the DAV handler. Again, this works just fine with
subversion 1.6.x.

I really don't get this, I checked my config and tried several things
to fix it but no luck so far. From the subversion changelog I can see
that they changed the way subversion interacts with the server:

http://subversion.apache.org/docs/release-notes/1.7.html#httpv2

--
Subversion 1.7 offers a simpler HTTP protocol variant that can be used
when connecting to supported servers. This simpler protocol (sometimes
referred to as HTTPv2) requires fewer client-server round trips to
establish a connection, making Subversion much more performant on high-
latency network connections.

As mentioned in the compatibility table, Subversion 1.7 will only use
HTTPv2 when connecting with a 1.7 (or greater) server, but will
continue to use existing protocols when communicating with earlier
versions of Subversion. Likewise, older clients will only use the old
protocol when connecting to a Subversion server, regardless of
version.
--

So the only explanation I have is that with 1.7 clients subversion
tries to do HTTPv2 and for some reason mod_wsgi decides to take over
even if it's not its job because the post goes to /repos, which should
be the DAV handler.

Could this be a bug in mod_wsgi?

Software versions (running on FreeBSD 8):

ap22-mod_wsgi-3.3_2
apache-2.2.22_5
py26-subversion-1.7.3
subversion-1.7.3
trac-0.12.3
python26-2.6.7_3

content of trac-netlabs.wsgi:
--
#!/usr/local/bin/python2.6
# -*- coding: utf-8 -*-

import os
def application(environ, start_request):
os.environ['TRAC_ENV_PARENT_DIR'] = '/data/trac/netlabs.org'
os.environ['PYTHON_EGG_CACHE'] = '/tmp/egg-cache'
from trac.web.main import dispatch_request
environ['trac.locale'] = 'sv_SE.UTF-8'
return dispatch_request(environ, start_request)
--

Any hints would be appreciated!

thanks

Adrian

[Graham Dumpleton]

Do you have the ability to control the order of the LoadModule lines
for DAV/SVN and mod_wsgi?

If both modules are configuring a handler of same type, if no explicit
ordering dependency has been defined within Apache modules, which
there will not be in this case, then handlers will be processed based
on order of LoadModule lines in Apache configuration.

Thus, try switching the order of the LoadModule lines around for the modules.

That is all I can think of.

Graham

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>

[Adrian Gschwend]

Hi,

> Do you have the ability to control the order of the LoadModule lines
> for DAV/SVN and mod_wsgi?

yeah it's my box so I can do whatever I want. Tried that, mod_wsgi was
the last module, switched it to be the first. No change.

> That is all I can think of.

ok if there are no other ideas I will try TRAC with fastcgi module to
see if the problem persists.

cu

Adrian

[Adrian Gschwend]

Hi Graham,

> Thus, try switching the order of the LoadModule lines around for the modules.
>
> That is all I can think of.

I finally had the time to switch to mod_fcgid, unfortunately the
problem persists. So this is definitely not a mod_wsgi problem but I'm
even more clueless than before.

Do you have any idea where this could go wrong? Is Apache itself
screwing something up maybe? I don't think this can be TRAC as TRAC
probably does not have an influence of the handlers at this point in
Apache, right?

I'm more than happy to do more tests and report bugs somewhere but I
need some ideas about where to do that :-)

cu

Adrian

[Christian Boos]

Just an idea: you could perhaps rebuild Subversion yourself, with the
same apr/apr-util libraries as the ones used by your Apache server
(using the proper --with-apxs=... setting).

-- Christian

[Adrian Gschwend]

Hi Christian,

> Just an idea: you could perhaps rebuild Subversion yourself, with the
> same apr/apr-util libraries as the ones used by your Apache server
> (using the proper --with-apxs=... setting).

I'm on FreeBSD and I use ports, so everything is built there.
Subversion is using the same APR.

cu

Adrian

[Adrian Gschwend]

Hi group,

Just as a followup, I posted it on the apache list:

http://article.gmane.org/gmane.comp.apache.user/99368

cu

Adrian

[Vishwajeet]

I am facing the same issue with django+apache+mod_wsgi+svn 1.7.
I posted it on svn mailing list and got following reply

"mod_wsgi is probably intercepting the POST request intended for
mod_dav_svn and not passing it on. Subversion 1.7.0 had a similar
problem, it tried to process non-Subversion POST requests and produced
an error. The Subversion problem was fixed in 1.7.2. A code change
in
mod_wsgi is probably required. It may be possible to reorder the
modules in the apache config file."

On Mar 23, 2:14 am, Adrian Gschwend <adrian.gschw...@comuno.org>
wrote:

[Graham Dumpleton]

I thought I already suggested playing with order of LoadModule lines
in Apache configuration, will need to check back in emails. I did
consider that likely order of one of the handlers, but based on
fiddling LoadModule lines not helping, seemed to be something else.

Graham

[vishwajeet singh]

On Tue, Mar 27, 2012 at 1:20 AM, Graham Dumpleton <graham.d...@gmail.com> wrote:

I thought I already suggested playing with order of LoadModule lines
in Apache configuration, will need to check back in emails. I did
consider that likely order of one of the handlers, but based on
fiddling LoadModule lines not helping, seemed to be something else.

Is there a way to exclude some paths from wsgi handler ??

Problem is that instead of going to /svn/ it's getting matched to / which the wsgi handler and this happens only for post requests which I fail to understand, any suggestions to workaround this issue.

--
Vishwajeet Singh
+91-9657702154 | dextr...@gmail.com | http://bootstraptoday.com
Twitter: http://twitter.com/vishwajeets | LinkedIn: http://www.linkedin.com/in/singhvishwajeet

[Graham Dumpleton]

If you are able to modify and recompile mod_wsgi source code in your
setup, try modifying:

static const char * const p1[] = { "mod_alias.c", NULL };

to:

static const char * const p1[] = { "mod_alias.c", "mod_dav_svn.c", NULL };

This will ensure that SVN module gets a look in first.

Graham

[vishwajeet singh]

On Tue, Mar 27, 2012 at 10:20 AM, Graham Dumpleton <graham.d...@gmail.com> wrote:

If you are able to modify and recompile mod_wsgi source code in your
setup, try modifying:

   static const char * const p1[] = { "mod_alias.c", NULL };

to:

   static const char * const p1[] = { "mod_alias.c", "mod_dav_svn.c", NULL };

This will ensure that SVN module gets a look in first.

I tried this but does not seem to work, one thing I have in my configuration that my svn configurations are inside named vhost if that makes a difference.

[Graham Dumpleton]

What are the names of all the DAV modules loaded in Apache?

It may not be mod_dav_svn that needs to be first but one of the core
DAV modules.

So in try mod_dav.c instead. Also maybe mod_dav_fs.

Graham

[Graham Dumpleton]

Did you get a chance to try with mod_dav.c etc instead?

Graham

[vishwajeet singh]

On Sun, Apr 1, 2012 at 3:51 AM, Graham Dumpleton <graham.d...@gmail.com> wrote:

Did you get a chance to try with mod_dav.c etc instead?

Yes I tried with both but it still did not worked, for time being I have moved svn & wsgi in different vhosts.

[Adrian Gschwend]

> > Did you get a chance to try with mod_dav.c etc instead?
>
> Yes I tried with both but it still did not worked, for time being I have
> moved svn & wsgi in different vhosts.

That was my "Plan B" as well but I would like to avoid doing that.

Graham do you have any other ideas where this could go wrong or how we
can debug it?

cu

Adrian

[Graham Dumpleton]

A prior post said:

"""
I am facing the same issue with django+apache+mod_wsgi+svn 1.7.
I posted it on svn mailing list and got following reply

"mod_wsgi is probably intercepting the POST request intended for
mod_dav_svn and not passing it on. Subversion 1.7.0 had a similar
problem, it tried to process non-Subversion POST requests and produced
an error. The Subversion problem was fixed in 1.7.2. A code change
in
mod_wsgi is probably required. It may be possible to reorder the
modules in the apache config file."
"""

If someone can find out what the change in that other code was that would help.

I suggested what would be need to do reordering but been told it doesn't work.

I would try and track down what the change in subversion 1.7.2 was,
and today I do have a very rare day to do some mod_wsgi stuff, but
time is a bit short as want to back port Apache 2.4 changes to
mod_wsgi 3.X.

If you happen to have a few minutes to look at subversion and work it
out that would be great.

What you will be looking for is changes in the equivalent of:

static void wsgi_register_hooks(apr_pool_t *p)
{

static const char * const p1[] = { "mod_alias.c", NULL };

static const char * const n1[]= { "mod_userdir.c",
"mod_vhost_alias.c", NULL };

static const char * const n2[] = { "core.c", NULL };

#if !defined(MOD_WSGI_WITH_AUTHN_PROVIDER)
static const char * const p3[] = { "mod_auth.c", NULL };
#endif
#if !defined(MOD_WSGI_WITH_AUTHZ_PROVIDER)
static const char * const n4[] = { "mod_authz_user.c", NULL };
#endif
static const char * const n5[] = { "mod_authz_host.c", NULL };

static const char * const p6[] = { "mod_python.c", NULL };

static const char * const p7[] = { "mod_ssl.c", NULL };

ap_hook_post_config(wsgi_hook_init, p6, NULL, APR_HOOK_MIDDLE);
ap_hook_child_init(wsgi_hook_child_init, p6, NULL, APR_HOOK_MIDDLE);

ap_hook_translate_name(wsgi_hook_intercept, p1, n1, APR_HOOK_MIDDLE);
ap_hook_handler(wsgi_hook_handler, NULL, NULL, APR_HOOK_MIDDLE);

#if defined(MOD_WSGI_WITH_DAEMONS)
ap_hook_post_config(wsgi_hook_logio, NULL, n2, APR_HOOK_REALLY_FIRST);

wsgi_header_filter_handle =
ap_register_output_filter("WSGI_HEADER", wsgi_header_filter,
NULL, AP_FTYPE_PROTOCOL);
#endif

#if !defined(MOD_WSGI_WITH_AUTHN_PROVIDER)
ap_hook_check_user_id(wsgi_hook_check_user_id, p3, NULL, APR_HOOK_MIDDLE);
#else
ap_register_provider(p, AUTHN_PROVIDER_GROUP, "wsgi",
AUTHN_PROVIDER_VERSION, &wsgi_authn_provider);
#endif
#if !defined(MOD_WSGI_WITH_AUTHZ_PROVIDER)
ap_hook_auth_checker(wsgi_hook_auth_checker, NULL, n4, APR_HOOK_MIDDLE);
#else
ap_register_provider(p, AUTHZ_PROVIDER_GROUP, "wsgi-group",
AUTHZ_PROVIDER_VERSION, &wsgi_authz_provider);
#endif
ap_hook_access_checker(wsgi_hook_access_checker, p7, n5, APR_HOOK_MIDDLE);
}

This is what sets up Apache handlers and order.

I will try and look later, but if someone can get to it first, even better.

Graham