SVNCacheRevProps and other performance tweaks.

[Terry Dooher]

Hi all,

 

I’ve just finished a dump/load cycle on about 500GB worth of repositories to bring in the more efficient changes for FSFS in 1.8 (We’re on 1.8.8 now). I’m looking at utilising some performance tweaks to make the most of the hardware the server is running on. Could someone clarify a couple of the more arcane directives for me, as they’re only tersely or not at all described in the docs, yet:

 

SVNPathAuthz – Currently ‘On’. To my understanding, the value ‘short_circuit’ will prevent any other path-based authorisation  being used for requests. I’m running Apache2.4/mod_dav_svn, with local Basic Auth and local Authz files. No network drives or other auth methods are configured for the server, which is dedicated to svn. Would short_circuit save me any processing time in this scenario or is it only useful for people with LDAP or other auth systems that might slow down request processing?

 

SVNCacheRevProps – Currently ‘Off’.  I’m told that it “should only be enabled under the conditions described in the documentation”, but I can’t find any reference to it in the source or the 1.8 book, yet. What are the conditions in which it should be used and what happens if I use it outside those conditions? Do I risk DB corruption or is it only of benefit in specific scenarios?

 

SVNInMemoryCacheSize – Just want to confirm this one: Is the overall server memory footprint (CacheSize*No. of worker processes) under mod_dav_svn? I’m using the prefork mpm (some minor PHP admin scripts on the box), and typically have 20-80 running child workers active. Bumping this up 256kb or 1MB would have a negligible effect, if this is the case.

 

Thanks,

 

Terry.

 

 

 

 

 

 

 

[Ben Reser]

On 5/14/14, 12:06 PM, Ben Reser wrote:
> Actually I think it's more that you really must have this enabled if you have
> revprop packing turned on. I can't think of a reason you don't want this turn
> off other than desiring to reduce resource usage at the cost of slower performance.

Oops double negative. I meant "I can't think of a reason you don't want to
turn this on..."

[Ben Reser]

On 5/14/14, 10:11 AM, Terry Dooher wrote:
> SVNPathAuthz – Currently ‘On’. To my understanding, the value ‘short_circuit’
> will prevent any other path-based authorisation being used for requests. I’m
> running Apache2.4/mod_dav_svn, with local Basic Auth and local Authz files. No
> network drives or other auth methods are configured for the server, which is
> dedicated to svn. Would short_circuit save me any processing time in this
> scenario or is it only useful for people with LDAP or other auth systems that
> might slow down request processing?

This setting controls the handling of authorization for what I refer to as
secondary paths. I.E. paths that are accessed other than the path that gets
directly accessed as the uri in the request line. A good example of this a
'svn log' request will check authorization for each file modified by a revision
when returning the log data for that revision in order to determine how much
information to return (some or all of the revision properties are omitted based
on this).

on (default): This causes the authorization check to generate a sub-request.
This means that a GET request for each of the secondary paths described above
is generated. This request is re-run through all of the authentication and
authorization processing. Usually this setting (despite being the default) is
not what you want. It is however the safest setting from a security
perspective. Unless you are using a httpd authorization module that also looks
at paths this added overhead does nothing for you. LDAP and other external
authentication makes the performance of this worse, however it's still a
measurable performance delay without such systems.

off: The secondary path setting does not run. It presumes if you have access
to the URI in the request-line you have access to all paths. This is the
setting you want if you are not using mod_authz_svn (i.e. not using path based
authorization).

short_circuit: The secondary path checks do not generate sub-requests but
rather short circuit the httpd authentication/authorization infrastructure and
directly as mod_authz_svn. This has significantly less overhead, while being
as secure as on provided that the only authorization module that is path aware
is mod_authz_svn. If you're using authz unless you have an extremely rare case
you want this setting.

In your scenario you want to use short_circuit.

> SVNCacheRevProps – Currently ‘Off’. I’m told that it “should only be enabled
> under the conditions described in the documentation”, but I can’t find any
> reference to it in the source or the 1.8 book, yet. What are the conditions in
> which it should be used and what happens if I use it outside those conditions?
> Do I risk DB corruption or is it only of benefit in specific scenarios?

Actually I think it's more that you really must have this enabled if you have
revprop packing turned on. I can't think of a reason you don't want this turn
off other than desiring to reduce resource usage at the cost of slower performance.

The documentation in question is the release notes:
http://subversion.apache.org/docs/release-notes/1.8.html#revprop-caching
http://subversion.apache.org/docs/release-notes/1.8.html#revprop-packing

> SVNInMemoryCacheSize – Just want to confirm this one: Is the overall server
> memory footprint (CacheSize*No. of worker processes) under mod_dav_svn? I’m
> using the prefork mpm (some minor PHP admin scripts on the box), and typically
> have 20-80 running child workers active. Bumping this up 256kb or 1MB would
> have a negligible effect, if this is the case.

First of all you probably want to read the 1.7 release notes on performance tuning:
http://subversion.apache.org/docs/release-notes/1.7.html#server-performance-tuning

Yes the amount of memory footprint of the cache (there will still be more
memory used for other uses by the server) will be CacheSize*No. of worker
process. Since the memory caches are per-process we do not recommend using
prefork with Subversion, but rather suggest that you use worker or event MPMs.

With the worker or event MPMs you are able to share the cache between the
threads within the same processes. This allows you to set a larger cache size
since you'll have fewer processes and more of the threads will gain access to
the cache, making it to be more effective.

I'd suggest you run the PHP in a separate instance of httpd or on a separate
machine. It's going to be difficult to tune httpd for Subversion with these
other uses in the same httpd processes.

[Terry Dooher]

From: Ben Reser Sent: 14 May 2014 20:07

> On 5/14/14, 10:11 AM, Terry Dooher wrote:

>> SVNPathAuthz - Would short_circuit save me any processing time in this

>> scenario or is it only useful for people with LDAP or other auth systems that
>> might slow down request processing?
>

> short_circuit: The secondary path checks do not generate sub-requests but
> rather short circuit the httpd authentication/authorization infrastructure
> and directly as mod_authz_svn. This has significantly less overhead, while
> being as secure as on provided that the only authorization module that is
> path aware is mod_authz_svn. If you're using authz unless you have an
> extremely rare case you want this setting.
>
> In your scenario you want to use short_circuit.

That's so much clearer now and sounds like something I can definitely use.
As I understand it, the downside is that the server will reveal path components
and filenames within restricted areas during log operations?

>> SVNCacheRevProps - Currently 'Off'. is it only of benefit in specific scenarios?

> Actually I think it's more that you really must have this enabled if you have
> revprop packing turned on. I can't think of a reason you don't want this
> turn off other than desiring to reduce resource usage at the cost of slower
> performance.
>
> The documentation in question is the release notes:
> http://subversion.apache.org/docs/release-notes/1.8.html#revprop-caching
> http://subversion.apache.org/docs/release-notes/1.8.html#revprop-packing

Makes sense. The dump/load cycle I've just finished was to bring in the
revprop packing changes, but I didn't appreciate the need for caching to
reduce CPU load.

>> SVNInMemoryCacheSize

> With the worker or event MPMs you are able to share the cache between the
> threads within the same processes. This allows you to set a larger cache
> size since you'll have fewer processes and more of the threads will gain
> access to the cache, making it to be more effective.

Ah, of course. I missed the k in the default value and figured that a
16-fold increase would be a small, suitable value for ~100 prefork processes;
I see now that setting it to 256MB per process might incur a slight overhead :)
I can run the PHP under lighttpd, easily, so moving to the worker mpm is
the best option.

Thanks for such a comprehensive response.

Terry.

[Ben Reser]

On 5/15/14, 1:15 AM, Terry Dooher wrote:
> That's so much clearer now and sounds like something I can definitely use.
> As I understand it, the downside is that the server will reveal path components
> and filenames within restricted areas during log operations?

Don't think you quite understand yet. If Subversion is revealing path
compontents/filenames within restricted areas during log then it can be
revealing file contents (by way of the REPORT method and bulk updates).

The short_circuit setting can only break security in this way if and only if
you have some other module than mod_authz_svn that looks at paths to determine
what access to provide. Many modules may have different configurations per
Location, but most don't look at the path like mod_authz_svn does and give
different answers within the same Location for the same request. In fact I'm
not aware of any such module that does this that comes with httpd, so you'd
have to install some 3rd party or custom module to run into this. Meaning for
the vast majority of users if they're using mod_authz_svn they should have
SVNPathAuthz set to short_circuit.

The off setting, should never be used with mod_authz_svn. It's probabbly
possible to configure things such that you only leak file paths, by disabling
bulk updates. But I'm not 100% sure that there isn't an edge case that
wouldn't be missed by that.

For the vast majority of users these rules are true:

Using mod_authz_svn with an AuthzSVNAccessFile set you want SVNPathAuthz set to
short_circuit.
Not using mod_authz_svn, no AuthzSVNAccessFile set, you want SVNPathAuthz set
to off.

The exception is if you use some weird authorization module like I explained above.

[Mark Phippard]

I recall there is a valid scenario for using "off" and that is when you are only using authz to restrict write access to some paths.  If all paths in the repository are readable, then I believe "off" works properly in all scenarios.  Meaning it is not possible to write to any paths that you do not have access to.

--
Thanks

Mark Phippard
http://markphip.blogspot.com/