SVN Property Value Size Limit

[Matthias Legeler]

In our subversion repositories we have merged many many times.

Now we have very large svn:mergeinfo property values (>8k).

The problem is now, that if we make a svn checkout, the property values will be truncated in the working copy by approximately 8k.

The svn propget command at the working copy folder returns a truncated value.

This occurs only for working copies there are created with the http protocol over the svn apache.

With the file protocol the property values are complete.

 

My question:

Is this a known behavior?

Is this limited by a setting in the Apache/WebDAV/SVN configuration?

 

regards,

Matthias

 

[Johan Corveleyn]

What client version? What server version?

--
Johan

[Johan Corveleyn]

[ Please use reply all to keep the list in the loop. Also, please
don't top-post, i.e. put your reply inline or below the thing you're
replying to. More below ... ]

On Thu, Jun 27, 2013 at 10:01 AM, Matthias Legeler
<M.Le...@intershop.de> wrote:
> Server 1.6.9
> Apache 2.2.11
> Client >=1.6.9

First thing to try is to see if this issue is still present when using
a more recent client. Can you try the latest from the 1.7.x series, or
the brand new 1.8.0?

I have never heard of this issue (I always thought there was no limit
on the size of a property value). So I'd first try to see if this
issue is still present with the most recent releases.

Note also that the 1.6 series is actually not suppored anymore (since
the release of 1.8 last week) [1]. But that doesn't mean we won't try
to help you of course.

[1] http://subversion.apache.org/docs/release-notes/#supported-versions

--
Johan

[Philip Martin]

No there is no limit. Run the propget on the http URL rather than the
working copy:

svn propget svn:mergeinfo URL

Does that return the full value?

--
Philip Martin | Subversion Committer
WANdisco | Non-Stop Data
www.wandisco.com

[Matthias Legeler]

yes, this returns the full correct value.
The svn checkout via http makes the problem, tested with server 1.6.9 and client 1.6.x and 1.8.0.

regards
Matthias

-----Original Message-----
From: Philip Martin [mailto:philip...@wandisco.com]
Sent: Donnerstag, 27. Juni 2013 12:00
To: Matthias Legeler
Cc: 'us...@subversion.apache.org'
Subject: Re: SVN Property Value Size Limit

[Daniel Shahaf]

Philip Martin wrote on Thu, Jun 27, 2013 at 11:00:23 +0100:
> Matthias Legeler <M.Le...@intershop.de> writes:
>
> > In our subversion repositories we have merged many many times.
> > Now we have very large svn:mergeinfo property values (>8k).
> > The problem is now, that if we make a svn checkout, the property values will be truncated in the working copy by approximately 8k.
> > The svn propget command at the working copy folder returns a truncated value.
> > This occurs only for working copies there are created with the http protocol over the svn apache.
> > With the file protocol the property values are complete.
> >
> > My question:
> > Is this a known behavior?
> > Is this limited by a setting in the Apache/WebDAV/SVN configuration?
>
> No there is no limit. Run the propget on the http URL rather than the
> working copy:
>

Actually, there is a bug in 1.8.0 FSFS servers whereby it's not possible
to retrieve properties larger than 16MB. The fix will be included in
1.8.1:

http://svn.apache.org/r1494437 (r1491770 on trunk)

The OP doesn't use a 1.8.0 server so that appears to be unrelated to his
problem (but mentioning this anyway for the archives).

Daniel

[Daniel Shahaf]

Matthias Legeler wrote on Thu, Jun 27, 2013 at 07:02:50 +0000:
> In our subversion repositories we have merged many many times.
> Now we have very large svn:mergeinfo property values (>8k).
> The problem is now, that if we make a svn checkout, the property values will be truncated in the working copy by approximately 8k.
> The svn propget command at the working copy folder returns a truncated value.

Where does the truncation happen? Do you get only the first 8KB of the
property? Do you get everything except the last 8KB? What is the
offset from the point of truncation to the start and to the (true) end
of the property? (Is that offset a power of 2?)

You can determine all that with 'svn propget --strict svn:mergeinfo ./'
and 'svn propget --strict svn:mergeinfo URL'.

Daniel

[Matthias Legeler]

'svn propget --strict svn:mergeinfo ./ ' gets the first 7895 characters
'svn propget --strict svn:mergeinfo URL' gets all 7959 characters

yes, 64 characters difference

Matthias

-----Original Message-----
From: Daniel Shahaf [mailto:dani...@elego.de]
Sent: Donnerstag, 27. Juni 2013 12:18
To: Matthias Legeler
Cc: 'us...@subversion.apache.org'
Subject: Re: SVN Property Value Size Limit

[Daniel Shahaf]

Matthias Legeler wrote on Thu, Jun 27, 2013 at 10:30:32 +0000:
> 'svn propget --strict svn:mergeinfo ./ ' gets the first 7895 characters
> 'svn propget --strict svn:mergeinfo URL' gets all 7959 characters
>
> yes, 64 characters difference
>

Interesting, thanks.

I guess the next step is to look at the response headers. (You can use
neon-debug-mask if you use neon, or wireshark/tcpdump if you don't use
SSL.) In particular, whether the response includes the whole property,
and whether metadata (eg: Content-Length response header) matches the
response.

BTW: does it happen under both serf and neon? (check 'svn --version' to
see which you have; use --config-option to switch the active library)

Daniel

[Philip Martin]

'Daniel Shahaf' <dani...@elego.de> writes:

> Matthias Legeler wrote on Thu, Jun 27, 2013 at 10:30:32 +0000:
>> 'svn propget --strict svn:mergeinfo ./ ' gets the first 7895 characters
>> 'svn propget --strict svn:mergeinfo URL' gets all 7959 characters
>>
>> yes, 64 characters difference

The Subversion project has svn:mergeinfo of about that size: 5915 bytes
on trunk, 7892 bytes on 1.8.x, 13838 bytes on 1.7.x.

> Interesting, thanks.
>
> I guess the next step is to look at the response headers. (You can use
> neon-debug-mask if you use neon, or wireshark/tcpdump if you don't use
> SSL.) In particular, whether the response includes the whole property,
> and whether metadata (eg: Content-Length response header) matches the
> response.

http://subversion.apache.org/docs/community-guide/debugging.html#net-trace

I'd suggest tracing the traffic for an empty checkout:

svn co --depth empty URL

That will reduce the traffic but still include the property. The
property value is tranferred as XML in the body of the response for the
final REPORT request.

[Klaus Mueller]

Hi,

I prepared a trace of the relevant part.

Compared transfer debug output for an element.

Both version generated with neon-debug-mask=130. Subversion version is "Version 1.6.12" on a Debian 6 system.


1.Checkout of element
---------------------

This is the version with the problem.

Call: svn co --depth empty https://URL core --username ...

Debug output shortend:
----
[status-line] < HTTP/1.1 200 OK^M
[hdr] Date: Fri, 28 Jun 2013 09:00:15 GMT^M
Header Name: [date], Value: [Fri, 28 Jun 2013 09:00:15 GMT]
[hdr] Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8k DAV/2 SVN/1.6.9 mod_perl/2.0.4 Perl/v5.10.0^M
Header Name: [server], Value: [Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8k DAV/2 SVN/1.6.9 mod_perl/2.0.4 Perl/v5.10.0]
[hdr] Transfer-Encoding: chunked^M
Header Name: [transfer-encoding], Value: [chunked]
[hdr] Content-Type: text/xml; charset="utf-8"^M
Header Name: [content-type], Value: [text/xml; charset="utf-8"]
[hdr] ^M
End of headers.
Running post_headers hooks
[chunk] < 21b7^M
Got chunk size: 8631
Reading 8192 bytes of response body.
Got 727 bytes.
Read block (727 bytes):
[<?xml version="1.0" encoding="utf-8"?>
<S:update-report xmlns:S="svn:" xmlns:V="http://subversion.tigris.org/xmlns/dav/" xmlns:D="DAV:" send-all="true">
<S:target-revision rev="146436"/>
<S:open-directory rev="146436">
<D:checked-in><D:href>/svneng/...</D:href></D:checked-in>
<S:set-prop name="svn:entry:committed-rev">145369</S:set-prop>
<S:set-prop name="svn:entry:committed-date">2013-06-21T09:00:08.455433Z</S:set-prop>
<S:set-prop name="svn:entry:last-author">dpal</S:set-prop>
<S:set-prop name="svn:entry:uuid">9fc60da8-8e6f-4eb0-9785-188b2ab5bd3a</S:set-prop>
<S:set-prop name="svn:ignore">.classpath
</S:set-prop>
<S:set-prop name="svn:mergeinfo">]
Reading 7904 bytes of response body.
Got 7904 bytes.
Read block (7904 bytes):
[/....
...
.../branches/e]
[chunk] < 47^M
Got chunk size: 71
Reading 71 bytes of response body.
Got 71 bytes.
Read block (71 bytes):
[</S:set-prop>
<S:prop></S:prop>
</S:open-directory>
</S:update-report>
]
[chunk] < 0^M
Got chunk size: 0
[hdr] ^M
End of headers.
Running post_send hooks
Request ends, status 200 class 2xx, error line:
200 OK
Running destroy hooks.
Request ends.
sess: Destroying session.
sess: Destroying session.
----

This is the truncated version of the property.

2. PROPGET svn:mergeinfo
------------------------

This is the working version of the requests.

Call: svn propget svn:mergeinfo https://URL --username ...

Debug output shortend:
----
^M
Sending request-line and headers:
Sending request body:
Body block (82 bytes):
[<?xml version="1.0" encoding="utf-8"?><propfind xmlns="DAV:"><allprop/></propfind>]
Request sent; retry is 1.
[status-line] < HTTP/1.1 207 Multi-Status^M
[hdr] Date: Fri, 28 Jun 2013 08:59:08 GMT^M
Header Name: [date], Value: [Fri, 28 Jun 2013 08:59:08 GMT]
[hdr] Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8k DAV/2 SVN/1.6.9 mod_perl/2.0.4 Perl/v5.10.0^M
Header Name: [server], Value: [Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8k DAV/2 SVN/1.6.9 mod_perl/2.0.4 Perl/v5.10.0]
[hdr] Transfer-Encoding: chunked^M
Header Name: [transfer-encoding], Value: [chunked]
[hdr] Content-Type: text/xml; charset="utf-8"^M
Header Name: [content-type], Value: [text/xml; charset="utf-8"]
[hdr] ^M
End of headers.
Running post_headers hooks
[chunk] < 2153^M
Got chunk size: 8531
Reading 8192 bytes of response body.
Got 531 bytes.
Read block (531 bytes):
[<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:ns0="DAV:">
<D:response xmlns:S="http://subversion.tigris.org/xmlns/svn/" xmlns:C="http://subversion.tigris.org/xmlns/custom/" xmlns:V="http://subversion.tigris.org/xmlns/dav/" xmlns:lp1="DAV:" xmlns:lp3="http://subversion.tigris.org/xmlns/dav/" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/svneng/esuite/!svn/bc/146436/.../</D:href>
<D:propstat>
<D:prop>
<S:ignore>.classpath
</S:ignore>
<S:mergeinfo>]
Reading 8000 bytes of response body.
Got 8000 bytes.
Read block (8000 bytes):
[/...
...
...:71506]
[chunk] < 463^M
Got chunk size: 1123
Reading 1123 bytes of response body.
Got 1123 bytes.
Read block (1123 bytes):
[</S:mergeinfo>
<lp1:resourcetype><D:collection/></lp1:resourcetype>
<lp1:getcontenttype>text/html; charset=UTF-8</lp1:getcontenttype>
<lp1:getetag>W/"145369//..."</lp1:getetag>
<lp1:creationdate>2013-06-21T09:00:08.455433Z</lp1:creationdate>
<lp1:getlastmodified>Fri, 21 Jun 2013 09:00:08 GMT</lp1:getlastmodified>
<lp1:checked-in><D:href>/svneng/esuite/!svn/ver/145369/...</D:href></lp1:checked-in>
<lp1:version-controlled-configuration><D:href>/svneng/esuite/!svn/vcc/default</D:href></lp1:version-controlled-configuration>
<lp1:version-name>145369</lp1:version-name>
<lp1:creator-displayname>dpal</lp1:creator-displayname>
<lp1:auto-version>DAV:checkout-checkin</lp1:auto-version>
<lp3:baseline-relative-path>...</lp3:baseline-relative-path>
<lp3:repository-uuid>9fc60da8-8e6f-4eb0-9785-188b2ab5bd3a</lp3:repository-uuid>
<lp3:deadprop-count>2</lp3:deadprop-count>
<D:lockdiscovery/>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
</D:multistatus>
]
[chunk] < 0^M
Got chunk size: 0
[hdr] ^M
End of headers.
Running post_send hooks
Request ends, status 207 class 2xx, error line:
207 Multi-Status
Running destroy hooks.
Request ends.
sess: Destroying session.
sess: Destroying session.
----

It seems the server is truncating the output before send the response from my point of view.

Klaus

> -----Original Message-----
> From: Philip Martin [mailto:philip...@wandisco.com]

> Sent: Friday, June 28, 2013 11:23 AM
> To: 'Daniel Shahaf'
> Cc: Matthias Legeler; us...@subversion.apache.org
> Subject: Re: SVN Property Value Size Limit
>

[Klaus Mueller]

An updated client generates the same output.

User-Agent: SVN/1.7.9 neon/0.29.6

I do not know how to generatet he output with a subversion 1.8.0 client as neon is replaced by serf.

Klaus

[Matthias Legeler]

Checkout with neon: property value truncated

Checkout with serf (svn co --config-option servers:global:http-library=serf URL): property value complete

[Daniel Shahaf]

Use ra_serf then :-)

[Christophe Broeckx]

Hi everybody.

I have similar problem now with ra_serf as well.
Subversion server version 1.5.2 (r32768)
Client version 1.8.8 (r1568071) using ra_serf 1.3.3
The offset between the local and server mergeinfo was 16 characters.

We have found a way to fix the problem. To do so, we need to reduce the size of the mergeinfo.
Here is how :
- ssh to svn server
- find the repository location (in this example, /srv/svn/repos/myrepo)
- do a local checkout of the branch with too long mergeinfo (not through http !)
   svn co file:///srv/svn/repos/myrepo/myproject/branches/2.4
- check the mergeinfo o the problematic file to know where you can simplify the mergeinfo
   e.g. here we had a lot of merge from branch 2.0 for the file path/path/path/Foo.java
   /myproject/branches/2.0/path/path/path/Foo.java:39126-43104,43469,44672,44705,44723,44736,44749,44794,44796,44805,44825,44831,44833,44852,44855,44865,44876,44879,44883,44887,44889,44905,44907,44912-44914,44919,44923,44947,44955,44957,44964,45007-45008,45069,45111,45114,45126,45128-45129,45142,45169,45202,45233,45341,45349,45351,45409,45425,45441,45460,45564,45568,45573,45575,45594-45595,45603,45626,45637,45654,45785,45827-45829,45832,45868,45871,45874,45877,45891,45917,45965,46000,46010,46014,46018,46024,46026,46030-46031,46038,46060,46073,46094,46109,46123,46130,46134,46159,46186,46190,46197,46210,46270,46304,46360,46363,46442-46443,46451,46470,46477-46478,46480-46481,46486-46488,46506,46509,46520,46539,46556,46561,46574,46582,46590,46595,46599,46604,46606,46752-46753,46755,46779-46780,46851-46852,46859,46864,46886,46903-46904,46923,46929,46984-46985,47011,47022,47040,47105,47120,47182,47205,47207-47208,47211,47238,47247,47253,47258,47288,47300,47302,47358,47377,47381,47407,47409-47410,47435,47514,47516,47522,47524,47536,47544,47553,47598,47605,47611,47615,47619,47622,47654,47711,47764,47923,47936,48023,48096,48164,48170-48171,48183,48290,48649,48666-48667,48755,48829,48897,48959,48962-48963,48982,48989,48991,49060,49068,49174,49184-49185,49205,49241,49269,49296,49312,49314,49319,49330,49343,49393,49400,49402,49413,49499,49509,49518-49519,49571,49582,49614,49756,49758,49768,49907-49908,50108,50252,50255,50259,50290,50398,50466,50490-50494,50497-50500,50708,50754,50756,50758-50760,50764-50765,50767-50770,50776,50781,51186,51284,51353,51357,51365,51521,51558,51574,51578,51665,51713,51752,51886,51889-51890,51980,51982-51985,51990,51997,52179,52205,52304,52357,52406,52413,52615,52620-52622,52703,52995,53007,53128,53205,53309-53310,53313,53315,53318-53323,53325,53328-53329,53533,53670,53879,53961,54027-54029,54131,54376,54526-54527,55374,55634,55653,55758,55808,56225,56450,57112,57421,57470,57631,57647,57734,57739,58308,58317,58357,58359,58378,58381,58421,58438-58440
- merge the complete range of the previously identified branch (use file protocol !)
   svn merge -r39126:58440 file:///srv/svn/repos/myrepo/myproject/branches/2.0
- check the merge (should be no other change than property changes) and that the mergeinfo is indeed reduced
   svn status
   svn propget svn:mergeinfo Foo.java
- if you don't have local commit write like here, switch to remote http protocol
   svn switch --relocate file:///srv/svn/repos/myrepo/myproject/branches/2.4 http://svn.server.url/repos/myrepo/myproject/branches/2.4
- commit
   svn commit -m "[MERGE:39126-58440] Reduced mergeinfo (merge range of already merged revisions)"

Now you can just update your remote repository, it's not necessary to do a clean checkout.

Note that even with this method the mergeinfo will one day be too long, because it contains merge info from all old branches too, so it will grow each time a new branch appears and gets merged.
Further cleaning could be done, like removing all closed branches and tags.

Hope this helps,
Christophe