svnsync on large files

[Stümpfig, Thomas]

Hi everybody,

i get an error with svnsync for large files. svnsync: E175002: REPORT request on

 

The rev to be synced is 2,291,973,385 Bytes large (Size in txn-protorevs directory. Of target mirror)

The sync is with svn 1.9.5 (Visual SVN Server) over both https

 

I tried to rdump, svnadmin load incremental (works) and adjust revprop info in rev 0 to the new imported files. But the I got a path already exists for following syncs.
Might not be related to my manipulations but could be the case.

 

Any Help is appreciated

 

Mit freundlichen Grüßen

Thomas Stümpfig

 

Siemens Indusry Software GmbH

Product Lifecycle Management

Global Sales & Services

 

Franz-Geuer Straße 10

50823 Köln

Franz-Geuer-Straße 10
50823 Köln Tel.: +49 (2153 )9107117

Fax.: +49 (221) 20802699

Mobil: +49 (175) 2205 712

thomas.s...@siemens.com            

www.siemens.com/plm

 

-----------------
Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564

[Pavel Lyalyakin]

Hello Thomas,

On Wed, Dec 7, 2016 at 8:54 PM, Stümpfig, Thomas
<thomas.s...@siemens.com> wrote:
>
> Hi everybody,
>
> i get an error with svnsync for large files. svnsync: E175002: REPORT request on

It seems to be only the first part of the error message. What's the
full wording of the error?

>
> The rev to be synced is 2,291,973,385 Bytes large (Size in txn-protorevs directory. Of target mirror)
>
> The sync is with svn 1.9.5 (Visual SVN Server) over both https

You should use file:/// access to the local repository instead of
HTTP(S). Here is an example:
[[[
svnsync sync "https://svn.example.com/svn/MyRepo"
"file:///C:/Repositories/MyRepo/"
]]]

Does the error occur when you use file:/// access to local repository?

> I tried to rdump, svnadmin load incremental (works) and adjust revprop info in rev 0 to the new imported files. But the I got a path already exists for following syncs.
> Might not be related to my manipulations but could be the case.
>
>
>
> Any Help is appreciated
>
>
>
> Mit freundlichen Grüßen
>
> Thomas Stümpfig
>
>
>
> Siemens Indusry Software GmbH
>
> Product Lifecycle Management
>
> Global Sales & Services
>
>
>
> Franz-Geuer Straße 10
>
> 50823 Köln
>
> Franz-Geuer-Straße 10
> 50823 Köln Tel.: +49 (2153 )9107117
>
> Fax.: +49 (221) 20802699
>
> Mobil: +49 (175) 2205 712
>
> thomas.s...@siemens.com
>
> www.siemens.com/plm
>
>
>
> -----------------
> Siemens Industry Software GmbH; Anschrift: Franz-Geuer-Str. 10, 50823 Köln; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 84564

--
With best regards,
Pavel Lyalyakin
VisualSVN Team

[Daniel Shahaf]

Pavel Lyalyakin wrote on Wed, Dec 07, 2016 at 20:59:24 +0300:
> On Wed, Dec 7, 2016 at 8:54 PM, Stümpfig, Thomas
> <thomas.s...@siemens.com> wrote:
> >
> > The rev to be synced is 2,291,973,385 Bytes large (Size in txn-protorevs directory. Of target mirror)
> >
> > The sync is with svn 1.9.5 (Visual SVN Server) over both https
>
>
> You should use file:/// access to the local repository instead of
> HTTP(S).

Why? There are perfectly good reasons to use http://localhost/, for
example, to have a one-stop-shop killswitch for disabling "everything"
from accessing the repository.

I'm guessing the dump/incremental-load/update-r0-revprops surgery went
wrong. I would (a) double-check that the revision that had been
manually synced did in fact sync correctly, (b) review the r0 revprops.
(Thomas: you can just post the 'proplist -v' here if you aren't sure
whether it's correct now.)

Also, as Pavel says, we need the full error message.

Cheers,

Daniel

[Stümpfig, Thomas]

Hi guys
Thank you for your support.
Here is the full error message.

E:\svnbackup>svnsync sync --source-username=myusername --source-password=xxxxx --sync-username= myusername --sync-password=xxxxx https://mydesturl/repository https://mysourceturl/repository
Transmitting file data .....svnsync: E175002: REPORT request on '/svn/projektablage
age/!svn/rev/54618' failed

(I modified the urls for infosec puposes ... )

E:\svnbackup>svn proplist --revprop -r 0 file:///E:/svnmirror/projektablage
Unversioned properties on revision 0:
svn:date
svn:sync-currently-copying
svn:sync-from-url
svn:sync-from-uuid
svn:sync-last-merged-rev

E:\svnbackup>svn propget --revprop -r 0 svn:sync-currently-copying file:///E:/svnmirror/projektablage
54618

E:\svnbackup>svn propget --revprop -r 0 svn:sync-last-merged-rev file:///E:/svnmirror/projektablage
54617

E:\svnbackup>svn info -r HEAD file:///E:/svnmirror/projektablage
Path: projektablage
URL: file:///E:/svnmirror/projektablage
Relative URL: ^/
Repository Root: file:///E:/svnmirror/projektablage
Repository UUID: e75c21af-ffb9-1a4d-8d4e-43c49f6e0dd9
Revision: 54617
Node Kind: directory
Last Changed Author: xxxxxxx
Last Changed Rev: 54617
Last Changed Date: 2016-12-05 11:38:18 +0100 (Mon, 05 Dec 2016)


This is the situation of the starting point. (I restored)

Regards
Thomas

[Stümpfig, Thomas]

Would you suggest the following command
svnsync initialize --allow-non-empty http://svn.example.com/svn-mirror \
NEW-SOURCE-URL

?

-----Original Message-----
From: Pavel Lyalyakin [mailto:pavel.l...@visualsvn.com]
Sent: Mittwoch, 7. Dezember 2016 18:59
To: Stümpfig, Thomas (DF PL S&SE DE PSM EAI) <thomas.s...@siemens.com>; us...@subversion.apache.org
Subject: Re: svnsync on large files

[Daniel Shahaf]

Stümpfig, Thomas wrote on Wed, Dec 07, 2016 at 23:32:38 +0000:
> This is the situation of the starting point. (I restored)

The steps should be this:

f() {
token="$USER@$(hostname):${RANDOM}:${RANDOM}"
svn propset --revprop -r0 -- svn:sync-lock $token
svn propget --revprop -r0 --strict svn:sync-lock | fgrep -q -- $token || return 1
svn propset --revprop -r0 svn:sync-currently-copying 54618
svnrdump dump -r54618 --incremental https://mysourceturl/repository >tmpfile
svnadmin load /svnmirror/projektablage <tmpfile
svn propset --revprop -r0 svn:sync-last-merged-rev 54618
svn propdel --revprop -r0 svn:sync-currently-copying
svn propdel --revprop -r0 svn:sync-lock
}

Note:

1. Using the same metadata revprops and order as svnsync.

2. The fgrep is to make up for the svn:sync-lock not being taken
atomically. (The cmdline client passes NULL for the ORIGINAL_PROPVAL
argument of svn_client_revprop_set2().)

3. Using a tmpfile since the dumpfile format isn't truncation-safe.

4. $RANDOM isn't in POSIX; if not available, substitute the first
N bytes from /dev/urandom.

5. Make sure to use the correct source URL on the dump command.

6. The hooks may need tweaking.

Cheers,

Daniel

[Stümpfig, Thomas]

Hi Daniel,
you are a guru. I don't know why but it worked!

I guess the magic is in the difference of network transmission of svnsync vs svnrdump / svnadmin load

Are there suggestions from your side to avoid this error in the future?

Thank you very much for your help!!!

Regards
Thomas

-----Original Message-----
From: Daniel Shahaf [mailto:dani...@apache.org]
Sent: Donnerstag, 8. Dezember 2016 07:20
To: Stümpfig, Thomas (DF PL S&SE DE PSM EAI) <thomas.s...@siemens.com>
Cc: Pavel Lyalyakin <pavel.l...@visualsvn.com>; us...@subversion.apache.org
Subject: Re: svnsync on large files

[Daniel Shahaf]

Stümpfig, Thomas wrote on Thu, Dec 08, 2016 at 07:36:10 +0000:
> Hi Daniel,
> you are a guru. I don't know why but it worked!
>
> I guess the magic is in the difference of network transmission of svnsync vs svnrdump / svnadmin load

That's pretty unlikely: svnsync and svnrdump use the very same network
interface. (svn_ra_replay*())

> Are there suggestions from your side to avoid this error in the future?

Check in the master's error log the reason for the E175002. I imagine
some limit was run into, in which case, have httpd.conf exempt svnsync
from that limit.

> Thank you very much for your help!!!
>

You're welcome.

Daniel

[Daniel Shahaf]

Daniel Shahaf wrote on Thu, Dec 08, 2016 at 06:20:20 +0000:
> The steps should be this:
>
> f() {
> token="$USER@$(hostname):${RANDOM}:${RANDOM}"
> svn propset --revprop -r0 -- svn:sync-lock $token
> svn propget --revprop -r0 --strict svn:sync-lock | fgrep -q -- $token || return 1
> svn propset --revprop -r0 svn:sync-currently-copying 54618
> svnrdump dump -r54618 --incremental https://mysourceturl/repository >tmpfile
> svnadmin load /svnmirror/projektablage <tmpfile
> svn propset --revprop -r0 svn:sync-last-merged-rev 54618
> svn propdel --revprop -r0 svn:sync-currently-copying
> svn propdel --revprop -r0 svn:sync-lock
> }
>
> Note:
>
> 1. Using the same metadata revprops and order as svnsync.

Looking at this again: svn:sync-lock and svn:sync-currently-copying
serve the same purpose; if we could combine them, we'd save two
round-trips per revision, which would be a win for small revisions.

Cheers,

Daniel

[Pavel Lyalyakin]

Hello Daniel,

On Thu, Dec 8, 2016 at 1:08 AM, Daniel Shahaf <dani...@apache.org> wrote:
>
> Pavel Lyalyakin wrote on Wed, Dec 07, 2016 at 20:59:24 +0300:
> > On Wed, Dec 7, 2016 at 8:54 PM, Stümpfig, Thomas
> > <thomas.s...@siemens.com> wrote:
> > >
> > > The rev to be synced is 2,291,973,385 Bytes large (Size in txn-protorevs directory. Of target mirror)
> > >
> > > The sync is with svn 1.9.5 (Visual SVN Server) over both https
> >
> >
> > You should use file:/// access to the local repository instead of
> > HTTP(S).
>
> Why? There are perfectly good reasons to use http://localhost/, for
> example, to have a one-stop-shop killswitch for disabling "everything"
> from accessing the repository.

"E175012: Connection timed out" error might occur when you use
`svnsync` and source and target repositories are both accessed by
HTTP(S). My guess was that Thomas was getting this error.

When both URLs are HTTP(S), `svnsync` works unreliable. It creates two
connections to the servers and the local server will drop the
connection on timeout in case there are delays on the side of remote
server. The local server has to wait untill the remote server fully
receives the transaction and commits the data and may drop the
connection if it takes a bit longer. In most cases, such problem could
occur when syncing larger revisions but it also depends on the bandwidth.

Using file:/// access to local repository helps avoid this problem.

> I'm guessing the dump/incremental-load/update-r0-revprops surgery went
> wrong. I would (a) double-check that the revision that had been
> manually synced did in fact sync correctly, (b) review the r0 revprops.
> (Thomas: you can just post the 'proplist -v' here if you aren't sure
> whether it's correct now.)
>
> Also, as Pavel says, we need the full error message.
>
> Cheers,
>
> Daniel
>
> > Here is an example:
> > [[[
> > svnsync sync "https://svn.example.com/svn/MyRepo"
> > "file:///C:/Repositories/MyRepo/"
> > ]]]
> >
> > Does the error occur when you use file:/// access to local repository?
> >
> > > I tried to rdump, svnadmin load incremental (works) and adjust
> > > revprop info in rev 0 to the new imported files. But the I got
> > > a path already exists for following syncs. Might not be related to
> > > my manipulations but could be the case.