svnadmin dump issue - E200015

[William Muriithi]

Hello,

I have a repository thats around 113 GB in size. Its on a VM and the
performance hasn't been that ideal. So we decided to source a new
hardware and set it up on its own dedicated system.

The current subversion system is subversion-1.8, and plan to move it
to subversion-1.9. I have en

source server:
subversion-1.8.13-1.x86_64

I have attemped svnadmin dump, svnadmin hotcopy and svnrdump. I am
getting the error before from all these utilities

svnadmin: E200015: Caught signal

or

* Dumped revision 2968.
svnrdump: E200015: Caught signal

My two questions are:

- Is it safe to use svnadmin dump on the source repository? As in
does it do it change in the source repo?

- What causes E200015? I don't think its permission as I have even
attempted to run svnadmin dump with root permissions? How can I
overcome the issue?

I have been blocked at this point since Wednesday. I have googled
extensively and don't seem to find anything that can help. Would be
glad if someone can point me to the right directions

Regards,

William

[Johan Corveleyn]

On Sun, Aug 7, 2016 at 8:41 PM, William Muriithi
<william....@gmail.com> wrote:
> Hello,
>
> I have a repository thats around 113 GB in size. Its on a VM and the
> performance hasn't been that ideal. So we decided to source a new
> hardware and set it up on its own dedicated system.
>
> The current subversion system is subversion-1.8, and plan to move it
> to subversion-1.9. I have en
>
> source server:
> subversion-1.8.13-1.x86_64
>
> I have attemped svnadmin dump, svnadmin hotcopy and svnrdump. I am
> getting the error before from all these utilities
>
> svnadmin: E200015: Caught signal
>
> or
>
> * Dumped revision 2968.
> svnrdump: E200015: Caught signal
>
> My two questions are:
>
> - Is it safe to use svnadmin dump on the source repository? As in
> does it do it change in the source repo?

Yes, it's absolutely safe to run on the source repository, it should
not cause anything to change in the source repo.

> - What causes E200015? I don't think its permission as I have even
> attempted to run svnadmin dump with root permissions? How can I
> overcome the issue?

Is that the full error message you gave?

"Caught signal" would be what you'd get if the process were aborted by
a signal, like aborting it with Ctrl-C, or if some other user with
enough permissions sends a kill signal to your process or something
like that.

Is it always aborting at the same revision?

Maybe someone else is interfering with your work, and (accidently)
killing your process? Or maybe your remote (ssh / whatever) session
gets disconnected which causes the dump process to abort?

--
Johan

[Nico Kadel-Garcia]

On Sun, Aug 7, 2016 at 4:45 PM, Johan Corveleyn <jco...@gmail.com> wrote:
> On Sun, Aug 7, 2016 at 8:41 PM, William Muriithi
> <william....@gmail.com> wrote:
>> Hello,
>>
>> I have a repository thats around 113 GB in size. Its on a VM and the
>> performance hasn't been that ideal. So we decided to source a new
>> hardware and set it up on its own dedicated system.
>>
>> The current subversion system is subversion-1.8, and plan to move it
>> to subversion-1.9. I have en
>>
>> source server:
>> subversion-1.8.13-1.x86_64
>>
>> I have attemped svnadmin dump, svnadmin hotcopy and svnrdump. I am
>> getting the error before from all these utilities

Break it up. Use svnadmin dump for 1000 commits at a time, to help
isolte if it's a specific commit that is killing you or if perhaps you
are running out of resources for such a large dump

In particular, 113 GB is pretty bulky. It's probably a good time to
think about trimming away some of the fat, of old branches and tags
that are no longer needed, or even to think about splitting the repo
up into smaller, individual projects. If that can be trimmed away to
proper use of svndumpfilter, more power to you. This violates the
tenet that the history, all of it, is the important thing, but I'm not
personally a big believer in it.,

There are also several nasty approaches to get something working
*now*, in parallel with your old repository. One is to do svn
export/import of the relevant code in a new working repository: you'd
need to reconcile the inevitable split brain after you get the full
repository copied, but if you're out of time it can save your ass.
(And yes, I'm bringing it up again, even though it consistently gets
me yelled at.)

There is also a nasty, but in a pinch workable, trick for getting
similar end results for unsophisticated repositories to get code up
and running if you're out of time. If you're not worried about
preserving svn:keywords, svn:ignore, or other added repository
attributes, you can get a fast clone working when svnadmin is giving
troubling by using git.

* Use "git svn" to make a "fast" copy.
* Delete unwanted debris, such as unuseful tags and branches.
* Use "git gc" to obliterate content that is no longer used.
* Use "git svn" to upload the content to a new, *much smaller* working
repository.

This flushes a great deal of attribute information and potentially
useful history, but it's often a much faster way to clear excess
debris and get a new, non-identical, but workable Subversion
repository with most of the important content history intact.

>> svnadmin: E200015: Caught signal
>>
>> or
>>
>> * Dumped revision 2968.
>> svnrdump: E200015: Caught signal
>>
>> My two questions are:
>>
>> - Is it safe to use svnadmin dump on the source repository? As in
>> does it do it change in the source repo?
>
> Yes, it's absolutely safe to run on the source repository, it should
> not cause anything to change in the source repo.

The big risk I see with a 113 GB repository is running out of
resources while doing "svnadmin dump".

>> - What causes E200015? I don't think its permission as I have even
>> attempted to run svnadmin dump with root permissions? How can I
>> overcome the issue?
>
> Is that the full error message you gave?
>
> "Caught signal" would be what you'd get if the process were aborted by
> a signal, like aborting it with Ctrl-C, or if some other user with
> enough permissions sends a kill signal to your process or something
> like that.
>
> Is it always aborting at the same revision?
>
> Maybe someone else is interfering with your work, and (accidently)
> killing your process? Or maybe your remote (ssh / whatever) session
> gets disconnected which causes the dump process to abort?

That... could also be a problem I would use "nohup" to wrap the dump
and get a usable output of both stdout and stderr.

If you're logging out in mid process and use a recent Linux, I'd also
be cautious about this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394

Recent versions of systemd kill user processes when the original user
login is disconnected, even if some process launched by that user is
backgrounded. It kills nohup'ed jobs, jobs in "screen" and "tmux", and
it doesn't log anywhere that it killed the process.

> --
> Johan

[William Muriithi]

Johan,

>> - Is it safe to use svnadmin dump on the source repository? As in
>> does it do it change in the source repo?
>
> Yes, it's absolutely safe to run on the source repository, it should
> not cause anything to change in the source repo.

Thanks. What about "svnadmin verify /var/svn/projects" ?

>
>> - What causes E200015? I don't think its permission as I have even
>> attempted to run svnadmin dump with root permissions? How can I
>> overcome the issue?
>
> Is that the full error message you gave?

Yes, that was it. There was nothing else that it threw at standard
error during the dump

>
> "Caught signal" would be what you'd get if the process were aborted by
> a signal, like aborting it with Ctrl-C, or if some other user with
> enough permissions sends a kill signal to your process or something
> like that.
>
> Is it always aborting at the same revision?

Yes, actually think its corruption related. I did manage dumping the
repo in full but have not been able to restore it. The restore fails
with the same error.

* editing path :
condor/trunk/work/design/ic/library/kl_condor/kl_vco_curmir_1to20mA/starrc/layout.oa
... done.
svnadmin: E200015: Caught signal
<<< Started new transaction, based on original revision 7001
* editing path :
condor/trunk/work/design/ic/library/yt_tmp/DM10_B_DUM ...svnadmin:
E160013: File not found: transaction '6490-50b', path
'/condor/trunk/work/design/ic/library/yt_tmp/DM10_B_DUM'

Note, its attempting to restore revision 7001 but the previous
revision is 6491. So, the file don't exist because it was created in
the future

>
> Maybe someone else is interfering with your work, and (accidently)
> killing your process? Or maybe your remote (ssh / whatever) session
> gets disconnected which causes the dump process to abort?

No, petty sure, I am the only one who have access to the new SVN
server but have seem E200015 error as seem above
>
> --
> Johan
Thanks a lot Johan

Regards,

William

[William Muriithi]

Nico,

>>> I have attemped svnadmin dump, svnadmin hotcopy and svnrdump. I am
>>> getting the error before from all these utilities
>
> Break it up. Use svnadmin dump for 1000 commits at a time, to help
> isolte if it's a specific commit that is killing you or if perhaps you
> are running out of resources for such a large dump
>

Thanks for this suggestion. Its precisely what I did and was able to
dump the full repo successfully. Well, now that I did look at the
logs, not too cleanly. I have a lot of warnings like this:

* Dumped revision 2062.
WARNING 0x0000: Referencing data in revision 1696, which is older than
the oldest dumped revision (r2001). Loading this dump into an empty
repository will fail.

* Dumped revision 3000.
WARNING 0x0000: The range of revisions dumped contained references to
copy sources outside that range.

WARNING 0x0001: Mergeinfo referencing revision(s) prior to the oldest
dumped revision (r57001). Loading this dump may result in invalid
mergeinfo.

What would this imply?

> In particular, 113 GB is pretty bulky. It's probably a good time to
> think about trimming away some of the fat, of old branches and tags
> that are no longer needed, or even to think about splitting the repo
> up into smaller, individual projects. If that can be trimmed away to
> proper use of svndumpfilter, more power to you. This violates the
> tenet that the history, all of it, is the important thing, but I'm not
> personally a big believer in it.,
>

Thanks, was planning to break it into smaller repos, but the process
hasn't been smooth, so for now, just want a simple dump and restore to
work first.

> There are also several nasty approaches to get something working
> *now*, in parallel with your old repository. One is to do svn
> export/import of the relevant code in a new working repository: you'd
> need to reconcile the inevitable split brain after you get the full
> repository copied, but if you're out of time it can save your ass.
> (And yes, I'm bringing it up again, even though it consistently gets
> me yelled at.)

Thats an interesting advice.. I can see the pain involved due to
split brain, but since the current repo is working, I think its better
to avoid this. But definally a great advice for firefighting

>
> There is also a nasty, but in a pinch workable, trick for getting
> similar end results for unsophisticated repositories to get code up
> and running if you're out of time. If you're not worried about
> preserving svn:keywords, svn:ignore, or other added repository
> attributes, you can get a fast clone working when svnadmin is giving
> troubling by using git.
>
> * Use "git svn" to make a "fast" copy.
> * Delete unwanted debris, such as unuseful tags and branches.
> * Use "git gc" to obliterate content that is no longer used.
> * Use "git svn" to upload the content to a new, *much smaller* working
> repository.
>
> This flushes a great deal of attribute information and potentially
> useful history, but it's often a much faster way to clear excess
> debris and get a new, non-identical, but workable Subversion
> repository with most of the important content history intact.
>

Interesting.. If I am not able to restore, I may just end up using
this.. I will leave it as last option and will have to see if
everybody can stomach it..

Thanks for sharing all this knowledge.

William

[Andreas Stieger]

> [...] svnadmin dump for 1000 commits at a time[...]
[...]

> * Dumped revision 2062.
> WARNING 0x0000: Referencing data in revision 1696, which is older than
> the oldest dumped revision (r2001). Loading this dump into an empty
> repository will fail.
>
> * Dumped revision 3000.
> WARNING 0x0000: The range of revisions dumped contained references to
> copy sources outside that range.
>
> WARNING 0x0001: Mergeinfo referencing revision(s) prior to the oldest
> dumped revision (r57001). Loading this dump may result in invalid
> mergeinfo.
>
> What would this imply?

Standard warning that if you dump part of the history, and that part references earlier history, the mergeinfo may be invalid (working, but not meaningfully so). If you load the dumped parts in the same sequence, this is not an issue.

Andreas

[William Muriithi]

Hi

Ah, thanks.  Great to hear this. Kind of now not certain what is triggering my load failure.

William

[Johan Corveleyn]

On Wed, Aug 10, 2016 at 6:43 PM, William Muriithi
<william....@gmail.com> wrote:
> Johan,
>
>>> - Is it safe to use svnadmin dump on the source repository? As in
>>> does it do it change in the source repo?
>>
>> Yes, it's absolutely safe to run on the source repository, it should
>> not cause anything to change in the source repo.
>
> Thanks. What about "svnadmin verify /var/svn/projects" ?

That's a read-only operation too (just like svnadmin dump), so should
be perfectly safe. BTW, both 'svnadmin dump' and 'svnadmin verify'
should run without problems on a busy repository (it doens't interfere
with commits and other write or read operations that happen in
parellel).

>>
>>> - What causes E200015? I don't think its permission as I have even
>>> attempted to run svnadmin dump with root permissions? How can I
>>> overcome the issue?
>>
>> Is that the full error message you gave?
>
> Yes, that was it. There was nothing else that it threw at standard
> error during the dump
>>
>> "Caught signal" would be what you'd get if the process were aborted by
>> a signal, like aborting it with Ctrl-C, or if some other user with
>> enough permissions sends a kill signal to your process or something
>> like that.
>>
>> Is it always aborting at the same revision?
> Yes, actually think its corruption related. I did manage dumping the
> repo in full but have not been able to restore it.

Hm, the fact that you *were* able to dump it in full (I guess by
splitting it up in chunks, as was suggested by Nico) indicates that
it's probably not a corruption in your source repository. The fact
that it works by dumping in chunks, but not in one giant dump, points
to a resource problem for creating or storing the giant dump.

> The restore fails
> with the same error.
>
> * editing path :
> condor/trunk/work/design/ic/library/kl_condor/kl_vco_curmir_1to20mA/starrc/layout.oa
> ... done.
> svnadmin: E200015: Caught signal

Here too I would guess you hit some system limit (RAM / swap space,
disk, maybe open file descriptors, ...). Perhaps you can split the
load process in even more steps, by specifying even smaller revision
ranges (but still using the dump files you created earlier). See
'svnadmin help load':

[[[
...
If --revision is specified, limit the loaded revisions to only those
in the dump stream whose revision numbers match the specified range.
]]]

--
Johan