SVN 1.6.17 dump is growing larger than repository size (approx. more than 10 times)

[arun prasath]

Hello Team,

I am creating Subversion 1.6.17 dump for a repository hosted in Linux server. SVNSERVE is serving the repository. We are migrating to SVN 1.9.2.

While creating the dump for repository of size 1.8 GB (revisions 3000+), the dump command completes revision 119 and hangs and keep updating the dumpfile which grows to 7-8 GBs. dump command is not moving to next revision but kept updating the dump file. So, i just closed the putty to stop creating the dump.

However, I ran the svnadmin verify which completes revision 119 and take some time to verify revision 120 and complete the verification successfully for all the repository revisions.

Since, there is no error output I have no logs to attach. Please suggest the work around for this situation. How can create the dump and migrate to target server. I am planning to use rsync from the server. I will post how it goes.

I am expecting the workaround to this situation and let me know if this is a known issue in SVN 1.6.17. This is my active repository and created the dump without stopping the svnserve program in the production server. 

Thanks,

Arun

[Nico Kadel-Garcia]

On Wed, Nov 25, 2015 at 4:25 AM, arun prasath <get2...@gmail.com> wrote:
> Hello Team,
>
> I am creating Subversion 1.6.17 dump for a repository hosted in Linux
> server. SVNSERVE is serving the repository. We are migrating to SVN 1.9.2.
>
> While creating the dump for repository of size 1.8 GB (revisions 3000+), the
> dump command completes revision 119 and hangs and keep updating the dumpfile
> which grows to 7-8 GBs. dump command is not moving to next revision but kept
> updating the dump file. So, i just closed the putty to stop creating the
> dump.

Which Linux? And are you using the verndor provided version, or a
locally compiled one? Subversion 1.6 was last up to 1.6.23: is there
any reason you can't upgrade that? Ideally with a hotcopy or "rsync"
based copy of the repository to somehwere else, for testing the copy?

> However, I ran the svnadmin verify which completes revision 119 and take
> some time to verify revision 120 and complete the verification successfully
> for all the repository revisions.

Hmm. You might consider skipping the svnadmin dump command, and simply
setting up an svnsync mirror on the Subversion 1.9 server. Even using
"rsync" to bring the old repository over should let you run a dump and
reload on the server with Subversion 1.9, as long as there's not some
other weird corruption with revision 119 or 120.

> Since, there is no error output I have no logs to attach. Please suggest the
> work around for this situation. How can create the dump and migrate to
> target server. I am planning to use rsync from the server. I will post how
> it goes.
>
> I am expecting the workaround to this situation and let me know if this is a
> known issue in SVN 1.6.17. This is my active repository and created the dump
> without stopping the svnserve program in the production server.

It's not one I've seen, but I don't let my repositories get that big.
And I'd look at revisions 119 and 120 to see if there were erroneous
commits of huge binary files, in which case I'd want to exclude them
from the backup.

[Barry Gershenfeld]

Unless svn's changed, you can look in your repositories directory on the server, e.g., repos/myproj/db/revs/0/ and you can see each revision stored there.  Under the db directory is revs and also revprops, and under each of those is one or more subdirectory (mine just had one called "0").   If you see one that is outrageously large, then dump is just trying to do its job.  If they are all reasonable size, then maybe there's a circular reference or some other bug.

Also, svnadmin dump just goes to standard-out, so you can send it to the screen, or through a grep filter or some other ingenious scheme to try to get a handle on what's happening.

Barry

[Bert Huijben]

By default svnadmin dump doesn’t produce deltas… so a small change on  a file will have it include the entire file in the dumpfile.

 

If you pass ‘--deltas' to svnadmin dump (see ‘svnadmin help dump’) your file will be smaller, at the cost of some extra processing during the dump creation.

 

                Bert

[arun prasath]

Hi Andreas,

I have used svnsync and it took a while for rev 120 but it went successfully. There is no space constraint but it was taking the longer time when compared to other repositories. So for now, i have migrated most repositories and will see how i am going to migrate the rest in coming months. rsync is not useful for me since I am moving from one to another server without altering the existing setup.

For -M options - is this is the cache for processing for dump files. do I need to increase the value in MB like -M 1024 to speed up.

Sorry for late reply.

regards,

Arun

On Wed, Nov 25, 2015 at 4:31 PM, Andreas Stieger <Andreas...@gmx.de> wrote:

Hello,

 

Yes the dump can be significantly larger than the repository size, due to delta encoding used in the on-disk representation. If you examine revision 120 I am sure you will fine a particularly complex change that would cause this, and your statement about 120 being larger than normal is in line with that.

 

This is not a problem, it is simply expected behaviour. If the size of the dumpstream is a problem for you, you can use "--deltas" to reduce size of the dump stream.

 

If things are slow for you, specify a memory cache size (e.g. 1-2 GB if you have it) to speed up operations.

 

  -M [--memory-cache-size] ARG : size of the extra in-memory cache in MB used to
                             minimize redundant operations. Default: 16.
                             [used for FSFS repositories only]

 

You can use svnsync for a more transparent migration. I am not sure how rsync should be relevant or useful here when doing dumps anyway.

 

Andreas

 

Gesendet: Mittwoch, 25. November 2015 um 10:25 Uhr
Von: "arun prasath" <get2...@gmail.com>
An: us...@subversion.apache.org
Betreff: SVN 1.6.17 dump is growing larger than repository size (approx. more than 10 times)

[Andreas Stieger]

On 12/12/15 13:54, arun prasath wrote:
> For -M options - is this is the cache for processing for dump files.
> do I need to increase the value in MB like -M 1024 to speed up.

Yes this affects processing dump files. This option is documented. It
increases the size of the memory cache used to reduce redundant
operations. Give it plenty if you have it.

Andreas

[arun prasath]

I will use this and give a try. 

Thank you very much for help information. 

regards,

Arun