Bug: Extrememly slow commits to USB drive on Windows 11

[Nition]

There seems to be something strange going on with TortoiseSVN, the "file://" protocol, and USB drives, at least on Windows 11.

I have a 3.68GB project:

• Creating a local SVN repo and committing the whole project to it, both located on the same SSD, takes a few minutes.
• Copying the project folder manually to a USB 3.0 drive takes a few minutes.
• Creating a GIT repo on the USB drive and committing the project to it takes a few minutes.
• Creating an SVN repo on the USB drive and committing the project to it takes 16 hours.

Tested with several versions: TortoiseSVN 1.10.5, TortoiseSVN 1.14.2, and the absolute latest 1.14.99.29376. Same result on all.

I found one other person mentioning the same issue here.

[Daniel Sahlberg]

It would be very interesting to see if you can reproduce the issue using the svn command line program (e.g. svn checkout file:///c:/repo e:\wc). I suspect the issue is in the Subversion code and not in the TortoiseSVN code. If you have the same problem with the svn command line programs, you can report it on the Subversion mailing lists: https://subversion.apache.org/mailing-lists.html

Subversion operations are relatively disk intensive and for example create a "pristine" copy of each file during checkout (in the .svn\pristine folder) and THEN copies that file to the final location in the working copy. Thus each file is written at least twice. If you have a low throughput disk device this means low performance, unfortunately.

There has been a number of improvements in the very latest version of Subversion (both regarding how data is written to the working copy as well as removing the "pristine copy" thus reducing disk space requirements).

Kind regards,

Daniel

[Nition]

Hi, yeah good call to try it with pure commandline.

I've now tried the same commit with commandline svn and, although I didn't wait for the whole thing to complete this time, it did seem to be similarly slow. So as you say, this sounds like an issue for SVN rather than for TortoiseSVN specifically. Thanks for your time anyway.

I understand that the commit may be much slower than a direct file copy, but in that case I'd be expecting maybe half an hour or even an hour, but not sixteen hours. I said something similar in an earlier comment that's still pending approval.

[Daniel Sahlberg]

torsdag 17 mars 2022 kl. 10:13:37 UTC+1 skrev Nition:

Hi, yeah good call to try it with pure commandline.

I've now tried the same commit with commandline svn and, although I didn't wait for the whole thing to complete this time, it did seem to be similarly slow. So as you say, this sounds like an issue for SVN rather than for TortoiseSVN specifically. Thanks for your time anyway.

I understand that the commit may be much slower than a direct file copy, but in that case I'd be expecting maybe half an hour or even an hour, but not sixteen hours. I said something similar in an earlier comment that's still pending approval.

Thanks for testing! It is good to know that the problem is the same in the command line client - then it is not TSVN specific and you might want to report it to us...@subversion.apache.org where the other devs might be able to provide more insight. Sorry that it didn't solve your problem.

If I understand it correctly from the other links, the problem is very much dependent on the type of USB drive. The fact that a drive has a USB 3.0 interface doesn't mean it has a fast storage chip inside.  You didn't mention what type of USB drive you are using but I expect there can be a significant difference betwen a cheap thumbdrive and an external SSD drive even if they both have a USB 3.0 interface.

Kind regards,

Daniel

PS. I mentioned changes in the very latest Subversion code. This is (at the time of writing) still only on /trunk (some parts even only on feature branches) and nothing is released yet.

[Johan Corveleyn]

Something to try: look into the (since 1.8) option of enabling
"exclusive-locking" for the internal SQLite database of the working
copy:
https://subversion.apache.org/docs/release-notes/1.8.html#exclusivelocking

I vaguely remember discussions on the d...@subversion.apache.org list
(or us...@s.a.o) about performance problems with working copies on
network drives (with USB drives it might be similar), after the 1.7
release (which introduced the central wc.db SQLite database for the
working copy metadata). After some research this exclusive locking
option was introduced (which actives an option for SQLite if I
remember correctly). The downside is "coarser grained locking" of the
entire working copy during operations (whereas, without this option,
certain operations can be performed concurrently). On an USB drive I
suppose concurrent operations are not very likely, so this might be a
big win (impact to be tested).

HTH,
--
Johan

On Fri, Mar 18, 2022 at 7:32 AM Bill via TortoiseSVN
<torto...@googlegroups.com> wrote:
>
> To be clear, I would understand if it took significantly longer than simply copying the files, like say 30 minutes. But to take 16 hours seems like it must be a bug of some kind. The person on Stack Overflow that I linked had theirs transferring even more slowly - 100MB over eight hours.

> --
> You received this message because you are subscribed to the Google Groups "TortoiseSVN" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tortoisesvn...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/tortoisesvn/bac5b522-e969-475b-b611-50973f0300b9n%40googlegroups.com.

[Nition]

Hi Johan

I just tried it again after enabling `exclusive-locking = true` in the config file, which hopefully did something.

I didn't do a very scientific test, but basically, it seems like may be a little faster but nothing major. Pretty interesting hack though - I've ever dug into the config file options before.

[Johan Corveleyn]

Hm, okay, that's disappointing.

But, I'm a bit confused actually. What are you putting on the USB disk?
* The 'svn repository' (the actual back-end database, created with
'svnadmin create', from which you checkout via the file:// protocol)
* A 'svn working copy' (a checkout of the central repository, which
can be on another filesystem accessed via the file:// protocol, or via
the http(s):// or svn:// protocols)

Keep in mind that in Subversion the repository (which keeps all
history) and a working copy (local checkout of some tree@revision of
the repository) are two entirely different things. This is different
from DVCS'es like GIT, where the (local clone of the) repository is at
the same time the set of files you'll be working on.

The Stackoverflow article you linked to [1] speaks about creating a
'checkout to an USB(3) drive'. In your first post you spoke about
'Creating an SVN repo on the USB drive and committing the project to
it takes 16 hours.'
Which of the two is it that you're putting on that USB drive? A
working copy (a checkout of a repo somewhere else) or an SVN
repository?

[1] https://stackoverflow.com/questions/68847008/checkout-speed-toirtoise-svn-on-usb-drive

--
Johan

> To view this discussion on the web visit https://groups.google.com/d/msgid/tortoisesvn/494f1e15-bc1b-439d-8682-67a34a8034b0n%40googlegroups.com.



--
Johan

[Nition]

The former, an SVN repository. My test process is:

- Create a new repository on the USB drive

- Check out the empty repository onto an SSD

- Add all the files (3.68GB)
- Commit all that as the initial commit (takes 16 hours)

I realise committing gigabytes at once is far from best practice. In this case I had a project which I'd worked on for a little while before putting it under version control.

[Johan Corveleyn]

On Tue, Mar 22, 2022 at 8:13 PM Nition via TortoiseSVN
<torto...@googlegroups.com> wrote:
>
> The former, an SVN repository. My test process is:
> - Create a new repository on the USB drive
> - Check out the empty repository onto an SSD
> - Add all the files (3.68GB)
> - Commit all that as the initial commit (takes 16 hours)
>
> I realise committing gigabytes at once is far from best practice. In this case I had a project which I'd worked on for a little while before putting it under version control.

Ah, wow! Okay, that makes this an entirely different issue than the
one you referred to on StackOverflow
(https://stackoverflow.com/questions/68847008/checkout-speed-toirtoise-svn-on-usb-drive),
which is about having a *working copy* on a USB drive.

So, naturally, my suggestion of using "exclusive locking" for the
client on the working copy, would not help at all.

I think this pretty much means this is not the ideal mailinglist for
this issue, as you are talking about a server-side / repository
performance problem (TortoiseSVN is an SVN *client*, offering a GUI
for the client-side of things). If you're using the file:// protocol
to access the repository then the client is also acting as an internal
"svn server". But also in that case, the issue is with the server-side
(repository) libraries contained in the client, and the way the
back-end files are organised inside the repository.

I suggest you repeat your question on the us...@subversion.apache.org
mailinglist. See
https://subversion.apache.org/mailing-lists.html#users-ml for
instructions on subscribing (or not, if you prefer -- you don't have
to be subscribed to post, but then do mention that you want to be
cc'ed). Several people (including me) follow both lists. Also, do
mention which protocol you use to access the repository (file:// ?),
and the precise version with which you created the repository (the
output of 'svnadmin --version' which you used to run 'svnadmin
create'), as well as the version of (Tortoise)SVN you're using at the
end.

I'm not sure whether you'll receive adequate suggestions there, but in
any case there are a bit more "server / back-end oriented" people on
there that might have experience with this sort of thing. Do keep in
mind that the repository back-end is not simply a filesystem where
these files are copied, you have to view it as a *database*. An
incoming commit is built up as a transaction, with a lot of operations
happening on the fly to optimize the storage etc (for instance
"deltification", compression, "representation sharing") -- some of
these things can be tuned for certain workloads to, for instance, get
you more speed at the cost of storage.

--
Johan

[Daniel Sahlberg]

I agree to what Johan say about the mailing list. I'm also following both lists and try to help on both sides.

onsdag 23 mars 2022 kl. 10:46:56 UTC+1 skrev Johan Corveleyn:

and the precise version with which you created the repository (the
output of 'svnadmin --version' which you used to run 'svnadmin
create'), as well as the version of (Tortoise)SVN you're using at the
end.

One small comment is that TortoiseSVN is capable of creating the repository through the GUI, so Nition might not be familiar with the svnadmin command. In that case, just report the TortoiseSVN version number. (If you use the command line tool as supplied with TortoiseSVN these should be the same except for the very latest TSVN 1.14.2 which still contains Subversion 1.14.1).

Kind regards,

Daniel 

[Nition]

Yes sorry, I actually didn't notice that the Stack Overflow post was checking out to a USB drive. The other factors matched (TortoiseSVN, USB drive, file:// protocol). And yes I usually use the TortoiseSVN GUI but I can use the commandline when I have to. :)

I don't know if I will bother asking on the mailing list to be honest. I have my big repository checked in now, and don't expect much trouble with future smaller commits. Large commits of many files to a flash drive certainly do seem slower than expected but the slowness makes some sense given the limitations of the medium - and honestly, I just have lots of other work to do. Thanks for your time helping out in this thread.