message "disk I/O error" when TurtoiseMerge is open (after update) and another updating by script/batch is done

[Arbeiter Ansgar]

after update an repository, i normally check actual changes (with TurtoiseMerge) so this tool and the update window are open.

when i then update again (by batch file) then i very often get this message:

 

svn: E200030: sqlite[S10]: disk I/O error

 

when i close TurtoiseMerge (and update window) before update again, then i never noticed it.

(i also checked for actually disk problems, but there are none)

 

batch file:

@echo off

svn --version > nul

if %ErrorLevel% gtr 0 (

  echo Sources can't be updated because the SVN-client not found. Please install the console tools TortoiseSVN.

) else (

  svn up ..\..\Axxx

  svn up ..\..\Cxxx

  svn up ..\..\Dxxx

  svn up ..

)

 

used versions:

 

TortoiseMerge 1.14.1, Build 29085 - 64 Bit , 2021/02/09 16:17:02

libsvn_diff 1.14.1,

apr 1.6.5

apr-util 1.6.1

 

 

TortoiseSVN 1.14.1, Build 29085 - 64 Bit , 2021/02/09 16:17:02

ipv6 enabled

Subversion 1.14.1, -release

apr 1.6.5

apr-util 1.6.1

serf 1.3.9

OpenSSL 1.1.1i  8 Dec 2020

zlib 1.2.11

SQLite 3.29.0

Mit freundlichen Grüßen / With kind regards

i.A. Ansgar Arbeiter
Graduate Engineer - Development

 

Böning Automationstechnologie GmbH & Co. KG
Am Steenöver 4
27777 Ganderkesee
Germany

Tel.: +49 4221 9475-51
Fax: +49 4221 9475-9051
ansgar....@boening.com

   

www.boening.com

Amtsgericht Oldenburg: HRA140737
Komplementär: Böning Verwaltungs GmbH
Amtsgericht Oldenburg: HRB141420
Geschäftsführer: Dipl.-Ing. Günther Böning
USt-ID Nr.: DE229750191 

[Nathan Hartman]

Hello,

It sounds like one of the tools may be holding the working copy
database open and/or locked in some way against access by other
programs (e.g., a file system based locking, a SQLite EXCLUSIVE_LOCK,
etc).

The error message "disk I/O error" may be misleading in this case;
i.e., it may not be an actual *disk* error, but rather inability to
open or operate on the database file for other reasons. (This error is
reported when the working copy's SQLite database reports SQLITE_IOERR.
The [S10] means the same thing as SQLITE_IOERR; it doesn't give any
more information. That is a generic code for a multitude of different
conditions.)

It's probably just best to close the Tortoise windows when you want to
use the command line client; however maybe someone else on this list
will know a better answer. You could also try asking at the
TortoiseSVN mailing list (see [1] below) whether the Tortoise client
is expected to prevent simultaneous access by other clients, and if
so, whether there is a way to avoid that.

[1] https://groups.google.com/g/tortoisesvn

Hope this helps,
Nathan

 

[Arbeiter Ansgar]

Hello Nathan,

 

thanks for reply,

sure, the "solution" is to avoid doing both together (although not convenient. especially as i have to do clean up afterwards if done accidently).

what i missed to mention it that doing the same in a previous version was working without any problem.

(but i can't say which versions it was. i guess about 1 year ago we updated svn and the version we used before had no problem)

i already tried to find known issues in turtoise bugtracker but did not find.

probably i should add one there.

 

Mit freundlichen Grüßen / With kind regards

i.A. Ansgar Arbeiter
Graduate Engineer - Development

 

Böning Automationstechnologie GmbH & Co. KG
Am Steenöver 4
27777 Ganderkesee
Germany

Tel.: +49 4221 9475-51
Fax: +49 4221 9475-9051
ansgar....@boening.com

   

www.boening.com

Amtsgericht Oldenburg: HRA140737
Komplementär: Böning Verwaltungs GmbH
Amtsgericht Oldenburg: HRB141420
Geschäftsführer: Dipl.-Ing. Günther Böning
USt-ID Nr.: DE229750191 

[Daniel Sahlberg]

I've tried to reproduce your problem but I can't. However you state that you get the error "very often" so maybe I'm just not lucky enough to catch it.

 Hello,

It sounds like one of the tools may be holding the working copy
database open and/or locked in some way against access by other
programs (e.g., a file system based locking, a SQLite EXCLUSIVE_LOCK,
etc).

The error message "disk I/O error" may be misleading in this case;
i.e., it may not be an actual *disk* error, but rather inability to
open or operate on the database file for other reasons. (This error is
reported when the working copy's SQLite database reports SQLITE_IOERR.
The [S10] means the same thing as SQLITE_IOERR; it doesn't give any
more information. That is a generic code for a multitude of different
conditions.)

It's probably just best to close the Tortoise windows when you want to
use the command line client; however maybe someone else on this list
will know a better answer. You could also try asking at the
TortoiseSVN mailing list (see [1] below) whether the Tortoise client
is expected to prevent simultaneous access by other clients, and if
so, whether there is a way to avoid that.

As far as I know, TortoiseSVN never touch the working copy by it self but only through calling Subversion's API. (I even checked the sources but I couldn't "findstr" ("grep" in Unix-speech) anything related to sqlite).

I seem to remember a similar question last year (?) where TortoiseSVN kept some resources open until the window was closed. I meant to look into it but never found the time. I think it was related to a temp file being held open, but I can't find the mail right now.

The svn command line client is doing one thing at a time (per process) whereas in TortoiseSVN you might start one thing (for example an update), then with that process still "alive" start other things, either in the same process or as separate processes. I think (hope) that there are some Subversion APIs that TortoiseSVN could call to cleanup things when whatever it does is finished instead of relying on cleaning up things 

If I find time, I'll try to look at it and submit patches to TortoiseSVN. As Nathan already wrote, you can also ask in the TortoiseSVN mailing lists [1].

 [1] https://groups.google.com/g/tortoisesvn

Kind regards,

Daniel Sahlberg

[Johan Corveleyn]

Note that there is a config setting in the client-side "config" file
that might have some impact here (setting "exclusive locking" for the
SQLite database):

http://subversion.apache.org/docs/release-notes/1.8.html#exclusivelocking

Also, look for "exclusive-locking" in this svnbook section:
https://svnbook.red-bean.com/nightly/en/svn.advanced.confarea.html#svn.advanced.confarea.opts.config

Perhaps you could test if that setting has any impact on the problem
you're seeing.

--
Johan

[Daniel Sahlberg]

Den tis 13 juli 2021 kl 11:16 skrev Johan Corveleyn <jco...@gmail.com>:

On Mon, Jul 12, 2021 at 10:32 PM Daniel Sahlberg
<daniel.l...@gmail.com> wrote:
> Den mån 12 juli 2021 kl 17:02 skrev Nathan Hartman <hartman...@gmail.com>:
>> On Mon, Jul 12, 2021 at 6:44 AM Arbeiter Ansgar <Ansgar....@boening.com> wrote:
>>>
>>> after update an repository, i normally check actual changes (with TurtoiseMerge) so this tool and the update window are open.
>>> when i then update again (by batch file) then i very often get this message:
>>>
>>> svn: E200030: sqlite[S10]: disk I/O error

[...]

 

Note that there is a config setting in the client-side "config" file
that might have some impact here (setting "exclusive locking" for the
SQLite database):

http://subversion.apache.org/docs/release-notes/1.8.html#exclusivelocking

Also, look for "exclusive-locking" in this svnbook section:
https://svnbook.red-bean.com/nightly/en/svn.advanced.confarea.html#svn.advanced.confarea.opts.config

Perhaps you could test if that setting has any impact on the problem
you're seeing.

I tried to reproduce the problem with exclusive-locking = true. I get another error message in this case, E200033:

[[[

C:\Devel\test>svn up

svn: E200033: Another process is blocking the working copy database, or the underlying filesystem does not support file locking; if the working copy is on a network filesystem, make sure file locking has been enabled on the file server

svn: E200033: sqlite[S5]: database is locked, executing statement 'PRAGMA case_sensitive_like=1;PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;PRAGMA foreign_keys=OFF;PRAGMA locking_mode = NORMAL;PRAGMA journal_mode = TRUNCATE;'

C:\Devel\test>

]]]

So it seems Subversion is properly detecting that the database is already locked (as indicated by the error code 5, SQLITE_BUSY, see [1]) and give another error message. In fact, E200030 is a fallback meaning "Sqlite returned an error for which we don't have a specific Subversion errorcode" [2].

That being said and combined with the fact that the error is intermittent, makes me wonder if you might have a problem with an antivirus/security software. Maybe TortoiseSVN is updating the database, which is caught by the security software triggering a scan, then when you run `svn up` the database is still locked exclusively by the security software. When you close TortoiseMerge, there is sufficent time to complete the scan so you don't notice? (I personally hate this explaination, but I've run into strange bugs where things start to work after some whitelisting, maybe whitelist the wc.db file?).

To test you could probably write a batch file along the following lines and run in a test wc.

[[[

svn up

echo "foo" >>test

svn add test

svn ci test -m test1

svn up

echo "bar">>test

svn ci test -m test2

svn up

]]]

Kind regards,

Daniel Sahlberg

[1] https://sqlite.org/rescode.html#ioerr

[2] https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_subr/sqlite.c, see the macro SQLITE_ERROR_CODE

[Arbeiter Ansgar]

Hello Daniel,

 

I will check if there is a Virus checker interacting.

 

>  there is sufficent time to complete the scan so you don't notice?

it does not matter if i wait 10 seconds or 2 hours.

relevant is that TurtoiseMerge is open while i update by script again.

so to me it looks like Turtoise is locking some data/files and releasing it first when i close it.

 

 

the result of your testing batch is the following:

 

L:\TestProject>test_svn_lock.bat

 

L:\TestProject>svn up

Updating '.':

U    branch\2.2.6\TestProject\src\eDef.pas

U    branch\2.2.6\TestProject\src\eDes.pas

U    branch\2.2.6\TestProject\src\eTyp.pas

U    branch\2.6.1\TestProject\src\eCon.pas

U    branch\2.6.1\TestProject\src\eDefs.pas

U    branch\2.6.1\TestProject\src\eDesCtrls.pas

U    branch\2.6.1\TestProject\src\eTyps.pas

U    trunk\src\eCon.pas

U    trunk\src\eDef.pas

U    trunk\src\eDes.pas

U    trunk\src\eTyp.pas

U    trunk\TestProjectTester.src\Tests\UnitTests\Test_7686.pas

 

Fetching external item into 'ASL':

Updated external to revision 1062.

 

 

Fetching external item into 'CommonLibs':

External at revision 606.

 

At revision 30790.

 

L:\TestProject>echo "foo"  1>>test

 

L:\TestProject>svn add test

A         test

 

L:\TestProject>svn ci test -m test1

Adding         test

Transmitting file data .done

Committing transaction...

Committed revision 30791.

 

L:\TestProject>svn up

Updating '.':

 

Fetching external item into 'ASL':

External at revision 1062.

 

 

Fetching external item into 'CommonLibs':

External at revision 606.

 

At revision 30791.

 

L:\TestProject>echo "bar" 1>>test

 

L:\TestProject>svn ci test -m test2

Sending        test

Transmitting file data .done

Committing transaction...

Committed revision 30792.

 

L:\TestProject>svn up

Updating '.':

 

Fetching external item into 'ASL':

External at revision 1062.

 

 

Fetching external item into 'CommonLibs':

External at revision 606.

 

At revision 30792.

 

L:\TestProject>

 

 

Mit freundlichen Grüßen / With kind regards

i.A. Ansgar Arbeiter
Graduate Engineer - Development

 

Böning Automationstechnologie GmbH & Co. KG
Am Steenöver 4
27777 Ganderkesee
Germany

Tel.: +49 4221 9475-51
Fax: +49 4221 9475-9051
ansgar....@boening.com

   

www.boening.com

Amtsgericht Oldenburg: HRA140737
Komplementär: Böning Verwaltungs GmbH
Amtsgericht Oldenburg: HRB141420
Geschäftsführer: Dipl.-Ing. Günther Böning
USt-ID Nr.: DE229750191 

Von: Daniel Sahlberg <daniel.l...@gmail.com>
Gesendet: Mittwoch, 21. Juli 2021 20:10

[Daniel Sahlberg]

Den tors 22 juli 2021 kl 13:12 skrev Arbeiter Ansgar <Ansgar....@boening.com>:

Hello Daniel,

 

I will check if there is a Virus checker interacting.

 

>  there is sufficent time to complete the scan so you don't notice?

it does not matter if i wait 10 seconds or 2 hours.

relevant is that TurtoiseMerge is open while i update by script again.

so to me it looks like Turtoise is locking some data/files and releasing it first when i close it.

Did you check the "exclusive locking" setting as suggested by Johan Corveleyn?

I think this thread should move to the TortoiseSVN support forums [1], since it it likely there is something within TortoiseMerge that keeps the connection open and it would have to be fixed there. (I'm reading that forum as well and can assist there as well).

If you want to prove things, you might use Process Explorer [2] to search for which process is holding wc.db open. This information as well as any information you come up with regarding when you get the error (ie, if the error can be reproduced consistently using some specific action) will be valuable.

the result of your testing batch is the following:

 

L:\TestProject>test_svn_lock.bat

[...]

 

Is L: a local drive or a network drive? TortoiseSVN explicitly advice against storing the WC in a network drive in their FAQ [3]. I have done it without any problems, but there are good reasons for this advice and YMMV.

Kind regards,

Daniel Sahlberg

 

[1] https://groups.google.com/g/tortoisesvn

[2] procexp.exe from http://live.sysinternals.com/

[3] https://tortoisesvn.net/faq.html#wconshare