strange E/W200033 (b)locking error/warning when trying to modify a svn 1.8 working copy over SMB share

[Tobias Bading]

Hi.

I'm having a problem with a workflow that used to work fine with Subversion
1.6 but stopped working with Subversion 1.8:

I'm using GNU Emacs on an Ubuntu Lucid machine to edit files on an AIX
machine
over a SMB share. Emacs uses the Subversion 1.8.8 command line client to
perform vc-related operations. Read-only operations like showing the log or
diffs work fine. However, operations like "svn resolved" or "svn update"
fail,
e.g. "svn resolved" with

svn: warning: W200033: sqlite[S5]: database is locked

and "svn update" with

svn: E200033: Another process is blocking the working copy database, or the
underlying filesystem does not support file locking; if the working copy
is on
a network filesystem, make sure file locking has been enabled on the file
server
svn: E200033: sqlite[S5]: database is locked

Sounds like a nice error message with good hints to probable causes, right?
I was about to send an email to our IT people, but decided to run
"strace svn
resolved <file>" first to get more details. The output is a bit
confusing, at
least to me ;-). Hopefully someone can shed a little light on this.
Here's an
extract of the trace:

--- snip ---

[...]
lstat("/smb/<path-to-share>/<wc-root>/.svn/wc.db",
{st_mode=S_IFREG|0600, st_size=100352, ...}) = 0
stat("/smb/<path-to-share>/<wc-root>/.svn/wc.db", {st_mode=S_IFREG|0600,
st_size=100352, ...}) = 0
open("/smb/<path-to-share>/<wc-root>/.svn/wc.db", O_RDWR|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0600, st_size=100352, ...}) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=100352, ...}) = 0
stat("/smb/<path-to-share>/<wc-root>/.svn/wc.db", {st_mode=S_IFREG|0600,
st_size=100352, ...}) = 0
lseek(3, 0, SEEK_SET) = 0
read(3, "SQLite format 3\0\4\0\1\1\0@ \0\0\2\315\0\0\0b"..., 100) = 100
fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824,
len=1}) = 0
fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741826,
len=510}) = 0
fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=1073741824,
len=1}) = 0
access("/smb/<path-to-share>/<wc-root>/.svn/wc.db-journal", F_OK) = -1
ENOENT (No such file or directory)
fstat(3, {st_mode=S_IFREG|0600, st_size=100352, ...}) = 0
access("/smb/<path-to-share>/<wc-root>/.svn/wc.db-wal", F_OK) = -1
ENOENT (No such file or directory)
fstat(3, {st_mode=S_IFREG|0600, st_size=100352, ...}) = 0
[...]
fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=1073741825,
len=1}) = 0
[...]
fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=1073741824,
len=1}) = 0

then 10 times

fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=1073741826,
len=510}) = -1 EACCES (Permission denied)
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({1, 0}, 0x7fff030c7ab0) = 0

followed by

svn: warning: W200033: sqlite[S5]: database is locked

--- snip ---

Three things are interesting/puzzling:
- fcntl F_SETLK with start=1073741824 when the wc.db file contains only
100352
bytes? I found nothing in the fcntl man page about this. Is this allowed?
What's the purpose?
- fcntl F_SETLK with start=1073741825 & len=1 works, but later it fails with
start=1073741826 & len=510? So the locking over SMB works in
principle, but
there's a problem a few bytes beyond the 1 GB boundary?!??
- "svn resolved" prints only a warning and returns 0, although the operation
did fail, i.e. the file is still in conflict state. "svn update" on the
other hand aborts with E200033 and returns 1. Shouldn't "svn resolved" do
the same?

Details of the svn client:
svn, version 1.8.8 (r1568071)
compiled Mar 28 2014, 09:55:49 on x86_64-unknown-linux-gnu

Copyright (C) 2013 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/

The following repository access (RA) modules are available:

* ra_svn : Module for accessing a repository using the svn network protocol.
- with Cyrus SASL authentication
- handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
- handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using
serf.
- using serf 1.3.4
- handles 'http' scheme
- handles 'https' scheme

System information:

* running on x86_64-unknown-linux-gnu
- Ubuntu 10.04.4 LTS (lucid) [Linux 2.6.32-44-generic]
* linked dependencies:
- APR 1.5.0 (compiled with 1.5.0)
- APR-Util 1.5.3 (compiled with 1.5.3)
- SQLite 3.8.4.2 (static)

Any hints are welcome...

Thanks,
Tobias

[Branko Čibej]

On 03.04.2014 17:06, Tobias Bading wrote:

Hi.

I'm having a problem with a workflow that used to work fine with Subversion
1.6 but stopped working with Subversion 1.8:

I'm using GNU Emacs on an Ubuntu Lucid machine to edit files on an AIX machine
over a SMB share. Emacs uses the Subversion 1.8.8 command line client to
perform vc-related operations. Read-only operations like showing the log or
diffs work fine. However, operations like "svn resolved" or "svn update" fail,
e.g. "svn resolved" with

svn: warning: W200033: sqlite[S5]: database is locked

and "svn update" with

svn: E200033: Another process is blocking the working copy database, or the
underlying filesystem does not support file locking; if the working copy is on
a network filesystem, make sure file locking has been enabled on the file
server
svn: E200033: sqlite[S5]: database is locked

You probably have the exclusive-locking option set in your ~/.subversion/config (or in /etc/subversion/config). Emacs vc-mode is probably running two (or more) instances of 'svn', one of which block the other(s).

-- Brane

--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. br...@wandisco.com

[Philip Martin]

Tobias Bading <tba...@web.de> writes:

> Three things are interesting/puzzling:
> - fcntl F_SETLK with start=1073741824 when the wc.db file contains only 100352
> bytes? I found nothing in the fcntl man page about this. Is this allowed?
> What's the purpose?

SQLite implements read-write locks and uses those high byte-ranges to
store state information in a way that works on read-only filesystems.

--
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

[Tobias Bading]

Hi Brane,

I'm quite sure Emacs doesn't fork more than one svn child process in parallel, at least not for simple things like "svn resolved" on a single file. Besides, I get the same errors when trying "svn resolved <file>" or "svn update" in a shell. I'll do a few tests tomorrow with a SMB share on a Windoze machine to see whether I get similar errors or not.

Thanks,
Tobias

[Stefan Sperling]

On Thu, Apr 03, 2014 at 07:12:26PM +0200, Tobias Bading wrote:
> I'm quite sure Emacs doesn't fork more than one svn child process in
> parallel,

It doesn't matter how many clients emacs is forking.

As soon as emacs runs an svn client that tries to access the working copy
at the same as any other svn client (run by emacs or something else), one
of the clients will run into an error if exclusive locking mode is enabled.

> single file. Besides, I get the same errors when trying "svn resolved
> <file>" or "svn update" in a shell. I'll do a few tests tomorrow with
> a SMB share on a Windoze machine to see whether I get similar errors
> or not.

Not sure if this is really relevant but using a windows network share
with sqlite is discouraged by the sqlite faq: http://sqlite.org/faq.html#q5

"""
People who have a lot of experience with Windows tell me that file
locking of network files is very buggy and is not dependable. If what
they say is true, sharing an SQLite database between two or more Windows
machines might cause unexpected problems.
"""

It's possible that this extends to Samba but the wording doesn't make
this entirely clear.

[Philip Martin]

Tobias Bading <tba...@web.de> writes:

> fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}) = 0
> fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741826, len=510}) = 0
> fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=1073741824, len=1}) = 0

There should be a second fcntl with F_UNLCK, start=0, len=0 which
unlocks the whole file. On my Linux system it is present whether or not
I use exclusive locking:

fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741824, len=1}) = 0
fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1073741826, len=510}) = 0
fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=1073741824, len=1}) = 0

fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0

> access("/smb/<path-to-share>/<wc-root>/.svn/wc.db-journal", F_OK) = -1
> ENOENT (No such file or directory)
> fstat(3, {st_mode=S_IFREG|0600, st_size=100352, ...}) = 0
> access("/smb/<path-to-share>/<wc-root>/.svn/wc.db-wal", F_OK) = -1
> ENOENT (No such file or directory)
> fstat(3, {st_mode=S_IFREG|0600, st_size=100352, ...}) = 0
> [...]
> fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=1073741825, len=1}) = 0
> [...]
> fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=1073741824, len=1}) = 0
>
> then 10 times
>
> fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=1073741826, len=510}) = -1 EACCES (Permission denied)

The missing F_UNLCK means the file is still locked and so this fails. I
don't know why SQLite is missing the F_UNLCK.

[Tobias Bading]

On 03.04.2014, at 19:28, Stefan Sperling wrote:
> On Thu, Apr 03, 2014 at 07:12:26PM +0200, Tobias Bading wrote:
>> I'm quite sure Emacs doesn't fork more than one svn child process in
>> parallel,
>
> It doesn't matter how many clients emacs is forking.
>
> As soon as emacs runs an svn client that tries to access the working copy
> at the same as any other svn client (run by emacs or something else), one
> of the clients will run into an error if exclusive locking mode is enabled.

Ok, maybe I should have been a little clearer on what I'm doing.
Somewhere in my home directory on the AIX machine there is a private
working copy created by svn 1.8 locally on the AIX machine. I'm using
Emacs on a Linux machine to edit files, resolve conflicts etc.
The working copy needs to be on the AIX machine because I need to test
the code there. I'm quite sure that nobody except me is accessing that
working copy. Forget that I mentioned Emacs for a second ;-). I simply
run "svn resolved <file>" in a shell on the Linux machine while the
current directory is the SMB-auto-mounted working directory.

Theoretically, I guess it would be possible that a smbd process on the
AIX machine is still holding a lock from some aborted operation days
ago or so. But I'm quite sure that Emacs forking "svn resolved" never
worked for me with a remote 1.8 working copy. I'll check that with a
fresh working copy tomorrow.

> Not sure if this is really relevant but using a windows network share
> with sqlite is discouraged by the sqlite faq: http://sqlite.org/faq.html#q5
>
> """
> People who have a lot of experience with Windows tell me that file
> locking of network files is very buggy and is not dependable. If what
> they say is true, sharing an SQLite database between two or more Windows
> machines might cause unexpected problems.
> """
>
> It's possible that this extends to Samba but the wording doesn't make
> this entirely clear.

Yep, using NFS might by an alternative.

[Tobias Bading]

Hi Philip,

I knew I should have appended the whole trace... :-(
Sorry for the confusion. The [...]s *do* include more locking related fcntl
calls, including unlocks. However, given that one process shouldn't be able
to lock itself out (if I read the man page correctly), I tried to reduce the
trace output to the parts I thought were most relevant.

I can post the whole trace tomorrow. I'll also try to reproduce the error
with a fresh working copy to make sure there's no stale lock held by an smbd
daemon messing things up.

Thanks,
Tobias

[Tobias Bading]

Hi,

follow-up regarding my problems with "database is locked" errors when trying
to access a Subversion 1.8 working copy located on an AIX machine
remotely via
SMB from a GNU/Linux machine:

- A Windows SMB share causes the same errors, so it's not specific to Samba.

- The problem can be circumvented by mounting the SMB share with option
'nobrl'. From the mount.cifs(8) man page:

nobrl
Do not send byte range lock requests to the server. This is
necessary
for certain applications that break with cifs style mandatory byte
range locks (and most cifs servers do not yet support requesting
advisory byte range locks).

I'm currently giving the 'nobrl' option in combination with

[working-copy]
exclusive-locking-clients = svn

in ~/.subversion/config a try. So far it seems to work just fine. :-)

Tobias

[Stefan Sperling]

Hi Tobias,

That's very nice!

Would be able to prepare a patch against our FAQ to document this?
Perhaps this entry could be extended:
http://subversion.apache.org/faq.html#nfs

Source code of the website is here:
http://svn.apache.org/repos/asf/subversion/site/publish/

[Tobias Bading]

Well, so far I didn't do much testing. Basic things like Emacs running a
"svn resolved" when the last conflict marker is gone works, as does a "svn
update" from the Linux machine. But it might be possible that I've managed
to outmaneuver SQLite's locking mechanism(s) entirely. A "strace svn update"
shows that calls like

fcntl(3, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=1073741826,

len=510}) = 0

work now, which is to be expected of course if the 'nobrl' option basically
converted these calls into no-ops. I had hoped though that the
'exclusive-locking-clients = svn' option would cause svn to lock the wc.db
file completely, but I can't see anything like that in the trace, just a few
byte-range locks via fcntl(F_SETLK). So my guess would be that at the
moment, svn commands running on the Linux machine don't lock the working
copy on the AIX machine at all. That's fine for my use cases, but not
something I would recommend to others in general.