NFS + SVN problem?

[Sean Bruno]

Not sure what this all means, but when I attempt to check out HEAD on an
NFS mount in the fbsd cluster (nfs server is a netapp filer), I'm
getting an odd failure error.

FreeBSD bhyve.freebsd.org 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r227883:
Wed Nov 23 06:08:40 PST 2011
sbr...@bhyve.freebsd.org:/usr/obj/var/tmp/temp/head/sys/GENERIC amd64

[sbruno@bhyve /dumpster/scratch/sbruno-scratch]$ svn co -q svn
+ssh://svn.freebsd.org/base/head
Enter passphrase for key '/home/sbruno/.ssh/id_rsa':
svn: E200030: disk I/O error, executing statement 'PRAGMA
synchronous=OFF;PRAGMA recursive_triggers=ON;'

Mounting the filer mount with the following entry in my fstab:

dumpster:/vol/volshscratch /dumpster/scratch nfs
rw,soft,bg,intr,nosuid 0 0

_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Dimitry Andric]

On 2011-11-23 17:41, Sean Bruno wrote:
> Not sure what this all means, but when I attempt to check out HEAD on an
> NFS mount in the fbsd cluster (nfs server is a netapp filer), I'm
> getting an odd failure error.
>
>
> FreeBSD bhyve.freebsd.org 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r227883:
> Wed Nov 23 06:08:40 PST 2011
> sbr...@bhyve.freebsd.org:/usr/obj/var/tmp/temp/head/sys/GENERIC amd64
>
>
> [sbruno@bhyve /dumpster/scratch/sbruno-scratch]$ svn co -q svn
> +ssh://svn.freebsd.org/base/head
> Enter passphrase for key '/home/sbruno/.ssh/id_rsa':
> svn: E200030: disk I/O error, executing statement 'PRAGMA
> synchronous=OFF;PRAGMA recursive_triggers=ON;'
>
>
> Mounting the filer mount with the following entry in my fstab:
>
> dumpster:/vol/volshscratch /dumpster/scratch nfs
> rw,soft,bg,intr,nosuid 0 0

Either downgrade to Subversion 1.6, or upgrade to NFSv4. I have found
nothing else that works. :(

Subversion 1.7 is a disaster in many areas, this is one of them...

[Rick Macklem]

Sean Bruno wrote:
> Not sure what this all means, but when I attempt to check out HEAD on
> an
> NFS mount in the fbsd cluster (nfs server is a netapp filer), I'm
> getting an odd failure error.
>
>
> FreeBSD bhyve.freebsd.org 10.0-CURRENT FreeBSD 10.0-CURRENT #0
> r227883:
> Wed Nov 23 06:08:40 PST 2011
> sbr...@bhyve.freebsd.org:/usr/obj/var/tmp/temp/head/sys/GENERIC amd64
>
>
> [sbruno@bhyve /dumpster/scratch/sbruno-scratch]$ svn co -q svn
> +ssh://svn.freebsd.org/base/head
> Enter passphrase for key '/home/sbruno/.ssh/id_rsa':
> svn: E200030: disk I/O error, executing statement 'PRAGMA
> synchronous=OFF;PRAGMA recursive_triggers=ON;'
>

I just did a checkout of head/sys to an NFS mount (off of a FreeBSD
server) and it worked ok. I have no idea what the above means, but
it suggests that the NFS mount point returned some error?

A couple of questions:
- Is the failure intermittent?
- Did it occur right when the svn was started or part way through it?

>
> Mounting the filer mount with the following entry in my fstab:
>
> dumpster:/vol/volshscratch /dumpster/scratch nfs
> rw,soft,bg,intr,nosuid 0 0
>

I'd try again without the "soft,intr" options. In particular, "soft"
can cause an I/O syscall to fail when the server is slow to respond and
apps. don't expect that to happen.

rick

[Rick Macklem]

Dinitry Andric wrote:
> On 2011-11-23 17:41, Sean Bruno wrote:
> > Not sure what this all means, but when I attempt to check out HEAD
> > on an
> > NFS mount in the fbsd cluster (nfs server is a netapp filer), I'm
> > getting an odd failure error.
> >
> >
> > FreeBSD bhyve.freebsd.org 10.0-CURRENT FreeBSD 10.0-CURRENT #0
> > r227883:
> > Wed Nov 23 06:08:40 PST 2011
> > sbr...@bhyve.freebsd.org:/usr/obj/var/tmp/temp/head/sys/GENERIC
> > amd64
> >
> >
> > [sbruno@bhyve /dumpster/scratch/sbruno-scratch]$ svn co -q svn
> > +ssh://svn.freebsd.org/base/head
> > Enter passphrase for key '/home/sbruno/.ssh/id_rsa':
> > svn: E200030: disk I/O error, executing statement 'PRAGMA
> > synchronous=OFF;PRAGMA recursive_triggers=ON;'
> >
> >
> > Mounting the filer mount with the following entry in my fstab:
> >
> > dumpster:/vol/volshscratch /dumpster/scratch nfs
> > rw,soft,bg,intr,nosuid 0 0
>
> Either downgrade to Subversion 1.6, or upgrade to NFSv4. I have found
> nothing else that works. :(
>

Ah, yes. I used an up-to-date NFS client but an old svn. If the upgrade
to NFSv4 helps, I suspect it is related to byte range locking, which I
think NFSv4 gets right.

rick

[Rick Macklem]

Dimitry Andric wrote:
> On 2011-11-23 17:41, Sean Bruno wrote:
> > Not sure what this all means, but when I attempt to check out HEAD
> > on an
> > NFS mount in the fbsd cluster (nfs server is a netapp filer), I'm
> > getting an odd failure error.
> >
> >
> > FreeBSD bhyve.freebsd.org 10.0-CURRENT FreeBSD 10.0-CURRENT #0
> > r227883:
> > Wed Nov 23 06:08:40 PST 2011
> > sbr...@bhyve.freebsd.org:/usr/obj/var/tmp/temp/head/sys/GENERIC
> > amd64
> >
> >
> > [sbruno@bhyve /dumpster/scratch/sbruno-scratch]$ svn co -q svn
> > +ssh://svn.freebsd.org/base/head
> > Enter passphrase for key '/home/sbruno/.ssh/id_rsa':
> > svn: E200030: disk I/O error, executing statement 'PRAGMA
> > synchronous=OFF;PRAGMA recursive_triggers=ON;'
> >
> >
> > Mounting the filer mount with the following entry in my fstab:
> >
> > dumpster:/vol/volshscratch /dumpster/scratch nfs
> > rw,soft,bg,intr,nosuid 0 0
>
> Either downgrade to Subversion 1.6, or upgrade to NFSv4. I have found
> nothing else that works. :(
>

I don't know if Dimitry tried this, but you could also try the
"nolockd" option, so that byte range locking is done locally in
the client and avoids the NLM.

Good luck with it and please let us know how it goes, rick

[Sean Bruno]

On Wed, 2011-11-23 at 09:58 -0800, Rick Macklem wrote:
> I don't know if Dimitry tried this, but you could also try the
> "nolockd" option, so that byte range locking is done locally in
> the client and avoids the NLM.
>
> Good luck with it and please let us know how it goes, rick

This seems to allow SVN 1.7 to do whatever nonsense it is trying to do.
I've modified my fstab on the test host in the cluster to:

dumpster:/vol/volshscratch /dumpster/scratch nfs
rw,soft,intr,bg,nolockd,nosuid 0 0

Removing soft,intr had no effect. This, I suspect will be problematic
for clusteradm@ if we start updating hosts in the cluster.

Sean

[Rick Macklem]

Sean Bruno wrote:
> On Wed, 2011-11-23 at 09:58 -0800, Rick Macklem wrote:
> > I don't know if Dimitry tried this, but you could also try the
> > "nolockd" option, so that byte range locking is done locally in
> > the client and avoids the NLM.
> >
> > Good luck with it and please let us know how it goes, rick
>
> This seems to allow SVN 1.7 to do whatever nonsense it is trying to
> do.
> I've modified my fstab on the test host in the cluster to:
>
> dumpster:/vol/volshscratch /dumpster/scratch nfs
> rw,soft,intr,bg,nolockd,nosuid 0 0
>

Yep. Unless you have multiple clients locking the same file concurrently,
doing locking locally within the client is the way to go. (Although I'm
not fond of "soft", I think the default timeout is pretty conservative,
so it would take a network partitioning or VERY SLOW server to trigger it.)
(The NLM and associated NSM protocols are fundamentally flawed in their
design, so no implementation can be expected to make them work "correctly".
Having said that, I am not familiar enough with the FreeBSD implementation
to try and fix specific scenarios.
I do have a patch submitted by John Dees that fixes the case where a process
with read access to a file attempts to read lock it. This case fails for the
code in head, because the nlm always tests for write access. I do plan on
getting this patch into head at some point.)

Oh, and don't hesitate to try NFSv4. It should do the locking correctly without
needing "nolockd" and the more testing it gets, the better.;-)

rick

> Removing soft,intr had no effect. This, I suspect will be problematic
> for clusteradm@ if we start updating hosts in the cluster.
>
> Sean

[Sean Bruno]

> Oh, and don't hesitate to try NFSv4. It should do the locking correctly without
> needing "nolockd" and the more testing it gets, the better.;-)
>
> rick
>
> > Removing soft,intr had no effect. This, I suspect will be problematic
> > for clusteradm@ if we start updating hosts in the cluster.
> >
> > Sean

Doesn't look like dumpster supports V4?

[tcp] dumpster:/vol/volshscratch: NFSPROC_NULL: RPC: Program/version
mismatch; low version = 2, high version = 3
mount_nfs: Cannot immediately mount dumpster:/vol/volshscratch,
backgrounding