files disappearing from ls on NFS

[Hartmut Brandt]

Hi,

I've updated one of my -current machines this week (previous update was in
february). Now I see a strange effect (it seems only on NFS mounts): ls or
even echo * will list only some files (strange enough the first files from
the normal, alphabetically ordered list). If I change something in the
directory (delete a file or create a new one) for some time the complete
listing will appear but after sime time (seconds to a minute or so) again
only part of the files is listed.

A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is called
only once (returning 4096). For a full listing getdirentries is called 5
times with the last returning 0.

I can still open files that are not listed if I know their name, though.

The NFS server is a Windows 2008 server with an OpenText NFS Server which
works without problems to all the other FreeBSD machines.

So what could that be?

Regards,
harti
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Freddie Cash]

There was just a security update that dealt with changes to getdirent or
something along those lines.

Check the security notices, and then see if reverting that change makes a
difference.

It was just in the past week here.

On Thu, May 2, 2013 at 9:11 AM, Hartmut Brandt <hartmut...@dlr.de>wrote:

> Hi,
>
> I've updated one of my -current machines this week (previous update was in
> february). Now I see a strange effect (it seems only on NFS mounts): ls or
> even echo * will list only some files (strange enough the first files from
> the normal, alphabetically ordered list). If I change something in the
> directory (delete a file or create a new one) for some time the complete
> listing will appear but after sime time (seconds to a minute or so) again
> only part of the files is listed.
>
> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is called
> only once (returning 4096). For a full listing getdirentries is called 5
> times with the last returning 0.
>
> I can still open files that are not listed if I know their name, though.
>
> The NFS server is a Windows 2008 server with an OpenText NFS Server which
> works without problems to all the other FreeBSD machines.
>
> So what could that be?
>
> Regards,
> harti

> ______________________________**_________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/**mailman/listinfo/freebsd-**current<http://lists.freebsd.org/mailman/listinfo/freebsd-current>
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@**
> freebsd.org <freebsd-curre...@freebsd.org>"
>



--
Freddie Cash
fjw...@gmail.com

[Hartmut Brandt]

On Thu, 2 May 2013, Freddie Cash wrote:

FC>There was just a security update that dealt with changes to getdirent or
FC>something along those lines.
FC>
FC>Check the security notices, and then see if reverting that change makes a
FC>difference.
FC>
FC>It was just in the past week here.

If you mean SA-13:05.nfsserver that seams to be related to the NFS server
only. My problem is in the client.

harti

FC>
FC>
FC>On Thu, May 2, 2013 at 9:11 AM, Hartmut Brandt <hartmut...@dlr.de>
FC>wrote:
FC> Hi,
FC>
FC> I've updated one of my -current machines this week (previous
FC> update was in february). Now I see a strange effect (it seems
FC> only on NFS mounts): ls or even echo * will list only some files
FC> (strange enough the first files from the normal, alphabetically
FC> ordered list). If I change something in the directory (delete a
FC> file or create a new one) for some time the complete listing
FC> will appear but after sime time (seconds to a minute or so)
FC> again only part of the files is listed.
FC>
FC> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
FC> called only once (returning 4096). For a full listing
FC> getdirentries is called 5 times with the last returning 0.
FC>
FC> I can still open files that are not listed if I know their name,
FC> though.
FC>
FC> The NFS server is a Windows 2008 server with an OpenText NFS
FC> Server which works without problems to all the other FreeBSD
FC> machines.
FC>
FC> So what could that be?
FC>
FC> Regards,
FC> harti
FC> _______________________________________________
FC> freebsd...@freebsd.org mailing list
FC> http://lists.freebsd.org/mailman/listinfo/freebsd-current
FC> To unsubscribe, send any mail to
FC> "freebsd-curre...@freebsd.org"
FC>
FC>
FC>
FC>
FC>--
FC>Freddie Cash
FC>fjw...@gmail.com
FC>
FC>

[Rick Macklem]

Someone else reported missing files returned via "ls" recently, when
they used a small readdirsize (below 8K). I haven't yet had a change to try
and reproduce it or do any snooping around.

There haven't been any recent changes to readdir in the NFS client,
except a trivial one that adds a check for vnode type being VDIR,
so I don't see that it can be a recent NFS change.

If you can increase the readdirsize, try that to see if it avoids
the problem. "nfsstat -m" shows you what the mount options end up
being after doing the mount. The server might be limiting the readdirsize
to 4K, so you should check, even if you specify a large value for
the mount.

rick

[Hartmut Brandt]

Hi Rick,

I checked. readdirsize is 64k.

I will try to do a binary search for the problematic commit next week. For
this I had to do a local checkout (usually I have the system sources on
the file server and this fails, of course).

harti

On Fri, 3 May 2013, Rick Macklem wrote:

RM>Hartmut Brandt wrote:
RM>> Hi,
RM>>
RM>> I've updated one of my -current machines this week (previous update
RM>> was in
RM>> february). Now I see a strange effect (it seems only on NFS mounts):
RM>> ls or
RM>> even echo * will list only some files (strange enough the first files
RM>> from
RM>> the normal, alphabetically ordered list). If I change something in the
RM>> directory (delete a file or create a new one) for some time the
RM>> complete
RM>> listing will appear but after sime time (seconds to a minute or so)
RM>> again
RM>> only part of the files is listed.
RM>>
RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
RM>> called
RM>> only once (returning 4096). For a full listing getdirentries is called
RM>> 5
RM>> times with the last returning 0.
RM>>
RM>> I can still open files that are not listed if I know their name,
RM>> though.
RM>>
RM>> The NFS server is a Windows 2008 server with an OpenText NFS Server
RM>> which
RM>> works without problems to all the other FreeBSD machines.
RM>>
RM>> So what could that be?
RM>>
RM>Someone else reported missing files returned via "ls" recently, when
RM>they used a small readdirsize (below 8K). I haven't yet had a change to try
RM>and reproduce it or do any snooping around.
RM>
RM>There haven't been any recent changes to readdir in the NFS client,
RM>except a trivial one that adds a check for vnode type being VDIR,
RM>so I don't see that it can be a recent NFS change.
RM>
RM>If you can increase the readdirsize, try that to see if it avoids
RM>the problem. "nfsstat -m" shows you what the mount options end up
RM>being after doing the mount. The server might be limiting the readdirsize
RM>to 4K, so you should check, even if you specify a large value for
RM>the mount.
RM>
RM>rick
RM>
RM>> Regards,
RM>> harti
RM>> _______________________________________________
RM>> freebsd...@freebsd.org mailing list
RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
RM>> To unsubscribe, send any mail to
RM>> "freebsd-curre...@freebsd.org"
RM>

[Hartmut Brandt]

On Fri, 3 May 2013, Daniel Braniss wrote:

DB>I don't know about current, but on 9.1-stable, the nfsstat -m only works
DB>for root! nfsstat can be run by anybody.

Same for current. It silently prints nothing. Took me some time
to figure out I should try as root...

[Daniel Braniss]

I don't know about current, but on 9.1-stable, the nfsstat -m only works

for root! nfsstat can be run by anybody.

>

[Rick Macklem]

Yep. I played is "safe" and only allowed root to do it.
I thought that some sysadmins might not want users to know what NFS
mounts are being done and I didn't see any need for non-root to be
able to do it.

Having said the above, I don't have a strong opinion on it or an obvious
example of a security risk caused by opening it up, so if the collective
thinks it should be doable by non-root, it can be changed.

rick

[Rick Macklem]

Hartmut Brandt wrote:
> On Fri, 3 May 2013, Daniel Braniss wrote:
>
> DB>I don't know about current, but on 9.1-stable, the nfsstat -m only
> works
> DB>for root! nfsstat can be run by anybody.
>
> Same for current. It silently prints nothing. Took me some time
> to figure out I should try as root...
>

Yea, I suppose it should either be opened up to non-root (see previous post)
or "only works as root" should be documented.

rick

[Rick Macklem]

Hartmut Brandt wrote:
> Hi Rick,
>
> I checked. readdirsize is 64k.
>

Since the server replied with 4K, I suspect you are seeing the same
problem the other reporter has.

I'll email if/when I have more information on it.

> I will try to do a binary search for the problematic commit next week.
> For
> this I had to do a local checkout (usually I have the system sources
> on
> the file server and this fails, of course).
>

Ok, if you succeed in isolating the commit, that would be great.

rick

[Rick Macklem]

Just in case..is the NFS mount in a union mount by any chance?
(There is a known bug for NFS readdir under a union mount, because the
NFS client readdir isn't setting the eofflag. Someone has sent me a patch
that looks fine, but it hasn't been checked into head yet.)

I'll also grep a head kernel to check to see if anything else (except the
NFS server) uses the eofflag argument to VOP_READDIR().

rick

[Daniel Braniss]

> Hartmut Brandt wrote:
> > On Fri, 3 May 2013, Daniel Braniss wrote:
> >
> > DB>I don't know about current, but on 9.1-stable, the nfsstat -m only
> > works
> > DB>for root! nfsstat can be run by anybody.
> >
> > Same for current. It silently prints nothing. Took me some time
> > to figure out I should try as root...
> >
> Yea, I suppose it should either be opened up to non-root (see previous post)
> or "only works as root" should be documented.
>

I have no strong opinios either, though security by obscurity was never my
favorit.
It should at least have said 'permition denied'.

cheers,
danny

[Hartmut Brandt]

On Fri, 3 May 2013, Rick Macklem wrote:

RM>Ok, if you succeed in isolating the commit, that would be great.

Hmm. I'm somewhat stuck. clang from yesterday can't compile clang from a
month ago...

harti

RM>
RM>rick
RM>
RM>> harti
RM>>

RM>> On Fri, 3 May 2013, Rick Macklem wrote:
RM>>

RM>> RM>Hartmut Brandt wrote:
RM>> RM>> Hi,
RM>> RM>>

RM>> RM>> I've updated one of my -current machines this week (previous

RM>> update
RM>> RM>> was in
RM>> RM>> february). Now I see a strange effect (it seems only on NFS
RM>> mounts):
RM>> RM>> ls or
RM>> RM>> even echo * will list only some files (strange enough the first
RM>> files
RM>> RM>> from
RM>> RM>> the normal, alphabetically ordered list). If I change something
RM>> in the
RM>> RM>> directory (delete a file or create a new one) for some time the
RM>> RM>> complete
RM>> RM>> listing will appear but after sime time (seconds to a minute or
RM>> so)
RM>> RM>> again
RM>> RM>> only part of the files is listed.

RM>> RM>>
RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is

RM>> RM>> called
RM>> RM>> only once (returning 4096). For a full listing getdirentries is
RM>> called
RM>> RM>> 5
RM>> RM>> times with the last returning 0.

RM>> RM>>
RM>> RM>> I can still open files that are not listed if I know their name,

RM>> RM>> though.
RM>> RM>>

RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS

RM>> Server
RM>> RM>> which
RM>> RM>> works without problems to all the other FreeBSD machines.

RM>> RM>>
RM>> RM>> So what could that be?
RM>> RM>>
RM>> RM>Someone else reported missing files returned via "ls" recently,

RM>> when
RM>> RM>they used a small readdirsize (below 8K). I haven't yet had a
RM>> change to try
RM>> RM>and reproduce it or do any snooping around.

RM>> RM>
RM>> RM>There haven't been any recent changes to readdir in the NFS client,

RM>> RM>except a trivial one that adds a check for vnode type being VDIR,
RM>> RM>so I don't see that it can be a recent NFS change.

RM>> RM>
RM>> RM>If you can increase the readdirsize, try that to see if it avoids

RM>> RM>the problem. "nfsstat -m" shows you what the mount options end up
RM>> RM>being after doing the mount. The server might be limiting the
RM>> readdirsize
RM>> RM>to 4K, so you should check, even if you specify a large value for
RM>> RM>the mount.

RM>> RM>
RM>> RM>rick
RM>> RM>

RM>> RM>> Regards,
RM>> RM>> harti
RM>> RM>> _______________________________________________
RM>> RM>> freebsd...@freebsd.org mailing list
RM>> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
RM>> RM>> To unsubscribe, send any mail to
RM>> RM>> "freebsd-curre...@freebsd.org"
RM>> RM>

[Rick Macklem]

Hartmut Brandt wrote:
> On Fri, 3 May 2013, Rick Macklem wrote:
>
> RM>Ok, if you succeed in isolating the commit, that would be great.
>
> Hmm. I'm somewhat stuck. clang from yesterday can't compile clang from
> a
> month ago...
>
> harti
>

Oh well. You could try this patch (which is the one to fix readdir for
union mounts), since I can see that VOP_VPTOCNP() will also be broken
without it. (I can't see how that would break "ls", but it breaks __getcwd()
and friends, so maybe it can affect "ls" somehow?)

It's a cut/paste under windows, so I'm afraid the whitespace will be
messed up, but it's pretty simple to apply by hand.

Index: nfs_clvnops.c
===================================================================
--- nfs_clvnops.c (revision 249568)
+++ nfs_clvnops.c (working copy)
@@ -2221,6 +2221,7 @@
!NFS_TIMESPEC_COMPARE(&np->n_mtime, &vattr.va_mtime)) {
mtx_unlock(&np->n_mtx);
NFSINCRGLOBAL(newnfsstats.direofcache_hits);
+ *ap->a_eofflag = 1;
return (0);
} else
mtx_unlock(&np->n_mtx);
@@ -2233,8 +2234,10 @@
tresid = uio->uio_resid;
error = ncl_bioread(vp, uio, 0, ap->a_cred);

- if (!error && uio->uio_resid == tresid)
+ if (!error && uio->uio_resid == tresid) {
NFSINCRGLOBAL(newnfsstats.direofcache_misses);
+ *ap->a_eofflag = 1;
+ }
return (error);
}

I haven't yet succeeded in reproducing the problem, but will be poking
at it some more, rick

[Rick Macklem]

Hartmut Brandt wrote:
> On Fri, 3 May 2013, Rick Macklem wrote:
>
> RM>Ok, if you succeed in isolating the commit, that would be great.
>
> Hmm. I'm somewhat stuck. clang from yesterday can't compile clang from
> a
> month ago...
>
> harti
>

Oh, and one other thing you can try is switching to the old client
"mount -t oldnfs ...".

rick

[Hartmut...@dlr.de]

Looks like the problem is in the new NFS code - the old code does the right thing. I've still to try your patch...

harti
________________________________________
From: Rick Macklem [rmac...@uoguelph.ca]
Sent: Sunday, May 05, 2013 12:49 AM
To: Brandt, Hartmut
Cc: cur...@freebsd.org
Subject: Re: files disappearing from ls on NFS

[Hartmut...@dlr.de]

Hi Rick,

the patch doesn't help. So how can I help to fix that? Of course, I can use the work-around with oldnfs, but ...

harti

-----Original Message-----
From: Rick Macklem [mailto:rmac...@uoguelph.ca]
Sent: Saturday, May 04, 2013 11:33 PM
To: Brandt, Hartmut
Cc: cur...@freebsd.org; Andrzej Tobola
Subject: Re: files disappearing from ls on NFS

[Rick Macklem]

Hartmut Brandt wrote:
> Hi Rick,
>
> the patch doesn't help. So how can I help to fix that? Of course, I
> can use the work-around with oldnfs, but ...
>

Well, I plan on going through the readdir code and seeing if I can spot
a case that would break for small RPC replies. If I can find something,
I'll email you a patch for testing. (I can't seem to reproduce the problem
here.)

The mysterious part for me is why it has shown up recently, because there
hasn't been any recent change committed that seems like it could cause this.
(Maybe it is just a co-incidence that it showed up recently and the bug has
been there all along?)

I'll admit my worst fear is that is somehow caused by the switch to clang for
certain arches. If that is the case, it could take a long time to isolate.

rick

[Hartmut Brandt]

On Mon, 6 May 2013, Rick Macklem wrote:

RM>Hartmut Brandt wrote:
RM>> Hi Rick,
RM>>
RM>> the patch doesn't help. So how can I help to fix that? Of course, I
RM>> can use the work-around with oldnfs, but ...
RM>>
RM>Well, I plan on going through the readdir code and seeing if I can spot
RM>a case that would break for small RPC replies. If I can find something,
RM>I'll email you a patch for testing. (I can't seem to reproduce the problem
RM>here.)
RM>
RM>The mysterious part for me is why it has shown up recently, because there
RM>hasn't been any recent change committed that seems like it could cause this.
RM>(Maybe it is just a co-incidence that it showed up recently and the bug has
RM> been there all along?)
RM>
RM>I'll admit my worst fear is that is somehow caused by the switch to clang for
RM>certain arches. If that is the case, it could take a long time to isolate.

I'm quite sure that I've build the system in February with clang already.
But in march or so a new clang version was committed.

harti

RM>> -----Original Message-----
RM>> From: Rick Macklem [mailto:rmac...@uoguelph.ca]
RM>> Sent: Saturday, May 04, 2013 11:33 PM
RM>> To: Brandt, Hartmut
RM>> Cc: cur...@freebsd.org; Andrzej Tobola
RM>> Subject: Re: files disappearing from ls on NFS

RM>>
RM>> Hartmut Brandt wrote:
RM>> > On Fri, 3 May 2013, Rick Macklem wrote:
RM>> >

RM>> > RM>Ok, if you succeed in isolating the commit, that would be great.
RM>> >
RM>> > Hmm. I'm somewhat stuck. clang from yesterday can't compile clang
RM>> > from
RM>> > a month ago...
RM>> >
RM>> > harti
RM>> >
RM>> Oh well. You could try this patch (which is the one to fix readdir for
RM>> union mounts), since I can see that VOP_VPTOCNP() will also be broken
RM>> without it. (I can't see how that would break "ls", but it breaks
RM>> __getcwd() and friends, so maybe it can affect "ls" somehow?)
RM>>
RM>> It's a cut/paste under windows, so I'm afraid the whitespace will be
RM>> messed up, but it's pretty simple to apply by hand.
RM>>
RM>> Index: nfs_clvnops.c
RM>> ===================================================================
RM>> --- nfs_clvnops.c (revision 249568)
RM>> +++ nfs_clvnops.c (working copy)
RM>> @@ -2221,6 +2221,7 @@
RM>> !NFS_TIMESPEC_COMPARE(&np->n_mtime, &vattr.va_mtime)) {
RM>> mtx_unlock(&np->n_mtx);
RM>> NFSINCRGLOBAL(newnfsstats.direofcache_hits);
RM>> + *ap->a_eofflag = 1;
RM>> return (0);
RM>> } else
RM>> mtx_unlock(&np->n_mtx); @@ -2233,8 +2234,10 @@
RM>> tresid = uio->uio_resid;
RM>> error = ncl_bioread(vp, uio, 0, ap->a_cred);
RM>>
RM>> - if (!error && uio->uio_resid == tresid)
RM>> + if (!error && uio->uio_resid == tresid) {
RM>> NFSINCRGLOBAL(newnfsstats.direofcache_misses);
RM>> + *ap->a_eofflag = 1;
RM>> + }
RM>> return (error);
RM>> }
RM>>
RM>> I haven't yet succeeded in reproducing the problem, but will be poking
RM>> at it some more, rick
RM>>

RM>> > RM>
RM>> > RM>rick
RM>> > RM>

RM>> > RM>> harti
RM>> > RM>>

RM>> > RM>> On Fri, 3 May 2013, Rick Macklem wrote:
RM>> > RM>>
RM>> > RM>> RM>Hartmut Brandt wrote:

RM>> > RM>> RM>> Hi,

RM>> > RM>> RM>>
RM>> > RM>> RM>> I've updated one of my -current machines this week

RM>> > (previous
RM>> > RM>> update
RM>> > RM>> RM>> was in
RM>> > RM>> RM>> february). Now I see a strange effect (it seems only on
RM>> > NFS
RM>> > RM>> mounts):
RM>> > RM>> RM>> ls or
RM>> > RM>> RM>> even echo * will list only some files (strange enough the
RM>> > first
RM>> > RM>> files
RM>> > RM>> RM>> from
RM>> > RM>> RM>> the normal, alphabetically ordered list). If I change
RM>> > something
RM>> > RM>> in the
RM>> > RM>> RM>> directory (delete a file or create a new one) for some
RM>> > time
RM>> > the
RM>> > RM>> RM>> complete
RM>> > RM>> RM>> listing will appear but after sime time (seconds to a
RM>> > minute
RM>> > or
RM>> > RM>> so)
RM>> > RM>> RM>> again
RM>> > RM>> RM>> only part of the files is listed.

RM>> > RM>> RM>>
RM>> > RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that

RM>> > getdirentries is
RM>> > RM>> RM>> called
RM>> > RM>> RM>> only once (returning 4096). For a full listing
RM>> > getdirentries
RM>> > is
RM>> > RM>> called
RM>> > RM>> RM>> 5
RM>> > RM>> RM>> times with the last returning 0.

RM>> > RM>> RM>>
RM>> > RM>> RM>> I can still open files that are not listed if I know their

RM>> > name,
RM>> > RM>> RM>> though.

RM>> > RM>> RM>>
RM>> > RM>> RM>> The NFS server is a Windows 2008 server with an OpenText

RM>> > NFS
RM>> > RM>> Server
RM>> > RM>> RM>> which
RM>> > RM>> RM>> works without problems to all the other FreeBSD machines.

RM>> > RM>> RM>>
RM>> > RM>> RM>> So what could that be?
RM>> > RM>> RM>>
RM>> > RM>> RM>Someone else reported missing files returned via "ls"

RM>> > recently,
RM>> > RM>> when
RM>> > RM>> RM>they used a small readdirsize (below 8K). I haven't yet had
RM>> > a
RM>> > RM>> change to try
RM>> > RM>> RM>and reproduce it or do any snooping around.

RM>> > RM>> RM>
RM>> > RM>> RM>There haven't been any recent changes to readdir in the NFS

RM>> > client,
RM>> > RM>> RM>except a trivial one that adds a check for vnode type being
RM>> > VDIR,
RM>> > RM>> RM>so I don't see that it can be a recent NFS change.

RM>> > RM>> RM>
RM>> > RM>> RM>If you can increase the readdirsize, try that to see if it

RM>> > avoids
RM>> > RM>> RM>the problem. "nfsstat -m" shows you what the mount options
RM>> > end
RM>> > up
RM>> > RM>> RM>being after doing the mount. The server might be limiting
RM>> > the
RM>> > RM>> readdirsize
RM>> > RM>> RM>to 4K, so you should check, even if you specify a large
RM>> > value
RM>> > for
RM>> > RM>> RM>the mount.

RM>> > RM>> RM>
RM>> > RM>> RM>rick
RM>> > RM>> RM>
RM>> > RM>> RM>> Regards,

RM>> > RM>> RM>> harti
RM>> > RM>> RM>> _______________________________________________
RM>> > RM>> RM>> freebsd...@freebsd.org mailing list
RM>> > RM>> RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
RM>> > RM>> RM>> To unsubscribe, send any mail to
RM>> > RM>> RM>> "freebsd-curre...@freebsd.org"

[Rick Macklem]

I've attached a patch that might be worth trying. It is a "shot in the dark",
but brings the new NFS client's readdir closer to the old one (which you
mentioned still works ok).

Please let me know how it goes, if you have a chance to test it, rick

[Hartmut Brandt]

On Sun, 12 May 2013, Rick Macklem wrote:

RM>Hartmut Brandt wrote:

RM>> Hi,
RM>>
RM>> I've updated one of my -current machines this week (previous update
RM>> was in
RM>> february). Now I see a strange effect (it seems only on NFS mounts):
RM>> ls or
RM>> even echo * will list only some files (strange enough the first files
RM>> from
RM>> the normal, alphabetically ordered list). If I change something in the
RM>> directory (delete a file or create a new one) for some time the
RM>> complete
RM>> listing will appear but after sime time (seconds to a minute or so)
RM>> again
RM>> only part of the files is listed.
RM>>
RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is
RM>> called
RM>> only once (returning 4096). For a full listing getdirentries is called
RM>> 5
RM>> times with the last returning 0.
RM>>
RM>> I can still open files that are not listed if I know their name,
RM>> though.
RM>>
RM>> The NFS server is a Windows 2008 server with an OpenText NFS Server
RM>> which
RM>> works without problems to all the other FreeBSD machines.
RM>>
RM>> So what could that be?
RM>>
RM>I've attached a patch that might be worth trying. It is a "shot in the dark",
RM>but brings the new NFS client's readdir closer to the old one (which you
RM>mentioned still works ok).
RM>
RM>Please let me know how it goes, if you have a chance to test it, rick

Hi Rick,

the patch doesn't help.

I wrote a small test program, which opens a directory, calls getdents(2)
in a loop and dumps that. I figured out, that the return of the system
call depends on the buffer size I pass to it. The directory has a block
size of 4k according to fstat(2). If I use that, I get some 300 of the
almost 500 directory entries. If I use 8k, I get just around 200 and if I
use 16k I get a handfull. If I dump the buffer in this case I see 0x200
bytes filled with directory entries, then a lot of zeros and starting from
0x1000 again data. This is of course ignored because of the zeros before.

All this looks very strange.

[Rick Macklem]

And for this case getdents(2) returned 16K? It is normal for getdents(2)
to return less than requested and when end of dir occurs, it should return 0.

But if it returns 16K, there shouldn't be zeroed space in the middle of
it.

And this always occurs or only after you wait a while? (You noted in the
above description that it would be ok for a little while after a directory
change and then would break, which suggests some kind of caching problem.)

rick

[Hartmut Brandt]

On Mon, 13 May 2013, Rick Macklem wrote:

RM>Hartmut Brandt wrote:

RM>> On Sun, 12 May 2013, Rick Macklem wrote:
RM>>

RM>> RM>Hartmut Brandt wrote:
RM>> RM>> Hi,
RM>> RM>>

RM>> RM>> I've updated one of my -current machines this week (previous

RM>> update
RM>> RM>> was in
RM>> RM>> february). Now I see a strange effect (it seems only on NFS
RM>> mounts):
RM>> RM>> ls or

RM>> RM>> even echo * will list only some files (strange enough the first
RM>> files
RM>> RM>> from
RM>> RM>> the normal, alphabetically ordered list). If I change something
RM>> in the
RM>> RM>> directory (delete a file or create a new one) for some time the
RM>> RM>> complete
RM>> RM>> listing will appear but after sime time (seconds to a minute or
RM>> so)
RM>> RM>> again

RM>> RM>> only part of the files is listed.

RM>> RM>>
RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that getdirentries is

RM>> RM>> called
RM>> RM>> only once (returning 4096). For a full listing getdirentries is
RM>> called
RM>> RM>> 5

RM>> RM>> times with the last returning 0.

RM>> RM>>
RM>> RM>> I can still open files that are not listed if I know their name,

RM>> RM>> though.
RM>> RM>>

RM>> RM>> The NFS server is a Windows 2008 server with an OpenText NFS

RM>> Server
RM>> RM>> which

RM>> RM>> works without problems to all the other FreeBSD machines.

RM>> RM>>
RM>> RM>> So what could that be?
RM>> RM>>
RM>> RM>I've attached a patch that might be worth trying. It is a "shot in

RM>> the dark",
RM>> RM>but brings the new NFS client's readdir closer to the old one
RM>> (which you
RM>> RM>mentioned still works ok).

RM>> RM>
RM>> RM>Please let me know how it goes, if you have a chance to test it,

RM>> rick
RM>>
RM>> Hi Rick,
RM>>
RM>> the patch doesn't help.
RM>>
RM>> I wrote a small test program, which opens a directory, calls
RM>> getdents(2)
RM>> in a loop and dumps that. I figured out, that the return of the system
RM>> call depends on the buffer size I pass to it. The directory has a
RM>> block
RM>> size of 4k according to fstat(2). If I use that, I get some 300 of the
RM>> almost 500 directory entries. If I use 8k, I get just around 200 and
RM>> if I
RM>> use 16k I get a handfull. If I dump the buffer in this case I see
RM>> 0x200
RM>> bytes filled with directory entries, then a lot of zeros and starting
RM>> from
RM>> 0x1000 again data. This is of course ignored because of the zeros
RM>> before.
RM>>
RM>And for this case getdents(2) returned 16K? It is normal for getdents(2)
RM>to return less than requested and when end of dir occurs, it should return 0.
RM>
RM>But if it returns 16K, there shouldn't be zeroed space in the middle of
RM>it.
RM>
RM>And this always occurs or only after you wait a while? (You noted in the
RM>above description that it would be ok for a little while after a directory
RM>change and then would break, which suggests some kind of caching problem.)

Today in the morning everything was fine. After waiting 5 minutes, again
only partial directories. When I do a read with 8k buffer size,
getdents(2) returns 8k, but starting from 0x200 until 0x1000 the buffer is
filled with zeros. The entry just before the zeroes ends exactly at 0x200
(that would be the first byte of the next entry) and at 0x1000 a new entry
starts. The rest of the buffer is fine. The next read returns only 4k and
seems to be fine - altough it contains some junk non-zero bytes in the
padding.

Ten minutes later again everything is fine. I tries to spy at the NFS
communication with tcpdump, but it seems unwilling to display something
useful about the NFS. Is it able to decode the readdir stuff?

[Rick Macklem]

Directory entries should never cross DIRBLKSIZ boundaries (512 or 0x200),
so it makes sense that one ends at 0x200 and one starts at 0x1000. What
doesn't make sense are the 0 bytes in between.

One difference between the old and new NFS clients, which the patch I sent
you changed to the way the old one does it, is filling in the last block.
The old NFS client just leaves the block short and depends on n_direofoffset
to recognize it is the last block with b_resid indicating where it ends.
For the new client (unless you've applied the patch I emailed you), it fills
the rest of the last block in with "empty directories". This was in the OpenBSD
code when I did the original NFSv4 stuff and port. I left it in, because I
thought it might avoid problems if n_direofoffset was ever bogus. That is why
there might be "different junk" at the end of the directory, but it shouldn't
matter.

It almost sounds like something else is bzero()ing out part of the buffer
cache block. Unless the directory has changed, the getdents() after 5 minutes
would just return the same buffer cache block that was read in 5 minutes
earlier (unless the buffer fell out of the cache and had to be re-read from
the server, which would only happen if there was a lot of other file I/O
going on during that 5minutes).

A couple of comments:
- You can run "nfsstat -m" as root to see what the mount it actually
configured to use. This might be worth looking at, to see if any
of the values are "weird".
- One other difference between the old and new NFS clients is the
value of NFS_DIRBLKSIZ. For the new one, it is 8K instead of 4K.
You could change this in fs/nfs/nfsport.h, where is is defined
and then rebuild the sources to see if it has any effect. I can't
see why it should matter, but??
- Maybe you could post your system configuration. Someone might spot
something that changed in Feb.->Mar. related to your hardware/setup?

> Ten minutes later again everything is fine. I tries to spy at the NFS
> communication with tcpdump, but it seems unwilling to display
> something
> useful about the NFS. Is it able to decode the readdir stuff?
>

To look at NFS packets you need wireshark. You can capture the packets
with tcpdump using the -w option. Something like:
# tcpdump -s 0 -w file.pcap host server
- Then look at file.pcap in wireshark. (Often more convenient than
installing wireshark on a particular machine.)
If you'd like, you can email me the file.pcap and I can look at it.

rick

[Hartmut...@dlr.de]

Now I've also changed NFS_DIRBLKSIZ to 4k - no change.

harti

-----Original Message-----
From: Rick Macklem [mailto:rmac...@uoguelph.ca]

Sent: Tuesday, May 14, 2013 2:50 PM
To: Brandt, Hartmut
Cc: cur...@freebsd.org

Subject: Re: files disappearing from ls on NFS

> RM>> block size of 4k according to fstat(2). If I use that, I get some
> RM>> 300

[Rick Macklem]

Hartmut Brandt wrote:
> Hi Rick,
>

> sorry for top-posting - this is Outlook :-(
>
> Attached is the system configuration. I use this more or less
> unchanged since years. The machine is an 8-core AMD64 with 144GByte
> memory.
>
> The nfsstats -m output for the two file systems I'm testing with is:
>
> knopfs01:/OP_UserUnix on /home
> nfsv3,tcp,resvport,hard,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=6126856,timeout=120,retrans=2
> knopfs01:/op_software on /software
> nfsv3,tcp,resvport,hard,cto,lockd,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=6126856,timeout=120,retrans=2
>
> I did the tcpdump/wireshark thing and I'm puzzled that I see no
> readdir requests. I see a lookup, followed by getattr, access and
> fsstat for the directory and that's it. Looks that even after hours
> the stuff returned by getdirents(2) comes from the cache. I assume
> that the NFS client uses getattr to check whether
> the directory has changed? If I knew what happens when calling
> getdirents() I could add some debugging printfs() here and there to
> figure out...
>
Yes. The NFS client will check the mtime on the directory to see if it has
changed and just use whatever is in the buffer cache otherwise.

Well, getdents() basically just calls kern_getdirentries() and it calls
VOP_READDIR() { which is called nfs_readdir() in the NFS clients }.
nfs_readdir() calls ncl_bioread() to do the real work of finding the
buffer cache blocks and copying the data out of them.
One thing you might check via printf()s is whether the buffer cache
has the zero'd data in it before it copies it to userland.

Since you get valid data sometimes and partially zero'd out data others,
I haven't a clue what is going on. One other person reported a problem
when they used a small readdirsize, but it is hard to say they saw the
same thing and no one else seems to be seeing this, so I have no idea
what it might be.

I remember you started seeing this after an upgrade of current. Do you
happen to have dates (or rNNNNNN) for the old working verion vs the one that broke this?
(All I can think to do is scan the commits that seem to somehow relate
to the buffer cache or copying to userland or ???)

rick

[Hartmut Brandt]

On Wed, 15 May 2013, Rick Macklem wrote:

RM>Well, getdents() basically just calls kern_getdirentries() and it calls
RM>VOP_READDIR() { which is called nfs_readdir() in the NFS clients }.
RM>nfs_readdir() calls ncl_bioread() to do the real work of finding the
RM>buffer cache blocks and copying the data out of them.
RM>One thing you might check via printf()s is whether the buffer cache
RM>has the zero'd data in it before it copies it to userland.

I now dump the data just before the call to vn_io_fault_iomove in
ncl_bioread(). So what I do:

1. reboot
2. login
3. ls
-> I see that it is moving 4 blocks 4k each to the user and they look
fine
4. wait half an hour
5. ls
-> now there is only one block, which contains zeros starting from
0x200.

Note that I don't do anything else on that machine during that time.

RM>Since you get valid data sometimes and partially zero'd out data others,
RM>I haven't a clue what is going on. One other person reported a problem
RM>when they used a small readdirsize, but it is hard to say they saw the
RM>same thing and no one else seems to be seeing this, so I have no idea
RM>what it might be.
RM>
RM>I remember you started seeing this after an upgrade of current. Do you
RM>happen to have dates (or rNNNNNN) for the old working verion vs the one that broke this?
RM>(All I can think to do is scan the commits that seem to somehow relate
RM> to the buffer cache or copying to userland or ???)

It looks like I had copied the old kernel before installing the new one
and it is from february 5th. There is no SVN revision in it - looks like
that feature was added only recently.

harti

RM>
RM>rick
RM>
RM>> harti
RM>>
RM>> -----Original Message-----
RM>> From: Rick Macklem [mailto:rmac...@uoguelph.ca]
RM>> Sent: Tuesday, May 14, 2013 2:50 PM

RM>> To: Brandt, Hartmut
RM>> Cc: cur...@freebsd.org

RM>> Subject: Re: files disappearing from ls on NFS

RM>>
RM>> Hartmut Brandt wrote:

RM>> > On Mon, 13 May 2013, Rick Macklem wrote:
RM>> >

RM>> > RM>Hartmut Brandt wrote:

RM>> > RM>> On Sun, 12 May 2013, Rick Macklem wrote:
RM>> > RM>>
RM>> > RM>> RM>Hartmut Brandt wrote:

RM>> > RM>> RM>> Hi,

RM>> > RM>> RM>>
RM>> > RM>> RM>> I've updated one of my -current machines this week

RM>> > (previous
RM>> > RM>> update
RM>> > RM>> RM>> was in
RM>> > RM>> RM>> february). Now I see a strange effect (it seems only on
RM>> > NFS
RM>> > RM>> mounts):
RM>> > RM>> RM>> ls or
RM>> > RM>> RM>> even echo * will list only some files (strange enough the
RM>> > first
RM>> > RM>> files
RM>> > RM>> RM>> from
RM>> > RM>> RM>> the normal, alphabetically ordered list). If I change
RM>> > something
RM>> > RM>> in the
RM>> > RM>> RM>> directory (delete a file or create a new one) for some
RM>> > time
RM>> > the
RM>> > RM>> RM>> complete
RM>> > RM>> RM>> listing will appear but after sime time (seconds to a
RM>> > minute
RM>> > or
RM>> > RM>> so)

RM>> > RM>> RM>> again
RM>> > RM>> RM>> only part of the files is listed.

RM>> > RM>> RM>>
RM>> > RM>> RM>> A ktrace on ls /usr/src/lib/libc/gen shows that

RM>> > getdirentries is
RM>> > RM>> RM>> called
RM>> > RM>> RM>> only once (returning 4096). For a full listing
RM>> > getdirentries
RM>> > is
RM>> > RM>> called
RM>> > RM>> RM>> 5
RM>> > RM>> RM>> times with the last returning 0.

RM>> > RM>> RM>>
RM>> > RM>> RM>> I can still open files that are not listed if I know their

RM>> > name,
RM>> > RM>> RM>> though.

RM>> > RM>> RM>>
RM>> > RM>> RM>> The NFS server is a Windows 2008 server with an OpenText

RM>> > NFS
RM>> > RM>> Server

RM>> > RM>> RM>> which
RM>> > RM>> RM>> works without problems to all the other FreeBSD machines.

RM>> > RM>> RM>>
RM>> > RM>> RM>> So what could that be?
RM>> > RM>> RM>>
RM>> > RM>> RM>I've attached a patch that might be worth trying. It is a

RM>> > "shot in
RM>> > RM>> the dark",
RM>> > RM>> RM>but brings the new NFS client's readdir closer to the old
RM>> > one
RM>> > RM>> (which you
RM>> > RM>> RM>mentioned still works ok).

RM>> > RM>> RM>
RM>> > RM>> RM>Please let me know how it goes, if you have a chance to test

RM>> > it,

RM>> > RM>> rick
RM>> > RM>>

RM>> > RM>> Hi Rick,
RM>> > RM>>
RM>> > RM>> the patch doesn't help.
RM>> > RM>>
RM>> > RM>> I wrote a small test program, which opens a directory, calls

RM>> > RM>> getdents(2)
RM>> > RM>> in a loop and dumps that. I figured out, that the return of the
RM>> > system
RM>> > RM>> call depends on the buffer size I pass to it. The directory has
RM>> > a
RM>> > RM>> block size of 4k according to fstat(2). If I use that, I get
RM>> > some
RM>> > RM>> 300
RM>> > of the
RM>> > RM>> almost 500 directory entries. If I use 8k, I get just around
RM>> > 200
RM>> > and
RM>> > RM>> if I
RM>> > RM>> use 16k I get a handfull. If I dump the buffer in this case I
RM>> > see
RM>> > RM>> 0x200
RM>> > RM>> bytes filled with directory entries, then a lot of zeros and
RM>> > starting
RM>> > RM>> from
RM>> > RM>> 0x1000 again data. This is of course ignored because of the
RM>> > zeros
RM>> > RM>> before.
RM>> > RM>>

RM>> > RM>And for this case getdents(2) returned 16K? It is normal for

RM>> > getdents(2)
RM>> > RM>to return less than requested and when end of dir occurs, it
RM>> > should
RM>> > return 0.
RM>> > RM>

RM>> > RM>But if it returns 16K, there shouldn't be zeroed space in the

RM>> > middle of
RM>> > RM>it.
RM>> > RM>

RM>> > RM>And this always occurs or only after you wait a while? (You noted

RM>> > in the
RM>> > RM>above description that it would be ok for a little while after a
RM>> > directory
RM>> > RM>change and then would break, which suggests some kind of caching
RM>> > problem.)
RM>> >
RM>> > Today in the morning everything was fine. After waiting 5 minutes,
RM>> > again only partial directories. When I do a read with 8k buffer
RM>> > size,
RM>> > getdents(2) returns 8k, but starting from 0x200 until 0x1000 the
RM>> > buffer is filled with zeros. The entry just before the zeroes ends
RM>> > exactly at
RM>> > 0x200
RM>> > (that would be the first byte of the next entry) and at 0x1000 a new
RM>> > entry starts. The rest of the buffer is fine. The next read returns
RM>> > only 4k and seems to be fine - altough it contains some junk
RM>> > non-zero
RM>> > bytes in the padding.
RM>> >
RM>> Directory entries should never cross DIRBLKSIZ boundaries (512 or
RM>> 0x200), so it makes sense that one ends at 0x200 and one starts at
RM>> 0x1000. What doesn't make sense are the 0 bytes in between.
RM>>
RM>> One difference between the old and new NFS clients, which the patch I
RM>> sent you changed to the way the old one does it, is filling in the
RM>> last block.
RM>> The old NFS client just leaves the block short and depends on
RM>> n_direofoffset to recognize it is the last block with b_resid
RM>> indicating where it ends.
RM>> For the new client (unless you've applied the patch I emailed you), it
RM>> fills the rest of the last block in with "empty directories". This was
RM>> in the OpenBSD code when I did the original NFSv4 stuff and port. I
RM>> left it in, because I thought it might avoid problems if
RM>> n_direofoffset was ever bogus. That is why there might be "different
RM>> junk" at the end of the directory, but it shouldn't matter.
RM>>
RM>> It almost sounds like something else is bzero()ing out part of the
RM>> buffer cache block. Unless the directory has changed, the getdents()
RM>> after 5 minutes would just return the same buffer cache block that was
RM>> read in 5 minutes earlier (unless the buffer fell out of the cache and
RM>> had to be re-read from the server, which would only happen if there
RM>> was a lot of other file I/O going on during that 5minutes).
RM>>
RM>> A couple of comments:
RM>> - You can run "nfsstat -m" as root to see what the mount it actually
RM>> configured to use. This might be worth looking at, to see if any
RM>> of the values are "weird".
RM>> - One other difference between the old and new NFS clients is the
RM>> value of NFS_DIRBLKSIZ. For the new one, it is 8K instead of 4K.
RM>> You could change this in fs/nfs/nfsport.h, where is is defined
RM>> and then rebuild the sources to see if it has any effect. I can't
RM>> see why it should matter, but??
RM>> - Maybe you could post your system configuration. Someone might spot
RM>> something that changed in Feb.->Mar. related to your hardware/setup?
RM>>
RM>> > Ten minutes later again everything is fine. I tries to spy at the
RM>> > NFS
RM>> > communication with tcpdump, but it seems unwilling to display
RM>> > something useful about the NFS. Is it able to decode the readdir
RM>> > stuff?
RM>> >
RM>> To look at NFS packets you need wireshark. You can capture the packets
RM>> with tcpdump using the -w option. Something like:
RM>> # tcpdump -s 0 -w file.pcap host server
RM>> - Then look at file.pcap in wireshark. (Often more convenient than
RM>> installing wireshark on a particular machine.) If you'd like, you can
RM>> email me the file.pcap and I can look at it.
RM>>
RM>> rick
RM>>
RM>> > harti

RM>> >
RM>> > _______________________________________________
RM>> > freebsd...@freebsd.org mailing list
RM>> > http://lists.freebsd.org/mailman/listinfo/freebsd-current

RM>> > To unsubscribe, send any mail to
RM>> > "freebsd-curre...@freebsd.org"

RM>>
RM>> _______________________________________________
RM>> freebsd...@freebsd.org mailing list
RM>> http://lists.freebsd.org/mailman/listinfo/freebsd-current

RM>> To unsubscribe, send any mail to
RM>> "freebsd-curre...@freebsd.org"
RM>

[John Baldwin]

On Wednesday, May 15, 2013 4:38:38 am Hartmut Brandt wrote:
> On Wed, 15 May 2013, Rick Macklem wrote:
>
> RM>Well, getdents() basically just calls kern_getdirentries() and it calls
> RM>VOP_READDIR() { which is called nfs_readdir() in the NFS clients }.
> RM>nfs_readdir() calls ncl_bioread() to do the real work of finding the
> RM>buffer cache blocks and copying the data out of them.
> RM>One thing you might check via printf()s is whether the buffer cache
> RM>has the zero'd data in it before it copies it to userland.
>
> I now dump the data just before the call to vn_io_fault_iomove in
> ncl_bioread(). So what I do:
>
> 1. reboot
> 2. login
> 3. ls
> -> I see that it is moving 4 blocks 4k each to the user and they look
> fine
> 4. wait half an hour
> 5. ls
> -> now there is only one block, which contains zeros starting from
> 0x200.
>
> Note that I don't do anything else on that machine during that time.

To be clear, you are still seeing zero'd data in your printfs? Also, it seems
like it is passing the wrong size now? Can you output the various size fields
(b_resid, n, n_direofoffset, uio_offset, and uio_resid)?

--
John Baldwin

[Rick Macklem]

Thanks to Hartmut's testing, a patch to fix this problem has just
been committed to head as r251079. The problem was caused by
vnode_pager_setsize() being called for directories (where the
size reported by the server can be smaller than the size of the
ufs-like directory created in the client from the RPCs XDR).

r251079 will be MFC'd to stable/9 in 1 week if things go smoothly.

You might see this problem for head kernels between r248567-r251078
and stable/9 kernels from r249078 (Apr. 4) until a week from now.

Sorry for any inconvenience and thanks go to Hartmut for his help
isolating this, rick