"server not responding" / "is alive again" NFS tunables
Steve Shorter
I have a dedicated NFS server with 16 nfsd's running, connected
to SCSI raid/softupdates and good network connectivity/switching. Under
moderate or even sometimes light load the clients(7 of them) log messages
nfs server 192.168.10.2:/mnt: not responding
nfs server 192.168.10.2:/mnt: is alive again
several times per minute. They always have the same timestamp. Performance
is not noticably impaired, but I am wondering if this situation will eventually
become a performance barrier as the system ramps up to full production, if
the above log messages mean that packets must be delayed or retransmitted.
I have experimented with both udp/tcp mounts with various rw sizes
ranging from 8192 - 32768. tcp with rw=32768 was the worst case wrt above, and
am currently using udp mounts with default rw. Running 8 or 16 nfsd gives no
noticable difference wrt the above either.
I can find no issues on the server side, and was wondering if
there is a timeout threshold on the client side that is triggering the
messages/condition and whether adjusting the compile time tunables in
sys/nfs/nfs.h can aleviate the problem(if it even is a problem).
The server is 4.5-PRE, the clients are 4.4-RELEASE. I know
that there have been a log of NFS changes in 4.5, but testing 4.5 on
the clients is problematic. Do any of those changes affect the above?
So which variable(s) in nfs.h affect the client side and contribute
to the server timeout situation. Or are there any other suggestions?
Here are the variables in nfs.h(4.5-RC) for your amusement. Thanks
/*
* Tunable constants for nfs
*/
#define NFS_MAXIOVEC 34
#define NFS_TICKINTVL 5 /* Desired time for a tick (msec) */
#define NFS_HZ (hz / nfs_ticks) /* Ticks/sec */
#define NFS_TIMEO (1 * NFS_HZ) /* Default timeout = 1 second */
#define NFS_MINTIMEO (1 * NFS_HZ) /* Min timeout to use */
#define NFS_MAXTIMEO (60 * NFS_HZ) /* Max timeout to backoff to */
#define NFS_MINIDEMTIMEO (5 * NFS_HZ) /* Min timeout for non-idempotent ops*/
#define NFS_MAXREXMIT 100 /* Stop counting after this many */
#define NFS_MAXWINDOW 1024 /* Max number of outstanding requests */
#define NFS_RETRANS 10 /* Num of retrans for soft mounts */
#define NFS_MAXGRPS 16 /* Max. size of groups list */
#ifndef NFS_MINATTRTIMO
#define NFS_MINATTRTIMO 3 /* VREG attrib cache timeout in sec */
#endif
#ifndef NFS_MAXATTRTIMO
#define NFS_MAXATTRTIMO 60
#endif
#ifndef NFS_MINDIRATTRTIMO
#define NFS_MINDIRATTRTIMO 30 /* VDIR attrib cache timeout in sec */
#endif
#ifndef NFS_MAXDIRATTRTIMO
#define NFS_MAXDIRATTRTIMO 60
#endif
#define NFS_WSIZE 8192 /* Def. write data size <= 8192 */
#define NFS_RSIZE 8192 /* Def. read data size <= 8192 */
#define NFS_READDIRSIZE 8192 /* Def. readdir size */
#define NFS_DEFRAHEAD 1 /* Def. read ahead # blocks */
#define NFS_MAXRAHEAD 4 /* Max. read ahead # blocks */
#define NFS_MAXUIDHASH 64 /* Max. # of hashed uid entries/mp */
#define NFS_MAXASYNCDAEMON 20 /* Max. number async_daemons runnable */
#define NFS_MAXGATHERDELAY 100 /* Max. write gather delay (msec) */
#ifndef NFS_GATHERDELAY
#define NFS_GATHERDELAY 10 /* Default write gather delay (msec) */
#endif
#define NFS_DIRBLKSIZ 4096 /* Must be a multiple of DIRBLKSIZ */
#ifdef _KERNEL
#define DIRBLKSIZ 512 /* XXX we used to use ufs's DIRBLKSIZ */
#endif
To Unsubscribe: send mail to majo...@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Jan Schlesner
some weeks ago, the same problem was posted in this newsgroup. Here is
the anwser from Ian Dowse:
--
These are a side-effect of the operation of the NFS dynamic retransmit
timeout code. The NFS client measures the request response time for
various types of operations and it sets a timeout based on the mean
and deviation of observed times.
The time taken by the server to perform some operations can vary
wildly though, so occasionally when a large number of operations
complete with very little delay, the response time estimate and
hence the timeout become very small. Then when one request is
unusually slow to complete (such as when the disk on the server is
busy), the client thinks that the server isn't responding and prints
those warnings. A fraction of a second later the request completes
and the client prints a an 'is alive again' message.
On non-soft mounts these messages are completely harmless because
the client will just wait for the server to eventually reply. On
soft mounts, the effect can cause problems because applications
occasionally see an EINTR error.
The dynamic retransmit timeout code can be disabled with the `-d'
flag to mount_nfs; this is often recommended for fast networks that
see very little packet loss.
--
On Wed, Jan 16, 2002 at 10:12:12AM -0500, Steve Shorter wrote:
>
> I have a dedicated NFS server with 16 nfsd's running, connected
> to SCSI raid/softupdates and good network connectivity/switching. Under
> moderate or even sometimes light load the clients(7 of them) log messages
>
> nfs server 192.168.10.2:/mnt: not responding
> nfs server 192.168.10.2:/mnt: is alive again
>
> several times per minute. They always have the same timestamp. Performance
> is not noticably impaired, but I am wondering if this situation will eventually
> become a performance barrier as the system ramps up to full production, if
> the above log messages mean that packets must be delayed or retransmitted.
--
[ gpg key: http://wwwnlds.physik.tu-berlin.de/~schlesner/jschlesn.gpg ]
[ key fingerprint: 4236 3497 C4CF 4F3A 274F B6E2 C4F6 B639 1DF4 CF0A ]
--
It's better to reign in hell,
than to serve in heaven...
Matthew Dillon
It should be fairly harmless, but you can try mounting the
clients with the 'dumbtimer' mount option to get rid of the
dynamic retransmit estimator.
If you are interested in helping me fix the dynamic retransmit
estimator, send me an email sometime after we release 4.5
and I'll give you some simple patch sets to try. (We don't
have time to fix it prior to the release and I'm busy with
release stuff right now).
I would also be interested in a tcpdump of the NFS traffic on the
client side that catches it in the act (one should see a
retransmitted nfs request in the dump at the same time the
kernel logs the responding/alive-again warning).
-Matt
Matthew Dillon
<dil...@backplane.com>
:Howdy!