Why is lsof so slow?
Uncle Jeff
the latest package from sunfreeware, v4.49. Here's the output of
a 'uname -a':
SunOS solaris 5.8 Generic sun4u sparc SUNW,Ultra-5_10
Now, on out linux boxes & our hp-ux box, a command like the following:
$ lsof +d /tmp/wrk
Responds back in a manner of seconds. We use this to check whether a
particular process has finished so we can make sure the temp files are
cleaned up properly:
$ temp-proc
$ while $(lsof +d /tmp/wrk | grep "temp-proc"); do sleep 1; done
$ rm -f temp-proc*
$ [continue on...]
And this works A-OK on the other systems. On our sun box the lsof hangs
around. I once timed it at about 50 minutes before came back. Of course
by that time the temp-proc was _long_ finished, but the lsof kept
everything else waiting.
And yes, before you say RTFM, I have. There are sections in the FAQ
explaining about how slow lsof can be, especially with the +d option.
But, _this_ slow?
Is there anything anybody can suggest?
--
( >- THKS %^>
/~\ Uncle Jeff (get rid of JUNK to email)
| \) Jeff.Ho...@JUNKfederated.ca
\_|_ Linux, the choice of a GNU generation
--
( >- THKS %^>
/~\ Uncle Jeff (get rid of JUNK to email)
| \) Jeff.Ho...@JUNKfederated.ca
\_|_ Linux, the choice of a GNU generation
Sent via Deja.com
http://www.deja.com/
Casper H.S. Dik - Network Security Engineer
Uncle Jeff <jeff....@federated.ca> writes:
>Hi all. We're having a problem with the lsof utility. We've installed
>the latest package from sunfreeware, v4.49. Here's the output of
>a 'uname -a':
> SunOS solaris 5.8 Generic sun4u sparc SUNW,Ultra-5_10
>Now, on out linux boxes & our hp-ux box, a command like the following:
>$ lsof +d /tmp/wrk
Which version of lsof are you using?
And which Solaris revision?
You may need an updated version of lsof if you have any kernel
patch installed.
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
Vic Abell
>Hi all. We're having a problem with the lsof utility. We've installed
>the latest package from sunfreeware, v4.49. Here's the output of
>a 'uname -a':
> SunOS solaris 5.8 Generic sun4u sparc SUNW,Ultra-5_10
>Now, on out linux boxes & our hp-ux box, a command like the following:
>$ lsof +d /tmp/wrk
You must mean "+D". The "+d" option tag is an obscure form of "-d"
which is used to select a file descriptor number, a file descriptor
range, or a file descriptor name (e.g., "cwd" or "rtd"). Lsof has
so many options (sigh) that case counts.
If I read what I think you meant, you are asking lsof to report on
any open files in the directory /tmp/wrk, whose proper lsof request
syntax is "+D /tmp/wrk".
> ...
>And this works A-OK on the other systems. On our sun box the lsof hangs
>around. I once timed it at about 50 minutes before came back.
> ...
>And yes, before you say RTFM, I have. There are sections in the FAQ
>explaining about how slow lsof can be, especially with the +d option.
>But, _this_ slow?
RFTM would have been my first reaction, but 50 minutes is clearly
an unacceptable time.
>Is there anything anybody can suggest?
When you run lsof, does it warn you that it was built for a different
version of Solaris?
Regardless of that, the lsof revision you got from sunfreeware is
five revisions out of date. Follow Casper Dik's suggestion and
get the latest lsof revision, 4.54, from:
ftp://vic.cc.purdue.edu/pub/tools/unix/lsof
Build it yourself from sources on the system where you intend to
use it.
Truss output from `lsof +D /tmp/wrk` might help to reveal where
lsof is being delayed. My hunch is that it is somehow stalled
unstacking the /tmp/wrk directory tree. Lsof's +D option has to
lstat(2) all the branches in the named tree, cache the results
(device and node number), then examine all open files.
One faint possibility is that the "safe" lstat(2) calls that lsof
makes are all timing out. Because lstat(2) can block in the kernel
when the named file system is unavailable -- e.g., an unresponsive
NFS server -- lsof does them "safely" in a child process and places
a time limit on how long it will wait for the child process to
complete. Each time-out is 30 seconds. This normally shouldn't
happen on a subdirectory of /tmp, but /tmp/wrk could be a symbolic
link to an NFS file system directory, I suppose. And if there are
a huge number of files in /tmp/wrk, the "safe" lstat(2) method may
be causing the delay. (And yes, there are lsof options for modifying
the "safe" lstat(2) behavior.)
Another faint possibility is that /tmp/wrk has a symbolic link loop
in it. Lsof's internal function that resolves links, Readlink(),
wasn't detecting symbolic link loops correctly until it was fixed
in revision 4.54. (That's another reason for updating.)
If, after updating lsof, you still see problems with +D, contact
me for some debugging suggestions. I tried to replicate your
problem, but /tmp/wrk on my Solaris 8 test system probably doesn't
have the configuration that triggers your delay.
Vic Abell, lsof author
Mike Marshall
>>And this works A-OK on the other systems. On our sun box the lsof hangs
>>around. I once timed it at about 50 minutes before came back.
This may be a tangent:
I'm used to lsof with no arguments taking a l-o-n-g time to
run on a busy server.
netstat -a runs slow too, compared to netstat -n -a.
Is there (I've looked through the man page on a few versions) a
flag to keep lsof from resolving all those symbolic names? Seems
like that would zip it up quite a bit.
Uncle Jeff
To: jeff....@federated.ca
Cc:
Bcc: Jeff Howie <Jeff....@federated.ca>
Subject: Re: Why is lsof so slow?
Reply-To: Jeff Howie <Jeff....@mail.federated.ca>
In-Reply-To: <2001012514...@x67.deja.com>; from
jeff....@federated.ca on Thu, Jan 25, 2001 at 08:38:46AM
X-Priority: 3 (Normal)
> Subject: Re: Why is lsof so slow?
> From: a...@cc.purdue.edu (Vic Abell)
> Date: 2001/01/25
> Newsgroups: comp.unix.solaris
> Uncle Jeff <jeff....@federated.ca> writes (in part):
>
>>$ lsof +d /tmp/wrk
>
> You must mean "+D". The "+d" option tag is an obscure form of "-d"
> which is used to select a file descriptor number, a file descriptor
> range, or a file descriptor name (e.g., "cwd" or "rtd"). Lsof has
> so many options (sigh) that case counts.
No, I do mean '+d'. From your man page :&> -
+d s This option causes lsof to search for all open
instances of directory s and the files and direc
tories it contains at its top level. This option
does NOT descend the directory tree, rooted at s,
nor does it follow symbolic links within it. The
+D D option may be used to request a full-descent
directory tree search, rooted at directory D.
> If I read what I think you meant, you are asking lsof to report on
> any open files in the directory /tmp/wrk, whose proper lsof request
> syntax is "+D /tmp/wrk".
Which would recursively search sub-directories, which there aren't any
anyway in this structure, but still a waste.
> When you run lsof, does it warn you that it was built for a different
> version of Solaris?
No. Our's is a 64 bit arch & it's the proper build.
jeffh:<pts/30>$ lsof -v
lsof version information:
revision: 4.49 -- find the latest revision at:
ftp://vic.cc.purdue.edu/pub/tools/unix/lsof
configuration info: 64 bit kernel
constructed: Sun May 14 20:55:07 EDT 2000
constructed by and on: steve@solaris
compiler: /opt/SUNWspro/bin/cc
compiler version: WorkShop Compilers 5.0 98/12/15 C 5.0
compiler flags: -Dsolaris=80000 -DHASPR_GWINDOWS -xarch=v9 -DHASIPv6
-DHAS_VSOCK -DLSOF_VSTR="5.8" -O
loader flags: -L./lib -llsof -lkvm -lelf -lsocket -lnsl
system info: SunOS solaris 5.8 Generic sun4u sparc SUNW,Ultra-5_10
> Regardless of that, the lsof revision you got from sunfreeware is
> five revisions out of date. Follow Casper Dik's suggestion and
> get the latest lsof revision, 4.54, from:
>
> ftp://vic.cc.purdue.edu/pub/tools/unix/lsof
>
> Build it yourself from sources on the system where you intend to
> use it.
I got the source yesterday, haven't had a chance to build it yet,
though.
> Truss output from `lsof +D /tmp/wrk` might help to reveal where
> lsof is being delayed. My hunch is that it is somehow stalled
> unstacking the /tmp/wrk directory tree. Lsof's +D option has to
> lstat(2) all the branches in the named tree, cache the results
> (device and node number), then examine all open files.
>
> One faint possibility is that the "safe" lstat(2) calls that lsof
> makes are all timing out. Because lstat(2) can block in the kernel
> when the named file system is unavailable -- e.g., an unresponsive
> NFS server -- lsof does them "safely" in a child process and places
> a time limit on how long it will wait for the child process to
> complete. Each time-out is 30 seconds. This normally shouldn't
> happen on a subdirectory of /tmp, but /tmp/wrk could be a symbolic
> link to an NFS file system directory, I suppose. And if there are
> a huge number of files in /tmp/wrk, the "safe" lstat(2) method may
> be causing the delay. (And yes, there are lsof options for modifying
> the "safe" lstat(2) behavior.)
Sorry for the misleading directory structure. I just used /tmp as an
example because I didn't think it would make any difference. The
directory actually is '/fi/dv/out/polprint', in this particular case,
but will vary slightly depending on the sub-process running.
Interesting. Here's what I get when I try the '-d' option:
jeffh:<pts/30>$ lsof -b +d /fi/dv/out/polprint
lsof: avoiding readlink(/fi/dv/out/polprint): -b was specified.
lsof: avoiding stat(/fi/dv/out/polprint): -b was specified.
lsof: WARNING: can't stat(/fi/dv/out/polprint): Resource temporarily \
unavailable
> Another faint possibility is that /tmp/wrk has a symbolic link loop
> in it. Lsof's internal function that resolves links, Readlink(),
> wasn't detecting symbolic link loops correctly until it was fixed
> in revision 4.54. (That's another reason for updating.)
No symlinks in these directories. Doesn't mean I don't want to
upgrade, though.
> If, after updating lsof, you still see problems with +D, contact
> me for some debugging suggestions. I tried to replicate your
> problem, but /tmp/wrk on my Solaris 8 test system probably doesn't
> have the configuration that triggers your delay.
I'll try to build it today. I tried once, but got the cc/gcc compiler
error about not being able to build 64 bit. I'm going to get & build
the latest gcc snapshot, but this is getting a little over my head.
I'll always try _anything_ once! :-0
--
thks.jeff
Uncle Jeff
Uncle Jeff <jeff....@federated.ca> wrote:
>>>$ lsof +d /tmp/wrk
> I'll always try _anything_ once! :-0
Like I said...
Couldn't get the lastest snapshot of of gcc built on my first attempt,
either! Not my day... ;*)
But, our sysadmin (we're working together on this) made a system
change. He disabled nfs logging & rebooted the server. And guess what,
IT WORKS GOOD & FAST NOW!!
But this is not a permanent solution, although it does give us an idea
as to where the problem lies, right? I see no reference to this in the
docs, though.
--
(>- THKS %^>
/~\ Uncle Jeff (get rid of JUNK to email)
| \) Jeff.Ho...@JUNKfederated.ca
\_|_ Linux, the choice of a GNU generation
Casper H.S. Dik - Network Security Engineer
Uncle Jeff <jeff....@federated.ca> writes:
>But, our sysadmin (we're working together on this) made a system
>change. He disabled nfs logging & rebooted the server. And guess what,
>IT WORKS GOOD & FAST NOW!!
This could point to a problem in older versions of lsof that have to
do with the new negative caching DNLC.
So get a newer version of lsof.
(Old lsof will run fast if there are few negative cache entries but
it will crawl and take many, many minutes to complete when there
are many negative cache entires)
>But this is not a permanent solution, although it does give us an idea
>as to where the problem lies, right? I see no reference to this in the
>docs, though.
Yes, get a newer version of lsof.
Vic Abell
Lsof's -n option does the same thing netstat's -n does -- it estops
DNS IP->hostname lookups. Lsof's -P prevents port number to service
name lookups. Lsof's -l option stifles UID to login name lookups.
Usually, lsof needs to do few readlink() calls -- mostly on mount
points -- and those shouldn't normally take a lot of time. You
can suppress them with -b, but other bad things can happen as a
result. You can tell lsof to do them and all stat(2) calls by
specifying -O, but that can cause the kernel to block lsof if the
file is on a file system that is currently unavailable -- e.g., an
NFS file system whose server is not responding. Consequently, -O
is listed as a risky option.
I've profiled lsof extensivelly and used truss to see where lsof
is spending its time on Solaris and have found that most of its
time is spent in gathering process information via kvm_getproc().
There's not much I can do to speed up that operation, short of a
significant design in the way lsof functions. A redesign would
probably disable other features.
Vic Abell, lsof author
Vic Abell
>No, I do mean '+d'. From your man page :&> -
> +d s This option causes lsof to search for all open
Jeff is right. Lsof has so many options I forgot about this
variant.
I'm moving the rest of this discussion with Jeff to e-mail so
we don't waste any more c.u.s. bandwidth.
Vic
Vic Abell
>In article <94pp4e$pkf$1...@nnrp1.deja.com>,
> Uncle Jeff <jeff....@federated.ca> wrote:
>But, our sysadmin (we're working together on this) made a system
>change. He disabled nfs logging & rebooted the server. And guess what,
>IT WORKS GOOD & FAST NOW!!
Ahhh, this *is* an NFS file system. The directory probably has a
lot of files. Calling stat(2) on them is going to take time. I'm
not sure what NFS logging does, but it must add significant time
to the stat(2) operation.
>But this is not a permanent solution, although it does give us an idea
>as to where the problem lies, right? I see no reference to this in the
>docs, though.
This is not so much an lsof problem as it is a Solaris performance
consideration. It's probably worth mention in lsof's FAQ, but it
will probably be a recommendation to avoid +d or +D on NFS-mounted
directories that contain a lot of files.
Vic
Mike Marshall
>Vic Abell, lsof author
Cool, a celebrity! When lsof came out and sysadmins started being
able to easily find out what process was responsible for the
LISTENer on each port, it was a great day. Nice program. Good job!
-Mike
unix_...@my-deja.com
> >Vic Abell, lsof author
>
> Cool, a celebrity! When lsof came out and sysadmins started being
> able to easily find out what process was responsible for the
> LISTENer on each port, it was a great day. Nice program. Good job!
>
> -Mike
I'm counting 2 celebrities here: Casper and Vic, but I am under the
impression that Casper would get top billing!
ua
Vic Abell
>>Uncle Jeff <jeff....@federated.ca> writes (in part):
>>>And this works A-OK on the other systems. On our sun box the lsof hangs
>>>around. I once timed it at about 50 minutes before came back.
>This may be a tangent:
>I'm used to lsof with no arguments taking a l-o-n-g time to
>run on a busy server.
>netstat -a runs slow too, compared to netstat -n -a.
Casper Dik reminded me of another variable that affects the
performance of lsof on Solaris 8 -- examining the kernel DNLC for
path name components.
There is a Solaris 8 patch that implements a negative DNLC strategy,
often causing the DNLC to be much larger and to contain entries
for which lsof's DNLC hashing code function produces the same value.
That can make lsof very slow. I worked up a fix for the problem
when Casper first notice the slowness, and the fix has been available
since lsof revision 4.51.
Even if you have 4.51 or later, if you want to squeeze more
performance from lsof, try using its "-C" option. That will make
lsof ignore the kernel DNLC. Of course, it will also prevent lsof
from finding any path name components.
Vic Abell