fail to rm directory

[Luo, Ming [CAR:5K36:EXCH]]

Hi,

I have an empty directory can not be removed. Is there any way to diagnose
the problem? I can use fsck to recover. But I want to pin down the cause.
The problem itself is hard to reproduce.

# ls
testdir1
# rm -rf testdir1
rm: Unable to remove directory testdir1: File exists
# cd testdir1
# ls -al
total 4
drwxr-xr-x 3 root other 512 Jun 23 14:45 .
drwxr-xr-x 3 mycomp mycomp 512 Jun 23 14:18 ..

thanks
Ming

[Luo, Ming [CAR:5K36:EXCH]]

[all mail refused]

In article <cbfba0$29t$1...@zcars0v6.ca.nortel.com>, Luo, Ming
[CAR:5K36:EXCH] wrote:

You got 50% extra free on your link count.

--
Elvis Notargiacomo master AT barefaced DOT cheek
http://www.notatla.org.uk/goen/

[Sebastian Tomac]

Hi Ming,
this is probably du to a filesystme inconsistency. Do a umount and fsck on
the filesystem.
/Sebastian www.tomac.se

"Luo, Ming [CAR:5K36:EXCH]" <lu...@americasm01.nt.com> wrote in message
news:cbfba0$29t$1...@zcars0v6.ca.nortel.com...

[Dennis Glover]

"Luo, Ming [CAR:5K36:EXCH]" <lu...@americasm01.nt.com> wrote in message
news:cbfba0$29t$1...@zcars0v6.ca.nortel.com...

That link count of 3, especially for the "." entry is very suspicious.
I think it ought to be 2. If you can umount the file system and
run "fsck -y" against its partition, you might find that some reference
count has become corrupted, though I don't know how such a thing
could happen.

You might try this, before running fsck:
cd <parent-of-testdir1>
/bin/tar cvf /tmp/testdir1.tar testdir # look for any errors here!
/bin/tar xvf /tmp/testdir1.tar # should force overwrite
/bin/ls -al testdir1 # check if the reference count has changed

Good luck!

[Darren Dunham]

Ouch. Your directory has 3 links. Normal for an empty directory is
2 (one from the . in the directory, and one from 'testdir1' in the
parent directory).

Somehow you have a 3rd link. Either a subdirectory was removed
inconsistently (did the machine crash?) or an extra link was made.

Grab the inode of the directory and do a
find <filesystem> -inum <inode> -print

If the directory shows up twice, then a hard link to the directory was
somehow made.

If 'fsck' cleans it up, I'd assume a crash caused a corruption. 'fsck'
might not clean it up, either. You might have to clear the inode with
clri and then fsck the filesytem to clean up.

--
Darren Dunham ddu...@taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

[Andreas Borchert]

Is this directory mounted somewhere by NFS?

There exists following possible scenario:

- You export the filesystem containing testdir1 by NFS.
- You mount it on another host.
- You create subdirectories or files within testdir.
- You have processes on the system that mounted this filesystem
by NFS that open these files and/or chdir to the subdirectories
within testdir1.
- Now, if you ``rm -fr testdir1'', files and/or directories
named ``.nfs<something>'' will be created within ``testdir1''
to keep them alive for the processes on the other system.

This causes ``rm -fr testdir1'' to fail with the error message
above and this may also cause ``.'' to have a link count greater
than 2 (in case of subdirectories).

To check if this is your problem, just wait some time after the
``rm -fr testdir1'' and check whether ``.nfs<something>'' files
or directories reappear.

Andreas.

--
Dr. Andreas F. Borchert, SAI, Universitaet Ulm | Habe Mut, dich deines
Helmholtzstrasse 18, E02, Tel +49 731 50-23572 | eigenen Verstandes zu
http://www.mathematik.uni-ulm.de/sai/borchert/ | bedienen! -- I. Kant

[Juhan Leemet]

Has your machine been cracked? Is it the real "ls"? Is it lying? What a
cracker typically does is to replace common utilities, so as to hide the
stuff that they are installing of changing on your system.

You might consider getting and running chkrootkit. That will scan to see
if it recognizes any standard kind of attack on your machine. There is a
copy available on www.sunfreeware.com. I thought there would be one on
www.blastwave.org but I don't see it there? Hmm.

Others here might have other (better?) suggestions.

If your machine has been cracked, I believe the best approach is to:
immediately remove it from networks, rebuild it from scratch (reinstalling
O/S, etc.). You can't know for sure what has or has not been "altered".

I hope that isn't the problem. Happened to me once on a (slow) Linux box,
though. I was lucky. Caught the bastard in the act! (hey, what are these
ftp sessions here, I'm not... hey, wait a minute?!?) Fortunately, my
Solaris machines were on another network behind that machine, and now are
behind one of the Linksys firewall/routers. Good investment, BTW.

--
Juhan Leemet
Logicognosis, Inc.

[Juhan Leemet]

I didn't know that... but shouldn't "ls -al" show all files, including
hidden files, such as .nfs<something>?

Ah, (dim light gimmers?) maybe this depends on which machine (NFS server,
or (one of) NFS client) one is working? I had assumed (ref. Benny Hill)...

> This causes ``rm -fr testdir1'' to fail with the error message
> above and this may also cause ``.'' to have a link count greater
> than 2 (in case of subdirectories).
>
> To check if this is your problem, just wait some time after the
> ``rm -fr testdir1'' and check whether ``.nfs<something>'' files
> or directories reappear.
>
> Andreas.

--
Juhan Leemet
Logicognosis, Inc.

[Andreas Borchert]

Juhan Leemet wrote:
> On Fri, 25 Jun 2004 00:38:30 +0200, Andreas Borchert wrote:
>> - Now, if you ``rm -fr testdir1'', files and/or directories
>> named ``.nfs<something>'' will be created within ``testdir1''
>> to keep them alive for the processes on the other system.
>
> I didn't know that... but shouldn't "ls -al" show all files, including
> hidden files, such as .nfs<something>?

Yes. However, it takes some time until these ``.nfs<something>''
entries reappear. And, if you are invoking ``ls -al'' on an NFS
client, it might take even longer.

> Ah, (dim light gimmers?) maybe this depends on which machine (NFS server,
> or (one of) NFS client) one is working?

Indeed!

[p...@icke-reklam.ipsec.nu]

What OS ?

It looks like an additional hardlink to the directory.

Do ls -i to find out inode number and see what the other link goes to.

> thanks
> Ming

--
Peter Håkanson
IPSec Sverige ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

[Joerg Schilling]

In article <GNHCc.77011$kw3....@newssvr29.news.prodigy.com>,
Darren Dunham <ddu...@redwood.taos.com> wrote:

>> I have an empty directory can not be removed. Is there any way to diagnose
>> the problem? I can use fsck to recover. But I want to pin down the cause.
>> The problem itself is hard to reproduce.
>
>> # ls
>> testdir1
>> # rm -rf testdir1
>> rm: Unable to remove directory testdir1: File exists
>> # cd testdir1
>> # ls -al
>> total 4
>> drwxr-xr-x 3 root other 512 Jun 23 14:45 .
>> drwxr-xr-x 3 mycomp mycomp 512 Jun 23 14:18 ..
>
>Ouch. Your directory has 3 links. Normal for an empty directory is
>2 (one from the . in the directory, and one from 'testdir1' in the
>parent directory).

For UFS, you are correct.

For POSIX in general, you are not correct. An empty directory on a POSIX
compliant system may have a link count of 1 and the link count does not
need to change if you create subdirectories.

This is why e.g. GNU find by default behaves POSIXly incorrect as it makes
assumptions on the link count of directories.

--
EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
j...@cs.tu-berlin.de (uni) If you don't have iso-8859-1
schi...@fokus.fraunhofer.de (work) chars I am J"org Schilling
URL: http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

[Doug Freyburger]

Darren Dunham wrote:

> Luo, Ming wrote:
>
> > I have an empty directory can not be removed.

> > # ls
> > testdir1
> > # rm -rf testdir1
> > rm: Unable to remove directory testdir1: File exists
> > # cd testdir1
> > # ls -al
> > total 4
> > drwxr-xr-x 3 root other 512 Jun 23 14:45 .
> > drwxr-xr-x 3 mycomp mycomp 512 Jun 23 14:18 ..
>
> Ouch. Your directory has 3 links. Normal for an empty directory is
> 2 (one from the . in the directory, and one from 'testdir1' in the
> parent directory).
>
> Somehow you have a 3rd link. Either a subdirectory was removed
> inconsistently (did the machine crash?) or an extra link was made.

This is why many versions of Unix supply an "ln" command that
checks if the hard link it is making is to a directory and
refuses. Some versions have even refued root.

> Grab the inode of the directory and do a
> find <filesystem> -inum <inode> -print
>
> If the directory shows up twice, then a hard link to the directory was
> somehow made.

Then delete one of the two hard links with "unlink" and use
"rm -r" on the other.

> If 'fsck' cleans it up, I'd assume a crash caused a corruption. 'fsck'
> might not clean it up, either. You might have to clear the inode with
> clri and then fsck the filesytem to clean up.

Before I saw the hard link count of 3, I figured it could be a
mount point. It would not be able to delete a mount point because
mount points aren't subject to unlink. To acheive the count of 3
it would take mounting a filesystem over a directory that already
has a subdir, but I think it would still show as a link count of
2. Every once in a while I'll encounter asymetry problems where
the directory that receives the mount point has significantly
different ownerships and permissions than the root of the mounted
filesystem, but that tends to make permission-denied not file-exists
errors.

[p...@icke-reklam.ipsec.nu]

In comp.unix.admin Joerg Schilling <j...@cs.tu-berlin.de> wrote:
> In article <GNHCc.77011$kw3....@newssvr29.news.prodigy.com>,
> Darren Dunham <ddu...@redwood.taos.com> wrote:

>>> I have an empty directory can not be removed. Is there any way to diagnose
>>> the problem? I can use fsck to recover. But I want to pin down the cause.
>>> The problem itself is hard to reproduce.
>>
>>> # ls
>>> testdir1
>>> # rm -rf testdir1
>>> rm: Unable to remove directory testdir1: File exists
>>> # cd testdir1
>>> # ls -al
>>> total 4
>>> drwxr-xr-x 3 root other 512 Jun 23 14:45 .
>>> drwxr-xr-x 3 mycomp mycomp 512 Jun 23 14:18 ..
>>
>>Ouch. Your directory has 3 links. Normal for an empty directory is
>>2 (one from the . in the directory, and one from 'testdir1' in the
>>parent directory).

> For UFS, you are correct.

> For POSIX in general, you are not correct. An empty directory on a POSIX
> compliant system may have a link count of 1 and the link count does not
> need to change if you create subdirectories.

I assume that the writer's system does not use POSIX-FS. In fact noone does.

> This is why e.g. GNU find by default behaves POSIXly incorrect as it makes
> assumptions on the link count of directories.

Sometimes RMS aggrres with me.

[Casper H.S. Dik]

p...@icke-reklam.ipsec.nu writes:

>I assume that the writer's system does not use POSIX-FS. In fact noone does.

"hsfs" on Solaris does not have traditional link count semantics
for directories and GNU find fails on it.

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

[Richard L. Hamilton]

In article <cbh589$pjb$1...@news.cs.tu-berlin.de>,

j...@cs.tu-berlin.de (Joerg Schilling) writes:
> In article <GNHCc.77011$kw3....@newssvr29.news.prodigy.com>,
> Darren Dunham <ddu...@redwood.taos.com> wrote:
>
>>> I have an empty directory can not be removed. Is there any way to diagnose
>>> the problem? I can use fsck to recover. But I want to pin down the cause.
>>> The problem itself is hard to reproduce.
>>
>>> # ls
>>> testdir1
>>> # rm -rf testdir1
>>> rm: Unable to remove directory testdir1: File exists
>>> # cd testdir1
>>> # ls -al
>>> total 4
>>> drwxr-xr-x 3 root other 512 Jun 23 14:45 .
>>> drwxr-xr-x 3 mycomp mycomp 512 Jun 23 14:18 ..
>>
>>Ouch. Your directory has 3 links. Normal for an empty directory is
>>2 (one from the . in the directory, and one from 'testdir1' in the
>>parent directory).
>
> For UFS, you are correct.
>
> For POSIX in general, you are not correct. An empty directory on a POSIX
> compliant system may have a link count of 1 and the link count does not
> need to change if you create subdirectories.
>
> This is why e.g. GNU find by default behaves POSIXly incorrect as it makes
> assumptions on the link count of directories.
>

One can give it the peculiar "-noleaf" option to prevent that assumption.

Since one can get the source, one could also modify it to prevent that
assumption by default; I haven't checked, but there might even be a
configuration option to that effect.

--
mailto:rlh...@smart.net http://www.smart.net/~rlhamil

[all mail refused]

In article <ixkzn6o...@dev.null>, Tim wrote:

>It screws up in ClearCase's MVFS filesystem as well. It fails to
>descend into this dir's subdirectories. Note that the link count
>doesn't match the number of directories shown by ls -l:
>
>$ ls -d .
>drwxrwxr-x 5 tbutler users 4313 Nov 10 2003 .
>$ ls -l | grep ^d | wc -l
> 12
>
>(I hadn't chased down the cause of this until this thread reminded
>me of it. Using the suggested option "-noleaf" does cause it to
>descend into all subdirs.)

Do you also have directories hardlinked to their grandparents ?
I've never been able to tell whether this is the fault of Clearcase
or the people in my company who configured it.

This caused me to write a find-like replacement that remembers
pairs of (st_dev,st_ino) and won't revisit them.

[Michael Paoli]

Since you included comp.unix.solaris, I'll presume this is a Solaris
system. Most UNIX and UNIX-like operating systems do allow superuser
(root) to create multiple hard links to directories. This is generally
restricted to superuser and is generally not recommended, due to the
problems it can cause.

If you don't have a corrupted filesystem (many would view multiple hard
links to a directory as a type of corruption), it would seem you have
multiple hard links to the directory. You might start by doing
something like:
# ls -ldi testdir1
# find mount_point_of_filesystem -xdev -inum inum_of_testdir1 -print

Take a close look at the man pages for link and unlink.

Generally speaking, one should never unlink a non-empty directory.
If, however, a directory contains an ancestor directory, one would
typically need to use unlink to remove the ancestor from the descendant.

A rough example of that type of scenario might be something like this:
# mkdir loop
# link loop loop/loop
Correction would typically require:
# unlink loop/loop

In your case it seems likely the extra hard link is somewhere else,
e.g.:
# mkdir testdir1
# link testdir1 testdir2
In that case,
# unlink testdir2
would typically remove the extra hard link, dropping the link count on
testdir1, such that testdir1 could then be removed via normal means.

Random note: I noticed that LINUX appears to disallow even superuser
from creating multiple hard links to directories (at least via link)
even on other UNIX filesystem types which clearly support such, and are
also supported under LINUX.

And here's example test (with some comments) under Solaris:

# uname -rs
SunOS 5.9
# mkdir /tmp/foo && cd /tmp/foo
# mkdir testdir1 && ls -lid testdir1
468350682 drwxr-xr-x 2 root root 69 Jul 17 14:00 testdir1
# link testdir1 testdir2
# ls -lid * */. */.. | sort
468350630 drwxr-xr-x 3 root root 151 Jul 17 14:00 testdir1/..
468350630 drwxr-xr-x 3 root root 151 Jul 17 14:00 testdir2/..
468350682 drwxr-xr-x 3 root root 69 Jul 17 14:00 testdir1
468350682 drwxr-xr-x 3 root root 69 Jul 17 14:00 testdir1/.
468350682 drwxr-xr-x 3 root root 69 Jul 17 14:00 testdir2
468350682 drwxr-xr-x 3 root root 69 Jul 17 14:00 testdir2/.
# rm -rf *

rm: Unable to remove directory testdir1: File exists

rm: Unable to remove directory testdir2: File exists
# rmdir *
rmdir: directory "testdir1": Directory not empty
rmdir: directory "testdir2": Directory not empty
# unlink testdir2
# rmdir testdir1
# mkdir d && link d d/d
# rm -rf d
Segmentation Fault - core dumped
# rm d/core
# unlink d/d
unlink: Invalid argument
... I wasn't expecting that to fail like that
# truss unlink d/d
...
unlink("d/d") Err#22_EINVAL
...
# unlink d
... but that removes the directory ... not sure if it left the
filesystem clean though (most notably did it decrement the link count on
the directory to zero, or did it create an unreferenced directory?)
# mkdir -p a/c
# link a a/c/a
# unlink a/c/a
# rmdir a/c a

"Luo, Ming [CAR:5K36:EXCH]" <lu...@americasm01.nt.com> wrote in message news:<cbfba0$29t$1...@zcars0v6.ca.nortel.com>...

[Frank Batschulat]

Michael Paoli wrote:
[...]

> # unlink d
> ... but that removes the directory ... not sure if it left the
> filesystem clean though (most notably did it decrement the link count on
> the directory to zero, or did it create an unreferenced directory?)

calling unlink(2) on a directory will not decrement the link count
of the unlinked directories parent directory. You will be unable
to remove the parent directory afterwards using rmdir(1) or rm(1) -rf,
untill fsck(1M) does clear things up.

Thus calling unlink(2) on a directory is not recommended.

While this may look like a bug in the first place, it is the consequence
of the unlink(2) behavior defined by various standards we have to comply with.

---
frankB