When will os.remove fail?

[Steve D'Aprano]

On Linux, if I call os.remove on a file which I own but don't have write
permission on, the file is still deleted:


py> f = open('/tmp/no-write', 'w')
py> os.path.exists('/tmp/no-write')
True
py> os.chmod('/tmp/no-write', 0) # Forbid ALL access.
py> os.remove('/tmp/no-write')
py> os.path.exists('/tmp/no-write')
False


It seems that os.remove on Linux will force the delete even if the file is
read-only or unreadable, provided you own the file.

Does os.remove work like this under Windows too?

Under what circumstances will os.remove fail to remove a file?

If you don't own the file and have no write permission, if it is on
read-only media, anything else?


Thanks,



--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[Lele Gaifax]

Steve D'Aprano <steve+...@pearwood.info> writes:

> Under what circumstances will os.remove fail to remove a file?
>
> If you don't own the file and have no write permission, if it is on
> read-only media, anything else?

I would say that what matter is the permission on the directory containing the
file, not on the file itself.

ciao, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
le...@metapensiero.it | -- Fortunato Depero, 1929.

[Larry Martell]

On Sun, Mar 12, 2017 at 1:48 PM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
> On Linux, if I call os.remove on a file which I own but don't have write
> permission on, the file is still deleted:
>
>
> py> f = open('/tmp/no-write', 'w')
> py> os.path.exists('/tmp/no-write')
> True
> py> os.chmod('/tmp/no-write', 0) # Forbid ALL access.
> py> os.remove('/tmp/no-write')
> py> os.path.exists('/tmp/no-write')
> False
>
>
> It seems that os.remove on Linux will force the delete even if the file is
> read-only or unreadable, provided you own the file.

In Linux, I believe you need write permission on the dir to delete a
file - the perms on the file do not matter.

[Alain Ketterlin]

Steve D'Aprano <steve+...@pearwood.info> writes:

> On Linux, if I call os.remove on a file which I own but don't have write
> permission on, the file is still deleted:
>
>
> py> f = open('/tmp/no-write', 'w')
> py> os.path.exists('/tmp/no-write')
> True
> py> os.chmod('/tmp/no-write', 0) # Forbid ALL access.
> py> os.remove('/tmp/no-write')
> py> os.path.exists('/tmp/no-write')
> False
>
>
> It seems that os.remove on Linux will force the delete even if the file is
> read-only or unreadable, provided you own the file.

Your permissions on the file do not really matters. It's all about your
permissions on the parent directory (removing a file is really modifying
the parent dir). Actually, it is even slightly more complicated. Here is
an excerpt of the unlink(2) call (which does the job) listing common
error cases:

| EACCES Write access to the directory containing pathname is not allowed
| for the process's effective UID, or one of the directories in
| pathname did not allow search permission. (See also path_reso‐
| lution(7).)
|
| ELOOP Too many symbolic links were encountered in translating path‐
| name.
|
| EPERM (Linux only)
| The filesystem does not allow unlinking of files.
|
| EPERM or EACCES
| The directory containing pathname has the sticky bit (S_ISVTX)
| set and the process's effective UID is neither the UID of the
| file to be deleted nor that of the directory containing it, and
| the process is not privileged (Linux: does not have the
| CAP_FOWNER capability).
|
| EROFS pathname refers to a file on a read-only filesystem.

> Does os.remove work like this under Windows too?

No, Windows has its own set of rules:

https://msdn.microsoft.com/en-us/library/bb727008.aspx

> Under what circumstances will os.remove fail to remove a file?
>
> If you don't own the file and have no write permission, if it is on
> read-only media, anything else?

I'm not sure which system you are asking about, here. See above.

-- Alain.

[Steve D'Aprano]

On Mon, 13 Mar 2017 05:45 am, Alain Ketterlin wrote:

> Steve D'Aprano <steve+...@pearwood.info> writes:

[...]

>> It seems that os.remove on Linux will force the delete even if the file
>> is read-only or unreadable, provided you own the file.
>
> Your permissions on the file do not really matters. It's all about your
> permissions on the parent directory (removing a file is really modifying
> the parent dir). Actually, it is even slightly more complicated. Here is
> an excerpt of the unlink(2) call (which does the job) listing common
> error cases:

Thanks for the detailed explanation, and to everyone else who corrected my
misunderstanding.

[eryk sun]

On Sun, Mar 12, 2017 at 5:48 PM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
>
> Does os.remove work like this under Windows too?

os.remove calls DeleteFile on Windows. This in turn calls NtOpenFile
to instantiate a kernel File object that has delete access and return
a handle to it. Next it calls NtSetInformationFile on the handle to
set the file's "DeleteFile" disposition. Either of these operations
can fail.

One hurdle to getting delete access is the sharing mode. If there are
existing File objects that reference the file, they all have to share
delete access. Otherwise the open fails with a sharing violation. This
is often the show stopper because the C runtime (and thus CPython,
usually) opens files with read and write sharing but not delete
sharing.

Another hurdle is paging files, but a program really has no business
trying to delete a critical system file. You'll get a sharing
violation if a file is currently mapped as one of the system's paging
files. Windows supports up to 16 paging files, each on a separate
volume, so it's not like this is a common problem.

Otherwise, getting delete access depends on the file system. If it
doesn't implement security (e.g. FAT32), then delete access is always
granted. If it does implement security (e.g. NTFS), then first I think
we need to discuss the basics of NT security.

----

Secured objects have a security descriptor that contains the security
identifier (SID) of the owner and (optionally) a primary group (not
used by Windows). It also may contain a discretionary access control
list (DACL) and a system access control list (SACL). Each of these
contains a list of access control entries (ACEs) that either allow or
deny the specified access mask for a given user/group SID
(canonically, deny ACEs should be sorted before allow ACEs). For use
with container objects, such as file-system directories, an ACE also
has the following flags to control inheritance: object inherit,
container inherit, no propagate inherit (i.e. clear the
object/container inherit flags on inherited ACEs), and inherent only
(i.e. don't apply this ACE to the object itself).

An access mask is a 32-bit value, with one right mapped to each bit.
The lower half defines up to 16 rights that are specific to an object
type, such as File, Process, or Thread. The next 8 bits are standard
rights (e.g. delete) that are defined the same for all objects. The
upper 4 bits are for generic read (GR), write (GW), execute (GE), and
all (GA) access. In an access check, generic rights should be
translated to standard and specific rights according to a generic
mapping that's defined for each type.

Each process has a primary token, and each thread in a process
optionally can have an impersonation token. An access token (and the
associated logon session) is usually created by the local security
authority (the lsass.exe process), which is coupled with the security
monitor in the kernel. A token contains a list of user/group SIDs,
which are used in an access check if they're flagged as either enabled
or use-for-deny-only (i.e. only for deny ACEs). A token also has a
list of privileges and their enable state. A token is itself a secured
object; it's an instance of the Token type.

A token is assigned an integrity level. The term "elevating" refers in
part to starting a process with a token that has a higher integrity
level, for which the available levels are low, medium (default), high,
and system. A secured object's SACL can contain a mandatory label ACE,
which sets its integrity level. The mask field of this ACE isn't a
32-bit access mask, but rather it consists of up to 3 flag values --
no-write-up, no-read-up, and no-execute-up -- that determine whether a
token at a lower integrity level is allowed write/delete, read, or
execute access. If a secured object doesn't have a mandatory label
ACE, then it implicitly has a medium integrity level with no-write-up.

----

OK, that covers the basics.

If the current token contains the privilege SeRestorePrivilege (e.g. a
token for an elevated administrator), and it's enabled, then delete
access will always be granted regardless of the file's security
descriptor. The restore privilege is quite empowering, so enable it
with caution.

A file's DACL will either allow or explicitly deny delete access. If
access isn't explicitly allowed, then it's implicitly denied. However,
delete access will still be granted if the parent directory grants
delete-child access. The delete-child right is specific to the File
type -- specifically for directories.

If the file's SACL mandatory access includes no-write-up, then a user
at a lower integrity level will not be able to open the file for
delete access, even if the file's DACL (discretionary access)
otherwise allows it. However, delete access is still granted if the
parent directory grants delete-child access to the user. Thus
parent-level discretionary access trumps object-level mandatory
access.

The next step is setting the file's "DeleteFile" disposition. First
let's take a close look at how this works.

----

In user mode, a kernel object such as a File instance is referenced as
a handle. Duplicating the handle creates a new reference to the
object. However, if you open the file/device again, you'll get a
handle for a new object. All File objects that reference the same data
file will share a file or link control block (FCB) structure that,
among other things, is used to track sharing of read, write, and
delete access and the file's delete disposition.

When the delete disposition is set, no new File references can be
instantiated (i.e. any level of access is denied, even just to read
the file attributes), but the file isn't immediately unlinked. It's
still listed in the parent directory. Any existing File reference that
has delete access can unset the delete disposition.

When all handle and pointer references to a File object are closed,
the file system decrements the reference count on the FCB. When the
FCB reference count drops to 0, if the delete disposition is currently
set, then finally the file is unlinked from the parent directory.

----

I can think of a couple of cases in which setting the delete
disposition will fail with access denied.

If the file is currently memory-mapped as code or data (e.g. an EXE or
DLL), but not for system paging, then you can open it for delete
access, but only renaming the file will succeed (it can be renamed
anywhere on the volume, but not to another volume). Setting the delete
disposition will fail with access denied. The system won't allow a
mapped file to be unlinked.

Finally, I'm sure most people are familiar with the read-only file
attribute. If this attribute is set you can still open a file with
delete access to rename it, but setting the delete disposition will
fail with access denied.

You may complain that the system has granted the user delete access,
so getting an access denied error is a breach of contract. You'd be
right, but the kernel actually returns STATUS_CANNOT_DELETE. The
Windows API maps this to ERROR_ACCESS_DENIED.

STATUS_CANNOT_DELETE = 0xC0000121
ERROR_ACCESS_DENIED = 0x0005

>>> ntdll.RtlNtStatusToDosError(STATUS_CANNOT_DELETE)
5

It's often the case that useful information is lost when mapping a
kernel status code to a Windows error code. This is an inherent
problem with layered APIs. Even more information is lost when the C
runtime maps the Windows error to a POSIX errno value such as EACCES.

[Marko Rauhamaa]

eryk sun <ery...@gmail.com>:

> On Sun, Mar 12, 2017 at 5:48 PM, Steve D'Aprano
> <steve+...@pearwood.info> wrote:
>>
>> Does os.remove work like this under Windows too?
>

> os.remove calls DeleteFile on Windows. [...]

Fascinating info, Eryk.

The difference between file removal in Linux and Windows is a bit like
the difference between object destrution in C++ and Python. In C++, you
destroy an object with a "delete" statement. In Python, you can't
destroy an object, you can simply lose all references to it.


Marko

[Steve D'Aprano]

On Mon, 13 Mar 2017 08:47 pm, eryk sun wrote:

> On Sun, Mar 12, 2017 at 5:48 PM, Steve D'Aprano
> <steve+...@pearwood.info> wrote:
>>
>> Does os.remove work like this under Windows too?
>
> os.remove calls DeleteFile on Windows.

[...]

Thanks for the unexpectedly detailed explanation! A few follow-up questions:

> One hurdle to getting delete access is the sharing mode. If there are
> existing File objects that reference the file, they all have to share
> delete access. Otherwise the open fails with a sharing violation. This
> is often the show stopper because the C runtime (and thus CPython,
> usually) opens files with read and write sharing but not delete
> sharing.

This is the famous "can't delete a file which is open under Windows"
problem, am I right?

I take it that you *can* delete open files, but only if the process that
opens them takes special care to use "delete sharing". Is that correct?

I don't have a machine to test this on, but I'd like to deal with this
situation in my (cross-platform) code. If I have one Python script do this:


with open("My Documents/foo") as f:
time.sleep(100000)


and while it is sleeping another script does this:

os.remove("My Documents/foo")


what exception will I get? Is that unique to this situation, or is a generic
exception that could mean anything?

My aim is to do:

try:
os.remove(thefile)
except SomeError:
# Could be a virus checker or other transient process.
time.sleep(0.2)
os.remove(thefile) # try again


Does that seem reasonable to you, as a Windows user?


[...]

> OK, that covers the basics.

The basics, he says :-)

> In user mode, a kernel object such as a File instance is referenced as
> a handle.

Out of curiosity, I only know the term "handle" from classic Macintosh
(pre-OS X) where a handle was a managed pointer to a pointer to a chunk of
memory. Being managed, the OS could move the memory around without the
handles ending up pointing to garbage. Is that the same meaning in Windows
land?


[...]

> When the delete disposition is set, no new File references can be
> instantiated (i.e. any level of access is denied, even just to read
> the file attributes), but the file isn't immediately unlinked. It's
> still listed in the parent directory. Any existing File reference that
> has delete access can unset the delete disposition.
>
> When all handle and pointer references to a File object are closed,
> the file system decrements the reference count on the FCB. When the
> FCB reference count drops to 0, if the delete disposition is currently
> set, then finally the file is unlinked from the parent directory.

So technically file deletions aren't atomic in Windows? In principle, I
could say:

delete file X

which then returns immediately, and if I try to open(X) it will fail. But I
can still see it if I do a dir() on the parent directory?

Eventually the last reference to X will go away, and then it is unlinked.
What happens if I pull the plug in the meantime? Will the file magically
come back on rebooting?


[...]

> Finally, I'm sure most people are familiar with the read-only file
> attribute. If this attribute is set you can still open a file with
> delete access to rename it, but setting the delete disposition will
> fail with access denied.

That was actually the situation I was thinking about when I started on this
question. From Python code, I was considering writing something like this:


def delete_readonly(thefile):
try:
os.remove(thefile)
except ReadOnlyFileError: # what is this really?
try:
# change the user permissions to unset Read-Only
old_perms = ... ?
set_perms(...)
except OSError:
pass
else:
# We changed Read-Only. Try deleting again. If it fails,
# revert the flags and fail.
try:
os.remove(thefile)
except OSError:
# Restore permissions
...
else:
# Success on the second attempt!
return
raise



That's intended to emulate the behaviour on Unix where deleting a file that
you own will succeed even if you don't have write access to the file.

(The bash rm command will ask you before deleting, but Python's os.remove
just removes it.)


Does this seem reasonable? Or overly complicated for not enough benefit?

[Chris Angelico]

On Tue, Mar 14, 2017 at 10:32 PM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
> On Mon, 13 Mar 2017 08:47 pm, eryk sun wrote:
>> One hurdle to getting delete access is the sharing mode. If there are
>> existing File objects that reference the file, they all have to share
>> delete access. Otherwise the open fails with a sharing violation. This
>> is often the show stopper because the C runtime (and thus CPython,
>> usually) opens files with read and write sharing but not delete
>> sharing.
>
> This is the famous "can't delete a file which is open under Windows"
> problem, am I right?
>
> I take it that you *can* delete open files, but only if the process that
> opens them takes special care to use "delete sharing". Is that correct?

Yes, but you can't always control the process that opens them. For
example, it's annoyingly difficult to update a running executable.

>> In user mode, a kernel object such as a File instance is referenced as
>> a handle.
>
> Out of curiosity, I only know the term "handle" from classic Macintosh
> (pre-OS X) where a handle was a managed pointer to a pointer to a chunk of
> memory. Being managed, the OS could move the memory around without the
> handles ending up pointing to garbage. Is that the same meaning in Windows
> land?

The old Mac meaning was very concrete and specific: it's a pointer to
a pointer. Since all *you* ever see is the first pointer, the OS is
free to (atomically) move the destination.

More generally, a handle is just an opaque cookie that represents a
thing. If you do cross-platform socket programming, you'll find that
on Unix, creating a socket returns a "file descriptor"; but on
Windows, it returns a "socket handle". Aside from the fact that socket
handles can only be used with socket functions (making them harder to
pass to other processes etc), the two concepts work broadly the same
way: you have a small integer value that represents a large resource.

>> When the delete disposition is set, no new File references can be
>> instantiated (i.e. any level of access is denied, even just to read
>> the file attributes), but the file isn't immediately unlinked. It's
>> still listed in the parent directory. Any existing File reference that
>> has delete access can unset the delete disposition.
>>
>> When all handle and pointer references to a File object are closed,
>> the file system decrements the reference count on the FCB. When the
>> FCB reference count drops to 0, if the delete disposition is currently
>> set, then finally the file is unlinked from the parent directory.
>
> So technically file deletions aren't atomic in Windows? In principle, I
> could say:
>
> delete file X
>
> which then returns immediately, and if I try to open(X) it will fail. But I
> can still see it if I do a dir() on the parent directory?
>
> Eventually the last reference to X will go away, and then it is unlinked.
> What happens if I pull the plug in the meantime? Will the file magically
> come back on rebooting?

Without actually testing it, my answers (based on experience of using
Windows) would be that the file doesn't disappear at all until it's
actually deleted. The "no new File refs can be instantiated" would
mean that it kicks back an error, not that it looks like the file
doesn't exist. If you pull the plug, it won't "magically come back" -
it'll have never gone.

> That's intended to emulate the behaviour on Unix where deleting a file that
> you own will succeed even if you don't have write access to the file.
>
> (The bash rm command will ask you before deleting, but Python's os.remove
> just removes it.)

(And the rm command won't ask if you say "-f".)

> Does this seem reasonable? Or overly complicated for not enough benefit?

I'm terrified to think what the interactions would be between this and
the Ubuntu subsystem in Win 10. What're the semantics of unlinkat()?
What about the common practice of creating a file and then unlinking
it so it doesn't show up in the directory? (And is that different on a
file system that supports hard links?) Or are heaps of POSIX features
simply not available?

ChrisA

[Jon Ribbens]

On 2017-03-14, Chris Angelico <ros...@gmail.com> wrote:
>> (The bash rm command will ask you before deleting, but Python's os.remove
>> just removes it.)
>
> (And the rm command won't ask if you say "-f".)

rm does not ask before deleting. However some Linux distributions
take it upon themselves to put "alias rm='rm -i'" in /etc/profile.

[Chris Angelico]

I have no such alias, but it still prompts.

'man rm':

If the -I or --interactive=once option is given, and there are more
than three files or the -r, -R, or --recursive are given, then rm
prompts the user for whether to proceed with the entire operation. If
the response is not affirmative, the entire command is aborted.

Otherwise, if a file is unwritable, standard input is a terminal, and
the -f or --force option is not given, or the -i or --interac‐
tive=always option is given, rm prompts the user for whether to remove
the file. If the response is not affirmative, the file is skipped.

This is the GNU coreutils rm command. Obviously behaviour may be
different with non-GNU rm's.

ChrisA

[Frank Millman]

"Chris Angelico" wrote in message
news:CAPTjJmrim9TJHfOLgynUJATx...@mail.gmail.com...

On Wed, Mar 15, 2017 at 12:30 AM, Jon Ribbens <jon+u...@unequivocal.eu>
wrote:
>>>

>>> (And the rm command won't ask if you say "-f".)
>>
>> rm does not ask before deleting. However some Linux distributions
>> take it upon themselves to put "alias rm='rm -i'" in /etc/profile.
>

> I have no such alias, but it still prompts.
>

On Fedora 22 (and for many previous versions) I have noticed that, if I log
in as 'root', it does prompt, but if I log in as an ordinary user, it does
not.

If I type 'alias' at the console, it lists current aliases. 'root' shows
exactly what Jon quoted above. 'frank' shows no alias for 'rm'.

I had a quick look to see what was setting it, but there is nothing in
/etc/profile or in /etc/bashrc. I don't know where else to check.

Frank Millman

[Chris Angelico]

On Wed, Mar 15, 2017 at 1:22 AM, Frank Millman <fr...@chagford.com> wrote:
> "Chris Angelico" wrote in message
> news:CAPTjJmrim9TJHfOLgynUJATx...@mail.gmail.com...
>
> On Wed, Mar 15, 2017 at 12:30 AM, Jon Ribbens <jon+u...@unequivocal.eu>
> wrote:
>>>>
>>>>

>>>> (And the rm command won't ask if you say "-f".)
>>>
>>>
>>> rm does not ask before deleting. However some Linux distributions
>>> take it upon themselves to put "alias rm='rm -i'" in /etc/profile.
>>
>>

>> I have no such alias, but it still prompts.
>>
>
> On Fedora 22 (and for many previous versions) I have noticed that, if I log
> in as 'root', it does prompt, but if I log in as an ordinary user, it does
> not.
>
> If I type 'alias' at the console, it lists current aliases. 'root' shows
> exactly what Jon quoted above. 'frank' shows no alias for 'rm'.
>
> I had a quick look to see what was setting it, but there is nothing in
> /etc/profile or in /etc/bashrc. I don't know where else to check.

Since root has the prompt, check /root/.profile and /root/.bashrc.
There may also be other files sourced from either of those, eg
/root/.bash_aliases.

ChrisA

[Frank Millman]

"Frank Millman" wrote in message news:oa8uaf$k9e$1...@blaine.gmane.org...

> On Fedora 22 (and for many previous versions) I have noticed that, if I
> log
in as 'root', it does prompt, but if I log in as an ordinary user, it does
not.

> If I type 'alias' at the console, it lists current aliases. 'root' shows
exactly what Jon quoted above. 'frank' shows no alias for 'rm'.

> I had a quick look to see what was setting it, but there is nothing in
/etc/profile or in /etc/bashrc. I don't know where else to check.

I should have looked a bit harder. It is in /root/.bashrc -

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

Frank

[Jon Ribbens]

On 2017-03-14, Chris Angelico <ros...@gmail.com> wrote:
> On Wed, Mar 15, 2017 at 12:30 AM, Jon Ribbens <jon+u...@unequivocal.eu> wrote:
>> rm does not ask before deleting. However some Linux distributions
>> take it upon themselves to put "alias rm='rm -i'" in /etc/profile.
>
> I have no such alias, but it still prompts.

I'm think you must be mistaken. What do you see if you type
'alias rm' and 'which rm'? Are you perhaps trying to delete
files to which you do not have write permission?

> 'man rm':
>
> If the -I or --interactive=once option is given, and there are more
> than three files or the -r, -R, or --recursive are given, then rm
> prompts the user for whether to proceed with the entire operation. If
> the response is not affirmative, the entire command is aborted.
>
> Otherwise, if a file is unwritable, standard input is a terminal, and
> the -f or --force option is not given, or the -i or --interac‐
> tive=always option is given, rm prompts the user for whether to remove
> the file. If the response is not affirmative, the file is skipped.

Yes, this describes the behaviour if you specify -I or -i,
as I mentioned - not if you don't specify either of those options.

> This is the GNU coreutils rm command.

I am talking about the GNU coreutils rm command (and POSIX for that
matter, since the behaviour of GNU rm appears to be POSIX compliant
as far as what we are talking about goes).

[Jon Ribbens]

On 2017-03-14, Frank Millman <fr...@chagford.com> wrote:
> If I type 'alias' at the console, it lists current aliases. 'root' shows
> exactly what Jon quoted above. 'frank' shows no alias for 'rm'.
>
> I had a quick look to see what was setting it, but there is nothing in
> /etc/profile or in /etc/bashrc. I don't know where else to check.

Perhaps one or more of:

/root/.profile
/root/.bash_profile
/root/.bashrc
/etc/profile.d/*

[Chris Angelico]

On Wed, Mar 15, 2017 at 1:30 AM, Jon Ribbens <jon+u...@unequivocal.eu> wrote:
> On 2017-03-14, Chris Angelico <ros...@gmail.com> wrote:
>> On Wed, Mar 15, 2017 at 12:30 AM, Jon Ribbens <jon+u...@unequivocal.eu> wrote:
>>> rm does not ask before deleting. However some Linux distributions
>>> take it upon themselves to put "alias rm='rm -i'" in /etc/profile.
>>
>> I have no such alias, but it still prompts.
>
> I'm think you must be mistaken. What do you see if you type
> 'alias rm' and 'which rm'? Are you perhaps trying to delete
> files to which you do not have write permission?

This conversation was specifically about files for which one does not
have write permission. So yes. (It's pretty common; .git/objects is
mostly or entirely made up of such files.)

ChrisA

[Lele Gaifax]

Jon Ribbens <jon+u...@unequivocal.eu> writes:

>> Otherwise, if a file is unwritable, standard input is a terminal, and
>> the -f or --force option is not given, or the -i or --interac‐
>> tive=always option is given, rm prompts the user for whether to remove
>> the file. If the response is not affirmative, the file is skipped.
>
> Yes, this describes the behaviour if you specify -I or -i,
> as I mentioned - not if you don't specify either of those options.

English is not my native language, but that's not how I understand that
paragraph: if -i is given, it always ask, regardless the writable bit,
otherwise it does when f is readonly and no -f is given.

I think Chris is right, consider:

$ rm --version
rm (GNU coreutils) 8.26

$ which rm
/bin/rm

$ alias | grep rm | wc -l
0

$ ls -l *file*
-rw-rw-r-- 1 lele lele 0 Mar 14 15:52 myfile
-r-------- 1 lele lele 0 Mar 14 15:52 myfile2
-rw-r--r-- 1 root root 0 Mar 14 15:52 othersfile

$ rm myfile

$ rm myfile2
rm: remove write-protected regular empty file 'myfile2'? y

$ rm othersfile
rm: remove write-protected regular empty file 'othersfile'? y

$ ls -l *file*
ls: cannot access '*file*': No such file or directory

my 0.02€

[Jon Ribbens]

On 2017-03-14, Lele Gaifax <le...@metapensiero.it> wrote:
> Jon Ribbens <jon+u...@unequivocal.eu> writes:
>>> Otherwise, if a file is unwritable, standard input is a terminal, and
>>> the -f or --force option is not given, or the -i or --interac‐
>>> tive=always option is given, rm prompts the user for whether to remove
>>> the file. If the response is not affirmative, the file is skipped.
>>
>> Yes, this describes the behaviour if you specify -I or -i,
>> as I mentioned - not if you don't specify either of those options.
>
> English is not my native language, but that's not how I understand that
> paragraph: if -i is given, it always ask, regardless the writable bit,
> otherwise it does when f is readonly and no -f is given.

You're understanding that paragraph correctly, but are
misunderstanding what I wrote - I didn't disagree with
what you're saying.

[eryk sun]

On Tue, Mar 14, 2017 at 11:32 AM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
> On Mon, 13 Mar 2017 08:47 pm, eryk sun wrote:
>
>> One hurdle to getting delete access is the sharing mode. If there are
>> existing File objects that reference the file, they all have to share
>> delete access. Otherwise the open fails with a sharing violation. This
>> is often the show stopper because the C runtime (and thus CPython,
>> usually) opens files with read and write sharing but not delete
>> sharing.
>
> This is the famous "can't delete a file which is open under Windows"
> problem, am I right?

If you aren't allowed shared delete access, the error you'll get is a
sharing violation (32). If the file security doesn't allow delete
access, then the error is access denied (5). In Python both of these
raise a PermissionError, so telling the difference requires checking
the winerror attribute.

Two more cases are (1) a read-only file and (2) a file that's
memory-mapped as code or data. In these two cases you can open the
file with delete access, which allows you to rename it, but setting
the delete disposition fails with access denied.

If you have the right to set the file's attributes, then you can at
least work around the read-only problem via os.chmod(filename,
stat.S_IWRITE).

For the mapped file case the best you can do is rename the file to
another directory. At least it gets it out of the way if you're doing
an upgrade to a running program. This workaround is rarely used -- I
think mostly due to people not knowing that it's possible.

> I take it that you *can* delete open files, but only if the process that
> opens them takes special care to use "delete sharing". Is that correct?

Unix programmers can't simply use delete sharing as a way to get
familiar semantics on Windows. A file pending delete is in limbo on
Windows. It can't be opened again, and as long as there are File
objects that reference the common file control block (FCB), it can't
be unlinked. One of those File objects may even be used to restore the
file (i.e. unset the delete disposition) if it has delete access.
Notably, this limbo file prevents the containing directory from being
deleted.

> I don't have a machine to test this on, but I'd like to deal with this
> situation in my (cross-platform) code. If I have one Python script do this:
>
> with open("My Documents/foo") as f:
> time.sleep(100000)
>
> and while it is sleeping another script does this:
>
> os.remove("My Documents/foo")
>
> what exception will I get? Is that unique to this situation, or is a generic
> exception that could mean anything?

This will fail with a sharing violation, i.e. winerror == 32.

> My aim is to do:
>
> try:
> os.remove(thefile)
> except SomeError:
> # Could be a virus checker or other transient process.
> time.sleep(0.2)
> os.remove(thefile) # try again
>
> Does that seem reasonable to you, as a Windows user?

If you get an access-denied error, then you can try to remove the
read-only attribute if it's set. Otherwise you can try to rename the
file to get it out of the way. But if you're locked out by a sharing
violation, there isn't anything you can do short of terminating the
offending program.

Virus scanners and other filter drivers generally aren't an immediate
problem with deleting. They share all access. But they might keep a
file from being immediately unlinked, which could cause problems in
functions like shutil.rmtree. Removing the parent directory can fail
if the file hasn't been unlinked yet.

>> In user mode, a kernel object such as a File instance is referenced as
>> a handle.
>
> Out of curiosity, I only know the term "handle" from classic Macintosh
> (pre-OS X) where a handle was a managed pointer to a pointer to a chunk of
> memory. Being managed, the OS could move the memory around without the
> handles ending up pointing to garbage. Is that the same meaning in Windows
> land?

That sounds similar to an HGLOBAL handle used with
GlobalAlloc/GlobalLock. This was a feature from 16-bit Windows, and
it's kept around for compatibility with older APIs that still use it.

I was talking about handles for kernel objects. Every process has a
table of handle entries that's used to refer to kernel objects. These
objects are allocated in the shared kernel space (the upper range of
virtual memory), so user-mode code can't refer to them directly.
Instead a program passes a handle to a system call, and kernel code
and drivers use the object manager to look up the pointer reference.

A kernel object has an associated type object (e.g. the "File" type),
and there's a "Type" metatype like in Python. The list of supported
methods is pretty basic:

DumpProcedure, OpenProcedure, ParseProcedure,
SecurityProcedure, QueryNameProcedure, OkayToCloseProcedure,
CloseProcedure, DeleteProcedure

The close method is called when a process closes a handle to the
object. It gets passed a reference to the Process that's closing the
handle and also the object's handle count, both in the process and
across all processes, to allow for whatever cleanup is required when
the last handle is closed both in the process and in the system. The
delete method is called when the object is no longer referenced, which
is for pointer references, but that implicitly includes handle
references as well.

For the File type, these methods reference the associated Device and
call into the device stack with an I/O request packet (IRP). For
closing a handle the major function is IRP_MJ_CLEANUP. For deleting
the object the major function is IRP_MJ_CLOSE. The Driver object [1]
has a MajorFunction table of function pointers for dispatching IRPs by
major function code.

If a File object [2] refers to a file-system file/directory (as
opposed to a device), then the file-system context is the shared file
control block (FCB) and usually a private context control block (CCB),
which are respectively the object members FsContext and FsContext2. If
the FCB is flagged as delete-on-close, then when the last reference is
cleaned up, the file system does the work of unlinking and whatever
else it has to do to really delete a file. If the CCB is flagged as
delete-on-close (e.g. CreateFile was called with
FILE_FLAG_DELETE_ON_CLOSE), then when the File object is cleaned up it
transfers this flag to the shared FCB.

If you want to see this in practice, look at the published cleanup
code for the fastfat driver [3].

[1]: https://msdn.microsoft.com/en-us/library/ff544174
[2]: https://msdn.microsoft.com/en-us/library/ff545834
[3]: https://github.com/Microsoft/Windows-driver-samples/blob/master/filesys/fastfat/cleanup.c

> In principle, I could say:
>
> delete file X
>
> which then returns immediately, and if I try to open(X) it will fail. But I
> can still see it if I do a dir() on the parent directory?

Yes.

> Eventually the last reference to X will go away, and then it is unlinked.
> What happens if I pull the plug in the meantime? Will the file magically
> come back on rebooting?

Setting the delete disposition is just a flag in the in-memory FCB
structure, so if you pull the plug the file hasn't actually been
unlinked. It's still there.

Bear in mind that sharing delete access is rare on Windows. Normally
when a file is deleted there's only a single File object referencing
it, such as from DeleteFile calling NtOpenFile. Then it immediately
calls NtSetInformationFile to set the delete disposition. When it
closes the handle, this triggers the CloseProcedure => IRP_MJ_CLEANUP,
DeleteProcedure => IRP_MJ_CLOSE sequence that actually unlinks the
file.

>> Finally, I'm sure most people are familiar with the read-only file
>> attribute. If this attribute is set you can still open a file with
>> delete access to rename it, but setting the delete disposition will
>> fail with access denied.
>
> That was actually the situation I was thinking about when I started on this
> question. From Python code, I was considering writing something like this:
>
> def delete_readonly(thefile):
> try:
> os.remove(thefile)
> except ReadOnlyFileError: # what is this really?

It's a PermissionError with winerror == 5, i.e. access denied. As
mentioned above, you can use os.chmod to remove the read-only file
attribute.

[eryk sun]

On Tue, Mar 14, 2017 at 1:07 PM, Chris Angelico <ros...@gmail.com> wrote:

> On Tue, Mar 14, 2017 at 10:32 PM, Steve D'Aprano
> <steve+...@pearwood.info> wrote:
>
>> I take it that you *can* delete open files, but only if the process that
>> opens them takes special care to use "delete sharing". Is that correct?
>

> Yes, but you can't always control the process that opens them. For
> example, it's annoyingly difficult to update a running executable.

One option is to move it out of the way. For example:

src = Path(sys.executable)
dst = Path('C:/Temp/python.exe')

>>> src.exists(), dst.exists()
(True, False)

>>> src.rename(dst)
>>> src.exists(), dst.exists()
(False, True)

>>> dst.rename(src)
>>> src.exists(), dst.exists()
(True, False)

> More generally, a handle is just an opaque cookie that represents a
> thing. If you do cross-platform socket programming, you'll find that
> on Unix, creating a socket returns a "file descriptor"; but on
> Windows, it returns a "socket handle". Aside from the fact that socket
> handles can only be used with socket functions (making them harder to
> pass to other processes etc), the two concepts work broadly the same
> way: you have a small integer value that represents a large resource.

Usually a socket is a handle for a File object, from opening the AFD
(ancillary function) device. You can use it with [Nt]ReadFile and
[Nt]WriteFile [1]. It's just better to use it with Winsock APIs like
WSARecv and WSASend, which make NtDeviceIoControlFile calls.

But there's also the headache of the deprecated layered service
providers, for which a socket isn't really a File object and using
File functions is inefficient.

[1]: https://msdn.microsoft.com/en-us/library/ms740522

Similarly for a console File handle you can use [Nt]ReadFile and
[Nt]WriteFile, which encode and decode byte streams using the console
input and output codepages. But it's generally better to use
ReadConsole and WriteConsole, which allow using the UTF-16 API via
NtDeviceIoControlFile. ReadConsole also allows defining a custom
control mask, if, for example, you want Ctrl+D to signal EOF.

> I'm terrified to think what the interactions would be between this and
> the Ubuntu subsystem in Win 10. What're the semantics of unlinkat()?
> What about the common practice of creating a file and then unlinking
> it so it doesn't show up in the directory? (And is that different on a
> file system that supports hard links?) Or are heaps of POSIX features
> simply not available?

WSL uses the VolFs file system for the Linux VFS. This should
implement native Linux file-system behavior.

The interesting interaction is for Windows drives that are made
available under /mnt. In the following example, in Windows Python I
set the delete disposition on a file named "spam". Then in WSL python3
I checked the directory listing and whether I could stat, remove, or
open the file:

>>> os.listdir()
['spam']

>>> os.stat('spam')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'spam'

>>> os.remove('spam')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'spam'

>>> open('spam')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'spam'

>>> open('spam', 'w')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'spam'

Back in Windows I called SetFileInformationByHandle to unset the
delete disposition. Then regular access was restored in WSL:

>>> s = os.stat('spam')
>>> s.st_ino, s.st_size
(35184372088857305, 10)

[Michael Felt]

On 13/03/2017 02:51, Steve D'Aprano wrote:
> On Mon, 13 Mar 2017 05:45 am, Alain Ketterlin wrote:
>
>> Steve D'Aprano <steve+...@pearwood.info> writes:
> [...]
>>> It seems that os.remove on Linux will force the delete even if the file
>>> is read-only or unreadable, provided you own the file.
>> Your permissions on the file do not really matters. It's all about your
>> permissions on the parent directory (removing a file is really modifying
>> the parent dir). Actually, it is even slightly more complicated. Here is
>> an excerpt of the unlink(2) call (which does the job) listing common
>> error cases:

Granted, I am a bit behind in the discussion - and I know nothing about
how Windows manages this since DOS 3.3 - there it also called unlink().

rm is the command we run. The system call it uses to remove a file is
unlink(). unlink() removes the "link" of the name in the directory to
the inode and lowers the count of the number of links to the file (a
hard link is an additional directory link to the inode). To modify the
inode (link counter) you need to own the file or have (super user)
elevated privelidges. To remove the directory link you need to have
write privilidge on the (special file type) directory.

On UNIX/Linux when the link count is zero the inode and the filesystem
blocks it provides access to are cleared if no process is holding the
file open. This means it is easy to unlink() (not remove!) an open file.
The inode is active and can be used "normally". FYI, the classic
mktemp() library routine on UNIX would create a file, open (better
create) the file, unlink the file, and then return file descriptor.
Regardless of how the program ended (crash or normal) the temp file
would be cleaned up on exit.

I would be guessing - but FAT and/or NTFS may not be written to clean up
on a file with no "links" - and this is why Windows behavior seems to be
more restrictive than POSIX/LINUX.

[Grant Edwards]

On 2017-03-13, eryk sun <ery...@gmail.com> wrote:

[An impressive 150-line explanation of file removal on Windows.]

Wow. I have two comments:

1. I think I can see the VMS heritage of Windows shining through.

2. You have my condolences regarding whatever it was that required you
to know all that...



VMS: for when just one overly complex solution to a simple problem
isn't enough.

--
Grant Edwards grant.b.edwards Yow! Finally, Zippy
at drives his 1958 RAMBLER
gmail.com METROPOLITAN into the
faculty dining room.

[Chris Angelico]

On Wed, Mar 15, 2017 at 5:28 AM, Michael Felt <mic...@felt.demon.nl> wrote:
> Granted, I am a bit behind in the discussion - and I know nothing about how
> Windows manages this since DOS 3.3 - there it also called unlink().
>
> rm is the command we run. The system call it uses to remove a file is
> unlink(). unlink() removes the "link" of the name in the directory to the
> inode and lowers the count of the number of links to the file (a hard link
> is an additional directory link to the inode). To modify the inode (link
> counter) you need to own the file or have (super user) elevated privelidges.
> To remove the directory link you need to have write privilidge on the
> (special file type) directory.
>
> On UNIX/Linux when the link count is zero the inode and the filesystem
> blocks it provides access to are cleared if no process is holding the file
> open. This means it is easy to unlink() (not remove!) an open file. The
> inode is active and can be used "normally". FYI, the classic mktemp()
> library routine on UNIX would create a file, open (better create) the file,
> unlink the file, and then return file descriptor. Regardless of how the
> program ended (crash or normal) the temp file would be cleaned up on exit.

You've fairly accurately summed up the POSIX model. The rm command
uses the unlink syscall, and the semantics are broadly as you
describe; the permissions question is handled by the standard
permission bits "rwxrwxrwx", but otherwise you're about right. And
yes, removing a file means removing one name from it, which is why a
log file growing to infinity can't simply be deleted when you run out
of disk space.

If the program ends while the OS is still alive, then yes, the temp
file will be cleaned up on exit. If the power is pulled, AIUI a fsck
will notice that there's a file with no directory entries, and clean
it up, although it may end up dropping it into lost+found rather than
just deleting it.

> I would be guessing - but FAT and/or NTFS may not be written to clean up on
> a file with no "links" - and this is why Windows behavior seems to be more
> restrictive than POSIX/LINUX.

The Windows semantics are very different. Firstly, FAT doesn't even
_have_ the concept of hardlinks (aka files with multiple directory
entries), so there's no way that it'll ever be able to do this kind of
thing. (And before you say "oh but FAT is dead these days", it's still
frequently used on removable media.) NTFS does have hardlinks, so in
theory it would be capable of POSIX semantics, but Windows semantics
are baked into a lot of programs' expectations, so I doubt that that's
going to change.

ChrisA

[Erik]

On 14/03/17 13:56, Chris Angelico wrote:
> On Wed, Mar 15, 2017 at 12:30 AM, Jon Ribbens <jon+u...@unequivocal.eu> wrote:

>> rm does not ask before deleting. However some Linux distributions
>> take it upon themselves to put "alias rm='rm -i'" in /etc/profile.
>

> I have no such alias, but it still prompts.

[snip]

> This is the GNU coreutils rm command. Obviously behaviour may be
> different with non-GNU rm's.

Yes, I believe this is something extra that particular versions of 'rm'
may do (over and above the POSIX/unlink() semantics discussed already).

Although one has permission at the OS/filesystem level to unlink the
file, unless '-f' is specified they may still ask ('-i' behaviour) if
that file is otherwise not modifiable by the user running the command,
as a safety net.

E.

[eryk sun]

On Tue, Mar 14, 2017 at 7:01 PM, Grant Edwards
<grant.b...@gmail.com> wrote:
>
> 1. I think I can see the VMS heritage of Windows shining through.

That's not surprising considering that VMS and NT have the same
architect -- Dave Cutler -- and that I/O system and file systems were
design by former DEC programmers that he brought with him, such as
Darryl Havens and Gary Kimura. The old joke is that WNT = VMS + 1.

[eryk sun]

On Tue, Mar 14, 2017 at 10:05 PM, Dennis Lee Bieber
<wlf...@ix.netcom.com> wrote:
> On Wed, 15 Mar 2017 00:07:32 +1100, Chris Angelico <ros...@gmail.com>

>
>>Yes, but you can't always control the process that opens them. For
>>example, it's annoyingly difficult to update a running executable.
>>

> I wouldn't be surprised if Windows mmap()'s running executables into
> swap space, rather than actually swapping in-core image to the normal swap
> file.

Executables are memory mapped, and this also applies to memory-mapped
data files (i.e. CreateFileMapping / MapViewOfFile) like with Python's
mmap module. Notice how the fastfat driver's function that sets the
delete disposition, FatSetDispositionInfo [1], has to check for this
by calling the kernel memory-manager function MmFlushImageSection [2]
(note that a flush type of MmFlushForDelete also checks the number of
views of the data section). If the mapped view prevents deleting the
file it returns STATUS_CANNOT_DELETE. This status code gets translated
to the Windows error code ERROR_ACCESS_DENIED, for which Python raises
a PermissionError. This is misleading because you were probably
granted delete access. The file just can't be deleted.

Typically you can still open such files with delete access to allow
renaming (relinking) them to another directory on the volume. You're
just not allowed to unlink them.

[1]: https://github.com/Microsoft/Windows-driver-samples/blob/master/filesys/fastfat/fileinfo.c#L2417
[2]: https://msdn.microsoft.com/en-us/library/ff549808

It's also interesting that the sharing mode is special cased. Normally
if write access isn't shared you're denied the right to open the file
with either write or append access. But with executables mapped by the
loader, you can still open them for append access.

The C runtime's POSIX (low) I/O layer has its own implementation of
append mode that requires opening the file with both write and append
access (O_WRONLY | O_APPEND), which is what Python's "a" mode uses. So
you have to call CreateFile directly to get append-only access. For
example, using a virtual environment copy of python.exe:

append = 4
h = _winapi.CreateFile(sys.executable, append, 7, 0, 3, 0, 0)
_winapi.WriteFile(h, b'spam')

f = open(sys.executable, 'rb')
f.seek(-4, os.SEEK_END)

>>> f.read()
b'spam'