[Python-Dev] Returning Windows file attribute information via os.stat()

[Ben Hoyt]

Hi folks,

As pointed out to me recently in an issue report [1] on my scandir
module, Python's os.stat() simply discards most of the file attribute
information fetched via the Win32 system calls. On Windows, os.stat()
calls CreateFile to open the file and get the dwFileAttributes value,
but it throws it all away except the FILE_ATTRIBUTE_DIRECTORY and
FILE_ATTRIBUTE_READONLY bits. See CPython source at [2].

Given that os.stat() returns extended, platform-specific file
attributes on Linux and OS X platforms (see [3] -- for example,
st_blocks, st_rsize, etc), it seems that Windows is something of a
second-class citizen here.

There are several questions on StackOverflow about how to get this
information on Windows, and one has to resort to ctypes. For example,
[4].

To solve this problem, what do people think about adding an
"st_winattrs" attribute to the object returned by os.stat() on
Windows?

Then, similarly to existing code like hasattr(st, 'st_blocks') on
Linux, you could write a cross-platform function to determine if a
file was hidden, something like so:

FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h

def is_hidden(path):
if startswith(os.path.basename(path), '.'):
return True
st = os.stat(path)
if hasattr(st, 'st_winattrs') and st.st_winattrs & FILE_ATTRIBUTE_HIDDEN:
return True
return False

I'd be interested to hear people's thoughts on this.

Thanks,
Ben.

[1]: https://github.com/benhoyt/scandir/issues/22
[2]: https://github.com/python/cpython/blob/master/Modules/posixmodule.c#L1462
[3]: https://docs.python.org/3.4/library/os.html#os.stat
[4]: http://stackoverflow.com/a/6365265
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Paul Moore]

On 10 June 2014 05:02, Ben Hoyt <ben...@gmail.com> wrote:
> To solve this problem, what do people think about adding an
> "st_winattrs" attribute to the object returned by os.stat() on
> Windows?

+1. Given the precedent of Linux- and OS X-specific attributes, this
seems like a no-brainer to me.

Paul

[Victor Stinner]

2014-06-10 6:02 GMT+02:00 Ben Hoyt <ben...@gmail.com>:
> To solve this problem, what do people think about adding an
> "st_winattrs" attribute to the object returned by os.stat() on
> Windows?

> (...)

> FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h
>

> if hasattr(st, 'st_winattrs') and st.st_winattrs & FILE_ATTRIBUTE_HIDDEN:

I don't like such API, it requires to import constants, use masks, etc.

I would prefer something like:

if st.win_hidden: ...

Or maybe:

if st.winattrs.hidden: ...

Victor

[MRAB]

On 2014-06-10 05:02, Ben Hoyt wrote:
[snip]

>
> FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h
>
> def is_hidden(path):
> if startswith(os.path.basename(path), '.'):
> return True
> st = os.stat(path)
> if hasattr(st, 'st_winattrs') and st.st_winattrs & FILE_ATTRIBUTE_HIDDEN:

That could be written more succinctly as:

if getattr(st, 'st_winattrs', 0) & FILE_ATTRIBUTE_HIDDEN:

> return True
> return False
>

[Ben Hoyt]

> > FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h
> >
> > if hasattr(st, 'st_winattrs') and st.st_winattrs & FILE_ATTRIBUTE_HIDDEN:
>
> I don't like such API, it requires to import constants, use masks, etc.
>
> I would prefer something like:
>
> if st.win_hidden: ...
>
> Or maybe:
>
> if st.winattrs.hidden: ...

Yes, fair call. However, it looks like the precent for the attributes
in os.stat()'s return value has long since been set -- this is
OS-specific stuff. For example, what's in "st_flags"? It's not
documented, but comes straight from the OS. Same with st_rdev,
st_type, etc -- the documentation doesn't define them, and it looks
like they're OS-specific values.

I don't think the st.win_hidden approach gains us much, because the
next person is going to ask for the FILE_ATTRIBUTE_ENCRYPTED or
FILE_ATTRIBUTE_COMPRESSED flag. So we really need all the bits or
nothing. I don't mind the st.st_winattrs.hidden approach, except that
we'd need 17 sub-attributes, and they'd all have to be documented. And
if Windows added another attribute, Python wouldn't have it, etc. So I
think the OS-defined constant is the way to go.

Because these are fixed-forever constants, I suspect in library code
and the like people would just KISS and use an integer literal and a
comment, avoiding the import/constant thing:

if getattr(st, 'st_winattrs', 0) & 2: # FILE_ATTRIBUTE_HIDDEN
...

-Ben

[Ben Hoyt]

>> if hasattr(st, 'st_winattrs') and st.st_winattrs &
>> FILE_ATTRIBUTE_HIDDEN:
>
> That could be written more succinctly as:
>
> if getattr(st, 'st_winattrs', 0) & FILE_ATTRIBUTE_HIDDEN:
>
>> return True
>> return False

Yes, good call. Or one further:

return getattr(st, 'st_winattrs', 0) & FILE_ATTRIBUTE_HIDDEN != 0

-Ben

[Paul Moore]

On 10 June 2014 13:19, Ben Hoyt <ben...@gmail.com> wrote:
> Because these are fixed-forever constants, I suspect in library code
> and the like people would just KISS and use an integer literal and a
> comment, avoiding the import/constant thing:

The stat module exposes a load of constants - why not add the
(currently known) ones there? Finding the values of Windows constants
if you don't have access to the C headers can be a pain, so having
them defined *somewhere* as named values is useful.

Paul

[Ben Hoyt]

> The stat module exposes a load of constants - why not add the
> (currently known) ones there? Finding the values of Windows constants
> if you don't have access to the C headers can be a pain, so having
> them defined *somewhere* as named values is useful.

So stat.FILE_ATTRIBUTES_HIDDEN and the like? Alternatively they could
go in ctypes.wintypes, but I think stat makes more sense in this case.

-Ben

[Paul Moore]

On 10 June 2014 13:58, Ben Hoyt <ben...@gmail.com> wrote:
> So stat.FILE_ATTRIBUTES_HIDDEN and the like?

Yep. (Maybe WIN_FILE_ATTRIBUTES_HIDDEN, but the Unix ones don't have
an OA name prefix, so I'd go with your original).

Paul

[Ethan Furman]

On 06/09/2014 09:02 PM, Ben Hoyt wrote:
>
> To solve this problem, what do people think about adding an
> "st_winattrs" attribute to the object returned by os.stat() on
> Windows?

+1 to the idea, whatever the exact implementation.

--
~Ethan~

[Zachary Ware]

On Tue, Jun 10, 2014 at 12:17 PM, Ethan Furman <et...@stoneleaf.us> wrote:
> On 06/09/2014 09:02 PM, Ben Hoyt wrote:
>> To solve this problem, what do people think about adding an
>> "st_winattrs" attribute to the object returned by os.stat() on
>> Windows?
>
>
> +1 to the idea, whatever the exact implementation.

Agreed.

--
Zach

[Ben Hoyt]

>> To solve this problem, what do people think about adding an
>> "st_winattrs" attribute to the object returned by os.stat() on
>> Windows?
>
> +1 to the idea, whatever the exact implementation.

Cool.

I think we should add a st_winattrs integer attribute (on Windows) and
then also add the FILE_ATTRIBUTES_* constants to stat.py per Paul
Moore.

What would be the next steps to get this to happen? Open an issue on
bugs.python.org and submit a patch with tests?

-Ben

[Zachary Ware]

On Tue, Jun 10, 2014 at 2:04 PM, Ben Hoyt <ben...@gmail.com> wrote:
>>> To solve this problem, what do people think about adding an
>>> "st_winattrs" attribute to the object returned by os.stat() on
>>> Windows?
>>
>> +1 to the idea, whatever the exact implementation.
>
> Cool.
>
> I think we should add a st_winattrs integer attribute (on Windows) and
> then also add the FILE_ATTRIBUTES_* constants to stat.py per Paul
> Moore.

Add to _stat.c rather than stat.py.

> What would be the next steps to get this to happen? Open an issue on
> bugs.python.org and submit a patch with tests?

Yep!

--
Zach

[Ben Hoyt]

>> What would be the next steps to get this to happen? Open an issue on
>> bugs.python.org and submit a patch with tests?
>
> Yep!

Okay, I've done step one (opened an issue on bugs.python.org), and
hope to provide a patch in the next few weeks if no-one else does
(I've never compiled CPython on Windows before):

http://bugs.python.org/issue21719

-Ben

[Terry Reedy]

On 6/11/2014 9:27 AM, Ben Hoyt wrote:
>>> What would be the next steps to get this to happen? Open an issue on
>>> bugs.python.org and submit a patch with tests?
>>
>> Yep!
>
> Okay, I've done step one (opened an issue on bugs.python.org), and
> hope to provide a patch in the next few weeks if no-one else does
> (I've never compiled CPython on Windows before):
>
> http://bugs.python.org/issue21719

If you have problems compiling, the core-mentorship list is one place to
ask. For 3.4+, I believe the devguide instructions are correct. If not,
say something.

--
Terry Jan Reedy