findfirst, findnext and finddata_t
Michel
I'm porting a Win32 application to linux. I'm looking for the
equivalent functions to _findfirst, _findnext and definitions
finddata_t on linux.
I do care about portability, so I don't think it's a good idea to use
the linux inherent opendir, readir, etc. to do the job instead.
The best thing would be to have a _findfirst, _findnext, _finddata_t
implementation for linux. Is there some library that already
implements them?
Thanks
Roger Leigh
> I'm porting a Win32 application to linux. I'm looking for the
> equivalent functions to _findfirst, _findnext and definitions
> finddata_t on linux.
> I do care about portability, so I don't think it's a good idea to use
> the linux inherent opendir, readir, etc. to do the job instead.
But, you're ignoring the fact that opendir(3), readdir(3) and
associated functions are part of the IEEE POSIX and SUSv3 standards.
If you want to write portable software, using the portable functions
is a good start.
From opendir(3): "opendir() is present on SVr4, 4.3BSD, and specified
in POSIX.1-2001", as well as the Single UNIX Specification.
You can read the SUS here:
http://www.unix.org/single_unix_specification/
Look in the System Interfaces section.
> The best thing would be to have a _findfirst, _findnext, _finddata_t
> implementation for linux. Is there some library that already
> implements them?
WINE might, but IMO your solution is backwards. You should really be
implementing (or using preexisting) standard functions on all
platforms, including Windows, such as those specified in IEEE POSIX
and SUSv3 (these are the same standards in all but name).
Reimplementing non-standard Microsoft-specific functionality is not
the way to achieve portability. Check out MinGW and Cygwin.
Microsoft also have what was previously Interix.
Regards,
Roger
--
.''`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
David Schwartz
> Michel <bmic...@gmail.com> writes:
> > The best thing would be to have a _findfirst, _findnext, _finddata_t
> > implementation for linux. Is there some library that already
> > implements them?
> WINE might, but IMO your solution is backwards. You should really be
> implementing (or using preexisting) standard functions on all
> platforms, including Windows, such as those specified in IEEE POSIX
> and SUSv3 (these are the same standards in all but name).
> Reimplementing non-standard Microsoft-specific functionality is not
> the way to achieve portability. Check out MinGW and Cygwin.
> Microsoft also have what was previously Interix.
Exactly. There's a lot of crappy Windows software out there because
experienced UNIX programmers tried to make Windows as much like UNIX
as possible. Let's not repeat this sin in the other direction by
producing crappy Linux software by trying to make Linux look like
Windows.
On Windows, you do things the Windows way. On Linux, do things the
Linux way.
DS
Rainer Weikusat
> On Jan 5, 9:44 am, Roger Leigh <rle...@whinlatter.ukfsn.org> wrote:
>> Michel <bmic...@gmail.com> writes:
>> > The best thing would be to have a _findfirst, _findnext, _finddata_t
>> > implementation for linux. Is there some library that already
>> > implements them?
>
>> WINE might, but IMO your solution is backwards. You should really be
>> implementing (or using preexisting) standard functions on all
>> platforms, including Windows, such as those specified in IEEE POSIX
>> and SUSv3 (these are the same standards in all but name).
>> Reimplementing non-standard Microsoft-specific functionality is not
>> the way to achieve portability. Check out MinGW and Cygwin.
>> Microsoft also have what was previously Interix.
>
> Exactly. There's a lot of crappy Windows software out there because
> experienced UNIX programmers tried to make Windows as much like UNIX
> as possible.
The root cause of this would be that a certain PC software development
company must absolutely move to a new level of NIH whenever a new
batch of university graduates very convinced to know everything better
by virtue of not knowing anything is hired.
But Microsoft is certainly in the position to fend of UNIX(*) by
burning money for a much longer time than the numerous, now defunct
minicomputer vendors of the 1980s.
David Schwartz
> The root cause of this would be that a certain PC software development
> company must absolutely move to a new level of NIH whenever a new
> batch of university graduates very convinced to know everything better
> by virtue of not knowing anything is hired.
I think that is a large fraction of the problem. But in all fairness,
part of it is actual Microsoft innovations, such as IOCP.
I can't defend Winsock1, that's a complete and utter abomination.
However, I can defend Winsock2. For example, you can argue that
Microsoft's pathetic emulation of 'select' is an example of NIH and an
attempt to make perfectly good code work badly. But in truth, 'select'
is the abomination (or at least, it has outlived its usefulness).
UNIXes have moved away from it.
DS
Uwe Bonnes
winelib is library implementing these function. Compiling your program as a
winelib program might be a first step.
--
Uwe Bonnes b...@elektron.ikp.physik.tu-darmstadt.de
Institut fuer Kernphysik Schlossgartenstrasse 9 64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------
phil-new...@ipal.net
So select(2) is now (officially?) depricated in favor of poll(2)?
--
-----------------------------------------------------------------------------
| Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ |
| (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ |
-----------------------------------------------------------------------------
phil-new...@ipal.net
| I'm porting a Win32 application to linux. I'm looking for the
| equivalent functions to _findfirst, _findnext and definitions
| finddata_t on linux.
| I do care about portability, so I don't think it's a good idea to use
| the linux inherent opendir, readir, etc. to do the job instead.
No, you would not want to use the ABI level kernel traps for these syscalls.
True portability would involve the POSIX API calls of the same names.
| The best thing would be to have a _findfirst, _findnext, _finddata_t
| implementation for linux. Is there some library that already
| implements them?
That's not portability. That would be emulation. Maybe one of the Windows
emulation packages have it.
Those are poorly chosen names if they are intended to gather a list of file
names in a given directory. They could just as easily be lookup tools for
any kind of container ADT.
Måns Rullgård
> On Sun, 6 Jan 2008 00:35:29 -0800 (PST) David Schwartz <dav...@webmaster.com> wrote:
> | On Jan 5, 10:50 pm, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
> |
> |> The root cause of this would be that a certain PC software development
> |> company must absolutely move to a new level of NIH whenever a new
> |> batch of university graduates very convinced to know everything better
> |> by virtue of not knowing anything is hired.
> |
> | I think that is a large fraction of the problem. But in all fairness,
> | part of it is actual Microsoft innovations, such as IOCP.
> |
> | I can't defend Winsock1, that's a complete and utter abomination.
> | However, I can defend Winsock2. For example, you can argue that
> | Microsoft's pathetic emulation of 'select' is an example of NIH and an
> | attempt to make perfectly good code work badly. But in truth, 'select'
> | is the abomination (or at least, it has outlived its usefulness).
> | UNIXes have moved away from it.
>
> So select(2) is now (officially?) depricated in favor of poll(2)?
Officially, not in the slightest way. I also fail to see what is so
abominable about it.
--
Måns Rullgård
ma...@mansr.com
phil-new...@ipal.net
After programming with poll(2) for a few years instead of select(2), I do
find it easier to use. That would not justify depricating it even if just
for new programs. But select(2) does have some negatives. That its list
of descriptors is a bitmap is one, IMHO. It's also necessary to have the
size of the map known at compile time even though it is possible to change
the limit on descriptors at boot time or later.
But even poll(2) has its detractions. A better method would be one that
includes the option to directly provide a list of descriptors that are now
ready, instead of having to scan the list or bitmap to look for them. Or
at least a call that would return one of them (which could be looped on to
get each ready queued descriptor). Most programs won't find much of a need
for this since most will have only a small set of working descriptors, so
the poll(2) list would be short, or the select(2) bitmap could be indexed
using a small list. Threaded programs could just not use nonblocking I/O
and let each thread block on I/O (but the inability to signal a specific
thread complicates that).
I'll just stick with poll(2).
Lew Pitcher
Your question presumes that the find*() family of functions is somehow
more standard and more portable than the *dir() family of functions.
This is not the case.
The *dir() functions are supported in all Posix environments, while
the find*() functions are only supported in the Microsoft Windows
environment. *dir() are inherently more portable (wrt the number of
systems that implement them) than the find*() functions are.
However, I can acknowledge that find*() are useful; this family of
functions can be emulated through the use of a proper regex() package
in conjunction with the standard *dir() functions.
David Schwartz
> On Sun, 6 Jan 2008 00:35:29 -0800 (PST) David Schwartz <dav...@webmaster.com> wrote:
> | I can't defend Winsock1, that's a complete and utter abomination.
> | However, I can defend Winsock2. For example, you can argue that
> | Microsoft's pathetic emulation of 'select' is an example of NIH and an
> | attempt to make perfectly good code work badly. But in truth, 'select'
> | is the abomination (or at least, it has outlived its usefulness).
> | UNIXes have moved away from it.
> So select(2) is now (officially?) depricated in favor of poll(2)?
I don't think there's any harm in using 'select' in an application
that has a very small number of descriptors. However, 'select' breaks
down horribly for applications that have large numbers of descriptors,
even if you're only trying to 'select' on a few of them. The API that
includes 'select' was designed at a time when 256 file descriptors was
a common application maximum.
In any event, 'select' is not the "best way" to do anything anymore. I
can't think of any case where 'select' would be pref3erred over
'poll'. On Linux, even 'poll' is not the 'right way' for applications
that deal with load or large numbers of descriptors.
I guess the unstated assumption in my post is that we're dealing with
cases where Winsock's 'select' emulation is insufficient or
problemmatic. In those same cases, 'select' sucks too.
DS
David Schwartz
> Officially, not in the slightest way. I also fail to see what is so
> abominable about it.
You are writing a library that uses 'select'. Please show me the
portable code that assures that your fd sets are large enough to
accept any value 'socket' might return.
DS
Rainer Weikusat
[...]
> The API that includes 'select' was designed at a time when 256 file
> descriptors was a common application maximum.
Actually, it was designed at the time when increasing the per-process
descriptor limit to 32 open descriptors was 'a feature going to be
implemented some day'.
phil-new...@ipal.net
| In any event, 'select' is not the "best way" to do anything anymore. I
| can't think of any case where 'select' would be pref3erred over
| 'poll'. On Linux, even 'poll' is not the 'right way' for applications
| that deal with load or large numbers of descriptors.
But there isn't a standard portable way to deal with large numbers of
descriptors?
phil-new...@ipal.net
There are some library functions available that handle file trees by
recursing through them. One (ftw) is (IMHO) not so clean by having a
call-back style API. Another (fts) is rather clunky with its options
(also IMHO). I wrote one (ftr) that has a simpler API and allows a
program to concurrently recurse multiple trees at once. Mine can be
seen in my LIBH package at http://libh.slashusr.org/
David Schwartz
> But there isn't a standard portable way to deal with large numbers of
> descriptors?
No, there isn't. When you're dealing with that kind of thing, you need
to use the best way on each particular platform. On Windows, that's
IOCP. On Linux, it's epoll. On FreeBSD, it's kqueue. On Solaris, I
think it's still /dev/poll (but it might be something else entirely).
And so on.
It definitely isn't 'select'. There isn't even a portable way to
'select' on a socket descriptor passed to a library function.
DS
Rainer Weikusat
> phil-new...@ipal.net writes:
>> On Sun, 6 Jan 2008 00:35:29 -0800 (PST) David Schwartz <dav...@webmaster.com> wrote:
>> | On Jan 5, 10:50 pm, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
[...]
>> | But in truth, 'select'is the abomination (or at least, it has
>> | outlived its usefulness).
>> | UNIXes have moved away from it.
>>
>> So select(2) is now (officially?) depricated in favor of poll(2)?
>
> Officially, not in the slightest way. I also fail to see what is so
> abominable about it.
Two reasons would be:
- it is more cumbersome to use than any of the alternatives,
because the interest set needs to be rebuilt on every
iteration of a select-loop
- the kernel/userspace interface is grossly inefficient
An example for the second point: Assuming I would be using select for
a small (3) number of descriptors whose numerical values a close-by.
Select takes three bitmask arguments, each of size 1024 (IIRC) for
Linux. This amounts to 384 bytes which need to be copied into the
kernel and out of the kernel when entering and leaving the call. Of
these 3072 bits, at most nine are used to carry actual information, so
the SDU/ PDU ratio is ~0.003, or there is 341 times more overhead than
useful data. In addition to the time needed copy all these useless
zero bits, doing so causes (for my usual target platform, worst case)
24 cache lines potentially holding useful data to be wiped clean.
When using poll to accomplish the same, only 24 bytes need to be
transfered between userspace and kernel space for each invocation of
and return from the syscall.
Using bitmaps to specify the select interest sets was a clever idea by
the time each of these sets easily fitted into a single machine word,
cf
Each process has a fixed size descriptor reference table,
where the size is returned by the getdtablesize call:
nds = getdtablesize();
result int nds;
and guaranteed to be at least 20. The entries in the
descriptor reference table are referred to by small inte-
gers; for example if there are 20 slots they are numbered 0 to
19.
(4.2BSD System Manual, p. 17 [pdf])
Chris Friesen
> An example for the second point: Assuming I would be using select for
> a small (3) number of descriptors whose numerical values a close-by.
> Select takes three bitmask arguments, each of size 1024 (IIRC) for
> Linux. This amounts to 384 bytes which need to be copied into the
> kernel and out of the kernel when entering and leaving the call.
Why would you need to copy the data for each of the three bitmasks? The
kernel takes pointers to the bitmaps, not the bitmaps themselves so it
only needs to copy the bitmasks that are actually being used. Also, it
only needs to copy the bits up to the highest descriptor value.
I admit that if you want to check descriptor 1023 for
read/write/exception then you'll have some innefficiency. But checking
descriptors 0/1/2 for reading requires copying three bitmask pointers
and a single word of the bitmask itself.
Chris
Rainer Weikusat
> Rainer Weikusat wrote:
>> An example for the second point: Assuming I would be using select for
>> a small (3) number of descriptors whose numerical values a close-by.
>> Select takes three bitmask arguments, each of size 1024 (IIRC) for
>> Linux. This amounts to 384 bytes which need to be copied into the
>> kernel and out of the kernel when entering and leaving the call.
>
> Why would you need to copy the data for each of the three bitmasks?
Because not specifying nfds as 'FD_SETSIZE' 'is just an
optimization', to quote another BSD clown.
Rainer Weikusat
A more sensible remark in this context would be that this (copying
only the used parts of an fd_set) indeed how the Linux-implementation
works. But 0, 534 and 1011 are three valid file descriptor numbers,
too. Additionally (ARM9), the code implementing this workaround for
an interface designed for an 'early eighties' UNIX(*) kernel figures
at ~723 bytes, making it nearly twice as large as all three
sets combined. This code needs to be loaded from memory, which is
'half of a copy' and doing so (assuming all is needed) amounts to
wiping 46 (i-)cache lines instead of 24 (d-cache lines).
While this would be difficult to measure, I am not convinced that not
just copying the 384 bytes wouldn't be more sensible.
BTW, it would be possible to use something with a triangular shape
instead of a wheel and rely on stronger motors to achieve comparable
speeds, presumably at the expense of a higher maintenance cost.
Linonut
"man ftw"
--
Ask not for whom the <CONTROL-G> tolls.
Linonut
> Michel <bmi...@gmail.com> writes:
>
>> I'm porting a Win32 application to linux. I'm looking for the
>> equivalent functions to _findfirst, _findnext and definitions
>> finddata_t on linux.
>> I do care about portability, so I don't think it's a good idea to use
>> the linux inherent opendir, readir, etc. to do the job instead.
>
> From opendir(3): "opendir() is present on SVr4, 4.3BSD, and specified
> in POSIX.1-2001", as well as the Single UNIX Specification.
>
>> The best thing would be to have a _findfirst, _findnext, _finddata_t
>> implementation for linux. Is there some library that already
>> implements them?
>
> WINE might, but IMO your solution is backwards. You should really be
> implementing (or using preexisting) standard functions on all
> platforms, including Windows, such as those specified in IEEE POSIX
> and SUSv3 (these are the same standards in all but name).
> Reimplementing non-standard Microsoft-specific functionality is not
> the way to achieve portability. Check out MinGW and Cygwin.
> Microsoft also have what was previously Interix.
Bleh. Interix.
Anyway, you will have to implement ftw-for-Windows yourself. I had to.
--
The wise man seeks everything in himself; the ignorant man tries to get
everything from somebody else.
phil-new...@ipal.net
The Win32 API appears, based on the little bit posted in this thread, to
have distinct functions to get the first (probably also initiating the
tree walk) and each subsequent item. The ftw "package" does not appear
to be like that at all. Perhaps the Win32 functions have some kind of
pointer to an instance that allows the findnext function (which ftw has
no equivalent for) to work. If that is so, then it would be possible to
do two or more separate tree walks across different trees at in parallel
at the same time. The ftw function cannot do that due to the way it is
designed as a callback function.
So I suggest that ftw is not a good equivalent. I have argued in the
past that ftw should even be depricated and eventually removed. There
are better APIs available, including fts that is already included, even
though it is rather difficult to use due to so many options, as well as
my own ftr functions. You can browse the source code of these functions
directly at: http://libh.slashusr.org/source/ftr/src/lib/h/
Do be sure to look at the ftr_header.h file since a number of macros
that would be used like functions are defined there.
Linonut
>| "man ftw"
>
> The Win32 API appears, based on the little bit posted in this thread, to
> have distinct functions to get the first (probably also initiating the
> tree walk) and each subsequent item. The ftw "package" does not appear
> to be like that at all. Perhaps the Win32 functions have some kind of
> pointer to an instance that allows the findnext function (which ftw has
> no equivalent for) to work. If that is so, then it would be possible to
> do two or more separate tree walks across different trees at in parallel
> at the same time. The ftw function cannot do that due to the way it is
> designed as a callback function.
Huh? Makes no sense to me. Each invocation of the callback has its own
stack of parameters.
Oh, I can see where you'd have trouble copying filenames into a global
object, though.
> So I suggest that ftw is not a good equivalent. I have argued in the
> past that ftw should even be depricated and eventually removed. There
> are better APIs available, including fts that is already included, even
> though it is rather difficult to use due to so many options, as well as
> my own ftr functions. You can browse the source code of these functions
> directly at: http://libh.slashusr.org/source/ftr/src/lib/h/
> Do be sure to look at the ftr_header.h file since a number of macros
> that would be used like functions are defined there.
Thanks for the heads up on fts and ftr. Just make sure you don't name
the next version "ftp" <grin>.
--
If you want to make God laugh, tell him about your plans.
-- Woody Allen
phil-new...@ipal.net
| * phil-new...@ipal.net peremptorily fired off this memo:
|
|>| "man ftw"
|>
|> The Win32 API appears, based on the little bit posted in this thread, to
|> have distinct functions to get the first (probably also initiating the
|> tree walk) and each subsequent item. The ftw "package" does not appear
|> to be like that at all. Perhaps the Win32 functions have some kind of
|> pointer to an instance that allows the findnext function (which ftw has
|> no equivalent for) to work. If that is so, then it would be possible to
|> do two or more separate tree walks across different trees at in parallel
|> at the same time. The ftw function cannot do that due to the way it is
|> designed as a callback function.
|
| Huh? Makes no sense to me. Each invocation of the callback has its own
| stack of parameters.
When letting the function do the tree walk, it does the iteration itself.
It is iterating over ONE tree. If you need to do parallel iteration over
two or more trees, ftw() cannot do it for you. When you call ftw(), you
give it ONE path, ONE function, and a maximum descriptor count.
By contrast, my ftr functions create an object instance for each path you
want to iterate/recurse/walk through. It returns with a pointer to that
object. Call it many times and get many pointers to many instances if you
want, for different paths, or even for the same path. Then you call other
functions to step through whichever instance you refer to in that call,
and also call functions to extract info about the most recently landed
file or filesystem object. You can get the path relative to the path
that created the instance, or with that path, or even a full path from
root. You can get stat data, or even have it opened for you. It also
provides a repeat of directories (with a different result code) when it
ascends back up (so you can choose to process a directory before and/or
after the descent). If you want to parallel iterator two trees, you
create an ftr instance for each, and then switch back and forth between
which you want to step. You can keep them in name sync by calling the
one that has the lower collating name to step it to the next path.
| Oh, I can see where you'd have trouble copying filenames into a global
| object, though.
Actually, that would not be a problem.
|
|> So I suggest that ftw is not a good equivalent. I have argued in the
|> past that ftw should even be depricated and eventually removed. There
|> are better APIs available, including fts that is already included, even
|> though it is rather difficult to use due to so many options, as well as
|> my own ftr functions. You can browse the source code of these functions
|> directly at: http://libh.slashusr.org/source/ftr/src/lib/h/
|> Do be sure to look at the ftr_header.h file since a number of macros
|> that would be used like functions are defined there.
|
| Thanks for the heads up on fts and ftr. Just make sure you don't name
| the next version "ftp" <grin>.
I'll watch out for that :-)
I do have a replacement for ftp in mind. Maybe I might even implement it.
I was going to call it fxp.
--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-200...@ipal.net |
|------------------------------------/-------------------------------------|