[perl #26787] read reports wrong eof under high system load
Michael Bell
# Please include the string: [perl #26787]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org:80/rt3/Ticket/Display.html?id=26787 >
This is a bug report for perl from michae...@cms.hu-berlin.de,
generated with the help of perlbug 1.34 running under perl v5.8.0.
-----------------------------------------------------------------
If a Linux system runs with a high system load then it can happen
that "read" returns a 0 but it is no EOF. The perldocumentation
"man perlfunc" includes the statement that a zero only happens at
EOF. We tested the same situation with a Solaris system which
does not produce this error.
We start studying the POSIX documentation and found a small
change. In the early years the interpretation of read was that
a not positive number signals an error or EOF because read is
usually blocking I/O. The actual POSIX documentation only
requires a non-negative return value if anything is ok. EOF is
not ok and must be tested seperately.
Nevertheless it looks like the implementations use the not so
restrictive POSIX specs for different interpretations. Linux can
return a zero under high load and this is no eof. Solaris
implements full blocking I/O and only returns positive numbers
of characters. They never return zero characters as successful
read operation.
There are two possible solution. First you can simply change the
documentation but this is not more than declaring a bug as a
feature. The second solution is to read until a positive number
of characters was returned from the read syscall.
Best regards
Michael
-----------------------------------------------------------------
---
Flags:
category=core
severity=low
---
This perlbug was built using Perl v5.8.0 - Thu Mar 13 22:21:38 UTC 2003
It is being executed now by Perl v5.8.0 - Thu Mar 13 22:14:08 UTC 2003.
Site configuration information for perl v5.8.0:
Configured by root at Thu Mar 13 22:14:08 UTC 2003.
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
Platform:
osname=linux, osvers=2.4.20, archname=i586-linux-thread-multi
uname='linux d20 2.4.20 #1 smp thu oct 10 18:10:26 utc 2002 i686
unknown
unknown gnulinux '
config_args='-ds -e -Dprefix=/usr -Dusethreads -Di_db -Di_dbm -Di_ndbm
-Di_gdbm -Duseshrplib=true'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O3 --pipe',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing'
ccversion='', gccversion='3.3 20030226 (prerelease) (SuSE Linux)',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =''
libpth=/lib /usr/lib /usr/local/lib
libs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil
perllibs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.3.2'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic
-Wl,-rpath,/usr/lib/perl5/5.8.0/i586-linux-thread-multi/CORE'
cccdlflags='-fPIC', lddlflags='-shared'
Locally applied patches:
---
@INC for perl v5.8.0:
/usr/lib/perl5/5.8.0/i586-linux-thread-multi
/usr/lib/perl5/5.8.0
/usr/lib/perl5/site_perl/5.8.0/i586-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.0
/usr/lib/perl5/site_perl
.
---
Environment for perl v5.8.0:
HOME=/home/michael
LANG=de_DE@euro
LANGUAGE (unset)
LC_COLLATE=POSIX
LD_LIBRARY_PATH=/usr/local/ssl/lib:/usr/local/ssl/lib:/usr/local/ssl/lib:/usr/lo
cal/ssl/lib:/usr/local/ssl/lib::/usr/local/lib:/usr/local/lib:/usr/local/lib:/us
r/local/lib:/usr/local/lib
LOGDIR (unset)
PATH=/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome2/bin:/opt
/gnome/bin:/opt/kde3/bin:/opt/kde2/bin:/usr/lib/java/bin:/opt/gnome/bin:/usr/loc
al/bin:/usr/local/OpenOffice.org1.0/program:/usr/local/bin:/usr/local/OpenOffice
.org1.0/program:/usr/local/bin:/usr/local/OpenOffice.org1.0/program:/usr/local/b
in:/usr/local/OpenOffice.org1.0/program:/usr/local/bin:/usr/local/OpenOffice.org
1.0/program
PERL_BADLANG (unset)
SHELL=/bin/bash
--
-------------------------------------------------------------------
Michael Bell Email: michae...@cms.hu-berlin.de
ZE Computer- und Medienservice Tel.: +49 (0)30-2093 2482
(Computing Centre) Fax: +49 (0)30-2093 2704
Humboldt-University of Berlin
Unter den Linden 6
10099 Berlin Email (private): michae...@web.de
Germany http://www.openca.org
Kurt Starsinic
> If a Linux system runs with a high system load then it can happen
> that "read" returns a 0 but it is no EOF. The perldocumentation
> "man perlfunc" includes the statement that a zero only happens at
> EOF. We tested the same situation with a Solaris system which
> does not produce this error.
>
> We start studying the POSIX documentation and found a small
> change. In the early years the interpretation of read was that
> a not positive number signals an error or EOF because read is
> usually blocking I/O. The actual POSIX documentation only
> requires a non-negative return value if anything is ok. EOF is
> not ok and must be tested seperately.
>
> Nevertheless it looks like the implementations use the not so
> restrictive POSIX specs for different interpretations. Linux can
> return a zero under high load and this is no eof. Solaris
> implements full blocking I/O and only returns positive numbers
> of characters. They never return zero characters as successful
> read operation.
>
> There are two possible solution. First you can simply change the
> documentation but this is not more than declaring a bug as a
> feature. The second solution is to read until a positive number
> of characters was returned from the read syscall.
Thanks for pointing out the lack of clarity. However, note
that Perl's read() is documented _not_ to be the libc read() --
one must use Perl's sysread() to expect POSIX semantics.
I favor letting read() return 0 bytes when 0 bytes are available,
as long as the filehandle is non-blocking. Thus, I would recommend
changing the documentation rather than the implementation.
- Kurt
Ton Hospel
Michael Bell (via RT) <perlbug-...@perl.org> writes:
> If a Linux system runs with a high system load then it can happen
> that "read" returns a 0 but it is no EOF. The perldocumentation
> "man perlfunc" includes the statement that a zero only happens at
> EOF. We tested the same situation with a Solaris system which
> does not produce this error.
>
you have code to reproduce this ?
Michael Bell
No, I have no small script to do this. We found this problem during
testing a batch system for a PKI (OpenCA). The problem only happens if
you have more than 90 percent total system load.
I don't think that it is a Linux bug because this behaviour is compliant
to the POSIX specs. I think that Perl is not aware of the latest POSIX
specs (blocking read can return zero characters and this is correct).
BTW I think that the real bug is in the POSIX specs because there is not
explicitly written that a zero always signals EOF if it is a blocking
read on a regular file. I like the Solaris behaviour much more than the
Linux one's.
My problem is that this beahviour means that there is no real blocking I
/O. The only chance to find a EOF after a POSIX read is now
pos = lseek (fildes, 0, SEEK_CUR);
fstat (fildes, buf);
if (pos >= buf.st_size) ...
This only works for regular files. Again I think that a returned zero
should signal EOF but the POSIX specs doesn't define this and the
question is now is this wanted by the authors or not?
Michael
Michael Bell
why.
Kurt Starsinic via RT wrote:
> Thanks for pointing out the lack of clarity. However, note
> that Perl's read() is documented _not_ to be the libc read() --
> one must use Perl's sysread() to expect POSIX semantics.
I know this fact but the docs of Perl imply that a zero means EOF.
> I favor letting read() return 0 bytes when 0 bytes are available,
> as long as the filehandle is non-blocking. Thus, I would recommend
> changing the documentation rather than the implementation.
Ok, I'm not a Perl guru and I never looked into the source code. I only
need a consistent state. If the documentation is changed then I can be
sure about what I have to do (means using the function eof).
Ton Hospel
Michael Bell <michae...@cms.hu-berlin.de> writes:
> (Ton Hospel) via RT wrote:
>> In article <rt-3.0.8-26787-783...@perl.org>,
>> Michael Bell (via RT) <perlbug-...@perl.org> writes:
>>
>>>If a Linux system runs with a high system load then it can happen
>>>that "read" returns a 0 but it is no EOF. The perldocumentation
>>>"man perlfunc" includes the statement that a zero only happens at
>>>EOF. We tested the same situation with a Solaris system which
>>>does not produce this error.
>>>
>>
>> Personally I'd rather see this reported as a linux bug. Do
>> you have code to reproduce this ?
>
> No, I have no small script to do this. We found this problem during
> testing a batch system for a PKI (OpenCA). The problem only happens if
> you have more than 90 percent total system load.
>
> I don't think that it is a Linux bug because this behaviour is compliant
> to the POSIX specs. I think that Perl is not aware of the latest POSIX
> specs (blocking read can return zero characters and this is correct).
> BTW I think that the real bug is in the POSIX specs because there is not
> explicitly written that a zero always signals EOF if it is a blocking
> read on a regular file. I like the Solaris behaviour much more than the
> Linux one's.
My reading only mentions a rather obscure "If any portion of a regular
file prior to the end-of-file has not been written, read() returns bytes
with value 0."
Apart form that this seems appropiate:
"If O_NONBLOCK is clear, read() will block the calling thread until
some data becomes available"
and
"If the starting position is at or after the end-of-file, 0 will be returned."
But anyways, I don't think perl CAN work around this, since it cannot
just retry the read. How often should you retry ? And you could still be
unlucky. Nor would the semantics be right since perl doesn't KNOW what
kind of fildescriptor is really behind the thing being read. You don't
want to do multiple reads from e.g. a tty where the user expects ^D to
"close" it instead of the read just being restarted.
So even if it's allowed in POSIX (which I don't currently agree with),
it would be one of these (fortunately rare) places where POSIX fucked
up, since there is no sane way to handle this situation. And the linux
people DO fix up such cases, so if you report this, I'm quite sure it
will be seen as a bug and fixed (see e.g. http://www.ussg.iu.edu/hypermail/linux/kernel/0207.2/0242.html and the discussion around that for the case that
close in principle may leave the filedescriptor open in case of an error
return, also leading to untenable semantics, and therefore fixed as a bug)
>
> My problem is that this beahviour means that there is no real blocking I
> /O. The only chance to find a EOF after a POSIX read is now
>
> pos = lseek (fildes, 0, SEEK_CUR);
> fstat (fildes, buf);
> if (pos >= buf.st_size) ...
>
> This only works for regular files. Again I think that a returned zero
> should signal EOF but the POSIX specs doesn't define this and the
> question is now is this wanted by the authors or not?
>
That's not even very dependable for regular files if they are still
actively being written to and truncated.
Dave Mitchell
From reading the read(2) manpage (on RH 7.2 and Fedora Core 1),
it will onlt ever return 0 on EOF. For a non-blocking filehandle, this
is handled by EAGAIN, or if interrupted by a signal before any bytes can be
read, then by EINTR. If Linux *is* returning 0 before EOF, then it's
disobeying its own documentation. And if POSIX does santion this
(I havn't loooked), then POSIX is broken.
Does the OP have any actual proof that the Linux read() system
call ever returns 0 in a situation where it isn't EOF (as opposed to say
where the file is still being actively written to, so it was briefly at
EOF but isn't now)?
Dave.
--
"I do not resent critisism, even when, for the sake of emphasis,
it parts for the time with reality".
-- Winston Churchill, House of Commons, 22nd Jan 1941.
Nick Ing-Simmons
>(Ton Hospel) via RT wrote:
>> In article <rt-3.0.8-26787-783...@perl.org>,
>> Michael Bell (via RT) <perlbug-...@perl.org> writes:
>>
>>>If a Linux system runs with a high system load then it can happen
>>>that "read" returns a 0 but it is no EOF. The perldocumentation
>>>"man perlfunc" includes the statement that a zero only happens at
>>>EOF. We tested the same situation with a Solaris system which
>>>does not produce this error.
>>>
>>
>> Personally I'd rather see this reported as a linux bug. Do
>> you have code to reproduce this ?
>
>No, I have no small script to do this. We found this problem during
>testing a batch system for a PKI (OpenCA). The problem only happens if
>you have more than 90 percent total system load.
>
>I don't think that it is a Linux bug because this behaviour is compliant
>to the POSIX specs. I think that Perl is not aware of the latest POSIX
>specs (blocking read can return zero characters and this is correct).
Hmm, I thought I was aware of this and so would have made Perl aware
of it.
I am still not clear what is at fault though - is it that perl's read()
returns 0 or perl guts' calls to PerlLIO_read() (i.e. read(2)) fail
to account for zero return?
>BTW I think that the real bug is in the POSIX specs because there is not
>explicitly written that a zero always signals EOF if it is a blocking
>read on a regular file.
But given that we don't really want to have to fstat() to find out it is
a regular file and fctrl() to find out it is blocking I would rather
calls to read(2) handled zero return in a manner that was consistent
with use on a non-blocking socket/pipe/tty. Which should make the
corner case for regular file work too.
James E Keenan via RT
here?
Thank you very much.
Jim Keenan
---
via perlbug: queue: perl5 status: open
https://rt.perl.org:443/rt3/Ticket/Display.html?id=26787
James E Keenan via RT
Dave Mitchell
EOF, and no-one has produced a reproducible example of linux doing
anything other than that, so I say the ticket should be rejected.
--
I before E. Except when it isn't.