4.2bsd eof flag in stdio
Ben Goldfarb
Why did Berkeley change stdio such that typing ^D (or whatever EOF character
one is using) on stdin causes that stream to reflect eof until a clearerr()
is done? Has this been discussed here before? If so, I apologize for
further belaboring the issue.
In any case, what is the correct approach to this problem? Obviously, we
can't expect the authors of programs that have been distributed with UNIX
since V7 to have provided for Berkeley's change; as it stands I've found
that addbib and learn are both broken because of the continual EOF. So I
patched
if (feof(stdin))
clearerr(stdin);
into both programs. I'm sure more are affected. Alternatively, I could have
"fixed" stdio, but how many Berkeley programs make use of this "feature?"
I'd appreciate some net wisdom on the subject.
--
Ben Goldfarb
University of Central Florida
uucp: {duke,decvax,princeton}!ucf-cs!goldfarb
ARPA: goldfar...@csnet.relay
csnet: goldfarb@ucf
Geoff Kuenning
>Why did Berkeley change stdio such that typing ^D (or whatever EOF character
>one is using) on stdin causes that stream to reflect eof until a clearerr()
>is done? Has this been discussed here before? If so, I apologize for
>further belaboring the issue.
>
>In any case, what is the correct approach to this problem?
We did this when I was at DEC because that's the way a file behaves, and it
is frequently easier to write a program to read the EOF twice. For example:
while ((ch = getchar ()) != EOF)
switch (ch)
{
case '\\':
switch (ch = getchar ())
{
case EOF:
break;
}
break;
}
Here, reading the EOF twice is a convenient way to handle the loop exit.
(Yes, there are other ways, notably using a goto. But in more complex code
this approach may be the cleanest). I never like assuming that I can unget
an EOF character (although it works on some systems).
One can also make a persuasive argument for the advantages of the other
approach, but I prefer this way because of consistency.
--
Geoff Kuenning
First Systems Corporation
...!ihnp4!trwrb!desint!geoff
Bill Shannon
> Why did Berkeley change stdio such that typing ^D (or whatever EOF character
> one is using) on stdin causes that stream to reflect eof until a clearerr()
> is done? Has this been discussed here before? If so, I apologize for
> further belaboring the issue.
>
> In any case, what is the correct approach to this problem? Obviously, we
> can't expect the authors of programs that have been distributed with UNIX
> since V7 to have provided for Berkeley's change; as it stands I've found
> that addbib and learn are both broken because of the continual EOF. So I
> patched
> if (feof(stdin))
> clearerr(stdin);
> into both programs. I'm sure more are affected. Alternatively, I could have
> "fixed" stdio, but how many Berkeley programs make use of this "feature?"
> I'd appreciate some net wisdom on the subject.
The change was made by Sun and bought back by Berkeley. I believe this
has been discussed on the net before. The change actually fixes another
bug. The bug was that without this change programs using fread on terminals
would never report an EOF condition to the user because internally fread
would just swallow the EOF and return a short record and the next fread
would go on reading past the EOF. We actually ran into this bug in some
existing program, I forget which one. Unfortunately, not all the programs
which depended on the old behaviour were fixed.
Bill Shannon
Sun Microsystems, Inc.
Doug Gwyn <gwyn>
> has been discussed on the net before. The change actually fixes another
> bug. The bug was that without this change programs using fread on terminals
> would never report an EOF condition to the user because internally fread
> would just swallow the EOF and return a short record and the next fread
> would go on reading past the EOF. We actually ran into this bug in some
> existing program, I forget which one. Unfortunately, not all the programs
> which depended on the old behaviour were fixed.
fread() returns 0 if there are 0 characters left in the terminal
input queue when the ^D is typed. What would you have it do?
Contrary to popular misconception, ^D is NOT an "EOF" character;
rather, it marks a delimiter for input canonicalization. If all
previous input has been consumed and a ^D is typed, then read()
returns a count of 0. This is often interpreted as EOF. If there
is some uncanonicalized input and ^D is typed, it acts much like
NEWLINE except of course no \n is appended.
If the 4.2BSD fread() was buggy, it should have been fixed rather
than introducing a significant incompatibility with other STDIOs.
Spencer W. Thomas
>fread() returns 0 if there are 0 characters left in the terminal
>input queue when the ^D is typed. What would you have it do?
that this would terminate input reading, but it does not -- you must
type another ^D to finish it off.
>
>Contrary to popular misconception, ^D is NOT an "EOF" character;
>rather, it marks a delimiter for input canonicalization. If all
>previous input has been consumed and a ^D is typed, then read()
>returns a count of 0. This is often interpreted as EOF. If there
>is some uncanonicalized input and ^D is typed, it acts much like
>NEWLINE except of course no \n is appended.
>
states that ^D is the *end-of-file* character. Perhaps the
documentation (unchanged since my memory) is "buggy"?
>If the 4.2BSD fread() was buggy, it should have been fixed rather
>than introducing a significant incompatibility with other STDIOs.
This bug is in ALL versions of fread (and getchar, and ...) *except*
4.2.
=Spencer
Doug Gwyn <gwyn>
> that this would terminate input reading, but it does not -- you must
> type another ^D to finish it off.
This is just what I expect. Why should the first ^D terminate input
reading, since the read will return 3 characters at that point?
> This is, of course, a matter of opinion, but all the documentation
> states that ^D is the *end-of-file* character. Perhaps the
> documentation (unchanged since my memory) is "buggy"?
Yup. Kernighan & Pike got it right in their book.
> This bug is in ALL versions of fread (and getchar, and ...) *except*
> 4.2.
The UNIX System V Release 2.0 fread() acts as I originally described,
which is what I would expect. In any case, judging from the number of
times people have had problems caused by this change, it was not a wise
move.
Bill Shannon
> input queue when the ^D is typed. What would you have it do?
Try this program on your favorite version of stdio:
#include <stdio.h>
char buf[256];
main()
{
register int n;
while (n = fread(buf, 1, sizeof buf, stdin))
fwrite(buf, 1, n, stdout);
printf("got EOF\n");
}
Run it and type (e.g.):
testing 1 2 3
^D
another test
Where ^D is your EOT character. If the program terminates
when you type ^D then your stdio works properly. The 4.1
version of stdio would "eat" the ^D and echo the first and
third lines. It would only terminate if you typed ^D twice
in a row.
> If the 4.2BSD fread() was buggy, it should have been fixed rather
> than introducing a significant incompatibility with other STDIOs.
Making EOF sticky was the fix. It seemed like the right thing to
do; the incompatibility was unfortunate. If you have a fix to
fread (filbuf, actually) that both fixes this bug and avoids the
incompatibility then please send it to me and/or post it to the
net. If this works properly in System V I would be interested to
hear that as well.
Ron Natalie
Looking in your beloved System V manuals you will find under READ(2):
A value of zero is returned when end-of-file has been reached.
and
When attempting to read a file associated with a tty that has
no data currently available ... the read will block until the
data becomes available.
And then looking at the documentation for the TTY driver, where is it
oh yes, it's called TERMIO and it's in the system administrators manual.
Of course, no ordinary user would ever want to change his terminal modes.
A line is delimited by a new-line (ASCII LF), an end-of-file
(ASCII-EOT), or an end-of-line character.
EOF - may be used to generate an end-of-file from a terminal.
Thus if there are no characters waiting, which is to say EOF
occurred at the beginning of line, zero characters will be
passed back, which is the standard end-of-file indication.
What this implies is the zero return from TTY reads are END-OF-FILE
and should be treated as such. It is possible to continue reading
past end of file on some devices such as TTY and Magtape, but that
doesn't mean you shouldn't handle EOF properly.
Fread states
Fread stops appending bytes if an end-of-file or error
condition occurs.
Ferror states
Feof returns non-zero when EOF has previously been detected
reading the named input stream.
Clearerr resets the error indicator and EOF indicator to zero.
It is obvious from this, that no distinction is made of EOT chars meaning
anything but the absolute end-of-file on TTY. If you were attempting to
write a Stdio using the definitions in the manual, you would have to implement
it this way.
You need to stop defining UNIX by whatever bugs AT&T has and penalize
Berkeley because they have fixed a legitimate bug in the original UNIX
code.
-Ron
Like I'm from the Mystic Valley.
Doug Gwyn <gwyn>
This is not the same as saying that EOT "means" EOF. Only in certain
contexts does it have that effect, as the TERMIO(7) manual entry says.
> What this implies is the zero return from TTY reads are END-OF-FILE
> and should be treated as such. It is possible to continue reading
> past end of file on some devices such as TTY and Magtape, but that
> doesn't mean you shouldn't handle EOF properly.
Agreed.
> Fread states
> Fread stops appending bytes if an end-of-file or error
> condition occurs.
And so it does! But this is on that call, not necessarily on future
calls. This feature works as advertised on UNIX System V Release 2.0.
> Ferror states
> Feof returns non-zero when EOF has previously been detected
> reading the named input stream.
> Clearerr resets the error indicator and EOF indicator to zero.
Again, this is the way it does work. The EOF is "latched" until
cleared, but fread() can read past EOF if there is data there.
I'm all for bugs being fixed, if they are really bugs and not just
different ideas about what should be happening. Perhaps 4.2BSD and
UNIX System V now agree about EOF behavior; that would be a pleasant
change.
Doug Gwyn <gwyn>
>
> char buf[256];
>
> main()
> {
> register int n;
>
> while (n = fread(buf, 1, sizeof buf, stdin))
> fwrite(buf, 1, n, stdout);
> printf("got EOF\n");
> }
>
> Run it and type (e.g.):
>
> testing 1 2 3
> ^D
> another test
>
> Where ^D is your EOT character. If the program terminates
> when you type ^D then your stdio works properly. The 4.1
> version of stdio would "eat" the ^D and echo the first and
> third lines. It would only terminate if you typed ^D twice
> in a row.
Thanks for the example, Bill. I guess we disagree about what is
expected here. The "EOFish" nature of the input is reflected in
fread()'s short return count; as expected the 0-length read forces
fread() to return prematurely. I see no reason for it to "stick"
at EOF, though. The programmer certainly can tell that he is at
EOF from the short count. Continuing to read the stream is a
programming error (that happens to work on "ordinary" files,
unless they are being dynamically appended to), and more than a bit
sloppy besides (just like the internals of most UNIX utilities).
I see the argument for the other interpretation; I just don't
agree with it.
Geoff Kuenning
>rather, it marks a delimiter for input canonicalization. If all
>previous input has been consumed and a ^D is typed, then read()
>returns a count of 0. This is often interpreted as EOF. If there
>is some uncanonicalized input and ^D is typed, it acts much like
>NEWLINE except of course no \n is appended.
-Doug Gwyn
Contrary to popular misconception, neither the design of the Unix kernel nor
its documentation was handed down on stone tablets from on high. I don't
really care whether Thompson and Ritchie chose to describe the behavior of
the original Unix TTY driver as "EOF" or "canonicalization". I strongly
suspect that their motivation was to describe the behavior of the code they
actually wrote, and the code was written for convenient implementation.
We need a way to indicate "end of data" to a program reading TTY input. It is
convenient for programmers to consider "end of file" as "end of data" when
reading file input. Since redirection of stdin is one of Unix's great
features, it is thus reasonable to simply provide a way for a TTY to indicate
"end of file". If T&R implemented it sloppily and documented it accurately,
that is no reason for us to slavishly follow their lead.
Once you decide to have ^D truly mean "end of file", it is only reasonable
to make it operate like a true EOF. That means that multiple reads return
multiple EOF indications, just like a disk. The original implementation
can be extremely disconcerting--I had a program a few days ago that wanted
two EOF's to terminate. It tested fine from a file, but "hung" when I
typed in the input and terminated it with ^D.
The fact that some programs have in the past misinterpreted this bug as a
feature and made use of it is unfortunate, but something we will have to
live with. It is just not that hard to grep for "EOF" and add "clearerr"
calls. In any case, any program that was doing this was already providing
incompatible behavior between files and TTY's. That's what you get when
you special-case TTY input :-).
Brandon Allbery
. . .
EOT (Control-D) may be used to generate an end of file from
a terminal. When an EOT is received, all the charac-
ters waiting to be read are immediately passed to the
program, without waiting for a new-line, and the EOT is
discarded. Thus if there are no characters waiting,
which is to say the EOT occurred at the beginning of a
line, zero characters will be passed back, and this is
the standard end-of-file indication.
\this is in the system manual; \i'd suggest both you
and \berkeley look it up (in a v7 manual if necessary).
fread() was \n\o\t designed for terminal \i/\o.
--bsa
--
Brandon Allbery @ North Coast Xenix | the.world!ucbvax!decvax!cwruecmp!
6504 Chestnut Road, Independence, Ohio | {atvax!}ncoast!{tdi1!}bsa
(216) 524-1416 \ 44131 | E1...@CSUOHIO.BITNET (friend's acct.)
---------------------------------------+---------------------------------------
Forgive; we just had a system crash & lost a month's worth of work and patches.
Robert Elz
|
| > This is, of course, a matter of opinion, but all the documentation
| > states that ^D is the *end-of-file* character. Perhaps the
| > documentation (unchanged since my memory) is "buggy"?
|
| Yup. Kernighan & Pike got it right in their book.
|
Rarely does anyone play into my hands quite no nicely. Now that
we have K&P cited as the absolute authority on this issue, I
will proceed to quote from page 204.
Structurally, readslow is identical to cat except that
it loops instead of quitting when it encounters the current
end of the input. It has to use low-leval I/O because the
standard library routines continue to report EOF after the
first end of file.
This immediately precedes the listing of the "readslow" program,
which is the authorised version of "tail -f" according to the
gosple of St Pike.
I'm not sure which particular version of "the standard library
routines" they were referring to - this was written before 4.2
was released. I always assumed that V8 had fixed the bug as
well, but I was (not too long ago) told that this was not so.
Would you care to clarify rob?
The above inclusion (from pervious articles) is, of course,
completely irrelevant to the original discussion under this
subject line. It makes absolutely no difference what ^D from
the terminal really does, or does not do. What is important,
is that stdio returns EOF from a getchar(), fread(), scanf()
or whatever. Not a zero length read, EOF. (And as EOF is
actually returned to mean a few other things, there is this
nifty macro "feof" that you can use to verify that this
really was "end of file").
I don't think its at all unreasonable for "end of file" to
be a "sticky" condition,
Kernighan & Pike got it right in their book.
Finally, it seems that their are two vocal groups of "anti-4.2"
people out there. There seems to be one group that complains
bitterly about all the "bugs" berkeley introduced, and all the
things that they "broke", and a second group that complains
bitterly about all the "bugs" left in the code, and the things
that weren't done. What's most amazing is that it seems often
that the most vocal members of each group are the same people.
Rather a double standard - they didn't fix the bugs that make
my life difficult, 'cause I have to fix them to run their code
on my hardware, but they did fix all the bugs I was relying on ...
Can we end this useless discussion now, and allow it to die
until someone else new "discovers" it again (in about a week)?
Robert Elz decvax!mulga!kre
Bill Shannon
> expected here. The "EOFish" nature of the input is reflected in
> fread()'s short return count; as expected the 0-length read forces
> fread() to return prematurely. I see no reason for it to "stick"
> at EOF, though. The programmer certainly can tell that he is at
> EOF from the short count. Continuing to read the stream is a
> programming error (that happens to work on "ordinary" files,
> unless they are being dynamically appended to), and more than a bit
> sloppy besides (just like the internals of most UNIX utilities).
If you think of fread as the stdio equivalent of read, and you are
prepared to handle input from a terminal, you will not think of a
short return count as cause for alarm. Certainly the manual page
gave you no reason to think otherwise.
Also, the manual said fread would return NULL on EOF. I've clearly
presented an example where it did not return NULL on EOF. We
considered that a bug, in the manual or in the code, and we chose the
code. The System V Release 2 manual page for fread has been rewritten
so that it corresponds to what the (non-4.2) code actually does.
This is just another example of the inconsistencies between the UNIX
manuals and the code. One group chose to fix the code ("people have
been programming according to the manual") while another chose to
fix the manual ("no one reads the manuals anyway, the CODE defines
UNIX").
> I see the argument for the other interpretation; I just don't
> agree with it.
The only good argument against the change was compatibility. That
may be a strong enough reason to change it back, now that AT&T has
clarified the operation of fread.
Bill Shannon
Sun Microsystems
Doug Gwyn <gwyn>
> >rather, it marks a delimiter for input canonicalization.
>
>
> We need a way to indicate "end of data" to a program reading TTY input. ...
Geoff, I think you missed the point: ^D (or whatever) from a terminal
DOES act like EOF if there is nothing between the previous delimiter
and this one, since read() will return a count of 0 on that record.
But I have made good use of the more general behavior of ^D in forcing
non-newline terminated input to the reading process.
The only reason repeated reading of an ordinary (disk) file keeps
returning 0 bytes (NOT "EOF"; there is no such thing in UNIX) is
that the file size is static. If the file is being appended to by
some other process, then continued reading should return data AFTER
the original "end of file". The same applies to magtape and terminals.
This is not only reasonable, it is quite useful.
I much prefer the thoughtful design of UNIX over the attempts to make
it look "safe and ordinary".
Whatever program you had that required two successive 0-length reads
("EOF" indication, by convention) to detect end of input was simply
WRONG. (Some old-time Pascal programmers may recognize the problem.)
Instead of trying to change UNIX by reducing its generality, why not
fix the erroneous program. There is no excuse for such sloppiness.
Chris Torek
things like
% cat -u
foo^D
exit. Not true! If you type
% cat -u
foo
^D
(assuming ^D is your EOF character) *then* cat will exit, but for the
former, it will print "foo" and keep reading. One more ^D (unless
preceded by other text) will cause it to terminate.
--
(This line accidently left nonblank.)
In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (301) 454-7690
UUCP: {seismo,allegra,brl-bmd}!umcp-cs!chris
CSNet: chris@umcp-cs ARPA: chris@maryland
Geoff Kuenning
>Geoff, I think you missed the point: ^D (or whatever) from a terminal
>DOES act like EOF if there is nothing between the previous delimiter
>and this one, since read() will return a count of 0 on that record.
>But I have made good use of the more general behavior of ^D in forcing
>non-newline terminated input to the reading process.
No, I didn't miss the point. I happen to think that the stdio package
should not embed features of UN*X into its assumptions about operating
systems. There are lots more operating systems with "hard" EOF's than with
soft ones. If you want to make use of the general behavior of ^D, write your
program to either use clearerr() or use UNIX I/O.
>The only reason repeated reading of an ordinary (disk) file keeps
>returning 0 bytes (NOT "EOF"; there is no such thing in UNIX) is
>that the file size is static. If the file is being appended to by
>some other process, then continued reading should return data AFTER
>the original "end of file". The same applies to magtape and terminals.
>This is not only reasonable, it is quite useful.
As to UNIX having EOF, try looking in read(2). In any case stdio is not UNIX,
and grepping stdio.h for EOF will succeed. The features you talk to about
(simultaneous read and write, for example) are not available on all OS's.
>I much prefer the thoughtful design of UNIX over the attempts to make
>it look "safe and ordinary".
Good, then use read(2) and write(2). Stdio is explicitly a compatibility
package. As such, it _ s_ h_ o_ u_ l_ d be safe and ordinary.
>Whatever program you had that required two successive 0-length reads
>("EOF" indication, by convention) to detect end of input was simply
>WRONG. (Some old-time Pascal programmers may recognize the problem.)
>Instead of trying to change UNIX by reducing its generality, why not
>fix the erroneous program. There is no excuse for such sloppiness.
I can do without the snottiness, Doug. I mentioned in my original posting
that I had already fixed the bug. My point was that an inconsistency in the
way stdio (_ n_ o_ t UNIX) is implemented caused the program to behave strangely
only when reading from a terminal.
It is unfortunate that this bug stayed in the system for so long that some
people mistook it for a feature. But that doesn't mean we shouldn't fix it.
--
Geoff Kuenning
...!ihnp4!trwrb!desint!geoff
br...@ism780.uucp
> ... Try this program on your favorite version of stdio:
>
> #include <stdio.h>
> char buf[256];
> main()
> {
> register int n;
> while (n = fread(buf, 1, sizeof buf, stdin))
> fwrite(buf, 1, n, stdout);
> printf("got EOF\n");
> }
>
> when you type ^D then your stdio works properly. ...
>
> ... If you have a fix to
> fread (filbuf, actually) that both fixes this bug and avoids the
> incompatibility then please send it to me and/or post it to the
> net. If this works properly in System V I would be interested to
> hear that as well.
I believe your test program doesn't produce the desired results because
it's buggy, not stdio. Try the following on a system that doesn't have a
buggered fread(), notice the call to feof I've inserted:
#include <stdio.h>
char buf[256];
main()
{
register int n;
while (!feof(stdin) && (n = fread(buf, 1, sizeof buf, stdin)))
fwrite(buf, 1, n, stdout);
printf("got EOF\n");
}
I've tested this on our VAX IS/3 system (System III stdio) and with our
vanilla SystemV stdio. Both versions produced the desired (i.e., correct)
behaviour.
Bruce Adler {sdcrdcf,uscvax,ucla-vax,vortex}!ism780!bruce
Interactive Systems decvax!yale-co!ima!bruce
j...@ism780b.uucp
>
>#include <stdio.h>
>
>char buf[256];
>
>main()
>{
> register int n;
>
> while (n = fread(buf, 1, sizeof buf, stdin))
> fwrite(buf, 1, n, stdout);
> printf("got EOF\n");
>}
>
>
>testing 1 2 3
>^D
>another test
fread() is not read(). Read() from a terminal is delimited by the newline
character, so that an EOF is always determined by a read that returns 0.
No such guarantee is offered by fread; show me the manual page for fread
that says that 0 is returned upon EOF! Had you used an fgets or getc loop,
the documentation states that NULL (fgets) or EOF (getc) indicates EOF on
the stream, and you can depend on that. All you can depend on with
fread is feof(). Thus your program is wrong, and rather than fix it you
broke the library.
>It seemed like the right thing to
>do; the incompatibility was unfortunate.
That is a pretty clear statement of BSD philosophy; it causes some problems.
>>fread() returns 0 if there are 0 characters left in the terminal
>>input queue when the ^D is typed. What would you have it do?
>The problem is if you type 'foo^D' with no newline. You would expect
>that this would terminate input reading, but it does not -- you must
>type another ^D to finish it off.
As an experienced UNIX user who has read tty(4) [termio(7) in SysV],
I certainly would not expect that.
>>Contrary to popular misconception, ^D is NOT an "EOF" character;
>>rather, it marks a delimiter for input canonicalization. If all
>>previous input has been consumed and a ^D is typed, then read()
>>returns a count of 0. This is often interpreted as EOF. If there
>>is some uncanonicalized input and ^D is typed, it acts much like
>>NEWLINE except of course no \n is appended.
>>
>This is, of course, a matter of opinion, but all the documentation
>states that ^D is the *end-of-file* character. Perhaps the
>documentation (unchanged since my memory) is "buggy"?
It of course *is not* a matter of opinion, and while the documentation
calls ^D the EOF character, the formal behavior described in the documentation
is less naive than the name:
EOF (Control-d or ASCII EOT) may be used to generate an end-of-file from
a terminal. When received, all the characters waiting to be read are
immediately passed to the program, without waiting for a new-line, and
the EOF is discarded. Thus, if there are no characters waiting, which
is to say *the EOF occurred at the beginning of a line*, zero
characters will be passed back, which is the standard end-of-file
indication.
(That is the >=SysIII text; the BSD text merely says that newline or ^D
terminate a line being read in cooked mode; nothing anywhere says
that simply entering a ^D will cause an end-of-file indication anywhere).
When discussing fine points of documentation, it is more accurate and less
embarrassing to use your eyeballs, not your memory. When something is
claimed to be a popular misconception, you should not be so arrogant as
to assume that you are not subject to such misconceptions without
verifying it.
>>If the 4.2BSD fread() was buggy, it should have been fixed rather
>>than introducing a significant incompatibility with other STDIOs.
>This bug is in ALL versions of fread (and getchar, and ...) *except*
>4.2.
Do you consider it a bug to be able to read() from a terminal after getting
an end-of-file indication? The behavior of fread was consistent with the
documentation. Changing it, whether desirable or not, is a change in
functionality. A change can only be considered a bug fix if it brings into
line behavior previously out of line with the documentation.
-- Jim Balter, INTERACTIVE Systems (ima!jim)
j...@ism780b.uucp
>anything but the absolute end-of-file on TTY.
Quite wrong. As you quoted, EOT terminates an input line, and *if that line
is empty*, the return value is zero, and the *zero return value from read*
is interpreted as EOF, not receipt of EOT. But that is not really relevant
to fread. What is relevant is that the current call to fread stops when it
encounters EOF; NOWHERE DOES IT SAY THAT FREAD RETURNS ZERO UPON END-OF-FILE.
It doesn't unless an EOF is encountered when fread tries to read its first
byte. Nowhere does it say that EOF latches, any more than it does for read.
It is the nature of UNIX terminals that you can read past the EOF;
that is why fread behaved as it does, totally consistent with the
documentation. To quote Bill Shannon,
"The bug was that without this change programs using fread on terminals
would never report an EOF condition to the user because internally fread
would just swallow the EOF and return a short record and the next fread
would go on reading past the EOF."
But that is exactly what fread should do: return a short record
(as documented; read returns 0, as documented) and go on reading
past the EOF on the next fread (just as read goes on reading beyond
EOF). Only improperly written programs that erroneously assume that
fread signals EOF with a zero return value (it doesn't;
it isn't documented to) have the "bug".
read is different from fread because it is delimited by newline, so that
EOT at the beginning of a line always causes a zero return, because the
first character of a line must be the first character read (although not
necessarily vice versa); this simply isn't true of fread.
For routines for which it is true, such as fgets or getc, then the return
value can be used to detect EOF.
You cannot show that the SysV fread is wrong by quoting the *read*
documentation, especially without understanding why the two are different.
Guy Harris
It isn't just the nature of UNIX terminals. Some DEC OSes use the same
behavior; EDT terminates input mode with a ^Z, their EOF character.
Actually, quoting the VMS manuals, "CTRL/Z - Echoes ^Z when CTRL/Z is
typed as a *read terminator*. *By convention*, CTRL/Z constitutes
end-of-file." This implies (although it may not be the case) that ^Z
works in VMS exactly like ^D does in UNIX. This is worth pointing out,
since it was stated in an earlier article that there are more OSes with
"hard" rather than "soft" EOFs. I hope their EOFs aren't too hard; most
systems I've seen will let you type in a bunch of text as input to a program
and type your favorite EOF character and end input to that program without
ending input to all programs that run from that terminal during that session.
(Admittedly, most systems I've seen are either UNIX or DEC OSes.)
By the way, I saw a later version of "stdio" for 4.2 that looked like
it had the change rescinded; was this the case? (In which case, a lot of
this discussion is somewhat moot.)
Guy Harris
{seismo,ihnp4,allegra}!rlgvax!guy
Kenneth Almquist
> feature and made use of it is unfortunate, but something we will have to
> live with.
Arggh!
The 4.2 BSD manual page for getchar states that, "These functions return
the integer constant EOF at end of file...." Now for a standard UNIX
file, the end of file is the location immediately above the last byte
written. Thus if getchar returns EOF, something is appended to the
input file, and getchar is called again, getchar should not return EOF
because the file pointer is no longer at end of file.
The fact that the 4.2 BSD implementation of getchar handles EOF differ-
ently not only from all other variants of UNIX, but also from the way
its own documentation says it should handle EOF, is indeed "unfortunate,
but something we will have to live with."
If we may believe Bill Shannon, the whole issue arose because nobody
could figure out how to make a straightforward change to fread. The
change could have been implemented as follows:
1) Add a new flag named _EOF_PUSHED_BACK to stdio.h.
2) When _filbuf is called with this flag set, have it clear the flag
and return EOF.
3) While a new routine called pushback which is just like ungetc except
that pushback(EOF, fp) should set _EOF_PUSHED_BACK and return.
4) When fread encounter EOF and it has read at least one item, have it
call pushback(EOF, fp) before returning.
Kenneth Almquist
Spencer W. Thomas
>It isn't just the nature of UNIX terminals. Some DEC OSes use the same
>behavior; EDT terminates input mode with a ^Z, their EOF character.
>Actually, quoting the VMS manuals, "CTRL/Z - Echoes ^Z when CTRL/Z is
>typed as a *read terminator*. *By convention*, CTRL/Z constitutes
>end-of-file." This implies (although it may not be the case) that ^Z
>works in VMS exactly like ^D does in UNIX.
Well, I don't know about VMS, but in TOPS-20, if you type ^Z, you see
the EOF, even if you type it in the middle of the line (unlike ^D on
Unix). Personally, no matter what the manual says about "terminating
input" on ^D, and so on, I find that it is very confusing to naive users
that they must SOMETIMES type ^D twice, but other times, typing it once
suffices. Just because you have gotten used to the behaviour, doesn't
mean it's right.
=Spencer
Bill Shannon
"show me the manual page for fread that says that 0 is returned upon EOF!"
Here's an excerpt from the 4.2BSD man page for fread, V7 is identical:
DESCRIPTION
Fread reads, into a block beginning at ptr, nitems of data
of the type of *ptr from the named input stream. It returns
the number of items actually read.
. . .
DIAGNOSTICS
Fread and fwrite return 0 upon end of file or error.
He also says,
"A change can only be considered a bug fix if it brings into
line behavior previously out of line with the documentation."
Thank you, Jim, for justifying our change. It seems apparent from your
argument that it was System III/V that did the wrong thing.
Jay Lepreau
> show me the manual page for fread that says that 0 is returned upon EOF!
> When discussing fine points of documentation, it is more accurate and less
> you should not be so arrogant as to assume that you are not subject to
> such misconceptions without verifying it.
> NOWHERE DOES IT SAY THAT FREAD RETURNS ZERO UPON END-OF-FILE.
Taking Jim's own humble advice on the use of eyeballs and arrogance
I found in the v7 manual under fread(3):
DIAGNOSTICS
Fread and fwrite return 0 upon end of file or error.
And so does the 4.2 manual, which is derived from 32v which is derived
from v7. Now, in Sys V (and Sys 3?), rather than change the code to
fix a bug they changed the documentation and removed that sentence.
Sun and UCB chose to fix the code. Fine. Arguments can be made both
ways. (Of course I have strong opinions as to which is preferable.)
However, the so-called issue of whether or not ^D other than at the
beginning of a line should mean EOF is a straw man, and is not at issue
(or shouldn't be, anyway). In any case it is orthogonal to the issue of
sticky-eof on ttys, and is just muddying the waters. It's about as
germane and likely to change as adding ^^ or BCD to C.
Jay Lepreau
Geoff Kuenning
>If we may believe Bill Shannon, the whole issue arose because nobody
>could figure out how to make a straightforward change to fread.
Bill said quite explicitly that the change arose because they wanted to make
the behavior of fread consistent. I am sure that Bill is capable of coming
up with the push_back_eof algorithm all by his little old self -- if, after
considering the design aspects of the situation, he decides that is the
behavior he wants.
If you intend to write portable software, don't assume you can continue
reading from a terminal after EOF. For my money, I would much rather pay
a small backwards-compatibility price to achieve a stdio implementation that
was truly portable.
In any case, most programs that expect to get more than one EOF from a
terminal are broken, because you will get different results if you redirect
from a file. Sure, there are special exceptions like slowread (aka tail -f
aka tra), but let's be honest, folks -- of all the files you access in a
day, how many do you access while they are growing? Normally, you make use
of existing, non-growing files, and a program expecting two EOF's from a
terminal will always get a null second file if it is redirected.
--
Geoff Kuenning
...!ihnp4!trwrb!desint!geoff
Henry Spencer
> Here's an excerpt from the 4.2BSD man page for fread, V7 is identical:
>
> Fread and fwrite return 0 upon end of file or error.
Not just a short count, mind you, but 0.
Kenneth Almquist
>>could figure out how to make a straightforward change to fread.
>
>Bill said quite explicitly that the change arose because they wanted to make
>the behavior of fread consistent.
That's what I said he said.
>I am sure that Bill is capable of coming
>up with the push_back_eof algorithm all by his little old self -- if, after
>considering the design aspects of the situation, he decides that is the
>behavior he wants.
You already stated the behavior that Bill wanted: he wanted to make the
behavior of fread match the description in the manual page. He did not
want to change the behavior of any other functions.
Of course Bill is capable of coming up with the push_back_eof algorithm
himself, but as it happens he did't. He asked in his posting how fread
could be made to correspond to the manual page description of it without
changing getc, and I answered him. None of this is intended as an attack
on Bill--any programmer is entitled to an occasional slip--but I wonder why
it wasn't caught before the release of 4.2.
>If you intend to write portable software, don't assume you can continue
>reading from a terminal after EOF.
And if I don't want to write software that is portable to anything other
than another UNIX system? And anyway, I have never heard of a system that
couldn't support reading on a terminal after EOF. Such a system would be
a bit awkward to use since every time you typed an EOF at your terminal
all programs, including the command processor, would presumably encounter
and EOF indication and you would be logged out.
>For my money, I would much rather pay
>a small backwards-compatibility price to achieve a stdio implementation that
>was truly portable.
Currently, stdio is not truly portable. Try to implement fseek on a
non-UNIX system some time. Stdio does hide differences between various
versions of UNIX and I am not suggesting that that should change.
>In any case, most programs that expect to get more than one EOF from a
>terminal are broken, because you will get different results if you redirect
>from a file.
Horrors, EMACS won't work if you redirect it's input to a file--I guess
we had better throw it out. Seriously, differences between UNIX variants
create problems for people. The idea that "they won't break very many
programs" is not a justification. Obviously nobody would have raised the
issue if no programs were affected.
I can appreciate Bill Shannon's postion on fread, but changing the
functioning of getc is a different issue.
Kenneth Almquist
Chris Torek
> we had better throw it out.
*Whose* Emacs won't work?
Doug Gwyn <gwyn>
> Fread and fwrite return 0 upon end of file or error.
Yes, it used to say that. It failed to clarify what would happen on
a "short read". Returning 0 in such a case is absolutely incompatible
with the DESIGN of fread(): that you tell it how many items you want
and it returns the number you got. Clearly the above portion of the
manual was not well thought out, since it led to this discussion.
AT&T has clarified this (and many similar oversights, ambiguities, and
confusion) in the manual; and in this particular case I think they did
it right (since it makes the function design make more sense than the
other interpretation).
ge...@desint.uucp
>And if I don't want to write software that is portable to anything other
>than another UNIX system? And anyway, I have never heard of a system that
>couldn't support reading on a terminal after EOF. Such a system would be
>a bit awkward to use since every time you typed an EOF at your terminal
>all programs, including the command processor, would presumably encounter
>and EOF indication and you would be logged out.
If you want to write non-portable software, use UNIX system calls. They
handle EOF in the UNIX way.
Just because you haven't heard of an operating system that has hard EOF's
doesn't mean one doesn't exist. Your presumption about logouts shows
a strong UNIX prejudice. *Very* few operating systems interpret EOF's to the
command processer as a logout indication. Furthermore, many operating
systems put the command processor in the kernel, so that an EOF delivered
to a user program is not at all the same as an EOF given to the command
processor. Indeed, this is frequently part of the reason they have "hard"
EOF's. (No, I don't like this design either -- shells should be user
processes. But such systems do exist.)
>Seriously, differences between UNIX variants
>create problems for people. The idea that "they won't break very many
>programs" is not a justification. Obviously nobody would have raised the
>issue if no programs were affected.
Yup, catching up with the real world is frequently painful. Check out the
heat that has risen over 6-character externals in the draft ANSI standard.
But in that case and this one, I would rather bite the bullet and do it the
way that will make life easier in the future.
BTW, I have an editor that is very similar to EMACS, and it does not object
at all if its descriptors are redirected to files. I added the feature
because I had a need for it.
--
Geoff Kuenning
...!ihnp4!trwrb!desint!geoff
b...@ncoast.uucp
+----------------
| > Here's an excerpt from the 4.2BSD man page for fread, V7 is identical:
| >
| > Fread and fwrite return 0 upon end of file or error.
|
Which is wrong. If you request 7 characters and it reads 4 before EOF,
you've either lost 4 characters or gotten 3 garbage characters, depending
on what fread returns and how your program deals with EOF. Sounds to
me like fread is wrong from square one.
--bsa
--
Brandon Allbery @ decvax!cwruecmp!ncoast!bsa (..ncoast!tdi1!bsa business)
6504 Chestnut Road, Independence, Ohio 44131 (216) 524-1416
<<<<<< An equal opportunity employer: I both create and destroy bugs :-) >>>>>>