Questions on reuse of FORMAT statements and newer I/O options such as ACCESS='STREAM'
John
A bit of background ...
It seemed time to review whether a local library of pre-f90 I/O
routines
should be updated, as many more I/O options are now available as part
of the standards. As everyone was rethinking what we could do with the
routines, we got to the "table" routine. This routine basically dumps
out arrays in various formats (ready for import into spread sheets,
as HTML, MIF, latex, "pretty" text tables, ...).
The issues ...
Anyhow, the result is we are having a local difference of opinion on
when to expect a newline to be generated when re-using a format when a
single non-advancing I/O and/or implied DO's write statement
"exhausts"
the format; ESPECIALLY if formatted stream I/O is used (in particular,
should a WRITE(IO,*) be equivalent to calling the NEW_LINE() function
when writing to a stream?).
So, should every "table" in the output files be identical or not?
That's
my current contention; but I am finding not all compilers (let alone
people :>) agree with me.
As an example, when I use GFORTRAN and STREAM I/O the DO loop example
has
a null character (shown as ^@) where I expect a space, some compilers
write nothing when a WRITE(3f) with no values is encountered in STREAM
mode, and so on. ( Note I couldn't run this without commenting some of
it out
on some compilers I have, they aren't all upgraded to supporting
STREAM I/O yet.)
I attached the GFORTRAN output. The first set of numbered lines looks
just
like I expected (written to UNIT=6) (I think every time you START
reusing
a format that the equivalent of a New_LINE(3f) call is made, but not
if you
just reach the end of it). The last place I expected a surprise was in
the
simple DO loop writing one value at a time, but I got it when I used
formatted
stream I/O ( starting at the line numbered "30" in the second set of
tables).
Other than that, this particular compiler seemed to do what I thought.
Others
think there should be less newlines in the file.
Thanks!
EOF
#!/bin/sh
cat >missouri.f <<\EOF
!
ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
program testit
open(unit=11,file="mystream",access="stream",form="formatted")
! STUDY: writing arrays (especially of unknown size) using newer I/O
options
! free-format -vs- stream -vs- non-advancing -vs-
! format (with f08 * repeat count, :, f08- repeat count,
! created with internal write or //)
! ISSUE:
! interpretation of expected standard behavior
! CASE:
! simplified basic TABLE(3f) "pretty" print
call writeit(11) ! write formatted STREAM
call writeit(6) ! write formatted to stdout
! NOTE: when I change the order of the above lines, "mystream" is
empty using gfortran
stop
end
!
ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
subroutine writeit(io)
parameter (isize=23)
character*80 myfmt
real ra(isize)
do i10=1,isize
ra(i10)=i10
enddo
!-----------------------------------------------------------------------
myfmt= '(7(''|'',f8.2,'' ''):,''|'')'
myfmt= '(7(''|'',f8.2,'' '') ,''|'')'
call line(io,myfmt,':array;up to 7 numbers on a line ')
write(io,myfmt)ra
!-----------------------------------------------------------------------
call line(io,myfmt,':array;up to 7 numbers on a line, no
advance')
write(io,myfmt,advance='no')ra
write(io,*)
!write(io,'(a)')new_line()
!-----------------------------------------------------------------------
call line(io,myfmt,':implied DO;up to 7 per line')
write(io,myfmt)(ra(i),i=1,isize)
!-----------------------------------------------------------------------
call line(io,myfmt,':implied DO;up to 7 per line,no advance')
write(io,myfmt,advance='no')(ra(i),i=1,isize)
write(io,*)
!
=======================================================================
! DO LOOP -- one value at a time to get full control from no advance
!-----------------------------------------------------------------------
myfmt='(f8.2,1x,''|'')'
call line(io,myfmt,':DO loop, advance=''no''')
write(io,'(''|'')',advance='no')
do i20=1,isize
write(io,myfmt,advance='no')ra(i20)
if(mod(i20-1,7).eq.6)write(io,'(/,a)',advance='no')'|'
enddo
write(io,*)
!-----------------------------------------------------------------------
stop
end
!
ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
subroutine line(io,fmt,header) ! write break and header
character*(*) fmt,header
write(io,'(80a1,/,a,a,/,
80a1)') &
& ('=',i=1,80),fmt(:len_trim(fmt)),header,('-',i=1,80)
return
end
!
ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
EOF
(
exec 2>&1
rm -f missouri mystream
f90 missouri.f -o missouri
#./missouri |cat -v -e -t -n
./missouri |cat -v -t -n
echo XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
STREAM
#cat -v -e -t -n mystream
cat -v -t -n mystream
) >>$0
exit
EXAMPLE OUTPUT:
+ gfortran -I. -J. -static -Wuninitialized -g -O missouri.f -o
missouri
+ exit
1
================================================================================
2 (7('|',f8.2,' ') ,'|'):array;up to 7 numbers on a line
3 --------------------------------------------------------------------------------
4 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
5 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
6 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
7 | 22.00 | 23.00 |
8
================================================================================
9 (7('|',f8.2,' ') ,'|'):array;up to 7 numbers on a line, no
advance
10 --------------------------------------------------------------------------------
11 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
12 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
13 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
14 | 22.00 | 23.00 |
15
================================================================================
16 (7('|',f8.2,' ') ,'|'):implied DO;up to 7 per line
17 --------------------------------------------------------------------------------
18 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
19 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
20 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
21 | 22.00 | 23.00 |
22
================================================================================
23 (7('|',f8.2,' ') ,'|'):implied DO;up to 7 per line,no advance
24 --------------------------------------------------------------------------------
25 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
26 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
27 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
28 | 22.00 | 23.00 |
29
================================================================================
30 (f8.2,1x,'|'):DO loop, advance='no'
31 --------------------------------------------------------------------------------
32 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
33 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
34 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
35 | 22.00 | 23.00 |
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX STREAM
1
================================================================================
2 (7('|',f8.2,' ') ,'|'):array;up to 7 numbers on a line
3 --------------------------------------------------------------------------------
4 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
5 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
6 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
7 | 22.00 | 23.00 |
8
================================================================================
9 (7('|',f8.2,' ') ,'|'):array;up to 7 numbers on a line, no
advance
10 --------------------------------------------------------------------------------
11 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
12 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
13 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
14 | 22.00 | 23.00 |
15
================================================================================
16 (7('|',f8.2,' ') ,'|'):implied DO;up to 7 per line
17 --------------------------------------------------------------------------------
18 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
19 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
20 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
21 | 22.00 | 23.00 |
22
================================================================================
23 (7('|',f8.2,' ') ,'|'):implied DO;up to 7 per line,no advance
24 --------------------------------------------------------------------------------
25 | 1.00 | 2.00 | 3.00 | 4.00 | 5.00 | 6.00
| 7.00 |
26 | 8.00 | 9.00 | 10.00 | 11.00 | 12.00 | 13.00 |
14.00 |
27 | 15.00 | 16.00 | 17.00 | 18.00 | 19.00 | 20.00 |
21.00 |
28 | 22.00 | 23.00 |
29
================================================================================
30 (f8.2,1x,'|'):DO loop, advance='no'
31 --------------------------------------------------------------------------------
32 | 1.00^@| 2.00^@| 3.00^@| 4.00^@| 5.00^@|
6.00^@| 7.00^@|
33 | 8.00^@| 9.00^@| 10.00^@| 11.00^@| 12.00^@|
13.00^@| 14.00^@|
34 | 15.00^@| 16.00^@| 17.00^@| 18.00^@| 19.00^@|
20.00^@| 21.00^@|
35 | 22.00^@| 23.00^@|
================================================================================
System I made this run on:
CYGWIN_NT-6.0 urbanjs-PC 1.5.25(0.156/4/2) 2008-06-12 19:34 i686
Cygwin
================================================================================
This is the GFORTRAN command I used:
+ gfortran -I. -J. -static -Wuninitialized -g -O -v missouri.f
Driving: gfortran -I. -J. -static -Wuninitialized -g -O -v missouri.f
-lgfortranbegin -lgfortran
Using built-in specs.
Target: i686-pc-cygwin
Configured with: /gnu/gcc/package/gcc4-4.3.2-2/src/gcc-4.3.2/configure
--srcdir=/gnu/gcc/package/gcc4-4.3.2-2/src/gcc-4.3.2
--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin
--sbindir=/usr/sbin --libexecdir=/usr/sbin --datadir=/usr/share
--localstatedir=/var --sysconfdir=/etc --infodir=/usr/share/info
--mandir=/usr/share/man --datadir=/usr/share --infodir=/usr/share/info
--mandir=/usr/share/man -v --with-gmp=/usr --with-mpfr=/usr
--enable-bootstrap --enable-version-specific-runtime-libs
--with-slibdir=/usr/bin --libexecdir=/usr/lib --enable-static
--enable-shared --enable-shared-libgcc --enable-__cxa_atexit
--with-gnu-ld --with-gnu-as --with-dwarf2 --disable-sjlj-exceptions
--enable-languages=ada,c,c++,fortran,java,objc,obj-c++ --disable-
symvers
--enable-libjava --program-suffix=-4 --enable-libgomp --enable-libssp
--enable-libada --enable-threads=posix AS=/opt/gcc-tools/bin/as.exe
AS_FOR_TARGET=/opt/gcc-tools/bin/as.exe LD=/opt/gcc-tools/bin/ld.exe
LD_FOR_TARGET=/opt/gcc-tools/bin/ld.exe
Thread model: posix
gcc version 4.3.2 20080827 (beta) 2 (GCC)
COLLECT_GCC_OPTIONS='-I.' '-J.' '-static' '-Wuninitialized' '-g' '-O'
'-v' '-mtune=generic'
/usr/lib/gcc/i686-pc-cygwin/4.3.2/f951.exe missouri.f -ffixed-form
-quiet -dumpbase missouri.f -mtune=generic -auxbase missouri -g
-O -Wuninitialized -version -J. -I. -fintrinsic-modules-path
/usr/lib/gcc/i686-pc-cygwin/4.3.2/finclude -o
/cygdrive/c/Users/urbanjs/AppData/Local/Temp/ccENuupd.s
GNU F95 (GCC) version 4.3.2 20080827 (beta) 2 (i686-pc-cygwin)
compiled by GNU C version 4.3.2 20080827 (beta) 2, GMP version
4.2.4, MPFR version 2.4.1.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-
heapsize=131072
COLLECT_GCC_OPTIONS='-I.' '-J.' '-static' '-Wuninitialized' '-g' '-O'
'-v' '-mtune=generic'
/usr/lib/gcc/i686-pc-cygwin/4.3.2/../../../../i686-pc-cygwin/bin/
as.exe
-v -I. -o /cygdrive/c/Users/urbanjs/AppData/Local/Temp/ccJRZ3Su.o
/cygdrive/c/Users/urbanjs/AppData/Local/Temp/ccENuupd.s
GNU assembler version 2.18.50 (i686-pc-cygwin) using BFD version (GNU
Binutils) 2.18.50.20080625
COMPILER_PATH=/usr/lib/gcc/i686-pc-cygwin/4.3.2/:/usr/lib/gcc/i686-pc-
cygwin/4.3.2/:/usr/lib/gcc/i686-pc-cygwin/:/usr/lib/gcc/i686-pc-cygwin/
4.3.2/:/usr/lib/gcc/i686-pc-cygwin/:/usr/lib/gcc/i686-pc-cygwin/
4.3.2/:/usr/lib/gcc/i686-pc-cygwin/:/usr/lib/gcc/i686-pc-cygwin/
4.3.2/../../../../i686-pc-cygwin/bin/
LIBRARY_PATH=/usr/lib/gcc/i686-pc-cygwin/4.3.2/:/usr/lib/gcc/i686-pc-
cygwin/4.3.2/:/usr/lib/gcc/i686-pc-cygwin/4.3.2/../../../:/lib/:/usr/
lib/
COLLECT_GCC_OPTIONS='-I.' '-J.' '-static' '-Wuninitialized' '-g' '-O'
'-v' '-mtune=generic'
/usr/lib/gcc/i686-pc-cygwin/4.3.2/collect2.exe -Bstatic
--dll-search-prefix=cyg /usr/lib/gcc/i686-pc-cygwin/4.3.2/../../../
crt0.o
/usr/lib/gcc/i686-pc-cygwin/4.3.2/crtbegin.o
-L/usr/lib/gcc/i686-pc-cygwin/4.3.2 -L/usr/lib/gcc/i686-pc-cygwin/
4.3.2
-L/usr/lib/gcc/i686-pc-cygwin/4.3.2/../../..
/cygdrive/c/Users/urbanjs/AppData/Local/Temp/ccJRZ3Su.o -
lgfortranbegin
-lgfortran -lgcc -lgcc_eh -lcygwin -luser32 -lkernel32 -ladvapi32
-lshell32 -lgcc -lgcc_eh /usr/lib/gcc/i686-pc-cygwin/4.3.2/crtend.o
+ exit
Richard Maine
> Anyhow, the result is we are having a local difference of opinion on
> when to expect a newline to be generated when re-using a format when a
> single non-advancing I/O and/or implied DO's write statement
> "exhausts"
> the format; ESPECIALLY if formatted stream I/O is used (in particular,
> should a WRITE(IO,*) be equivalent to calling the NEW_LINE() function
> when writing to a stream?).
First, it makes no difference whether you are using stream or not. You
won't find a hint of any such difference anywhere in the standard.
Stream doesn't even change the default for the advance specifier. See
9.5.1.3 (ADVANCE= specifier in a data transfer statement) in F2003. The
last sentence is
"If this specifier is omitted from an input/output statement that
allows the specifier, the default is YES."
Note that there are no conditions relating to stream versus sequential.
The default is just YES. The condition about statements that allow the
specifier is for things like internal I/O.
Second, even if one explicitly specifies ADVANCE='no', that has no
effect on reversion of format. Advancing versus non-advancing I/O
affects only whether the record is terminated at the end of the output
statement. It has no effect on new records arising from explicit "/"
edit descriptors or reversion of format (or from writing newline
characters in stream output).
See the 3rd para after Note 10.5 in F2003. That's the para that
describes format reversion. Skipping over the bits that describe when
reversion happens and how it interacts with I/O list elements, we find
".. the file is positioned in a manner identical to the way it is
positioned what a slash edit descriptor is procesed..."
Note there are no conditions relating to advance='no, stream access, or
anything of the sort. It just has the effect of a slash. Look in the
section about slash. That would be 10.7.2. Not only doesn't it exclude
stream I/O, it specifically includes it.
"On output to a file connected for sequential or stream acess, a new
empty record is created following the current record."
I don't see how one could plausibly misread this stuff. Now if one were
to try to just guess the behavior, or possibly extrapolate from
nonstandard extensions to f77, one might come to different conclusions.
But I don't see much wiggle room on this in the standard.
Do note that there is a difference between format reversion (what you
are calling reuse) and just comming to the end of the i/o list. Format
reversion always causes a new record; there are no exceptions. Comming
to the end of the i/o list causes a new record depending on the setting
of advance (which does not depend on stream versus sequential).
--
Richard Maine | Good judgment comes from experience;
email: last name at domain . net | experience comes from bad judgment.
domain: summertriangle | -- Mark Twain
John
no exceptions"; backed up by
your chain of standard references seems to have quelled the dissent,
we will report the various
deviations by some compilers as bugs. As a footnote, some of the
confusion came from the f2008
section C.7.1, which uses the term "format rescan" instead of "format
reversion" and, as you guessed,
from extension behaviors such as "$" in a format and non-standard
stream I/O behavior. Plus, not
everyone had used reversion much (and some have ~50 years FORTRAN
experience).
On Dec 28, 12:55 am, nos...@see.signature (Richard Maine) wrote:
glen herrmannsfeldt
> Plus, not everyone had used reversion much (and some have ~50
> years FORTRAN experience).
I believe that reversion of the whole format has a reasonable
amount of use, but reversion on inner levels of parentheses
has much less usage.
In a post not so long ago, I remembered the new record on whole
format reversion, but forgot about the new record on inner
levels, which applies when there is not a repetition factor
on the parenthesized group.
It is convenient for printing arrays when you need a different format
for the first row than for succeeding rows.
I believe this all goes back to Fortran I.
-- glen
Richard Maine
> In a post not so long ago, I remembered the new record on whole
> format reversion, but forgot about the new record on inner
> levels, which applies when there is not a repetition factor
> on the parenthesized group.
Format reversion works the same in all cases. There aren't different
rules depending on whether or not the reversion goes back to the
beginning of the format. The rules about parens control where in the
format the reversion goes to, but they have no bearing on what happens
during reversion. You should not be thinking about these as different
cases at all.
Now what I get confused about is the distinction between formatted
stream and formatted sequential in general (not having to do with
formats or reversion). Seems to me that one could just forget about
formatted sequential and always use formatted stream in its place
without loosing much of anything, and gaining some minor options. But
then, I never did really figure out what the advantage of formatted
stream was anyway. Unformatted stream I'm a big fan of, but formatted
stream seems not to add much to me. I did hear some advocates explain
advantages, but the explanations didn't do much for me. (For example,
there were explanations about how necessary buffers caused problems with
long records for formatted sequential, but I never really understood how
stream did much that related to that issue.)
glen herrmannsfeldt
> glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
>> In a post not so long ago, I remembered the new record on whole
>> format reversion, but forgot about the new record on inner
>> levels, which applies when there is not a repetition factor
>> on the parenthesized group.
> Format reversion works the same in all cases. There aren't different
> rules depending on whether or not the reversion goes back to the
> beginning of the format. The rules about parens control where in the
> format the reversion goes to, but they have no bearing on what happens
> during reversion. You should not be thinking about these as different
> cases at all.
Yes, but the remembering of the rules is different. The OP said that
reversion wasn't used that much. In the case of whole format
reversion, it isn't so hard to remember. Whole format, whole record.
It works very well for printing out cases where all records are
the same. I know I have done that many times.
The case of reversion on inner parentheses is much less common.
The only one I can remember using was in a program I didn't write.
There is also the interesting difference between Fortran and PL/I
formatted (edit directed) output. In Fortran, descriptors that don't
use an I/O list item are still applied after the last list item,
where in PL/I they are not.
> Now what I get confused about is the distinction between formatted
> stream and formatted sequential in general (not having to do with
> formats or reversion). Seems to me that one could just forget about
> formatted sequential and always use formatted stream in its place
> without loosing much of anything, and gaining some minor options. But
> then, I never did really figure out what the advantage of formatted
> stream was anyway.
To me, it makes some sense in that stream is the way C does it.
Well, that and the character oriented unix file system. Personally,
I prefer out-of-band record indicators, but then I don't always
have that choice.
> Unformatted stream I'm a big fan of, but formatted
> stream seems not to add much to me. I did hear some advocates explain
> advantages, but the explanations didn't do much for me. (For example,
> there were explanations about how necessary buffers caused problems
> with long records for formatted sequential, but I never really
> understood how stream did much that related to that issue.)
I would have to go back to be sure, but it might have to do with
the T format descriptor. In record oriented I/O, where the buffer
is at least one record long, it is easy to position anywhere within
the record, even backwards. With stream, especially unbuffered,
(not so uncommon with unix) once a character is out you don't
expect to be able to change it. Thinking in terms of serial terminals
and paper tape readers and punches that can physically do character
at a time I/O, the distinction can be important.
With record oriented I/O, one can think in terms of an actual
buffer big enough for a whole record. (Independent of the actual
implementation.)
-- glen
Richard Maine
> Richard Maine <nos...@see.signature> wrote:
> > glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
> > Format reversion works the same in all cases. There aren't different
> > rules depending on whether or not the reversion goes back to the
> > beginning of the format. The rules about parens control where in the
> > format the reversion goes to, but they have no bearing on what happens
> > during reversion. You should not be thinking about these as different
> > cases at all.
>
> Yes, but the remembering of the rules is different.
I guess I don't understand what you are saying because that was exactly
my point - that you don't have to remember different rules because they
aren't different. If you try to remember it as two different sets of
(identical, as it happens) rules, then you are just making life harder
for yourself.
Again, I'm talking about the rules for what reversion does - such as
starting a new record. Those were the rules under discussion and they
are not different in the two cases. I'm *NOT* talking about the rules to
figure out what set of parens to go back to; yes, there is a rule for
that, and if you want to use it you'll have to learn that rule. But that
doesn't mean you also have to learn a separate set of rules for things
line record ending any more than it means you have to learn a new set of
rules for, say, what F edit descriptors do during format reversion. They
do the same thing as always.
By the way, yes, I've used the parens form of format reversion. It is
handy when you want the first line of an output to be different from
subsequent lines, as is pretty common. For example, I often deal with
time series data where each record has a time and a set of values that
go with the time. When printing such data, the time is often formatted
specially and appears only on the first line; also it can be handy to
indent subsequent lines that go with the same time. I recall seeing some
pretty messy code to get this result when people didn't realize how
simple it was to do with format reversion. Try without format reversion
to do the equivalent of
write(*,'(i10,3x,4e15.5:/(13x,4e15.5))') time, data
where time is an integer (not usually, but let's simplify that part),
and data is an array of arbitrary size. Of course you can do it. I've
seen it done. But I think you'll find it quite a lot messier. Do be sure
to cover all of the edge cases, including those where it all fits on the
first line (and for that matter, where data is of size 0). The output
might sometimes fit on one line, but the code won't. :-)
> > Unformatted stream I'm a big fan of, but formatted
> > stream seems not to add much to me. I did hear some advocates explain
> > advantages, but the explanations didn't do much for me. (For example,
> > there were explanations about how necessary buffers caused problems
> > with long records for formatted sequential, but I never really
> > understood how stream did much that related to that issue.)
>
> I would have to go back to be sure, but it might have to do with
> the T format descriptor. In record oriented I/O, where the buffer
> is at least one record long, it is easy to position anywhere within
> the record, even backwards. With stream, especially unbuffered,
> (not so uncommon with unix) once a character is out you don't
> expect to be able to change it.
I don't see anything in the Fortran standard that supports this as a
distinction. I'm talking about the difference between formatted stream
and formatted sequential in the Fortran standard. Yes, there are various
ways to implement I/O. But unless you can point to something in the
Fortran standard that relates stream vs sequential to those ways, then I
fail to see the relevance. No, I don't see any distinction in regards to
the T edit descriptor, so just sugesting that it might relate to that
doesn't do it unless you can point to a specific distinction.
I might add that no, T does not let you position anywhere within a
record. There is such a thing as the left tab limit. For advancing I/O,
that will generally be the beginning of the record. For nonadvancing
I/O, it won't be. But it works the same for sequential and stream, so I
have a bit of trouble seeing how that explains the difference between
the two. Yes, by the way, formatted stream files can and usually do have
records.
Also note that nothing in the standard talks about bufferred versus
unbuffered, so again, if you want to use that as a rationalization for
the distinction in the standard, you'd need to actually show some
relationship to the standard. I do recall bufferred versus nonbufferred
being mentioned as a large part of the reason why some vendors were
pushing for formatted stream, so maybe it is there somewhere. But I just
don't see the distinction actually made in the standard.
I suppose it could be one of those funny kinds of things where the
standard doesn't actually make a distinction, but there is an "informal
understanding" that the vendors make different implementation choices
for sequential versus stream.
glen herrmannsfeldt
(snip)
> I don't see anything in the Fortran standard that supports this as a
> distinction. I'm talking about the difference between formatted stream
> and formatted sequential in the Fortran standard. Yes, there are various
> ways to implement I/O. But unless you can point to something in the
> Fortran standard that relates stream vs sequential to those ways, then I
> fail to see the relevance. No, I don't see any distinction in regards to
> the T edit descriptor, so just sugesting that it might relate to that
> doesn't do it unless you can point to a specific distinction.
The main distinction, at the beginning of chapter 9, seems to be:
"A file is composed of either a sequence of file storage units
(9.2.4) or a sequence of records, which provide an extra level
of organization to the file. A file composed of records is
called a record file. A file composed of file storage units
is called a stream file. A processor may allow a file to be
viewed both as a record file and as a stream file; in this
case the relationship between the file storage units when
viewed as a stream file and the records when viewed as a
record file is processor dependent."
Two cases that I can see that might give different results are file
positioning and writing line terminating characters to the file.
Stream files, formatted or unformatted, can be positioned based
on file units (such as characters). In the C standard, text files
may only be positioned (using fseek) to positions previously returned
from ftell. One reason for that restriction is that the character
count will be off for systems using CRLF line terminators.
Another is that some systems don't position files in characters.
VM/CMS, for example, keeps track of records and position within
a record, but not byte offset. The only way to find a byte offset
on a variable record file is to read from the beginning. The value
returned by ftell for variable length record files is something
like record*32768+offset. For fixed length records, the byte offset
can be computed from the record number. Some of the restrictions and
featers of C streams carry forward to Fortran formatted stream.
(For example, no record length limit.)
In the case of record oriented file systems, again such as VM/CMS,
one can write all possible characters to the file, and expect
to read them back again. For stream file systems using line
terminating characters, that isn't always true.
Some of the system dependencies will occur writing files one way
and reading them the other.
-- glen
Richard Maine
> Richard Maine <nos...@see.signature> wrote:
> (snip)
>
> > I don't see anything in the Fortran standard that supports this as a
> > distinction. I'm talking about the difference between formatted stream
> > and formatted sequential in the Fortran standard. Yes, there are various
> > ways to implement I/O. But unless you can point to something in the
> > Fortran standard that relates stream vs sequential to those ways, then I
> > fail to see the relevance. No, I don't see any distinction in regards to
> > the T edit descriptor, so just sugesting that it might relate to that
> > doesn't do it unless you can point to a specific distinction.
>
> The main distinction, at the beginning of chapter 9, seems to be:
>
> "A file is composed of either a sequence of file storage units
> (9.2.4) or a sequence of records, which provide an extra level
> of organization to the file. A file composed of records is
> called a record file. A file composed of file storage units
> is called a stream file. A processor may allow a file to be
> viewed both as a record file and as a stream file; in this
> case the relationship between the file storage units when
> viewed as a stream file and the records when viewed as a
> record file is processor dependent."
Yes, but that's just a general description. It doesn't translate into
any actual difference in usage, particularly in the case where, as
mentioned above, the same file can be opened as either formatted stream
or formatted sequential. (Which is likely to be the usual case - very
usual - namely in that I'm not sure there will ever be any other cases.)
Look at the actual I/O statements. Both the Fortran syntax and the
resulting files. If you overlook the access='stream' in the open, you
might have trouble telling the difference in either the syntax or the
results.
> Two cases that I can see that might give different results are file
> positioning and writing line terminating characters to the file.
>
> Stream files, formatted or unformatted, can be positioned based
> on file units (such as characters). In the C standard, text files
> may only be positioned (using fseek) to positions previously returned
> from ftell. One reason for that restriction is that the character
> count will be off for systems using CRLF line terminators.
Fortran is the same for the same reason (plus issues of C
compatibility).
But I don't see that as an inherent difference between formatted
sequential and formatted stream. That's just a new capability, which
could trivially have also been added to formated sequential. It doesn't
need a whole new kind of connection. And yes, the capability, like
almost everything in I/O can be unavailable for some files. In fact that
optionality is the case for stream now; the standard specifically says
that a stream file might or might not be positionable. A formatted
stream file that is not positionable is different from a formatted
character file how? (Again, I'm asking about differences in terms of the
Fortran standard - not different implementation choices that vendors
might happen to make for reasons not reflected in the standard.)
Writing line terminating characters isn't an incompatible difference
either. Both forms do the same thing for new records caused by advancing
I/O, slash edit descriptors, or format reversion. The only difference is
in writing newline characters as part of the data. For that case, stream
output defines the action, while sequential output leaves it up to the
processor what to do. That doesn't sound very incompatible to me. And,
in fact, most processors will do exactly the same thing for sequential
as for stream. (Odds are they use the same support library). After all,
the usual formatted sequential file formats don't allow any way of
escaping the line terminator characters. (There are implementations that
do allow arbitrary characters in formatted records, but those aren't
common any more). So the implementation basically has 2 reasonable
choices - throw an error or just write the line terminator to the file
and let it make a new record.
glen herrmannsfeldt
(snip on differences between record and stream formatted I/O)
(snip)
> Writing line terminating characters isn't an incompatible difference
> either. Both forms do the same thing for new records caused by advancing
> I/O, slash edit descriptors, or format reversion. The only difference is
> in writing newline characters as part of the data. For that case, stream
> output defines the action, while sequential output leaves it up to the
> processor what to do. That doesn't sound very incompatible to me. And,
> in fact, most processors will do exactly the same thing for sequential
> as for stream. (Odds are they use the same support library). After all,
> the usual formatted sequential file formats don't allow any way of
> escaping the line terminator characters. (There are implementations that
> do allow arbitrary characters in formatted records, but those aren't
> common any more). So the implementation basically has 2 reasonable
> choices - throw an error or just write the line terminator to the file
> and let it make a new record.
Maybe not so common, but much more common than sign magnitude
machines, and the standard (also C89) still allows for them.
I believe VMS also has a record oriented file system. Between
IBM's MVS (z/OS) and VM/CMS, and HPs VMS they aren't that uncommon.
(I don't know the details of VMS file system as closely as
VMS and CMS, though.)
It only takes one for the difference to matter...
-- glen
Richard Maine
> Richard Maine <nos...@see.signature> wrote:
> (snip on differences between record and stream formatted I/O)
> > Writing line terminating characters isn't an incompatible difference
> > either....
> > The only difference is
> > in writing newline characters as part of the data. For that case, stream
> > output defines the action, while sequential output leaves it up to the
> > processor what to do. That doesn't sound very incompatible to me. And,
> > in fact, most processors will do exactly the same thing for sequential
> > as for stream....
>
> Maybe not so common, but much more common than sign magnitude
> machines, and the standard (also C89) still allows for them.
[examples of record oriented file systems]
Yes, those are cases where there are multiple possibilities. However, I
still see no convincing connection between the Fortran standard and
these things. Nor do I recall anything like that being mentioned when
stream I/O was being added. About the only thing I recall being
mentioned was unlimitted record length issue, which, as noted before, I
never really understood the connection to.
Yes, I was there for the debates. I even wrote the original proposal for
stream I/O. My proposal had only unformatted stream. The biggest
objection to my proposal was that several people wanted formatted stream
as well. But just because I was there doesn't mean I understood every
point made. About all I can say for sure is to fall back on the old "it
was added because that's the way the votes went."
As noted above, formatted stream output defines what happens when you
try to write a newline character to the file. Formatted sequential
leaves it processor dependent. There exist no file structures that are
not equally compatible with both of those specifications. A vendor could
choose two different implementations, but nothing in the standard
specifies such a difference. That's what I meant when I said above that
it isn't an incompatible difference.
In particular, the "usual" rationalization for why a vendor is allowed
to just let writing a newline character to a sequential file create a
new record is that the standard allows the vendor to prohibit output of
some control characters to a formatted file. If the newline character is
declared to be one of the prohibitted ones, then a program that writes
one is violating the standard in a way that allows the processor to do
anything (and in particular, "anything" could be making a new record.)
A vendor with a record-oriented file system could allow arbitrary
control characters in a formatted sequential file; I've used systems
that did so. But that is a vendor choice - not a specification of the
standard. And any program that takes advantage of such a choice is going
to be very non-portable. I think I recall having to deal with porting
some such programs in the past; it can easily come up if you try to use
a formatted file as a "bit bucket" for non-character data (as, for
example, by using "A" edit descriptors for non-character data, which was
allowed for Hollerith in f66 and was occasionally taken advantage of in
nonstandard ways for non-Hollerith data.)
glen herrmannsfeldt
(snip)
> Yes, those are cases where there are multiple possibilities. However, I
> still see no convincing connection between the Fortran standard and
> these things. Nor do I recall anything like that being mentioned when
> stream I/O was being added. About the only thing I recall being
> mentioned was unlimitted record length issue, which, as noted before, I
> never really understood the connection to.
> Yes, I was there for the debates. I even wrote the original proposal for
> stream I/O. My proposal had only unformatted stream. The biggest
> objection to my proposal was that several people wanted formatted stream
> as well. But just because I was there doesn't mean I understood every
> point made. About all I can say for sure is to fall back on the old "it
> was added because that's the way the votes went."
I might believe that character oriented file systems are the worst
thing that has happened in file system design. As far as I know, that
goes back to DEC. With character oriented systems, it is easy to
position to a specific character in a file, but difficult to position
to a record, requiring reading from the beginning counting line
terminator characters. Very slow as files get bigger.
With record oriented file systems, it is easy to position to a
record, but difficult to position to a character offset. Because
of the differences in line terminators, C restricts positioning
of text files, but not of binary files. For variable record length
files on some IBM systems, the only way to position to a byte offset
is to read from the beginning. It isn't necessary to examine every
character, but just read the length field of the record headers.
I do agree that the difference is most important in the case of
non-text files, but some differences still show up in text files.
> As noted above, formatted stream output defines what happens when you
> try to write a newline character to the file. Formatted sequential
> leaves it processor dependent. There exist no file structures that are
> not equally compatible with both of those specifications. A vendor could
> choose two different implementations, but nothing in the standard
> specifies such a difference. That's what I meant when I said above that
> it isn't an incompatible difference.
That works when using only Fortran I/O. If you have to read a file
written by another program, you have less control over the contents
of that file. I agree that the differences are small. It is still
nice to seem them acknowledged by the standard.
> In particular, the "usual" rationalization for why a vendor is allowed
> to just let writing a newline character to a sequential file create a
> new record is that the standard allows the vendor to prohibit output of
> some control characters to a formatted file. If the newline character is
> declared to be one of the prohibitted ones, then a program that writes
> one is violating the standard in a way that allows the processor to do
> anything (and in particular, "anything" could be making a new record.)
Since Fortran was designed around record oriented file systems,
it isn't too surprising to see it still there. Even so, it isn't
always easy. There is no good way in Fortran 66, I am not sure about
later standards, to find the length of a record read from a formatted
file. With fixed length records you remove the trailing blanks
and count what is left. With variable length records there could be
trailing blanks that are part of the record.
> A vendor with a record-oriented file system could allow arbitrary
> control characters in a formatted sequential file; I've used systems
> that did so. But that is a vendor choice - not a specification of the
> standard. And any program that takes advantage of such a choice is going
> to be very non-portable. I think I recall having to deal with porting
> some such programs in the past; it can easily come up if you try to use
> a formatted file as a "bit bucket" for non-character data (as, for
> example, by using "A" edit descriptors for non-character data, which was
> allowed for Hollerith in f66 and was occasionally taken advantage of in
> nonstandard ways for non-Hollerith data.)
Well, for that matter depending on eight bit bytes is not-portable,
but a fairly safe bet these days.
-- glen
Ron Shepard
glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
> Personally,
> I prefer out-of-band record indicators, but then I don't always
> have that choice.
I'm trying to follow this discussion, but there are a few terms I
may not understand. In this context, "out-of-band" records are are
the ones determined by the format, right? (either explicit "/" or
implicit format reversion). An "in-band" record control would be an
embedded control character in the data (such as CR, or LF, or one
of the other ASCII characters related to records, or some
combination of the above, such as in DOS/Windows file systems). Is
this right?
The above applies to formatted I/O. I'm not sure about how
unformatted I/O (stream or normal) fits into this yet.
$.02 -Ron Shepard
Ken Fairfield
wrote:
> In article <hhbjk5$rn...@naig.caltech.edu>,
> glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
>
> > Personally,
> > I prefer out-of-band record indicators, but then I don't always
> > have that choice.
>
> I'm trying to follow this discussion, but there are a few terms I
> may not understand. In this context, "out-of-band" records are are
> the ones determined by the format, right? (either explicit "/" or
> implicit format reversion). An "in-band" record control would be an
> embedded control character in the data (such as CR, or LF, or one
> of the other ASCII characters related to records, or some
> combination of the above, such as in DOS/Windows file systems). Is
> this right?
That's essentially correct.
To use the example of VMS's file system (which Glen alluded to),
there are *many* different record formats available for files.
The most common is a variable length record (VAR) file where the
file system keeps the length of the record in the first 16 bit
word of each record (for a maximum record length of 32767
characters). There is also VFC record format (variable with
fixed control). Both of these have out-of-band record control
in that the end-of-record condition is not included in the
record data, but is a separate count. In fact, in general
there are no CR's or LF's in present in these types of file.
There is an "FTN" (Fortran) record format as well, which was
cooked up early on to support essentially unlimited record length
binary (unformatted) files for Fortran input/output. Again, the
end-of-record condition is out-of-band, almost by definition since
you need to be able to handle anything and everything in the data
stream.
(We'll leave indexed (essentially ISAM) and "relative" file
formats as they're not terribly relevant to the present
discussion...)
"Modern" versions of VMS (since about 1990 :-) support
stream record formats, "STMLF" being the usual unix
LF-terminated records, "STMCR" being CR-terminated records
(like Macintosh?), and "STM" being Windows style CRLF-terminated
records.
All the standard VMS file untilies (editors, TYPE, COPY,
RENAME, DELETE, BACKUP, DIFERENCE, etc., etc.) transparently
handle the variety of file record formats. The one place the
user does see a difference is an attempt to do a TYPE/TAIL
(like unix "tail") on one of the stream format files. It
wouldn't be impossible to make the utility do that, but it
would be very inefficient since the file system would
essentially be forced to read the file from the beginning in
order to find the last "n" records: not very pleasant with a
GB-sized file! *That* is why out-of-band record indicators are
useful: the ability to efficiently position within a file without
reading all the data.
-Ken
Ken Fairfield
> On Dec 29, 8:53 am, Ron Shepard <ron-shep...@NOSPAM.comcast.net>
> wrote:
> > I'm trying to follow this discussion, but there are a few terms I
> > may not understand. In this context, "out-of-band" records are are
> > the ones determined by the format, right? (either explicit "/" or
> > implicit format reversion). An "in-band" record control would be an
> > embedded control character in the data (such as CR, or LF, or one
> > of the other ASCII characters related to records, or some
> > combination of the above, such as in DOS/Windows file systems). Is
> > this right?
[...]
> "Modern" versions of VMS (since about 1990 :-) support
> stream record formats, "STMLF" being the usual unix
> LF-terminated records, "STMCR" being CR-terminated records
> (like Macintosh?), and "STM" being Windows style CRLF-terminated
> records.
I neglected to add the obvious, here, that these three stream
file record formats do, indeed, have in-band record terminators,
explicit LF's and/or CR's in the data stream to define the
end-of-record condition.
Which is why I took pains to note:
[...] The one place the
Steve Lionel
>
> To use the example of VMS's file system (which Glen alluded to),
> there are *many* different record formats available for files.
snip
> there are no CR's or LF's in present in these types of file.
>
> There is an "FTN" (Fortran) record format as well, which was
> cooked up early on to support essentially unlimited record length
> binary (unformatted) files for Fortran input/output. Again, the
> end-of-record condition is out-of-band, almost by definition since
> you need to be able to handle anything and everything in the data
> stream.
Not that this changes the argument, but there is no FTN recordtype in
VMS. The recordtype used by VMS Fortran for its "segmented" unformatted
files is VFC, where two of the sixteen bits in the control field were
used to indicate first/last/both/neither segment. Other users of the VFC
recordtype included the SOS editor, which put line numbers in the
control field, and the CDD preprocessor for Fortran which inserted error
codes.
FTN is a carriagecontrol attribute for files in VMS, but that can apply
to any record type.
--
Steve Lionel
Developer Products Division
Intel Corporation
Nashua, NH
For email address, replace "invalid" with "com"
User communities for Intel Software Development Products
http://software.intel.com/en-us/forums/
Intel Software Development Products Support
http://software.intel.com/sites/support/
My Fortran blog
http://www.intel.com/software/drfortran
glen herrmannsfeldt
(snip)
> Not that this changes the argument, but there is no FTN recordtype in
> VMS. The recordtype used by VMS Fortran for its "segmented" unformatted
> files is VFC, where two of the sixteen bits in the control field were
> used to indicate first/last/both/neither segment. Other users of the VFC
> recordtype included the SOS editor, which put line numbers in the
> control field, and the CDD preprocessor for Fortran which inserted error
> codes.
There is currently a discussion in comp.sys.pdp10 on SOS for the PDP-10
which puts five digit line numbers in the first word of a line, and sets
the low bit. (Five 7 bit ASCII characters per 36 bit word, with one
bit left over.) The low bit tells programs that read the file
(compilers, for example) that line numbers are there. Those numbers
are used in error messages instead of the position of the line in
the file.
-- glen
Ken Fairfield
[...]
> Not that this changes the argument, but there is no FTN recordtype in
> VMS. The recordtype used by VMS Fortran for its "segmented" unformatted
> files is VFC, where two of the sixteen bits in the control field were
> used to indicate first/last/both/neither segment.
Yes, of course, sorry. I had just taken a too-quick look at
HELP SET FILE /ATTRIBUTES for the record attributes and saw
the FTN without really *reading*. :-( I actually wrote some
code once that had to know about Fortran and segmented records.
Thought it ws really cool at the time. :-)
> Other users of the VFC
> recordtype included the SOS editor, which put line numbers in the
> control field, and the CDD preprocessor for Fortran which inserted error
> codes.
>
> FTN is a carriagecontrol attribute for files in VMS, but that can apply
> to any record type.
I also omitted fixed-length and "print" record formats,
but I think enough has been said already.
-Ken
Gordon Sande
One of my early exposures to time sharing was with a PDP-10 that was
running TOPS 10 (if memory serves correectly). I had a source
file in the editor which had been a data file earlier in it provenance.
The result was line numbers in the data portion of the record. I recall
having to edit every line to delete the data line numbers so the editor
could insert its line numbers with their marker bit. A painfull introduction
to either an inflexible computer or a lack of available local expertise and
documentation. The program was in Basic and was probably the paper tape
output of another time sharing system that was running GE235s (Dartmouth
time sharing system if memory is still working).
Whenever I hear about someone assuming 8 bit ASCII I am tempted to tell
my story of the trouble that 7 bit ASCII caused for me. I am also tempted
to bring up the Univac Exec 8 which had ASCII in 9 bit characters/bytes/
whatevers as there were 4 things in a 36 bit word although I can not claim
direct personal experience. And then there is the old 6 bit CDC code but that
was not ASCII.
Jerry DeLisle
--- snip ---
> ! NOTE: when I change the order of the above lines, "mystream" is
> empty using gfortran
> stop
> end
--- snip ---
Regarding the above comment about reversing the order.
You have a stop statement at the end of the writeit subroutine. The program
stops after execution of the first call to the subroutine. I don't know if
anyone else caught this in the thread, so I apologize if I am repeating
information. I did not want to take the time to read any further.
Correcting that problem and testing with gfortran 4.5, (4.4 is probably OK too)
there are no problems with NULs and the output to UNIT=6 is identical to the
output to the mystream file (UNIT=11)
> !-----------------------------------------------------------------------
> stop
> end
> !
> ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
> subroutine line(io,fmt,header) ! write break and header
--- snip ---
J.F. Cornwall
Aye, the fun looks you could get from describing the 9-bit ASCII you
used on Univac machines (I spent 7 years coding on 'em for Uncle Sam).
Things could get interesting when you were doing conversions to/from
those quarter-word bytes to the Fielddata 6-bit bytes, or to 5-level
Baudot, or to the other character sets and binary protocols we had to
deal with. Fun, though. :)
Jim
Terence
forgive me if I am repeating something already potsted in reply.
1) I agree with the usefullness oh having a set of "table" output
routines for interfacing other systems as listed (and I've done it for
years, including RTF, DBF and Market Research standard ascii file
record layouts like Quantum card code). Then you just pass the line
string and its optional desired ending string to the routine,
requesting the appropriate conversion code (you usually need a prefixe
block and a postfix end-of-file block as well for any of these
formats).
2) the reason that you cannot expect a crlf on exhausting a Format
statement, is for the old Fotrans requirement to allow the "same line"
override Format control of '\' (or sometimes '$') to permit either
adding to the end of an output line, or finally ending it without more
text.
A new-line string is otherwise output only on meeting a control code
of '/', or restarting the complete Format statement to exhaust the
output variable list.
3) For text outputs I recommend using a combination of Formatted write-
to-string and then unformatted stream output of the result, where the
end-of-output-line string (lf, cr, or cr-lf) is written as characters
with the appropriate bit pattern(s).
If you want a final end-of-file marker (e.g. #1A) you should write
(output) one; otherwise end-of-file position and signal (on next file
read) is usually computed from the byte count stored in the directory
entry (something Fortran itself does not know about). But looking for
that #1a has its uses on old systems.