direct access with Intel fortran
John Paine
I'm struggling with a problem in trying to perform direct access read/write
to a file using Intel 11.1.
What I want to do is:
1. Open a data file
2. Write a header block to the file defining the data in the file (this
block is of known size, but unknown content when I open the file)
3. Write multiple blocks of binary data of various types sequentially to the
file and accumulate statistics about the data such as range of values, size
of data blocks etc
4. Once all of the data blocks have been written, I then want to rewrite the
header block with the updated statistics and data block sizes etc
5. Close the file
In CVF, I used to do this using the dfwin interface (I'm not working
cross-platform) to Windows the routines lcreat, lwrite, llseek and lclose
and all worked just fine. Since I am now migrating my code to the Intel
compiler, I'm cleaning up my code base and would like to eliminate the
Windows routines and use standard Fortran IO.
Simplifying the problem to the barest bones to illustrate the task
schematically, the code looks like this:
implicit none
integer*4 i,j
integer*4 numy,numx(40)
real*4 zmin,zmax
real*4 zval(40)
c 1: open the file
open(10,file='test.dat',form='binary')
c 2: initialise and write out the header
numy=40
zmin=1e32
zmax=-1e32
do j=1,numy
numx(j)=0
end do
write(10)numy,zmin,zmax
write(10)(numx(j),j=1,numy)
c 3: loop over the data
do j=1,numy
c do stuff to create numx(j) data values in array zval
c NB numx(j) is determined as part of the calculation
numx(j)=j
do i=1,numx(j)
zval(i)=i
zmin=min(zmin,zval(i))
zmax=min(zmax,zval(i))
end do
c write out the data
write(10)(zval(i),i=1,numx(j))
end do
c 4: rewrite the header
rewind(10)
write(10)numy,zmin,zmax
write(10)(numx(j),j=1,numy)
c 5: close the file
close(10)
c reopen the file and read the data
open(10,file='test.dat',form='binary')
c this step works ok and the header is read correctly
read(10)numy,zmin,zmax
read(10)(numx(j),j=1,numy)
c this loop fails with an "end-of-file during read" error
do j=1,numy
read(10)(zval(i),i=1,numx(j))
end do
close(10)
end
This all works fine, except that step 4 truncates the file after the write
so all of the data records are lost. If I omit step 4, the data in the file
is exactly what I want, but the data counts and statistics in the header
record are not valid. Note, this example is greatly simplified, so answers
along the lines of "calculate the statistics before writing the data" while
perfectly valid, will not solve my difficulty. In the real case, the data
written out in step 3 is created in the loop and is expensive to calculate
and there is a lot of it. What I really want is to be able to rewrite the
header record which will not truncate the data file. Separating the header
and data is also an option, but my preference is to keep the two together in
the one file to ensure that the data can be read by other applications
further down the processing stream with no danger of the header file being
misplaced thus rendering the data file useless.
The behavior of the 'binary' write truncating the file is documentated by
Intel. They also include a mention of "direct access" which suggests that
direct access be used. This is a possibility as I could open the file in
step 1 with access='direct',recl=1 specified. But I then need to keep track
of where I am in the file in order to specify the record number to write
out. This would be OK for the above case as I know the position of the
header and can count the bytes as they are written out. But really I'd
prefer not to keep track of this myself (unless I really have to) as it
would be so much simpler if I could somehow stop the truncation of the file
when rewriting the header. The open statement will allow the inclusion of
the Carriagecontrol='none' clause, but this does not make any difference.
Thanks in advance for any suggestions.
John
glen herrmannsfeldt
> I'm struggling with a problem in trying to perform direct
> access read/write to a file using Intel 11.1.
> 4. Once all of the data blocks have been written, I then want
> to rewrite the header block with the updated statistics and
> data block sizes etc
> open(10,file='test.dat',form='binary')
> write(10)numy,zmin,zmax
> write(10)(numx(j),j=1,numy)
Your subject says "Direct Access" but your sample code doesn't.
You need the ACCESS='DIRECT' and RECL= options on the OPEN,
and the REC= on the READ/WRITE statements. RECL specifies
the record length that all records will have.
If your data doesn't naturally have a fixed length then you
have to do some work to divide it up into fixed length units.
(Possibly wasting the end of some of the records.)
Note that in the case of statements like:
write(10)numy,zmin,zmax
write(10)(numx(j),j=1,numy)
Two records will be used, while it can be written:
write(10)numy,zmin,zmax,(numx(j),j=1,numy)
and use only one record. It can then be read with:
read(10)numy,zmin,zmax,(numx(j),j=1,numy)
(But note that these are not direct access I/O statements
without the REC= option.
-- glen
Louis Krupp
> Hi all,
>
> I'm struggling with a problem in trying to perform direct access
> read/write to a file using Intel 11.1.
>
> What I want to do is:
>
> 1. Open a data file
> 2. Write a header block to the file defining the data in the file (this
> block is of known size, but unknown content when I open the file)
> 3. Write multiple blocks of binary data of various types sequentially to
> the file and accumulate statistics about the data such as range of
> values, size of data blocks etc
> 4. Once all of the data blocks have been written, I then want to rewrite
> the header block with the updated statistics and data block sizes etc
> 5. Close the file
You may be overlooking the easy way to do this:
1. Write your multiple blocks of binary data to a scratch file,
accumulating statistics as you go.
2. Rewind the scratch file.
3. Open the output data file and write the header with everything you need.
4. Read records from the scratch file and write them to the output file.
5. Close the output file. Remove the scratch file.
Unless your files are really, really big (for some definition of "big"),
this is likely to be fast enough *and* easier to code. Plus, it will
let the header size vary, which might be convenient at some point.
A couple more thoughts:
If your header size and/or format change, make sure that new versions of
the file that read the file can read old versions or at least fail
gracefully. Including a version number in the first few bytes of the
header is probably a good way to do this. You'll probably also want to
make sure that old versions of the program know which version(s) of the
file they can read.
I'm not sure I'd recommend this practice, but as you may know, PDF uses
a cross-reference and a trailer record at the end of the file. Programs
that read PDF files start by reading a bunch of bytes at the end of the
file and scanning for a magic marker string.
Louis
James Van Buskirk
news:4b6f518f$0$32133$afc3...@news.optusnet.com.au...
> I'm struggling with a problem in trying to perform direct access
> read/write to a file using Intel 11.1.
C:\gfortran\clf\streamtest>type streamtest.f90
program streamtest
implicit none
type head
integer i
real x
character(3) c
end type head
type(head) header
integer array(10)
integer i
array = [(i,i=1,size(array))]
header = head(0,0,'0')
open(10,file='streamtest.dat',access='stream')
write(10) header, array
header = head(2,3.14,'CAT')
write(10,POS=1) header
close(10)
open(10,file='streamtest.dat',access='stream')
header = head(0,0,'0')
array = 0
read(10), header, array
write(*,*) 'header = ', header
write(*,*) 'array = ', array
end program streamtest
C:\gfortran\clf\streamtest>gfortran streamtest.f90 -ostreamtest
C:\gfortran\clf\streamtest>streamtest
header = 2 3.1400001 CAT
array = 1 2 3 4 5
6 7 8 9 10
--
write(*,*) transfer((/17.392111325966148d0,6.5794487871554595D-85, &
6.0134700243160014d-154/),(/'x'/)); end
Richard Maine
> What I want to do is:
....
> 4. Once all of the data blocks have been written, I then want to rewrite the
> header block with the updated statistics and data block sizes etc
...
> In CVF, I used to do this using the dfwin interface
..
> Since I am now migrating my code to the Intel
> compiler, I'm cleaning up my code base and would like to eliminate the
> Windows routines and use standard Fortran IO.
[example using form='binary']
Well form='binary' is not standard Fortran, so if that is the objective,
this would not achieve it even if it did act like you wanted.
> This all works fine, except that step 4 truncates the file after the write
> so all of the data records are lost....
> The behavior of the 'binary' write truncating the file is documentated by
> Intel. They also include a mention of "direct access" which suggests that
> direct access be used.
Yes, you could certainly do it with direct access, but there are lots of
complications - more than you mentioned. For a start,
> step 1 with access='direct',recl=1 specified.
The use of recl=1 with direct access is a hack recognized by some
compilers, but it is not standard. Direct access is standard, but the
common special-case interpretation of recl=1 is not. There are other
complications as well.
> Thanks in advance for any suggestions.
I'd recommend against going with direct access. It certainly can be
done; I've done that kind of thing in the past. But the many gotchas of
direct access are a large part of why I was a big pusher for stream
access in f2003.
I recommend using stream access. James gave some presumably fine example
code (I didn't check in detail, but I suspect it is fine). I just
thought I'd supply some English to supplement his Fortran. :-)
The form='binary' and access='direct',recl=1 are both nonstandard
variants of stream access. They date from before stream was
standardized. Now that stream is standardized, I recommend using it
instead of those nonstandard variants. As an additional "side" benefit,
the standard requires that it act like you want (no truncation), at
least as long as you stick to unformatted stream. (Formatted is a
different story).
The approach that Louis mentioned (using a temporary intermediate file)
can also work well; that's your choice to make.
--
Richard Maine | Good judgment comes from experience;
email: last name at domain . net | experience comes from bad judgment.
domain: summertriangle | -- Mark Twain
John Paine
Thanks for the comments. I wasn't too sure what to put as the subject, so
just opted for the flavor of the task and did not intend it to be
interpreted as relating to the access='direct' clause. (Besides, that's also
how it is referred to in the Intel documentation, though that's not of a
defence).
I did include a remark about the possibility of using access='direct', but
would prefer not to do that for a number of reasons. Primarily, I'd like the
flexibility of rewriting various portions of the binary files I use and
frequently the actual position will depend on the data that precedes the
location where the data is to be written. I know that I can keep track of
this position (and can even use fseek to directly tell me where I am in the
file), but wanted to check first that there wasn't some obvious option for
the open statement that I was missing that would allow me to rewrite a
portion of the file without truncating it. Also, having to have a fixed
record length would make it much harder to keep track of where I actually
want to write the data and I'd rather revamp my interface to the relevant
system calls than rewrite my code to manage extraneous task of counting
records.
John
"glen herrmannsfeldt" <g...@ugcs.caltech.edu> wrote in message
news:hknlej$4tk$1...@naig.caltech.edu...
John Paine
Thanks for the response.
I did consider this route as it is akin to the separate header/data files,
but avoids the problem of separate files. While it is certainly a feasible
solution, it's not really one I'd opt for. As per my comments to Glen's
response, I was hoping for something that would allow me to simply rewrite
data within an existing file using standard fortran and (although unstated
in my oriiginal email) with as little rewriting as possible. The task is
indeed somewhat similar to the PDF problem as in that case (if I remember
correctly) you need to know where the bytes defining the file structure are
at the end of the file, and you don't know that position until you've
finished writing the contents file. It is a pretty common task to rewrite
information in a header block, so I was hoping that Fortran would allow me
to do it directly without having to resort to the Windows API or writing the
functionality as a C library.
Many thanks
John
"Louis Krupp" <lkrupp...@indra.com.invalid> wrote in message
news:eZqdnTGRr5zM-PLW...@indra.net...
John Paine
"James Van Buskirk" <not_...@comcast.net> wrote in message
news:hknqce$h1m$1...@news.eternal-september.org...
Hi James,
I haven't had any experience with stream files, so didn't try using that
type of access as the Intel documentation didn't sound very promising.
However your sample looks like it does the job, so I rewrote the code as
below and it does exactly what I want.
The one point that troubles me is that the "pos=" keyword uses a number that
"indicates a file position in file storage units in a stream file ". This
seems to be bytes, but I thought that I saw a comment in the Intel
documentation (which I can't locate at the moment), that the "storage unit"
is system dependant. Also, I can't find any way to inquire what the unit
might be. I'm happy to ignore this for now as it allows me to implement a
simple replacement of the API calls with my own equivalents for a stream
file, but I'm sure that at some stage the unit size will come back to bite
me.
Many thanks
John
implicit none
integer*4 i,j
integer*4 numy,numx(40)
real*4 zmin,zmax
real*4 zval(40)
c 1: open the file
open(10,file='test_stream.dat',access='stream')
c 2: initialise and write out the header
numy=40
zmin=1e32
zmax=-1e32
do j=1,numy
numx(j)=0
end do
write(10)numy,zmin,zmax,(numx(j),j=1,numy)
c 3: loop over the data
do j=1,numy
c do stuff to create numx(j) data values in array zval
c NB numx(j) is determined as part of the calculation
numx(j)=j
do i=1,numx(j)
zval(i)=i
zmin=min(zmin,zval(i))
zmax=max(zmax,zval(i))
end do
c write out the data
write(10)(zval(i),i=1,numx(j))
end do
c 4: rewrite the changed avlues in the header
write(10,pos=5)zmin,zmax,(numx(j),j=1,numy)
c 5: close the file
close(10)
c reopen the file and read the data
open(10,file='test_stream',access='stream')
c this step works ok and the header is read correctly
read(10)numy,zmin,zmax,(numx(j),j=1,numy)
c this loop correctly reads the data from the file
John Paine
"Richard Maine" <nos...@see.signature> wrote in message
news:1jdk4w1.162yhg4tspypsN%nos...@see.signature...
> John Paine <johnp...@optusnet.com.au> wrote:
>
>> What I want to do is:
> ....
>> 4. Once all of the data blocks have been written, I then want to rewrite
>> the
>> header block with the updated statistics and data block sizes etc
> ...
>> In CVF, I used to do this using the dfwin interface
> ..
>> Since I am now migrating my code to the Intel
>> compiler, I'm cleaning up my code base and would like to eliminate the
>> Windows routines and use standard Fortran IO.
> [example using form='binary']
>
> Well form='binary' is not standard Fortran, so if that is the objective,
> this would not achieve it even if it did act like you wanted.
>
I knew I was vulnerable on that front and was hoping that nobody would
notice. I know it's not standard, but I'm assuming that it'd be a brave
vendor who left it out in the forseeable future.
>> This all works fine, except that step 4 truncates the file after the
>> write
>> so all of the data records are lost....
>> The behavior of the 'binary' write truncating the file is documentated by
>> Intel. They also include a mention of "direct access" which suggests that
>> direct access be used.
>
> Yes, you could certainly do it with direct access, but there are lots of
> complications - more than you mentioned. For a start,
>
>> step 1 with access='direct',recl=1 specified.
>
> The use of recl=1 with direct access is a hack recognized by some
> compilers, but it is not standard. Direct access is standard, but the
> common special-case interpretation of recl=1 is not. There are other
> complications as well.
>
I agree, and that was one of the considerations for not going down that
route. Though if it had worked (and I did try it), I wouldn't have been
posting.
>> Thanks in advance for any suggestions.
>
> I'd recommend against going with direct access. It certainly can be
> done; I've done that kind of thing in the past. But the many gotchas of
> direct access are a large part of why I was a big pusher for stream
> access in f2003.
>
> I recommend using stream access. James gave some presumably fine example
> code (I didn't check in detail, but I suspect it is fine). I just
> thought I'd supply some English to supplement his Fortran. :-)
>
> The form='binary' and access='direct',recl=1 are both nonstandard
> variants of stream access. They date from before stream was
> standardized. Now that stream is standardized, I recommend using it
> instead of those nonstandard variants. As an additional "side" benefit,
> the standard requires that it act like you want (no truncation), at
> least as long as you stick to unformatted stream. (Formatted is a
> different story).
>
> The approach that Louis mentioned (using a temporary intermediate file)
> can also work well; that's your choice to make.
I will be going down the stream route and James' code was more than adequate
to put me on the right path. As an added benefit, it now seems that I will
be using standard fortran to do what I want. Very pleasing!
Richard Maine
> "Richard Maine" <nos...@see.signature> wrote in message
> news:1jdk4w1.162yhg4tspypsN%nos...@see.signature...
> > Well form='binary' is not standard Fortran, so if that is the objective,
> > this would not achieve it even if it did act like you wanted.
> >
> I knew I was vulnerable on that front and was hoping that nobody would
> notice. I know it's not standard, but I'm assuming that it'd be a brave
> vendor who left it out in the forseeable future.
Lots of vendors already don't have it. Pretty much all of them have some
kind of stream-like functionality, but the details do vary already. If a
large majority of vendors already did it the same way, that way probably
would have been adopted as the standard instead of introducing stream.
But they don't, so it wasn't.
Now if you are talking about the odds of Intel in specific dropping it,
I'd agree those odds would be low. But the odds of some other random
vendor happening to have it are much more questionable.
(I see you are happy with the suggestion to use stream, so these points
are perhaps a bit academic).
Richard Maine
> The one point that troubles me is that the "pos=" keyword uses a number that
> "indicates a file position in file storage units in a stream file ". This
> seems to be bytes, but I thought that I saw a comment in the Intel
> documentation (which I can't locate at the moment), that the "storage unit"
> is system dependant. Also, I can't find any way to inquire what the unit
> might be.
In that area, the f2003 standard has something unusual compared to
practice in previous Fortran standards. It makes a recommendation. I
recall having to convince some people that this was ok to do in a
standard. ISO specifically allows for it, but it just wasn't done in
previous Fortran standards.
In this case, the recommendation is that the storage unit be an octet
(which is standard-speak for an 8-bit byte). In practice, I don't think
you will ever find a compiler that doesn't at least have an option for
the storage unit to be that. The waffling in the standard is to
accommodate systems where it might be impractical for that to be the
storage unit. Such systems have existed in the past (for example, CDCs
where things were stored in 60-bit words and 8 bits wasn't a natural
size for anything). But you aren't really going to see them again in the
forseeable future. Making the 8-bit storage unit a recommendation
instead of a requirement amounted to a cover-your-ass type of exercise
so that the Fortran standard would not demand specific hardware sizes.
There was variation among vendors in the storage unit size in f77.
However, this is an area where there has been convergence. Those vendors
who used to use other storage unit sizes have all (I think it is all,
but I'll admit to having not checked every case) at least made an option
to support the 8-bit choice. I expect f2003 to finish that transition,
with vendors making it the default. This is because
1. It helps compatibility with other vendors.
2. The recommendation in the standard means that the vendors would find
themselves having to defend why they didn't follow it.
3. Some of the features of stream I/O would probably be a PITA to
implement otherwise. The POS= is high on that list. You can write
individual characters and need to be able to position to them. No vendor
is going to get by with a hack like padding all characters out to 32
bits or some such thing. The standard allows a vendor to declare that
some (or all) files are not positionable, but no vendor is going to try
that kind of excuse for ordinary disk files. (That's more for things
like serial ports that really inherently are not positionable).
Prior to f2003, there were ways to probably determine the storage unit
size, but none of them were actually guaranteed to work by the standard.
Knowing the ways that things could likely go wrong allowed one to do it
with reasonable confidence, but not with a guarantee supported by the
standard.
For an f2003 compiler, see the file_storage_size paameter in the
iso_fortran_env intrinsic module. I don't recall whether Intel has that
module yet, but I'd expect them to. After all, it is a pretty trivial
module to implement.
John Paine
Once again I amazed by your detailed knowledge and willingness to put even
trivial questions such as mine into the much broader context of language
standards and their evolution. The FILE_STORAGE_SIZE value in the Intel
intrinsic module (which I didn't even know existed, let alone what it might
contain) is indeed 8 and the documentation tells me that it is measured in
bits. So I do have an ability now to ensure that I am reading/writing at the
position I expect in the stream. Though from the context you provide, it's
going to be pretty unlikely that it'll be anything I don't expect it to be.
Many thanks
John
"Richard Maine" <nos...@see.signature> wrote in message
news:1jdke64.1yssjyt24letcN%nos...@see.signature...
John Paine
"Richard Maine" <nos...@see.signature> wrote in message
> John Paine <johnp...@optusnet.com.au> wrote:
>
>> "Richard Maine" <nos...@see.signature> wrote in message
>> news:1jdk4w1.162yhg4tspypsN%nos...@see.signature...
>
>> > Well form='binary' is not standard Fortran, so if that is the
>> > objective,
>> > this would not achieve it even if it did act like you wanted.
>> >
>> I knew I was vulnerable on that front and was hoping that nobody would
>> notice. I know it's not standard, but I'm assuming that it'd be a brave
>> vendor who left it out in the forseeable future.
>
> Lots of vendors already don't have it. Pretty much all of them have some
> kind of stream-like functionality, but the details do vary already. If a
> large majority of vendors already did it the same way, that way probably
> would have been adopted as the standard instead of introducing stream.
> But they don't, so it wasn't.
>
> Now if you are talking about the odds of Intel in specific dropping it,
> I'd agree those odds would be low. But the odds of some other random
> vendor happening to have it are much more questionable.
>
> (I see you are happy with the suggestion to use stream, so these points
> are perhaps a bit academic).
That sums up the situation pretty well. I guess a better way for me to have
exressed it is as a more selfish one: pre-stream I wouldn't have used a
compiler that didn't do what I wanted it to (or at least I would not have
used it if it required too much rewriting of code). Post-stream, the same
comments would probably still apply.
So it's not so much about bravery of vendors (as from your comments, they've
moved on and I haven't), but more about my slowness in changing vendors (I
still use CVF for most of my code), moving with the times (I still code
mostly F77) and narrowness of platform (solely Windows). But even with my
slowness and change-resistance, I'm still managing to keep my clients pretty
happy and there are much more interesting things to do than rewrite code
that already works reasonably well (even if it is non-standard).
Thanks again
John
Tim Prince
> John Paine<johnp...@optusnet.com.au> wrote:
>
>> The one point that troubles me is that the "pos=" keyword uses a number that
>> "indicates a file position in file storage units in a stream file ". This
>> seems to be bytes, but I thought that I saw a comment in the Intel
>> documentation (which I can't locate at the moment), that the "storage unit"
>> is system dependant. Also, I can't find any way to inquire what the unit
>> might be.
> In this case, the recommendation is that the storage unit be an octet
> (which is standard-speak for an 8-bit byte). In practice, I don't think
> you will ever find a compiler that doesn't at least have an option for
> the storage unit to be that.
For ifort, that option is
-assume byterecl
(replace space separator by : for Windows)
(in case it hasn't already been mentioned somewhere in the thread)
Tim Prince
>
>> The one point that troubles me is that the "pos=" keyword uses a number that
>> "indicates a file position in file storage units in a stream file ". This
>> seems to be bytes, but I thought that I saw a comment in the Intel
>> documentation (which I can't locate at the moment), that the "storage unit"
>> is system dependant. Also, I can't find any way to inquire what the unit
>> might be.
> In this case, the recommendation is that the storage unit be an octet
> (which is standard-speak for an 8-bit byte). In practice, I don't think
> you will ever find a compiler that doesn't at least have an option for
> the storage unit to be that.
John Paine
"Tim Prince" <tpr...@computer.org> wrote in message
news:7ta4qb...@mid.individual.net...
It hadn't been mentioned, but I picked that up in the documentation for the
the ISO_FORTRAN_ENV intrinsic module:
"FILE_STORAGE_SIZE: Is the size of the file storage unit expressed in bits.
To use this constant, compiler option assume byterecl must be enabled."
James Van Buskirk
> Once again I amazed by your detailed knowledge and willingness to put even
> trivial questions such as mine into the much broader context of language
> standards and their evolution. The FILE_STORAGE_SIZE value in the Intel
> intrinsic module (which I didn't even know existed, let alone what it
> might contain) is indeed 8 and the documentation tells me that it is
> measured in bits. So I do have an ability now to ensure that I am
> reading/writing at the position I expect in the stream. Though from the
> context you provide, it's going to be pretty unlikely that it'll be
> anything I don't expect it to be.
That was one reason why I created a structure that mapped to the whole
header: there was no need to know the details of how to move around
in the file if we were just going to replop the whole blob of data
there at the beginning. Also it modularized the code so that the
WRITE statements that created the dummy and real header wouldn't have
to change even the the header changed. Assuming the header isn't
itself huge, the extra cost of rewriting parts of the header that
were known at file creation time should be negligible.
Richard Maine
> "John Paine" <johnp...@optusnet.com.au> wrote in message
> news:4b6fc223$0$12922$afc3...@news.optusnet.com.au...
>
> > The FILE_STORAGE_SIZE value in the Intel
> > intrinsic module (which I didn't even know existed, let alone what it
> > might contain) is indeed 8 and the documentation tells me that it is
> > measured in bits. So I do have an ability now to ensure that I am
> > reading/writing at the position I expect in the stream. Though from the
> > context you provide, it's going to be pretty unlikely that it'll be
> > anything I don't expect it to be.
>
> That was one reason why I created a structure that mapped to the whole
> header: there was no need to know the details of how to move around
> in the file if we were just going to replop the whole blob of data
> there at the beginning. Also it modularized the code so that the
> WRITE statements that created the dummy and real header wouldn't have
> to change even the the header changed. Assuming the header isn't
> itself huge, the extra cost of rewriting parts of the header that
> were known at file creation time should be negligible.
Also see the POS= specifier in the inquire statement. I should have
mentioned that before, but I got distracted by more directly answering
the question of the file storage unit size instead. By using POS=, you
don't have to compute the position; you just ask what it is and you'll
get the right answer, avoiding all kinds of potential system
dependencies (that mostly won't happen, but with POS= you don't have to
worry about the rare cases).
glen herrmannsfeldt
(snip)
> That was one reason why I created a structure that mapped to the whole
> header: there was no need to know the details of how to move around
> in the file if we were just going to replop the whole blob of data
> there at the beginning.
Two popular file formats that need header information not available
until later, TeX's DVI and the ever popular PDF, put the header data
at the end. On many systems it is easier to seek and read the end
of the file first than it is to overwrite the beginning.
> Also it modularized the code so that the
> WRITE statements that created the dummy and real header wouldn't have
> to change even the the header changed. Assuming the header isn't
> itself huge, the extra cost of rewriting parts of the header that
> were known at file creation time should be negligible.
PDF has the additional feature that one can change or append to
the file, in which case a new Table of Contents is written at the
end describing both the old and new data. As the TOC has a variable
size, it is especially convenient to have it at the end.
-- glen
glen herrmannsfeldt
(snip)
> Also see the POS= specifier in the inquire statement. I should have
> mentioned that before, but I got distracted by more directly answering
> the question of the file storage unit size instead. By using POS=, you
> don't have to compute the position; you just ask what it is and you'll
> get the right answer, avoiding all kinds of potential system
> dependencies (that mostly won't happen, but with POS= you don't have to
> worry about the rare cases).
While the OP uses UNFORMATTED in the examples, there is also FORMATTED
STREAM. In that case, from 9.5.1.10:
"If the file is connected for formatted stream access, the
file position specified by POS= shall be equal to either 1
(the beginning of the file) or a value previously returned by a
POS= specifier in an INQUIRE statement for the file."
This restriction is similar to the restriction C places on text files.
I believe that C also allows fseek() to the end of the file,
even in the case of text files.
-- glen
Gordon Sande
> James Van Buskirk <not_...@comcast.net> wrote:
> (snip)
>
>> That was one reason why I created a structure that mapped to the whole
>> header: there was no need to know the details of how to move around
>> in the file if we were just going to replop the whole blob of data
>> there at the beginning.
>
> Two popular file formats that need header information not available
> until later, TeX's DVI and the ever popular PDF, put the header data
> at the end. On many systems it is easier to seek and read the end
> of the file first than it is to overwrite the beginning.
The PostScript desription that I found indicated that you needed to put
the page
count at the beginning but a permitted count was "count at end". So technically
no but pragmatically yes. It sounds like they would like the count at the
beginning but are realistic enough to do the work of reading to the end if
instructed. PDF is a number of PS jobs in one file with extra structure. It
was easier for me to generate the PS and use one of the PS to PDF utilites
to get to PDF. (It also lowered the number of cook books I had to read!)
This is a nice example of the use of a flag value. If the page count, or any
other size, is known give it but allow a flag value that causes the size to
be determined by a file scan.
Richard Maine
> Richard Maine <nos...@see.signature> wrote:
> (snip)
>
> > Also see the POS= specifier in the inquire statement....
>
> While the OP uses UNFORMATTED in the examples, there is also FORMATTED
> STREAM.
Yes, but that's a whole different story. Not only does the OP not happen
to use it, but it would be completely unsuited to his needs for many
reasons, one of them being the one that prompted his question in the
first place. He had a problem with Intel's form='binary' truncating the
file. Formatted stream can do exactly the same thing. Unformatted stream
does not.
I might add, even though it has nothing to do with the OP's issues, that
the restriction on POS= values in formatted stream is a typical piece of
standard-speak. It isn't as though any compiler is actually likely to
enforce the requirement that POS= values in a READ must come from
previous inquire statements. That isn't anticipated and I'd be surprised
to find any compiler bothering to try to enforce it. Seems like that
would be a lot of fuss for no useful purpose. The point is much more
that if you compute a value some other way (such as by adding up
expected byte counts) then you might not end up where you expected
(particularly if you didn't correctly account for possible
system-dependent record terminators). Accidentally positioning to the
middle of a multi-byte datum could be particularly awkward. As long as
you have managed to correctly compute a "reasonable" position, I'd be
quite surprised if any compiler rejected it. Sure it would be possible
for a compiler to keep track of what values had been returned by
inquire, but what a pointless implementation mess that would be unless
the file structure were strange enough that it took something like that
to make POS= work (in which case one might think it more likely that
such files would just be declared non-positionable.)
glen herrmannsfeldt
> glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
(snip)
>> While the OP uses UNFORMATTED in the examples, there is also
>> FORMATTED STREAM.
> I might add, even though it has nothing to do with the OP's issues, that
> the restriction on POS= values in formatted stream is a typical piece of
> standard-speak. It isn't as though any compiler is actually likely to
> enforce the requirement that POS= values in a READ must come from
> previous inquire statements. That isn't anticipated and I'd be surprised
> to find any compiler bothering to try to enforce it. Seems like that
> would be a lot of fuss for no useful purpose. The point is much more
> that if you compute a value some other way (such as by adding up
> expected byte counts) then you might not end up where you expected
> (particularly if you didn't correctly account for possible
> system-dependent record terminators). Accidentally positioning to the
> middle of a multi-byte datum could be particularly awkward. As long as
> you have managed to correctly compute a "reasonable" position, I'd be
> quite surprised if any compiler rejected it. Sure it would be possible
> for a compiler to keep track of what values had been returned by
> inquire, but what a pointless implementation mess that would be unless
> the file structure were strange enough that it took something like that
> to make POS= work (in which case one might think it more likely that
> such files would just be declared non-positionable.)
Specifically, in some cases it would be nice to seek to the end
of the file, back up a little bit, and then start reading.
That is, for example, what the unix tail command does. On most
systems the confustion of LF, CR, CRLF record terminators is all
that you would run into. It isn't so obvious what the I/O library
would do if you seek to the middle of a CRLF and start reading.
On the other hand, there are some IBM systems that don't keep track
of file positions as byte offsets, but as record number and record
offset. On at least one such system, ftell() returns the
value 32768*(record)+(offset), and fseek() expects those values.
(Records have a maximum length of 32767.) In that case, you might
find very strange results from adding or subtracting to the ftell()
(or INQUIRE) value.
-- glen
Richard Maine
> On the other hand, there are some IBM systems that don't keep track
> of file positions as byte offsets, but as record number and record
> offset. On at least one such system, ftell() returns the
> value 32768*(record)+(offset), and fseek() expects those values.
> (Records have a maximum length of 32767.) In that case, you might
> find very strange results from adding or subtracting to the ftell()
> (or INQUIRE) value.
I didn't know about that particular IBM scheme, but that's at least in
the general direction of what I was thinking about when I mentioned file
structures "strange" enough that you really only could use POS= values
returned from an inquire.
In your IBM example, the position still managed to get encoded into a
single integer and one still could in theory compute an appropriate
value as long as you knew the scheme.
I was imagining cases where the position could not reasonably be encoded
into a single default integer. In that case, I could imagine that
inquire could create a database of positions inquired about and that
POS= might just index into that database of inquire results. I don't
know of any actual systems that do anything like that, but I can imagine
such a thing. What a mess it would be though.
glen herrmannsfeldt
> glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
>> On the other hand, there are some IBM systems that don't keep track
>> of file positions as byte offsets, but as record number and record
>> offset. On at least one such system, ftell() returns the
>> value 32768*(record)+(offset), and fseek() expects those values.
>> (Records have a maximum length of 32767.) In that case, you might
>> find very strange results from adding or subtracting to the ftell()
>> (or INQUIRE) value.
> I didn't know about that particular IBM scheme, but that's at least in
> the general direction of what I was thinking about when I mentioned file
> structures "strange" enough that you really only could use POS= values
> returned from an inquire.
> In your IBM example, the position still managed to get encoded into a
> single integer and one still could in theory compute an appropriate
> value as long as you knew the scheme.
> I was imagining cases where the position could not reasonably be encoded
> into a single default integer.
Well, that is almost the case above, as the number of records is not
limited to 131072. Also, in the case of variable record length binary
files the above systems is still used, contrary to the C standard.
> In that case, I could imagine that
> inquire could create a database of positions inquired about and that
> POS= might just index into that database of inquire results. I don't
> know of any actual systems that do anything like that, but I can imagine
> such a thing. What a mess it would be though.
Or read through the whole file from the beginning and store the
record offsets into a big table. Then use that table for seeks.
Slow at the beginning but fast after that.
OS/360 direct access files are fixed record length, and unblocked.
Inefficient for small block sizes.
-- glen
Ron Shepard
glen herrmannsfeldt <g...@ugcs.caltech.edu> wrote:
> Specifically, in some cases it would be nice to seek to the end
> of the file, back up a little bit, and then start reading.
How portable is this:
open(unit,...position='append'...)
rewind unit
read(unit) last_info
If something like this works, and is efficient and portable, then this
is what I think I would recommend. Almost all file types allow
overwriting records at the end of the file. For things like magnetic
tape devices, this is the only possible way to overwrite information.
Is is sort of an odd concept to overwrite records at the beginning or in
the middle of a file and expect all of the information afterwards to
remain accessible. In fortran, direct access allows this, but only for
fixed-length records, not for arbitrary mixtures of record lengths. If
you think about how information is written to most external devices
(magnetic or optical in nature), you can see why this is the case. Even
a file stored to magnetic disk (a direct access device) is restricted to
write an entire sector at a time. If you want to overwrite only part of
the sector, and expect the original information to be retained, then
either you or the I/O library must first read the original information,
replace the part you are overwriting, and then the full sector must be
replaced. Although most external devices now are disk-like rather than
tape-like, some effort is required to make things appear "as if" they
are byte- or word-addressable.
So, unless you would prefer to write two separate files (one with the
data, the other with the last_info record describing how the data are
written), I would recommend putting the header information at the end of
the file rather than the beginning. Especially if the above works, it
is difficult to imagine anything that is simpler.
$.02 -Ron Shepard
Dick Hendrickson
> In article<hkptjr$rs9$6...@naig.caltech.edu>,
> glen herrmannsfeldt<g...@ugcs.caltech.edu> wrote:
>
>> Specifically, in some cases it would be nice to seek to the end
>> of the file, back up a little bit, and then start reading.
>
> How portable is this:
>
> open(unit,...position='append'...)
> rewind unit
I also wondered why the OP didn't do this.
Dick Hendrickson
robin
|
| I did include a remark about the possibility of using access='direct', but
| would prefer not to do that for a number of reasons. Primarily, I'd like the
| flexibility of rewriting various portions of the binary files I use and
| frequently the actual position will depend on the data that precedes the
| location where the data is to be written.
access = "direct" is a good way of updating individual records of a file.
John Paine
"Dick Hendrickson" <dick.hen...@att.net> wrote in message
news:7tbssi...@mid.individual.net...
> On 2/8/10 4:38 PM, Ron Shepard wrote:
>> In article<hkptjr$rs9$6...@naig.caltech.edu>,
>> glen herrmannsfeldt<g...@ugcs.caltech.edu> wrote:
>>
>>> Specifically, in some cases it would be nice to seek to the end
>>> of the file, back up a little bit, and then start reading.
>>
>> How portable is this:
>>
>> open(unit,...position='append'...)
>> rewind unit
> I bet you mean "backspace unit"
>
> I also wondered why the OP didn't do this.
>
This approach would work if I placed the header block at the end of the
file, but not if it is at the start of the file as writes using 'binary'
format truncate the file. Binary stream files don't have this problem. But
for such files I can do exactly what I want using the POS= keyword on the
write, so there's no need to change what I have already implemented.
Naturally if my header record was not a fixed size, then placing it at the
end of the file would probably be the best solution as otherwise I'd have to
allocate extra space for the header at the start of the file or adopt a two
pass process of writing the data to a scratch file, writing the required
header after the data had been created and then appending the data from the
scratch file. But in my case, I know the size of the header in advance and
just need to update the contents once I have all of the data.
At a broader level, I wanted to avoid solutions like this (ie moving the
header to the end or to a separate file) partly because some of the file
formats are not mine and the required information has to be at specific
locations in the file. But even for formats I do control, I prefer having
the information at the start of the file as I can then set up all of the
data arrays immediately after opening the file and before actually reading
the data. Some of the files can be quite large, so reading to the end to
find the defining header record seems somehow wrong to me. I recognize that
this approach is certainly not difficult and is also not uncommon (see
Glen's comments about pdf and DVI files), so this truly is a preference
rather than a necessity. I prefer to have programs operate the way that I
want them to rather than be limited by what the compiler allows me to do
(within reason).
From comments by other posters, I gather that some of the basic tenets of
file access are still grounded in the exigencies of reading tapes, and in
that context truncate after write presumably makes perfect sense (though I'm
sure that there was a way to rewrite records on tape too, but it may have
involved using an assembler routine to stop the tape writing the EOF
marker). However it's been a long time since I or anyone I work for have
used tape units as part of a data processing stream, so being able to
rewrite a portion of a file without truncating the file seems to me to be a
perfectly reasonable thing to do and something the computer itself is also
perfectly capable of performing. My failure was that I have lagged badly in
keeping up with changes in the language and knew nothing about stream files.
Thankfully the newsgroup is alive and well and put me on the right path very
quickly.
Ron Shepard
Dick Hendrickson <dick.hen...@att.net> wrote:
> > open(unit,...position='append'...)
> > rewind unit
> I bet you mean "backspace unit"
>
> I also wondered why the OP didn't do this.
Oops, yes. I meant backspace there rather than rewind. I
originally had a longer example in mind
(open,backspace,read,rewind,read,write) and decided to shorten it,
but I deleted the wrong line.
$.02 -Ron Shepard
Ron Shepard
"John Paine" <johnp...@optusnet.com.au> wrote:
> >> open(unit,...position='append'...)
> >> rewind unit
> > I bet you mean "backspace unit"
> >
> > I also wondered why the OP didn't do this.
Yes, the example should have been
open(unit,...position='append'...)
backspace unit
read(unit) last_info
>
>[...]
> Some of the files can be quite large, so reading to the end to
> find the defining header record seems somehow wrong to me.[...]
My example was not clear because of my mistake of using rewind
rather than backspace. As you can see, this does not read through
the entire file.
$.02 -Ron Shepard
glen herrmannsfeldt
(snip)
> From comments by other posters, I gather that some of the basic tenets of
> file access are still grounded in the exigencies of reading tapes, and in
> that context truncate after write presumably makes perfect sense (though I'm
> sure that there was a way to rewrite records on tape too, but it may have
> involved using an assembler routine to stop the tape writing the EOF
> marker). However it's been a long time since I or anyone I work for have
> used tape units as part of a data processing stream, so being able to
> rewrite a portion of a file without truncating the file seems to me to be a
> perfectly reasonable thing to do and something the computer itself is also
> perfectly capable of performing. My failure was that I have lagged badly in
> keeping up with changes in the language and knew nothing about stream files.
> Thankfully the newsgroup is alive and well and put me on the right path very
> quickly.
The reason is more complicated than that. The format of data on disks,
including gaps between blocks, is carefully designed such that one can
fully overwrite a block, yet not write over the beginning of the
following block. There have been tape systems that did that, but
many did not. DECtape, a tape system used by many early DEC computer,
was designed that way. That is, as a direct access tape system.
Disk low level formats have a block header with the address
(cylinder, track, sector) for that block positioned such that the
header can be read and then, at just the right time, the new
data written. As the disk speed may vary with time or between
drives, the size of the gap following a data block is very
important. The drive must write 'idle' data long enough to
overwrite the end of a block previously written at the high end
of the speed tolerance, while now writing at the low end.
-- glen
Louis Krupp
There are variations on PDF that provide for some information at the
beginning of the file (such as enough info to display the first page
when the file is opened by a web browser), but in general, everything
you really need to know about the PDF file is at the end. (Except for
the PDF version, which is at the beginning. And which may or may not be
accurate.) Specifically, you wouldn't want to look for a page count at
the beginning of a PDF file.
There is a lot of Postscript, and stuff that looks and parses like
Postscript, embedded in a PDF file, but PDF is more (or less, depending
on your point of view) than a collection of Postscript jobs. It's a
collection of objects, some of which might be encrypted. Some of these
objects are nodes in a page tree (you have to traverse the non-leaf
nodes to get the page count), and others contain data (including page
commands whose syntax mimics Postscript).
The file is not usually read from beginning to end unless one of the
cross reference tables is damaged (this happens), in which case the
reader program has to scan for strings like '<obj #> <gen #> obj' and
note which objects appear where in the file.
A PDF file generated from Postscript might have a lot of stuff in it
that looks familiar, but the structure is different in strange and
significant ways.
Louis