OPEN Stream File w/ Position='APPEND'

[Gary Scott]

I'm not seeing why opening a stream access file with the position as
append should not work. I receive no error, but a write statement
appears to overwrite rather than append. I have more testing to do, but
is there some obvious incompatibility?

[steve kargl]

Quite a bit of detail missing. So, I suspect the answer is 'yes/no/maybe'.

--
steve

[urba...@comcast.net]

Seems to work as expected with a simple test program. Can you make this show your problem?

program testit
integer :: IOS
character(len=256) :: message
open(unit=10,file='testfile')
write(10,'(a)')'OLD DATA '
close(unit=10)

open( &
& UNIT=10, &
& FILE='testfile', &
& ACCESS='stream', &
& ACTION='readwrite', &
& FORM='unformatted', &
& IOMSG=message, &
& IOSTAT=ios, &
& POSITION='append', &
& STATUS='old' )

if (ios < 0) then
! Perform end-of-file processing on the file connected to unit 3.
stop
elseif (ios > 0) then
! Perform error processing
write(*,'(a)')trim(message)
stop
else
write(10)'NEW DATA '
endif
end program testit

$ cat testfile
OLD DATA
NEW DATA

[Gary Scott]

this reflects my open statement except omitted form=, so defaulted. All
io statements are without format statements (write(lfn)variable

It is replacing rather than overwriting. I probably have something
wrong somewhere. Ran out of time, will test further over the weekend.

[Gary Scott]

This was of course an obvious coding error that I have fixed and now
have working properly. But a related issue is one of convenience and
efficiency. So, I first want to open the file at the beginning and
overwrite a time stamp. Then I want to position to the end and begin
appending new records. That works fine. However, I appear to have to
close the file and then reopen the file specifying append position. I
would rather just "wind" to the end (opposite of rewind) and begin
writing. It seems inefficient to both have to close the file and reopen
with append status or to manually read forward in a loop until EOF is
hit, then begin writing. I see a POS= keyword on the write statement
but that requires a numerical position. This would be more useful if
you could specify something like "-1" to indicate go to the append
position. However, I'd prefer not to add the extra overhead of
processing that on the write statement and have a separate "wind"
statement. Other options?

[steve kargl]

Do you really need to close the file? F2018 says

An OPEN statement initiates or modifies the connection between an
external file and a specified unit. The OPEN statement may be used
to connect an existing file to a unit, create a file that is preconnected,
create a file and connect it to a unit, or change certain modes of a
connection between a file and a unit.

Have simply tried using

OPEN(unit=fd, position="APPEND")

where 'fd' is an already connected UNIT?

--
steve

[steve kargl]

This does what one expects.

program foo
integer fd
open(newunit=fd, file='tmp.txt', access='stream')
write(fd) 'abc'
open(fd, position='rewind')
write(fd) 'z'
open(fd, position='append')
write(fd) 'xy'
end program foo

% gfcx -o z a.f90 && ./z
% hexdump -C tmp.txt
00000000 7a 62 63 78 79 |zbcxy|
00000005

[Gary Scott]

Cool, still seems like there would be more overhead than a simple
directed positioning statement, but I will change to this if IVF likes it.

[Gary Scott]

Interestingly, ifort issues this message (in both cases):

104 incorrect POSITION= specifier value for connected file, unit -181,
file C:\Fortr...

Also interestingly, 104 is not listed in the help file. It skips from 98
to 106


98

Severe (98): cannot allocate memory for the file buffer - out of memory

FOR_S_NOMEMFORIO. This error often occurs during a file I-O operation
such as OPEN, READ, or WRITE. Either increase the amount of memory
available to the program, or reduce its demand.

106

Severe (106): FORT_BLOCKSIZE environment variable has erroneous syntax

FOR_S_INVBLOCKSIZE. Syntax for specifying the default block size value
was incorrect. For correct syntax, see Environment Variable FORT_BLOCKSIZE.


-----------------------------------

open(newUnit=lfnData,file=trim(filePathData),access='stream',position='REWIND',status='UNKNOWN',action='READWRITE',share='DENYNONE',ioStat=iStat)

...

open(newUnit=lfnInsert,file=trim(filePathInsert),access='stream',position='REWIND',status='UNKNOWN',action='READWRITE',share='DENYNONE',ioStat=iStat)

write(lfnData,ioStat=iStat)maxTS !If in server mode...

write(lfnInsert,ioStat=iStat)maxTS !If in server mode...

open(lfnData,position='APPEND',ioStat=iStat,iomsg=message)

write(10,*)istat,message

open(lfnInsert,position='APPEND',ioStat=iStat)

write(10,*)istat

[urba...@comcast.net]

Without a specific example cannot say. As noted, you should not have to close the file if nothing else is changing, like changing from an unformatted string to a formatted stream or sequential file. Since you are replacing a line and not using a direct access filel if I assume you are using an unformatted stream throughout and that the line at the top you are replacing is the same length as the new line
you should be able to do

program testit
integer :: IOS
character(len=256) :: message

! make sample file
open(unit=10,file='testfile')
write(10,'(a)')'OLD DATE',('OLD DATA ',i=1,10)
close(unit=10)
! now edit the file, simulating your program description

open( &
& UNIT=10, &
& FILE='testfile', &
& ACCESS='stream', &

& IOMSG=message, &
& IOSTAT=ios, &

& STATUS='old' )

if (ios .ne. 0) then

! Perform error processing
write(*,'(a)')trim(message)
stop
else

write(10)'NEW DATE'
endif
open(unit=10,position='append')
! possibly build the string using a format into a character variable
write(10)('NEW DATA AT END',new_line('A'),i=1,10)
end program testit


In a lot of other cases you would have to close and reopen, such as changing to FORM='formatted' or switching to sequential I/O. The above is an example that, as noted in the previous post, you not have to always close a file to do subsequent OPEN(3f). Depends on what you are changing. As far as efficiency goes
closing and opening the file is not inefficient. If the file is big doing READ(3f)s to get to the end could be. But I am unclear on whether you sometimes want to treat the file as a stream and other times want it to be a sequential formatted file or not. The output of the above should be

NEW DATE
OLD DATA
OLD DATA
OLD DATA
OLD DATA
OLD DATA
OLD DATA
OLD DATA
OLD DATA
OLD DATA
OLD DATA
NEW DATA AT END
NEW DATA AT END
NEW DATA AT END
NEW DATA AT END
NEW DATA AT END
NEW DATA AT END
NEW DATA AT END
NEW DATA AT END
NEW DATA AT END
NEW DATA AT END

It is important that you can assume the new date is the same length as the old date.

[Gary Scott]

On 10/27/2018 1:58 PM, urba...@comcast.net wrote:
> Without a specific example cannot say. As noted, you should not have to close the file if nothing else is changing, like changing from an unformatted string to a formatted stream or sequential file. Since you are replacing a line and not using a direct access filel if I assume you are using an unformatted stream throughout and that the line at the top you are replacing is the same length as the new line
> you should be able to do

yes of course. all is working fine. I tried changing to a simple open
with a position statement and got an error with IVF, still
investigating. See other post.

[Gary Scott]

This form of open with position= as the only change does not seem to be
supported in IVF 2017.5.267. I can first close and reopen with a
position= keyword, but it doesn't seem to accept the keyword except on a
new open operation...it was an interesting idea, but seems not worth
wasting more time on at this time since I have more important changes.

[Gary Scott]

On 10/27/2018 2:29 PM, Gary Scott wrote:
snip

>>> Cool, still seems like there would be more overhead than a simple
>>> directed positioning statement, but I will change to this if IVF
>>> likes it.
>> Interestingly, ifort issues this message (in both cases):
>>
>> 104 incorrect POSITION= specifier value for connected file, unit -181,
>> file C:\Fortr...

snip

>>
>> -----------------------------------
>>
>> open(newUnit=lfnData,file=trim(filePathData),access='stream',position='REWIND',status='UNKNOWN',action='READWRITE',share='DENYNONE',ioStat=iStat)

>> open(newUnit=lfnInsert,file=trim(filePathInsert),access='stream',position='REWIND',status='UNKNOWN',action='READWRITE',share='DENYNONE',ioStat=iStat)
>>
>>
>>        write(lfnData,ioStat=iStat)maxTS !If in server mode...
>>
>>        write(lfnInsert,ioStat=iStat)maxTS !If in server mode...
>>
>>        open(lfnData,position='APPEND',ioStat=iStat,iomsg=message)
>>
>> write(10,*)istat,message
>>
>>        open(lfnInsert,position='APPEND',ioStat=iStat)
>>
>> write(10,*)istat
>
> This form of open with position= as the only change does not seem to be
> supported in IVF 2017.5.267.  I can first close and reopen with a
> position= keyword, but it doesn't seem to accept the keyword except on a
> new open operation...it was an interesting idea, but seems not worth
> wasting more time on at this time since I have more important changes.

Steve Lionel (Ret.) posted:

Your program violates the Fortran standard. gfortran may allow it as an
extension. Here's what the standard (F2018 FDIS) says (emphasis mine) If
the file to be connected to the unit is the: same as the file to which
the unit is connected, a new connection is not established and values
for any changeable modes (12.5.2) specified come into effect for the
established connection the current file position is unaffected. Before
any effect on changeable modes, a wait operation is performed for any
pending asynchronous data transfer operations for the specified unit. If
the POSITION= specifier appears in such an OPEN statement, the value
specified shall not disagree with the current position of the file. If
the STATUS= specifier is included in such an OPEN statement, it shall be
specified with the value OLD. Other than ERR=, IOSTAT=, and IOMSG=, and
the changeable modes, the values of all other specifiers in such an OPEN
statement shall not differ from those in effect for the established
connection. In your example, the current position is in the middle of
the file, not at the end. What I think you can do is this: Open the file
POSITION='APPEND'. Do an INQUIRE (ifnData,POS=lastpos) ! Declare lastpos
as integer REWIND (ifnData) Do your writes Do a WRITE
(ifnData,POS=lastpos) ! with no I/O list Append your records. I tested
this as follows: open
(1,file='stm.dat',form='unformatted',access='stream') do i=1,10 write
(1) i end do inquire (1,POS=ipos) rewind (1) write (1) 11 write (1) 12
write (1,POS=ipos) write (1) 13 write (1) 14 rewind (1) do read
(1,end=99) k print *, k end do 99 end And got: D:\Projects>t.exe 11 12 3
4 5 6 7 8 9 10 13 14

[steve kargl]

Gary Scott wrote:

> This form of open with position= as the only change does not seem to be
> supported in IVF 2017.5.267. I can first close and reopen with a
> position= keyword, but it doesn't seem to accept the keyword except on a
> new open operation...it was an interesting idea, but seems not worth
> wasting more time on at this time since I have more important changes.

Looks like a bug report to Intel may be appropriate as position='append' has
been a part of the Fortran standard since Fortran 2003.

--
steve

[Steve Lionel]

Gary Scott posted my reply from the Intel forum, though the formatting
got lost.

Quoting from F2018 FDIS 12.5.6.1 (OPEN statement > General), emphasis mine:

If the file to be connected to the unit is the: same as the file to
which the unit is connected, a new connection is not established and
values for any changeable modes (12.5.2) specified come into effect for
the established connection the current file position is unaffected.
Before any effect on changeable modes, a wait operation is performed for
any pending asynchronous data transfer operations for the specified

unit. *If the POSITION= specifier appears in such an OPEN statement, the

value specified shall not disagree with the current position of the

file.* If the STATUS= specifier is included in such an OPEN statement,

it shall be specified with the value OLD. Other than ERR=, IOSTAT=, and
IOMSG=, and the changeable modes, the values of all other specifiers in
such an OPEN statement shall not differ from those in effect for the
established connection.

The program does "open on a connected unit" (probably my least favorite
feature in the language) with POSITION='APPEND' but the current position
is (presumably - it's an existing file) not at the end. Processors are
not required to detect this violation, and if gfortran supports this as
an extension, that's fine.

However, it does sound like the Intel implementation provides suboptimal
diagnostics in this case....

I suggested an alternate approach, where one opens the file initially
with 'APPEND' rather than 'REWIND', uses INQUIRE to get the current
position, REWIND, write the initial records and then do a WRITE with
POS= the saved position (and no I/O list). This is standard-conforming
and accomplishes what he asked for. (Except that Gary doesn't like this
solution for reasons not yet explained.)

--
Steve Lionel
Retired Intel Fortran developer/support
Email: firstname at firstnamelastname dot com
Twitter: @DoctorFortran
LinkedIn: https://www.linkedin.com/in/stevelionel
Blog: http://intel.com/software/DrFortran

[Steve Lionel]

On 10/27/2018 8:22 PM, Steve Lionel wrote:
> However, it does sound like the Intel implementation provides suboptimal
> diagnostics in this case....

No, it's fine. I don't get the sort of results Gary does. Just:

forrtl: severe (104): incorrect POSITION= specifier value for connected
file, unit 1, file D:\Projects\stm.dat

That's a perfectly acceptable error message. I don't see other messages,
but maybe Gary ran different code.

[urba...@comcast.net]

Does look like the standard precludes not doing the close but so far I haven't tried a compiler that enforces that, so it seems like a somewhat arbitrary limit;
especially since the standard requires it to be allowed at run time if the position matches the current position (meaning it has to do a run-time check anyway, so why not just position it to the desired spot?). I can see where if you had to implement a run-time warning that the requested position did not map the current position when you did the open on the connected unit it would just be as easy to jus position it where the open requested.

Just for grins I made an "unwind" procedure as mentioned in the initial post that basically uses INQUIRE and queries all the parameters, closes the file and reopens the file with APPEND mode that I believe is standard-conforming and meets the original goal. I did not think I would use it, but I am starting to think of other uses for it and for a TYPE that basically holds the current state of a file so I might go further down the rathole.

[FortranFan]

On Saturday, October 27, 2018 at 8:22:54 PM UTC-4, Steve Lionel wrote:

> ..
> I suggested an alternate approach, where one opens the file initially
> with 'APPEND' rather than 'REWIND', uses INQUIRE to get the current
> position, REWIND, write the initial records and then do a WRITE with
> POS= the saved position (and no I/O list). This is standard-conforming
> and accomplishes what he asked for. (Except that Gary doesn't like this
> solution for reasons not yet explained.)

> ..


The advantage of "STREAM IO" is that one can easily perform data transfer in almost any order.

Given this facility, why bother to open with 'APPEND' to save the position, then 'REWIND' to write some initial data followed by transfer of the extra data starting at the saved position plus 1?

OP writes on the Intel forum, "I was attempting to first open in the rewind position and overwrite a time stamp, then simply reposition to the end and begin appending records."

So why not just open with 'APPEND', write the extra data, then transfer the updated initial data (say time stamp) at initial position which is simply 1? See below where subprogram FPRINT mimics OP's stated desire. The code works fine with both Intel Fortran as well as gfortran:

--- begin example ---
program p
use, intrinsic :: iso_fortran_env, only : compiler_version, iostat_end, iostat_eor
implicit none
character(len=*), parameter :: fname = "C:\temp\tmp.dat"
character(len=*), parameter :: msg = " Hello World! "
character(len=8) :: sdate
character(len=10) :: stime
print *, "Compiler Version: ", compiler_version()
call date_and_time( sdate, stime )
call iprint()
call rfile()
call fprint()
call rfile()
stop
contains
subroutine iprint()
integer :: lun
open( newunit=lun, file=fname, access="stream", form="unformatted", status="replace" )
write( lun ) sdate, stime, msg
close( lun )
return
end subroutine
subroutine fprint()
integer :: lun
integer :: istat
character(len=2048) :: imsg
character(len=8) :: ldate
character(len=10) :: ltime
character(len=*), parameter :: emsg = "I'm glad Fortran introduced stream access!"
! Open with position="append"
open( newunit=lun, file=fname, access="stream", form="unformatted", status="old", &
position="append", iostat=istat, iomsg=imsg )
if ( istat /= 0 ) then
print *, "fprint: open failed - iostat=", istat
print *, trim(imsg)
stop
end if
write( lun ) emsg ! write extra data
call date_and_time( ldate, ltime )
write( lun, pos=1 ) ldate, ltime ! update time stamp at beginning of file
close( lun )
return
end subroutine
subroutine rfile()
integer :: lun
integer :: istat
integer :: i
integer, parameter :: iostat_zero = 0
character(len=2048) :: imsg
open( newunit=lun, file=fname, access="stream", form="unformatted", status="old", &
position="rewind", iostat=istat, iomsg=imsg )
if ( istat /= 0 ) then
print *, "fprint: open failed - iostat=", istat
print *, trim(imsg)
stop
end if
i = 0
imsg = ""
loop_read: do
i = i + 1
if ( i > len(imsg) ) then
print *, "File content exceeds buffer, read aborted."
exit loop_read
end if
read( lun, pos=i, iostat=istat ) imsg(i:i)
select case ( istat )
case ( iostat_zero )
! continue reading
case ( iostat_end, iostat_eor )
exit loop_read
case default
print *, "Error reading file at position ", i
stop
end select
end do loop_read
close( lun )
print *, "Contents of ", trim(fname), ":"
print *, trim(imsg)
return
end subroutine
end program p
--- end example ---

Upon execution with gfortran:
Compiler Version: GCC version 9.0.0 20181014 (experimental)
Contents of C:\temp\tmp.dat:
20181027231708.121 Hello World!
Contents of C:\temp\tmp.dat:
20181027231708.128 Hello World! I'm glad Fortran introduced stream access!


Upon execution with Intel Fortran:
Compiler Version:
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 19.0.0.117 Build 20180804
Contents of C:\temp\tmp.dat:
20181027231735.491 Hello World!
Contents of C:\temp\tmp.dat:
20181027231735.499 Hello World! I'm glad Fortran introduced stream access!

[Gary Scott]

I didn't say I didn't like it. I said I would look into it further. It
does seem that all of the proposed solutions are somewhat inelegant.
But it isn't that important that I can't use one of them. FF proposed
probably the best option which is to open as append and write all of the
appended records, THEN return to position 1 and update the timestamp.
That seems sort of obvious, but is reverse of what the logical serial
task implies. But in the end is probably superior because the multiple
accessing shared file readers would continue to get the old time stamp
and other control information and not attempt to read new content before
the new content has been written.

[Gary Scott]

On 10/27/2018 7:28 PM, Steve Lionel wrote:
> On 10/27/2018 8:22 PM, Steve Lionel wrote:
>> However, it does sound like the Intel implementation provides
>> suboptimal diagnostics in this case....
>
> No, it's fine. I don't get the sort of results Gary does. Just:
>
> forrtl: severe (104): incorrect POSITION= specifier value for connected
> file, unit 1, file D:\Projects\stm.dat
>
> That's a perfectly acceptable error message. I don't see other messages,
> but maybe Gary ran different code.
>

Thats the only message I received. I was merely reporting that error
104 was missing from my list of error messages in the help file.

[Ron Shepard]

On 10/28/18 8:00 AM, Gary Scott wrote:
> [...] It

> does seem that all of the proposed solutions are somewhat inelegant. But
> it isn't that important that I can't use one of them. FF proposed
> probably the best option which is to open as append and write all of the
> appended records, THEN return to position 1 and update the timestamp.
> That seems sort of obvious, but is reverse of what the logical serial

> task implies. [...]

It seems that you have the file positions for both the beginning and the
end of the file, so you are free to write the information in any order
you want. Fortran provides you with a character addressable file, I
don't know how you can be any more elegant than that. To me, that seems
more elegant than any other possible combinations of OPEN, CLOSE,
REWIND, READ, BACKSPACE, and WRITE statements that you could put together.

You apparently have a running program, and you want the information
communicated within it to be available to other programs through this
file. This raises the separate issue of what else is required to achieve
that other than writing the information to the file? Must the running
program complete execution? If not, then can you just close the file? Is
it sufficient to just write the information, or would a WRITE followed
by a FLUSH be sufficient? If it is a short file, then it is almost
certainly held in i/o buffers within the fortran library, and the OS may
buffer file operations even beyond that. If it is a shared file system
of some kind, then the computer the program in executing on must also
communicate to the remote computer or the remote file server, and then
the other computers must sync with that remote file server to get the
updated information. Exactly how much of that process is under control
of the fortran program?

$.02 -Ron Shepard

[Gary Scott]

All applications are 100% Fortran. This is the "backup" client server
communication process. If the server (application, not the shared
folder) is down, clients can "publish" to the server directory so that
when the server is back up, it can process the missed content.
Similarly for server to client communications. Could be
obsessive/compulsive, but the available server hardware is way
underpowered and iffy and I like redundancy. It was working very well,
but I decided that the structure would cause the one file to grow
indefinitely and needed to be changed to address that.

I've not had any issue with flushing (I thought I did at one point but
was a different issue).

I've decided to redesign as direct access files. Not as efficient use
of storage space, but makes it easier to create a content index by an
initial sequential read process and then seek directly to the item of
interest (comparison match) and for storing, to overwrite rather than
append the same UUID at the end with newer values. Almost done. Yes I
could have done all that with stream access, but not as easily. I also
included some unused/reserve space in each record for growth.

>
> $.02 -Ron Shepard

[Louis Krupp]

On Sun, 28 Oct 2018 14:35:18 -0500, Gary Scott
<garyl...@sbcglobal.net> wrote:

<snip>

>
>All applications are 100% Fortran. This is the "backup" client server
>communication process. If the server (application, not the shared
>folder) is down, clients can "publish" to the server directory so that
>when the server is back up, it can process the missed content.
>Similarly for server to client communications. Could be
>obsessive/compulsive, but the available server hardware is way
>underpowered and iffy and I like redundancy. It was working very well,
>but I decided that the structure would cause the one file to grow
>indefinitely and needed to be changed to address that.
>
>I've not had any issue with flushing (I thought I did at one point but
>was a different issue).
>
>I've decided to redesign as direct access files. Not as efficient use
>of storage space, but makes it easier to create a content index by an
>initial sequential read process and then seek directly to the item of
>interest (comparison match) and for storing, to overwrite rather than
>append the same UUID at the end with newer values. Almost done. Yes I
>could have done all that with stream access, but not as easily. I also
>included some unused/reserve space in each record for growth.

If the transaction volume is reasonable (for some definition of the
word), you might want to consider creating a new file, with a name
based on the current date and time, for each transaction. Your server
could sort the files by name and open and delete them one at a time.
This might be more robust than reading and writing one file and
worrying about locks and race conditions.

If the volume is higher than what seems "reasonable," you might look
at something like this:

https://www.rabbitmq.com

I haven't used this product. It advertises APIs for C and C++, so
you'd have to interface one of those to Fortran. Your application
would be more complicated, but if what you're currently doing doesn't
work as well as you'd like, it might make sense to take advantage of
the work that other people have already done.

Louis

[Gary Scott]

I already do that for the client files. They consist of a true UUID in
hex (with some other syntax). Each client export operation will have a
new name, but everything that's available for export will be in the
file. The server automatically deletes them once they are processed.