Re: [gfortran43] opening file twice...
dpb
> Hi,
>
> I googled on this issue but I did not find anything relevant.
>
> Say I have to open a file twice (or more):
>
> open(unit=20, file='a.dat', form='unformatted', access='stream')
> open(unit=21, file='a.dat', form='unformatted', access='stream')
>
> Running my program, I get:
>
> Fortran runtime error: File already opened in another unit
>
> How could I fix it ?
Can't have same file open for two units simultaneously.
> Option to gfortan or special flag to open() ?
Nope, don't do that.
--
Dave Allured
>
> Hi,
>
> I googled on this issue but I did not find anything relevant.
>
> Say I have to open a file twice (or more):
>
> open(unit=20, file='a.dat', form='unformatted', access='stream')
> open(unit=21, file='a.dat', form='unformatted', access='stream')
>
> Running my program, I get:
>
> Fortran runtime error: File already opened in another unit
>
> How could I fix it ?
>
>
It seems to me that this should be legal. My training was that
Unix-like file systems allow multiple read access on the same file. I
think this is a gfortran issue.
I ran into something related (platform = Mac OS 10):
open(unit=20, file='a.dat')
[lots of processing, but no close statement]
open(unit=20, file='a.dat')
Sun fortran and XLF 8.1 accept this without complaint. Gfortran gives a
run time warning message like "file is already open" on the second open
statement. But then it proceeds to do the right thing, re-open the file
at the beginning. HTH.
--Dave
dpb
> dpb <no...@non.net> a écrit :
...
>> Can't have same file open for two units simultaneously.
> And my threads have to read the same file.
>
> Works fine with ifort, though.
That's an extension to the Standard.
From CVF OPEN
OPEN
Statement: Connects an external file to a unit, creates a new file and
connects it to a unit, creates a preconnected file, or changes certain
properties of a connection.
...
Only one unit at a time can be connected to a file, but multiple OPENs
can be performed on the same unit. ...
> Or... is there another work around ?
...
Would take using the system APIs directly I think.
I generally tend to use memory-mapped files for such things but that
also circumvents "normal" Fortran I/O.
--
GaryScott
> fred wrote:
> > dpb <n...@non.net> a écrit :
In CVF and IVF, you can use the same unit number and :
SHARE = shr
shr - Is a scalar default character expression that evaluates to one
of the following values:
'DENYRW' Indicates deny-read/write mode. No other process can open
the file.
'DENYWR' Indicates deny-write mode. No process can open the file with
write access.
'DENYRD' Indicates deny-read mode. No process can open the file with
read access.
'DENYNONE' Indicates deny-none mode. Any process can open the file
in any mode.
Dick Hendrickson
> fred wrote:
>> Hi,
>>
>> I googled on this issue but I did not find anything relevant.
>>
>> Say I have to open a file twice (or more):
>>
>> open(unit=20, file='a.dat', form='unformatted', access='stream')
>> open(unit=21, file='a.dat', form='unformatted', access='stream')
>>
>> Running my program, I get:
>>
>> Fortran runtime error: File already opened in another unit
>>
>> How could I fix it ?
>>
>> Option to gfortan or special flag to open() ?
>>
>> TIA.
>
> It seems to me that this should be legal. My training was that
> Unix-like file systems allow multiple read access on the same file. I
> think this is a gfortran issue.
No, it is specifically forbidden by the standard. Mostly, because it's
impossible to specify what it could mean on a wide variety of systems.
What should happen if there are non-advancing reads on the two
"different" units? A rewind on one of them? Or a write or a close?
If 2 is OK, why not a hundred? It's a can of worms that can't
work portably.
>
> I ran into something related (platform = Mac OS 10):
>
> open(unit=20, file='a.dat')
> [lots of processing, but no close statement]
> open(unit=20, file='a.dat')
This is specifically allowed. There are a set of changeable connection
mode specifiers which allow certain file properties, such as the BLANK
interpretation mode, to be changed. In your example, you aren't
changing any of them, which is a little unusual.
>
> Sun fortran and XLF 8.1 accept this without complaint. Gfortran gives a
> run time warning message like "file is already open" on the second open
> statement. But then it proceeds to do the right thing, re-open the file
> at the beginning. HTH.
I don't think it should reopen the file at the beginning. I believe the
position specifier default is ASIS and a reopen is not allowed to change
this.
You also need to be careful about wording. "re-open" is a tricky word,
the standard does not say the file is closed and then opened; it says
that certain connection properties are potentially changed. That's
important if you expect the file to be repositioned or want to do reads
after writes or anything else that depends on a close taking place.
Dick Hendrickson.
>
> --Dave
dpb
...
> In CVF and IVF, you can use the same unit number and :
>
> SHARE = shr
...
Yes, ooops...I'd forgotten it.
Note to OP, though -- it _is_ an extension to Standard.
--
Richard Maine
Of course, recall also that the OP said he needed this because of
threads. Basically anything relating to threads is an extension. I/O is
one area known to have complications. If you want to have multiple
tghreads doing I/O to the same file, you pretty much had better be aware
that the whole business is nonstandard and has potential portability
issues.
That's not to say one shouldn't do it. But just that you aren't going to
find an answer in the Fortran standard, and whatever answer you might
find elsewhere might not be portable.
--
Richard Maine | Good judgment comes from experience;
email: last name at domain . net | experience comes from bad judgment.
domain: summertriangle | -- Mark Twain
dpb
> dpb <no...@non.net> wrote:
>
>> GaryScott wrote:
>> ...
>>
>>> In CVF and IVF, you can use the same unit number and :
>>>
>>> SHARE = shr
>> ...
>>
>> Yes, ooops...I'd forgotten it.
>>
>> Note to OP, though -- it _is_ an extension to Standard.
>
> Of course, recall also that the OP said he needed this because of
> threads. Basically anything relating to threads is an extension. I/O is
> one area known to have complications. If you want to have multiple
> tghreads doing I/O to the same file, you pretty much had better be aware
> that the whole business is nonstandard and has potential portability
> issues.
>
> That's not to say one shouldn't do it. But just that you aren't going to
> find an answer in the Fortran standard, and whatever answer you might
> find elsewhere might not be portable.
Also recall I told OP I normally did such things w/ OS APIs, typically
memory mapped files. Hardly Standard-conforming.
:)
--
GaryScott
>
>
> An "extension to Standard" means that GNU compiler does not take in
> account this extension ?
>
I'm not aware of any compilers supporting the share= extension other
than IVF/CVF and ancient versions of MS Fortran. I don't believe you
will be able to find a method portable across all compilers, even
those that support bind(c) syntax to interface with windows (since
windows isn't a C-based API anyway (it's essentially pascal-based)
> Cheers,
>
> --
> Fred
Lynn McGuire
> than IVF/CVF and ancient versions of MS Fortran. I don't believe you
> will be able to find a method portable across all compilers, even
> those that support bind(c) syntax to interface with windows (since
> windows isn't a C-based API anyway (it's essentially pascal-based)
Open Watcom F77 ( www.openwatcom.org ) supports SHARE =.
From http://www.openwatcom.org/ftp/manuals/1.5/fuguide.pdf :
The SHARE= specifier can be used to indicate the manner in
which subsequent processes are allowed to access the file
while the file is open. The values allowed for the SHARE=
specifier are the following.
’COMPAT’ no other process may open the file
’DENYRW’ other processes are denied read and write access
’DENYWR’ other process are denied write access (allowed read-only access)
’DENYRD’ other process are denied read access (allowed write-only access)
’DENYNONE’ other processes are allowed read and write access
Lynn
Gordon Sande
> dpb <no...@non.net> a écrit :
>
>> Note to OP, though -- it _is_ an extension to Standard.
> Hmmm, not sure to understand you right.
>
> An "extension to Standard" means that GNU compiler does not take in
> account this extension ?
>
>
> Cheers,
Extension to the standard means that you will have to read the
manual for THAT version of the compiler carefully to see what
it says. It will not apply to any other version of that or any
other compiler, except when you are lucky. Given the topic you
can not expect to be very lucky. I.E. It will not be very
portable. It may work poorly and you can not expect other users
to be very helpful.
Sometimes an extension will apply to several compilers and be
sort-of and on-good-days portable.
If it is standard then you can complain to the compiler vendor
if it does not work. Otherwise you can grumble and hope they
got the manual and the code to match and don't change it too much
in the next version.
Vendors hope you will use their nonstandard convenience extension
so you will not be able to use any other vendors compiler. But a few
have been burned in the past by trying to advance guess the standards
committee so that game is not a strong as in the past. Or maybe the
users are smarter as they also got burned.
JB
>> That's not to say one shouldn't do it. But just that you aren't going to
>> find an answer in the Fortran standard, and whatever answer you might
>> find elsewhere might not be portable.
It should matter, as you in this thread have hopefully learned a
lesson in what happens when you go beyond the standard. :)
> So... what's the answer ? ;-)
The obvious answer would be to do all I/O from the master thread.
If you really need parallel I/O, you should probably look into
something like MPI-I/O (which as the name might imply, means you need
MPI rather than or in addition to OpenMP), and use a parallel file
system like Lustre.
> As it is non standard, I can't open a file by multiple unit _with GNU compiler_, right?
Correct. Nor can you do it with most other compilers, and those that
can probably all do it in mutually incopatible ways.
--
JB
JB
>> memory mapped files. Hardly Standard-conforming.
You write your I/O code in C using things like mmap()
http://www.opengroup.org/onlinepubs/000095399/functions/mmap.html
and use the ISO C BINDING feature to call your I/O code from
Fortran. While you're in the C/POSIX world, you can have multiple file
descriptors per file if you want, for that matter, and you can use
read()/write() on them as well if you dont' want to use mmap().
--
JB
Richard Maine
> Extension to the standard means that you will have to read the
> manual for THAT version of the compiler carefully to see what
> it says. It will not apply to any other version of that or any
> other compiler, except when you are lucky. Given the topic you
> can not expect to be very lucky. I.E. It will not be very
> portable. It may work poorly and you can not expect other users
> to be very helpful.
And further, also given the topic, even if a vendor supports the
particular SHARE feature mentioned, that doesn't mean it will work with
threads, or that much of anything will work with threads. Making things
work properly with threads is many times more complicated than the
particular question of opening the same file multiple times. Given the
level of the questions being asked, I seriously doubt the viability of
pursuing that direction.
I also wouldn't count on the code running correctly under ifort, as the
OP seems to think it does. It is actually far from trivial to even
verify that a multi-threaded code is correct and will work consistently.
Seeing that a handful of test cases appear to work is not at all
adequate to demonstrate that all is correct. It requires a fair amount
if sophistication in understanding the issues relating to
multithreading. I'm not seeing that level of sophistication here. Maybe
it does all happen to work, but showing that convincingly is nontivial
for a multi-threaded program. They have a tendency to pass test cases
and then fail miserably in actual application.
(And yes, I'm reasonably sure that you are aware of all this, Gordon; it
isn't directed at you.)
To forestall the half-expected question, no I don't have any concrete
suggestions. Sort of my whole point is that the problem isn't ameanable
to solution by taking a few quick suggestions from a newsgroup. One
needs to spend a fair amount of time learning about the issues involved
in multi-threading. Makes me wonder whether multi-threading is really
the best match for the OP's needs.
dpb
>> memory mapped files. Hardly Standard-conforming.
Well, somewhere else I saw you were/are on a Linux variant and we moved
everything over to the MS world after OS/2 so I don't have much of
anything very pertinent to your platform, sorry.
I'll note it also involved for multiple threads (in our case actually
independent processes, but the synchronization issues are the same)
critical sections, semaphores, etc., etc., ... All that was wrapped up
in a set of OS-specific routines called from the Fortran application(s)
w/ a fairly simple user API. It took a fair amount of effort to develop
originally; however, it was pretty quick to make the modifications of
the OS API calls from OS/2 to NT (but one would expect that given their
somewhat related history at that juncture). To take the same and try to
port to Linux-like OS, I've no idea--never looked in detail. I now the
same kinds of features are available but how they work comparatively, no
clue.
As (I think Richard?) noted, I'd not lay odds that even if the IVF SHARE
keyword were implemented by your compiler that it would work for
multi-thread application (correctly, anyway)...
--
GaryScott
I use the technique (SHARE=) in both a multiprocessing and
multithreaded application. I do however, use synchronization
techniques, including various data integrity checks. One of those
includes a data record head and tail that must match per the algorithm
in use. Others include the id of the owning/writing thread/process/
user (mainly for short term write-access write operations, as its not
usually a problem with read-only records).
e p chandler
>
>
> An "extension to Standard" means that GNU compiler does not take in
> account this extension ?
1, No. Here is a counter example. gfortran allows real variables as
the index variable in a DO loop unless you use either -std=f95 or -
std=f2003. This is no longer standard.
2. Saying "This code is standard because compiler X compiles it." will
get you flamed in this newsgroup :-).
3. AFAIK there is NO reference compiler for Fortran.
- e
Dave Allured
>
> Dave Allured wrote:
> > fred wrote:
> >> Hi,
> >>
> >> I googled on this issue but I did not find anything relevant.
> >>
> >> Say I have to open a file twice (or more):
> >>
> >> open(unit=20, file='a.dat', form='unformatted', access='stream')
> >> open(unit=21, file='a.dat', form='unformatted', access='stream')
> >>
> >> Running my program, I get:
> >>
> >> Fortran runtime error: File already opened in another unit
> >>
> >> How could I fix it ?
> >>
> >> Option to gfortan or special flag to open() ?
> >>
> >> TIA.
> >
> > It seems to me that this should be legal. My training was that
> > Unix-like file systems allow multiple read access on the same file. I
> > think this is a gfortran issue.
>
> No, it is specifically forbidden by the standard. Mostly, because it's
> impossible to specify what it could mean on a wide variety of systems.
> What should happen if there are non-advancing reads on the two
> "different" units? A rewind on one of them? Or a write or a close?
> If 2 is OK, why not a hundred? It's a can of worms that can't
> work portably.
Consider this definition for read access only, and for conventional
files on disk rather than more exotic devices. Each instance of an open
statement with a different unit number on the same file name defines a
unique and separate connection to the file. All connection information
such as access mode parameters, file position, and error status is
maintained separately for each connection. From the perspective of the
calling program, it as if there are multiple files that happen to
contain identical information. Any number of connections to the same
file are allowed.
This should easily handle all cases that you mentioned except for the
write. I think this is exactly how multiple connections were
traditionally handled by compilers such as XLF 8.1 and the SunWS
compiler. I am not an expert and can't immediately back this up with
references.
I see no significant problem for standardizing this behavior for read
access. If you still see a contradiction then please describe it.
Remaining problems are the inquire command by file name, and write
access. I don't see either of these as difficult for ordinary files on
disk. Fodder for other conversations if anyone is interested. I expect
a standards group kicked all of this around a few years ago; I just
happen to not like the current result.
--Dave
Dick Hendrickson
There probably are no contradictions or unavoidable problems. The
file and access needs to be pretty severely restricted. It's not clear
what should happen when there is a close statement. Many (most? all?)
current systems treat units 5 and 6 as synonyms for the * units. But,
there is no common agreement about what should happen to * when 5 or 6
is closed. It's worse when 5 or 6 are subsequently connected to a
different file. It's not that there can't be a definition, it's that
there are many good ones in use. That's mostly why Fortran invented the
INPUT_UNIT, etc., file names. It allows a programmer to avoid conflicts.
>
> Remaining problems are the inquire command by file name, and write
> access. I don't see either of these as difficult for ordinary files on
> disk. Fodder for other conversations if anyone is interested. I expect
> a standards group kicked all of this around a few years ago; I just
> happen to not like the current result.
>
Almost all of the Fortran access methods contain a phrase like "whether
or not you can use this feature is system dependent." Personally, I
think a restriction to "read-only from disk files" is too restrictive to
be useful. I can also see your point of view.
Dick Hendrickson
> --Dave
Richard Maine
> Consider this definition for read access only, and for conventional
> files on disk rather than more exotic devices....
Realize that when Fortran I/O was first defined, "conventional" I/O was
for tapes, where multiple independent access is pretty clearly not
reasonably viable.
> I expect
> a standards group kicked all of this around a few years ago; I just
> happen to not like the current result.
My guess is more that it was left unchanged from the previous
definition, rather than that it was very actively debated. I don't
particularly recall any debates on the subject when I was on the
committee, though there might have been some from before then (I joined
in about 1991), or for that matter, there might have been some that were
quickly enough disposed of that I don't recall them.
The odds of the standard talking specifically about disk files seem...
low. It doesn't get to that level of device-specific specification for
anything else. I can imagine a note explaining that disk files were a
motivation or a likely place for some particular feature to apply, but I
can't imagine the standard actually specifying that something applies to
disk files (it could take quite a bit of work to precisely define
exactly what such a thing would mean from a standard perspective; surely
one would not want to say hardware-related things that would rule out
ramdisks, SSDs, etc.)
What I could more imagine would be an optional specification about how
such a thing would work on the files for which it was allowed, leaving
it system dependent to specify what files it was allowed on.
But that turns out not to be incredibly different from the current
situation. Optional stuff is often like that. Currently, you have no
guarantee that multiple opens of the same file will work. It might
happen to work as an extension on some machines/compilers. If this was
added as an optional property that some files could have, then you would
still have no guarantee that any particular machine/compiler would give
that property to any particular file. Some might; some might not. I have
trouble distinguishing these in practice.
I suppose the biggest difference might just be in the implied
encouragement of vendors to support such a thing as opposed to the
implied discouragement in the current standard. But I don't see a
concrete difference in the actual requirements. On the other hand, I do
admit that such implications have been enough to justify other additions
to the standard.
This does assume it would be an optional property, but I think that is
pretty much a given. Pretty much *EVERYTHING* in I/O is technically
optional. Fortunately, vendors don't tend to take the obstructionist
positions that a lawyer could probably manage to defend. (Well, I've
seen it done to try to defend what I regard as non-conforming
implementations of complex kinds, but not so much in I/O). It would be
really out of place to have everything else about I/O be optional, but
that one property not be; seems unlikely.
Perhaps it could be done as a recommendation rather than a requirement.
That might have a better chance of flying, I'd think. It wouldn't get
your guarantees, but it might improve the odds.
Glen Herrmannsfeldt
(snip on opening the same file on multiple units at the same time)
> Consider this definition for read access only, and for conventional
> files on disk rather than more exotic devices. Each instance of an open
> statement with a different unit number on the same file name defines a
> unique and separate connection to the file. All connection information
> such as access mode parameters, file position, and error status is
> maintained separately for each connection. From the perspective of the
> calling program, it as if there are multiple files that happen to
> contain identical information. Any number of connections to the same
> file are allowed.
That means it is up to the OS what happens, which is the way
it should be. It would seem reasonable for an OS to allow it, but
there is no requirement for it. Many older single task OS
may not have the ability to keep separate file pointers for the
same file. I don't know about files, but I do remember that
in MS-DOS the current directory is global. It would seem also
possible that file positions could also be global.
-- glen
Thomas Koenig
> It would be
> really out of place to have everything else about I/O be optional, but
> that one property not be; seems unlikely.
I can well imagine an embedded system which doesn't even have files.
I would imagine that C would be the usual language for these, but
in a non-hosted environment (which means that I/O doesn't have to
be available), but there's no reason that Fortran could not be used
instead.
Gary Scott
Fortran was used in some embedded environments. It is still listed as a
component of some embedded development environments but few of those are
actively used or marketed (or updated).
--
Gary Scott
mailto:garylscott@sbcglobal dot net
Fortran Library: http://www.fortranlib.com
Support the Original G95 Project: http://www.g95.org
-OR-
Support the GNU GFortran Project: http://gcc.gnu.org/fortran/index.html
If you want to do the impossible, don't hire an expert because he knows
it can't be done.
-- Henry Ford
George
> Thomas Koenig wrote:
>
>> On 2008-10-31, Richard Maine <nos...@see.signature> wrote:
>>
>>>It would be
>>>really out of place to have everything else about I/O be optional, but
>>>that one property not be; seems unlikely.
>>
>>
>> I can well imagine an embedded system which doesn't even have files.
>> I would imagine that C would be the usual language for these, but
>> in a non-hosted environment (which means that I/O doesn't have to
>> be available), but there's no reason that Fortran could not be used
>> instead.
> Fortran was used in some embedded environments. It is still listed as a
> component of some embedded development environments but few of those are
> actively used or marketed (or updated).
But it takes more than a syntax and a bunch of chips to make an integrated
circuit. It takes less effort and money to do this using windows and C
than one might believe, with development environments like Win AVR.
comp.arch.embedded has lively discussions of relevant issues. I'm wiring
the fans in my hallway so that a microchip could run them, which is to say,
I'm finishing the hallway so that wires can be fished through without
grief.
--
George
If you're sick and tired of the politics of cynicism and polls and
principles, come and join this campaign.
George W. Bush
Picture of the Day http://apod.nasa.gov/apod/
Gary Scott
> On Sat, 01 Nov 2008 10:46:41 -0500, Gary Scott wrote:
>
>
>>Thomas Koenig wrote:
>>
>>
>>>On 2008-10-31, Richard Maine <nos...@see.signature> wrote:
>>>
>>>
>>>>It would be
>>>>really out of place to have everything else about I/O be optional, but
>>>>that one property not be; seems unlikely.
>>>
>>>
>>>I can well imagine an embedded system which doesn't even have files.
>>>I would imagine that C would be the usual language for these, but
>>>in a non-hosted environment (which means that I/O doesn't have to
>>>be available), but there's no reason that Fortran could not be used
>>>instead.
>>
>>Fortran was used in some embedded environments. It is still listed as a
>>component of some embedded development environments but few of those are
>>actively used or marketed (or updated).
>
>
> But it takes more than a syntax and a bunch of chips to make an integrated
> circuit. It takes less effort and money to do this using windows and C
> than one might believe, with development environments like Win AVR.
those embedded-suitable fortran compilers had all the necessary
extensions such that it wasn't any easier necessarily. More important
is what the OS API is written in.
>
> comp.arch.embedded has lively discussions of relevant issues. I'm wiring
> the fans in my hallway so that a microchip could run them, which is to say,
> I'm finishing the hallway so that wires can be fished through without
> grief.
i've done a lot of pulling wires of all sorts through inner and outer
walls (i hate outer walls, at least in some roof designs)
Gary Scott
ooops, one of us has a time zone problem
George
> Gary Scott wrote:
>
>> George wrote:
>>
>>> On Sat, 01 Nov 2008 10:46:41 -0500, Gary Scott wrote:
>>>
>>>
>>>> Thomas Koenig wrote:
>>>>
>>>>
>>>>> On 2008-10-31, Richard Maine <nos...@see.signature> wrote:
>>>>>
>>>>>
>>>>>> It would be
>>>>>> really out of place to have everything else about I/O be optional, but
>>>>>> that one property not be; seems unlikely.
>>>>>
>>>>>
>>>>>
>>>>> I can well imagine an embedded system which doesn't even have files.
>>>>> I would imagine that C would be the usual language for these, but
>>>>> in a non-hosted environment (which means that I/O doesn't have to
>>>>> be available), but there's no reason that Fortran could not be used
>>>>> instead.
>>>>
>>>>
>>>> Fortran was used in some embedded environments. It is still listed
>>>> as a component of some embedded development environments but few of
>>>> those are actively used or marketed (or updated).
>>>
>>>
>>>
>>> But it takes more than a syntax and a bunch of chips to make an
>>> integrated
>>> circuit. It takes less effort and money to do this using windows and C
>>> than one might believe, with development environments like Win AVR.
>>
>>
>> those embedded-suitable fortran compilers had all the necessary
>> extensions such that it wasn't any easier necessarily. More important
>> is what the OS API is written in.
I was unaware that such tools existed for fortran. The advantage to WinAVR
is the price: free, and having a group of people out there that you can
talk to about it.
>>
>>>
>>> comp.arch.embedded has lively discussions of relevant issues. I'm wiring
>>> the fans in my hallway so that a microchip could run them, which is to
>>> say,
>>> I'm finishing the hallway so that wires can be fished through without
>>> grief.
>>
>>
>> i've done a lot of pulling wires of all sorts through inner and outer
>> walls (i hate outer walls, at least in some roof designs)
That's why they have hammer drills and I have hearing loss.
>>
> ooops, one of us has a time zone problem
It's seven bells on my machine, which agrees when I google for time.
Maybe you're getting nervous for your thesis defense and best distract
yourself. (I'm hyper-aware of time when I'm nervous about having to do
something.)
Might be a good night to go see "W."
--
George
When I take action, I'm not going to fire a $2 million missile at a $10
empty tent and hit a camel in the butt. It's going to be decisive.
Gary Scott
Was referring to the out of order sorting of these threads in google groups.
George
> Was referring to the out of order sorting of these threads in google groups.
One of us could be in the wrong time-place:
Path: uni-berlin.de!individual.net!not-for-mail
From: George <geo...@example.invalid>
Newsgroups: comp.lang.fortran
Subject: Re: [gfortran43] opening file twice...
Date: Sat, 1 Nov 2008 19:06:57 -0700
But I really do believe that I'm seven hours off greenwich time and that
it's nineteen hundred local.
Path:
uni-berlin.de!fu-berlin.de!nlpi057.nbdc.sbc.com!prodigy.net!flpi088.ffdc.sbc.com!prodigy.com!flpi107.ffdc.sbc.com!flpi144.ffdc.sbc.com.POSTED!cfe18fef!not-for-mail
From: Gary Scott <garyl...@sbcglobal.net>
X-UserInfo1:
FKPGW]CEGJPKR_DYABNLNFXBWR\HPCTL@XT^OBPLAH[\RWICYFWUQBKZQLYJX\_ITFD_KFVLUN[DOM_A_NSYNWPFWNS[XV\I]PZ@BQ[@CDQDPCL^FKCBIPC@KLGEZEFNMDYMKHRL_YYYGDSSODXYN@[\BK[LVTWI@AXGQCOA_SAH@TPD^\AL\RLGRFWEARBM
Date: Sat, 01 Nov 2008 20:31:01 -0500
Xref: uni-berlin.de comp.lang.fortran:205928
On my news client, this is reported as 6:31 pm, which prima facie is a half
hour earlier. I guess I don't get the time stamp at all, because I read
that as ten o'clock eastern time.
--
George
Every nation in every region now has a decision to make. Either you are
with us, or you are with the terrorists.
George
> Was referring to the out of order sorting of these threads in google groups.
It was daylight savings. I got to enjoy an extra hour of a hangover today.
--
George
People make suggestions on what to say all the time. I'll give you an
example; I don't read what's handed to me. People say, 'Here, here's your
speech, or here's an idea for a speech.' They're changed. Trust me.
Gary Scott
> On Sat, 01 Nov 2008 20:31:01 -0500, Gary Scott wrote:
>
>
>>Was referring to the out of order sorting of these threads in google groups.
>
>
> It was daylight savings. I got to enjoy an extra hour of a hangover today.
same here, well, not really, i only had 4 beers but I don't drink much
so the impact is a little bigger
Dave Allured
It seems fairly obvious that in what I described, a close statement
would simply close the logical connection between the file and the
specified unit number. What happens to files on other unit numbers is
irrelevant.
> Many (most? all?)
> current systems treat units 5 and 6 as synonyms for the * units. But,
> there is no common agreement about what should happen to * when 5 or 6
> is closed. It's worse when 5 or 6 are subsequently connected to a
> different file.
Special unit number assignments are outside the scope of my proposal.
> It's not that there can't be a definition, it's that
> there are many good ones in use. That's mostly why Fortran invented the
> INPUT_UNIT, etc., file names. It allows a programmer to avoid conflicts.
>
> >
> > Remaining problems are the inquire command by file name, and write
> > access. I don't see either of these as difficult for ordinary files on
> > disk. Fodder for other conversations if anyone is interested. I expect
> > a standards group kicked all of this around a few years ago; I just
> > happen to not like the current result.
> >
> Almost all of the Fortran access methods contain a phrase like "whether
> or not you can use this feature is system dependent." Personally, I
> think a restriction to "read-only from disk files" is too restrictive to
> be useful. I can also see your point of view.
Yes, after more contemplation I agree with that. With appropriate care
in the application, both read and write access through multiple unit
numbers can be reasonable and useful. I would change the device
constraint to include your "system dependent" wording and to also say
"the availability of multiple file connections is system dependent by
type of device".
--Dave
Dave Allured
>
> Dave Allured <nos...@nospam.com> wrote:
>
> > Consider this definition for read access only, and for conventional
> > files on disk rather than more exotic devices....
>
> Realize that when Fortran I/O was first defined, "conventional" I/O was
> for tapes, where multiple independent access is pretty clearly not
> reasonably viable.
Agreed.
> > I expect
> > a standards group kicked all of this around a few years ago; I just
> > happen to not like the current result.
>
> My guess is more that it was left unchanged from the previous
> definition, rather than that it was very actively debated. I don't
> particularly recall any debates on the subject when I was on the
> committee, though there might have been some from before then (I joined
> in about 1991), or for that matter, there might have been some that were
> quickly enough disposed of that I don't recall them.
>
> The odds of the standard talking specifically about disk files seem...
> low. It doesn't get to that level of device-specific specification for
> anything else. I can imagine a note explaining that disk files were a
> motivation or a likely place for some particular feature to apply, but I
> can't imagine the standard actually specifying that something applies to
> disk files (it could take quite a bit of work to precisely define
> exactly what such a thing would mean from a standard perspective; surely
> one would not want to say hardware-related things that would rule out
> ramdisks, SSDs, etc.)
Good point.
> What I could more imagine would be an optional specification about how
> such a thing would work on the files for which it was allowed, leaving
> it system dependent to specify what files it was allowed on.
I think that generic wording about system dependency would be sufficient
for the standard, with no further elaboration. Something like
"availability of multiple file connections is system dependent by type
of device". I might throw in ordinary random access hard disk files as
an illustrative example, if that did not confilct with the writing style
of the standard.
> But that turns out not to be incredibly different from the current
> situation. Optional stuff is often like that. Currently, you have no
> guarantee that multiple opens of the same file will work. It might
> happen to work as an extension on some machines/compilers. If this was
> added as an optional property that some files could have, then you would
> still have no guarantee that any particular machine/compiler would give
> that property to any particular file. Some might; some might not. I have
> trouble distinguishing these in practice.
>
> I suppose the biggest difference might just be in the implied
> encouragement of vendors to support such a thing as opposed to the
> implied discouragement in the current standard. But I don't see a
> concrete difference in the actual requirements. On the other hand, I do
> admit that such implications have been enough to justify other additions
> to the standard.
So here we come to the exact problem with an explicit prohibition on
multiple connections in the language standard. Apparently several
vendors decided that it was reasonable to allow multiple connections as
an extension, presumably to give Fortran programmers direct access to a
useful and commonly available access method. However, gfortran authors
interpreted the statement strictly and without an exception mechanism
simply because the standard said so, therefore denying what some of us
think is reasonable usage.
> This does assume it would be an optional property, but I think that is
> pretty much a given. Pretty much *EVERYTHING* in I/O is technically
> optional. Fortunately, vendors don't tend to take the obstructionist
> positions that a lawyer could probably manage to defend.
Gfortran did on this one. However I expect it was simply due to an
intent to be true to the standard, rather than to deliberately obstruct
useful applications.
> (Well, I've
> seen it done to try to defend what I regard as non-conforming
> implementations of complex kinds, but not so much in I/O). It would be
> really out of place to have everything else about I/O be optional, but
> that one property not be; seems unlikely.
>
> Perhaps it could be done as a recommendation rather than a requirement.
> That might have a better chance of flying, I'd think. It wouldn't get
> your guarantees, but it might improve the odds.
Because of this thread and also a live application of mine, I have given
this a lot of thought. My conclusions are (1) I should file a feature
request to gfortran to allow multiple file connections, probably by
permission from a command line option; and (2) multiple connections as a
system and device dependent feature should be added to the Fortran
standard, without an incursion into device descriptions.
Thanks to everyone for your stimulating participation on this topic. I
have been educated. ;-)
--Dave
Dave Allured
I expect that virtually all OS's, old and new, support random access for
disk files, regardless of whether they also maintain file pointers for
certain kinds of OS supported sequential access. Therefore it should be
straightforward for Fortran run time systems to maintain a separate file
pointer, and other appropriate state info, for each instance of an open
Fortran file connection. I suspect that most modern Fortran run times
internally perform the translation from file pointer to random file
address, without depending on an OS file pointer.
Perhaps someone with real knowledge of a Fortran I/O handler could
comment.
--Dave
Dick Hendrickson
Sometime after F2008 is more or less finalized, there will be an open
call for proposed feature and enhancements for F2013 (14? 15? ...).
This would be a reasonable request. I'll try to remember to do
something about it, but if you can, you should also send something in.
A brief description of why you need it, as well as a high level
description, pretty much like your last sentence.
Dick Hendrickson
Glen Herrmannsfeldt
(snip)
> So here we come to the exact problem with an explicit prohibition on
> multiple connections in the language standard. Apparently several
> vendors decided that it was reasonable to allow multiple connections as
> an extension, presumably to give Fortran programmers direct access to a
> useful and commonly available access method. However, gfortran authors
> interpreted the statement strictly and without an exception mechanism
> simply because the standard said so, therefore denying what some of us
> think is reasonable usage.
Did you put ACTION='READ' on the OPEN statement?
I don't know if it makes a difference for gfortran, but note that
the default is 'READWRITE' and that opening the same file for output
on two different units can cause problems on many systems.
It seems reasonable to reject OPEN for the same file on
two different units without ACTION='READ' on both. (Because of
buffering, once one has changed it is hard to say what you would
get from a READ on the other.)
It might not be gfortran that is refusing the open, but the OS file
system may refuse an exclusive lock more than once. (It should also
fail if two different programs try to open the file for writing.)
If opened with ACTION='READ' it should open for shared access.
Windows file locking seems to be the default. Unix and unix-like
systems mostly leave it up to the user to avoid file access problems.
-- glen
Glen Herrmannsfeldt
(snip)
> Yes, after more contemplation I agree with that. With appropriate care
> in the application, both read and write access through multiple unit
> numbers can be reasonable and useful. I would change the device
> constraint to include your "system dependent" wording and to also say
> "the availability of multiple file connections is system dependent by
> type of device".
Because of buffering, in both the Fortran library and system (multiple
systems for network drives) multiple access (same or different program)
is very system dependent. Use of the FLUSH statement would help, but
note even that has the 'processor dependent' clause. For network
(remote) file systems (NFS, SMB, Appleshare) it becomes much more
difficult to keep data synchronized, even with only one writing.
As noted in another post, you need ACTION='READ' for read only
access, the default being 'READWRITE'.
-- glen
GaryScott
> Dave Allured wrote:
>
> (snip)
>
> > Yes, after more contemplation I agree with that. With appropriate care
> > in the application, both read and write access through multiple unit
> > numbers can be reasonable and useful. I would change the device
> > constraint to include your "system dependent" wording and to also say
> > "the availability of multiple file connections is system dependent by
> > type of device".
>
> Because of buffering, in both the Fortran library and system (multiple
> systems for network drives) multiple access (same or different program)
> is very system dependent. Use of the FLUSH statement would help, but
> note even that has the 'processor dependent' clause. For network
> (remote) file systems (NFS, SMB, Appleshare) it becomes much more
> difficult to keep data synchronized, even with only one writing.
>
it can be a bit messy. I do it by writing explicit "whose in write
control" record content. If the id isn't your own, you assume read
access. If data integrity checks fail, there is a self-healing
process to restore correct synchronization. In reality, all of the
applications have write access, they just don't exercise it unless
they successfully subscribe to the record to be written. So far,
there have been no data integrity failures in ten years of constant
use.
Glen Herrmannsfeldt
(snip)
> I don't find it particularly difficult to synchronize NFS files, but
> it can be a bit messy. I do it by writing explicit "whose in write
> control" record content. If the id isn't your own, you assume read
> access. If data integrity checks fail, there is a self-healing
> process to restore correct synchronization. In reality, all of the
> applications have write access, they just don't exercise it unless
> they successfully subscribe to the record to be written. So far,
> there have been no data integrity failures in ten years of constant
> use.
It might be that within one implementation (one vendor) it
can work well. If you read comp.protocols.nfs (not a busy
newsgroup) you find complaints about problems working between
different implementations, such as between Solaris and Linux.
How often do you have to do the self-healing process?
Also, have you tried it long distance through the public internet?
Not having looked recently, I don't remember if it guarantees that
changes are not reordered. (Such as due to network packet ordering.)
For NFS/TCP that should not be as much of a problem is NFS/UDP.
-- glen
GaryScott
> GaryScott wrote:
>
> (snip)
>
> > I don't find it particularly difficult to synchronize NFS files, but
> > it can be a bit messy. I do it by writing explicit "whose in write
> > control" record content. If the id isn't your own, you assume read
> > access. If data integrity checks fail, there is a self-healing
> > process to restore correct synchronization. In reality, all of the
> > applications have write access, they just don't exercise it unless
> > they successfully subscribe to the record to be written. So far,
> > there have been no data integrity failures in ten years of constant
> > use.
>
> It might be that within one implementation (one vendor) it
> can work well. If you read comp.protocols.nfs (not a busy
> newsgroup) you find complaints about problems working between
> different implementations, such as between Solaris and Linux.
>
> How often do you have to do the self-healing process?
Only if one of the applications abnormally terminates during a record
write process and leaves a record half written. It's extremely rare.
The writing application reads the written record again under some
circumstances to ensure it successfully wrote the record and retries
at least one time (only under circumstances where network bandwidth
consumption would not be excessive). The record structure is fixed
length (direct access), but there is an encoding scheme to pack
variable length records into the fixed length records. So
"blobs" (like images) are not automatically retried to reduce
bandwidth requirements. They can be manually refreshed by the user
however. It is fairly robust with one exception...searching requires
a local download for larger database sizes, else the search process
(directly from an NFS file) is to sluggish.
<snip>
> -- glen
JB
> I expect that virtually all OS's, old and new, support random access for
> disk files, regardless of whether they also maintain file pointers for
> certain kinds of OS supported sequential access.
No. At least in unix, and IIRC in windows too, the canonical system
interfaces are some variation of read(), write(), seek(), tell() etc.,
where for seekable files (as opposed to streams, terminals etc.) the
reads or writes begin at a position that the OS kernel maintains for
each file descriptor (and of course, read/write/seek modify the
position). Note that opening a new file descriptor on the same file
gives an independent file position (at least in unix).
Modern unixes typically also support pread/pwrite interfaces as well,
where the functions have an extra position argument, and hence these
functions ignore the kernel-maintained file position. These interfaces
were added in order to support multithreaded programs reading/writing
from multiple locations without having to put an exclusive lock on any
code that seeks and reads/writes, but they are quite recent and old
systems don't support them.
> Therefore it should be
> straightforward for Fortran run time systems to maintain a separate file
> pointer, and other appropriate state info, for each instance of an open
> Fortran file connection.
Yes, that would be no problem, if this maps to native OS semantics. If
the OS doesn't support this, multiplexing multiple Fortran units on
top of a single OS file descriptor sounds like a major pain, and would
likely suffer poor performance as well especially if the OS also
doesn't support the above mentioned pread/pwrite style interfaces.
> I suspect that most modern Fortran run times
> internally perform the translation from file pointer to random file
> address, without depending on an OS file pointer.
I find this hard to believe. However, depending how often the file
position is needed, the runtime library might choose to keep track of
the file position internally, in order to avoid the syscall overhead
of asking the OS for the position.
--
JB
Dave Allured
>
> On 2008-11-03, Dave Allured <nos...@nospam.com> wrote:
> > I expect that virtually all OS's, old and new, support random access for
> > disk files, regardless of whether they also maintain file pointers for
> > certain kinds of OS supported sequential access.
>
> No. At least in unix, and IIRC in windows too, the canonical system
> interfaces are some variation of read(), write(), seek(), tell() etc.,
> where for seekable files (as opposed to streams, terminals etc.) the
> reads or writes begin at a position that the OS kernel maintains for
> each file descriptor (and of course, read/write/seek modify the
> position). Note that opening a new file descriptor on the same file
> gives an independent file position (at least in unix).
>
> Modern unixes typically also support pread/pwrite interfaces as well,
> where the functions have an extra position argument, and hence these
> functions ignore the kernel-maintained file position. These interfaces
> were added in order to support multithreaded programs reading/writing
> from multiple locations without having to put an exclusive lock on any
> code that seeks and reads/writes, but they are quite recent and old
> systems don't support them.
"... old systems don't support them." This is the part I find odd,
since I have dealt with random access to disk files in Fortran as far
back as the late 60's. I need to go read up on the Unix API before I
demonstrate any further ignorance.
> > Therefore it should be
> > straightforward for Fortran run time systems to maintain a separate file
> > pointer, and other appropriate state info, for each instance of an open
> > Fortran file connection.
>
> Yes, that would be no problem, if this maps to native OS semantics. If
> the OS doesn't support this, multiplexing multiple Fortran units on
> top of a single OS file descriptor sounds like a major pain, and would
> likely suffer poor performance as well especially if the OS also
> doesn't support the above mentioned pread/pwrite style interfaces.
So, um, let's forget about multiplexing. Support multiple file
connections through a Unix facility that was made to do exactly that,
via mapping separate Fortran unit numbers to separate OS file
descriptors. Originally it was the Unix file access model that I
implied Fortran should emulate.
> > I suspect that most modern Fortran run times
> > internally perform the translation from file pointer to random file
> > address, without depending on an OS file pointer.
>
> I find this hard to believe. However, depending how often the file
> position is needed, the runtime library might choose to keep track of
> the file position internally, in order to avoid the syscall overhead
> of asking the OS for the position.
Yes I got ahead of myself on that one. I should go read some gfortran
run time source or something. Thanks for your perspective.
--Dave
JB
> JB wrote:
>>
>> On 2008-11-03, Dave Allured <nos...@nospam.com> wrote:
>> > I expect that virtually all OS's, old and new, support random access for
>> > disk files, regardless of whether they also maintain file pointers for
>> > certain kinds of OS supported sequential access.
>>
>> No. At least in unix, and IIRC in windows too, the canonical system
>> interfaces are some variation of read(), write(), seek(), tell() etc.,
>> where for seekable files (as opposed to streams, terminals etc.) the
>> reads or writes begin at a position that the OS kernel maintains for
>> each file descriptor (and of course, read/write/seek modify the
>> position). Note that opening a new file descriptor on the same file
>> gives an independent file position (at least in unix).
>>
>> Modern unixes typically also support pread/pwrite interfaces as well,
>> where the functions have an extra position argument, and hence these
>> functions ignore the kernel-maintained file position. These interfaces
>> were added in order to support multithreaded programs reading/writing
>> from multiple locations without having to put an exclusive lock on any
>> code that seeks and reads/writes, but they are quite recent and old
>> systems don't support them.
>
> "... old systems don't support them." This is the part I find odd,
> since I have dealt with random access to disk files in Fortran as far
> back as the late 60's.
The canonical read/write/seek/tell/etc. interface is perfectly capable
of supporting random access. In fact, that's the entire point of
seek() and tell(). E.g. the pwrite(Write some data starting at
position 123) is roughly equivalent to the follow pseudo-C code:
int fd = ... // The file descriptor, roughly equivalent to
// the Fortran unit number.
current_position = tell(fd); // where am I?
seek(fd, 123);
write(fd, some data);
seek(fd, current_position); // seek back to the original position
So pread/pwrite don't add any fundamental new capability to the
system. In fact, they are more limited than the traditional read/write
since they don't support non-seekable files. As I mentioned above,
their usefulness lies in being a performance optimization for certain
file-access patterns in multithreaded programs.
>> Yes, that would be no problem, if this maps to native OS semantics. If
>> the OS doesn't support this, multiplexing multiple Fortran units on
>> top of a single OS file descriptor sounds like a major pain, and would
>> likely suffer poor performance as well especially if the OS also
>> doesn't support the above mentioned pread/pwrite style interfaces.
>
> So, um, let's forget about multiplexing. Support multiple file
> connections through a Unix facility that was made to do exactly that,
> via mapping separate Fortran unit numbers to separate OS file
> descriptors. Originally it was the Unix file access model that I
> implied Fortran should emulate.
I see. But the point still stands; if the OS (non-unix in this case)
doesn't support such semantics, mandating that Fortran supports it
places a burden on the runtime library implementer (for a rather small
benefit, IMHO). Now, I'm not familiar enough with windows or the
plethora of weird and wonderful OS's where someone might want to run
Fortran in order to tell whether this kind of semantics is universal
outside the Unix world or not. Though I suspect this is the kind of
issue where OS's might very well differ.
--
JB