gfortran stack size warning

[Beliavsky]

The program

program xstack_size
implicit none
integer, parameter :: n = 2*10**8
double precision :: x(n)
call random_number(x)
print*,"kind(x), n, average =",kind(x),n,sum(x)/n
end program xstack_size

compiles and runs with gfortran 10.0.0 20190929 from equation.com on Windows 10, but says, if the -Wall option is used,

Warning: Array 'x' at (1) is larger than limit set by '-fmax-stack-var-size=', moved from stack to static storage. This makes the procedure unsafe when called recursively, or concurrently from multiple threads. Consider using '-frecursive', or increase the '-fmax-stack-var-size=' limit, or change the code

If I follow the advice and compile with -frecursive, an executable is created, but it does not give any output when run. The same thing happens when I compile with -fmax-stack-var-size=1000000000000 . If I reduce that size by a factor of 10, the program compiles and runs, but I still get the compiler warning message.

What options should I use to compile with -Wall and not get warnings but still have the program run properly?

[Steve Lionel]

On 12/29/2019 4:43 PM, Beliavsky wrote:
> If I follow the advice and compile with -frecursive, an executable is created, but it does not give any output when run. The same thing happens when I compile with -fmax-stack-var-size=1000000000000 . If I reduce that size by a factor of 10, the program compiles and runs, but I still get the compiler warning message.
>
> What options should I use to compile with -Wall and not get warnings but still have the program run properly

Don't use options. Instead, make large arrays ALLOCATABLE and allocate
them to the desired size.

--
Steve Lionel
ISO/IEC JTC1/SC22/WG5 (Fortran) Convenor
Retired Intel Fortran developer/support
Email: firstname at firstnamelastname dot com
Twitter: @DoctorFortran
LinkedIn: https://www.linkedin.com/in/stevelionel
Blog: http://intel.com/software/DrFortran
WG5: https://wg5-fortran.org

[robin....@gmail.com]

On Monday, December 30, 2019 at 11:46:38 AM UTC+11, Steve Lionel wrote:
> On 12/29/2019 4:43 PM, Beliavsky wrote:
> > If I follow the advice and compile with -frecursive, an executable is created, but it does not give any output when run. The same thing happens when I compile with -fmax-stack-var-size=1000000000000 . If I reduce that size by a factor of 10, the program compiles and runs, but I still get the compiler warning message.
> >
> > What options should I use to compile with -Wall and not get warnings but still have the program run properly
>
> Don't use options. Instead, make large arrays ALLOCATABLE and allocate
> them to the desired size.

Sort of departs from the KISS principle.

An ALLOCATE statement needs to have STAT option or an ERRMSG
option when the array is large.

Automatic arrays should work for all sizes that the compiler can handle.

[Ron Shepard]

On 12/31/19 3:58 AM, robin....@gmail.com wrote:
> On Monday, December 30, 2019 at 11:46:38 AM UTC+11, Steve Lionel wrote:
>> On 12/29/2019 4:43 PM, Beliavsky wrote:
>>> If I follow the advice and compile with -frecursive, an executable is created, but it does not give any output when run. The same thing happens when I compile with -fmax-stack-var-size=1000000000000 . If I reduce that size by a factor of 10, the program compiles and runs, but I still get the compiler warning message.
>>>
>>> What options should I use to compile with -Wall and not get warnings but still have the program run properly
>>
>> Don't use options. Instead, make large arrays ALLOCATABLE and allocate
>> them to the desired size.
>
> Sort of departs from the KISS principle.

Yet there are many practical differences to using allocatable rather
than static or automatic arrays. If things are simplified too much, the
programmer has no control left over those differences.

> An ALLOCATE statement needs to have STAT option or an ERRMSG
> option when the array is large.

Perhaps so, but many times the programmer just wants the program to fail
and stop, and that is what typically happens by default when those
options are not specified.

I have debugged programs in the past where the programmer had used the
status options on i/o statements or with open and close statements, but
did not follow through well enough afterwards to track down and print
the appropriate error message. Simply removing those ISTAT= arguments
and letting the compiler and operating system do its thing was sometimes
the quickest way to understand what was the problem.

There is also the complication involving lazy allocation, where the
allocate() statement always succeeds, and it is only later when too much
of the array is actually referenced when the error occurs.

> Automatic arrays should work for all sizes that the compiler can handle.

It depends on how much control the programmer wants over the process.
There are practical differences between static, heap, and stack
allocation, regarding both size limits and efficiency, and the
programmer might not want to give up too much control over that process
to the compiler. This is not something that is specified by the language
standard (maybe it should be?), so it must be accomplished outside the
standard in possibly nonportable ways, such as with compiler options or
with operating system settings.

In addition to automatic, static, and allocatable arrays that are
declared explicitly by the programmer in some way, there is also the
issue of compiler-generated intermediates that are allocated and
deallocated on the fly. It would sometimes be useful to give the
programmer some control over how that is done (e.g. stack or heap
allocation, specifying limits, etc.).

One of the things I have advocated in the past is for the ability to
query the system to see what is the current memory allocation statistics
and what are the current limits. That would allow the programmer to make
appropriate choices of block sizes, array allocations, and algorithm
choices in order for the program to fit into whatever constraints are
applicable at that moment. When the compiler does too much, and doesn't
reveal what it has done, the programmer loses that control. Most
supercomputers today work with RAM memory only, with no swap space, so
this is just as important today as it was 40 or 50 years ago in the
early days of time-sharing.

Other languages have taken the KISS approach to extremes and sacrificed
programmer control and efficiency for simplicity, and they came and went
like a leisure suit fad. At least fortran is still around and thriving.

$.02 -Ron Shepard

[JCampbell]

On Wednesday, January 1, 2020 at 4:01:25 AM UTC+11, Ron Shepard wrote:
>
> There are practical differences between static, heap, and stack
> allocation, regarding both size limits and efficiency, and the
> programmer might not want to give up too much control over that process
> to the compiler. This is not something that is specified by the language
> standard (maybe it should be?), so it must be accomplished outside the
> standard in possibly nonportable ways, such as with compiler options or
> with operating system settings.
>

Ron,

I agree with your comment, but there is more to this, especially when using OpenMP.

With OpenMP, where there are (lots of) small private arrays, it is best to have these arrays in the local thread stack, rather than shared within a single L1 cache buffer.
For large arrays, cache conflicts are less of a performance issue and so they can either be on the local stack or shared heap.
With gFortran, managing arrays between the stacks and heap is vague, depending on which OS you are using and how the compiler options may or may not be implemented. As an example, try to find how to change the thread stack size in gFortran for your OS implementation.

I do think the original example of a 1.6 Gbyte local array being placed on the stack, rather than as an allocatable array as a poor choice. Making it allocatable is far easier than adjusting the stack size on most compilers and is especially the case with gFortran.

When redefining the stack size, there is no uniformity of parameter between compilers:
Is the value decimal or hex ?
Is the value bytes, kbytes or mbytes ?
Is this for the main stack only or applies to all threads ?

Very rarely is there an explained example.
ALLOCATE looks to be the far simpler option.

If someone would explain these options for gFortran, I would appreciate it, especially:
An example of changing the stack in a gFortran build and with OpenMP
An example of specifying controlling stack vs heap allocation.
Does the compiler recognise the array size limit for stack allocation in -fmax-stack-var-size= ?
In a large build, can some routines use the heap for large arrays, but others use the stack ? ( separate .f90 files )

The answers to these are not easily located in the supplied documentation.

[robin....@gmail.com]

On Wednesday, January 1, 2020 at 4:01:25 AM UTC+11, Ron Shepard wrote:

> On 12/31/19 3:58 AM, r.....@gmail.com wrote:
> > On Monday, December 30, 2019 at 11:46:38 AM UTC+11, Steve Lionel wrote:
> >> On 12/29/2019 4:43 PM, Beliavsky wrote:
> >>> If I follow the advice and compile with -frecursive, an executable is created, but it does not give any output when run. The same thing happens when I compile with -fmax-stack-var-size=1000000000000 . If I reduce that size by a factor of 10, the program compiles and runs, but I still get the compiler warning message.
> >>>
> >>> What options should I use to compile with -Wall and not get warnings but still have the program run properly
> >>
> >> Don't use options. Instead, make large arrays ALLOCATABLE and allocate
> >> them to the desired size.
> >
> > Sort of departs from the KISS principle.
>
> Yet there are many practical differences to using allocatable rather
> than static or automatic arrays. If things are simplified too much, the
> programmer has no control left over those differences.
>
> > An ALLOCATE statement needs to have STAT option or an ERRMSG
> > option when the array is large.
>
> Perhaps so, but many times the programmer just wants the program to fail
> and stop,

Without any decent error message? Not likely programmers want that.
At least with STAT option and test, programmer can provide himself
an error report including details of what statement failed.

With ERRMSG, programmer receives a message; might not be detailed enough;
depemds on compiler.

> and that is what typically happens by default when those
> options are not specified.

It is inappropriate to have the program crash.

> I have debugged programs in the past where the programmer had used the
> status options on i/o statements or with open and close statements, but
> did not follow through well enough afterwards to track down and print
> the appropriate error message.

That's what happens with a lazy or unskilled programmer.
At least, you could have added to the error diagnostic
to provide the info needed.

> Simply removing those ISTAT= arguments
> and letting the compiler and operating system do its thing was sometimes
> the quickest way to understand what was the problem.
>
> There is also the complication involving lazy allocation, where the
> allocate() statement always succeeds, and it is only later when too much
> of the array is actually referenced when the error occurs.

either that's the fault of the compiler, or it's a programming error (subscript
out-of range).

[ga...@u.washington.edu]

In the case of static or automatic allocation, there is no way to
be notified and be able to print your own message. On many systems,
the program fails in mysterious ways.

It is a convenience of ALLOCATABLE and of allocating for POINTER
that the program has the ability to control the message, but as noted
also the ability to leave the message to the system.

One small thing, though. In the case of static or automatic arrays,
one gives the size with the declaration, but for ALLOCATABLE, one
does not have that ability. It would be a small convenience to allow
one to specify a default value for the allocation size with the
declaration. This is especially true when one is using ALLOCATABLE
in places where automatic allocation would also work, as it reduces
the number of changes needed.

[Ron Shepard]

On 1/1/20 3:44 AM, ga...@u.washington.edu wrote:
> One small thing, though. In the case of static or automatic arrays,
> one gives the size with the declaration, but for ALLOCATABLE, one
> does not have that ability. It would be a small convenience to allow
> one to specify a default value for the allocation size with the
> declaration. This is especially true when one is using ALLOCATABLE
> in places where automatic allocation would also work, as it reduces
> the number of changes needed.

Discussions of this feature go back about 10 years, at least.

If an allocatable array could be declared with a default size, including
size zero, and also with default values, then many programs could be
simplified significantly. In many cases, there would be no need to test
for the status of those variables (which is a frequent source of
errors), and no need to ever have an allocate() or deallocate()
statement in the entire program (because of allocate on assignment
semantics). That ability to initialize the state of allocatable varibles
would eliminate a whole class of programming errors. It would be more
than just a small convenience.

Some of these same benefits would also apply to the initialization of
pointers. I think this might have been addressed in the latest f2018
revision, but it is not yet implemented in the compilers I'm using.

Here are a few examples of what I think should be allowed:

integer, allocatable :: a(1) ! default size with no value.
integer, allocatable :: b(2)=[2,1] ! default size with values.
integer, allocatable :: c(1:0) ! size zero.

All of this should also apply to allocatable scalars, including
character strings.

There are some complications associated with this feature when the SAVE
attribute is added, so those would need to be worked out. Also, if this
declaration is a component of a derived type, then there are some
complications. But in any case, I think this would still be a nice
feature to add to the language.

$.02 -Ron Shepard

[robin....@gmail.com]

On Wednesday, January 1, 2020 at 8:44:36 PM UTC+11, ga...@u.washington.edu wrote:
> In the case of static or automatic allocation, there is no way to
> be notified and be able to print your own message.

A STATIC array is static, and the compiler itself can issue a message
at compile time.

An AUTOMATIC array should always work, provided that the extent
does not exceed what is available at run time. In which case,
the run-time should issue an error message.

> On many systems,

such as?

> the program fails in mysterious ways.

> It is a convenience of ALLOCATABLE and of allocating for POINTER
> that the program has the ability to control the message, but as noted
> also the ability to leave the message to the system.

which may not be helpful, particularly as to the ALLOCATE statement
in question.

> One small thing, though. In the case of static or automatic arrays,
> one gives the size with the declaration, but for ALLOCATABLE, one
> does not have that ability.

In PL/I the bounds of a CONTROLLED [i.e., ALLOCATABLE] array
can be supplied at compile time and/or at run time.

> It would be a small convenience to allow
> one to specify a default value for the allocation size with the
> declaration. This is especially true when one is using ALLOCATABLE
> in places where automatic allocation would also work, as it reduces
> the number of changes needed.

PL/I traps requests for excessive storage, and raises the STORAGE
condition, in which case the programmer can regain control and
continue execution (with, possibly, a reduced storage request).

In Fortran, users of ALLOCATABLE arrays are advised to use the STAT or ERRMSG
options on ALLOCATE statements.

[robin....@gmail.com]

On Thursday, January 2, 2020 at 5:08:35 AM UTC+11, Ron Shepard wrote:

> On 1/1/20 3:44 AM, g....@u.washington.edu wrote:
> > One small thing, though. In the case of static or automatic arrays,
> > one gives the size with the declaration, but for ALLOCATABLE, one
> > does not have that ability. It would be a small convenience to allow
> > one to specify a default value for the allocation size with the
> > declaration. This is especially true when one is using ALLOCATABLE
> > in places where automatic allocation would also work, as it reduces
> > the number of changes needed.
>
> Discussions of this feature go back about 10 years, at least.
>
> If an allocatable array could be declared with a default size, including
> size zero, and also with default values, then many programs could be
> simplified significantly. In many cases, there would be no need to test
> for the status of those variables (which is a frequent source of
> errors), and no need to ever have an allocate() or deallocate()
> statement in the entire program (because of allocate on assignment
> semantics).

which is probably the reason that default sizes and values are
not available.
Without a specific ALLOCATE statement, there is no means for
the programmer to test for errors via STAT Aand ERRMSG options.

[ga...@u.washington.edu]

On Wednesday, January 1, 2020 at 5:35:07 PM UTC-8, robin...@gmail.com wrote:
> On Wednesday, January 1, 2020 at 8:44:36 PM UTC+11, ga...@u.washington.edu wrote:
> > In the case of static or automatic allocation, there is no way to
> > be notified and be able to print your own message.

> A STATIC array is static, and the compiler itself can issue a message
> at compile time.

Large static arrays might make the load module too big to load,
and the system should give a message at load time. Not all systems
do that.

> An AUTOMATIC array should always work, provided that the extent
> does not exceed what is available at run time. In which case,
> the run-time should issue an error message.

Automatic might be stack or heap, and systems may or may not figure
that out. As noted above, for systems with lazy allocation, the
problem can come at an unexpected time.

As for ALLOCATABLE, if the only thing to do is print a message
and die, might as well let the Fortran library do that.

[Ron Shepard]

On 1/1/20 7:39 PM, robin....@gmail.com wrote:
> On Thursday, January 2, 2020 at 5:08:35 AM UTC+11, Ron Shepard wrote:

[...]>> If an allocatable array could be declared with a default size,

including
>> size zero, and also with default values, then many programs could be
>> simplified significantly. In many cases, there would be no need to test
>> for the status of those variables (which is a frequent source of
>> errors), and no need to ever have an allocate() or deallocate()
>> statement in the entire program (because of allocate on assignment
>> semantics).
>
> which is probably the reason that default sizes and values are
> not available.
> Without a specific ALLOCATE statement, there is no means for
> the programmer to test for errors via STAT Aand ERRMSG options.

I'm not sure what is the point of this statement. Fortran already allows
allocation of arrays without specific allocate() statements.

Are you suggesting that allocate-on-assignment be removed from the language?

The point of this discussion is the ability to initialize an allocatable
array in the same way that other arrays can be initialized. I was not
suggesting that useful features be removed from the language, I was
suggesting that useful features be added.

$.02 -Ron Shepard

[JCampbell]

It appears to me that some comments are missing the initial point of this thread.
The problem is not about the static, local, automatic or allocatable arrays and how Robin thinks they must be defined as the Fortran standard states. (Does the standard differentiate between heap and stack ?)
When this program does not run, it is because the stack is too small. In general, when using large arrays, the stack is always going to be too small, especially for 64-bit. This does not happen with the heap which is extendable at run time. This is why compilers provide options like -fmax-stack-var-size to switch local and automatic arrays from the stack to the heap, although there are implementation limitations when deciding “what is a large array”. Compilers should not need options. They should be smart enough to fix this problem.

For me when writing multi-thread calculations, there can be performance gains using a stack that is separate for each thread. Sometimes we want to allocate Private variables and arrays to the same small area, when we want to locate these values on the L1 cache, so a local stack is useful. Again, compilers provide options like -fstack-arrays for this case. However the design of the stack, to not be extendable, means that this stack option is never a robust solution.

I thought I had a good OMP solution, where I managed each thread stack size and gained a noticeable performance advantage (using gFortran) with -fstack-arrays. But there is always a project where the fixed stack is not big enough and the program crashes. Unfortunately this always occurs when the project is late and rebuilding the program is the last distraction you want.

Why can't the stack be extendable ?

Perhaps the answer could be multiple heaps with the primary heap for shared arrays, then secondary heaps for private arrays. After all the memory addresses are virtual.

Then again, isn’t this problem the fault of the compiler.
Surely space for a 1.6 GB local array should be allocated at run time and placed on the heap.
If the array doesn’t fit on the stack then give it an address on the heap. Reporting stack overflow is a lazy response.

[ga...@u.washington.edu]

On Thursday, January 2, 2020 at 12:18:05 AM UTC-8, JCampbell wrote:
> It appears to me that some comments are missing the initial
> point of this thread.

(snip)

> When this program does not run, it is because the stack is too small.
> In general, when using large arrays, the stack is always going to be
> too small, especially for 64-bit.

There really isn't a good reason for that, and less with 64 bit.

The question is how much of the address space to give to the stack.

For x86, the stack grows down. Usual method, at least for non-virtual
systems, is to start the stack at the top of whatever address space
is available, and the heap at the bottom.

Well, one reason to limit stack size, is that is what limits
infinite recursion.

> This does not happen with the heap which is extendable at run time.
> This is why compilers provide options like -fmax-stack-var-size to
> switch local and automatic arrays from the stack to the heap,
> although there are implementation limitations when deciding
> “what is a large array”. Compilers should not need options.
> They should be smart enough to fix this problem.

(snip)

Part of the reason is that C programmers don't put large
data items on the stack. At least traditionally (I believe
it has changed), only fixed size data items went on the
stack. Even with variable sized automatic data, C programmers
don't tend to use it.

OS were designed around the needs of C programmers.

With multiple threads, all run in the same address space, but
with their own stack. The most threads you can have is the
address space size divided by the stack size per thread.

It used to be that systems would allocate backing store
(page file space) to the whole virtual address space, but now
most don't do that. Unix-like systems seem to have a soft
and hard limit to stack size, where the hard limit is set
in some configuration file, and soft limit can be set by the
user up to the hard limit.

The system where I am writing this seems to have the hard
limit set to 65532K, or about 64 Megabytes.

> I thought I had a good OMP solution, where I managed each thread
> stack size and gained a noticeable performance advantage
> (using gFortran) with -fstack-arrays. But there is always a project
> where the fixed stack is not big enough and the program crashes.
> Unfortunately this always occurs when the project is late and
> rebuilding the program is the last distraction you want.

> Why can't the stack be extendable ?

x86, and I suspect others, grow the stack down. You start at some
large address, larger for virtual storage systems. Increasing the
stack size later means moving everything to higher addresses.
You can't move data that might have pointers to it, which is usual
for stack (and heap) data.

But even if the stack grew up, it would have to be below
everything already in the heap, and even more, not run into
any other stack.

I suspect that 8192K or 65532K limits were set on 32 bit systems,
to still allow for a reasonable number of threads, and not increased
with 64 bit systems.

[Ron Shepard]

On 1/2/20 2:18 AM, JCampbell wrote:
> For me when writing multi-thread calculations, there can be performance gains using a stack that is separate for each thread.

Stack allocation is always more efficient than heap allocation, whether
single- or multiple-thread. To push or pop the stack, just a couple of
storage locations need to be modified. Allocation on the heap requires
overhead associated with searching for a segment of the correct size,
while deallocation requires combining adjacent deallocated fragments
into a single fragment. This overhead is called "garbage collection",
and it applies only to heap storage, not to stack storage.

I do not think that fixed stack size is an intrinsic property of stacks.
I think that is just historical precedent. When we all had to do memory
management manually in f77, it was not uncommon to work with two stack
pointers, one increasing and the other decreasing, within the same
workspace. That could be done also at the OS level rather than the user
level, and it would give the fortran programmer control over the limits
directly. Elsethread, someone mentioned allocating multiple stacks from
within the heap, that is the same idea.

$.02 -Ron Shepard

[Gary Scott]

I've never quite understood why systems using a stack could not detect
the stack overflow at run time and punt to heap. That seems a
preferable behavior in most cases.

[Gary Scott]

On 1/2/2020 2:18 AM, JCampbell wrote:
> It appears to me that some comments are missing the initial point of this thread.
> The problem is not about the static, local, automatic or allocatable arrays and how Robin thinks they must be defined as the Fortran standard states. (Does the standard differentiate between heap and stack ?)
> When this program does not run, it is because the stack is too small. In general, when using large arrays, the stack is always going to be too small, especially for 64-bit. This does not happen with the heap which is extendable at run time. This is why compilers provide options like -fmax-stack-var-size to switch local and automatic arrays from the stack to the heap, although there are implementation limitations when deciding “what is a large array”. Compilers should not need options. They should be smart enough to fix this problem
>

> For me when writing multi-thread calculations, there can be performance gains using a stack that is separate for each thread. Sometimes we want to allocate Private variables and arrays to the same small area, when we want to locate these values on the L1 cache, so a local stack is useful. Again, compilers provide options like -fstack-arrays for this case. However the design of the stack, to not be extendable, means that this stack option is never a robust solution.
>
> I thought I had a good OMP solution, where I managed each thread stack size and gained a noticeable performance advantage (using gFortran) with -fstack-arrays. But there is always a project where the fixed stack is not big enough and the program crashes. Unfortunately this always occurs when the project is late and rebuilding the program is the last distraction you want.
>
> Why can't the stack be extendable ?
>
> Perhaps the answer could be multiple heaps with the primary heap for shared arrays, then secondary heaps for private arrays. After all the memory addresses are virtual.
>
> Then again, isn’t this problem the fault of the compiler.
> Surely space for a 1.6 GB local array should be allocated at run time and placed on the heap.
> If the array doesn’t fit on the stack then give it an address on the heap. Reporting stack overflow is a lazy response.
>

The application programmer should never have to think about whether
something is on the stack or not. That should be internal optimization
decision. At run time, the system should determine whether the stack
size is sufficient or not and if not, punt to heap.

[ga...@u.washington.edu]

On Thursday, January 2, 2020 at 10:14:50 AM UTC-8, Ron Shepard wrote:

(snip)

> Stack allocation is always more efficient than heap allocation, whether
> single- or multiple-thread. To push or pop the stack, just a couple of
> storage locations need to be modified. Allocation on the heap requires
> overhead associated with searching for a segment of the correct size,
> while deallocation requires combining adjacent deallocated fragments
> into a single fragment. This overhead is called "garbage collection",
> and it applies only to heap storage, not to stack storage.

Yes. Stack allocation is nice for things that need to be fast,
and is especially popular for function call arguments and return
addresses. It also works well for a small collection of automatic
variables allocated on function entry. (Note that many of the
buffer overruns that are favorite for hackers come from C allocating
on the same stack as return addresses.)

I don't know that it was intentional, but C programmers normally
use auto for small arrays, and malloc() for large ones, especially
ones with size unknown at compile time.

It might be that Fortran should check the size at run time,
and allocate larger automatic arrays on the heap.

Reminds me of the FREQUENCY statement in early Fortran, which allowed
one to tell the compiler how many times a DO loop might run, and
the relative likelihood of IF branches. That allowed compilers to make
some optimizing decisions using that information.

There is no way for a compiler to guess the size at run time of
variable sized arrays. Probably the best choice is changing with
time, as people adapt to larger memory machines.

> I do not think that fixed stack size is an intrinsic property of stacks.
> I think that is just historical precedent. When we all had to do memory
> management manually in f77, it was not uncommon to work with two stack
> pointers, one increasing and the other decreasing, within the same
> workspace. That could be done also at the OS level rather than the user
> level, and it would give the fortran programmer control over the limits
> directly. Elsethread, someone mentioned allocating multiple stacks from
> within the heap, that is the same idea.

Fixed stack size is intrinsic on systems with a single address space
for one user program. (I believe that is all except OS/2.)

With a large virtual address space, one can leave enough room
for large stacks, before running into heap allocated items or
other stacks. The cost is fairly small.

The multiple segment addressing in x86 processors since the 80286
is very convenient for things like stacks of unknown size, but
all except OS/2 put the same segment into all selectors, essentially
one address space.

For MS-DOS, the stack is at the top of available memory and
grows down, the heap starts at the bottom. Routines can check
before pushing onto the stack that they don't go into something
else, but many don't.

Multithreading makes it more complicated, though.

[ga...@u.washington.edu]

On Thursday, January 2, 2020 at 10:17:10 AM UTC-8, Gary Scott wrote:

(snip)

> I've never quite understood why systems using a stack could not detect
> the stack overflow at run time and punt to heap. That seems a
> preferable behavior in most cases.

The main use for the stack is subprogram arguments and return
addresses. Subroutine call is designed to put the return address
on a stack (on many processors) and find it there for return.

It is convenient to allocate AUTO variables at the same time,
but I suspect that Fortran should not always do that. Using the
stack allows for stack pointer relative addressing, which is
convenient. Not doing that requires a register, or memory location
(and indirect addressing) for auto variables.

The question in another discussion related to temporary variables
used by the compiler, and often not known by the programmer.
Putting them on the stack is especially surprising to users.

[ga...@u.washington.edu]

On Thursday, January 2, 2020 at 10:20:57 AM UTC-8, Gary Scott wrote:

(snip)

> The application programmer should never have to think about whether
> something is on the stack or not. That should be internal optimization
> decision. At run time, the system should determine whether the stack
> size is sufficient or not and if not, punt to heap.

There are many things that application programmers should not have
to think about, but for optimization reasons, they do. Some system
require programs to give a size when opening a new file. That allows
the file system to make better choices in where to allocate them.

The ability to choose stack or heap at run time requires more
complicated code, and likely extra registers, than stack only.
The compiler needs to know to allow for that.

[Steve Lionel]

On 1/2/2020 3:18 AM, JCampbell wrote:
> Why can't the stack be extendable ?

It "can" be. OpenVMS VAX, for example, had an automatically extendable
stack whose only limit was when it bumped up against a small amount of
storage allocated to the command processor. But this took advantage of
the P0 and P1 address spaces of VAX and there was a hard limit of about
1GB. OpenVMS Alpha does something similar, but I forget the details.

How the stack works is fundamental to the design of the OS. On Windows,
the linker sets aside a fixed amount of storage for the stack, 10MB by
default if I recall correctly. You can increase this using a linker
option, but the maximum is 1GB, even on x64.

Linux has something it calls "unlimited" stack size, but it is really up
to a maximum specified when the kernel was compiled.

There's nothing to prevent all automatic things from being
heap-allocated - except that this is MUCH (100X or more) slower than
stack allocation, and for a lot of programs, using the stack works just
fine. (Heap allocation also requires deallocation, whereas this is
essentially free when the stack is used.)

That said, Fortran 2018's change, that procedures are recursive by
default, is going to put increasing pressure on stack allocation in
compilers, and a drive to do more with heap allocation.

The standard, of course, is completely silent on how automatic variables
are implemented. It specifies the behavior, not the mechanism, so the
concepts of "heap" and "stack" do not appear.

[Beliavsky]

On Thursday, January 2, 2020 at 3:58:20 PM UTC-5, Steve Lionel wrote:

> That said, Fortran 2018's change, that procedures are recursive by
> default, is going to put increasing pressure on stack allocation in
> compilers, and a drive to do more with heap allocation.

Few of the procedures I use are recursive, and those are labeled as such, as versions of Fortran earlier than F2018 require. Will the F2018 change cause programmers to pay in CPU time for something they are not using?

[ga...@u.washington.edu]

On Thursday, January 2, 2020 at 1:12:06 PM UTC-8, Beliavsky wrote:

(snip)

> Few of the procedures I use are recursive, and those are labeled
> as such, as versions of Fortran earlier than F2018 require.
> Will the F2018 change cause programmers to pay in CPU time for
> something they are not using?

For some years now, there has been work toward speeding up
function calls, such as keeping some arguments in registers.

This at the same time that processors are getting much faster.

But as noted in the part that I snipped, this is mostly moving
from static allocation to automatic. I suspect that most are already
doing automatic allocation for local variables, but maybe someone
will report actual details.

Many languages, such as C and Pascal, have always allowed for
recursion without specifying it.

[JCampbell]

Ron,

Yes stack is always more efficient than heap, but is significant only for a few types of problems; mainly those that do lots of allocation of small arrays.

If I am allocating large GByte arrays, these can go on the heap with negligible performance issues. It is just another address and set of memory pages.

Most of my cases when there is a stack overflow, it is large arrays that cause the problem. What is needed is when large arrays are to be allocated on the stack, then -fmax-stack-var-size=n needs to work. I suspect that this is not always the case for automatic arrays, and the compiler takes the easy way out. This test should be implemented at run time and not at compile time. After all, it is just an address being allocated ( plus a heap or stack status when deallocate takes place )

With 64-bit, compilers need to be smarter when allocating local or automatic arrays. It appears this is more of an issue with Fortran than with C and possibly why gFortran might have problems in this area.

Also; I tested the OP example and it may be a bug in gFortran. When I ran it with a larger array WerFault.exe started reporting an error, rather than the application xstack_size.exe.

Basically gFortran needs to better manage how large local arrays are allocated. Local are an easier case than automatic which cause most problems. Also, how PRIVATE arrays are managed in !$OMP becomes a more important but complex case. Is this addressed in later versions of OpenMP ?

[Steve Lionel]

I don't think so. As noted elsewhere, compilers nowadays tend to put
local variables on the stack already. Variables marked SAVE won't be
affected. You can always mark procedures as NON_RECURSIVE if you believe
it to be an issue.

[robin....@gmail.com]

[robin....@gmail.com]

On Thursday, January 2, 2020 at 12:50:03 PM UTC+11, ga...@u.washington.edu wrote:

> On Wednesday, January 1, 2020 at 5:35:07 PM UTC-8, r.....@gmail.com wrote:
> > On Wednesday, January 1, 2020 at 8:44:36 PM UTC+11, ga...@u.washington.edu wrote:
> > > In the case of static or automatic allocation, there is no way to
> > > be notified and be able to print your own message.
>
> > A STATIC array is static, and the compiler itself can issue a message
> > at compile time.
>
> Large static arrays might make the load module too big to load,

A large static array may make the object module large.
It's unlikely that a load module is too large, given the
size of memory.

Again, if the size is a problem, the compiler can diagnose it.

> and the system should give a message at load time. Not all systems
> do that.

If the loader cannot handle an object/load module,
it can give an error message.

> > An AUTOMATIC array should always work, provided that the extent
> > does not exceed what is available at run time. In which case,
> > the run-time should issue an error message.
>
> Automatic might be stack or heap, and systems may or may not figure
> that out.

Whereever it is at run-time, an automatic array should always work.

> As noted above, for systems with lazy allocation, the
> problem can come at an unexpected time.

Such wojuld be a defective implementation, as noted apreviously.

> As for ALLOCATABLE, if the only thing to do is print a message
> and die, might as well let the Fortran library do that.

As noted above, such system messages are generally useless,
or at best, non-informative.

An error message (describing the problem and noting which statement
caused the error is always going to be better than any system error message.

[robin....@gmail.com]

On Thursday, January 2, 2020 at 7:14:59 PM UTC+11, Ron Shepard wrote:

> On 1/1/20 7:39 PM, r......@gmail.com wrote:
> > On Thursday, January 2, 2020 at 5:08:35 AM UTC+11, Ron Shepard wrote:
> [...]>> If an allocatable array could be declared with a default size,
> including
> >> size zero, and also with default values, then many programs could be
> >> simplified significantly. In many cases, there would be no need to test
> >> for the status of those variables (which is a frequent source of
> >> errors), and no need to ever have an allocate() or deallocate()
> >> statement in the entire program (because of allocate on assignment
> >> semantics).
> >
> > which is probably the reason that default sizes and values are
> > not available.
> > Without a specific ALLOCATE statement, there is no means for

> > the programmer to test for errors via STAT and ERRMSG options.

>
> I'm not sure what is the point of this statement. Fortran already allows
> allocation of arrays without specific allocate() statements.

True, but if done as you suggest for ALLOCATABLE arrays,
then without a specific ALLOCATE statement for the size that you
require, the opportunity to trap allocation errors vanishes.

> Are you suggesting that allocate-on-assignment be removed from the language?
>
> The point of this discussion is the ability to initialize an allocatable
> array in the same way that other arrays can be initialized. I was not
> suggesting that useful features be removed from the language, I was
> suggesting that useful features be added.

It's not a useful feature being added. You suggested that the only
means of trapping errors arising from an ALLOCATE statement be
eliminated.

[robin....@gmail.com]

On Thursday, January 2, 2020 at 7:18:05 PM UTC+11, JCampbell wrote:
> It appears to me that some comments are missing the initial point of this thread.
> The problem is not about the static, local, automatic or allocatable arrays and how Robin thinks they must be defined as the Fortran standard states. (Does the standard differentiate between heap and stack ?)
> When this program does not run, it is because the stack is too small.

Use the appropriate compiler option(s).

> In general, when using large arrays, the stack is always going to be too small, especially for 64-bit. This does not happen with the heap which is extendable at run time. This is why compilers provide options like -fmax-stack-var-size to switch local and automatic arrays from the stack to the heap,

This is an implementer's choice.

> although there are implementation limitations when deciding “what is a large array”. Compilers should not need options.

Most compilers need options.

> They should be smart enough to fix this problem.

You mean, the compiler writer(s) should be smart enough to make it work.

> For me when writing multi-thread calculations, there can be performance gains using a stack that is separate for each thread. Sometimes we want to allocate Private variables and arrays to the same small area, when we want to locate these values on the L1 cache,

Is an L1 cache a standard Fortran feature?

[Ron Shepard]

On 1/3/20 10:54 PM, robin....@gmail.com wrote:
>> The point of this discussion is the ability to initialize an allocatable
>> array in the same way that other arrays can be initialized. I was not
>> suggesting that useful features be removed from the language, I was
>> suggesting that useful features be added.
> It's not a useful feature being added. You suggested that the only
> means of trapping errors arising from an ALLOCATE statement be
> eliminated.

I'm not suggesting that anything currently in the language be
eliminated, I'm advocating a new feature that has the potential to
eliminate a large class of programming errors.

$.02 -Ron Shepard

[robin....@gmail.com]

On Saturday, January 4, 2020 at 6:28:28 PM UTC+11, Ron Shepard wrote:

That seems extremely unlikely.
In reality, it would encourage a practice that leads to
run-time errors.

Better to ask for the storage that you want.

Why do you think that STAT and ERRMSG options were included
as options for the ALLOCATE statement.
They were added for a reason.

[ga...@u.washington.edu]

On Sunday, January 5, 2020 at 4:51:15 PM UTC-8, robin...@gmail.com wrote:

(snip)

> That seems extremely unlikely.
> In reality, it would encourage a practice that leads to
> run-time errors.

> Better to ask for the storage that you want.

> Why do you think that STAT and ERRMSG options were included
> as options for the ALLOCATE statement.
> They were added for a reason.

Most systems will print a message and terminate when a request
can't be satisfied. In most cases, I suspect the same message that
is returned for ERRMSG. Not much reason, then, if you are going
to terminate anyway, not to let the system do it.

If there is something else to do when you find not enough memory
available, then use the features.

Somewhere in this discussion is the suggestion to use ALLOCATABLE
arrays in place of automatic arrays. As automatic arrays don't give
one the ability to handle allocation failure, when replacing it with
ALLOCATABLE arrays, and assuming reasonable messages from the run-time
system, not much reason to change.

As I noted previously, it would be easiest to make the transition
if ALLOCATABLE allowed one to put a default size on the declaration.
(Which might be constant or variable.) That would reduce the changes
needed, and increase the probability of getting it right the first time.

[JCampbell]

For 64-bit, handling ALLOCATE errors is not very useful.
The practicality is that ALLOCATE rarely fails. (It often did on 32-bit but that is the past!)
It can fail if the requested memory to be allocated exceeds the addressable memory or the virtual memory allowance, but these are typically very large.

What is a more practical limit is when the requested memory to be allocated exceeds the available physical memory. This can result in a significant performance penalty and most often is not what is wanted. But this is not reported as an error or preferably a warning.

What we don't have is any controlled response when local, automatic or private arrays exceed their available stack.
This is where better control or response needs to be provided.
It is frustrating to most users when running on a pc with say 32 gb of memory available, most of it unused and you get an error that the stack has overflowed and your application crashes.

The Fortran language gives us nothing in this case. It is left to the O/S to report little, then back to the compiler to rebuild, often with obscure rebuild options. We need to locate then shift the offending arrays from the stack to the heap by recompiling and rebuilding the application.

The much easier response is to shift this array to the heap at run time.
Often the cause of this problem is a number of large arrays so a working version of -fmax-stack-var-size=n (at run time) should be the default.

I agree with Ron that stack arrays are preferable in some cases so an option (extension) is needed to specify where local or automatic arrays should be directed. This can be an even more significant issue for PRIVATE arrays in !$OMP. Perhaps a heap/stack option in !$OMP PRIVATE (...)

There needs to be more intelligence provided for overcoming stack size warnings or stack overflow errors, especially given that the solution is frustratingly obvious at run time and only 20% of memory is being used.

I am not finding much support for my assessment. If you disagree, please let me know why.

I hope something could be progressed in this area.

[Gary Scott]

Well, I had reason to stick with 32-bit because 3 party libraries fixed
problems first in the 32-bit libraries. I found the 64-bit libraries
always bringing up the rear. That could probably change going forward,
but I've about come to the end of my development career (or perhaps just
a break).

[robin....@gmail.com]

On Tuesday, January 7, 2020 at 10:28:42 AM UTC+11, ga...@u.washington.edu wrote:

> On Sunday, January 5, 2020 at 4:51:15 PM UTC-8, r......@gmail.com wrote:
>
> (snip)
>
> > That seems extremely unlikely.
> > In reality, it would encourage a practice that leads to
> > run-time errors.
>
> > Better to ask for the storage that you want.
>
> > Why do you think that STAT and ERRMSG options were included
> > as options for the ALLOCATE statement.
> > They were added for a reason.
>
> Most systems will print a message and terminate when a request
> can't be satisfied. In most cases, I suspect the same message that
> is returned for ERRMSG.

You don't know.

> Not much reason, then, if you are going
> to terminate anyway, not to let the system do it.

Since what the system does cannot be guaranteed.
far better to use either ERRMSS / STAT options
produce the messages.

That CAN be guaranteed.

> If there is something else to do when you find not enough memory
> available, then use the features.

To do that, you need to use STAT option, as i said.

[robin....@gmail.com]

A stack is not a Fortran language element.

> This is where better control or response needs to be provided.
> It is frustrating to most users when running on a pc with say 32 gb of memory available, most of it unused and you get an error that the stack has overflowed and your application crashes.

That's because the compiler is broken.
Tell the manufacturer and/or use another compiler.

> The Fortran language gives us nothing in this case. It is left to the O/S to report little, then back to the compiler to rebuild, often with obscure rebuild options. We need to locate then shift the offending arrays from the stack to the heap by recompiling and rebuilding the application.
>
> The much easier response is to shift this array to the heap at run time.
> Often the cause of this problem is a number of large arrays so a working version of -fmax-stack-var-size=n (at run time) should be the default.
>
> I agree with Ron that stack arrays are preferable in some cases so an option (extension) is needed to specify where local or automatic arrays should be directed. This can be an even more significant issue for PRIVATE arrays in !$OMP. Perhaps a heap/stack option in !$OMP PRIVATE (...)
>
> There needs to be more intelligence provided for overcoming stack size warnings or stack overflow errors, especially given that the solution is frustratingly obvious at run time and only 20% of memory is being used.

Use the available compiler options.

[FortranFan]

On Saturday, January 4, 2020 at 2:28:28 AM UTC-5, Ron Shepard wrote:

> .. I'm advocating a new feature that has the potential to

> eliminate a large class of programming errors.

> ..

@Ron Shepard,

Please join up with the other Fortranners who are already pursuing options to improve Fortran language by developing proposals at this GitHub site:

https://github.com/j3-fortran/fortran_proposals

I suggest you advocate your suggestions with ALLOCATE and utilities for available system memory, etc. at this GitHub site developed by Ondrej Certik, you will find good follow-up on your ideas.

It will be very highly valuable to have your input and feedback to improve Fortran for scientific and technical programming, especially with all the other proposals you will notice there.

[spectrum]

On Thursday, January 2, 2020 at 3:08:35 AM UTC+9, Ron Shepard wrote:
>
> If an allocatable array could be declared with a default size, including
> size zero, and also with default values, then many programs could be
> simplified significantly. In many cases, there would be no need to test
> for the status of those variables (which is a frequent source of
> errors), and no need to ever have an allocate() or deallocate()
> statement in the entire program (because of allocate on assignment

> semantics). That ability to initialize the state of allocatable varibles
> would eliminate a whole class of programming errors. It would be more
> than just a small convenience.
>
> Some of these same benefits would also apply to the initialization of
> pointers. I think this might have been addressed in the latest f2018
> revision, but it is not yet implemented in the compilers I'm using.
>
> Here are a few examples of what I think should be allowed:
>
> integer, allocatable :: a(1) ! default size with no value.
> integer, allocatable :: b(2)=[2,1] ! default size with values.
> integer, allocatable :: c(1:0) ! size zero.
>
> All of this should also apply to allocatable scalars, including
> character strings.
>

I think even something as simple as

integer, allocatable :: c(0) ! size zero.

would be useful as an alternative way to declare an allocatable array
(which I think causes no extra overhead because of no heap allocation).
With an option for bounds check on, this can give an error message like
"the index greater than the upper bound 0", which is easy to understand.
It can also be used directly as a starting point for list-like usage (e.g., c = [c, x]),
with no need for doing allocate( c(0) ) at some later stage.

I guess "allocation + initialization at once" like below

> integer, allocatable :: b(2)= [2,1] ! default size with values.

may also be possible if the initialization value can be determined
at compile time (e.g., something similar to those allowed for PARAMETER).

RE https://github.com/j3-fortran/fortran_proposals, I also believe it would be great
if Ron would make such a proposal with explanations based on actual
experiences / needs... (<-- IMO, I guess this last information is quite important).

[Ron Shepard]

On 1/7/20 2:01 PM, spectrum wrote:
> I guess "allocation + initialization at once" like below
>
>> integer, allocatable :: b(2)= [2,1] ! default size with values.
>
> may also be possible if the initialization value can be determined
> at compile time (e.g., something similar to those allowed for PARAMETER).

I guess I should point out a possible conflict if b(:) is local array in
a subroutine. What SHOULD happen in that case is that the array is
initialized, and possibly modified, and then becomes deallocated upon
return, like any other local allocatable array.

But f2003 made a mistake and defined such initialized variables to have
the SAVE attribute automatically. Initializing a SAVEd allocatable dummy
array doesn't make sense, and you cannot turn off that SAVE attribute to
avoid the problem. That leaves the programmer in the position of always
using a DEALLOCATE() of that variable -- that defeats the purpose of my
proposal, which is to avoid the need for either ALLOCATE() or
DEALLOCATE() statements for certain types of well-behaved arrays.

That default SAVE attribute was controversial from the beginning, but it
has been in the language for 17 years, so it likely isn't going to be
removed now.

$.02 -Ron Shepard

[qol...@gmail.com]

Actually I think it was the 1991 standard that defined the "implicit save" behaviour...

[Ron Shepard]

On 1/8/20 3:47 AM, qol...@gmail.com wrote:
> Actually I think it was the 1991 standard that defined the "implicit save" behaviour...

Maybe you are right, but I remember some period where initialization
without the SAVE behavior was undefined. Then the implicit save behavior
was added in order to avoid that situation and to make previously
nonconforming code standard. I think the correct approach would have
been for the compiler to print a warning and to require the programmer
to add the SAVE. If that had been done, then initialized allocatable
variables could behave as expected, namely the same as other local
allocatable variables, without requiring any exceptions to the general
declaration and initialization rules. I thought that was done in f2003,
but I might be off by a decade.

Allocatable scalars and arrays are one of the nicest features of modern
fortran, but there are some unnecessary restrictions to their use:

1) The inability to initialize them to a defined state that is
allocated, possibly with values.

2) The inability to read an alloctable array of unknown size; that
should just work in the obvious way.

3) Perform i/o on a structure with an allocatable component without
jumping through hoops; I think that operation also should just work in
the obvious way.

4) The ability to query the environment at runtime to determine the
total allocatable memory usage. This would allow the programmer to
choose algorithms, or to adjust parameters within an algorithm, in order
to optimize the execution.

$.02 -Ron Shepard

[Ian Harvey]

On 2020-01-09 17:51, Ron Shepard wrote:
> On 1/8/20 3:47 AM, qol...@gmail.com wrote:
>> Actually I think it was the 1991 standard that defined the "implicit
>> save" behaviour...
>
>
> Maybe you are right, but I remember some period where initialization
> without the SAVE behavior was undefined. Then the implicit save behavior
> was added in order to avoid that situation and to make previously
> nonconforming code standard. I think the correct approach would have
> been for the compiler to print a warning and to require the programmer
> to add the SAVE. If that had been done, then initialized allocatable
> variables could behave as expected, namely the same as other local
> allocatable variables, without requiring any exceptions to the general
> declaration and initialization rules. I thought that was done in f2003,
> but I might be off by a decade.
>
> Allocatable scalars and arrays are one of the nicest features of modern
> fortran, but there are some unnecessary restrictions to their use:
>
> 1) The inability to initialize them to a defined state that is
> allocated, possibly with values.

As previously discussed, this requires Fortran processors to have
additional underlying machinery to support this feature (when execution
of a program starts, some initialization code would need to be executed
for any scoping unit with such an initialized allocatable). That's ok -
often new features require additional machinery, but it is not zero cost
in terms of processor complexity.

(You do need to be careful that you don't permit initializers that
result in a circular ordering requirement for that initialization code.
Existing rules around ordering of declarations and the like for constant
expressions might be sufficient - not sure.)

> 2) The inability to read an alloctable array of unknown size; that
> should just work in the obvious way.

The obvious way is perhaps not that obvious.

> 3) Perform i/o on a structure with an allocatable component without
> jumping through hoops; I think that operation also should just work in
> the obvious way.

And that obvious way is...?

> 4) The ability to query the environment at runtime to determine the
> total allocatable memory usage. This would allow the programmer to
> choose algorithms, or to adjust parameters within an algorithm, in order
> to optimize the execution.

It would be easy enough for a runtime to track and report this (you can
track it now using Fortran source, for source that you have control
over), but there might not be that much of a link between the use of
memory by allocatable objects and the total use of memory of a program.
The necessary accounting will also cause a minor performance hit,
particularly in the face of things like threaded or multi-image
execution, depending on the precise definition of the thing you want
reportable.

[Ron Shepard]

On 1/9/20 3:37 PM, Ian Harvey wrote:
> On 2020-01-09 17:51, Ron Shepard wrote:

[...]

>> 2) The inability to read an alloctable array of unknown size; that
>> should just work in the obvious way.
>
> The obvious way is perhaps not that obvious.

You have an array, and you want to allocate and fill it with whatever is
the input data. I know how to do this by hand, but it requires extra
memory allocations and transfers. The i/o library knows already how to
do all of that without the extra work. It is already doing the hard
work, but then hiding it from you. Just make that capability available
directly to the programmer.

>
>> 3) Perform i/o on a structure with an allocatable component without
>> jumping through hoops; I think that operation also should just work in
>> the obvious way.
>
> And that obvious way is...?

There is a well-defined way when the components are not allocatable.
Reading and writing allocatable components in the same way seems obvious
enough.

I'm assuming here on reading that all of the components have been
allocated prior to the i/o operation.

>
>> 4) The ability to query the environment at runtime to determine the
>> total allocatable memory usage. This would allow the programmer to
>> choose algorithms, or to adjust parameters within an algorithm, in
>> order to optimize the execution.
>
> It would be easy enough for a runtime to track and report this (you can
> track it now using Fortran source, for source that you have control
> over), but there might not be that much of a link between the use of
> memory by allocatable objects and the total use of memory of a program.
> The necessary accounting will also cause a minor performance hit,
> particularly in the face of things like threaded or multi-image
> execution, depending on the precise definition of the thing you want
> reportable.

Whatever it is that causes your program to crash (stack overflow, swap
space limit, heap overflow, batch queue quota limit, etc.), that's what
needs to be available to the programmer. This information is usually
available in a machine-specific nonportable way, just standardize an API
to an intrinsic subprogram and make it portable.

$.02 -Ron Shepard

[ga...@u.washington.edu]

On Thursday, January 9, 2020 at 11:03:19 PM UTC-8, Ron Shepard wrote:
> On 1/9/20 3:37 PM, Ian Harvey wrote:
> > On 2020-01-09 17:51, Ron Shepard wrote:

> >> 2) The inability to read an alloctable array of unknown size; that
> >> should just work in the obvious way.

> > The obvious way is perhaps not that obvious.

> You have an array, and you want to allocate and fill it with whatever is
> the input data. I know how to do this by hand, but it requires extra
> memory allocations and transfers. The i/o library knows already how to
> do all of that without the extra work. It is already doing the hard
> work, but then hiding it from you. Just make that capability available
> directly to the programmer.

The I/O library doesn't know any more than you do.

The library could, as you say, allocate and reallocate until the
array was big enough to hold the available data.

Well, C can use realloc() which has the ability, though no
guarantee, to increase the size of an array in place.

But note that you can only realloc() in place an array at the end
of memory, which can only be one array.

On the other hand, and as the subject of this thread, the library
could loop and put data on the stack until it was all read in,
then allocate the array and copy from the stack. Of course that might
result in stack overflow in the case of really big arrays.

In most cases, the programmer (you) have some idea, maybe only a
rough idea, of the expected size of the data. Normal I/O libraries
have no such idea.

But this is also the problem with stack size. At some point, a
stack size needs to be specified. It might be from the compiler,
from the linker, a shell parameter, or system boot-time option.

In the case of dynamically allocated arrays, or temporary arrays
based on dynamically allocated data, there is no way for the system
and/or library to guess the size.

I suspect, though, that after all this discussion, the best way would
be for dynamically sized local arrays and unknown size temporaries
to be heap allocated.

[Ron Shepard]

On 1/10/20 2:02 AM, ga...@u.washington.edu wrote:
> On Thursday, January 9, 2020 at 11:03:19 PM UTC-8, Ron Shepard wrote:
>> On 1/9/20 3:37 PM, Ian Harvey wrote:
>>> On 2020-01-09 17:51, Ron Shepard wrote:
>
>>>> 2) The inability to read an alloctable array of unknown size; that
>>>> should just work in the obvious way.
>
>>> The obvious way is perhaps not that obvious.
>
>> You have an array, and you want to allocate and fill it with whatever is
>> the input data. I know how to do this by hand, but it requires extra
>> memory allocations and transfers. The i/o library knows already how to
>> do all of that without the extra work. It is already doing the hard
>> work, but then hiding it from you. Just make that capability available
>> directly to the programmer.
>
> The I/O library doesn't know any more than you do.

Actually it does. For unformatted records, for example, it knows the
record lengths. Knowing that, it could use that information to allocate
my unallocated allocatable array to the correct size and fill it up.

> The library could, as you say, allocate and reallocate until the
> array was big enough to hold the available data.

When I do this myself, I usually use a linked list to allocate and fill
buffers, and then at the end, when the array size is known, copy those
buffers back into my newly allocated array. That would be a reasonable
way for the i/o library to do the same thing. Only it would be more
efficient because the i/o library is already allocating temporary arrays
internally to transfer from whatever external device is connected, so it
could cut out the middle man in the process, me.

> Well, C can use realloc() which has the ability, though no
> guarantee, to increase the size of an array in place.
>
> But note that you can only realloc() in place an array at the end
> of memory, which can only be one array.

That's not the way I would do it.

>
> On the other hand, and as the subject of this thread, the library
> could loop and put data on the stack until it was all read in,
> then allocate the array and copy from the stack. Of course that might
> result in stack overflow in the case of really big arrays.

This is closer to what I'm suggesting. When I do this with linked lists,
I'm using allocatable arrays, which typically live in the heap, not the
stack. Using the stack would be more efficient, but as you say, possibly
limited due to stack size limitations. As a fortran programmer, I
currently have no way to query the stack to determine any of these
limits, but the i/o library does not have that limitation, it could
freely mix stack and heap allocation to optimize the whole process.

> In most cases, the programmer (you) have some idea, maybe only a
> rough idea, of the expected size of the data. Normal I/O libraries
> have no such idea.

But the i/o library has access to all of that information. The fortran
programmer doesn't.

We *SHOULD* have that access, but we don't. That is a separate issue.

> In the case of dynamically allocated arrays, or temporary arrays
> based on dynamically allocated data, there is no way for the system
> and/or library to guess the size.

They should not need to guess the size, they should just work. The
fortran programmer doesn't have any super powers that allow him to do
things the i/o library can't. It is rather more likely the other way around.

$.02 -Ron Shepard

[ga...@u.washington.edu]

On Friday, January 10, 2020 at 5:49:20 PM UTC-8, Ron Shepard wrote:

(snip, I wrote)

> > The I/O library doesn't know any more than you do.

> Actually it does. For unformatted records, for example, it knows the
> record lengths. Knowing that, it could use that information to allocate
> my unallocated allocatable array to the correct size and fill it up.

Yes, for unformatted it knows the record length, and might be able
to figure out the array size, but might not. Consider:

READ(1) N,(A(I),I=1,N),M,(B(I),I=1,M)

knowing the record length doesn't tell you the length of A and B.

On the other hand, if you do:

READ(1) N,M,(A(I),I=1,N),(B(I),I=1,M)

you can:

READ(1) N,M
BACKSPACE 1
ALLOCATE (A(N),B(M))
READ(1) N,M,(A(I),I=1,N),(B(I),I=1,M)

though I usually try to avoid BACKSPACE

But also, you could read from a terminal, paper tape,
or (more likely) pipe, which is not seekable and has no
way to know how much date is coming up.

> > The library could, as you say, allocate and reallocate until the
> > array was big enough to hold the available data.

> When I do this myself, I usually use a linked list to allocate and fill
> buffers, and then at the end, when the array size is known, copy those
> buffers back into my newly allocated array. That would be a reasonable
> way for the i/o library to do the same thing. Only it would be more
> efficient because the i/o library is already allocating temporary arrays
> internally to transfer from whatever external device is connected, so it
> could cut out the middle man in the process, me.

Which takes twice as much memory as the data needs, but then again,
that is probably usual now.

> > Well, C can use realloc() which has the ability, though no
> > guarantee, to increase the size of an array in place.

> > But note that you can only realloc() in place an array at the end
> > of memory, which can only be one array.

> That's not the way I would do it.

> > On the other hand, and as the subject of this thread, the library
> > could loop and put data on the stack until it was all read in,
> > then allocate the array and copy from the stack. Of course that might
> > result in stack overflow in the case of really big arrays.

> This is closer to what I'm suggesting. When I do this with linked lists,
> I'm using allocatable arrays, which typically live in the heap, not the
> stack. Using the stack would be more efficient, but as you say, possibly
> limited due to stack size limitations. As a fortran programmer, I
> currently have no way to query the stack to determine any of these
> limits, but the i/o library does not have that limitation, it could
> freely mix stack and heap allocation to optimize the whole process.

I am still remembering when memory wasn't so cheap, but yes, now
you can usually do that.

> > In most cases, the programmer (you) have some idea, maybe only a
> > rough idea, of the expected size of the data. Normal I/O libraries
> > have no such idea.

> But the i/o library has access to all of that information. The fortran
> programmer doesn't.

Well, the library can read the whole file, figure out how big
things are, and then allocate, rewind, and read again. But only
for seekable input. You can't do that with pipes or terminals.

> We *SHOULD* have that access, but we don't. That is a separate issue.

> > In the case of dynamically allocated arrays, or temporary arrays
> > based on dynamically allocated data, there is no way for the system
> > and/or library to guess the size.

> They should not need to guess the size, they should just work. The
> fortran programmer doesn't have any super powers that allow him to do
> things the i/o library can't. It is rather more likely the other way around.

I suppose so, but often the optimal way is different for big and
small, so someone has to decide.

The reason we have Fortran instead of Python, is that we get more
control over the low-level details, but also have a price in handling
those details.

[Ian Harvey]

On 2020-01-10 16:33, Ron Shepard wrote:
> On 1/9/20 3:37 PM, Ian Harvey wrote:
>> On 2020-01-09 17:51, Ron Shepard wrote:
> [...]
>>> 2) The inability to read an alloctable array of unknown size; that
>>> should just work in the obvious way.
>>
>> The obvious way is perhaps not that obvious.
>
> You have an array, and you want to allocate and fill it with whatever is
> the input data. I know how to do this by hand, but it requires extra
> memory allocations and transfers. The i/o library knows already how to
> do all of that without the extra work. It is already doing the hard
> work, but then hiding it from you. Just make that capability available
> directly to the programmer.

When you do it by hand, you implicitly make choices about the
implementation. I don't consider all those choices obvious.

I think that when you start down the path putting together the specific
list of requirements and behaviours for this sort of feature, it ends up
getting rather messy. Messy does not mean impossible, but it might be
undesirable.

I have an input file connected to unit `unit`.

1, 2, 3, 4

I have an I/O statement, where a and b are integer, allocatable rank one
arrays.

READ (unit, *) a, b

What happens? Be specific. Consider all the variations. Be specific.
Now try and do that without having many pages of complicated
requirements with lots of inscrutable special cases.

>>> 3) Perform i/o on a structure with an allocatable component without
>>> jumping through hoops; I think that operation also should just work
>>> in the obvious way.
>>
>> And that obvious way is...?
>
> There is a well-defined way when the components are not allocatable.
> Reading and writing allocatable components in the same way seems obvious
> enough.
>
> I'm assuming here on reading that all of the components have been
> allocated prior to the i/o operation.

That last requirement certainly wasn't obvious to me.

In all other places within the language, the value of an object of
derived type includes the "allocation status, ... dynamic type and type
parameters, bounds and value" ... of its components.

I personally figured that I/O of an object of derived type would include
I/O of all those aspects of its value (maintaining a consistent approach
to encapsulation of information within objects of the type). Putting
aside dynamic type and type parameters, you have a number of choices for
how you represent "is this component allocated, and if so here are its
bounds", choices ripe for disagreement between users as to what should
happen. I figured that this would be where the debate about
"obviousness" lay... but we didn't even make it there!

Type and type parameters then increase complexity further.

(Note that if are not dealing with a polymorphic component, and you know
the type parameters and bounds of what would otherwise be an allocatable
component ahead of an I/O statement... then it is questionable whether
the component needs to be allocatable at all. I suspect for a large
number of use cases that fit the requirement of "component allocate
before hand", this is an already solved problem.)

Another magnitude of complexity (is it a reference? is it a value?)
with pointer components.

Out of the blocks - you are saying it should work one way, I'm saying it
should work some other way, and I think that is very representative of
the situation, so rather than pick one or the other the standard just
says "here are some tools (UDDTIO), but otherwise sort it out yourselves".

>>> 4) The ability to query the environment at runtime to determine the
>>> total allocatable memory usage. This would allow the programmer to
>>> choose algorithms, or to adjust parameters within an algorithm, in
>>> order to optimize the execution.
>>
>> It would be easy enough for a runtime to track and report this (you
>> can track it now using Fortran source, for source that you have
>> control over), but there might not be that much of a link between the
>> use of memory by allocatable objects and the total use of memory of a
>> program. The necessary accounting will also cause a minor performance
>> hit, particularly in the face of things like threaded or multi-image
>> execution, depending on the precise definition of the thing you want
>> reportable.
>
> Whatever it is that causes your program to crash (stack overflow, swap
> space limit, heap overflow, batch queue quota limit, etc.), that's what
> needs to be available to the programmer. This information is usually
> available in a machine-specific nonportable way, just standardize an API
> to an intrinsic subprogram and make it portable.

I assumed by "allocatable memory" you were talking about memory used by
objects within the Fortran program that have the ALLOCATABLE attribute.
This is a retrospective measurement - "what has been allocated".

A retrospective query doesn't help a program understand what it might
get away with in future.

~~

(Fortran processor implementations could practically solve stack
overflow issues by not using the stack (underlying operating system
calls and CPU architecture requirements aside). Implementations don't
do that completely... because execution performance suffers massively.
Implementations don't even avoid stack usage partially, without compiler
options, due to performance concerns.)

[Ron Shepard]

On 1/11/20 5:10 PM, Ian Harvey wrote:
> On 2020-01-10 16:33, Ron Shepard wrote:
>> On 1/9/20 3:37 PM, Ian Harvey wrote:
>>> On 2020-01-09 17:51, Ron Shepard wrote:
>> [...]
>>>> 2) The inability to read an alloctable array of unknown size; that
>>>> should just work in the obvious way.
>>>
>>> The obvious way is perhaps not that obvious.
>>
>> You have an array, and you want to allocate and fill it with whatever
>> is the input data. I know how to do this by hand, but it requires
>> extra memory allocations and transfers. The i/o library knows already
>> how to do all of that without the extra work. It is already doing the
>> hard work, but then hiding it from you. Just make that capability
>> available directly to the programmer.
>
> When you do it by hand, you implicitly make choices about the
> implementation.  I don't consider all those choices obvious.

Of course, all implementations of a feature make such choices, but they
can still conform to the programmer interface, whatever that is. I am
not suggesting what that interface is, I'm just advocating for the
underlying functionality.

And just because some feature is implemented as part of the language
does not preclude the programmer from writing something better for
special cases. For an unrelated example of this, a programmer might use
both matmul() and dgemm() to do matrix-matrix products. There is nothing
wrong with that.

> I think that when you start down the path putting together the specific
> list of requirements and behaviours for this sort of feature, it ends up
> getting rather messy.  Messy does not mean impossible, but it might be
> undesirable.
>
> I have an input file connected to unit `unit`.
>
>   1, 2, 3, 4
>
> I have an I/O statement, where a and b are integer, allocatable rank one
> arrays.
>
>   READ (unit, *) a, b

That seems ambiguous to me, and to you, no doubt, which is why you are
using it in an argument against implementing this functionality. What
I'm requesting could just as well be implemented as

call read_and_allocate( unit, a )

in which case there is no such ambiguity. You would expect the
allocatable array a(:) to end up being allocated with 4 elements and
filled with the obvious data. Perhaps optional arguments could be added
to set the lower bounds, perhaps formats in addition to list directed,
and of course it should be generic for any data type.

> What happens?  Be specific.  Consider all the variations.  Be specific.
> Now try and do that without having many pages of complicated
> requirements with lots of inscrutable special cases.

You are looking for excuses to not implement the feature rather than
looking for ways to do so.

>>>> 3) Perform i/o on a structure with an allocatable component without
>>>> jumping through hoops; I think that operation also should just work
>>>> in the obvious way.
>>>
>>> And that obvious way is...?
>>
>> There is a well-defined way when the components are not allocatable.
>> Reading and writing allocatable components in the same way seems
>> obvious enough.
>>
>> I'm assuming here on reading that all of the components have been
>> allocated prior to the i/o operation.
>
> That last requirement certainly wasn't obvious to me.
>
> In all other places within the language, the value of an object of
> derived type includes the "allocation status, ... dynamic type and type
> parameters, bounds and value" ... of its components.

My request is for something simpler than all that. Suppose you have a
derived types

type xxx
integer :: a(4)
end type xxx

type yyy
integer, allocatable :: a(:)
end type yyy

type(xxx) :: x
type(yyy) :: y
allocate( y%a(4) )

From this point on, I think you should be able to use the derived types
x and y in pretty much the same way, including such things as

read(unit,*) x
read(unit,*) y

That's it. nothing deep and profound, just treat allocatable components
like nonallocatable ones when it is appropriate. This already applies to
assignments and expressions with array operators, but for some reason
not with i/o statements with derived types.

[...]

> Out of the blocks - you are saying it should work one way, I'm saying it
> should work some other way, and I think that is very representative of
> the situation, so rather than pick one or the other the standard just
> says "here are some tools (UDDTIO), but otherwise sort it out yourselves".

UDDTIO was slow to be implemented, so I do not have much experience with
it. But from what I've read, it seems the same as writing your own
subroutine to write out the data, and then invoking it through a
read/write statement rather than a call statement.

Is that right, is it just a thin layer of syntactic sugar?

In any case, I'm not suggesting that UDDTIO be removed from the
language, but rather than the obvious default behavior be invoked when
doing i/o on derived types. When you delete an old component or add a
new component to a derived type, then the obvious thing should happen
when a read/write is executed, you should not need to dig into a
user-level subprogram and manually synchronize with the new definition.
If you want to do something that is not obvious, then UDDTIO is there
for that.

>>>> 4) The ability to query the environment at runtime to determine the
>>>> total allocatable memory usage. This would allow the programmer to
>>>> choose algorithms, or to adjust parameters within an algorithm, in
>>>> order to optimize the execution.
>>>
>>> It would be easy enough for a runtime to track and report this (you
>>> can track it now using Fortran source, for source that you have
>>> control over), but there might not be that much of a link between the
>>> use of memory by allocatable objects and the total use of memory of a
>>> program. The necessary accounting will also cause a minor performance
>>> hit, particularly in the face of things like threaded or multi-image
>>> execution, depending on the precise definition of the thing you want
>>> reportable.

There should be no extra overhead because everything that I'm requesting
is already computed anyway. I'm just advocating that it be made visible
to the programmer through a standard interface.

As for doing it manually, yes you can do that for allocatable arrays.
Every time you allocate, deallocate, or allocate-on-assignment for such
an array, you could call a user-level subprogram to add or subtract from
the total. That would be a major hassle, to say the least, and as I said
above it would be redundant with what the runtime libraries are already
doing.

But the more serious problem is that there are other memory allocations
done by the compiler that you have no direct control over. So if you are
trying to avoid hitting some memory usage limit, keeping track of just a
subset of the memory usage would not solve that problem. The compiler
generates array temporaries in many places, in expressions, in order to
optimize common subexpressions over several statements, and during
subprogram calls. Sometimes the way this is done depends on the compiler
optimization level. The only real way to monitor the memory usage is to
query the runtime libraries that are doing all that.

>> Whatever it is that causes your program to crash (stack overflow, swap
>> space limit, heap overflow, batch queue quota limit, etc.), that's
>> what needs to be available to the programmer. This information is
>> usually available in a machine-specific nonportable way, just
>> standardize an API to an intrinsic subprogram and make it portable.
>
> I assumed by "allocatable memory" you were talking about memory used by
> objects within the Fortran program that have the ALLOCATABLE attribute.
> This is a retrospective measurement - "what has been allocated".

Yes, of course you also need to query to see what limits are in effect
so that you can tune your future allocations, or select the appropriate
algorithms, in order to fit within the constraints.

$.02 -Ron Shepard

[ga...@u.washington.edu]

On Saturday, January 11, 2020 at 6:22:15 PM UTC-8, Ron Shepard wrote:
> On 1/11/20 5:10 PM, Ian Harvey wrote:

(snip)

> > I think that when you start down the path putting together the specific
> > list of requirements and behaviours for this sort of feature, it ends up
> > getting rather messy.  Messy does not mean impossible, but it might be
> > undesirable.

> > I have an input file connected to unit `unit`.

> >   1, 2, 3, 4

> > I have an I/O statement, where a and b are integer, allocatable rank one
> > arrays.

> >   READ (unit, *) a, b

> That seems ambiguous to me, and to you, no doubt, which is why you are
> using it in an argument against implementing this functionality. What
> I'm requesting could just as well be implemented as

> call read_and_allocate( unit, a )

> in which case there is no such ambiguity. You would expect the
> allocatable array a(:) to end up being allocated with 4 elements and
> filled with the obvious data. Perhaps optional arguments could be added
> to set the lower bounds, perhaps formats in addition to list directed,
> and of course it should be generic for any data type.

Yes there is less ambiguity with only one array.

But even there, do you only read one line? Read to EOF?
Read until a certain value appears in one array element?

With two arrays, though, there are many possibilities.

It might read two elements for A, then two for B,
and if the file has more lines, on down the line.

Or alternate A then B? Three of A and one of B?

what does call read_and_allocate( unit, a, b) do?

From before dynamic memory, we set a nice large size for all
the arrays, and complain if they are not big enough.

With dynamic memory, we might have the user set a large
enough, but maybe not exact size.

I have in some programs sized arrays of size that I didn't
know in advance based on the size of some that I did.
There is enough correlation in size that it usually works.

But library routines won't know that.

[robin....@gmail.com]

Won't what happens depend on the current dimensions of both arrays?

> and to you, no doubt, which is why you are
> using it in an argument against implementing this functionality. What
> I'm requesting could just as well be implemented as
>
> call read_and_allocate( unit, a )

Why is this any different?
What bounds will array 'a' have? Surely it will have those bounds
that it has before the CALL is executed?

> in which case there is no such ambiguity. You would expect the
> allocatable array a(:) to end up being allocated with 4 elements

Why? Where does the bound 4 come from?

> and filled with the obvious data. Perhaps optional arguments could be added
> to set the lower bounds, perhaps formats in addition to list directed,
> and of course it should be generic for any data type.

Just use existing facilities.
No ambiguity then.

> > What happens?  Be specific.  Consider all the variations.  Be specific.
> > Now try and do that without having many pages of complicated
> > requirements with lots of inscrutable special cases.
>
> You are looking for excuses to not implement the feature rather than
> looking for ways to do so.

This is something that we don't need.

Simple statements like the previous two are easier to write and
to understand.

> That's it. nothing deep and profound, just treat allocatable components
> like nonallocatable ones when it is appropriate.

You have to be joking.

> This already applies to
> assignments and expressions with array operators, but for some reason
> not with i/o statements with derived types.

Does not seem to be transferrable to I/O.

[Thomas Koenig]

Ron Shepard <nos...@nowhere.org> schrieb:

> On 1/11/20 5:10 PM, Ian Harvey wrote:

>> I have an I/O statement, where a and b are integer, allocatable rank one
>> arrays.
>>
>>   READ (unit, *) a, b
>
> That seems ambiguous to me, and to you, no doubt, which is why you are
> using it in an argument against implementing this functionality.

If people want to allocation on read, I think list-directed I/O should
be excluded. Having a special format for this might make more sense.

>What
> I'm requesting could just as well be implemented as
>
> call read_and_allocate( unit, a )
>
> in which case there is no such ambiguity. You would expect the
> allocatable array a(:) to end up being allocated with 4 elements and
> filled with the obvious data. Perhaps optional arguments could be added
> to set the lower bounds, perhaps formats in addition to list directed,
> and of course it should be generic for any data type.
>
>> What happens?  Be specific.  Consider all the variations.  Be specific.
>> Now try and do that without having many pages of complicated
>> requirements with lots of inscrutable special cases.
>
> You are looking for excuses to not implement the feature rather than
> looking for ways to do so.

As a user, I would like to have something like this, but as somebody
who could have a hand implementing this, I would need clear rules
on what to do :-)

Consider a file containing

1 2 3 a

and you would like to read in an allocatable integer array. What
would you propose? Allocating an array of size three? Should there
be an error condition? If there is an error condition, should a partial
read succeed?

Now, consider a file containing

1 2 3
4 5 6


What should the program read? Three elements or six?

And what about a file with

1 2 3

4 5 6

?

[spectrum]

On Sunday, January 12, 2020 at 11:22:15 AM UTC+9, Ron Shepard wrote:
>
> (...) Suppose you have a derived types

>
> type xxx
> integer :: a(4)
> end type xxx
>
> type yyy
> integer, allocatable :: a(:)
> end type yyy
>
> type(xxx) :: x
> type(yyy) :: y
> allocate( y%a(4) )
>
> From this point on, I think you should be able to use the derived types
> x and y in pretty much the same way, including such things as
>
> read(unit,*) x
> read(unit,*) y

Indeed, if allocatable array components (like 'a') have already been allocated
before I/O, I think the above lines would correspond to

read(unit, *) x % a(:)
read(unit, *) y % a(:)

If there are more components, for example,

type zzz
integer :: n

integer, allocatable :: a(:)
end type

type(zzz) :: z

I would probably expect that

read(unit, *) z

corresponds to

read(unit, *) z % n, z % a(:)

(i.e., sequential read for each components, as declared in the type).

So, I am wondering why the former is currently illegal (and needs a custom I/O routine).
Possibly, is it to avoid some complexity due to possible pointer components, or
OO features like inheritance...?

[spectrum]

Because reading data is more complicated, I guess it is not surprising that derived-type
input is not provided by default. But, my (bigger) question is why derived-type output
is also made illegal by default...

For example, in the following code, I can print / read 'f1' (which has no allocatable
components) via default list-directed I/O, while cannot do a similar thing for 'f2'
(which has an allocatable component), even if 'f2 % a' is allocated beforehand.

program main
implicit none

type Foo1_t
integer :: n = 1
integer :: a( 2 ) = [10, 20]
endtype

type Foo2_t
integer :: n = 1
integer, allocatable :: a(:) !! = [30, 40] (cannot be assigned here)
endtype

type(Foo1_t) :: f1
type(Foo2_t) :: f2

f2 % a = [30, 40] !! or, allocate( f2 % a, source = [30, 40] )

!! Print components.

print *, "f1 = ", f1 !! this works

!-- print *, "f2 = ", f2 !! error
print *, "f2 = ", f2% n, f2% a !! so print each component

!! Read in components from stdin.

print *, "input f1"
read *, f1 !! this works
print *, "f1 = ", f1

print *, "input f2"
!-- read *, f2 !! error
read *, f2% n, f2% a(:) !! so read in each component
print *, "f2 = ", f2% n, f2% a(:)
end


$ gfortran-9 test.f90 && ./a.out

f1 = 1 10 20
f2 = 1 30 40
input f1
7 100 200 <-- my input from stdin
f1 = 7 100 200
input f2
8 300 400 <-- another input
f2 = 8 300 400

[Ron Shepard]

On 1/12/20 1:46 AM, robin....@gmail.com wrote:
>>>   READ (unit, *) a, b
>> That seems ambiguous to me,
> Won't what happens depend on the current dimensions of both arrays?

If the arrays have been previously allocated, then the standard already
defines what happens, and yes, it does depend on the current dimensions.

>
>> and to you, no doubt, which is why you are
>> using it in an argument against implementing this functionality. What
>> I'm requesting could just as well be implemented as
>>
>> call read_and_allocate( unit, a )
> Why is this any different?
> What bounds will array 'a' have? Surely it will have those bounds
> that it has before the CALL is executed?

You are asking a question about the syntax, not the functionality, but
here is how I think it should fit into the language. If this is
implemented through normal i/o statements

read(unit,*) a

then a(:) would need to be in an unallocated state for this to be
invoked. If a(:) were previously allocated, then the current rules
regarding i/o would apply. I'm not advocating removing or changing any
current behavior, I'm advocating additional new functionality. If a(:)
were unallocated, then the read statement would read the data, determine
the length, allocate the array, and fill the contents of that array with
the data. In the list-directed case above, the length would be
determined either by a "/" in the input or by the end of file. If this
were applied to unformatted i/o, then it would be the record length that
determines the array size. If it were applied also to namelist, then the
length determination could also include a subsequent variable name.

&input
a=1,2,3,4
b=...whatever...
/

If instead this were implemented through a separate intrinsic subroutine
call, then there might be more flexibility. For example, if a(:) were
previously allocated then maybe it would be required to deallocate it
first, and then proceed to read and allocate the array. I.e. in analogy
to the way move_alloc() works, or allocate-on-assignemnt.

But these are all syntax questions, the basic functionality I'm
advocating is simply the ability to read an array of unknown length
where the length is determined by the input data. That is a common
situation, not only in my own programming, but also it is a frequent
topic of discussion here in c.l.f.

>> in which case there is no such ambiguity. You would expect the
>> allocatable array a(:) to end up being allocated with 4 elements
> Why? Where does the bound 4 come from?

I was assuming an end-of-file after the 4th integer. In practice the
input might also be something like

1, 2, 3, 4 /
...other stuff...

and I would think that could also work. That would allow reading
multiple arrays of unknown length from the same file (but one at a time).

>
>> and filled with the obvious data. Perhaps optional arguments could be added
>> to set the lower bounds, perhaps formats in addition to list directed,
>> and of course it should be generic for any data type.
> Just use existing facilities.
> No ambiguity then.

That is how I do this now, either with an overallocated temporary array
or with linked lists, using current fortran facilities. Both of these
cases are inefficient compared to what the i/o library could do
directly. The overallocation depends, of course, on what I think at the
time is a practical upper bound; that choice sometimes needs to be
reexamined from time to time, so this approach is also error prone. The
efficient implementation of this functionality in the i/o library would
eliminate that possible error in addition to increasing the efficiency
(by eliminating one unnecessary memory copy).

[...on i/o involving allocatable components...]

> My request is for something simpler than all that. Suppose you have a
>> derived types
>>
>> type xxx
>> integer :: a(4)
>> end type xxx
>>
>> type yyy
>> integer, allocatable :: a(:)
>> end type yyy
>>
>> type(xxx) :: x
>> type(yyy) :: y
>> allocate( y%a(4) )
>>
>> From this point on, I think you should be able to use the derived types
>> x and y in pretty much the same way, including such things as
>>
>> read(unit,*) x
>> read(unit,*) y

Just in case it was not clear before, that first statement is currently
allowed, the second one is not.

> Simple statements like the previous two are easier to write and
> to understand.

I know, easier to read too. That is why I'm advocating this new
functionality.

>> That's it. nothing deep and profound, just treat allocatable components
>> like nonallocatable ones when it is appropriate.
> You have to be joking.

You just said above that it would be easier to write and understand, and
you were right.

>> This already applies to
>> assignments and expressions with array operators, but for some reason
>> not with i/o statements with derived types.
> Does not seem to be transferrable to I/O. [...]

It currently is not. But if it were extended in this way, then code
would be easier to read, write, and understand, and it would be less
error prone because the compiler would always know what was in the
derived type, it would not require manual updating of the user-written
i/o subroutine to keep it current with the definition.

$.02 -Ron Shepard

[Ron Shepard]

On 1/12/20 6:59 AM, Thomas Koenig wrote:
> Ron Shepard <nos...@nowhere.org> schrieb:
[...]

> As a user, I would like to have something like this, but as somebody
> who could have a hand implementing this, I would need clear rules
> on what to do :-)
>
> Consider a file containing
>
> 1 2 3 a
>
> and you would like to read in an allocatable integer array. What
> would you propose?

I would suggest starting with something simple, just reading one array
at a time, or maybe requiring the array to be the last item in the i/o
list. Then if there is programmer demand, it could be extended in the
future to allow more extensive parsing of the input data as would be
required for this example.

If this were done within namelist i/o, then that parsing is already done
for fixed-length arrays, so it seems like a small step to allow it also
for allocatable arrays.

> Now, consider a file containing
>
> 1 2 3
> 4 5 6
>
>
> What should the program read? Three elements or six?

That is a good question. I would vote to read until a "/" is found or
until an end-of-file is encountered. But reading one record at a time
would be useful too. The programmer could then easily splice together
multiple lines into a single output array if that was the final goal.

$.02 -Ron Shepard

[Ron Shepard]

On 1/12/20 7:09 AM, spectrum wrote:
> I would probably expect that
>
> read(unit, *) z
>
> corresponds to
>
> read(unit, *) z % n, z % a(:)
>
> (i.e., sequential read for each components, as declared in the type).
>
> So, I am wondering why the former is currently illegal (and needs a custom I/O routine).
> Possibly, is it to avoid some complexity due to possible pointer components, or
> OO features like inheritance...?

When the array is not allocatable, then the standard requires the
components to be read (and written) in the order they are declared. What
I am proposing is that this rule be extended also to allocatable arrays
that have been allocated.

Just to preempt comments about storage order, the standard already
allows the storage order within a derived type to be rearranged by the
processor, so regardless of whether or not the components are
allocatable, the processor must be prepared to rearrange the output from
storage order into the declared order when doing i/o. Extending the
capability to allocatable components would not add any additional burden
to the i/o library in this respect.

$.02 -Ron Shepard

[Gary Scott]

lots of opportunity to dramatically muck up IO performance. I prefer to
explicitly define my file record structure to be unambiguous as to data
type with full documentation of the structure so that it can be
reconstructed by others if necessary. I also try to structure it for
minimum negative impact on performance.

[FortranFan]

On Saturday, January 11, 2020 at 9:22:15 PM UTC-5, Ron Shepard wrote:

> On 1/11/20 5:10 PM, Ian Harvey wrote:

> ..

> > What happens?  Be specific.  Consider all the variations.  Be specific.
> > Now try and do that without having many pages of complicated
> > requirements with lots of inscrutable special cases.
>
> You are looking for excuses to not implement the feature rather than
> looking for ways to do so.

> ..

@Ron Shepard should take heart in this WG5 document:
https://isotc.iso.org/livelink/livelink?func=ll&objId=19530634&objAction=Open&viewType=1

and take note of the second most popular item in the user feedback to WG5 survey, "Automatic Allocation on READ into ALLOCATABLE character or array," with a Score of 5.04 that sounds quite similar to what @Ron Shepard is suggesting in this thread and which is also what @Ron Shepard has suggested often previously on comp.lang.fortran (CLF) forum.

@Ron Shepard would serve Fortran well by getting on to the new GitHub site for Fortran proposals and posting all these ideas there: https://github.com/j3-fortran/fortran_proposals.

[robin....@gmail.com]

On Monday, January 13, 2020 at 5:47:49 AM UTC+11, Ron Shepard wrote:

> On 1/12/20 1:46 AM, r......@gmail.com wrote:
> >>>   READ (unit, *) a, b
> >> That seems ambiguous to me,
> > Won't what happens depend on the current dimensions of both arrays?
>
> If the arrays have been previously allocated, then the standard already
> defines what happens, and yes, it does depend on the current dimensions.
>
> >
> >> and to you, no doubt, which is why you are
> >> using it in an argument against implementing this functionality. What
> >> I'm requesting could just as well be implemented as
> >>
> >> call read_and_allocate( unit, a )
> > Why is this any different?
> > What bounds will array 'a' have? Surely it will have those bounds
> > that it has before the CALL is executed?
>
> You are asking a question about the syntax, not the functionality, but
> here is how I think it should fit into the language. If this is
> implemented through normal i/o statements
>
> read(unit,*) a
>
> then a(:) would need to be in an unallocated state for this to be
> invoked. If a(:) were previously allocated, then the current rules
> regarding i/o would apply. I'm not advocating removing or changing any
> current behavior, I'm advocating additional new functionality. If a(:)
> were unallocated, then the read statement would read the data,

And, pray, how would it do that?, given that nothing is known about the number of elements in the array?

> determine the length, allocate the array,

And how would it do that? Still nothing is still not known about the
number of elements in the array.

> and fill the contents of that array with
> the data. In the list-directed case above, the length would be
> determined either by a "/" in the input or by the end of file.

So, a different action depending on whether an array has been allocated
or not, and leaving a '/' dangling or not in the input stream.

Surely, this is ambiguous, is it not?

> If this
> were applied to unformatted i/o, then it would be the record length that
> determines the array size.

Oh?

And might the WRITE encounter limits as to the length of an
unformatted record?

> If it were applied also to namelist, then the
> length determination could also include a subsequent variable name.
>
> &input
> a=1,2,3,4
> b=...whatever...
> /

oh, no ...

> If instead this were implemented through a separate intrinsic subroutine
> call, then there might be more flexibility.

groan ...

> For example, if a(:) were
> previously allocated then maybe it would be required to deallocate it
> first,

And not, something contradictory when using an intrinsic subrouting?

> and then proceed to read and allocate the array. I.e. in analogy
> to the way move_alloc() works, or allocate-on-assignemnt.
>
> But these are all syntax questions, the basic functionality I'm
> advocating is simply the ability to read an array of unknown length
> where the length is determined by the input data.

You haven't proposed anything where the length of the input
could be determined.

> That is a common
> situation, not only in my own programming, but also it is a frequent
> topic of discussion here in c.l.f.
>
> >> in which case there is no such ambiguity. You would expect the
> >> allocatable array a(:) to end up being allocated with 4 elements
> > Why? Where does the bound 4 come from?
>
> I was assuming an end-of-file after the 4th integer. In practice the
> input might also be something like
>
> 1, 2, 3, 4 /

As I said before, this is ambiguous.

> ...other stuff...
>
> and I would think that could also work. That would allow reading
> multiple arrays of unknown length from the same file (but one at a time).
>
> >
> >> and filled with the obvious data. Perhaps optional arguments could be added
> >> to set the lower bounds, perhaps formats in addition to list directed,
> >> and of course it should be generic for any data type.
> > Just use existing facilities.
> > No ambiguity then.
>
> That is how I do this now, either with an overallocated temporary array
> or with linked lists, using current fortran facilities. Both of these
> cases are inefficient compared to what the i/o library could do
> directly.

Again, your proposal is, at best, ambiguous.

[robin....@gmail.com]

On Monday, January 13, 2020 at 5:59:45 AM UTC+11, Ron Shepard wrote:
> On 1/12/20 6:59 AM, Thomas Koenig wrote:
> > Ron Shepard <nos...@nowhere.org> schrieb:
> [...]
> > As a user, I would like to have something like this, but as somebody
> > who could have a hand implementing this, I would need clear rules
> > on what to do :-)
> >
> > Consider a file containing
> >
> > 1 2 3 a
> >
> > and you would like to read in an allocatable integer array. What
> > would you propose?
>
> I would suggest starting with something simple, just reading one array
> at a time, or maybe requiring the array to be the last item in the i/o
> list.

Oh no, groan.

Then if there is programmer demand, it could be extended in the
> future to allow more extensive parsing of the input data as would be
> required for this example.
>
> If this were done within namelist i/o, then that parsing is already done
> for fixed-length arrays,

and not for dynamic arrays?

> so it seems like a small step to allow it also
> for allocatable arrays.
>
> > Now, consider a file containing
> >
> > 1 2 3
> > 4 5 6
> >
> >
> > What should the program read? Three elements or six?
>
> That is a good question. I would vote to read until a "/" is found or
> until an end-of-file is encountered.

That is ambiguous. see my previous post.

[robin....@gmail.com]

On Monday, January 13, 2020 at 6:24:01 AM UTC+11, Ron Shepard wrote:
> On 1/12/20 7:09 AM, spectrum wrote:
> > I would probably expect that
> >
> > read(unit, *) z
> >
> > corresponds to
> >
> > read(unit, *) z % n, z % a(:)
> >
> > (i.e., sequential read for each components, as declared in the type).
> >
> > So, I am wondering why the former is currently illegal (and needs a custom I/O routine).
> > Possibly, is it to avoid some complexity due to possible pointer components, or
> > OO features like inheritance...?
>
> When the array is not allocatable, then the standard requires the
> components to be read (and written) in the order they are declared. What
> I am proposing is that this rule be extended also to allocatable arrays
> that have been allocated.

Doesn't Fortran do this already?

[ga...@u.washington.edu]

On Sunday, January 12, 2020 at 4:48:18 PM UTC-8, FortranFan wrote:

(snip)

> and take note of the second most popular item in the user feedback
> to WG5 survey, "Automatic Allocation on READ into ALLOCATABLE
> character or array," with a Score of 5.04 that sounds quite similar
> to what @Ron Shepard is suggesting in this thread and which is also
> what @Ron Shepard has suggested often previously on
> comp.lang.fortran (CLF) forum.

Fancy input routines are more usual with interpreted languages
like Mathematic and R. Consider the R routine read.table().

read.table(file, header = FALSE, sep = "", quote = "\"'",
dec = ".", numerals = c("allow.loss", "warn.loss", "no.loss"),
row.names, col.names, as.is = !stringsAsFactors,
na.strings = "NA", colClasses = NA, nrows = -1,
skip = 0, check.names = TRUE, fill = !blank.lines.skip,
strip.white = FALSE, blank.lines.skip = TRUE,
comment.char = "#",
allowEscapes = FALSE, flush = FALSE,
stringsAsFactors = default.stringsAsFactors(),
fileEncoding = "", encoding = "unknown", text, skipNul = FALSE)


It has a fair number of parameters to specify the options needed for
different file formats. One common format has row and column
headers, which it knows how to read and create a data frame from.

If you want just the data, you use as.matrix() to convert to
a matrix without all the extra stuff for a data frame.

Note, though, that R has dynamic typing such that array elements can
be numeric, string, or maybe a few more types. Fortran doesn't do that,
so you would have to do something different. You could read into a
matrix of CHARACTER variables, and later convert to numeric.

But okay, in the more usual case you just want to read in a matrix
of unknown rows and columns, hopefully with the same number of
items in each row. Options like blank, comma, or tab separated
might be available.

R also has read.csv and read.xls, I presume everyone knows what
the latter does. That might be useful for some users.

[Ron Shepard]

On 1/12/20 7:23 PM, robin....@gmail.com wrote:
>> In practice the
>> input might also be something like
>>
>> 1, 2, 3, 4 /
> As I said before, this is ambiguous.

How so?

$.02 -Ron Shepard

[Ron Shepard]

On 1/12/20 7:29 PM, robin....@gmail.com wrote:
>> When the array is not allocatable, then the standard requires the
>> components to be read (and written) in the order they are declared. What
>> I am proposing is that this rule be extended also to allocatable arrays
>> that have been allocated.
> Doesn't Fortran do this already?

No. I think it should, but it doesn't.

$.02 -Ron Shepard

[robin....@gmail.com]

On Monday, January 13, 2020 at 8:38:18 PM UTC+11, Ron Shepard wrote:

> On 1/12/20 7:29 PM, r......@gmail.com wrote:
> >> When the array is not allocatable, then the standard requires the
> >> components to be read (and written) in the order they are declared. What
> >> I am proposing is that this rule be extended also to allocatable arrays
> >> that have been allocated.
> > Doesn't Fortran do this already?
>
> No. I think it should, but it doesn't.

It does already. Try running:-

real, allocatable :: a(:)
allocate (a(1:2))

read (*, *) a

print *, a

end

[robin....@gmail.com]

On Monday, January 13, 2020 at 8:29:01 PM UTC+11, Ron Shepard wrote:

> On 1/12/20 7:23 PM, r......@gmail.com wrote:
> >> In practice the
> >> input might also be something like
> >>
> >> 1, 2, 3, 4 /
> > As I said before, this is ambiguous.
>
> How so?

Read my earlier reply in the same post and which you have elided.

[Ron Shepard]

This discussion is about allocatable components of derived types. I
already gave code examples with both fixed and allocatable components
that demonstrate the difference.

$.02 -Ron Shepard

[Ron Shepard]

I did read your other reply, and you were not explicit there either.
There should be no need for me to ask the same question twice. But I
guess there is.

In the context that I described the allocate-on-read behavior, how is
the above input ambiguous? Please be explicit. Show your work. Give an
example.

$.02 -Ron Shepard

[Ian Harvey]

On 12/01/2020 11:04 pm, spectrum wrote:
> Because reading data is more complicated, I guess it is not surprising that derived-type
> input is not provided by default. But, my (bigger) question is why derived-type output
> is also made illegal by default...
>
> For example, in the following code, I can print / read 'f1' (which has no allocatable
> components) via default list-directed I/O, while cannot do a similar thing for 'f2'
> (which has an allocatable component), even if 'f2 % a' is allocated beforehand.
>
> program main
> implicit none
>
> type Foo1_t
> integer :: n = 1
> integer :: a( 2 ) = [10, 20]
> endtype
>
> type Foo2_t
> integer :: n = 1
> integer, allocatable :: a(:) !! = [30, 40] (cannot be assigned here)
> endtype
>
> type(Foo1_t) :: f1
> type(Foo2_t) :: f2
>
> f2 % a = [30, 40] !! or, allocate( f2 % a, source = [30, 40] )
>
> !! Print components.
>
> print *, "f1 = ", f1 !! this works
>
> !-- print *, "f2 = ", f2 !! error
> print *, "f2 = ", f2% n, f2% a !! so print each component

From the point of view of the language, the value of an object of type
Foo1_t (i.e. f1) is just the combination of the values of its components
- it is equivalent to f1%n and f1%a.

But for a type with allocatable components such as Foo2_t, the value of
an object of the type (i.e. f2) also includes the allocation status of
any allocatable component and its bounds (and dynamic type and type
parameters, if relevant). The value of f2 is more than f2%n and f2%a,
i.e. it includes the concept of ALLOCATED(f2%a), and if f2%a is
allocated, LBOUND(f2%a) and UBOUND(f2%a). The value of f2 is also
considered well defined even if f2%a is not allocated.

Don't take my word for it - you can read it for yourself in section
7.5.8 of F2018. Been the same since F2003, when allocatable components
first rocked up.

So what you propose is inconsistent with a fundamental concept in the
language.

Languages invariably have inconsistencies in them, the rules around
around I/O of objects of type with allocatable components could be
changed to permit what you propose (or changed to permit the "full"
definition of the value of the type to be written out, which appeals
more to me - but as I said upthread - what do you do then about
polymorphic things or pointers). But inconsistencies do come with some
cost in terms of language complexity.

For further contrast, consider the fragment:

type Foo3_t(len)
integer, len :: len
integer :: n = 1
integer :: a(len)
end type

type(Foo3_t(2)) :: f3
f3%a = [50, 60]
print *, f3

The type and type parameters of an object determine the nature of the
values that an object can have, but they are not considered part of the
value. The value of f3 is again just the combination of f3%n and f3%a.

There's no problem reading f3 either...

Why is component `a` allocatable in your Foo2_t type? Does it need to be?



> ...

[Ian Harvey]

I don't need an excuse - I don't have to implement the feature.

I am trying to highlight that quite a bit more thought is required for
specification of this feature, than perhaps it might first seem.
Perhaps someone can come up with a specification that works well - in
which case that's great. But most of the ideas thrown out there in
previous discussions about this sort of feature just don't seem to fit
very well.

Assignment of objects of derived type preserves value as per the
definition of value of an object of derived type - it is fine to have an
unallocated component (the "unallocated" status is copied across), if
the component is an array and is allocated then its bounds are also
copied. But what's being proposed here for I/O is inconsistent with the
definition of the value of an object of derived type.

I don't follow the array operator bit.

> [...]
>> Out of the blocks - you are saying it should work one way, I'm saying
>> it should work some other way, and I think that is very representative
>> of the situation, so rather than pick one or the other the standard
>> just says "here are some tools (UDDTIO), but otherwise sort it out
>> yourselves".
>
> UDDTIO was slow to be implemented, so I do not have much experience with
> it. But from what I've read, it seems the same as writing your own
> subroutine to write out the data, and then invoking it through a
> read/write statement rather than a call statement.
>
> Is that right, is it just a thin layer of syntactic sugar?

Yes - you write a subroutine to define the input or output. No - it is
not just a thin layer of syntactic sugar. Child input output statements
behave differently to parent input/output statements - you cannot
emulate those differences outside of UDDTIO. Integration with things
like namelist input/output is also not particularly "thin".

> In any case, I'm not suggesting that UDDTIO be removed from the
> language, but rather than the obvious default behavior be invoked when
> doing i/o on derived types. When you delete an old component or add a
> new component to a derived type, then the obvious thing should happen
> when a read/write is executed, you should not need to dig into a
> user-level subprogram and manually synchronize with the new definition.
> If you want to do something that is not obvious, then UDDTIO is there
> for that.

My view of the "obvious default" is strongly towards the formal
definition of the value of the object.

Anyway as alluded to up-thread and in the recent post, just use a length
type parameter, and all's peachy (well, unless you need to MOVE_ALLOC
the component).

There are multiple storage-like resources that might be limiting - it is
not clear to me how to specify something that is both portable and
practically useful.

[Ron Shepard]

On 1/14/20 6:06 AM, Ian Harvey wrote:
> Why is component `a` allocatable in your Foo2_t type?  Does it need to be?

Yes, it sometimes does. It is not uncommon to need an allocatable
component in an array. For example, it might be an application in which
the size changes during the program execution, a situation that does not
apply to PDTs.

In general, PDTs not really a substitute for allocatable arrays, so your
question seems to be somewhat odd in that respect.

In any case, it would still be useful and simple if it were possible to
write out derived types with allocatable components the same way that
one can write out derived types with fixed length arrays. In my
proposal, it would only apply to arrays that have been allocated and
have defined values, so your comments about allocation status would not
apply directly. I do not see any ambiguity.

$.02 -Ron Shepard

[Ron Shepard]

Can you explain why it is inconsistent or ambiguous?

$.02 -Ron Shepard

[ga...@u.washington.edu]

On Sunday, January 12, 2020 at 11:24:01 AM UTC-8, Ron Shepard wrote:

(snip)

> When the array is not allocatable, then the standard requires the
> components to be read (and written) in the order they are declared. What
> I am proposing is that this rule be extended also to allocatable arrays
> that have been allocated.

> Just to preempt comments about storage order, the standard already
> allows the storage order within a derived type to be rearranged by the
> processor, so regardless of whether or not the components are
> allocatable, the processor must be prepared to rearrange the output from
> storage order into the declared order when doing i/o. Extending the
> capability to allocatable components would not add any additional burden
> to the i/o library in this respect.

I might be remembering this from some years ago, and maybe it doesn't
make sense anymore, or maybe never did.

Consider an UNFORMATTED I/O statement, which sort of by definition
reads and writes the bytes as they are stored in memory. Most of the
time that is fine, but in the case of an allocatable array, there
is a descriptor with, more or less, the bounds and address of the
actual allocated memory.

Note, for example that a BIND(C) structure should match up with a
C struct, and C will just write out the bytes. (I don't know that
there is a way to interoperate allocatable structure members with
C, though.)

That doesn't really apply in the case of formatted I/O, though,
but as well as I know it, it is excluded for consistency.

I don't know about possible reordering, or for that matter, padding
of structures, but in any case it should be possible to WRITE one
out and READ it back in.

And what do you do about pointers in structures in I/O statements?

[Ron Shepard]

On 1/16/20 1:14 AM, ga...@u.washington.edu wrote:
> And what do you do about pointers in structures in I/O statements?

I think pointers have always been problematic with i/o. Sometimes it
makes sense to write out memory addresses and then read them back in
during the same execution. But between two different executions of a
program, the memory locations in one have nothing to do with the memory
locations in the other, so it just doesn't make sense. And even within
the same execution, the programmer somehow must know that the memory
addresses have remained well defined in between the write and the
subsequent read. That seems difficult to define and enforce. The other
parts of the pointers, the ranks and array bounds, maybe that does make
sense, but the actual memory addresses don't.

In some ways allocatable arrays are like tame pointers, at least in the
low level implementation, so maybe some of this carries over. But in
most other ways, allocatables are more like regular arrays with the
extra feature that they can be resized at runtime. I guess allocatable
scalars fit in this too, in particular allocatable character strings. In
this sense, it seems more consistent with the language, once they are
allocated, to treat them like regular arrays within a structure,
including i/o, intrinsic assignment, and so on.

Just to give an example, suppose you have a linked list implemented as a
derived type with allocatable (rather than pointer) links. Is it a well
defined operation to write and read that linked list with a single i/o
statement? Or is this something that the user should be required to
handle at a low level (say with UDDTIO)?

$.02 -Ron Shepard

[Ron Shepard]

On 1/16/20 1:14 AM, ga...@u.washington.edu wrote:

> Consider an UNFORMATTED I/O statement, which sort of by definition
> reads and writes the bytes as they are stored in memory. Most of the
> time that is fine, but in the case of an allocatable array, there
> is a descriptor with, more or less, the bounds and address of the
> actual allocated memory.

All arrays have descriptors with memory location, rank, and bounds
information, including regular nonallocatable, nonpointer arrays, and
the programmer can access that information with size(), lbound(),
ubound(), etc. intrinsics for all types of arrays. When you do i/o on
regular arrays, that descriptor information is not written out. It is up
to the programmer to ensure that what was written out is consistent with
what is later read in. I think there would be no inconsistency to make
that the same requirement on the programmer for allocatable arrays.

$.02 -Ron Shepard

[ga...@u.washington.edu]

On Thursday, January 16, 2020 at 10:12:44 AM UTC-8, Ron Shepard wrote:
(I wrote)

> > And what do you do about pointers in structures in I/O statements?

> I think pointers have always been problematic with i/o. Sometimes it
> makes sense to write out memory addresses and then read them back in
> during the same execution. But between two different executions of a
> program, the memory locations in one have nothing to do with the memory
> locations in the other, so it just doesn't make sense. And even within
> the same execution, the programmer somehow must know that the memory
> addresses have remained well defined in between the write and the
> subsequent read. That seems difficult to define and enforce. The other
> parts of the pointers, the ranks and array bounds, maybe that does make
> sense, but the actual memory addresses don't.

In C if you write out a struct array:

fwrite(s, sizeof(s), n, stdout);

it writes out the (C definition of) bytes it contains.

fwrite gets a pointer and length and writes out bytes.

The definition of Fortran UNFORMATTED tends to be that it
does something similar, but from the time before pointers
and allocatables.

> In some ways allocatable arrays are like tame pointers, at least in the
> low level implementation, so maybe some of this carries over. But in
> most other ways, allocatables are more like regular arrays with the
> extra feature that they can be resized at runtime. I guess allocatable
> scalars fit in this too, in particular allocatable character strings. In
> this sense, it seems more consistent with the language, once they are
> allocated, to treat them like regular arrays within a structure,
> including i/o, intrinsic assignment, and so on.

Consider a structure with one member, an allocatable array.
It doesn't seem so strange to just write out the array.
(I think for now consider FORMATTED output.)

Now how about a struct containing two allocatable arrays.
Just write them out one after another, but now it is already
confusing. When you read them, you don't know where one ends
and the next one starts.

Maybe for list-directed, it should write out square brackets
around the array, which would then tell you where one ends and
the next begins. For explicit format, you have to supply the
brackets, but you don't know where to put them.

I don't know UDDTIO very well, but it seems to me that it
is the solution to this problem. You define where you want the
square brackets, or otherwise indicate boundaries.

> Just to give an example, suppose you have a linked list implemented as a
> derived type with allocatable (rather than pointer) links. Is it a well
> defined operation to write and read that linked list with a single i/o
> statement? Or is this something that the user should be required to
> handle at a low level (say with UDDTIO)?

You could define it to write out the elements, but there are no
visible boundaries, so it is pretty much not human readable.
(Or machine readable, either.)

I believe with allocatable, you can't make a circular linked list,
but with pointers, you can. How do you print one of those?

[ga...@u.washington.edu]

On Thursday, January 16, 2020 at 10:23:37 AM UTC-8, Ron Shepard wrote:
(I wrote)

The question is the number of indirections.

In most cases of pointers and allocatables, the operation is defined
to use the referenced data, not the allocatable data structure itself
or the pointer itself. (Except for pointer assignment.)

I think that means that we can consider how it treats assignment,
to simpify things.

Say we have two structures, A and B, and we:

B=A

we expect all the contents of A to transfer to B. If A contains
pointers, maybe (having never tried) this, the pointer is copied.

What does it do if I contains allocatables? You can't copy an
allocatable in the same way as a pointer. Should (does?) it allocate
(on assignment) a new array and copy the contents?

Does it do this recursively if the new array is a structure
containing allocatables?

[Ron Shepard]

On 1/16/20 2:10 PM, ga...@u.washington.edu wrote:
> Say we have two structures, A and B, and we:
>
> B=A
>
> we expect all the contents of A to transfer to B. If A contains
> pointers, maybe (having never tried) this, the pointer is copied.
>
> What does it do if I contains allocatables? You can't copy an
> allocatable in the same way as a pointer. Should (does?) it allocate
> (on assignment) a new array and copy the contents?

Yes, the same as simple assignment of an allocatable array. Pointers are
different, only the metadata is copied, and it points to the same target
as in a pointer assignment, =>.

> Does it do this recursively if the new array is a structure
> containing allocatables?

Yes, the allocation and assignment works all the way down the chain. I
think i/o with nested allocatables could work the same way (i.e. the
same as nested derived types with regular arrays).

$.02 -Ron Shepard

[Ron Shepard]

On 1/16/20 1:56 PM, ga...@u.washington.edu wrote:
[...]

> The definition of Fortran UNFORMATTED tends to be that it
> does something similar, but from the time before pointers
> and allocatables.

In fortran, the structure information is never written out, whether it
is formatted or unformatted i/o. It is just the data, written in
whatever order the programmer specifies using implied do-loops or array
stride notation (or storage order, by default).

>> In some ways allocatable arrays are like tame pointers, at least in the
>> low level implementation, so maybe some of this carries over. But in
>> most other ways, allocatables are more like regular arrays with the
>> extra feature that they can be resized at runtime. I guess allocatable
>> scalars fit in this too, in particular allocatable character strings. In
>> this sense, it seems more consistent with the language, once they are
>> allocated, to treat them like regular arrays within a structure,
>> including i/o, intrinsic assignment, and so on.
>
> Consider a structure with one member, an allocatable array.
> It doesn't seem so strange to just write out the array.
> (I think for now consider FORMATTED output.)
>
> Now how about a struct containing two allocatable arrays.
> Just write them out one after another, but now it is already
> confusing. When you read them, you don't know where one ends
> and the next one starts.

In my proposed implementation, the programmer must allocate the array
components correctly before the read, so it is the same as a regular
array after that point.

> Maybe for list-directed, it should write out square brackets
> around the array, which would then tell you where one ends and
> the next begins. For explicit format, you have to supply the
> brackets, but you don't know where to put them.

None of that should be necessary. The sizes are all known when the read
statement is executed, so it would be the same as a regular array.

> I don't know UDDTIO very well, but it seems to me that it
> is the solution to this problem. You define where you want the
> square brackets, or otherwise indicate boundaries.

Yes, then you can write out self defining formats, even something like
hdf or xml if you want. But that's not what I'm proposing for this
simple case.

$.02 -Ron Shepard

[ga...@u.washington.edu]

On Thursday, January 16, 2020 at 9:59:40 PM UTC-8, Ron Shepard wrote:

(snip, I wrote)

> > Consider a structure with one member, an allocatable array.
> > It doesn't seem so strange to just write out the array.
> > (I think for now consider FORMATTED output.)

> > Now how about a struct containing two allocatable arrays.
> > Just write them out one after another, but now it is already
> > confusing. When you read them, you don't know where one ends
> > and the next one starts.

> In my proposed implementation, the programmer must allocate the array
> components correctly before the read, so it is the same as a regular
> array after that point.

OK, there is some discussion above for allocate on read, which
would be nice in some cases, but isn't so easy.

> > Maybe for list-directed, it should write out square brackets
> > around the array, which would then tell you where one ends and
> > the next begins. For explicit format, you have to supply the
> > brackets, but you don't know where to put them.

> None of that should be necessary. The sizes are all known when the read
> statement is executed, so it would be the same as a regular array.

I think I agree, but consider in list-directed output of arrays.
One can:

PRINT *,'[', A, '], [', B, ']'

such that one can tell, reading the printed output, where one
array ends and the next starts. I don't know if anyone does that.

> > I don't know UDDTIO very well, but it seems to me that it
> > is the solution to this problem. You define where you want the
> > square brackets, or otherwise indicate boundaries.

> Yes, then you can write out self defining formats, even something like
> hdf or xml if you want. But that's not what I'm proposing for this
> simple case.

It seemed that it might be nice, but it doesn't follow tradition.

OK, just write out the structure members.

[Ian Harvey]

On 2020-01-16 15:08, Ron Shepard wrote:
> On 1/14/20 6:06 AM, Ian Harvey wrote:
>> Why is component `a` allocatable in your Foo2_t type?  Does it need to
>> be?
>
> Yes, it sometimes does. It is not uncommon to need an allocatable
> component in an array. For example, it might be an application in which
> the size changes during the program execution, a situation that does not
> apply to PDTs.

Deferred type parameters can vary at runtime. What allocatable
components offer is that you can vary the allocation status of a
component without having to redefine all other components. Allocatable
components also offer the possibility of:

- movement of allocation;
- polymorphism (but that is not being addressed by this input/output
proposal);
- not being allocated (but that is not being considered by this
input/output proposal);
- parent object value incorporating array bounds information (but that
is not being considered by this input/output proposal).

> In general, PDTs not really a substitute for allocatable arrays, so your
> question seems to be somewhat odd in that respect.

They can be used for a reasonable subset of current uses of allocatable
components. I think that subset would significantly overlap with use
cases that are also suitable for this input/output proposal.

> In any case, it would still be useful and simple if it were possible to
> write out derived types with allocatable components the same way that
> one can write out derived types with fixed length arrays. In my
> proposal, it would only apply to arrays that have been allocated and
> have defined values, so your comments about allocation status would not
> apply directly. I do not see any ambiguity.

I don't see ambiguity - but I do see inconsistency.

[ga...@u.washington.edu]

On Friday, January 17, 2020 at 11:39:23 PM UTC-8, Ian Harvey wrote:

> Deferred type parameters can vary at runtime. What allocatable
> components offer is that you can vary the allocation status of a
> component without having to redefine all other components. Allocatable
> components also offer the possibility of:

> - movement of allocation;
> - polymorphism (but that is not being addressed by this input/output
> proposal);
> - not being allocated (but that is not being considered by this
> input/output proposal);
> - parent object value incorporating array bounds information (but that
> is not being considered by this input/output proposal).

I now realize that with MOVE_ALLOC you can make a circular linked list
out of allocatables. Be careful with them.

[Ron Shepard]

On 1/18/20 1:39 AM, Ian Harvey wrote:
> They can be used for a reasonable subset of current uses of allocatable
> components.  I think that subset would significantly overlap with use
> cases that are also suitable for this input/output proposal.

The way PDTs are defined, can you now do I/O on them without UDDTIO?
Both formatted and unformatted?

>
>> In any case, it would still be useful and simple if it were possible
>> to write out derived types with allocatable components the same way
>> that one can write out derived types with fixed length arrays. In my
>> proposal, it would only apply to arrays that have been allocated and
>> have defined values, so your comments about allocation status would
>> not apply directly. I do not see any ambiguity.
>
> I don't see ambiguity - but I do see inconsistency.

I guess this depends in part on the answer to the above question. If you
can do I/O on a parameterized derived type, but you cannot do I/O on a
derived type with an allocatable component, then my proposal would
actually make things more consistent, not less. It would treat derived
types with fixed length arrays, parameterized derived types, and derived
types with allocatable components all equally.

$.02 -Ron Shepard

[Ron Shepard]

I have not thought about doing this, but I'm not sure this is correct.
The from argument, which would be the head of the list before the circle
is closed, is deallocated when move_alloc() is invoked. It seems like
that would preclude creating a circular linked list. Or maybe this is an
exception to the deallocation side effect?

$.02 -Ron Shepard

[ga...@u.washington.edu]

On Saturday, January 18, 2020 at 12:06:20 AM UTC-8, Ron Shepard wrote:

(snip, I wrote)

> > I now realize that with MOVE_ALLOC you can make a circular linked list
> > out of allocatables. Be careful with them.

> I have not thought about doing this, but I'm not sure this is correct.
> The from argument, which would be the head of the list before the circle
> is closed, is deallocated when move_alloc() is invoked. It seems like
> that would preclude creating a circular linked list. Or maybe this is an
> exception to the deallocation side effect?

Yes, you would need a pointer to one element of the list.

And yes, it probably isn't a good idea.

The deallocation side effect is necessary because there can be
only one actual allocatable, but there can be many pointers to it.
Only the real one can be deallocated or passed as an actual argument
to an allocatable dummy.

[Ian Harvey]

On 18/01/2020 5:27 pm, Ron Shepard wrote:
> On 1/18/20 1:39 AM, Ian Harvey wrote:
>> They can be used for a reasonable subset of current uses of
>> allocatable components.  I think that subset would significantly
>> overlap with use cases that are also suitable for this input/output
>> proposal.
>
> The way PDTs are defined, can you now do I/O on them without UDDTIO?
> Both formatted and unformatted?

Yes, if the PDT does not contain an allocatable or pointer component.

>>> In any case, it would still be useful and simple if it were possible
>>> to write out derived types with allocatable components the same way
>>> that one can write out derived types with fixed length arrays. In my
>>> proposal, it would only apply to arrays that have been allocated and
>>> have defined values, so your comments about allocation status would
>>> not apply directly. I do not see any ambiguity.
>>
>> I don't see ambiguity - but I do see inconsistency.
>
> I guess this depends in part on the answer to the above question. If you
> can do I/O on a parameterized derived type, but you cannot do I/O on a
> derived type with an allocatable component, then my proposal would
> actually make things more consistent, not less. It would treat derived
> types with fixed length arrays, parameterized derived types, and derived
> types with allocatable components all equally.

I have a half prepared response to a query elsethread about my
perception of inconsistency that I will finish off next weekend to
address this.

[spectrum]

On Tuesday, January 14, 2020 at 9:06:03 PM UTC+9, Ian Harvey wrote:
> (... a lot of stuff... )

> Why is component `a` allocatable in your Foo2_t type? Does it need to be?

Though other discussions/talks are on-going here, I will keep my post minimal,
but the reason why I do not use PDT is rather clear... That is, I will use
only "stable" language features for production codes, partly because of
the availability of compilers, and also for fear of possible bugs for new features.
But, since the debugging of compilers are also important, I try to use
new features for relatively smaller programs (for which I can predict the result
and so notice wrong results relatively easily). For example, I've just started
to use the ASSOCIATE construct only very recently (though this is also partly because
I recently decided to put compilers not supporting ASSOCIATE "on-hold" ...)

Another reason why I prefer allocatables may be simply that it can be a large array.
RE the output of DT objects containing allocatables, I will write more in a different thread
(so that the topic will not be mixed with the input thing).