Filling part of an input array via namelist

[spectrum]

Although namelist in Fortran is often thought to be an archaic feature
(and something that should not be used for portability?), I use it very
often for setting parameters from a file. In a recent case, I used
an array for input and tried to fill part of it, like:

&param_inp
...
arr(1:3) = 1.0 2.0 3.0
...
/

Here, the array "arr" is an allocatable array that was allocated with
a sufficiently large size (e.g. arr(1:100)) before namelist read.
The above code worked as expected, but it also seems the following
patterns also work in the same way:

arr(:) = 1.0 2.0 3.0

or

arr = 1.0 2.0 3.0

Does this mean that, if the number of entities on the right-hand side
of assignment is smaller than size(arr), it is okay to
assume that the remaining part of "arr" (e.g. arr(4:)) in the above case
is kept intact in the calling routine?

[Ron Shepard]

On 6/24/21 5:32 PM, spectrum wrote:
> Although namelist in Fortran is often thought to be an archaic feature

> (and something that should not be used for portability?), [...]

Where would you get that idea?

[...]

> Does this mean that, if the number of entities on the right-hand side
> of assignment is smaller than size(arr), it is okay to
> assume that the remaining part of "arr" (e.g. arr(4:)) in the above case
> is kept intact in the calling routine?

I believe that is correct. Also, you can fill in specific array elements
and use stride notation.

arr(1:3)= 1. 2. 3.
arr(5) = 5.
arr(7:8) = 7. 8.
arr(10:14:2) = 10. 12. 14.

and so on.

You can also use derived types with members specified in the obvious
way. As far as I know, all of this is standard and portable.

However, you cannot use commas to skip entries, or use repeat counts.
Those are features of list-directed i/o, but not namelist.

$.02 -Ron Shepard

[FortranFan]

On Friday, June 25, 2021 at 12:41:18 AM UTC-4, Ron Shepard wrote:

> ..
> However, you cannot use .. repeat counts.
> Those are features of list-directed i/o, but not namelist. ..

By "repeat counts", if the following is meant, then they are indeed allowed:

integer :: a(3)
namelist / dat / a
character(len=:), allocatable :: inp
inp = "&dat a=1,2*2 /" !<-- notice the n*v entry where n* serves as a repeat count
read( inp, nml=dat )
print *, "a = ", a, "; expected is 1, 2, 2 "
end

C:\Temp>p.exe
a = 1 2 2 ; expected is 1, 2, 2

[spectrum]

On Friday, June 25, 2021 at 1:41:18 PM UTC+9, Ron Shepard wrote:
> I believe that is correct. Also, you can fill in specific array elements
> and use stride notation.
>
> arr(1:3)= 1. 2. 3.
> arr(5) = 5.
> arr(7:8) = 7. 8.
> arr(10:14:2) = 10. 12. 14.
>
> and so on.

Thanks very much for your info! It is reassuring to know that
the above usage is valid, because I've already used it in a routine way
in my codes (based on some small test codes). Up to now,
those codes are working just fine and I experienced no problem,
but I was not sure whether this is really a valid usage, so asked anyway above.

On 6/24/21 5:32 PM, spectrum wrote:
>> Although namelist in Fortran is often thought to be an archaic feature
>> (and something that should not be used for portability?), [...]

> Where would you get that idea?

I remember I saw at least 3 or 4 times in the Stackoverflow site
that someone wrote a comment like "namelist is not portable".
So I imagined the actual implementations in various compilers
vary somewhat (due to vendor extensions etc), so there may have been
problematic cases for "portability" (not in the sense of the standards,
but for using one person's namelist in different environments).

RE general opinions about namelist, I remember I saw several related pages,
but I don't remember each site... One site I remember is
the following one (which explains disadvantages of namelist
as compared to other formats):
http://degenerateconic.com/fortran-configuration-file-formats/

Indeed, I also feel formats like JSON or TOML are very useful
(and can be used in different languages on the same footing),
so in that sense it may be reasonable to make the input format "common"
for one's application (rather than using language-specific formats).

> You can also use derived types with members specified in the obvious
> way. As far as I know, all of this is standard and portable.

Yeah, this is exactly what I'm using :)
I define a parameter type (like Param_t), which contains only parameters
for the calculation/simulation, and just read it from namelist like

type(Param_t) :: par !! dummy argument
namelist /param_inp/ par
read(..., nml= param_Inp )
...

This is very convenient because I don't have to specify the components
explicitly in the definition of namelist, while allowing specific inputs like

&param_inp
...
par%arr(1:3) = 1.0 2.0 3.0
...
/

[Steve Lionel]

On 6/25/2021 12:41 AM, Ron Shepard wrote:
> However, you cannot use commas to skip entries, or use repeat counts.
> Those are features of list-directed i/o, but not namelist.

This is not true - both of those are specified by the standard as valid
in namelist input.

"13.11.3.3 Namelist input values
Each value is either a null value (13.11.3.4), c, r*c, or r*, where c is
a literal constant, optionally signed if integer or real, and r is an
unsigned, nonzero, integer literal constant. A kind type parameter shall
not be specified for c or r. The constant c is interpreted as though it
had the same kind type parameter as the corresponding effective item.
The r*c form is equivalent to r successive appearances of the constant
c, and the r* form is equivalent to r successive null values. Neither of
these forms shall contain embedded blanks, except where permitted within
the constant c."

13.11.3.4 Null values
A null value is specified by
• the r* form,
• blanks between two consecutive nonblank value separators following
an equals,
• a value separator that is the first nonblank character following an
equals, or
• two consecutive nonblank value separators.

A null value has no effect on the definition status of the corresponding
input list item. If the namelist group object list item is defined, it
retains its previous value; if it is undefined, it remains undefined. A
null value shall not be used as either the real or imaginary part of a
complex constant, but a single null value may represent an entire
complex constant."

--
Steve Lionel
ISO/IEC JTC1/SC22/WG5 (Fortran) Convenor
Retired Intel Fortran developer/support
Email: firstname at firstnamelastname dot com
Twitter: @DoctorFortran
LinkedIn: https://www.linkedin.com/in/stevelionel
Blog: https://stevelionel.com/drfortran
WG5: https://wg5-fortran.org

[gah4]

On Friday, June 25, 2021 at 4:43:27 PM UTC-7, spectrum wrote:

(snip)

> I remember I saw at least 3 or 4 times in the Stackoverflow site
> that someone wrote a comment like "namelist is not portable".
> So I imagined the actual implementations in various compilers
> vary somewhat (due to vendor extensions etc), so there may have been
> problematic cases for "portability" (not in the sense of the standards,
> but for using one person's namelist in different environments).

Yes NAMELIST is not very portable if you are using 50 year old
compilers.

I believe NAMELIST goes back to the beginning of IBM Fortran IV,
from before the Fortran 66 standard. Many IBM features didn't get into
the standard, but were instead extensions, and then similar features were
extensions in other competing compilers. So, in the 1960's and 1970's
it wasn't surprising to find different implementations. (IBM was always
good at using shading in their manuals to mark extensions to the standard.)

But since it was added to the standard, it should be implemented in
the standard way.

Among others, the IBM form uses &END to end the input data stream,
where the standard now uses /. I believe some other impelmentations
uses other ending characters.

Well, it wasn't yet in Fortran 77, and I am not sure now which version it
did first appear.

[Thomas Koenig]

gah4 <ga...@u.washington.edu> schrieb:

> I believe NAMELIST goes back to the beginning of IBM Fortran IV,
> from before the Fortran 66 standard. Many IBM features didn't get into
> the standard, but were instead extensions, and then similar features were
> extensions in other competing compilers. So, in the 1960's and 1970's
> it wasn't surprising to find different implementations. (IBM was always
> good at using shading in their manuals to mark extensions to the standard.)
>
> But since it was added to the standard, it should be implemented in
> the standard way.

There are currently 106 test cases in the gfortran testsuite
whose name contains "namelist"

This is beaten by "class" (174 test cases), "typebound" (124 test
cases) and "proc_ptr" (123 test cases) but is less than "fmt"
(93 test cases).

By this metric at least, namelist is a feature that is _very_ hard to
get right.

[Ron Shepard]

Oops, I stand corrected. I thought these were extensions.

As for namelist being nonportable, I did not mention namelist output.
There, the output format itself is not specified fully by the standard.
The programmer has little control over things like line lengths,
upper/lower case of the variable names (namelist output always uses
upper case), spaces surrounding equals signs, and where line breaks
occur (the latter is the same situation as with list directed output).
However, for the most part, whatever is written by namelist output
should be able to be read by namelist input. I think there are some odd
cases involving character strings with embedded quotes where input and
output forms might not match, but like the above repeat count issue, I
might not be up to date on what is standard and what is an extension.

I almost never use namelist output for this reason. It is relatively
simple to write your own output records with normal formatted output
where you have complete control over the output format. I also don't use
namelist output for a different reason: it writes out EVERYTHING. The
way I use namelist, my input files are always relatively short,
specifying only a small subset of the possible values, and those are
always the ones I am interested in also on output. This has nothing to
do with the language, it is the way I use namelist in my programs that
results in this feature.

There are two other nice features of namelist input. One is that you can
embed comments. They are delimited with ! exclamation marks, the same
way source code comments are embedded in source lines. The other is that
you can specify an input item multiple times. The last specification
override any previous specifications. This is useful for temporarily
overriding some input value and then easily reverting back to the
previous value.

$.02 -Ron Shepard

[gah4]

On Sunday, June 27, 2021 at 4:00:20 AM UTC-7, Ron Shepard wrote:

(snip)

> I almost never use namelist output for this reason. It is relatively
> simple to write your own output records with normal formatted output
> where you have complete control over the output format. I also don't use
> namelist output for a different reason: it writes out EVERYTHING. The
> way I use namelist, my input files are always relatively short,
> specifying only a small subset of the possible values, and those are
> always the ones I am interested in also on output. This has nothing to
> do with the language, it is the way I use namelist in my programs that
> results in this feature.

Well, not quite everything.

(For comparison purposes only.)

The PL/I equivalent of namelist, data directed I/O, has an option to omit the
list of data items, in which case everything in scope is included.

I did use it once with the input form as part of a debugging system, which
would let me change variables at some point in a program. But it also
works for output, such that it will print all variables in scope.
(Maybe not statement label variables and some others that don't have
readable values.)

But yes, NAMELIST output could be useful for a program that wants to write
output, to be read in by a later program, but also human readable.
Otherwise, I usually use unformatted I/O.

[Robin Vowels]

On Monday, June 28, 2021 at 10:09:49 AM UTC+10, gah4 wrote:
> On Sunday, June 27, 2021 at 4:00:20 AM UTC-7, Ron Shepard wrote:
>
> (snip)
> > I almost never use namelist output for this reason. It is relatively
> > simple to write your own output records with normal formatted output
> > where you have complete control over the output format. I also don't use
> > namelist output for a different reason: it writes out EVERYTHING. The
> > way I use namelist, my input files are always relatively short,
> > specifying only a small subset of the possible values, and those are
> > always the ones I am interested in also on output. This has nothing to
> > do with the language, it is the way I use namelist in my programs that
> > results in this feature.
> Well, not quite everything.
>
> (For comparison purposes only.)
>
> The PL/I equivalent of namelist, data directed I/O, has an option to omit the
> list of data items, in which case everything in scope is included.
>
> I did use it once with the input form as part of a debugging system, which
> would let me change variables at some point in a program.

.
The simple form of input ( GET DATA; ) allows the input of one or more
values to be assigned to any variable in scope.
It allows the user to change the value of as little as one variable,
leaving all the others unchanged. ( e.g., the data stream is simply K = 5 ; )
If no input is required, only the semicolon is given to indicate a null list.

.
> But it also
> works for output, such that it will print all variables in scope.

.
That's very handy when there is a moderate number of variables in scope.

.
> (Maybe not statement label variables and some others that don't have
> readable values.)
>
> But yes, NAMELIST output could be useful for a program that wants to write
> output, to be read in by a later program, but also human readable.
> Otherwise, I usually use unformatted I/O.

.
PL/I allows the input of a data stream that has been output using
a PUT DATA statement.

[Steve Lionel]

On 6/26/2021 7:42 PM, gah4 wrote:
> I believe NAMELIST goes back to the beginning of IBM Fortran IV,
> from before the Fortran 66 standard. Many IBM features didn't get into
> the standard, but were instead extensions, and then similar features were
> extensions in other competing compilers. So, in the 1960's and 1970's
> it wasn't surprising to find different implementations. (IBM was always
> good at using shading in their manuals to mark extensions to the standard.)

NAMELIST was in IBM FORTRAN by the late 70s. It was one of my first
tasks at DEC to add NAMELIST, in a compatible way, to VAX FORTRAN.
NAMELIST did not get added to the standard until Fortran 90, but the
committee chose a slightly different syntax for the group start and end
delimiters, so we had to support both!

NAMELIST input, on its own, should be quite portable, as long as you
stick to the standard. (For example, don't type a real value when the
variable is integer, and don't mix LOGICAL with numeric, as DEC
implementations did.) NAMELIST output, just like list-directed, can be
very implementation-dependent.

[Gary Scott]

On 6/28/2021 7:50 AM, Steve Lionel wrote:
> On 6/26/2021 7:42 PM, gah4 wrote:
>> I believe NAMELIST goes back to the beginning of IBM Fortran IV,
>> from before the Fortran 66 standard.  Many IBM features didn't get into
>> the standard, but were instead extensions, and then similar features were
>> extensions in other competing compilers.  So, in the 1960's and 1970's
>> it wasn't surprising to find different implementations.   (IBM was always
>> good at using shading in their manuals to mark extensions to the
>> standard.)
>
> NAMELIST was in IBM FORTRAN by the late 70s. It was one of my first
> tasks at DEC to add NAMELIST, in a compatible way, to VAX FORTRAN.
> NAMELIST did not get added to the standard until Fortran 90, but the
> committee chose a slightly different syntax for the group start and end
> delimiters, so we had to support both!
>
> NAMELIST input, on its own, should be quite portable, as long as you
> stick to the standard. (For example, don't type a real value when the
> variable is integer, and don't mix LOGICAL with numeric, as DEC
> implementations did.) NAMELIST output, just like list-directed, can be
> very implementation-dependent.
>

That was one of the most annoying things about it. Should allow user to
specify an integer value for a real variable. I used to have to add
processing to allow that by marking up the input stream and applying it
to character variable and then read it again using namelist. I'd like
to see some added processing be added to report errors at least on read.
Maybe an array of integer return codes and character variable names.
This would allow it to be adapted for many parsing needs.

[Steve Lionel]

On 6/28/2021 9:57 AM, Gary Scott wrote:
> Should allow user to specify an integer value for a real variable.

You can do that. But specifying a real (for example, 3.0) for an integer
is not allowed.

"When the next effective item is of type real, the input form of the
input value is that of a numeric input field. A numeric input field is a
field suitable for F editing (13.7.2.3.2) that is assumed to have no
fractional digits unless a decimal symbol appears within the field."

[Robin Vowels]

On Monday, June 28, 2021 at 11:57:27 PM UTC+10, Gary Scott wrote:
> On 6/28/2021 7:50 AM, Steve Lionel wrote:
> > On 6/26/2021 7:42 PM, gah4 wrote:
> >> I believe NAMELIST goes back to the beginning of IBM Fortran IV,
> >> from before the Fortran 66 standard. Many IBM features didn't get into
> >> the standard, but were instead extensions, and then similar features were
> >> extensions in other competing compilers. So, in the 1960's and 1970's
> >> it wasn't surprising to find different implementations. (IBM was always
> >> good at using shading in their manuals to mark extensions to the
> >> standard.)
> >
> > NAMELIST was in IBM FORTRAN by the late 70s. It was one of my first
> > tasks at DEC to add NAMELIST, in a compatible way, to VAX FORTRAN.
> > NAMELIST did not get added to the standard until Fortran 90, but the
> > committee chose a slightly different syntax for the group start and end
> > delimiters, so we had to support both!
> >
> > NAMELIST input, on its own, should be quite portable, as long as you
> > stick to the standard. (For example, don't type a real value when the
> > variable is integer, and don't mix LOGICAL with numeric, as DEC
> > implementations did.) NAMELIST output, just like list-directed, can be
> > very implementation-dependent.
> >
> That was one of the most annoying things about it. Should allow user to
> specify an integer value for a real variable.

.
The equivalent in PL/I (with GET DATA statement) allows exactly that,
and has done so since 1966.
Being able to do this is even more important than would seem at first glance
because with the simplest form of statement ( GET DATA; ) without a list of variables,
the only indication of a variable name is in the data stream (e.g., K = 5; ).
The data type of the variable is not specified in the data stream, and
in general, the user does not need to know what the type of the variable is.

.
> I used to have to add
> processing to allow that by marking up the input stream and applying it
> to character variable and then read it again using namelist. I'd like
> to see some added processing be added to report errors at least on read.
> Maybe an array of integer return codes and character variable names.

.
Groan. It doesn't need any of that.
.
PL/I has been handing it automatically for 55+ years.

[Robin Vowels]

On Tuesday, June 29, 2021 at 2:33:28 AM UTC+10, Steve Lionel wrote:
> On 6/28/2021 9:57 AM, Gary Scott wrote:
> > Should allow user to specify an integer value for a real variable.
> You can do that. But specifying a real (for example, 3.0) for an integer
> is not allowed.

.
Why? It should be allowed.

[Steve Lionel]

On 6/28/2021 10:08 PM, Robin Vowels wrote:
> On Tuesday, June 29, 2021 at 2:33:28 AM UTC+10, Steve Lionel wrote:
>> On 6/28/2021 9:57 AM, Gary Scott wrote:
>>> Should allow user to specify an integer value for a real variable.
>> You can do that. But specifying a real (for example, 3.0) for an integer
>> is not allowed.
> .
> Why? It should be allowed.

The standard does not allow it, but I know that DEC/Compaq/Intel Fortran
does. The standard requires that the input for an integer variable be
suitable for I format editing, which "3.0" would not. But what would you
want it to do with, say, "3.5"? Round? Truncate? Something else?

[Robin Vowels]

On Wednesday, June 30, 2021 at 2:45:06 AM UTC+10, Steve Lionel wrote:
> On 6/28/2021 10:08 PM, Robin Vowels wrote:
> > On Tuesday, June 29, 2021 at 2:33:28 AM UTC+10, Steve Lionel wrote:
> >> On 6/28/2021 9:57 AM, Gary Scott wrote:
> >>> Should allow user to specify an integer value for a real variable.
> >> You can do that. But specifying a real (for example, 3.0) for an integer
> >> is not allowed.
> > .

> > Why? It should be allowed..

> The standard does not allow it, but I know that DEC/Compaq/Intel Fortran
> does. The standard requires that the input for an integer variable be
> suitable for I format editing, which "3.0" would not. But what would you
> want it to do with, say, "3.5"? Round? Truncate? Something else?

.
The same as would happen in the assignment statement K = R
(where K is integer and R is real).
It's obvious. (And it would not need a special rule just for NAMELIST data.)

[Ron Shepard]

It is not uncommon to see programs read floating point numbers with
integer values and then immediately assign them to integers.

read(n,'(f4.0)') r
int = r ! or = nint(r)

The reason is that the input can be written as "3. ", " 3. ", or " 3."
within the four-character field width and the correct value would be
read and then transferred to the integer. You can't do that with an
integer field, so, especially for input records with many values, that
requires counting columns exactly in the input to get the integer value
right.

Even with modern fortran, I don't understand why i0 cannot be specified
as an input field width for this purpose (to mirror what i0 does on output).

$.02 -Ron Shepard

[gah4]

On Tuesday, June 29, 2021 at 6:10:14 PM UTC-7, Ron Shepard wrote:

> integer values and then immediately assign them to integers.

> read(n,'(f4.0)') r
> int = r ! or = nint(r)

> The reason is that the input can be written as "3. ", " 3. ", or " 3."
> within the four-character field width and the correct value would be
> read and then transferred to the integer. You can't do that with an
> integer field, so, especially for input records with many values, that
> requires counting columns exactly in the input to get the integer value
> right.

It used to be that both F and I input treated blanks the same as zero.
This meant that in both cases, one might be surprised about the value,
depending (for F) on the position of the implied decimal point. That
worked mostly well on cards, but less well on a terminal.

One way to get around that is to read F format, with explicit
decimal point, as you note.

Some years ago, the default was changed, with an option on OPEN
for the old treatment.

> Even with modern fortran, I don't understand why i0 cannot be specified
> as an input field width for this purpose (to mirror what i0 does on output).

There are interesting difference between Fortran formatted input
and C's scanf(). In scanf, input stops for any character that doesn't
satisfy the input descriptor. No error, it just goes onto the next one.

One can read a series of integers, separated by blanks (the usual way)
with scanf("%d%d%d%d%d", &i, &j, &k, &l, &m);

DEC used to have some interesting extensions for their Fortran systems,
related to terminal input.
I don't think it is close to a mirror of i0 on output, though, but if you did
something like C's %d on input, that might be nice.

Note another difference in C and Fortran format. For output, C format
descriptors expand to the width of the item. %d will always expand, where
the given value is the minimum width. Some C programs used "19%d" for
outputting years, such than in year 2000 we get 19100.

(I always wonder about printf("%f",1e3000L); in C.)

So, I think you can do I0, but you have to specify the exact meaning
regarding what follows in on the same line.

[jfh]

The only relevant archaic thing is that arrays in f95 namelists could not be allocatable. In f2003 or later they can be. The 'arr = 1 2 3' below is not an assignment (which would reallocate the array with size 3). My test program below, followed by its output, confirmed the OP's assumption when I compiled it with gfortran. An annoying feature is that even in f2018 the read statement had to access the variable alist, not the constant avalue.

program main
implicit none
real, allocatable :: arr(:)
namelist /param_inp/ arr
character(*),parameter:: avalues = '&param_inp arr = 1 2 3 /'
character(len(avalues)):: alist = avalues
arr = [4,5,6,7] ! allocation during assignment
read(alist,NML=param_inp)
write(*,NML=param_inp)
end program main

&PARAM_INP
ARR= 1.00000000 , 2.00000000 , 3.00000000 , 7.00000000 ,
/

[gah4]

On Wednesday, June 30, 2021 at 5:48:23 PM UTC-7, jfh wrote:

(snip)

> An annoying feature is that even in f2018 the read statement had to access the variable alist, not the constant avalue.

Nothing against Fortran, but I do find some of these restrictions funny.

I believe also, or maybe for the same reason, you can't put a string constant in.

read('5',*) i

Error: UNIT specification at (1) must be an INTEGER expression or a CHARACTER variable

[Ron Shepard]

On 6/30/21 7:48 PM, jfh wrote:
> On Friday, June 25, 2021 at 10:32:26 AM UTC+12, spectrum wrote:
>> Although namelist in Fortran is often thought to be an archaic feature
>> (and something that should not be used for portability?), I use it very
>> often for setting parameters from a file. In a recent case, I used
>> an array for input and tried to fill part of it, like:
>>
>> &param_inp
>> ...
>> arr(1:3) = 1.0 2.0 3.0
>> ...
>> /
>>
>> Here, the array "arr" is an allocatable array that was allocated with
>> a sufficiently large size (e.g. arr(1:100)) before namelist read.
>> The above code worked as expected, but it also seems the following
>> patterns also work in the same way:
>>
>> arr(:) = 1.0 2.0 3.0
>>
>> or
>>
>> arr = 1.0 2.0 3.0
>>
>> Does this mean that, if the number of entities on the right-hand side
>> of assignment is smaller than size(arr), it is okay to
>> assume that the remaining part of "arr" (e.g. arr(4:)) in the above case
>> is kept intact in the calling routine?
>
> The only relevant archaic thing is that arrays in f95 namelists could not be allocatable. In f2003 or later they can be.

Allocatables in f95 were hobbled in several ways. The "Allocatable TR"
was published at the same time as the f95 standard, which fixed most of
those issues, and that is what eventually was incorporated into f2003.
Allocatables have several important advantages over pointers, so it was
unfortunate timing that f95 itself did not include all of that
functionality. That resulted in an 8 to 10 year delay in the widespread
use of allocatables within the language.

> The 'arr = 1 2 3' below is not an assignment (which would reallocate the array with size 3). My test program below, followed by its output, confirmed the OP's assumption when I compiled it with gfortran. An annoying feature is that even in f2018 the read statement had to access the variable alist, not the constant avalue.

I think it would be a useful feature in the language if there were some
way to invoke allocation of both scalars and arrays during i/o
operations, including list-directed and namelist input. The workarounds
for this missing feature all suffer from practical inefficiencies, such
as reading the input file twice, or copying from an intermediate
linked-list data structure into the final allocatable array. This could
be done in ways that are consistent with the previous standard behavior,
and thus it would not invalidate any existing code.

$.02 -Ron Shepard

[gah4]

On Thursday, July 1, 2021 at 3:15:20 AM UTC-7, Ron Shepard wrote:

(snip)

> I think it would be a useful feature in the language if there were some
> way to invoke allocation of both scalars and arrays during i/o
> operations, including list-directed and namelist input. The workarounds
> for this missing feature all suffer from practical inefficiencies, such
> as reading the input file twice, or copying from an intermediate
> linked-list data structure into the final allocatable array. This could
> be done in ways that are consistent with the previous standard behavior,
> and thus it would not invalidate any existing code.

I suppose so, but it isn't obvious that it avoids the inefficiencies,
but most likely just moves them into library routines.

In some cases, knowing the file size is enough, or close enough, to find
the size. I believe that ENQUIRE can do that, but I never tried.

In the case of reading in formatted numeric data, as you note, either
reading in twice or temporary storage of some kind.

A loop using MOVE_ALLOC isn't so bad in most cases, though something
closer to C's realloc() might be nice.

[Robin Vowels]

.
Why scalars?
.

> and arrays during i/o
> operations, including list-directed and namelist input. The workarounds
> for this missing feature all suffer from practical inefficiencies, such
> as reading the input file twice, or copying from an intermediate
> linked-list data structure into the final allocatable array.

.
How often do you read in namelist data?
How often do you read in such large quantities of data
that you can't define an array that is large enough to contain
all elements? It is trivial then to process the elements that have been read in.

.
> This could
> be done in ways that are consistent with the previous standard behavior,
> and thus it would not invalidate any existing code.

.
It doesn't need new language features to achieve this,
which trivial straightforward programming can do.

[Gary Scott]

On 7/1/2021 5:32 AM, gah4 wrote:
> On Thursday, July 1, 2021 at 3:15:20 AM UTC-7, Ron Shepard wrote:
>
> (snip)
>
>> I think it would be a useful feature in the language if there were some
>> way to invoke allocation of both scalars and arrays during i/o
>> operations, including list-directed and namelist input. The workarounds
>> for this missing feature all suffer from practical inefficiencies, such
>> as reading the input file twice, or copying from an intermediate
>> linked-list data structure into the final allocatable array. This could
>> be done in ways that are consistent with the previous standard behavior,
>> and thus it would not invalidate any existing code.
>
> I suppose so, but it isn't obvious that it avoids the inefficiencies,
> but most likely just moves them into library routines.
>
> In some cases, knowing the file size is enough, or close enough, to find
> the size. I believe that ENQUIRE can do that, but I never tried.

In some OS, you would also need to inquire the file record structure
type for those that aren't just a bucket-o-bits.

[Ron Shepard]

On 7/1/21 5:32 AM, gah4 wrote:
> On Thursday, July 1, 2021 at 3:15:20 AM UTC-7, Ron Shepard wrote:
>
> (snip)
>
>> I think it would be a useful feature in the language if there were some
>> way to invoke allocation of both scalars and arrays during i/o
>> operations, including list-directed and namelist input. The workarounds
>> for this missing feature all suffer from practical inefficiencies, such
>> as reading the input file twice, or copying from an intermediate
>> linked-list data structure into the final allocatable array. This could
>> be done in ways that are consistent with the previous standard behavior,
>> and thus it would not invalidate any existing code.
>
> I suppose so, but it isn't obvious that it avoids the inefficiencies,
> but most likely just moves them into library routines.

This is probably right for some cases. If the user is reading into a
linked list, allocating the final array, moving the linked list into the
array, and then deleting the linked list, then yes, that is probably
what the fortran i/o library would be doing anyway. It just simplifies
the task for the programmer.

> In some cases, knowing the file size is enough, or close enough, to find
> the size. I believe that ENQUIRE can do that, but I never tried.
>
> In the case of reading in formatted numeric data, as you note, either
> reading in twice or temporary storage of some kind.

Yes, these are all workarounds that are available now and that are used.
In f77, dimensioning all the arrays at their maximum size was the common
approach. Now that memory is relatively cheap compared to then, it might
be argued that this is now a practical solution, but it still just seems
like a workaround to me. Fortran now has allocatable entities, both
scalar and arrays, so why not use them to their full advantage.

> A loop using MOVE_ALLOC isn't so bad in most cases, though something
> closer to C's realloc() might be nice.

If the realloc cannot be done in-place, then this requires up to N**2
data moves to read an array of length N. Not so bad if done in blocks
maybe rather than individual elements, but still O(N**2) scaling. I
think Fortran deserves a better solution to this common problem than that.

$.02 -Ron Shepard

[gah4]

On Friday, July 2, 2021 at 10:01:16 AM UTC-7, Ron Shepard wrote:

(snip, I wrote)

> > A loop using MOVE_ALLOC isn't so bad in most cases, though something
> > closer to C's realloc() might be nice.

> If the realloc cannot be done in-place, then this requires up to N**2
> data moves to read an array of length N. Not so bad if done in blocks
> maybe rather than individual elements, but still O(N**2) scaling. I
> think Fortran deserves a better solution to this common problem than that.

I always write mine to increase the size exponentially. I think my favorite
(I haven't done it recently) increased by 5/3 each time. So the number
of reallocations is O(log N), and so the elements moved is O(N logN).

But it does remind me of a (C) program I worked on many years ago,
where someone wrote an input routine which would read data a
line at a time, and strcat() it into a big (already allocated) array.
It turns out that, even without the reallocation that you note,
a strcat() loop is O(N**2). With N only (?) in the millions, it was
the dominant time for that program.

I also remember working on R program, which read in data and create
a matrix column by column, (or row by row) appending to the existing matrix.
That is, as you note, O(N**2). And many programs do this, even when the
size is known in advance.

Fortran fixed size (large) arrays were first done with single-task systems,
where there was no reason not to use all the memory. But then after
some years, there were multi-task real memory systems, where there
was a cost. (Though in the interests of keeping fragmentation down,
there weren't that many different sizes you could use.)

But then there was virtual memory, where the costs of large arrays
are mostly virtual storage. And with lazy allocation, many systems
don't even allocate any backing store for large arrays, until it
is used.

I now have Spice 2g6, the last of the Fortran Spice programs, running
with gfortran. From the notes, it looks like it was last done with f2c,
and then some of the C routines were modified later. In any case,
it was last used in the times of virtual memory systems, and uses the
the large (200000) element array system.

This is from the Fortran 77 days, when there is a large array, which is
then divided up for the various uses, keeping an index where each
array starts in the large array. For reasons that I don't know, it
fails with even -O1 in gfortran, but works with -O0.

[Robin Vowels]

.
What's the matter with using an allocatable array?
Allocate an array, read the data, then re-assign it with allocate on assignment.
I think that that is one extra [trivial] statement.

[gah4]

On Friday, July 2, 2021 at 6:09:02 PM UTC-7, Robin Vowels wrote:
> On Saturday, July 3, 2021 at 3:01:16 AM UTC+10, Ron Shepard wrote:

(snip)

> > If the realloc cannot be done in-place, then this requires up to N**2
> > data moves to read an array of length N. Not so bad if done in blocks
> > maybe rather than individual elements, but still O(N**2) scaling. I
> > think Fortran deserves a better solution to this common problem than that.

> What's the matter with using an allocatable array?
> Allocate an array, read the data, then re-assign it with allocate on assignment.
> I think that that is one extra [trivial] statement.

With 64 bit addressing (though most hardware doesn't supply all the bits),
and lazy allocation, you can almost allocate way bigger than you ever expect,
and then not worry about any waste.

Without lazy allocation, it has to allocate backing store, which might take some time.

You still have to pick a size, though.