Printing large arrays in Fortran
fai...@googlemail.com
Can anyone tell me how to print a large array in Fortran. In fact, I
have an 3000x3000 array, and would like to write it to an external
file, so that I can access it from MATLAB. I did it in the following
way
open(18,file='a.dat')
DO i=1,N
write(18,*)( A(i,j), j=1,N)
ENDDO
close(18)
But MATLAB could not read the dat file as an NxN matrix.
Any help would be appreciated,,,,,
Thank you for your help in advance...
fai...@googlemail.com
Beliavsky
fai...@googlemail.com wrote:
> Can anyone tell me how to print a large array in Fortran. In fact, I
> have an 3000x3000 array, and would like to write it to an external
> file, so that I can access it from MATLAB. I did it in the following
> way
>
> open(18,file='a.dat')
> DO i=1,N
> write(18,*)( A(i,j), j=1,N)
You could use a format string with a "repeat edit descriptor", for
example
write (18,"(3000(1x,f0.6))") ( A(i,j), j=1,N)
Some compilers may not allow a repetition count as large as 3000, in
which case you can use a smaller count combined with ADVANCE = "no".
Btw if you are using Fortran 90 or 95 compiler the implied do loop can
be replaced with an array subscript, giving
write (18,"(3000(1x,f0.6))") A(i,1:N)
> ENDDO
> close(18)
>
> But MATLAB could not read the dat file as an NxN matrix.
Did you look at the contents of the dat file before posting your
question :) ?
fai...@googlemail.com
Thank you for your replay.
> You could use a format string with a "repeat edit descriptor", for
> example
>
> write (18,"(3000(1x,f0.6))") ( A(i,j), j=1,N)
I did it as you suggest, but it seems that there is a problem with the
compiler (NAG F95 compiler). This is what I get
Buffer overflow on output
Program terminated by fatal I/O error
> Some compilers may not allow a repetition count as large as 3000, in
> which case you can use a smaller count combined with ADVANCE = "no".
I did this way as well, but get error too.
write(18,175)( A(i,j), j=1,100)
write(18,175,ADVANCE='NO')(A(i,j), j=101,201)
175 FORMAT(100(1x,d10.6))
glen herrmannsfeldt
>> You could use a format string with a "repeat edit descriptor", for
>> example
>> write (18,"(3000(1x,f0.6))") ( A(i,j), j=1,N)
> I did it as you suggest, but it seems that there is a problem with the
> compiler (NAG F95 compiler). This is what I get
> Buffer overflow on output
> Program terminated by fatal I/O error
There are limits you may run into, one being the longest line
(record) the compiler allows one to write. In this case I would
probably do something like:
DO I=1,N
WRITE(18,*) (A(I,J),J=1,N)
WRITE(18,*) "XY"
ENDDO
The put the output through the unix tr command, (versions
exist for Windows, too), to convert existing newlines to spaces
and XY to the appropriate line ending characters, something like:
tr '\r\nXY' ' \r\n' < temporaryfile > outputfile
Yes, its a kludge, but it does get around record limitations
for file systems using line ending characters.
-- glen
dpb
fai...@googlemail.com wrote:
> Can anyone tell me how to print a large array in Fortran. In fact, I
> have an 3000x3000 array, and would like to write it to an external
Do you have to see it as a text file as well as being able to read it
into Matlab? If not, how about writing it as an unformatted file? How
to do so w/ F77 will be compiler-dependent, but virtually all compilers
have a way. Look in the compiler doc for "stream" or "binary" output
or in the OPEN statement section. Failing that, post back the
particular compiler and OS and somebody here will undoubtedly know the
proper incantation.
Then, you can open the file from Matlab w/ the "b" switch and suck the
data in. This way will probably have noticeable performance advantages
on both the Fortran and the Matlab side as well.
me...@skyway.usask.ca
What kind of format does MATLAB read ... e.g. if you had data
from another source.
Does it want comma delimited, does it have a restriction
on the line length , or if no restriction, do you
have to specify it...
You can write any format you like from Fortran ... but
you have to know which one you like.
Chris
Ancient_Hacker
You have a nine million element array, so it may be worthwhile to try
to write it out more efficiently.
First, see if MatLab can read a plain binary file of 32-bit floats. I
imagine there is a way.
Then use an unformatted write. You may have to tweak soem parametes on
the OPEN statement to get a pure binary write. But that should not
only run much faster, it will take up much less disk space, and much
more likely to read into matlab.
I'd first try a test program with like 100x100 !!
Terence
data.
Do you need to read as floating-point image values of 4 or 8 byte
words?
Or will scientific notation in Ascii be acceptable and serve the
purpose?
And will MATLIB read until it has satisfied the column count for a row,
irrespective of how many column elements per record are given at a
time?
If this latter case than write the elements formatted, say 10 at a
time.
Else if the former case, you need to write the file in Unformatted
Binary mode.
One simple "legal" way is to open the file as direct unformatted with a
record length of (element size in bytes, times N) where N an integer
fraction of the number of elements in the matrix row (e.g. 10 columns).
Then just write out the column elements for the row and repeat for all
the rows.
Mark B Stucky
<fai...@googlemail.com> wrote in message
news:1165945070....@j44g2000cwa.googlegroups.com...
As others have mentioned that you might be better off reading and
writing binary or unformatted files...
The following is untested, but should give you the general idea...
Fortran :
integer N,M
parameter (N = 300)
parameter (M = 100)
real A(N,M)
open(18,file='a.dat',form='UNFORMATTED')
write(18) N, M
write(18) ((A(i,j), i=1,N), j=1,M)
close(18)
Matlab:
fname = 'a.dat';
byteorder = 'ieee-be'; % IEEE Big-Endian format (user
"ieee-le" for Little-Endian)
[fid,msg] = fopen(fname, 'r', byteorder) ;
% read the number of records...
[nrec,count] = fread(fid,1,'int32');
[n,count] = fread(fid,1,'int32');
[m,count] = fread(fid,1,'int32');
% read the trailing number of records...
[nrec,count] = fread(fid,1,'int32');
% read the number of records...
[nrec,count] = fread(fid,1,'int32');
[Ain,count] = fread(fid,n*m, 'float32');
% read the trailing number of records...
[nrec,count] = fread(fid,1,'int32');
A = reshape(Ain, n, m);
clear Ain
Hopefully that should give you a start...
--Mark
James Giles
...
> You could use a format string with a "repeat edit descriptor", for
> example
>
> write (18,"(3000(1x,f0.6))") ( A(i,j), j=1,N)
>
> Some compilers may not allow a repetition count as large as 3000, in
> which case you can use a smaller count combined with ADVANCE = "no".
Advance="no" would not help in this case. It allows processing
of long records by not assuming an end of record when the I/O
list has been exhausted. But, it still assumed record marks
everywhere else that would normally cause one. In this case
if the count is less than 3000 format "reversion" will occur.
Such "reversion" still causes a record termination.
This should work for writing the data in one line (though whether
MATLIB reads it is still something I don't know):
DO i=1,N
DO j=1,N-1
write(18, '(1x,f0.6)', advance='no') A(i,j)
END DO
write(18, '(1x,f0.6)') A(i,j) ! last of the line advances
END DO
Of course, implementations may have arbitrary limits on the
length of a record even if you're using non-advancing I/O
(or even if you're using F2003 stream I/O - a redundant feature
for formatted transfers).
(Since the OP said he was working with F77, he can't use
non-advancing I/O anyway. I'm just correcting your error
in describing the feature.)
--
J. Giles
"I conclude that there are two ways of constructing a software
design: One way is to make it so simple that there are obviously
no deficiencies and the other way is to make it so complicated
that there are no obvious deficiencies." -- C. A. R. Hoare
James Giles
...
> write(18, '(1x,f0.6)') A(i,j) ! last of the line advances
Sorry, that should have been:
write(18, '(1x,f0.6)') A(i,N) ! last of the line advances
Too quick on the drag and drop. :-(
Dmitry V. Gorbatovsky
It seems to me like ideal case for using hdf
http://www.hdfgroup.org/HDF5/doc/
Matlab has "native" support for hdf.
Regards, Dmitry
Dylan Sung
If you're not using MATLAB, and just want the data values to be stored in
another fashion, then I suggest converting it to a one dimensional array
where B(k) = A(i,j). If you have a 3000 by 3000 array, let max=3000, and so
B must have max*max values. We also set up two integer arrays to store the
grid references in A as M and N and these also have max*max values.
parameter (max=3000)
integer X(max*max),Y(max*max)
read A(max,max),B(max*max)
k = 0
do 101 i = 1, max
do 201 j = 1, max
k = k + 1
B(k)=A(i,j)
M(k)=i
N(k)=j
201 continue
101 continue
To reverse that, you can write
real C(max,max)
do 203 k = 1, max*max
C(M(k),N(k))=B(k)
203 continue
where C has taken on the values of A for each and every grid ref and the two
are identical.
Dyl.
Ian Bush
I must be missing something here ...
Dylan Sung wrote:
>
> I suggest converting it to a one dimensional array
> where B(k) = A(i,j). If you have a 3000 by 3000 array, let max=3000, and
> so B must have max*max values. We also set up two integer arrays to store
> the grid references in A as M and N and these also have max*max values.
>
> parameter (max=3000)
> integer X(max*max),Y(max*max)
> read A(max,max),B(max*max)
> k = 0
> do 101 i = 1, max
> do 201 j = 1, max
> k = k + 1
> B(k)=A(i,j)
> M(k)=i
> N(k)=j
> 201 continue
> 101 continue
(Ignoring the obvious typo above)
You've quadrupled the memory usage, indexed A the wrong way round, and introduced
a significantly less easily read data structure for what gain ?
Ian
Beliavsky
fai...@googlemail.com wrote:
> Thank you for your replay.
>
> > You could use a format string with a "repeat edit descriptor", for
> > example
> >
> > write (18,"(3000(1x,f0.6))") ( A(i,j), j=1,N)
>
> I did it as you suggest, but it seems that there is a problem with the
> compiler (NAG F95 compiler). This is what I get
>
> Buffer overflow on output
> Program terminated by fatal I/O error
Have you read the answer to the NAG FAQ at
http://www.nag.co.uk/nagware/NP/doc/faq.asp#FAQQ8 :
"What does "Buffer overflow on output" mean?
This means that your program attempted a formatted write, in a single
record, with more bytes than will fit into the file buffer for that
unit.
The default buffer size for formatted i/o is 1024 characters.
To increase the buffer size you can use the RECL= keyword in the OPEN
statement for the file you wish to write. For example:
OPEN(13,FILE="myfile",FORM="formatted",RECL=2048)
will establish a buffer size of 2048 bytes on unit 13.
Note that Fortran 95 provides the IOLENGTH keyword in the INQUIRE
statement to discover the needed value for RECL. For example:
INQUIRE(IOLENGTH=IRECL) MY_BIG_ARRAY
OPEN(13,FILE="dump",FORM="formatted",RECL=IRECL)
will establish a sufficiently large buffer for unit 13 to allow the
entirety of MY_BIG_ARRAY to be read or written in one i/o statement."
The following program, which writes a 3000x3000 matrix to a file, works
fine with g95, gfortran and Intel Fortran, taking between 30 and 60
seconds to run on my PC:
program xrepeat_edit
integer, parameter :: nr = 3000, nc = 3000, iu = 18
real :: A(nr,nc)
integer :: i,j
call random_seed()
call random_number(A)
open (unit=iu,file="junk.txt",action="write",status="replace")
do i=1,nr
write (iu,"(3000(1x,f0.6))") (A(i,j),j=1,nc)
end do
end program xrepeat_edit
Here are my experiences with the NAG compiler.
Your "Buffer overflow on output" problem occurred in a production code
with which I was testing the NAG compiler, and which works with g95,
gfortran, Lahey/Fujitsu, and Intel Fortran.
Before that, I had a problem that for NAG, the SYSTEM routine is in the
F90_UNIX_PROC module, so I need to write
use F90_UNIX_PROC, only: SYSTEM
for NAG to make my code work, unlike other compilers. I understand that
SYSTEM is a Fortran 95 extension and that I could write a wrapper for
it.
With the existence of free Fortran 95 compilers such as g95 and
gfortran, and faster compilers such as Intel Fortran, I don't see why I
should pay money and jump through hoops to make my codes run with NAG
Fortran.
Gordon Sande
> With the existence of free Fortran 95 compilers such as g95 and
> gfortran, and faster compilers such as Intel Fortran, I don't see why I
> should pay money and jump through hoops to make my codes run with NAG
> Fortran.
Perhaps because it has better debugging facilities. The ones you mention
come up a weaker when compared to the "-C=undefined" option of NAG.
The capability was not there until version 5. Lahey and Salford provide
similar facilities but only for one platform rather than the multiple
platforms of NAG.
Tighter checking makes for fewer problems in the future. Multiple compilers
to do portability checking is often recommended. Choosing at least one
"performance" vendor and one "debugging" vendor for the first two makes
a lot of sense.
Beliavsky
Gordon Sande wrote:
> On 2006-12-13 11:06:54 -0400, "Beliavsky" <beli...@aol.com> said:
>
> > With the existence of free Fortran 95 compilers such as g95 and
> > gfortran, and faster compilers such as Intel Fortran, I don't see why I
> > should pay money and jump through hoops to make my codes run with NAG
> > Fortran.
>
> Perhaps because it has better debugging facilities. The ones you mention
> come up a weaker when compared to the "-C=undefined" option of NAG.
> The capability was not there until version 5. Lahey and Salford provide
> similar facilities but only for one platform rather than the multiple
> platforms of NAG.
Lahey has compilers for Intel Linux and Intel Windows, which I would
count as two platforms. G95 has the freal=NaN (setting reals to NaN by
default) and similar options for integer and logical variables, but
this is not a full substitute for a "-C=undefined" option.
Running a code with multiple compilers can help one detect the use of
uninitialized variables (the results will be different) and other bad
code, and my complaint against NAG is that they do little things to
make that more difficult for me.
Gordon Sande
Explicit undefined checking beats noticing differences in results.
Particularly if the code uses self correcting numerical algorithms
which will just grind a bit longer.
Every time I have had NAG complain I discovered that they were following
the exact letter of the Fortran standard. For one offs on rush days that
is a nuisance but it sure beats having to fix it later when one has
forgotten the details.
Michael Prager
[old quotes snipped]
> Every time I have had NAG complain I discovered that they were following
> the exact letter of the Fortran standard. For one offs on rush days that
> is a nuisance but it sure beats having to fix it later when one has
> forgotten the details.
Let me add a note of caution to this lovefest for FTN95.
Though the undefined variable checking is impressive, I gave up
on FTN95 several years ago and returned it for a refund. I was
running into errors with perfectly valid (and not particularly
fancy) F95 code. The problems were acknowledged by Salford. I
had neither time nor inclination to help them work those bugs
out.
Since the FTN95 error checking is so very good, perhaps even
better than Lahey (which I use regularly), I am interested to
hear if Gordon (and others) think that it now has been more
thoroughly debugged.
--
Mike Prager, NOAA, Beaufort, NC
Address spam-trapped; remove color to reply.
* Opinions expressed are personal and not represented otherwise.
* Any use of tradenames does not constitute a NOAA endorsement.
Herman D. Knoble
Silverfrost just came out with V5 of FTN95. It fixes
some annoying bugs (but does not diagnose some of our tests (same
ones at Polyhedron). I look at FTN95 as an alternate compiler. It's
Windowed debugger is better than any other vendor's. I too
got a bit frustrated when trying to debug a large (> 100,000 lines)
code. But if we users of compilers do not faithfully report bugs,
how in the heck will compiler developers correct their compilers?
They do maintain test suits of codes, but there always seems to
be new bugs in any compiler. While you may not have the time
at a given time, I'd suggest trying to make time sometime:-)
Note that the Lahey compiler (LF95) with the right combinaton
of compiler options (either under Windows or Linux):
-chk (aesux) -chkglobal -co -nsav -pca -warn -trace -stchk -trap diou -nli
will give run-time diagnostics about as good as Salford's.
(But the windowed debugger is not near as useful as Salford's).
Soap Box:
Finally - and Intel please take notee - the availability of compiler
options to cause run-time diagnostics to flag all types of undefined variables,
subscripts and strings out of range, illegal subprogram argument/parameter
use, storage leaks (a la g95), etc., do help add significant integrity
to Fortran codes that end up helping to do important (mission critical)
human services. That is, we all accept that robust run-time diagnostic
capability may cause compiled code to run significantly slower than
highly optimized code. Correct results of a code, surely helped
significantly by good debugging tools, are more important than
speed during the debugging phase of Fortran program development.
Regards.
Skip Knoble
On Wed, 13 Dec 2006 12:37:55 -0500, Michael Prager <Mike.Prag...@noaa.gov> wrote:
-|Let me add a note of caution to this lovefest for FTN95.
-|
-|Though the undefined variable checking is impressive, I gave up
-|on FTN95 several years ago and returned it for a refund. I was
-|running into errors with perfectly valid (and not particularly
-|fancy) F95 code. The problems were acknowledged by Salford. I
-|had neither time nor inclination to help them work those bugs
-|out.
-|
-|Since the FTN95 error checking is so very good, perhaps even
-|better than Lahey (which I use regularly), I am interested to
-|hear if Gordon (and others) think that it now has been more
-|thoroughly debugged.
-|
-|
-|--
-|Mike Prager, NOAA, Beaufort, NC
-|Address spam-trapped; remove color to reply.
-|* Opinions expressed are personal and not represented otherwise.
-|* Any use of tradenames does not constitute a NOAA endorsement.
Beliavsky
> Gordon Sande <g.s...@worldnet.att.net> wrote:
>
> [old quotes snipped]
>
> > Every time I have had NAG complain I discovered that they were following
> > the exact letter of the Fortran standard. For one offs on rush days that
> > is a nuisance but it sure beats having to fix it later when one has
> > forgotten the details.
>
> Let me add a note of caution to this lovefest for FTN95.
One person mentioning a positive attribute of the Salford compiler does
not constitute a "lovefest". The comments you quoted were about NAG, a
different compiler.
>
> Though the undefined variable checking is impressive, I gave up
> on FTN95 several years ago and returned it for a refund. I was
> running into errors with perfectly valid (and not particularly
> fancy) F95 code. The problems were acknowledged by Salford. I
> had neither time nor inclination to help them work those bugs
> out.
I had the same problem when I tried Salford, but I think the bugs were
later fixed.
> Since the FTN95 error checking is so very good, perhaps even
> better than Lahey (which I use regularly), I am interested to
> hear if Gordon (and others) think that it now has been more
> thoroughly debugged.
Salford is Windows-only, which is not directly a problem for me, but a
compiler that also runs on Linux and/or OSX will automatically have
more users/bug reporters. The compiler was sold to Silverfrost, and the
Salford web site http://www.salfordsoftware.co.uk/ does not mention
Fortran. I wonder how many active developers of the compiler remain.
When I emailed Silverfrost some time ago, they said there were no plans
for a Fortran 2003 version of the compiler. The compiler does not
support the Allocatable TR , and forcing oneself to use pointers when
one could use allocatable arrays would defeat the goal of writing less
buggy code.
Gordon Sande
<Mike.Prag...@noaa.gov> said:
> Gordon Sande <g.s...@worldnet.att.net> wrote:
>
> [old quotes snipped]
>
>> Every time I have had NAG complain I discovered that they were following
>> the exact letter of the Fortran standard. For one offs on rush days that
>> is a nuisance but it sure beats having to fix it later when one has
>> forgotten the details.
>
> Let me add a note of caution to this lovefest for FTN95.
>
> Though the undefined variable checking is impressive, I gave up
> on FTN95 several years ago and returned it for a refund. I was
> running into errors with perfectly valid (and not particularly
> fancy) F95 code. The problems were acknowledged by Salford. I
> had neither time nor inclination to help them work those bugs
> out.
>
> Since the FTN95 error checking is so very good, perhaps even
> better than Lahey (which I use regularly), I am interested to
> hear if Gordon (and others) think that it now has been more
> thoroughly debugged.
>
Salford is the vendor for FTN95
NAG is the vendor for F95
Salford is a university in the UK. NAG is a commercial firm based
in the UK with an operation in the US (Chicago suburbs). NAG also
sell a library like the IMSL (whatever it is called now - Visual
Numerics?) library.
At an early stage Salford resold the NAG F90 then they redid their own
Fortran 95. CheckMate is impressive when it works. FTN95 had lots of
problems. I got to the point of emailing the compiler develper directly
at one point. Salford's older F77 is evidently rock solid and uses the
same CheckMate. Salford now use a reseller named SilverFrost. They offer
a zero cost personal edition.
NAGWare F95 is a different beast. Up to Version 4 it had the usual subscript
checking. With Version 5 they added undefined variable checking. I have
observed no problems with the NAGWare F95. NAG is typically the first vendor
out for a new standard. They use the GCC backend to allow them to concentrate
on the frontend analysis. The result is an early debugging strict compiler
that only optimizes as well as GCC and that supports many platforms. I only
got NAG after they came out with V5 for Mac. (GCC is a memory hog when
asked to optimize and I keep meaning to get more. That paging is a scheme to
make a computer run at the speed of its slowest component is all too true!)
I have both Salford and Lahey on my PC. Salford is quite OK on conservative,
looks like F77, code but my problems were on the various heads down conversions
from C. All pointers and allocates so not like typical Fortran. Lahey has
restrictions on undefineds in allocated arrays (or at least the last time I
looked) and had problems with large modules. I converted some F77 by adding
a contains at the top to get arguement checking and it tanked on too many
relocation elements. I also have DEC/Compaq and Intel to make a full panel.
Final compiles are with Intel.
I run NAG on my Mac/OsX and it is my everyday workhorse. No problems and solid
debugging. The Unix compatibilty stuff does not work with undefined but the
one show stopper (GETENV) was supplied by their support. Hopefully they will
make it all work and put it in a supplied module. (I use GETENV because I
could not trust GETARG to be standard and I wanted the totally operating
system dependent checking for file presence etc to be done in the scripts.)
So Salford FTN95 on a PC is not the same as NAGWare F95 on lots including Mac.
I prefer to debug interesting problems so I really like systems that catch
silly trivial mistakes like typos and forgotten initializations. My problems
tend to allow for test beds that allow definitional testing of the results
so I find that about 1/3 of my errors are in the test bed but they help make
sure I actually understand my problems. I have made enough varied mistakes
to know that I need all the help I can get. I am willing to have separate
debugging and production compilers as that seems to be the way the world is.
I shudder when I see reviews that are only of compile and exection speed. The
Polyhedron tables of error checking are a great step forward.
Michael Prager
I'm glad that G. and B. cleared up my confusion between two
different compilers. Skip -- your soapbox speech is right on
target.
Mike
robin
You need to ask yourself how does MATLAB require the numbers?
As integers? As numbers with a decimal point?
As numbers with exponents?
Then choose an appropriate specific format specification.
When something like this doesn't work,
first try it with a small array (3 x 3).