GFortran - OpenMP problem with Windows XP memory
Cirilo S. Bresolin
I am experiencing a problem with a simple multithread program and
WindowsXP.
The program is as follows. It doesn't do nothing, because I'm just
testing and learning the OpenMP directives.
program test
use omp_lib
implicit none
integer :: i,j
integer, parameter :: n=361
double precision :: a(n,n), b(n,n)
call omp_set_num_threads(2)
!$omp parallel shared(a,b) private(i,j)
!$omp do
do i=1,n
do j=1,n
a(j,i) = dble(i)**2.d0 + dble(j)**2.d0 !This
operations is just to spend some cpu time
b(j,i) = dble(i)**4.d0 + dble(j)**4.d0
a(j,i) = b(j,i)/(b(j,i)*b(j,i)) + a(j,i)/(a(j,i)*a(j,i))
end do
end do
!$omp end do
!$omp end parallel
end program
I compile using: gfortran file.f95 - fopenmp
System: Windows XP Professional SP2
Machine: Intel Core 2 Duo E6550 2.33 GHz 2Gb RAM
The problem: If I set n = 361 the windows crashs the program, if it is
set to 360 it runs ok. Without the -fopenmp flag, I can set n up to
11169, but with 11170 it crashs. In the last case I understand that is
a memory limit problem, but in the first I don't. Why this happen? n
is to small to be memory problem.
An interesting behavior. The same program with single precision has
the limits: n=510 with -fopemmp flag and n=15795 without the flag. The
ratio 510/360 = 15795/11169 = sqrt(2).
Some light over this problem?
Anony
"Cirilo S. Bresolin" <cirilo....@gmail.com> wrote in message
news:eae7bce9-523f-49e1...@a1g2000hsb.googlegroups.com...
> Hi!
>
> I am experiencing a problem with a simple multithread program and
> WindowsXP.
>
> The program is as follows. It doesn't do nothing, because I'm just
> testing and learning the OpenMP directives.
>
> program test
> use omp_lib
> implicit none
> integer :: i,j
> integer, parameter :: n=361
> double precision :: a(n,n), b(n,n)
>
A workaround ... Modify the above statement to
double precision, save :: a(n,n), b(n,n)
glen herrmannsfeldt
> I am experiencing a problem with a simple multithread program and
> WindowsXP.
> The program is as follows. It doesn't do nothing, because I'm just
> testing and learning the OpenMP directives.
Even though is doesn't do anything useful, it should still
make sense in terms of OpenMP. Since A and B are shared,
I would expect each thread to operate on only part of the
array, such as even elements for one, odd for the other.
Since you don't do that, both threads are changing A
and B, possibly generating undesired values, such as
zero, for A or B. Since you divide by A and B...
What does it do when it crashes? Any message?
> program test
> use omp_lib
> implicit none
> integer :: i,j
> integer, parameter :: n=361
> double precision :: a(n,n), b(n,n)
> call omp_set_num_threads(2)
> !$omp parallel shared(a,b) private(i,j)
> !$omp do
> do i=1,n
> do j=1,n
> a(j,i) = dble(i)**2.d0 + dble(j)**2.d0 !This
> operations is just to spend some cpu time
> b(j,i) = dble(i)**4.d0 + dble(j)**4.d0
> a(j,i) = b(j,i)/(b(j,i)*b(j,i)) + a(j,i)/(a(j,i)*a(j,i))
> end do
> end do
> !$omp end do
> !$omp end parallel
> end program
(snip)
> The problem: If I set n = 361 the windows crashs the program, if it is
> set to 360 it runs ok. Without the -fopenmp flag, I can set n up to
> 11169, but with 11170 it crashs. In the last case I understand that is
> a memory limit problem, but in the first I don't. Why this happen? n
> is to small to be memory problem.
361 seems to be just below each 1MB for A and B.
-- glen
Greg Lindahl
Cirilo S. Bresolin <cirilo....@gmail.com> wrote:
> Why this happen? n is to small to be memory problem.
The stack size in OpenMP programs is often limited.
A good OpenMP runtime will tell you explicitly when you run out of
stack in one of the threads.
-- greg
Anony
"glen herrmannsfeldt" <g...@ugcs.caltech.edu> wrote in message
news:_KqdnchSOLPTm7fV...@comcast.com...
> Even though is doesn't do anything useful, it should still
> make sense in terms of OpenMP. Since A and B are shared,
> I would expect each thread to operate on only part of the
> array, such as even elements for one, odd for the other.
>
> Since you don't do that, both threads are changing A
> and B, possibly generating undesired values, such as
> zero, for A or B. Since you divide by A and B...
>
> What does it do when it crashes? Any message?
>
This is not a problem with OpenMP. The problem is from gfortran failure to
allocate memory. With the attribute "save", everything is OK. For example,
C:\TEMP>type bresolin.f90
program test
use omp_lib
implicit none
integer :: i,j
integer, parameter :: n=361
double precision, save :: a(n,n), b(n,n)
call omp_set_num_threads(2)
!$omp parallel shared(a,b) private(i,j)
!$omp do
do i=1,n
do j=1,n
a(j,i) = dble(i)**2.d0 + dble(j)**2.d0
b(j,i) = dble(i)**4.d0 + dble(j)**4.d0
a(j,i) = b(j,i)/(b(j,i)*b(j,i)) + a(j,i)/(a(j,i)*a(j,i))
end do
end do
!$omp end do
!$omp end parallel
end program
C:\TEMP>gfortran -fopenmp bresolin.f90 -o bresolin.exe
C:\TEMP>bresolin
The program is complete without an error message. I even tested it with
n=5000 without any problem.
Steven G. Kargl
"Anony" <no-e...@equation.com> writes:
>
> "glen herrmannsfeldt" <g...@ugcs.caltech.edu> wrote in message
> news:_KqdnchSOLPTm7fV...@comcast.com...
>> Even though is doesn't do anything useful, it should still
>> make sense in terms of OpenMP. Since A and B are shared,
>> I would expect each thread to operate on only part of the
>> array, such as even elements for one, odd for the other.
>>
>> Since you don't do that, both threads are changing A
>> and B, possibly generating undesired values, such as
>> zero, for A or B. Since you divide by A and B...
>>
>> What does it do when it crashes? Any message?
>>
>
> This is not a problem with OpenMP. The problem is from gfortran failure to
> allocate memory. With the attribute "save", everything is OK. For example,
It's not a problem with gfortran. It's a problem with reading the
available documentation and using a limited OS. From gfortran.info
* `-fopenmp' implies `-frecursive', i.e., all local arrays will be
allocated on the stack. When porting existing code to OpenMP, this
may lead to surprising results, especially to segmentation faults
if the stacksize is limited.
Here's your program running without any special modifications or
options on FreeBSD
last pid: 67464; load averages: 1.26, 1.16, 0.75 up 34+05:19:22 15:46:16
74 processes: 4 running, 70 sleeping
CPU states: 81.2% user, 0.0% nice, 17.7% system, 0.4% interrupt, 0.8% idle
Mem: 2656M Active, 4386M Inact, 434M Wired, 190M Cache, 214M Buf, 212M Free
Swap: 17G Total, 184K Used, 17G Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
67463 kargl 101 0 1536M 1273M RUN 1 0:06 27.10% {z}
67463 kargl 99 0 1536M 1273M RUN 0 0:06 17.58% {initial thread}
Well, I did change n to 10000.
In your case, you probably need the -fmax-stack-var-size option and
to set your OS's resource limits to some appropriate value(s).
Anony
"Steven G. Kargl" <ka...@troutmask.apl.washington.edu> wrote in message
news:g0d6db$b2c$1...@gnus01.u.washington.edu...
> In article <%ToWj.7867$Uz2.1122@trnddc06>,
> It's not a problem with gfortran. It's a problem with reading the
> available documentation and using a limited OS. From gfortran.info
>
> * `-fopenmp' implies `-frecursive', i.e., all local arrays will be
> allocated on the stack. When porting existing code to OpenMP, this
> may lead to surprising results, especially to segmentation faults
> if the stacksize is limited.
>
You should refer to the original post of Bresolin, without -fopenmp, the
problem still appears. It does not matter if -fopenmp is appiled or not.
ifort on Windows never has the problem. ***Whether gfortran properly
allocates memory is an issue.***
glen herrmannsfeldt
> "glen herrmannsfeldt" <g...@ugcs.caltech.edu> wrote in message
> news:_KqdnchSOLPTm7fV...@comcast.com...
>>Even though is doesn't do anything useful, it should still
>>make sense in terms of OpenMP. Since A and B are shared,
>>I would expect each thread to operate on only part of the
>>array, such as even elements for one, odd for the other.
>>Since you don't do that, both threads are changing A
>>and B, possibly generating undesired values, such as
>>zero, for A or B. Since you divide by A and B...
> This is not a problem with OpenMP. The problem is from gfortran failure to
> allocate memory. With the attribute "save", everything is OK. For example,
I believe you are right in this case, but it could
cause problems in other cases. In general, one must
be sure that only one thread is changing any variable,
and that other threads aren't using the value of variables
that one thread changes, without synchronization in between.
Otherwise, stack size has always been a problem with DOS
and Windows. I still remember when the default stack size
was 2K bytes.
-- glen
Steven G. Kargl
Again, gfortran has no problem with memory allocation. It is a resource
limit. gfortran, as does many other compilers, puts temporary variables
on the stack. The OP needs either to use the option I pointed out or
get a better OS or both.
Anony
> In article <QNpWj.22089$5b3.10143@trnddc05>,
>
> Again, gfortran has no problem with memory allocation. It is a resource
> limit. gfortran, as does many other compilers, puts temporary variables
> on the stack. The OP needs either to use the option I pointed out or
> get a better OS or both.
>
What is the definition of "temporary variables"?
The matrices, a(n,n) & b(n,n) in Bresolin's main program, are two key
variables, and exist from the beginning to the end. They are not temporary.
In your opinion, gfortran treats every variable as a temporary variable, and
put on the stack.
It would be a user's choice whether to use the option -fmax-stack-var-size,
as you suggested. That would be a burden for user to estimate stack size. In
my experiences, the magic word "save", as shown in previous example, can
resolve the problem and ease the work.
Steven G. Kargl
"Anony" <no-e...@equation.com> writes:
>
> "Steven G. Kargl" <ka...@troutmask.apl.washington.edu> wrote in message
> news:g0da88$ehe$1...@gnus01.u.washington.edu...
>> In article <QNpWj.22089$5b3.10143@trnddc05>,
>>
>> Again, gfortran has no problem with memory allocation. It is a resource
>> limit. gfortran, as does many other compilers, puts temporary variables
>> on the stack. The OP needs either to use the option I pointed out or
>> get a better OS or both.
>>
>
> What is the definition of "temporary variables"?
>
> The matrices, a(n,n) & b(n,n) in Bresolin's main program, are two key
> variables, and exist from the beginning to the end. They are not temporary.
> In your opinion, gfortran treats every variable as a temporary variable, and
> put on the stack.
a(j,i) = b(j,i)/(b(j,i)*b(j,i)) + a(j,i)/(a(j,i)*a(j,i))
Does gfortran create one or more temporary arrays for the RHS? It's
a rhetorical question. I know the answer.
> It would be a user's choice whether to use the option -fmax-stack-var-size,
> as you suggested. That would be a burden for user to estimate stack size. In
> my experiences, the magic word "save", as shown in previous example, can
> resolve the problem and ease the work.
You can still hit a resource limit with 'save'.
troutmask:kargl[268] limit
cputime unlimited
filesize unlimited
datasize 8388608 kbytes <-- If this is too small, 'save' won't help no.
stacksize 1048576 kbytes <-- This is restricts -fmax-stack-var-size.
coredumpsize unlimited
memoryuse unlimited
vmemoryuse unlimited
Cirilo S. Bresolin
I spend some time today studying the problem, but I didn't understand
if the problem is with gfortran or OpenMP or neither.
As suggested by Anony, using:
<code> double precision, save :: a(n,n), b(n,n)
c:\gfortran file.f95 -fopenmp</code>
worked well up to n=11169. Either as suggested by Steven G. Kargl:
<code>double precision :: a(n,n), b(n,n)
c:\gfortran file.f95 -fopenmp -fmax-stack-var-size=2</code>
worked with any size greater than 2 worked. Also worked:
<code>double precision :: a(n,n)=0.d0, b(n,n)=0.d0
c:\gfortran file.f95 -fopenmp</code>
that implies implicit <cite>save</save> attribute, since explicit
initialization, but max n=7198 (nearly 1Gb RAM). And
<code>double precision :: a(n,n), b(n,n)
c:\gfortran file.f95 -fopenmp -fno-automatic</code>
that, as -fmax-stack-var-size, overwrites the -frecursive flag imposed
by -fopenmp flag.
Should work too but I didn't test, set the environment variable
GOMP_STACKSIZE to some value.
Now from documentation:
From OpenMP API v2.5
<cite>In both Fortran 90 and Fortran 95, variables with explicit
initialization have the SAVE
attribute implicitly. This is not the case in Fortran 77. However, a
compliant OpenMP
Fortran implementation must give such a variable the SAVE attribute,
regardless of the
underlying base language version.</cite>
and Gfortran.info, as previusly previusly by Steven G. Kargl
<cite>`-fopenmp' implies `-frecursive', i.e., all local arrays will
be
allocated on the stack. When porting existing code to OpenMP, this
may lead to surprising results, especially to segmentation faults
if the stacksize is limited. </cite>
Now the questions:
All shared variable needs have in some way the SAVE attribute to work,
as said in the OpenMP API? If this is the case all shared variables,
in subroutines/functions called many times, must have explicit
initialization at least, to avoid side-effects.
The -frecursive flag imposed by -fopenmp, is something general from
the OpenMP API or something particular from gfortran/gomp
implementation, to force stack allocation?
Anony
"Cirilo S. Bresolin" <cirilo....@gmail.com> wrote in message
news:95999025-2b18-4172...@s50g2000hsb.googlegroups.com...
> Hi!
>
> I spend some time today studying the problem, but I didn't understand
> if the problem is with gfortran or OpenMP or neither.
> As suggested by Anony, using:
>
> <code> double precision, save :: a(n,n), b(n,n)
>
> c:\gfortran file.f95 -fopenmp</code>
>
> worked well up to n=11169.
>
I assume you have 32-bit gfortran (because 64-bit gfortran for windows works
and also supports openMP, but has optimization problems. Possibly, not so
many people use 64-bit gfortran for windows presently). When n=11169, the
memory space for a(n,n) & b(n,n), double precision, requires about 2GB. That
almost reaches the limit of 32-bit memory address. The 32-bit memory address
put a limit here. For n --> bigger, you need 64-bit OS and 64-bit compiler.
>
> ....................................
> that, as -fmax-stack-var-size, overwrites the -frecursive flag imposed
> by -fopenmp flag.
>
A little confuse here. -fmax-stack-var-size does not overwrite -frecursive,
but set a stack size.
>
> and Gfortran.info, as previusly previusly by Steven G. Kargl
> <cite>`-fopenmp' implies `-frecursive', i.e., all local arrays will
> be
> allocated on the stack. When porting existing code to OpenMP, this
> may lead to surprising results, especially to segmentation faults
> if the stacksize is limited. </cite>
>
It may be better for you to pay attention on which variables are private and
which variables are shared. OpenMP manual focus on "private" and "shared"
variables. It is really unnecessary for you to pay attention which "local
variable" is on the stack. What you need to deal with is "private" and
"shared" veriables, not stack.
How to allocate a memory space for a variable is compiler's job. Compiler
should do that for you. In my personal opinion, gfortran fails to properly
allocate memory for variables. You can play the magic word "save" or set a
sufficient stack size, which is totally up to you.
Tobias Burnus
> How to allocate a memory space for a variable is compiler's job. Compiler
> should do that for you. In my personal opinion, gfortran fails to properly
> allocate memory for variables. You can play the magic word "save" or set a
> sufficient stack size, which is totally up to you.
I partially agree. Using -fopenmp gfortran puts the local variables in
subroutines and functions on the stack as it has no way of knowing
whether the procedure is called simultaneously in multiple threads. In
so far gfortran behaves correctly.
The gfortran buglet* is that is also puts the local variables of the
PROGRAM on the stack although it can be safely put in static memory.
I think gfortran is line with other compilers (ignoring the buglet);
if one knows that a procedure is not called simultaneously, one can
mark its variables as SAVE. I think this user friendlier than
requiring the user to mark all needed procedures as RECURSIVE. (Using
the stack for all local variables makes it more likely that the
correct result is produced although more memory might be used than
needed. In the other case, one likely gets hard to track bugs which
depends on the exact condition and vanish if one uses (e.g. for
debugging) a single thread.)
Tobias
Anony
news:c3aeb831-554a-425e...@m73g2000hsh.googlegroups.com...
>
> I partially agree. Using -fopenmp gfortran puts the local variables in
> subroutines and functions on the stack as it has no way of knowing
> whether the procedure is called simultaneously in multiple threads. In
> so far gfortran behaves correctly.
>
I agree that it is correct to put variables declared in recursive subroutine
(or function), as a default, on the stack. Programmer should know what he is
doing.
The Brosolin's example that raises this topic is a main program. The key
variables a(n,n) & b(n,n) are declared in the main program, and exist from
the beginning to the end. They are not temporary in a subroutine. Whether it
is proper for gfortran to treat a(n,n) & b(n,n) as temporary variables and
put them on the stack is an issue. If n goes up, we need a monster-sized
stack.