Problems regarding large memory: -mcmodel=medium, HYPRE, etc.

[Nadeem Malik]

Hi neks, 

From the previous thread, I have successfully compiled and produced the ppiclF+nek500 executable -- but the mystery seems to deepen here because when I run the code, there are runtime ... errors. I have tried many variations to no avail. 

The basic problem is that I want to drive the code to maxmium precision by setting lx1=lx2=lx3=15 -- this has previously worked for the nek5000 (i.e. fluid) part of the code. This requires large memory, so I had to  add the following to the makenek:

PPLIST="HYPRE"

and to the *.par file:

[PROBLEMTYPE]

equation = incompNS

[VELOCITY]

residualTol = 1e-08

density = 1.0

viscosity = 0.00766

[PRESSURE]

preconditioner = semg_amg_hypre

residualTol = 1e-08

residualProj = yes

Now I am trying to add the particle ppiclF to the code -- which I have done successfully for lx1=lx2=lx3=8. No problem with standard settings - good results.

But when I try lx1=lx2=lx3=15 (I have also tried with 13), memory problems arise.

I was advised to add " -mcmodel=medium" to the compiler options, which I did and it compiles without problems, as noted above.

In fact, I have done both, i.e. add -mcmodel=model, and also  "HYPRE" to makenek. It does not complain -- so I guess this must be ok, but some one please clarify what the difference is?

But now, I get runtime errors whatever I do. I have increased the number of ranks from 64 to 128; and tried lx1=15 and lx1=13. Here are some of the error messages that I receive:

(1) For a job with 64 ranks, lx1=15, HYPRE, mcmodel=medium:

/work2/06396/tg856952/stampede2/LocalCodeDir/TestCaseDir/Case4/nek5000: error while loading shared libraries: libpthread.so.0: cannot map zero-fill pages: Cannot allocate memory

(2) For a job with 128 ranks, lx1=13, HYPRE, mcmodel=medium:

[proxy:0:1...@c403-053.stampede2.tacc.utexas.edu] HYDU_create_process (../../utils/launch/launch.c:825): execvp error on file /work2/06396/tg856952/stampede2/LocalCodeDir/TestCaseDir/Case5/nek5000 (Argument list too long)

I've tried to run with just -mcmodel=medium, but the problem still arises.

Incidentally, is there a -mcmodel=large option (or similar)?

I would like to sort this out asap.

Thanks

Nadeem

[Nadeem Malik]

Hi all,

The problem has been resolved. The problem was in the appropriate choice of  PPICLF_LPART which is the maximum number of particles per rank! I had a value of 5000, which works with lx1=8, lxd=10, and 64 ranks on a 16^3=4096 elemental grid.

I have now successfully run the code for: lx1=10,  lxd=10, Nranks=128,  PPICLF_LPART=3000.

Incidentally. I did not the options "-mcmodel=medium" or "-mcmodel=large ", so this is a bonus -- I still have plenty of memory space to increase the simulation size, which I am sure that I will do in due time.

From now on, I shift focus to numerical analysis, accuracy, and simulation of the actual physics of the problem that I am working on. But in the meantime, I want to raise several questions -- some technical, some fundamental  -- which may be of general interest, so I will raise each question as a separate thread.

Thanks to all.

Nadeem

[nkul...@ncsu.edu]

Hi Nadeem,

Could you please tell me what type of fluid you are using for this simulation? I have been trying to set up a dimensional run with water as the fluid in a square channel of dimensions 0.01x0.01x0.1m. I can't get the simulation to work at Re = 1000. I have been using the following set up:

      subroutine useric(ix,iy,iz,ieg) ! set up initial conditions
      include 'SIZE'
      include 'TOTAL'
      include 'NEKUSE'

      ux   = 0.0
      uy   = 0.089
      uz   = 0.0

[PROBLEMTYPE]
equation = incompNS
variableProperties = no

[PRESSURE]
residualTol = 1e-05
residualProj = yes

[VELOCITY]
residualTol = 1e-05
density = 1000.0
viscosity = 0.00089
residualProj = no

The figure above shows the velocity profile after 50 s of runtime at the end of the channel. Could you please tell me if I'm missing something?

Thank you

Nilay

[N M]

Hi Nilay,

 

I implement a simpler useric:

 

c-----------------------------------------------------------------------

      subroutine useric (ix,iy,iz,eg)

      include 'SIZE'

      include 'TOTAL'

      include 'NEKUSE'

 

      common /exacu/ ue(lx1,ly1,lz1,lelt),ve(lx1,ly1,lz1,lelt), we(lx1,ly1,lz1,lelt),pe(lx2,ly2,lz2,lelt)

      common /exacd/ ud(lx1,ly1,lz1,lelt),vd(lx1,ly1,lz1,lelt), wd(lx1,ly1,lz1,lelt),pd(lx2,ly2,lz2,lelt)

 

      integer e,eg, eqnum

      integer icalld

      save    icalld

      data    icalld  /0/

 

 

      n = nx1*ny1*nz1*nelv

      if (icalld.eq.0) then

         icalld = icalld + 1

         time = 0.

         u0   = 0.0

         v0   = 0.0

         w0   = 0.0

      endif

 

      ie = gllel(ieg)

      ux=ue(ix,iy,iz,ie)

      uy=ve(ix,iy,iz,ie)

      uz=we(ix,iy,iz,ie)

      temp=0

 

      return

      end

c-----------------------------------------------------------------------

 

… and the *.par file:

 

….

[PROBLEMTYPE]

equation = incompNS

 

[PRESSURE]

preconditioner = semg_xxt

residualTol = 1e-09

residualProj = yes

 

[VELOCITY]

residualTol = 1e-09

density = 1.0

viscosity = 0.001532

residualProj = yes

 

 

I hope that this helps.

Nadeem

--
You received this message because you are subscribed to a topic in the Google Groups "Nek5000" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nek5000/D5Ya7ur3xjI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nek5000+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/3a5a5f45-630b-43bd-8063-6bd80748eec6n%40googlegroups.com.

 

[Nilay Atul Kulkarni]

Hi Nadeem,

Could you please explain what does exacu and ue, ve and we are for? And how are you calculating the velocity and fluid properties at a given Reynolds number? Currently, I was calculating the uy based on Re = 1,000, rho = 1,000 mu = 0.00089 L= 0.01. So u = Re*mu/(rho*L). 

Thank you

Nilay

[nadeem...@cantab.net]

(ue,ve,we) is from an older version of the code and was the initial velocity field. I no longer need this and I will probably take it out, so please ignore it.

Once the fluid properties are read from *.par, it remains constant in the code.

You have to set some initial velocity field -- whatever your study entails.

L=1 is believe by default in the Nek5000, unless explicitly reset -- I do not know how to reset it for a given problem, can someone clarify this point?

(On the basis of numbers, your velocity scale should be U=0.89 with L=1, and U=89 with L=0.01.)

The velocity field is automatically calculated in userchk at the start of every timestep.

Nadeem

... and the *.par file:

 

....

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/CAHqO89P0PK%3D1GiqCY9jUsvs6V2KbO0RqS3MBVFJWqMKWEorEwQ%40mail.gmail.com.

[nkul...@ncsu.edu]

Nadeem,

Thank you for your help. The case is working now.