Initiating problems -- how to stagger initial and flow and particle seeding?

[Nadeem Malik]

HI,

I have come up against a specific problem regarding initiating my flow with seeded particles. The problem is that initing both the flow and thousands of particles simultaneously causes the code to crash. 

(I have seen this by first turning off the particles -- even then the flow converges with an initial very small timestep. When I introduce a small number of particles, the code only just converges taking a huge amount of CPU on just the first timestep. For larger numbers of particles the code crashes.)

Thus, what I would like to do is to introduce the particles after the flow has first converged -- (maybe after just the first timestep?). Here is a schema that I have in mind:

(1)  Start the initial flow configuration at t=0, with small timestep dt=1E-04 (or smaller).

(2)  Wait for code to converge, after a few timesteps -- maybe Nt1=1 - 10.

(3)  Then, at Nt1+1, introduce the particles (ppiclF). The ppiclF is cuploed to the Nek5000 via subrutine usrdat2 -- so essentially usrdat2 must be called first at N=Nt1+1, and subsequently thereafter.

(4) After the ppiclF+Nek5000 has converged, say at timestep N=Nt2, I want to increase the timestep Dt to a more convenient larger value. 

(5)  And I want to reset my time to t=0 at N=Nt2; and this means recording at regular times,  writeinterval = 100, after this (Otherwise there would be a effective offset to the time).

Can someone help out here please?

Thanks

Nadeem

[Fischer, Paul]

Hi Nadeem,

You should be able to run your code without particles and save the output.

Then restart.  You can reset the time when you restart if you wish, via:

            1 PRESOLVE/RESTART OPTIONS  *****

myfile0.f00001 time=0.0

hth,

Paul

---

From: nek...@googlegroups.com <nek...@googlegroups.com> on behalf of Nadeem Malik <nadeem....@gmail.com>
Sent: Sunday, December 19, 2021 12:28 AM
To: Nek5000 <nek...@googlegroups.com>
Subject: [nek5000] Initiating problems -- how to stagger initial and flow and particle seeding?

 

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/299c0c97-19cd-4b3d-a876-9ba2c31d26f1n%40googlegroups.com.

[Nadeem Malik]

Hi Paul,

The situation is a bit more complicated. I just noticed that if I turn off the particle seeding, the solver is bypassed altogether (?!) and returns trivial zero velocity etc.

Currently, if I seed with a small number of particles, say 12,800, then the ppiclF+nek5000 converges with dt=1E-04 after about 10 steps. 

If I try to seed with say 50,000 particles, the solver crashes.

Thus, the simplest would be to start off with a small number of particles, even npart=1, and let the flow converge after say N=100 steps, so that T1=1E-02.

But then, I need a method to add particles at any random time, say all at once with 10^5 particle at N=101, or stagger the seeding itself at periodic intervals until it has all converged. 

I recon that that this could all be done within N=400 steps with dt=1E-04. (But I will test with smaller dt still and see what happens.)

After N=500 (or so?) I must recenter to N=0, and t=0.

Can you help out here?

Thanks

Nadeem

[Nadeem Malik]

Hi neks,

Some good news, I have progressed significantly on this problem in the last hour.

(1) First, I can confirm that this problem is very much an initial configuration issue -- because when I changed the timestep down to dt=1E-05 -- the first iteration at t=0 returned identical convergence and error levels as before:

 call userchk
          0  Error Hmholtz psi      900   2.5079E-07   7.3732E+00   1.0000E-09
          0  Error Hmholtz psi      900   3.9999E-05   7.2968E+00   1.0000E-09
          0  Error Hmholtz psi      900   6.2106E-07   1.0265E+00   1.0000E-09
   0    0.000000E+00  t Time
  11    6.005508E-02  2.691722E-02  1.455912E-03  Div er
  11    2.656657E+00  3.551325E+00  5.473797E-01  2.000102E+00  3.464260E+00  2.000178E+01 u,V err

(2) Secondly, on a hunch that the convergence problem was aggravated by the physical size and distribution of initial particle seeding, I (i) reduced the diameter of each particle from dp=2E-03 to 1E-03;  (2) I increased the distance between particles from dx=0.02 (each coord) to dx=0.05 (total distance of about 0.09). This did the trick! I have now gone up from 12,800 particles to 204,800 particles in my [-pi,pi]^3 domain -- I could probably go higher.

This is on a grid of 20^3 spectral elements, using 128 cores (ranks). 

I will eventually go to 24^4 or even 30^3 -- then there will be many more ranks -- maybe an order of magnitude more -- on which to distribute the particles.

For the moment, a dilute distribution 200,000 particles (hopefully more) is OK -- I can start at this end of the parametric study.

However, eventually I would like to seed clusters of hundreds of particles close together in chose regions -- my scientific  interest is in particle-flow structure interactions.

So the problem of how to release particles staggered, or continuously in time, is still important -- so can I ask the experts to resolve this issue for me (and I suspect others) as soon as possible.

There are some residual points of interest with respect to error messages and tolerances. For example, I see:

             L1/L2 DIV(V)           -1.8960E-16   1.8436E-03
             L1/L2 QTL                4.0471E-08    2.2188E-03
             L1/L2 DIV(V)-QTL  -4.0471E-08    1.1301E-03

What are these metrics referring to? And what is a 'good' value for them be?

My dt=1E-06. Here is a sample of the log file:

        0  0.0000E+00 Write checkpoint
       FILE:
 /work2/06396/tg856952/stampede2/LocalCodeDir/TestCaseDir/Run_20-10-01/vorunifor
 m0.f00001

        0  0.0000E+00 done :: Write checkpoint
                              file size =     370.E+00MB
                              avg data-throughput =   191.6MB/s
                              io-nodes =   128

div: davg:  0.0000E+00  0.0000E+00  0.0000E+00
        0  0.00000E+00  0.00000E+00  0.00000E+00     Infinity     Infinity  cdiv
        0        0 -1.259E+00  1.268E+00  0.000E+00  0.000E+00   Infinity  0.000E+00divmnmx
PPICLF: *********** PPICLF OUTPUT *****************
PPICLF:  *Begin General Info
PPICLF:   -Cycle                  :         0
PPICLF:   -Output Freq.           :         5
PPICLF:   -Time                   :   0.000000E+00
PPICLF:   -dt                     :   1.000000E-06
PPICLF:   -Global particles       :    204800
PPICLF:   -Local particles (Max)  :     15400
PPICLF:   -Local particles (Min)  :         0
PPICLF:   -Local particles (Ideal):      1600
PPICLF:   -Total ranks            :       128
PPICLF:   -Problem dimensions     :         3
PPICLF:   -Integration method     :         1
PPICLF:   -Number of bins total   :       125
PPICLF:   -Number of bins (x)     :         5
PPICLF:   -Number of bins (y)     :         5
PPICLF:   -Number of bins (z)     :         5
PPICLF:   -Bin xl coordinate      :  -3.141593E+00
PPICLF:   -Bin xr coordinate      :   3.141593E+00
PPICLF:   -Bin yl coordinate      :  -3.141593E+00
PPICLF:   -Bin yr coordinate      :   3.141593E+00
PPICLF:   -Bin zl coordinate      :  -3.141593E+00
PPICLF:   -Bin zr coordinate      :   3.141593E+00
PPICLF:   End General Info
PPICLF:  *Begin Ghost Info
PPICLF:   -Global ghosts          :         0
PPICLF:   -Local ghosts (Max)     :         0
PPICLF:   -Local ghosts (Min)     :         0
PPICLF:   End Ghost Info
PPICLF:  *Begin Grid Info
PPICLF:   -Orig. Global cells     :      8000

PPICLF:   -Orig. Local cells (Max):        63
PPICLF:   -Orig. Local cells (Min):        62
PPICLF:   -Map Global cells       :     17064
PPICLF:   -Map Local cells (Max)  :       184
PPICLF:   -Map Local cells (Min)  :         0
PPICLF:   End Grid Info

        0  0.0000E+00 Write checkpoint
       FILE:
 /work2/06396/tg856952/stampede2/LocalCodeDir/TestCaseDir/Run_20-10-01/ptwunifor
 m0.f00001

        0  0.0000E+00 done :: Write checkpoint
                              file size =     423.E+00MB
                              avg data-throughput =   196.4MB/s
                              io-nodes =   128

 done :: userchk

gridpoints unique/tot:      10648000     13824000

 Initialization successfully completed    241.55     sec

Starting time loop ...

     DT/DTCFL/DTFS/DTINIT   0.100E-05   0.000E+00   0.000E+00   0.100E-05
Step      1, t= 1.0000000E-06, DT= 1.0000000E-06, C=  0.001 0.0000E+00 0.0000E+00
             Solving for fluid
 New CG1-tolerance (Neumann) =   2.027793701767070E-009
 New CG1-tolerance (Neumann) =   2.027793701767070E-009
    1 1.00000E-09 1.93766E+00 1.04826E+01 1.84846E-01       1 Divergence
    2 1.00000E-09 5.95294E-01 1.04826E+01 5.67890E-02       1 Divergence
    3 1.00000E-09 1.65180E-01 1.04826E+01 1.57577E-02       1 Divergence
    4 1.00000E-09 5.56430E-02 1.04826E+01 5.30816E-03       1 Divergence
    5 1.00000E-09 2.12153E-02 1.04826E+01 2.02387E-03       1 Divergence
    6 1.00000E-09 7.21750E-03 1.04826E+01 6.88525E-04       1 Divergence
    7 1.00000E-09 2.65618E-03 1.04826E+01 2.53391E-04       1 Divergence
    8 1.00000E-09 8.98624E-04 1.04826E+01 8.57257E-05       1 Divergence
    9 1.00000E-09 3.00934E-04 1.04826E+01 2.87081E-05       1 Divergence
   10 1.00000E-09 1.00568E-04 1.04826E+01 9.59385E-06       1 Divergence
   11 1.00000E-09 3.65411E-05 1.04826E+01 3.48589E-06       1 Divergence
   12 1.00000E-09 1.21528E-05 1.04826E+01 1.15933E-06       1 Divergence
   13 1.00000E-09 4.01449E-06 1.04826E+01 3.82968E-07       1 Divergence
   14 1.00000E-09 1.35404E-06 1.04826E+01 1.29171E-07       1 Divergence
   15 1.00000E-09 4.48609E-07 1.04826E+01 4.27957E-08       1 Divergence
   16 1.00000E-09 1.53761E-07 1.04826E+01 1.46683E-08       1 Divergence
   17 1.00000E-09 5.14530E-08 1.04826E+01 4.90844E-09       1 Divergence
   18 1.00000E-09 1.93103E-08 1.04826E+01 1.84214E-09       1 Divergence
   19 1.00000E-09 7.21820E-09 1.04826E+01 6.88592E-10       1 Divergence
   20 1.00000E-09 2.36338E-09 1.04826E+01 2.25458E-10       1 Divergence
   21 1.00000E-09 8.28725E-10 1.04826E+01 7.90575E-11       1 Divergence

          1  PRES gmres        21   8.2872E-10   1.0483E+01   1.0000E-09   7.9258E-01   1.4079E+00    F
          1  Hmholtz VELX       1   9.4973E+00   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELX       2   1.1010E-03   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELX       3   2.4603E-07   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELX       4   5.6786E-11   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELX       3   5.6786E-11   9.4973E+00   1.0000E-09
          1  Hmholtz VELY       1   1.1805E+01   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELY       2   1.2788E-03   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELY       3   2.7341E-07   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELY       4   5.7184E-11   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELY       3   5.7184E-11   1.1805E+01   1.0000E-09
          1  Hmholtz VELZ       1   8.5999E+00   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELZ       2   1.1186E-03   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELZ       3   2.5847E-07   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELZ       4   6.2952E-11   7.6600E-03   1.0000E-09   1.0000E+06   F
          1  Hmholtz VELZ       3   6.2952E-11   8.5999E+00   1.0000E-09
             L1/L2 DIV(V)          -1.8917E-16   1.7135E-03
             L1/L2 QTL              3.2635E-10   2.2200E-03
             L1/L2 DIV(V)-QTL      -3.2635E-10   1.4062E-03
          1  Fluid done  1.0000E-06  1.9463E+00
        1  1.00000E-06  0.00000E+00  0.00000E+00     Infinity     Infinity  cdiv
        1        0 -1.268E+00  1.276E+00  0.000E+00  0.000E+00   Infinity  0.000E+00divmnmx
Step      2, t= 2.0000000E-06, DT= 1.0000000E-06, C=  0.001 1.3514E+02 1.3514E+02
             Solving for fluid
          2  PRES gmres        22   5.7682E-10   1.4188E+01   1.0000E-09   7.6861E-01   1.3727E+00    F
          2  Hmholtz VELX       3   7.5901E-12   2.0631E+00   1.0000E-09
          2  Hmholtz VELY       3   6.4960E-12   2.1591E+00   1.0000E-09
          2  Hmholtz VELZ       3   4.6088E-12   1.4918E+00   1.0000E-09
             L1/L2 DIV(V)          -1.8945E-16   1.7379E-03
             L1/L2 QTL              3.2324E-09   2.2207E-03
             L1/L2 DIV(V)-QTL      -3.2324E-09   1.2939E-03
          2  Fluid done  2.0000E-06  1.8904E+00
        2  2.00000E-06  0.00000E+00  0.00000E+00     Infinity     Infinity  cdiv
        2        0 -1.270E+00  1.279E+00  0.000E+00  0.000E+00   Infinity  0.000E+00divmnmx
Step      3, t= 3.0000000E-06, DT= 1.0000000E-06, C=  0.001 2.6982E+02 1.3467E+02
             Solving for fluid
          3  PRES gmres        21   5.4393E-10   7.8997E+00   1.0000E-09   7.3393E-01   1.3009E+00    F
          3  Hmholtz VELX       3   4.8131E-12   2.4302E+00   1.0000E-09
          3  Hmholtz VELY       3   4.1281E-12   2.5525E+00   1.0000E-09
          3  Hmholtz VELZ       3   2.9701E-12   1.7118E+00   1.0000E-09
             L1/L2 DIV(V)          -1.8876E-16   1.7799E-03
             L1/L2 QTL              1.3456E-08   2.2201E-03
             L1/L2 DIV(V)-QTL      -1.3456E-08   1.2084E-03
          3  Fluid done  3.0000E-06  1.8173E+00
        3  3.00000E-06  0.00000E+00  0.00000E+00     Infinity     Infinity  cdiv
        3        0 -1.271E+00  1.280E+00  0.000E+00  0.000E+00   Infinity  0.000E+00divmnmx
Step      4, t= 4.0000000E-06, DT= 1.0000000E-06, C=  0.001 4.0445E+02 1.3464E+02
             Solving for fluid

          4  PRES gmres        20   7.0281E-10   3.3339E+00   1.0000E-09   6.9923E-01   1.2314E+00    F
          4  Hmholtz VELX       3   4.1956E-12   2.2054E+00   1.0000E-09
          4  Hmholtz VELY       3   3.5940E-12   2.2866E+00   1.0000E-09
          4  Hmholtz VELZ       3   2.6383E-12   1.4223E+00   1.0000E-09
             L1/L2 DIV(V)          -1.8964E-16   1.8180E-03
             L1/L2 QTL              2.6998E-08   2.2194E-03
             L1/L2 DIV(V)-QTL      -2.6998E-08   1.1579E-03
          4  Fluid done  4.0000E-06  1.7473E+00
        4  4.00000E-06  0.00000E+00  0.00000E+00     Infinity     Infinity  cdiv
        4        0 -1.271E+00  1.281E+00  0.000E+00  0.000E+00   Infinity  0.000E+00divmnmx
Step      5, t= 5.0000000E-06, DT= 1.0000000E-06, C=  0.001 5.3895E+02 1.3450E+02
             Solving for fluid
          5  PRES gmres        17   6.4232E-10   5.9034E-01   1.0000E-09   5.9304E-01   1.0297E+00    F
          5  Hmholtz VELX       3   3.5713E-12   2.0290E+00   1.0000E-09
          5  Hmholtz VELY       3   3.0723E-12   2.0674E+00   1.0000E-09
          5  Hmholtz VELZ       3   2.2586E-12   1.1560E+00   1.0000E-09
             L1/L2 DIV(V)          -1.8960E-16   1.8436E-03
             L1/L2 QTL              4.0471E-08   2.2188E-03
             L1/L2 DIV(V)-QTL      -4.0471E-08   1.1301E-03
          5  Fluid done  5.0000E-06  1.6500E+00

        5  5.0000E-06 Write checkpoint
       FILE:
 /work2/06396/tg856952/stampede2/LocalCodeDir/TestCaseDir/Run_20-10-01/vorunifor
 m0.f00002

        5  5.0000E-06 done :: Write checkpoint
                              file size =     370.E+00MB
                              avg data-throughput =   195.1MB/s
                              io-nodes =   128

        5  5.00000E-06  0.00000E+00  0.00000E+00     Infinity     Infinity  cdiv
        5        0 -1.271E+00  1.280E+00  0.000E+00  0.000E+00   Infinity  0.000E+00divmnmx

Thanks

Nadeem