Pop 3 Maker Segfault

22 views
Skip to first unread message

Nathan Lai

unread,
Sep 27, 2024, 6:15:07 PM9/27/24
to enzo-dev
Hi all,
I am running into a segfault when using pop3_maker.F heres the debug:

Thread 1 "enzo.exe" received signal SIGSEGV, Segmentation fault.
0x00005555557bd6eb in pop3_maker (nx=14, ny=28, nz=<optimized out>, d=..., dm=..., h2d=..., h2diss=..., kph=..., temp=..., u=..., v=..., w=..., cooltime=..., dt=0.013145641993152993, r=..., metal=...,
    dx=0.0062499999999999917, t=2172.3355615849296, z=0, procnum=0, d1=1.6730000000000106e-20, x1=3.0857e+18, v1=9778179.1678549927, t1=315570000000, nmax=2945, xstart=0.44375000000000003,
    ystart=0.081250000000000003, zstart=0.14375000000000002, ibuff=3, imetal=0, imethod=0, h2crit=0.00050000000000000001, metalcrit=0.0001, odthresh=<optimized out>, starmass=100, level=3, np=0, xp=...,
    yp=..., zp=..., up=..., vp=..., wp=..., mp=..., tdp=..., tcp=..., metalf=..., type=..., ctype=5, justburn=0, iradtrans=0) at pop3_maker.F:195
195                    if (kph(i,j,k) .gt. tiny) goto 10
(gdb) bt
#0  0x00005555557bd6eb in pop3_maker (nx=14, ny=28, nz=<optimized out>, d=..., dm=..., h2d=..., h2diss=..., kph=..., temp=..., u=..., v=..., w=..., cooltime=..., dt=0.013145641993152993, r=..., metal=...,
    dx=0.0062499999999999917, t=2172.3355615849296, z=0, procnum=0, d1=1.6730000000000106e-20, x1=3.0857e+18, v1=9778179.1678549927, t1=315570000000, nmax=2945, xstart=0.44375000000000003,
    ystart=0.081250000000000003, zstart=0.14375000000000002, ibuff=3, imetal=0, imethod=0, h2crit=0.00050000000000000001, metalcrit=0.0001, odthresh=<optimized out>, starmass=100, level=3, np=0, xp=...,
    yp=..., zp=..., up=..., vp=..., wp=..., mp=..., tdp=..., tcp=..., metalf=..., type=..., ctype=5, justburn=0, iradtrans=0) at pop3_maker.F:195
#1  0x0000555555775089 in grid::StarParticleHandler (this=0x55555beaaff0, SubgridPointer=<optimized out>, level=<optimized out>, level@entry=3, dtLevelAbove=dtLevelAbove@entry=0.035201639594406316,
    TopGridTimeStep=0) at Grid_StarParticleHandler.C:961
#2  0x0000555555634c04 in EvolveLevel (MetaData=MetaData@entry=0x7fffffffccb0, LevelArray=LevelArray@entry=0x7fffffffcb20, level=level@entry=3, dtLevelAbove=<optimized out>,
    Exterior=Exterior@entry=0x7fffffffd020, ImplicitSolver=ImplicitSolver@entry=0x7fffffffcbf8, dt0=dt0@entry=0.12755533538867594, SiblingGridListStorage=SiblingGridListStorage@entry=0x7fffffffc780)
    at EvolveLevel.C:694
#3  0x0000555555635128 in EvolveLevel (MetaData=MetaData@entry=0x7fffffffccb0, LevelArray=LevelArray@entry=0x7fffffffcb20, level=level@entry=2, dtLevelAbove=<optimized out>,
    Exterior=Exterior@entry=0x7fffffffd020, ImplicitSolver=ImplicitSolver@entry=0x7fffffffcbf8, dt0=dt0@entry=0.12755533538867594, SiblingGridListStorage=SiblingGridListStorage@entry=0x7fffffffc780)
    at EvolveLevel.C:818
#4  0x0000555555635128 in EvolveLevel (MetaData=MetaData@entry=0x7fffffffccb0, LevelArray=LevelArray@entry=0x7fffffffcb20, level=level@entry=1, dtLevelAbove=<optimized out>,
    Exterior=Exterior@entry=0x7fffffffd020, ImplicitSolver=ImplicitSolver@entry=0x7fffffffcbf8, dt0=dt0@entry=0.12755533538867594, SiblingGridListStorage=SiblingGridListStorage@entry=0x7fffffffc780)
    at EvolveLevel.C:818
#5  0x0000555555635128 in EvolveLevel (MetaData=MetaData@entry=0x7fffffffccb0, LevelArray=LevelArray@entry=0x7fffffffcb20, level=level@entry=0, dtLevelAbove=dtLevelAbove@entry=0.12755533538867594,
    Exterior=Exterior@entry=0x7fffffffd020, ImplicitSolver=ImplicitSolver@entry=0x7fffffffcbf8, dt0=dt0@entry=0.12755533538867594, SiblingGridListStorage=SiblingGridListStorage@entry=0x7fffffffc780)
    at EvolveLevel.C:818
#6  0x0000555555629fca in EvolveHierarchy (TopGrid=..., MetaData=..., Exterior=Exterior@entry=0x7fffffffd020, ImplicitSolver=0x7fffffffcbf8, LevelArray=LevelArray@entry=0x7fffffffcb20,
    Initialdt=<optimized out>) at EvolveHierarchy.C:512
#7  0x000055555559f466 in main (argc=<optimized out>, argv=<optimized out>) at enzo.C:793


Was wondering if anyone had any suggestions. I may be just running out of memory but I hope that is not the issue.

Thanks!

Nathan Lai

unread,
Oct 6, 2024, 11:26:25 PM10/6/24
to enzo-dev
I have narrowed this down to an issue with gfortran, does anyone know what other fortran compilers work with Enzo/recommendations?

John Wise

unread,
Oct 7, 2024, 10:30:27 AM10/7/24
to enzo...@googlegroups.com

Hi Nathan,

Sorry to hear that you're having issues with the Pop III star formation routine.  It's failing when it's trying to access the LW radiation field.  It should only check this field when you have both radiation transport on and MultiSpecies > 1 (H2 cooling).  Do you have both of these parameters set like this?  Can you send your parameter file?

Thanks,

John

On 10/6/24 23:26, Nathan Lai wrote:

You don't often get email from natha...@gmail.com. Learn why this is important

--
You received this message because you are subscribed to the Google Groups "enzo-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to enzo-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/enzo-dev/3f057ed3-263e-48bb-9fa1-c2c31bd6971fn%40googlegroups.com.
-- 
John Wise
Director, Center for Relativistic Astrophysics
Professor, School of Physics
Georgia Institute of Technology
http://cosmo.gatech.edu

Matthew Abruzzo

unread,
Oct 8, 2024, 8:33:37 AM10/8/24
to enzo...@googlegroups.com
Hi Nathan,

I don't have much experience with classic Enzo (and I haven't spent any time looking into your particular issue). But, what optimization flag are you using to compile the fortran code? It might be worth it to try passing the `-O1` flag. I'm skeptical that this will fix things, but it's an easy thing to try out.

Anecdotally, I once had a Grackle-issue related to gfortran that led to segmentation faults unless I passed the `-O1` flag. That was a fairly unique bug relating to passing around passing a string from Fortran code to C routines (essentially gfortran made certain optimizations where it deallocated the memory to hold a string too early because it couldn't properly infer the lifetime of the string). Again, I'm a little skeptical that this will fix things for you, but it may be worth a try since it's easy to do. (I don't think any of Enzo's Fortran routines call C code).

What version of gfortran are you using? 

To answer your question, I think some people may use Intel's fortran compiler (ifort). But, you may want to use that with Intel's c++ compilers. (Again, I haven't really used classic Enzo so I'm not completely sure).

Let us know whether anything works.

Best,

Matthew



--

Nathan Lai

unread,
Oct 9, 2024, 11:10:15 PM10/9/24
to enzo-dev

Hi all,

Thanks for the suggestions. I was indeed missing the radiative transfer in my parameter file. However, now it appears pop3_maker.F is not being called even though it seems the criteria for a star to form are met. I will paste my parameter file below. I have just modified the CollapseTest code with the goal of simulating simple star formation. I appreciate the help, thanks again.


#
#  define problem
#
ProblemType                = 27         // Collapse test
TopGridRank                = 3
TopGridDimensions          = 20 20 20
SelfGravity                = 1          // gravity on
TopGridGravityBoundary     = 0          // periodic
LeftFaceBoundaryCondition  = 3 3 3      // periodic
RightFaceBoundaryCondition = 3 3 3
#
# problem parameters
#
CollapseTestRefineAtStart   = 1        
CollapseTestNumberOfSpheres = 1
CollapseTestUseParticles    = 1
#CollapseTestInitialTemperature =     // temperature of the background gas
CollapseTestSpherePosition[0]   = 0.5 0.5 0.5
CollapseTestSphereVelocity[0]   = 0.0 0.0 0.0
CollapseTestSphereRadius[0]     = 0.1
#CollapseTestSphereCoreRadius[0] =   // only used with sphere type 5
CollapseTestSphereDensity[0]    = 1000   // sphere density, the background density is 1
CollapseTestSphereTemperature[0] = 5    // put sphere in pressure equilibrium (rho * T is constant)
CollapseTestSphereType[0]       = 1     // constant density
                                        // 1: uniform
// 2: r^-2 power-law
// 3: NFW
// 4: Gaussian
// 5: r^-2 power-law with a core
#
#  no cosmology for this run
#
ComovingCoordinates   = 0              // Expansion OFF
#
#  units
#
DensityUnits          = 1.673e-20      // 10^4 g cm^-3
LengthUnits           = 3.0857e+18     // 1 pc in cm
TimeUnits             = 3.1557e+11     // 10^4 yrs
GravitationalConstant = 1.39698e-3     // 4*pi*G_{cgs}*DensityUnits*TimeUnits^2

#
#  set I/O and stop/start parameters
#
StopTime          = 10000
dtDataDump        = 100
StopCycle   = 999999999
NewMovieParticleOn = 1
DataDumpDir       = DD
DataDumpName      = DD
OutputTemperature = 1                  // Output temperature field.
OutputGriddedStarParticle = 1
#
#  set hydro parameters
#
Gamma                       = 1.6667
PPMDiffusionParameter       = 0        // diffusion off
DualEnergyFormalism         = 1        // use total & internal energy
InterpolationMethod         = 1        // SecondOrderA
CourantSafetyNumber         = 0.5
FluxCorrection              = 1
ConservativeInterpolation   = 1
HydroMethod                 = 0        // PPM
RiemannSolver = 5
ParticleCourantSafetyNumber = 0.8
RiemannSolverFallback = 1
UseMinimumPressureSupport   = 0

#
#  chemistry/cooling
#
MultiSpecies                = 2        
RadiativeCooling            = 1  
RadiativeTransfer = 1


     
#
#  set grid refinement parameters
#
StaticHierarchy                       = 0
MaximumRefinementLevel                = 3
MaximumGravityRefinementLevel         = 3
MaximumParticleRefinementLevel        = 4
RefineBy                              = 2
CellFlaggingMethod                    = 2 4
MinimumOverDensityForRefinement       = 8.0 8.0
MinimumMassForRefinementLevelExponent = 0.0 0.0
MinimumEfficiency                     = 0.35




#
# star formation parameters
#
StarParticleCreation          = 8
#StarParticleFeedback          = 8
StarMakerOverDensityThreshold = 100
StarMakerMassEfficiency       = 0.25
StarMassEjectionFraction      = 0.25
StarMetalYield                = 0.1
StarEnergyToThermalFeedback   = 1e-5
#StarFeedbackDistRadius        = 1
#StarFeedbackDistCellStep      = 3

John Wise

unread,
Oct 11, 2024, 10:39:28 AM10/11/24
to enzo...@googlegroups.com

Hi Nathan,

I made a few changes to your parameter file, and when I run it, one Pop III star particle forms at t ~ 1.71. The main problem was that you set the star particle creation density to the initial density of the cloud.  In that case, all of the cells inside the cloud will trigger star particle creation, but there is a hard-coded safety check in the Pop III formation routine to prevent many star particles from forming within some radius.

I changed the SF density criterion from 1000 to 1e5, allowing the cloud to collapse before it forms stars, which is a more realistic setup.  I also changed the Courant factor from 0.5 to 0.25, which is needed with feedback, and the StopTime to 2.5 and output cadence to 0.25.  I'm attaching the modified parameter file, so you can test it out.  The Pop III star particles will have its "particle_type" field equal to 5.  When it forms, the stdout should give you the following information

"""

StarParticle[birth]: L3, r = 0.00625 pc, M = 1017, Z2/Z3 = 0/0
         mass = 100 (100% cold) SolarMass,
         rho = 9.77818e+06 g/cm3, tdyn = 0 Myr
         vel = -0.000000 -0.000000 -0.000000 (-0.000000 -0.000000 -0.000000)
         pos = 0.500000 0.500000 0.500000

"""

I hope this works for you, and let us know if you have any more questions.

Thanks,

John

P3.enzo

Nathan Lai

unread,
Oct 11, 2024, 4:04:11 PM10/11/24
to enzo-dev
Hi all,

Thanks. I was able to replicate this successfully. Would it be possible to upscale this simulation to get more stars, create conditions similar to a nebula/large molecular cloud, or would that be better served with a different setup, perhaps a cosmological simulation?

John Wise

unread,
Oct 14, 2024, 2:20:03 PM10/14/24
to enzo...@googlegroups.com

Yes, you can use the CollapseTest to simulate a metal-free molecular cloud instead of a cosmological simulation.  I'd suggest using a published setup and trying to reproduce their results.  Most of the parameters are there to control the initial conditions of the cloud.  The only thing that is lacking is a good turbulence generator.  The one in CollapseTest only does local random motions, but the MHD CollapseTest generates turbulent conditions from some power spectrum.  I'm not too familiar with the latter.

Thanks,

John

Nathan Lai

unread,
Oct 15, 2024, 12:24:33 PM10/15/24
to enzo-dev
Sorry to bother you again,

I worry that I am misunderstanding this Enzo output. The output file (in this case DD0018.hierarchy) says "NumberOfStarParticles      = 72" However when I look in DD0018.cpu0000, there is only 1 listed. Am I misinterpreting NumberOfStarParticles?

John Wise

unread,
Oct 15, 2024, 4:48:04 PM10/15/24
to enzo...@googlegroups.com

No worries. When Pop III star particles are created, it merges any particles together within some radius in a given timestep. Since the bookkeeping of particle IDs in Enzo makes it difficult to delete particles, we just set the particle masses to a very small number.  The NumberOfStarParticles parameter is used to set the star particle IDs, and we only increment it.  This is why that .hierarchy entry is larger than the actual star particles.  You should use the data provided with yt / .cpuXXXX files.

Thanks,

John

Reply all
Reply to author
Forward
0 new messages