WRF CHEM CBMZ MOSAIC ERROR

[sanjukta ghosh]

Respected Sir/Ma'am,

I am using CBMZ MOSAIC chem option (chem_opt=9) in HPC . But I am getting the error :

d01 2015-01-01_00:18:45  *** exceeded lsodes failure limit =        1000
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:      87
 *** exceeded lsodes failure limit =        1000
-------------------------------------------
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 91

I have attached the namelist and rsl.error file. I have used CBMZ-MOSAIC mapping for mozbc. Kindly help.

[Mary Barth]

Sanjuka,

That error means that the chemistry solver could not find a solution to
the set of ODEs. Likely it is because a trace gas species has a bad
mixing ratio (e.g. a value not realistic for the conditions). One thing
you can do is look at the wrfinput and wrfchemi files to see if
something looks odd. If you have a wrfout file from a time just before
the error that would be the first file to review the results to see if
there are any clues. And the rsl.out or rsl.error file can help you
locate where to look (i.e. the top of the rsl.out file tells you which
grid points are being solved by that CPU).

Good luck!
Mary

--
^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^
Mary Barth Phone: 303-497-8186
Senior Scientist email: bar...@ucar.edu

National Center for Atmospheric Research
P.O. Box 3000
Boulder, CO 80307

https://staff.ucar.edu/users/barthm
DC3 Web Site: http://www2.acom.ucar.edu/dc3/
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

[sanjukta ghosh]

But one day simulation ran fine with the same setup and same wrfchemi files. I just changed the mapping of MOZBC. Earlier I used MOZART mapping. Can that be a problem?

Sanjukta Ghosh
Student of Atmospheric Sciences
---------------------------------------------
✉: 18sanju...@gmail.com

☏: +91-6000083669


   

[Ankan Sarkar]

Dear Sanjukta,

I am also having the same error as you faced in the past with the CBMZ-MOSAIC option. My parent domain run was completed successfully. But when I am performing the nested domain simulation, I am getting this error. As the above conversation is 2 years old, I hope you solved that problem. So, can you please guide me if you know how to solve it? That will be really helpful for me. Thank you for your time and consideration.

With regards,

Ankan

[Mary Barth]

Ankan,

The original error was that the chemistry ODE could not solve the chemistry. The reason can vary from one person's simulation to another. That is why I responded with the following as there is no simple fix to this error.

That error means that the chemistry solver could not find a solution to
the set of ODEs. Likely it is because a trace gas species has a bad
mixing ratio (e.g. a value not realistic for the conditions). One thing
you can do is look at the wrfinput and wrfchemi files to see if
something looks odd. If you have a wrfout file from a time just before
the error that would be the first file to review the results to see if
there are any clues. And the rsl.out or rsl.error file can help you
locate where to look (i.e. the top of the rsl.out file tells you which
grid points are being solved by that CPU).

Mary

^--^--^--^--^--^--^--^--^--^--^--^--^--^--^-^--^--^--^-^--^

Mary Barth 

Senior Scientist   

phone: 303-497-8186    email: bar...@ucar.edu

National Center for Atmospheric Research

P.O. Box 3000

Boulder, CO 80307

Staff web page

SIMA Web Site

MUSICA Web Site

DC3 Web Site 

^--^--^--^--^--^--^--^--^--^--^--^--^--^--^-^--^--^--^-^--^

[Ankan Sarkar]

Dear Mam,

Thank you for your response. In my case, I am running WRF-Chem v3.9.1 using the CBMZ-MOSAIC chemistry option (chem_opt=32). The horizontal resolutions of my domains are 15 km and 3 km (i.e., 1:5 parent_grid_ratio), so I set time_step 4*dx, i.e., time_step=60 for the parent domain and ndown run, and time_step=12 for the nested domain run. My parent domain simulation was completed successfully. However, the nested domain run was stopped after throwing some errors in the rsl.error.0022 and rsl.out.0022 files. The errors are related to 'points exceeded cfl=2' and 'gasodesolver_lsodes failure'.  As the CFL errors are coming, I thought reducing the time step could solve the problem and gave it some test runs by setting time_step as the multiplication of 6 or less than 6 with dx (i.e, 3 km). The results of the test runs using varying time_steps are as follows:

Test1: time_step = 12 (4*dx)
"BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES" came in the wrfrun.log, and the simulation stopped after throwing some errors related to points exceeded cfl=2 and gasodesolver_lsodes failure
Test2:  time_step = 18 (6*dx)

The simulation stopped after printing the following error only at the end of rsl.error.0022 (just like Sanjukta got in the past)

" -------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:      76

 *** exceeded lsodes failure limit =        1000
-------------------------------------------

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 22" . Also, 'points exceeded cfl=2' error came.

Test3-4:  time_step = 15 (5*dx), and 9 ( 3*dx)

Seems like the simulation was stucked, as I noticed no more lines were added in the rsl files, but wrf.exe was running. So, I had to kill the simulation.  And this time also, the errors related to 'points exceeded cfl=2' and 'gasodesolver_lsodes failure' came. 

Test5:  time_step = 6 (2*dx)

This time simulation was running a little bit longer (14 minutes) than before. But, again, the simulation got stuck. This time, only the two lines of errors related to CFL are printed at the end of rsl.error.0022 file:

"6  points exceeded cfl=2 in domain d01 

MAX AT i,j,k:          294         219           3  vert_cfl,w,d(eta)=   2.85836554      0.969382226       1.29999518E-02"

Test6:  time_step = 3 (1*dx)

The simulation is still running without any error, but much slower than before. And I don't know if the simulation will be successful or not.

So, is the error coming due to time_step settings in the namelist in my case? Or anything else? Can you please guide me in this regard? That will be really helpful. Thank you for your time and consideration.

With regards,

Ankan

[Mary Barth]

Hello Ankan,

It seems to me that the CFL error is one to solve and you are doing the right thing by reducing the time step. A 3-s time step is small for dx=3km. I would hope that you could get a successful simulation with dt=9 to 12 seconds. I suggest checking the WRF forum for any other suggestions on addressing CFL errors. It's good to review your namelist in the dynamics section. Hopefully you're using advection option 2 for advection of chem, tracer, moist variables. And check the damping options. 

Good luck,

Mary

^--^--^--^--^--^--^--^--^--^--^--^--^--^--^-^--^--^--^-^--^

Mary Barth 

Senior Scientist   

phone: 303-497-8186    email: bar...@ucar.edu

National Center for Atmospheric Research

P.O. Box 3000

Boulder, CO 80307

Staff web page

SIMA Web Site

MUSICA Web Site

DC3 Web Site 

^--^--^--^--^--^--^--^--^--^--^--^--^--^--^-^--^--^--^-^--^

[Ankan Sarkar]

Dear Mam,

Thank you for getting back to me. I asked the same question at https://forum.mmm.ucar.edu/ and am still waiting for a reply. I set advection option 2 for the advection of chem, tracer, and moist variables. My namelist.input settings for the 12-second time_step are as follows:

&domains
 time_step                           = 12,
 time_step_fract_num                 = 0,
 time_step_fract_den                 = 1,
 max_dom                             = 1,
 e_we                                = 426,
 e_sn                                = 286,
 e_vert                              = 35,    
 p_top_requested                     = 5000,
 num_metgrid_levels                  = 32,
 num_metgrid_soil_levels             = 4,
 dx                                  = 3000,
 dy                                  = 3000,  
 grid_id                             = 1,  
 parent_id                           = 1,
 i_parent_start                      = 179,
 j_parent_start                      = 97,    
 parent_grid_ratio                   = 1,    
 parent_time_step_ratio              = 1,    
 feedback                            = 0,
 smooth_option                       = 0,
 interp_method_type                  = 2,
 eta_levels                          = 1.000, 0.993, 0.983, 0.970, 0.954,
                                       0.934, 0.909, 0.880, 0.845, 0.807,
                                       0.765, 0.719, 0.672, 0.622, 0.571,
                                       0.520, 0.468, 0.420, 0.376, 0.335,
                                       0.298, 0.263, 0.231, 0.202, 0.175,
                                       0.150, 0.127, 0.106, 0.088, 0.070,
                                       0.055, 0.040, 0.026, 0.013, 0.000
 /

 &physics
 mp_physics                          = 10,    10,    
 ra_lw_physics                       = 4,     4,    
 ra_sw_physics                       = 4,     4,    
 radt                                = 20,    20,    
 sf_sfclay_physics                   = 1,     1,    
 bl_pbl_physics                      = 1,     1,    
 bldt                                = 0,     0,    
 sf_surface_physics                  = 2,     2,    
 cu_physics                          = 3,     3,    
 cudt                                = 0,     0,    
 sf_urban_physics                    = 0,          
 isfflx                              = 1,
 ifsnow                              = 1,
 icloud                              = 1,
 surface_input_source                = 3,
 num_soil_layers                     = 4,
 num_land_cat                        = 21,
 do_radar_ref                        = 1,
 cu_rad_feedback                     = .true., .true.,
 slope_rad                           = 0,     0,
 topo_shading                        = 0,     0,
 cu_diag                             = 1,     1,
 progn                               = 1,     1,
 /

 &fdda
 /

 &dynamics
 w_damping                           = 0,
 rk_ord                              = 3,
 diff_opt                            = 1,      1,      
 km_opt                              = 4,      4,      
 diff_6th_opt                        = 0,      0,      
 diff_6th_factor                     = 0.12,   0.12,  
 base_temp                           = 290.
 damp_opt                            = 0,
 zdamp                               = 5000.,  5000.,  
 dampcoef                            = 0.2,    0.2,    
 khdif                               = 0,      0,      
 kvdif                               = 0,      0,  
 smdiv                               = 0.1,    0.1,
 emdiv                               = 0.01,   0.01,
 epssm                               = 0.1,    0.1,
 non_hydrostatic                     = .true., .true.,
 moist_adv_opt                       = 2,      2,          
 scalar_adv_opt                      = 2,      2,      
 chem_adv_opt                        = 2,      2,
 tke_adv_opt                         = 2,      2,
 h_mom_adv_order                     = 5,      5,
 v_mom_adv_order                     = 3,      3,
 h_sca_adv_order                     = 5,      5,
 v_sca_adv_order                     = 3,      3,
 /

 &bdy_control
 spec_bdy_width                      = 5,
 spec_zone                           = 1,
 relax_zone                          = 4,
 specified                           = .true.,
 nested                              = .false., .true.,
 /

 &grib2
 /

 &chem
 kemit                               = 1,
 chem_opt                            = 32,      32,
 bioemdt                             = 6,      6,
 photdt                              = 6,      6,
 chemdt                              = 0,       0,
 emiss_inpt_opt                      = 101,       101,
 emiss_opt                           = 4,       4,
 io_style_emissions                  = 2,
 chem_in_opt                         = 0,       0,
 phot_opt                            = 2,       2,
 gas_drydep_opt                      = 1,       1,
 aer_drydep_opt                      = 1,       1,
 bio_emiss_opt                       = 3,       3,
 gas_bc_opt                          = 1,       1,
 gas_ic_opt                          = 1,       1,
 aer_bc_opt                          = 1,       1,
 aer_ic_opt                          = 1,       1,
 gaschem_onoff                       = 1,       1,
 aerchem_onoff                       = 1,       1,
 wetscav_onoff                       = 1,       1,
 cldchem_onoff                       = 1,       1,
 vertmix_onoff                       = 1,       1,
 chem_conv_tr                        = 1,       1,
 conv_tr_wetscav                     = 1,       1,
 conv_tr_aqchem                      = 1,       1,
 seas_opt                            = 2,
 dust_opt                            = 13,
 dmsemis_opt                         = 1,
 biomass_burn_opt                    = 3,       3,
 scale_fire_emiss                    = .false., .false.,
 plumerisefire_frq                   = 60,      60,
 have_bcs_chem                       = .true.,  .true.,
 have_bcs_upper                      = .false.,  .false.,
 fixed_ubc_inname                    = "ubvals_b40.20th.track1_1996-2005.nc",
 aer_ra_feedback                     = 1,       1,
 aer_op_opt                          = 1,       1,
 ne_area                             = 400,
 opt_pars_out                        = 1,      
 chemdiag                            = 1,       1,
/

 &namelist_quilt
 nio_tasks_per_group = 0,
 nio_groups = 1,
 /

Is my namelist.input settings okay? Please take a look at it and let me know if it is correct or not. Thank you for your time and consideration.

With regards,

Ankan

[Andrea Ximena Miranda Corzo]

Hi Ankan, could yo resolve yout issue? 

I'm having the same problems. In my case I am working with 3 domains for 9, 3 and 1. Running the model for only 9 km works fine, for two domains as well but when I run the model for the 3 domains the execution stops and in the same way no mistake. 

[Ankan Sarkar]

Hi Andrea,

If you are getting CFL errors, then the first thing you should try is to run some simulations by reducing the timesteps one by one until you don't get errors. If this doesn't work for you, you should look at your namelist.input (specifically '&dynamics' option), as Mary mam suggested earlier. In my case, changing the time steps and setting the following options in '&dynamics' section of namelist.input worked for me:

&dynamics
 w_damping                           = 1,

 rk_ord                              = 3,
 diff_opt                            = 1,      1,
 km_opt                              = 4,      4,
 diff_6th_opt                        = 0,      0,
 diff_6th_factor                     = 0.12,   0.12,
 base_temp                           = 290.

 damp_opt                            = 1,

 zdamp                               = 5000.,  5000.,
 dampcoef                            = 0.2,    0.2,
 khdif                               = 0,      0,
 kvdif                               = 0,      0,
 smdiv                               = 0.1,    0.1,
 emdiv                               = 0.01,   0.01,

 epssm                               = 0.2,    0.2,

 non_hydrostatic                     = .true., .true.,
 moist_adv_opt                       = 2,      2,
 scalar_adv_opt                      = 2,      2,
 chem_adv_opt                        = 2,      2,
 tke_adv_opt                         = 2,      2,
 h_mom_adv_order                     = 5,      5,
 v_mom_adv_order                     = 3,      3,
 h_sca_adv_order                     = 5,      5,
 v_sca_adv_order                     = 3,      3,
 /

Mainly, You should check w_damping, damp_opt and epssm options in &dynamics section.

I hope that it will work for you. Good luck.

Regards,

Ankan