"ASTEM internal steps exceeded 200" error when running wrf.exe with MOZART+MOSAIC

[ying...@sjsu.edu]

Hello, WRFchemhelp,

I got error msg: "ASTEM internal steps exceeded         200" when running wrf.exe with MOZART +MOSAIC 

I attached the rsl.error.0000 and my namelist.input. 

I use chem_opt =201 and set my namelist.input follow https://www2.acom.ucar.edu/sites/default/files/wrf-chem/MOZART_MOSAIC_V3.6.readme_dec2016.pdf and 2015 best practice. But I feel the error is related to my namelist.input setting. Please help me check. 

Thank you very much! 

Ying

[Gabriele Pfister]

Hi Ying, 

this error happens in the MOSAIC aerosol routine and we are unfortunately not able to help you in this case. MOSAIC has been developed by PNNL and I hope they have an answer to your problem. 

Gabriele

--

================================
Gabriele Pfister

Atmospheric Chemistry Observations & Modeling

National Center for Atmospheric Research

Email: pfi...@ucar.edu

Phone: +1 303 497 2915

Web: https://staff.ucar.edu/users/pfister

[ying...@sjsu.edu]

Thanks for helping me contact PNNL people, Gabi!

Hi, Jerome, I tried to use different physics scheme and urban physical option, the error is still the same. Model crashed after running about 5 mins. I think it is a time step problem but don't sure how to fix it. Please help me check it. 

Thank you very much!

Ying

[Ying Song]

Thanks for the reply, Jerome!

I reduced both time step(36 ->12) and chemdt (6.0, 6.0 ->2.0, 2.0). The model crashed even earlier, after running 2 mins. 

I checked the rsl.error.0000 and do find some error msg at the initial time, such as:

*****************************************

d01 2016-07-07_00:00:00 module_io.F (md_calls.m4) : in wrf_get_dom_ti_real_sca

d01 2016-07-07_00:00:00  NetCDF error: NetCDF: Attribute not found

d01 2016-07-07_00:00:00  NetCDF error in ext_ncd_get_dom_ti.code REAL, line          83  Element STAND_LON

d01 2016-07-07_00:00:00  input_wrf: wrf_get_dom_ti_real for STAND_LON returns   -123.0000

d01 2016-07-07_00:00:00 module_io.F (md_calls.m4) : in wrf_get_dom_ti_real_sca

d01 2016-07-07_00:00:00  NetCDF error: NetCDF: Attribute not found

d01 2016-07-07_00:00:00  NetCDF error in ext_ncd_get_dom_ti.code REAL, line          83  Element POLE_LAT

d01 2016-07-07_00:00:00  input_wrf: wrf_get_dom_ti_real for POLE_LAT returns    90.00000

......

 date 2016-07-07_00:00:00

 ds            1           1           1

 de          102           8         102

 ps            1           1           1

 pe            7           8           7

 ms           -4           1          -4

 me           14           8          14

d01 2016-07-07_00:00:00 module_io.F: in wrf_read_field

d01 2016-07-07_00:00:00  NetCDF error: NetCDF: Variable not found

d01 2016-07-07_00:00:00  NetCDF error in wrf_io.F90, line        2883  Varname E_ORGI_A

 input_wrf.F reading 4d real E_ORGI_A Status =        -1021

input_wrf.F reading 4d real E_ORGJ_A

 date 2016-07-07_00:00:00

 ds            1           1           1

 de          102           8         102

 ps            1           1           1

 pe            7           8           7

 ms           -4           1          -4

 me           14           8          14

d01 2016-07-07_00:00:00 module_io.F: in wrf_read_field

d01 2016-07-07_00:00:00  NetCDF error: NetCDF: Variable not found

d01 2016-07-07_00:00:00  NetCDF error in wrf_io.F90, line        2883  Varname E_ORGJ_A

 input_wrf.F reading 4d real E_ORGJ_A Status =        -1021

input_wrf.F reading 4d real E_CO_BB

*******************************************************

E_ORGI_A and E_ORGJ_A are  "Additional CBMZ and MOSAIC emission variables"

Does that mean there are missing variables in my wrfinput? I run WRF-Chem with MOZART+MOSAIC, why I need CBMZ emission variables.  Or my namelist.input setting has problem?   

Thank you so much for the help. 

Ying

On Fri, Mar 2, 2018 at 11:20 AM, Fast, Jerome D <Jerom...@pnnl.gov> wrote:

We have tested MOSAIC for a wide range of conditions, and our group rarely encounters the error message in ASTEM you encountered.  Generally, if the code is crashing in MOSAIC is is likely something may be going badly elsewhere.  You will need to look at the aerosol, trace gas, and meteorology fields at the time step before the code crashes to make sure that they look realistic.  It could be just one grid cell is going bad with some sort of bad input.

 

You could try and change your timestep, but you should also try changing the chemdt timestep.

 

Jerome

[Ying Song]

Hi, don't bother by the last email I replied. 

I rerun the real.exe and find the "NetCDF error: NetCDF: Attribute not found" errors already exist when running real.exe. I guess my met_em* files has something wrong.  Let me work on it first. Will submit the question through forum again if I can not solve it. 

Thanks! 

Ying

[Gabriele Pfister]

Ying 

these are warnings that show up frequently but not necessarily mean a problem. As Jerome suggested, closely examine the output files before your model crashed for any crazy chemical or meteorological variables and look at the input files for this time step. 

You stated: chemdt (6.0, 6.0 ->2.0, 2.0). Does this mean you run with the same chemistry time step for both domains? You should have a smaller time step for the inner domain. E.g., if your d01 is 12 km and d02 is 4km (which I believe it is) I suggest:
time_step = 72
chemdt = 1.2, 0.4

Gabriele

[ying...@sjsu.edu]

Hello,

Thanks for the reply. 

Yes. I adjusted the time step and chemdt as suggested. Here are some new details about this error. It's a KPP runtime error. I got the error msg:

.......

Forced exit from Rosenbrock due to the following error:

 Step size too small

 T=  4.778828614911929E-023 and H=  4.778828614911929E-023

 Forced exit from Rosenbrock due to the following error:

 Step size too small

 T=  4.778177334672780E-026 and H=  4.778177334672780E-026

 Forced exit from Rosenbrock due to the following error:

 Step size too small

 T=  4.778813111895551E-026 and H=  4.778813111895551E-026

......

I find related poster online saying some reaction rates are too small.(http://forum.wrfforum.com/viewtopic.php?f=41&t=8919). The mozart_mosaic_4bin.eqn in/chem/KPP/mechanisms/ needs to be modified. Not sure if it is the correct way to fix? So I write this email to ask before I cut any reactions in the script. 

Attachments are rsl files, namelist.input and mozart_mosaic_4bin.eqn file.

Thank you very much! 

Ying

[Gabriele Pfister]

Ying Song, 

the problem is not in the chemistry but rather lies in your setup and/or inputs. You should not modify any of the KPP files unless you have a very good scientific reason to do so. 

I took a look at your namelist.input. You run for 2 domains with 12km for d01 and 4km for d02, respectively. Your setting of time_step = 72 is correct, i.e you have 1.2 min for d01 and 0.4 min for d02. But then you set chemdt = 6, 2; five times higher than time_step. 

Try running with chemdt the same as time_step and see how this works. Also carefully check your inputs and the output created just before the model halts. 

Gabi

--

================================
Gabriele Pfister, Deputy Director

Atmospheric Chemistry Observations & Modeling

National Center for Atmospheric Research

Email: pfi...@ucar.edu

Phone: +1 303 497 2915

Web: https://staff.ucar.edu/users/pfister

[Ying Song]

Thanks for the reply, Gabi, 

Yes, I tried to set chemdt =1.2, 0.4, the error is still the same. Looks like the error is related to KPP? Yes. I don't want to edit any chemistry unless I clearly  understand the error. The error occurs after model running 24 sec , The time step is 72, so the problem should be in input data. I checked the wrfinput, meteo fields are okay. Ozone concentration shows some odd pattern at surface in wrfinput. So I should double WPS and mozbc?   

---------------------------------------------------------   

d02 2016-05-20_00:00:24 calling conv transport for chemical species

d02 2016-05-20_00:00:24 calling calc_het_n2o5

d02 2016-05-20_00:00:24 chem_driver(02): Calling dchm_tstep_init

d02 2016-05-20_00:00:24 calling kpp_mechanism_driver

d02 2016-05-20_00:00:24 kpp_mechanism_driver: calling mozart_mosaic_4bin_interface

 Forced exit from Rosenbrock due to the following error:

 Step size too small

 T=  1.291853748993375E-029 and H=  1.207075151954502E-029

 Forced exit from Rosenbrock due to the following error:

 Step size too small

 T=  1.291605233216736E-029 and H=  1.291605233216736E-029

--------------------------------------

Ying

[Stacy Walters]

Ying,

This error is from the gas phase solver routines associated with KPP.  This error

indicates that there is something seriously out of order with either the reaction

rate constants or the initial gas phase concentrations or both.  

Stacy

[Stacy Walters]

Ying,

Could you please send your latest namelist.input file as an attachment?

Stacy

[Ying Song]

Hi Stacy,

Thanks for your reply! Here is the newest namelist.input

Ying

[Ying Song]

Hello, I checked the wrfinput data and find some abnormal chemistry data near the boundary. So I tried to run only one and smaller domain to avoid the initial abnormal data.

I recompiled WRF and compile.log is attached. all .exe files are generated fine. but there is an error msg:"

 Found file chem/KPP/mechanisms/cbm4/cbm4_wrfkpp.equiv

ERROR: variable name for third body in KPP species file is expected to be  M, but was not found in cbm4 species file

"

Someone posted the same error here:(https://groups.google.com/a/ucar.edu/forum/?hl=en#!topic/wrf-chem/o9fSgUudd_o) and Stacy replied that it is not a problem. I just run WRF with my new configuration. However, I still got the same error after 48 sec model run:

***********************************

d01 2016-07-27_00:00:48 calling kpp_mechanism_driver

d01 2016-07-27_00:00:48 kpp_mechanism_driver: calling mozart_mosaic_4bin_interface

 Forced exit from Rosenbrock due to the following error:

 Step size too small

 T=  2.357378159939805E-018 and H=  2.357378159939805E-018

 Forced exit from Rosenbrock due to the following error:

 Step size too small

......

......

 T=  2.434632994509025E-022 and H=  2.434632994509025E-022

 Forced exit from Rosenbrock due to the following error:

 Step size too small

 T=  2.437667227292376E-018 and H=  2.437667227292376E-018

d01 2016-07-27_00:00:48 chem_driver(01): Calling dchm_tstep_set

d01 2016-07-27_00:00:48 no gocart so2-so4 conversion

d01 2016-07-27_00:00:48 aerosols_driver calling mosaic_aerchem_driver

 ASTEM internal steps exceeded         200

 ASTEM internal steps exceeded         200

***********************************

Attachment is my latest namelist.input, rsl.out msg, and additional fort.67 generated when running wrf.exe.

Stacy, could you check this newest namelist if you haven't gotten chance to see the previous one. Also, I use NAM 12km data as initial and set res = 4km in namelist.input. 

Not sure what is next direction I should try to fix this problem. Any comments and suggestions are highly appreciated. Thank you very much. 

Ying

[Gabriele Pfister]

Ying, 

if you mention strange looking values already in your wrfinput file, then this tells me that there is some problem already in your setup. I suggest starting fresh, recreating your wrfinput and wrfbdy files, carefully check them and then try again. 

Gabi

[Ying Song]

Thanks for the reply, Gabi, 

Yes and I have already redo everything with smaller domain to avoid the odd initial value in wrfinput. The wrfinput file looks fine this time. However, the problem is still the same. The namelist and rsl files I attached in last email are from the new run.  

I will redo everything, from compiling, on Cheyenne. hope the problem can be fixed. 

Ying 

[Xin Zhang]

Hi YingSong and Gabi,

I got the same error as YingSong's:

Forced exit from Rosenbrock due to the following error:

 No of steps exceeds maximum bound
 T=   72.0000000000000      and H=   72.0000000000000

 Forced exit from Rosenbrock due to the following error:

 No of steps exceeds maximum bound
 T=   72.0000000000000      and H=   72.0000000000000

After changing debug_level to 1000, I find this error showed up after calling kpp_mechanism_driver.

I attached my namelist.input (the new mechanism r2smh works fine on other machine), rsl.error.0000 and rsl.out.0000.

2014_prep.log file contains some information about min,mean,max of different substances.

Could you help me with this problem?

Best regards,

Xin

[Gabriele Pfister]

Hi, 

what this tells is that the model fails in solving the chemical equations. There is somewhere a problem in your inputs or your configuration. Sorry, that is really all I can do from our side. 

Gabriele

[Xin Zhang]

Thank you Gabriele!

I compiled WRF-Chem without KPP and it works fine. But, the input files are as same as before.

So, it's related to KPP. I'll read about the KPP code tomorrow and try to figure out why.

Regards,

Xin

[Xin Zhang]

Hi Gabriele,

Finally, I find that it's related to optimization level.

Setting the optimization level from '-O2' to '-O1' works.

You can check that here.

Cheers,

Xin

[Caterina Mogno]

Hi,

I am actually experiencing the same "ASTEM internal steps exceeded 200" and "Forced exit from Rosenbrock" when running ./wrf.exe. 

I am using WRF-Chem 4.0.2, compiled with gfortran, and using chem_opt=202.

Since I red the thread it may be an input problem. When I check the wrfinput_d01 and wrfbdy_d01 created with MOZBC (using outputs from MOZART-4), I see that some fields are empty (eg PM2.5_DRY)

It is necessary to have a 1 to 1 corrispondence between the variables in the  wrfinput_d01, wrfbdy_d01, the speciation map defined in MOZBC and the wrfout_* files? Or the default settings of WRF-Chem are anyway ok? 

I attach here also the namelist.input.

Thanks in advance!

Caterina

[Gabriele Pfister]

Caterina

it is not necessary to have all the chemical species in the wrfinput
and wrfbdy, specifically not short-lived species. And you definitely
will not need PM2_5_DRY, which is a diagnostic species.

Gabriele

[Caterina Mogno]

Hi Ying,

did you manage to solve the problem eventually? If yes, how? I am exactly having your same problem in running ./wrf.exe.

Thanks in advance!

Caterina

[manish soni]

Hi Caterina,

open chem/module_mosaic_therm.F

try to find these lines in this fortran file.

! astem parameters
      nmax_astem      = 200		! max number of time steps in astem
      alpha_astem     = 0.05		! choose a value between 0.01 and 1.0

Try to increase the value of nmax_astem and recompile and run.

--
To unsubscribe from this group and stop receiving emails from it, send an email to wrf-chem-run...@ucar.edu.

--

Regards

-- -- -- 
Manish Soni
BIT Mesra, Jaipur Campus

[Anahita Amiri Farahani]

Hi Caterina,

Did you find out what was the problem?

Thanks,

Ana

[Gabriele Pfister]

Dear Ankan

this error (ASTEM internal steps exceeded) has been discussed within
the forum in previous conversations. Please look through those.

Gabi

On Sun, Jan 29, 2023 at 11:07 PM Ankan Sarkar <ankan.s...@gmail.com> wrote:
>
> Hello everyone,
> I am getting almost the same error (as mentioned in the previous thread here) after running wrf.exe. I checked the 'rsl.out.0000' file and found that the simulation stopped at 2 minutes and 6 seconds when the following lines showed up:
> entering mosaic_cloudchem_driver - ktau = 8
> leaving mosaic_cloudchem_driver - ktau = 8 0
> d02 2015-12-22_00:02:06 aerosols_driver calling mosaic_aerchem_driver
> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry

> ASTEM internal steps exceeded 200

> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry

> ASTEM internal steps exceeded 200

> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry

> ASTEM internal steps exceeded 200
> ASTEM internal steps exceeded 200

> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry

> ASTEM internal steps exceeded 200

> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry
> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry
> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry
> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry
> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry

> ASTEM internal steps exceeded 200
> ASTEM internal steps exceeded 200

> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry
> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry
> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry

> ASTEM internal steps exceeded 200

> d02 2015-12-22_00:02:06 mosaic_aerchem_driver: back from aerchemistry

> ASTEM internal steps exceeded 200

> On a server with 32 processors (RAM = 378 GB), I'm running WRF-Chem version 3.9.1.1 with the MOZART-MOSAIC chemistry option (chem_opt = 202). Looking at the previous threads, I set the time steps according to my case. Since I am using two domains- d01 (15 km) and d02 (3 km). So, I set the time_step to 90 (6 * 15) since the WRF User's Guide says that for a typical case, the time step for integration should be 6 * dx in km.
> &domains
> time_step = 90,
> time_step_fract_num = 0,
> time_step_fract_den = 1,
> And then, I set 'chemdt' option in the '&chem' section in the namlist to 1.5 for d01 and 0.3 for d02. Since my anthropogenic emission input files, i.e., wrfchemi* files (generated using the EDGARv5_MOZART_dataset) and my fire emission input files, i.e., wrffirechemi* files (using FINN datasets) are hourly input files, I set the following option in &time_control namelist:
> auxinput5_interval_m = 60, 60,
> auxinput7_interval_m = 60, 60,
> I am also attaching my input files here. Is it because of namelist option? Or is it because of a memory issue? Or is there any need to edit chem/module_mosaic_therm.F and recompile and run? Can anyone please guide me in this regard? Any help on this will be greatly appreciated. Thank you.
> With regards,
> Ankan

--
================================
Gabriele Pfister, Deputy Director
Atmospheric Chemistry Observations & Modeling

National Center for Atmospheric Research (NCAR)
Email: pfi...@ucar.edu
Temporary Home Office Phone: +1 303 241 1329
Work Phone: +1 303 497 2915
Web: https://staff.ucar.edu/users/pfister

[Ankan Sarkar]

Okay Mam. Thank you for the response.

With regards,

Ankan

[Setareh Rahimi]

Hi Xin,

I am running WRF-Chem version 4.4 and using chem-opt=202, and encountered the same problem:

ASTEM internal steps exceeded 200

and after some hours of simulation the model stopped.
I compiled the model using gfortran compiler. Now I want to install the model with intel compiler ( I would like to install WRF-Chem version 4.2.1 on a HPC. On HPC intel compiler has been installed before ( ifort version 19.0.4.243)) . But after setting the below environmental variables I cannot install libpng and netcdf fortan, while I could install them using gfortran.
export CC=icc
export CXX=icpc
export FC=ifort
export F77=ifort
export F90=ifort
export MPIFC=mpiifort
export MPIF77=mpiifort
export MPIF90=mpiifort
export MPICC=mpiicc
export MPICXX=mpiicpc

So would you please advice me on how to sort this issue? What other environmental variables should be added, and also what changes should be considered regarding configure.wrf file please?

Thanks in advance,
best wishes,

[Setareh Rahimi]

Hi Manosh,

As you suggested I increased the value of nmax_astem to 9000, and recompiled, but still got the ASTEM error!

What should I do, please?

Thanks.