Calibration stops after 3 iterations

[ERICK RAUL OLVERA PRADO]

Hello WRF-hydro community,

I am conducting an automated calibration exercise using the PyWrfHydroCalib-5.1.2-Beta source code on a personal computer. I specified the calibration of several parameters over a range that is specified by default in the calib_parms.tbl file. As I understand, this should produce several iterations over the calibration period, however after I run calibOrchestrator.py, only 3 iterations are run and then the CALIBRATION FOR JOB ID: 20 COMPLETE message is shown. Is it supposed to run only 3 iterations or something may be wrong?, Do you think that running in a personal computer (with mpirun) is influencing the number of iterations done due to memory issues? Is it weird  because there is no error message in the log files.

I would appreciate your feedback on this issue.

Best,

Erick Olvera

[Arezoo RafieeiNasab]

Hi Erick, 

Great news! That means you were able to run the calibration with no error! The calibration options will be setup and tuned in this file: https://github.com/NCAR/PyWrfHydroCalib/blob/master/setup_files/setup.parm, regarding your specific question, here is the relevant part: 

# Specify number of model iterations to calibrate over
numIter = 3

The setup provided is just a sample, and the user needs to read through the setup file and modify it as required. Please modify the number of iteration to what you could/want to use. We usually do a couple of hundreds (300 to 400), however, that depends on how long the model simulation takes for a given iteration. You do a quick calculation of how long the N iteration will take for your setup and domain, and decide a number that meets your needs and timeline. 

I hope this helps, let me know if you have any follow up questions. 

Thanks!

Arezoo

--
You received this message because you are subscribed to the Google Groups "wrf-hydro_users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wrf-hydro_use...@ucar.edu.
To view this discussion visit https://groups.google.com/a/ucar.edu/d/msgid/wrf-hydro_users/66f20517-b0cc-4c0d-86a6-1d1b3e3cbc8en%40ucar.edu.

--

-------------------------------------------------------------------------------------------------

My working day may not be your working day. Please do not feel obliged to reply to this email outside of your normal working hours.

-------------------------------------------------------------------------------------------------

Arezoo Rafieei Nasab, Ph.D.

Project Scientist II

NCAR Research Applications Laboratory

are...@ucar.edu

[ERICK RAUL OLVERA PRADO]

Hi Arezo,

Thanks for the clarification, I have changed the value of numIter in the setup.parm file and now the calibration has finished after several iterations.

Now, another issue came out when I run validOrchestrator.py, since the message VALIDATION FOR JOB ID: 21 COMPLETE. is shown without any output

in the /RUN.VALID/OUTPUT directory, neither any error or log files. I wonder if you know what could be happening? I looked at the setup.parm file for any

parameter that could be related to turn on or off validation but I couldn't find anything.

Many thanks in advance for your feedback.

Best,

Erick

[Arezoo RafieeiNasab]

Hi Erick, 

Any chance that this is a second run you are attempting to do, and have not used a new database? If you have had a database and you have already completed the validation, it has a flag that shows it as complete then the workflow will not do anything further. 

If this is not the case, could you share the setup file you have with me? I can take a look at it. 

Thanks!

Arezoo

[ERICK RAUL OLVERA PRADO]

Hi Arezo,

Thank you for your quick response. No, it was the first time when I run validOrchestrator.py that the "VALIDATION FOR JOB ID: 21 COMPLETE" appeared, that is, just after the calibration finished.

I don't know if this is relevant, but I would like to mention that after initializing every experiment, I had to copy my database (wrfHydroCalib.db) to the PyWrfHydroCalib-5.1.2-Beta directory, then

after the spinup, again I had to copy the updated database to PyWrfHydroCalib-5.1.2-Beta, this in order to proceed with the next step, otherwise a message like "spinup is not completed" appears.

In this case, during calibration, I noticed the database is updated on every iteration in the PyWrfHydroCalib-5.1.2-Beta directory, so I let the last database in the folder and run validOrchestrator.py.

I attach the setup.parm file as requested. Let me know i f I need to provide further information.

Many thanks in advance.

Best,

Erick

[Arezoo RafieeiNasab]

Hi Erick, 

You do not need to copy the database, and in fact you should not. The database needs to be created once by the workflow (step 1) and gets updated in every step and every iteration. All the information is written in the database and you do not want to mess with it if you do not know the content of it. Would you paste all the commands that you use to setup, run spinup, calibration and validation here?

Thanks!

Arezoo

[ERICK RAUL OLVERA PRADO]

Hi Arezo,

Thanks for your response. Yes, I agree that this should be the workflow, not having to copy the database, but after several attempts I realize that I had to do it in order to continue with the procedure. 

Perhaps I am doing something wrong with the commands. I paste an example of them below:

python ../PyWrfHydroCalib-5.1.2-Beta/initDB.py --optDbPath wrfHydroCalib.db
python ../PyWrfHydroCalib-5.1.2-Beta/inputDomainMeta.py domainMeta.csv --optDbPath wrfHydroCalib.db
python ../PyWrfHydroCalib-5.1.2-Beta/jobInit.py setup.parm --optExpID 8 --optDbPath wrfHydroCalib.db
python ../PyWrfHydroCalib-5.1.2-Beta/spinOrchestrator.py 8 --optDbPath wrfHydroCalib.db
python ../PyWrfHydroCalib-5.1.2-Beta/calibOrchestrator.py 8 --optDbPath wrfHydroCalib.db 

python ../PyWrfHydroCalib-5.1.2-Beta/validOrchestrator.py 8 --optDbPath wrfHydroCalib.db 

Many thanks again.

Erick

[Arezoo RafieeiNasab]

Hi Eric, 

I would recommend using the full path for both the *.py files and also Database, domainMeta and setup fille. I do not use the symlink or relative path, which should not really be an issue but I am just suggesting avoiding it since I am pretty sure the full path works. 

Also another hint, the name of the database could be changed, you do not need to stick to the wrfHydroCalib.db, this way you could make sure EACH experiment has its own database, and do not mix them. 

Finally, when you want to start a new experiment, you want to start clean meaning do not use the database from an old run, if the DB exists please remove it before you start a new setup. Could you try again with these suggestions (using a full path, new database) and let me know if that helps?

Thanks!

Arezoo

 

[Arezoo RafieeiNasab]

Hi Erick, 

I was just curious if you tried to restart from the beginning and use the full path and whether that helped you. 

Thanks!

Arezoo