RE: Doubt about p and r-factors
Abbaspour, Karim
p-factor, r-factor, NS, deterministic and stochastic calibration, …
I think it is important to first understand the difference between a “deterministic” and a “stochastic” calibration, and to realize that there is no room for a deterministic calibration in natural sciences.
Example of a deterministic optimization is “trial and error”. Meaning you just keep adjusting parameters until you get some kind of a reasonable match between simulation and observation. Reporting this as a calibrated model, in my opinion is wrong, and will not stand in any court of law, if it comes to that.
In stochastic calibration, at least we recognize the errors and uncertainties in our modeling work and try to capture, to some degree, our ignorance and lack of understanding of the processes in natural systems. I have written many times that there is an intimate relationship between calibration and uncertainty. Reporting the uncertainty is not a luxury in modeling, it is a necessity. Without the uncertainty, calibration is meaningless and misleading.
In SUFI2, we want that our model result envelops the observation. Observation, is what we have measured in the natural system. Observation is important because it is the culmination of all the processes taking place in the region of study. The argument, however naïve, is that if we capture the observation correctly with our model, then we are somehow capturing correctly all the processes leading to that observation. The problem, of course, is that often a combination of wrong processes in our model may also produce good simulation results. For this reason, the more variables (representing different processes) we include in the objective function, the more likely we are to avoid the wrong processes.
In stochastic optimization, we have distributions representing our uncertain parameters (remember that parameters represent processes). Propagation of the uncertainties in these input parameters produce output uncertainties. The output uncertainties are often expressed by the “95% prediction uncertainties”, or the so call 95PPUs. It is important to note that the “95PPU” is our model’s output results. When we calibrate a model stochastically, we calibrate the 95PPU. The aim is not to get a single signal, historically and quite wrongly, referred to as the “best simulation”. Trying to capture the observations by these 95PPUs is the philosophy behind the SUFI2 program.
To quantify the fit between simulation result, expressed as 95PPU, and observation expressed as a single signal (with some error associated with it) we came up with two statistics: P-factor and R-factor (see Abbaspour 2004, 2007 references provided in the reference list of SWAT-CUP). P-factor is the percentage of observed data enveloped by our modeling result, the 95PPU. R-factor is the thickness of the 95PPU envelop. In SUFI2, we try to get reasonable values of these two factors. While we would like to capture most of our observations in the 95PPU envelop, we would at the same time like to have a small envelop. No hard numbers exist for what these two factors should be, similar to the fact that no hard numbers exist for R2 or NS. The larger they are, the better they are. For P-factor, we suggested a value of >70% for discharge, while having R-factor of around 1. For sediment, a smaller P-factor and a larger R-factor could be acceptable.
SUFI2 operates by performing several iterations, usually at most <5. In each iteration, the parameter ranges get smaller zooming on a region of the parameter space, which produced better results in the previous iteration. Naturally, as parameter ranges get smaller, the 95PPU envelop gets smaller, leading to smaller P-factor and smaller R-factor. As each iteration zooms into a better region of the parameter space, obtained by the previous iteration, it is going to find a better “best” solution. So, if you have NS as your objective function, then you will get a better NS in subsequent iterations, but the P-factor and R-factor will decrease because of narrower parameter ranges. But the idea is not to find that so called “best simulation”. Because, 1) there are always better simulations, and 2) the difference between the “best simulation” and the “next best simulation” and the “next next best simulation” is usually statistically insignificant (e.g., NS=0.83 vs NS=0.81 are probably not significantly different), meaning that they could both be identified as the best simulations. But while the differences are insignificant in terms of the objective function value, they are very significant in terms of parameters values. Therefore, the next-best solution cannot be ignored.
In conclusion, our final solution must contain all the reasonable solutions to the problem, because we do not know better! I suggest we leave out the idea of a “best solution” from our minds, we can then see better how to deal with uncertainty in our model. Once you calibrate your model by calibrating parameter ranges, then whatever you do with your model must be the result of propagating those parameter ranges. So, for example, if you do some landuse change analysis and get a new SWAT model, then copy the new TxtInOut in your SWAT-CUP project directory and propagate the calibrated parameter ranges by doing an iteration of say 500 simulations, extract the variables you want to look at and see the 95PPU of those variables and compare them with the previous scenario (use the No_Observation option). No reason to put a single parameter set you identified as the best, back into the SWAT database for further analysis!
Sorry for the long response. This is a very common recurring question, I hope that this explanation sheds more light on the calibration problem without making it more confusing! I am updating the SWAT-CUP manual with more detailed explanations.
Best, Karim
-------------------------------------------------
Dr. K.C. Abbaspour
Eawag, Swiss Federal Institute for Aquatic Science and Technology
Ueberlandstr. 133, P.O. Box 611, 8600 Duebendorf, Switzerland
email: abba...@eawag.ch
phone: +41 58 856 5359
fax: +41 58 856 5375
http://www.eawag.ch/index_EN
From: Eliete Nazare [mailto:eliete...@gmail.com]
Sent: Monday, May 04, 2015 6:35 PM
To: Abbaspour, Karim
Subject: Doubt about p and r-factors
Dear Dr. Abbaspour,
If possible, I'd like clarify one question.
After several attempts to improve the parameters I have three results that I consider good, my doubt is because of p and r factors. I noticed that by improving the NS, p and r-factors worsen.
1 - p-factor = 0.89, r-factor = 1.4, NS = 0.74, = 3.9 PBIAS
2 - p-factor = 0.83, r-factor = 0.61, NS = 0.81, = 1.6 PBIAS
3 - p-factor = 0.79, r-factor = 0:49, NS = 0.83, = 0.5 PBIAS
What the best calibration?
Thank you!
Regards!
--
Eliete Nazaré Eduardo
Eng. Agr., Master in Agronomy Soil Science - Federal Rural University of the Rio de Janeiro
Doctoral Student in Soil Science - Federal University of Lavras
Visiting Scholar in the Department of Agronomy - Purdue University
Jim Almendinger
To: "SWAT-CUP" <swat...@googlegroups.com>
Cc: "Eliete Nazare" <eliete...@gmail.com>
Sent: Tuesday, May 5, 2015 7:08:08 AM
Subject: RE: Doubt about p and r-factors
NOTE: This message was trained as non-spam. If this is wrong, please correct the training as soon as possible.
Spam
Not spam
Forget previous vote
You received this message because you are subscribed to the Google Groups "SWAT-CUP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swat-cup+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
St. Croix Watershed Research Station
Science Museum of Minnesota
16910 152nd St N
Marine on St. Croix, MN 55047
tel: 651-433-5953 ext 19
