Re: Strategic Decision Making Multiobjective Decision Analysis With Spreadsheets
Melanie Council
Economic value diagram results using the dependent dataset for normative decisions made following bias-corrected deterministic forecasts (GEFS member 1) and calibrated probabilistic forecasts from GEFS.
The (a) VS and (b) POD for all test values for the threshold of opposing risk, for a user with C/L = 0.01. The arrows indicate the lowest test value at which there was no significant difference between the results from the altered decisions (dashed) and the control results (solid).
Strategic Decision Making Multiobjective Decision Analysis With Spreadsheets
Download File https://tinourl.com/2yLC57
Contingency table results for all user behaviors for C/L = 0.1 showing (a) hits, (b) false alarms, (c) misses, and (d) correct rejections. Results for user behaviors that ignore ambiguity are lightly shaded while those that employ ambiguity are heavily shaded. The use of different scales among the plots is intended to emphasize differences in results between the user behaviors.
This study explores the objective application of ambiguity information, that is, the uncertainty in forecast probability derived from an ensemble. One application approach, called uncertainty folding, merges ambiguity with forecast uncertainty information for subsequent use in standard risk-analysis decision making. Uncertainty folding is found to be of no practical benefit when tested in a low-order, weather forecast simulation. A second approach, called ulterior motives, attempts to use ambiguity information to aid secondary decision factors not considered in the standard risk analysis, while simultaneously maintaining the primary value associated with the probabilistic forecasts. Following ulterior motives, the practical utility of ambiguity information is demonstrated on real-world ensemble forecasts used to support decisions concerning the preparation for freezing temperatures paired with a secondary desire for the reduction in repeat false alarms. Sample products for communicating ambiguity to the user are also presented.
The benefits of incorporating probabilistic forecast information, or first-order uncertainty, into the decision process are well established. Following the principles of risk analysis, the objective application of skilled probability forecasts can enable a user of the information to optimize decisions (i.e., limit negative consequences and minimize expenses) over the long run (Thompson 1950, 1952). Murphy (1977) further demonstrated that even moderately unreliable probabilistic forecasts had greater value than categorical and climatological forecasts, indicating the benefits of expressing day-to-day forecasts in terms of probabilities.
The skill of probabilistic forecasts has improved with the introduction of ensemble forecasts that can capture the flow dependence of first-order uncertainty. Wilks and Hamill (1995) demonstrated utility from ensemble-based forecast probability (pe) for both static and dynamic decisions, but used hypothetical ensemble forecasts. Using actual pe from the European Centre for Medium-Range Weather Forecasts (ECMWF) Ensemble Prediction System (EPS), Richardson (2000) found considerable added economic value throughout the medium forecast range relative to deterministic forecasts and suggested that the value of the EPS information is equivalent to years of development to the forecast model and data assimilation system. Palmer (2002) detailed the benefits of ensemble forecasts on time scales from days to seasons to decades and emphasized establishing ties between forecasters and users to maximize the potential economic value. Zhu et al. (2002) concluded that even when the ensemble employs a lower model resolution than the deterministic forecast, pe presents value to a wider range of users than do the deterministic forecasts. Using a wave model coupled to the ECMWF EPS, Saetra and Bidlot (2004) demonstrated the economic benefits to a specific user (oil rig operations) who employs probabilistic wave height forecasts. Keith (2003) demonstrated the economic value of using reliable probabilistic forecasts of visibility and cloud bases over categorical forecast information for airline operations when making fuel payload decisions. Maria-Helena et al. (2007) described the benefits found by users of 10-day flood warnings when decision tools combined deterministic and ensemble forecast information from the European Flood Forecast System.
The primary causes of ambiguity are sampling error (from a finite number of ensemble members) and deficiencies in ensemble design that leave weaknesses and/or gaps in the ability to account for all sources of forecast uncertainty, as described in Eckel et al. (2011, hereafter E11), which is a companion paper to the present work. Three methods of estimating an ambiguity distribution (the possible values of true forecast probability) and total ambiguity (the 90% confidence interval about the best-guess, calibrated forecast probability) were introduced in E11:
Ensemble of ensembles (EoE) is a theoretically accurate yet impractical method that involves running many different but equally likely versions of the original ensemble to produce multiple forecast probability density functions (PDFs), from which an ambiguity distribution can be produced.
Calibrated error sampling (CES) is a practical method that constructs an ambiguity distribution by calculating possible errors in forecast probability based on the ensemble spread and possible errors (defined from past ensemble performance) in the first two moments of the calibrated forecast PDF.
Randomly calibrated resampling (RCR) is a second practical method that generates an ambiguity distribution using bootstrap resampling of the ensemble members to account for sampling error and then applying a random calibration to each resampled dataset to account for ensemble design deficiencies.
Rather than a purely theoretical application of ambiguity, this study is concerned with potential real-world utility. E11 concluded that the two practical ambiguity estimation methods, CES and RCR, displayed some skill but also exhibited notable limitations. The question then for this study is whether CES or RCR, even given their limitations, can provide valuable information for decision making.
Two possible approaches for benefiting from ambiguity information are explained and demonstrated in section 2. Section 3 discusses possible ways of communicating ambiguity to the user. Our conclusions are given in section 4.
Uncertainty folding aims to improve decision making by following the same normative process as the control, but rather than risk defined as , risk is derived from the ambiguity distribution. While the focus of this study is on the real-world application of ambiguity information, uncertainty folding is explored here using a simulation of weather forecasting, which permits testing of all three ambiguity estimation methods. Since EoE cannot be applied to real-world ensemble forecasting, the low-order dynamical system used in E11, called L96, is employed here. L96, introduced by Lorenz (1996), mimics atmospheric processes with a set of symmetric, coupled equations that evolve large-scale (X) and small-scale variables. E11 simulated atmospheric modeling with a flawed model of L96 (using lower-order numerical truncation, a relatively coarse time step, and the parameterization of small-scale variables) and a flawed ensemble (using ensemble Kalman filter initial conditions and limited perturbations to the model parameterizations).
These findings are explained by the fact that in EoE, is a random sample of the ambiguity distribution so E(pe) presents different and valuable information. For CES and RCR, the ambiguity distribution is essentially constructed as the potential error about so that . It is therefore unfortunate for operational application that uncertainty folding is only effective with the impractical ambiguity estimation method EoE and ineffective with the practical methods of CES or RCR.
Since uncertainty folding provides no benefit when used with CES or RCR, and processing ambiguity information from EoE adds no value relative to the probability information obtained directly from the ensemble data, the overall conclusion is that a user cannot benefit from ambiguity information via uncertainty folding.
7fc3f7cf58