SummerSim 2017: Untangling Uncertainty With Common Random Numbers: A Simulation Study - Reviewer Karel Van den Bosch
Jacob Barhak
Discussion of “Untangling uncertainty with common random numbers: a simulation study”
by Karel Van den Bosch,
Belgian Federal Planning Bureau
The paper is about quantifying the uncertainty in microsimulation the effects of health interventions, and in particular about reducing this uncertainty through the technique of common random numbers. It focuses on two kinds of uncertainty. The first is random or stochastic variability in outcomes due essentially to the fact that random numbers are used in the simulation, which implies that two simulations using the same model and data can produce different outcomes. The second kind is parameter uncertainty, i.e. the uncertainty in the parameter values that are inputs to the model. The main point of the paper is that naive calculations of total uncertainty could overstate the degree of uncertainty, and that this can be avoided by using common random numbers.
The issue of uncertainty in the results of microsimulation studies is both important and difficult, and papers on this are wellcome. Yet, though this short paper is at first sight quite straightforward and clear, I found it somewhat confusing when reading it more closely, and trying to figure out what its message exactly is. Perhaps this is because my background is largely in the analysis of surveys, rather than in simulation studies. In this piece I will try to set out the issues with which this paper is concerned as I see them from my background. It is therefore more a discussion than a review. I hope it is still useful to both authors and readers.
In my view the authors overlook (or at least do not mention) the fact that there is a fundamental difference between these two kinds of uncertainty: stochastic uncertainty is something that is generated within the simulation study, and that analysts should try to reduce as much as possible. For this reason, simulation studies typically generate very large samples, or the simulation is replicated a very large number of times. By contrast, parameter uncertainty is a given from outside, which the analyst can do little about. What she/he should do is to quantify this source of uncertainty.
CRN can be useful to reduce stochastic uncertainty in studies where expanding the sample, or increasing the number of replications is not practical, and/or where simulated cases go through a sequence of random events. But in the examples used in this study, and given the capabilities of current computers, there is no reason why stochastic uncertainty should not be reduced to a very small amount. Here CRN seems to be most useful to correctly quantify the effect of parameter uncertainty. This could perhaps be brought out more clearly in the paper.
Two further general remarks. First, note that there is a peculiar thing about the set-up of this study. The intervention simulated (a vaccine) eliminates completely one kind of diarrhea (ETEC-caused), but the effect of the intervention is measured in terms of the overall incidence of all kinds of diarrhea. The proportion of ETEC-caused diarrhea in all kinds of diarrhea is a random variable. So the effect of the intervention is random in terms of the overall incidence of diarrhea, even without parameter uncertainty. In my view, this set-up is equivalent to a situation where the effect of the intervention itself is uncertain or random. This is not a criticism, but something to keep in mind when interpreting the results of the study.
Secondly, from my point of view, in the fairly simple models used in this study, the use of common random numbers (CRN) serves to ensure that the baseline and the intervention are simulated on the same sample, instead of on different samples. Note that when using CRN the i-th case in one generated sample is exactly the same as the the i-th case in another sample. As they are for all practical purposes the same sample, the covariance between the variable of interest in the baseline scenario and in the intervention scenario is taken into account. If the covariance is large (which is usually the case), the variance of the difference (i.e. the effect) is much reduced. (See Stout and Goldie, 2008; Goedemé et al. (2013), “Testing the Statistical Significance of Microsimulation Results: A Plea”, International Journal of Microsimulation, 6(3) 50-77.) Making this point explicit in the paper may make it easier to understand for some readers (it would for me).
Further remarks:
Abstract: The abstract needs an additional sentence between the second and the third one. The 2nd sentence says that one must account for uncertainty. But the third sentence says that CRN can be used to reduce variance. A sentence is needed to link these two ideas, to the effect that naive quantifications of uncertainty may overestimate it.
Introduction, 1st para.: The lack of a strong correlation between expenditure on health care and life expectancy is not only, or even not mainly due to differences in the efficiency of health care, but also because differences in environment and lifestyle play a major role (see Joumard I, André C, Nicq C, Chatal O. Health Status Determinants: Lifestyle, Environment, Health Care Resources and Efficiency. OECD Publishing; 2008. Available at: http://ideas.repec.org/p/oec/ecoaaa/627-en.html.). In the context of this paper, this is of course a minor point.
2 Methods, 2nd para., “An example is instructive to make clear the distinction we see between these sources of uncertainty.”: After this sentence I was expecting the presentation in a few lines of a simple example to make this clear. Instead the paragraph continues about parameter uncertainty, and “An example” presumably refers to the paper as a whole. The reader should not be put on the wrong track in this way.
2 Methods, 2nd para., “The GBD estimates are represented .... “ I guess this representation is by the authors, not by the authors of the GBD study? This should be made clear.
2.1 Simple model. The sample size is set at 1000. It is not clear to me why it is set at such a low level, which of course leads – unnecessarily – to a fairly high level of stochastic uncertainty. Or is it supposed to be similar to the size of the communities in Kenya?
2.4 A philosophical challenge: I would call this a problem in theoretical statistics. Perhaps you should try to find a theoretical statistician to comment on this issue.
2.4 A philosophical challenge, 4th para: “Simple proposed alternatives ...”. As suggested above, simulating more individuals, or more replications could virtually eliminate stochastic uncertainty, so there would be no need to quantify it.
3. Results: Tables 1-5 could be merged into a single table, for better overview.
3. Results, Tables 1-5. What is lacking is a table with parameter uncertainty only, using CRN. In my perspective (as sketched above), that would be the most relevant table.
4 Discussion, 2nd para: Given what I wrote above, this paragraph seems to be somewhat misleading. Expanding the sample size, increasing the number of replications and the use of CRN can all serve to reduce stochastic uncertainty. (Whether to chose one or the other, or all three, depends on a number of considerations, cf. Murphy et al., 2013). The remaining stochastic uncertainty should be correctly reported. The formulation of the authors could suggest to some readers that stochastic uncertainty is an unknown parameter, that should be estimated as accurately as possible. That, it is not of course, it is a feature of the set-up of the simulation study. By contrast, parameter uncertainty is imposed from outside the study, and its implications for uncertainty about the outcomes should indeed be estimated as accurately as possible.
4.1 Limitations and directions for further research, last para: How to communicate to decision makers that results are uncertain is indeed an important challenge for researchers. (Survey analysts have had to contend with this for decades.) But first we should be very clear within the simulation research community about how to reduce and quantify the uncertainty best.
Jacob Barhak
Below is additional review by Karel Van den Bosch after the response of the authors made public in:
https://groups.google.com/forum/#!topic/public-scientific-reviews/_W8IQj0TkRM
Here is my response (quickly to make your job easier, and also because I do not want to spend much more time on this paper).
In my view, the authors have more or less adequately addressed the minor points of my review, but have given evasive (though polite) answers to my main point.
My main point was that:
Stochastic uncertainty is something that is generated within the simulation study, and that analysts should try to reduce as much as possible. CRN can be useful to reduce stochastic uncertainty in studies where expanding the sample, or increasing the number of replications is not practical, and/or where simulated cases go through a sequence of random events. But in the examples used in this study, and given the capabilities of current computers, there is no reason why stochastic uncertainty should not be reduced to a very small amount.
The authors give in fact three different antwers to this point:
1) (in the response ) However, we believe that stochastic uncertainty is not simply an artifact of simulation and therefore it is not desirable to eliminate it entirely. On the contrary, some amount of stochastic uncertainty is “real” and should be included in total uncertainty estimates produced by simulation.
2) (in their next paragraph in the response) However, we believe that stochastic uncertainty should not be reduced to as small an amount as possible. Rather, it should be reduced as small as appropriate, but no smaller.
3) (in the revised text) Although it may be tempting to use these or other approaches to eliminate stochastic uncertainty entirely from the estimates, we propose that these approaches are not valid alternatives to CRN when our goal is to quantify stochastic uncertainty, and not to eliminate it.
All three answers are puzzling to me. Regarding the first one, it is not clear to me why or how some amount of stochastic uncertainty could be “real”? An example would help. The second answer begs the question of what is an ‘appropriate’ amount (if it is not as low as possible)? Both answers seem to have the surprising implication that simulation studies that use very large samples or a very large number of replications to reduce stochastic uncertainty to virtually nil, would *underestimate* the ‘real’ or ‘appropriate’ amount of uncertainty.
The third answer is perhaps most to the point, except that, in my view, it is not very useful to quantify stochastic uncertainty, if you could also eliminate it.
Of course, not all points of contention need to be resolved before a study can be published. But if my point reflects the dominant view (as the authors state they believe), one would hope for more informative answers than those given.
Best regards,
Karel Van den Bosch
Jacob Barhak
The Response for the second review by Karel is posted by the authors and available publicly in:
https://groups.google.com/forum/#!topic/public-scientific-reviews/_W8IQj0TkRM
Jacob Barhak
Below is the Response by Karel Van den Bosch and an email exchange with Jacob Barhak regarding the paper after the second author response to the second review.
Karel Van den Bosch
Well, I made my point, and as the authors suspected, I am not completeley satisfied with their answer, but it is of no use to continue the discussion here.
Their last answer vaguely suggest situations where there is ‘real-world’ stochastic uncertainty. The paper would have been much more convincing if the authors have would have used an example where this was the case. In the example that is used, it is not at all clear to me what the added-value of CRN is. But I guess, they just wanted to show how it works.
Best regards,
Karel
-----
Jacob Barhak
Yet you have to make a call.
To your opinion, is it ok to publish considering that the discussion is public including your reservations?
I will make my question above public as well as your answer.
------------
Karel Van den Bosch
Dear Jacob,
OK, I wasn’t sure what you meant by call.
For me it is ok to publish this paper as a SummerSim paper.
Regards, Karel