SummerSim 2017: Untangling Uncertainty With Common Random Numbers: A Simulation Study - Response for Reviewers
Jacob Barhak
Response to Reviewers for Untangling uncertainty with common random numbers: a simulation study
Reviewer 1: Jacob Barhak
The paper discusses some of the uncertainty aspects in modeling.
I myself would recommend accepting the paper after some modifications. The reasons for my decision are:
- There are clearly not enough experts in the field. I contacted over 30 potential reviewers before getting a positive response. Many returned back with a busy answer – in the large scope this may mean that there are not enough experts in the field and they need reinforcement.
- It is clear that the author team did some non trivial work towards quantifying uncertainty and it should be noted. Even if there is a debate on details, I rather have the debate take place and get published rather than ignore and forget it.
We thank the reviewer for this positive assessment.
On the other side, this paper does seem to stir some discussion. I would very much like the authors to engage in this discussion and transform the paper as much as time permits - and time is relatively short - a couple of weeks perhaps. The authors should revise the paper and try to satisfy reviewers as much as possible. Since this is public non-blind review – the authors can choose to engage with the reviewers and the transcript of conversations be made public to supplement the paper. This may be advantageous since time is relatively short.
I do have several specific points I wish to touch upon:
- The
authors should acknowledge in further details that the CRN approach will
not work on more complicated models since after a few iterations the state
in which individuals are will be different – therefore the approach is
quite limited.
On the contrary, we believe that CRN will work on highly complicated models, and although the state will diverge, there will be substantial, and appropriate, reductions in variance in the estimates of interest. We have added a cautious sentence about this at the end of the methods section.
- The
authors should reference the availability of High Performance Computing
and Moores’ Law– this may reduce many types of uncertainties in the future
– and although computing power is still limited, some of the philosophical
debates brought here can be dealt with sheer brute computing force. This
development happened way after some of the papers quoted and although
cannot eliminate uncertainly, it may be a key factor to consider.
We have added a sentence on this important point in the discussion, although our conclusion is different than the reviewers: deploying additional computational resources as they become available cannot adequately address stochastic uncertainty.
- Figure
2 confuses me, I assume these are different interventions in different
locations – the names mean little to me. And then the results section is
showing a simple model – I am unsure how are these connected – please
elaborate there.
We agree that this figure is confusing, although we are not sure how to simplify it without derailing the focus of the paper. This figure actually does not show the intervention at all, with could be represented by a change to the incidence rate of ETEC or rota, or a change to the excess mortality rate of diarrhea due to any etiology. We have added some additional words to the caption in the hopes of making it at least a little more clear.
- Your
programmer must have written large portions of the paper since there are several capitalization issues. I
recognize this phenomenon from first hand since I myself tent to
capitalize text differently after writing so much code. So I collected
some examples of capitalization that need fixing, yet the authors should
look at the paper again after modifications : Page 1: Cost-effectiveness
Analysis Microsimulation , Page3: Capitalization : Results Section , Page
4: The Simple Model
We have reviewed the capitalization and edited it for consistency, and thank the reviewer for this careful attention to detail.
- The CRN approach, despite its disadvantages, has an advantage
with regards to reproducibility of results - the authors should mention
that. And if possible, provide links to reproducible code - a paper
grounded by reproducible results has an advantage in a field where
reproducibility is not a requirement and seems like a rare practice.
We see CRN as distinct from reproducible results in randomized computation. Although CRN requires an attention to making results reproducible, we strongly advocate that researchers set the random seed and take other steps to make their results reproducible even if they do not choose to use CRN for reducing variance.
We have created a “replication archive” so that others can generate the results in this paper, and added a link to it. (Side note: this revealed that we failed to set the random seed for the permutation calculation in Section 3.2, and we have adjusted the values in Table 6 to match a more reproducible version of this calculation.)
Reviewer 2: MA Al-Mamun
Accept with some corrections
The paper presents an example of the quantification of parameter uncertainty
using CRN method in microsimulation study. The authors chose the
microsimulation of health interventions to eradicate diarrhea. This kind of
paper is really appreciated in the field of stochastic modeling, especially
related to cost-effectiveness of some public health interventions. But in my
opinion, the paper needs further revision of the purpose, presentation, and
discussion. Please address the following issues
Major issues:
1. First of all, the paper needs to state the main purpose and implications
clearly. The current abstract needs a sentence or two about the purpose and
implications. One assumption: if the model shows the reduction in the variance
of input parameter uncertainty, it should be wise to relate that with the
implication of the cost-effectiveness!
We have added a sentence to the abstract to make clear what is at stake, and also added this in the body of the paper.
2. Second, I understand that the authors wanted to choose the simple model to
show the simpler CRN method, but audience may find difficulties to understand
the interventions to one kind of diarrhea and all kinds of diarrhea. One
suggestion would be to add clarification of the problem simply in the method section.
We have added a clarifying sentence to Section 2.1.
3. Third, I am sure the audience will
like the first section of the pencil and paper calculation. But, they may get
more confused by looking at the further results. The best thing would be show the
results graphically (i.e table 2,3,4)
We appreciate this advice, and in general completely agree that the visual display of quantitative information is usually preferred to tables. However, in this case, it seems clearer to present the numbers in order, and a graphic would present them all at once.
4. Fourth, authors have given a good understanding of the impact of parameter uncertainty to the cost-effectiveness of the interventions. To maintain the flow of the paper, authors should link the parameter uncertainty of this simple model to the implications of cost-effectiveness. The audience will expect at least one figure or table which will tell them how this CRN reduce the variance of uncertainty and how that can be inferred towards the efficacy of decisions/interventions
We see this as related to the Major Issue #1 raised by the reviewer, and have added sentences to the paragraphs after Table 3 and Table 5 to further develop the claim in the abstract. Again, we appreciate the advice that we do this with a figure, but feel that it is really better stated in words in this case.
5. Finally, the results section should be expanded with some more results. Also, when the authors say this method is simpler, than we need a comparison with the previous literature (probably with murphy et al., (2013)).
We are unsure just how to respond to this suggestion, and while we are always in favor of including more results, it is not clear to us what direction the reviewer thinks would be best. We must also balance our desire to include everything we could do (such as additional interventions, additional disease models, additional health behaviors) against the page limits for a SummerSim publication.
We are similarly unsure just what the reviewer is referring to when they request a comparison to previous literature---it is not our intention to claim in this paper that we have a CRN approach that is simpler than other approaches (although perhaps we do, but to defend this claim we would need to introduce some notions of program complexity that are far afield from our present focus). Indeed, the only time we say one approach is simpler than another in this paper, it is CRN that is the more complicated approach (and we say this in reference to previous literature, such as (Murphy et al. 2013), although a typo which we have now corrected may have obscured our intended meaning).
Reviewer 3: Karel Van den Bosch, Belgian Federal Planning Bureau
The paper is about quantifying the uncertainty in microsimulation the effects of health interventions, and in particular about reducing this uncertainty through the technique of common random numbers. It focuses on two kinds of uncertainty. The first is random or stochastic variability in outcomes due essentially to the fact that random numbers are used in the simulation, which implies that two simulations using the same model and data can produce different outcomes. The second kind is parameter uncertainty, i.e. the uncertainty in the parameter values that are inputs to the model. The main point of the paper is that naive calculations of total uncertainty could overstate the degree of uncertainty, and that this can be avoided by using common random numbers.
The issue of uncertainty in the results of microsimulation studies is both important and difficult, and papers on this are wellcome. Yet, though this short paper is at first sight quite straightforward and clear, I found it somewhat confusing when reading it more closely, and trying to figure out what its message exactly is. Perhaps this is because my background is largely in the analysis of surveys, rather than in simulation studies. In this piece I will try to set out the issues with which this paper is concerned as I see them from my background. It is therefore more a discussion than a review. I hope it is still useful to both authors and readers.
Definitely useful! We appreciate all of the reviewers work on this.
In my view the authors overlook (or at least do not mention) the fact that there is a fundamental difference between these two kinds of uncertainty: stochastic uncertainty is something that is generated within the simulation study, and that analysts should try to reduce as much as possible. For this reason, simulation studies typically generate very large samples, or the simulation is replicated a very large number of times. By contrast, parameter uncertainty is a given from outside, which the analyst can do little about. What she/he should do is to quantify this source of uncertainty.
We believe that this is the dominant view, and agree with most of it. However, we believe that stochastic uncertainty is not simply an artifact of simulation and therefore it is not desirable to eliminate it entirely. On the contrary, some amount of stochastic uncertainty is “real” and should be included in total uncertainty estimates produced by simulation.
CRN can be useful to reduce stochastic uncertainty in studies where expanding the sample, or increasing the number of replications is not practical, and/or where simulated cases go through a sequence of random events. But in the examples used in this study, and given the capabilities of current computers, there is no reason why stochastic uncertainty should not be reduced to a very small amount. Here CRN seems to be most useful to correctly quantify the effect of parameter uncertainty. This could perhaps be brought out more clearly in the paper.
We hope that our previous point helps to explain again why we slightly disagree with the assessment of the reviewer. We agree that one benefit of CRN to avoid expanding the sample or increasing the replications, and although the simple example in this study can be run many times with a very large sample, more complicated simulations will not have this luxury. However, we believe that stochastic uncertainty should not be reduced to as small an amount as possible. Rather, it should be reduced as small as appropriate, but no smaller.
Two further general remarks. First, note that there is a peculiar thing about the set-up of this study. The intervention simulated (a vaccine) eliminates completely one kind of diarrhea (ETEC-caused), but the effect of the intervention is measured in terms of the overall incidence of all kinds of diarrhea. The proportion of ETEC-caused diarrhea in all kinds of diarrhea is a random variable. So the effect of the intervention is random in terms of the overall incidence of diarrhea, even without parameter uncertainty. In my view, this set-up is equivalent to a situation where the effect of the intervention itself is uncertain or random. This is not a criticism, but something to keep in mind when interpreting the results of the study.
We appreciate the reviewer’s clear assessment of this aspect of the setup, and hope that others will also interpret things this way. This, together with the relatively large, but carefully quantifies, parameter uncertainty in the total level of diarrhea as well as the fraction cause by ETEC may be a unique aspect of this sort of global health cost-effectiveness simulation work.
Secondly, from my point of view, in the fairly simple models used in this study, the use of common random numbers (CRN) serves to ensure that the baseline and the intervention are simulated on the same sample, instead of on different samples. Note that when using CRN the i-th case in one generated sample is exactly the same as the the i-th case in another sample. As they are for all practical purposes the same sample, the covariance between the variable of interest in the baseline scenario and in the intervention scenario is taken into account. If the covariance is large (which is usually the case), the variance of the difference (i.e. the effect) is much reduced. (See Stout and Goldie, 2008; Goedemé et al. (2013), “Testing the Statistical Significance of Microsimulation Results: A Plea”, International Journal of Microsimulation, 6(3) 50-77.) Making this point explicit in the paper may make it easier to understand for some readers (it would for me).
[We appreciate this perspective, and have added this alternative explanation of why CRN works in the methods section.]
Further remarks:
Abstract: The abstract needs an additional sentence between the second and the third one. The 2nd sentence says that one must account for uncertainty. But the third sentence says that CRN can be used to reduce variance. A sentence is needed to link these two ideas, to the effect that naive quantifications of uncertainty may overestimate it.
We have added such a sentence and hope that the abstract will now be clearer to readers.
Introduction, 1st para.: The lack of a strong correlation between expenditure on health care and life expectancy is not only, or even not mainly due to differences in the efficiency of health care, but also because differences in environment and lifestyle play a major role (see Joumard I, André C, Nicq C, Chatal O. Health Status Determinants: Lifestyle, Environment, Health Care Resources and Efficiency. OECD Publishing; 2008. Available at: http://ideas.repec.org/p/oec/ecoaaa/627-en.html.). In the context of this paper, this is of course a minor point.
We appreciate this point, but would prefer to continue to open our paper this way. Although it is unreasonable to expect that all of the differences in life expectancy can be addressed by changes to how health dollars are spent, we believe that there is substantial room for improvement, particularly in settings where health resources are (much) more constrained than they are in OECD settings.
2 Methods, 2nd para., “An example is instructive to make clear the distinction we see between these sources of uncertainty.”: After this sentence I was expecting the presentation in a few lines of a simple example to make this clear. Instead the paragraph continues about parameter uncertainty, and “An example” presumably refers to the paper as a whole. The reader should not be put on the wrong track in this way.
We have changed this sentence and hope that it will no longer mislead readers.
2 Methods, 2nd para., “The GBD estimates are represented .... “ I guess this representation is by the authors, not by the authors of the GBD study? This should be made clear.
Thank you for raising this issue of attribution, we have edited in an effort to make clearer what results we are referencing from GBD.
2.1 Simple model. The sample size is set at 1000. It is not clear to me why it is set at such a low level, which of course leads – unnecessarily – to a fairly high level of stochastic uncertainty. Or is it supposed to be similar to the size of the communities in Kenya?
This sample size is chosen in part for convenience, it is a nice round number that is small enough to yield substantial stochastic uncertainty. However it does correspond roughly to the number of under-5-year-old children in a community that a local health decision-maker might be concerned with in some cases.
2.4 A philosophical challenge: I would call this a problem in theoretical statistics. Perhaps you should try to find a theoretical statistician to comment on this issue.
We agree that it would indeed be interesting to discuss these issues with a theoretical statistician, and we hope that the publication of this work will be the basis for pushing forward such conversations. To be responsive to the reviewer’s suggestion, but without making the claim that this is a problem of interest to theoretical statisticians, we have changes the section title to “a theoretical problem”.
2.4 A philosophical challenge, 4th para: “Simple proposed alternatives ...”. As suggested above, simulating more individuals, or more replications could virtually eliminate stochastic uncertainty, so there would be no need to quantify it.
We agree with the reviewer, although as has come up in previous responses above, we feel it is useful to quantify the stochastic uncertainty and would be potentially misleading to eliminate it. Perhaps it is worthwhile to try to explain why we hold this view, although I’m not sure it can fit into our paper. Accurate quantification of uncertainty is likely important for making informed decisions. If there is indeed substantial stochastic uncertainty and simulation provides a decision-maker with incorrectly precise predictions by overlooking this uncertainty, the decision-maker will be unable to “hedge” appropriately against this uncertainty. It is likely that this change in action will take very different forms in different settings (and with different decision-makers!), but we believe that including an appropriate level of stochastic uncertainty is important to allowing informed decision making under uncertainty.
3. Results: Tables 1-5 could be merged into a single table, for better overview.
It was suggested by Reviewer 2 that these be merged into a figure, but we prefer the flow of presenting them one at a time. Perhaps it is a matter of taste, and we are open to changing this, although our current taste is for having multiple separate tables, instead of trying to combine all the tables (and all the captions!) into one larger table or figure.
3. Results, Tables 1-5. What is lacking is a table with parameter uncertainty only, using CRN. In my perspective (as sketched above), that would be the most relevant table.
This table is excluded because the parameter uncertainty is the same with or without CRN. [we have added a comment explaining this to the text.]
4 Discussion, 2nd para: Given what I wrote above, this paragraph seems to be somewhat misleading. Expanding the sample size, increasing the number of replications and the use of CRN can all serve to reduce stochastic uncertainty. (Whether to chose one or the other, or all three, depends on a number of considerations, cf. Murphy et al., 2013). The remaining stochastic uncertainty should be correctly reported. The formulation of the authors could suggest to some readers that stochastic uncertainty is an unknown parameter, that should be estimated as accurately as possible. That, it is not of course, it is a feature of the set-up of the simulation study. By contrast, parameter uncertainty is imposed from outside the study, and its implications for uncertainty about the outcomes should indeed be estimated as accurately as possible.
This is either a point on which we fundamentally disagree, or a point which did not present clearly in our submitted draft and hope we have now cleared up somewhat in previous responses.
4.1 Limitations and directions for further research, last para: How to communicate to decision makers that results are uncertain is indeed an important challenge for researchers. (Survey analysts have had to contend with this for decades.) But first we should be very clear within the simulation research community about how to reduce and quantify the uncertainty best.
We completely agree, and have added this as an additional direction for future research.
Jacob Barhak
Second Response to Reviewers for Untangling uncertainty with common random numbers: a simulation study
Reviewer 1: Karel Van den Bosch
Here is my response (quickly to make your job easier, and also because I do not want to spend much more time on this paper).
We thanks the reviewer and appreciate all of your efforts.
In my view, the authors have more or less adequately addressed the minor points of my review, but have given evasive (though polite) answers to my main point.
We will attempt to remedy this directly now, and appreciate the opportunity to do so.
My main point was that:
Stochastic uncertainty is something that is generated within the simulation study, and that analysts should try to reduce as much as possible. CRN can be useful to reduce stochastic uncertainty in studies where expanding the sample, or increasing the number of replications is not practical, and/or where simulated cases go through a sequence of random events. But in the examples used in this study, and given the capabilities of current computers, there is no reason why stochastic uncertainty should not be reduced to a very small amount.
We believe that we substantially disagree with the reviewer’s main point; it is our claim that stochastic uncertainty is not only something generated within simulation that we should strive to remove---we claim that stochastic uncertainty is “real”, in the sense that we can make better decisions by accounting for it than we can by eliminating it. We have not attempted to prove this claim directly in the present paper, but perhaps we should have.
The authors give in fact three different antwers to this point:
1) (in the response ) However, we believe that stochastic uncertainty is not simply an artifact of simulation and therefore it is not desirable to eliminate it entirely. On the contrary, some amount of stochastic uncertainty is “real” and should be included in total uncertainty estimates produced by simulation.
2) (in their next paragraph in the response) However, we believe that stochastic uncertainty should not be reduced to as small an amount as possible. Rather, it should be reduced as small as appropriate, but no smaller.
3) (in the revised text) Although it may be tempting to use these or other approaches to eliminate stochastic uncertainty entirely from the estimates, we propose that these approaches are not valid alternatives to CRN when our goal is to quantify stochastic uncertainty, and not to eliminate it.
All three answers are puzzling to me. Regarding the first one, it is not clear to me why or how some amount of stochastic uncertainty could be “real”? An example would help. The second answer begs the question of what is an ‘appropriate’ amount (if it is not as low as possible)? Both answers seem to have the surprising implication that simulation studies that use very large samples or a very large number of replications to reduce stochastic uncertainty to virtually nil, would *underestimate* the ‘real’ or ‘appropriate’ amount of uncertainty.
We also believe that an example would help, but developing a concise example seems beyond the time available in this review period, and explaining it in the present paper might likely take us beyond the page limit. A sketch of how we would do so is included here: first, we must introduce a decision framework, where at least two alternative interventions are under consideration, on the basis of their cost-effectiveness; this decision framework must include some notion of “risk” so that it is more complicated than selecting the intervention with the lowest expected cost-effectiveness ratio. Second, we must show that the ranking of interventions is different when stochastic uncertainty is excluded than when included (at the appropriate level). And third, we must show that the ranking when stochastic uncertainty is included is better than when it is excluded.
To make this example completely convincing, it would be best to use realistic (if stylized) simulation scenarios, so that it is clear that this issue is of real importance and not simply pathological.
The third answer is perhaps most to the point, except that, in my view, it is not very useful to quantify stochastic uncertainty, if you could also eliminate it.
If and when we generate our convincing example, we will be sure to call it to the reviewer’s attention (although we understand if they have spent all the time that they want to on this line of research already!)
Of course, not all points of contention need to be resolved before a study can be published. But if my point reflects the dominant view (as the authors state they believe), one would hope for more informative answers than those given.
We again thank the reviewer for their time, and hope that this proven less evasive than our last round of comments, while acknowledging that the review has likely found it still not completely satisfying.
Jacob Barhak
Third Response to Reviewers for Untangling uncertainty with common random numbers: a simulation study
Reviewer 1: Jacob Barhak
The authors did address the issues I raised - I am content.
We thank the reviewer.
So from a content point of view, this paper is acceptable. This is a safe decision considering that the authors have the support of 2 reviewers – Karel and myself, and since the third reviewer was previously generally supportive even before revisions. I am still waiting for the third reviewer opinion while writing these lines, yet it is safe to accept the paper.
Thanks again for all of your work on this.
However, for the paper to be published it has to fit the author kit format – otherwise it may be stopped by the publishing chair or the publisher. The current version is 13 pages long – I would ask that the paper be truncated in size to the allowed 12 pages – this can be done by reducing font size of the code at the end and reducing appendices – please do not remove content.
We have removed the code listing in Appendix B (but kept the url to the online version of it) and now the paper fit comfortably within the 12 page limit.
Yet also, I will ask that the authors provide links to the public reviews for this paper. This paper stirred some discussion and readers may be interested in the different arguments around this paper beyond what the authors are claiming. Since the review is public it is highly appropriate to present the entire picture alongside the reviews opinions.
We have added a shortlink to the public reviews for this paper.
Reviewer 2: Below is final response from Mohammad Abdullah Al-Mamun
I think, I agree with you, the future discussions can have enough meat from this paper as a proof of concept.
We thank the reviewer for this assessment, and for their time reviewing our paper.
Reviewer 3: Karel Van den Bosch
Well, I made my point, and as the authors suspected, I am not completely satisfied with their answer, but it is of no use to continue the discussion here.
We appreciate the reviewer engaging with this topic, and hope that in future work we will demonstrate our position more completely.
Their last answer vaguely suggest situations where there is ‘real-world’ stochastic uncertainty. The paper would have been much more convincing if the authors have would have used an example where this was the case. In the example that is used, it is not at all clear to me what the added-value of CRN is. But I guess, they just wanted to show how it works.
Thank you again to the conference chairs and all the reviewers for their work.