SummerSim 2017: Untangling Uncertainty With Common Random Numbers: A Simulation Study - Reviewer Jacob Barhak

[Jacob Barhak]

Reviewer: Jacob Barhak

 

The paper discusses some of the uncertainty aspects in modeling.

 

I myself would recommend accepting the paper after some modifications. The reasons for my decision are:

 

1. There are clearly not enough experts in the field.  I contacted over 30 potential reviewers before getting a positive response. Many returned back with a busy answer – in the large scope this may mean that there are not enough experts in the field and they need reinforcement.

2. It is clear that the author team did some non trivial work towards quantifying uncertainty and it should be noted. Even if there is a debate on details, I rather have the debate take place and get published rather than ignore and forget it.

 

On the other side, this paper does seem to stir some discussion. I would very much like the authors to engage in this discussion and transform the paper as much as time permits - and time is relatively short - a couple of weeks perhaps. The authors should revise the paper and try to satisfy reviewers as much as possible. Since this is public non-blind review – the authors can choose to engage with the reviewers and the transcript of conversations be made public to supplement the paper. This may be advantageous  since time is relatively short.

I do have several specific points I wish to touch upon:

1. The authors should acknowledge in further details that the CRN approach will not work on more complicated models since after a few iterations the state in which individuals are will be different – therefore the approach is quite limited.

2. The authors should reference the availability of High Performance Computing and Moores’ Law– this may reduce many types of uncertainties in the future – and although computing power is still limited, some of the philosophical debates brought here can be dealt with sheer brute computing force. This development happened way after some of the papers quoted and although cannot eliminate uncertainly, it may be a key factor to consider.

3. Figure 2 confuses me, I assume these are different interventions in different locations – the names mean little to me. And then the results section is showing a simple model – I am unsure how are these connected – please elaborate there.

4. Your programmer must have written large portions of the paper since  there are several capitalization issues. I recognize this phenomenon from first hand since I myself tent to capitalize text differently after writing so much code. So I collected some examples of capitalization that need fixing, yet the authors should look at the paper again after modifications : Page 1: Cost-effectiveness Analysis Microsimulation , Page3: Capitalization : Results Section , Page 4: The Simple Model

5. The CRN approach, despite its disadvantages, has an advantage with regards to reproducibility of results - the authors should mention that. And if possible, provide links to reproducible code - a paper grounded by reproducible results has an advantage in a field where reproducibility is not a requirement and seems like a rare practice.

[Jacob Barhak]

Below is the Response by Jacob Barhak to the changes made by the authors

The authors did address the issues I raised - I am content.

So from a content point of view, this paper is acceptable. This is a safe decision considering that the authors have the support of 2 reviewers - Karel and myself, and since the third reviewer was previously generally supportive even before revisions. I am still waiting for the third reviewer opinion while writing these lines, yet it is safe to accept the paper.

However, for the paper to be published it has to fit the author kit format - otherwise it may be stopped by the publishing chair or the publisher. The current version is 13 pages long - I would ask that the paper be truncated in size to the allowed 12 pages - this can be done by reducing font size of the code at the end and reducing appendices - please do not remove content.

Yet also, I will ask that the authors provide links to the public reviews for this paper. This paper stirred some discussion and readers may be interested in the different arguments around this paper beyond what the authors are claiming. Since the review is public it is highly appropriate to present the entire picture alongside the reviews opinions.