SummerSim 2017: Using Sensitivity analysis to examine the effects of an Ebola vaccine - Response for Reviewers
Jacob Barhak
However, there are problems with the framing and analysis in the context of the recent ebolavirus outbreak. First, the authors discuss ebolavirus eradication. This is inappropriate as “eradication” has a specific meaning in public health policy (Dowdle 1998) and eradication is not possible for zoonoses unless the pathogen is also eliminated from animal reservoirs. A more appropriate way to frame this question is to ask “under what conditions is the elimination of an ebolavirus strain circulating within a human population likely?”.
More importantly, analysis of eradication from an endemic equilibrium with a deterministic model is not appropriate for a disease that is dominated by stochastic zoonotic outbreaks with dynamics far from equilibrium. Their treatment ignores the transient immune dynamics during outbreaks that make post-outbreak elimination more likely in finite populations. The authors in the abstract and discussion note the relevance of stochastic effects, but do not quantify them. (A recent paper that the authors couldn’t have known about prior to submission examines this for ebolavirus.) Also, by assuming endemic equilibrium with Ro near 1, the authors implicitly assumed that the mean age of infection in an unvaccinated population is comparable to a typical human lifetime (exponentially-distributed in the model) (Keeling and Rohani p 33), and thus they also implicitly assumed their conclusion that immunity would need to persist for 50 years to guarantee elimination of an endemic strain.
If this was a journal publication, I would recommend rejection as the analysis as presented isn’t appropriate to the disease as currently framed by the authors. However, as this is a conference proceeding, a lower bar for work in progress is reasonable, and so my recommendation is Borderline: acceptance at the discretion of the conference organizers. Language clarifying the meaning of “eradication” in the context of human ebolavirus outbreaks can be fixed, but a stochastic model for realistic finite population sizes (or at least an ODE in the post-outbreak transient regime) is more appropriate than an endemic ODE model for studying ebolavirus elimination. Minimally, I would recommend reframing the analysis as applying in contexts where an ebola-like pathogen escapes early control efforts prior to vaccine development.
Response: This is an excellent point. Modellers can be too quick to talk about eradication when what we really mean is disease control. We have reworded throughout to be more accurate. The word "eradication" no longer appears. As to the second point, although developing a stochastic model is out of the scope of our manuscript, we have nevertheless added more investigation of the effects of random variations in the parameters using Latin Hypercube Sampling and Monte Carlo simulations. We have added a new figure (Figure 5) that looks at the variation here even when parameters are chosen that suggest disease control, in order to illustrate the effect that such randomness can have.
Major comments
How was waning effect = 0.02 chosen to define “can be theoretically eradicated?” There are samples of Ro>1 at lower values of the waning effect in Figure 4.
Response: We chose that value as it represented a case when >95% of simulated results predicted Rp<1. However, we have made some changes to the model due to other reviewers' comments, so this no longer applies.
I don’t understand how there exists a disease-free equilibrium in this model with finite c_m (exposure to infected meat). Because c_m is very small (new zoonotic outbreaks are rare), it is reasonable to discuss the equilibrium in the c_m goes to zero limit (especially as the propagation of this exposure to a measurable outbreak is inherently stochastic such that the ODE equilibrium isn’t meaningful anyway). If this is the assumption being made, the paper needs to state it, and explain what (if any) role is played by c_m in the analysis because it is described as a relevant parameter in the methods.
Response: You are correct. Since c_m was small, we had ignored it for the purposes of calculating the disease-free equilibrium, but we agree that this was not clear. Consequently, we have decided to remove the c_m term from the model. (Page 4)
From the abstract and methods, I expected that this paper was addressing an interesting question that is often overlooked in setting vaccination policies: under what conditions can introducing a vaccine increase long-term disease burden over the unvaccinated state? However, this perverse behavior is only briefly mentioned in the discussion as not relevant for ebolavirus. If this point is important enough to remain in the paper, I would like to see the results for analyzing vaccinated contact behavior. How large an increase in contact rates would be required at what levels of vaccination coverage for perverse outcomes to occur?
Response: You are right, we had not investigated this option as fully as we could have. We have added more analysis to this part of the manuscript and have added a new figure (Figure 2) to illustrate the result. (Page 7)
The figures need explanatory captions.
Response: We have added detailed captions to all figures.
Minor comments
Top of page 5 and throughout: Replace “the vaccine may wane over time” with “immunity from vaccination may wane over time” and similar usages.
Response: Done. (Pages 3, 6)
Page 10: Numerical simulations: is it 1000 times per parameter or 1000 times for all parameters?
Response: The latter. We have clarified. (Page 8)
Figure 4-6: bigger dots. The current symbols are distractingly small when viewed on my monitor.
Response: Done. (Figures 4 and 5)
In general, I think the information in figures 3-6 would be better represented as two bivariate scatter plots with Ro as the color (see Fig 1 of McCarthy et al for example). This would show the correlations of waning, contact, and transmission as well as their effects on Ro in a manner more informative to the reader.
Response: We like the idea of colour grading that you suggest and so have done this in Figures 4 and 5. We have also revised the boxplot figure to show a comparison of the key interventions. (Figure 3)
Reference 18 is missing “HIV” from the title.
Response: Fixed. Good catch.
Second reviewer:
I enjoy going through the paper. It addresses an important subject – assessment of one of the more potent vaccines developed for the Ebola outbreak. The model developed by the authors is ambitious in its scope, both in terms of its number of compartments as well as in terms of the realistic way that the vaccine effects are integrated into the model. At this point in time I would recommend to deem the paper as “Borderline” until some of the issues that need to be clarified get resolved. Let me list my concerns in the order of appearance in the paper:
1) The Introduction section summarizes well the key features of the Ebola disease and the societal and economic misery caused by the last Ebola outbreak. It makes a strong case for the urgent need to develop vaccines. To that effect the Introduction points out to multiple important references. What the Introduction has not done so far, but it should, is to discuss issues that are relevant for the mathematical modeling done later in the paper: What are other references that model the last or earlier Ebola outbreaks? In what way is the new model attempts to improve on existing models? What is known about simpler models of Ebola – for instance, were the pairwise infection, recovery and death coefficients estimated for some of the models based on given infection and mortality figures? It seems from the Bibliography that the only reference to other modeling efforts is that by Salem and Smith (reference 14), and explanation is provided regarding the differences in the models.
Response: This is a good point. We have added a significant number of citations to several modelling papers and listed issues that other models have been concerned with (Pages 3–4)
2) On page 2 of the paper the following curious statement is made “…during the Ebola virus outbreak in West Africa, there was an observed increase in children dying from other vaccine-preventable diseases, such as measles,..”. I was a little disappointed discovering later on that the above effect was not captured by the authors’ model.
Response: We hadn't anticipated this response, but it makes sense. We thought it was an interesting tangential fact to mention, but can understand how it might be misleading. We have removed the sentence. (Page 2)
3) On page 3 of the paper there is a statement “Once infected, with an incubation period averaging 11 days, individuals begin to experience the onset of symptoms…”. It is indeed well known, even in the simplest SIR model, that proper time delay effects must be added to any spread of infectious diseases model. The model proposed by the authors has no delay effects.
Response: We have an exposed (incubating) class for both unvaccinated and vaccinated individuals (Eu and Ev, respectively). Such a class has the effect of introducing a delay. (Page 4)
4) Numbers should be added to each of the formulas.
Response: The SummerSim house style is to only number equations that are referenced in the text. Since we do not explicitly reference any of the equations, they do not take numbers.
5) For the model on page 5 all the variables and coefficients seem to be explained, with the exception of ?, associated with the amount of vaccines.
Response: Subsequent discussion suggested that the symbol represented by ? was Λ. We would like to clarify that Λ is in fact the birth/immigration rate, not the amount of vaccines. However, the reviewer is correct to call us out on not including this parameter, precisely for the misinterpretation that such an absence can cause! We have clarified. (Page 4) We have also listed all parameters, along with description, in the table, which we have now placed closer to the model. (Page 6)
6) On page 7 the authors mention that “Using previous Ebola outbreaks and their corresponding populations at the time, an average contact rate was calculated.” Let me suggest that the authors explain it a little more. Obviously, earlier outbreak mathematical models involve less compartment due to the lack of any vaccines at the time. Contact rate coefficients are always estimated by multiple simulation runs, while least squares fitting simulation outputs (like simulated mortality figures) to the respective actual data (say actual mortality figures). In such parameter fitting the correct delay effects must be added. The authors further mention that several other contact rate parameters were simply taken from the paper by Salem and Smith. Again, it is important to understand how these were obtained.
Response: The parameter in question has been dropped from the model, after the comments of the first reviewer, so this no longer applies.
7) For the formulas on page 8, involving non-obvious extensions to the “Basic Reproduction Ratio Numbers” some more detailed derivations must be provided, and references as well (possibly such as the paper by Dickermann et al, J. Math. Biology, 1990 and others).
Response: Due to page restrictions, we have streamlined some of the analysis here and just presented the results. We have added a citation. (Page 7)
8) The p variable in the dV/dt equation is a typo error.
Response: This is not an error (p is the proportion of people vaccinated), but the misunderstanding may have occurred due to our not explicitly defining Λ.
9) Figures 4-6 are not visible. All I can see is a faint line at zero, which obviously does not explain at all the far reaching observation made on page 11 relating the eradication of the Ebola virus to the waning effect of the vaccine.
Response: We have made the dots more visible and colour coded them for further visibility. (Figures 4 and 5) We have softened our claim of eradication, as mentioned in the response to the first reviewer.
Third reviewer:
This is a well written paper that explained well the following: 1) what the authors are typing to do – figure out how much vaccines affect Ebola 2) Mathematical model used – with great amounts of details to the point that a non infectious disease modeler can understand the basics 3) how they reached the conclusions.
I would recommend accepting the paper after some minor revisions that were pointed out by the other reviewers. I will therefore ask the authors to prepare another version for the paper that addresses all the concerns of the reviewers so far.
Response: Done.
I read the other reviews and do find them helpful. For example, the figures are indeed hard to see as the last reviewer mentioned – perhaps making the points there bigger would help. It will also help adding the notations to the x axis.
Response: We have made the figures more visible and added colour to increase visibility of the results.
And I would ask the authors to consider to create a table with a list of notations – there are many symbols floating around – I believe that the authors know what they are doing, yet it would help readers if such a table exists as reference – think about someone trying to reproduce this work. I did notice table 1, yet it comes near the end, it would help if it would be near the equation declarations and include other notations.
Response: This is a good suggestion. We have done this.
And since the second reviewer pointed a potential error in one of the equations, I suggest inspecting them all again to make sure that they are ok – we do want to prevent future readers from being stuck on an issue due to a typo in an equation. Incidentally, if the authors are releasing code publicly this would be wonderful as this will eliminate all sorts of issues and allow reproducibility – hopefully they can be persuaded.
Response: We have done so and noticed that a few terms had ben omitted. Specifically, the dead from either class can now infected both classes, which was an oversight, and there was an extraneous background death rate in one of the terms. These have been fixed (Page 4) and the analysis re-done.
We are not opposed to releasing the code publicly, but it contains 40 separate programs, making this unwieldy.
And from a non expert observed point f view, can you add a few words on what is the waning effect of vaccines developed just to get an idea of ranges that can be expected – I know this is not definite, yet some orientation will help here for a novice reader.
Response: The issue at hand no longer applies, since the errors in the model resulted in different outcomes, so the waning of the vaccine is no longer a key factor for disease control. However, this is an excellent point, so we have added this as an example for the novice when explaining the inverse relationship between parameters and the length of time spent in compartments. (Page 6)
Finally, the paper does not seem to follow the author kit format for SummerSim – this is a fix that has to be made to technically allow publication – better be made sooner than later.
Response: Done.
In summary, we feel that these revisions have addressed all the points raised by the reviewers and hope that the manuscript is now acceptable.
Yours sincerely,
Stephanie Wiafe, Emma Veilleux-Gravel and Robert Smith?
Jacob Barhak
Here is transcript of additional communications between the author and reviewers:
On May 5, 2017 2:11 PM, "Robert Smith?" <rsmi...@uottawa.ca> wrote:
Hi Jacob,
I don't know how to respond to the reviewer directly, but I have a
clarification question for Zvi Roth:
5) For the model on page 5 all the variables and coefficients seem to be
explained, with the exception of ?, associated with the amount of vaccines.
The symbol represented by ? did not transfer through google docs, so could you
ask what symbol is being referred to? We don't have one that describes the
amount of vaccines, so we're a bit confused.
Thanks,
- Robert Smith?
On May 5, 2017, at 1:25 PM, Jacob Barhak <jacob....@gmail.com> wrote:
Hi Zvi,
Robert Smith? Has a question about your review of his paper, see below.
Please feel free to communicate directly.
Please CC me to any communication you may have so I can post it later as part
of the public review. So please press reply all rather than just reply to this
message.
Hopefully you can answer quickly.
Jacob
Zvi Roth
May 5 (12 days ago)
to me, Robert
I think that it was a Greek capital lambda. It's on the first term of the dx/dt
equation, denoting the amount of vaccinations.
Zvi
Sent from my iPhone
########################
There was also communications regarding the authors kit that was explored
within SCS - here it is below:
Do you know when the author's kit for 2017 will be updated? Currently, the
LaTeX kit is still for SpringSim 2016:
http://scs.org/authorskit/
I can make an approximation, but I'm not sure whether it's best to adapt the
SpringSim one for SummerSim or the SummerSim one from last year to 2017.
Cheers,
- Robert Smith?
Jacob Barhak <jacob....@gmail.com>
May 8 (9 days ago)
to Andrea, José, Robert
Hi Robert,
Are you using this file?
http://scs.org/wp-content/uploads/2016/10/SCS17LatexPaper.zip
If so, I assume you just need to uncomment line #117 in scs17paper.tex and
change the dates and conference location in the following lines:
#127: July 9-12, 2017
#130: Bellevue, WA, USA
I am not a Latex user, yet is seems these are the actions to take.
Yet you are right, perhaps it is the role of the organizers. I am CCing Jose
and Andrea since I thin it would be better done by the organizers so everyone
has the same template with the correct location.
Jose, Andrea, please correct me if you have a better solution.
Andrea D'Ambrogio
May 9 (8 days ago)
to me, Robert, José
Hi Jacob,
your solution is correct, there is a single template for all SCS
conferences/events and it is stored in a single repository.
Authors are thus requested to uncomment lines and modify text accordingly for
a specific event (this is rather simple for LaTeX users, see %AUTHORS comments
on the template).
All the best.
Robert Smith?
May 9 (8 days ago)
to Andrea, me, José
Okay, got it. Although it was helpful in this exchange to learn that the
conference was in Bellevue, rather than Seattle, as I had in my head :-)
Also, is it SummerSim-ANSS? The template has that last bit, but the website
doesn't, so I just wanted to check.
Cheers,
- Robert Smith?
________________________________________
From: Andrea D'Ambrogio [dam...@uniroma2.it]
Sent: Tuesday, 9 May 2017 1:12 AM
To: Jacob Barhak
Cc: Robert Smith?; José Luis Risco Martín
Subject: Re: Authors kit?
Jacob Barhak <jacob....@gmail.com>
May 9 (8 days ago)
to Robert, Andrea, José
So Robert,
You should just use SummerSim.
The word template does not have ANSS after SpringSim in the footer - and it
seems this will have to be changed in all papers after submission - I do not
think anyone noticed so far. I know I didn't and the word template needs change
as well.
By the way we had similar issues last year with the template.
I think that you should not bother too much. Once all the paper are submitted
someone will make sure all accepted papers are standardized.
Andrea, who is responsible for sending this message for the accepted paper
authors?
Hopefully next year this confusion can be avoided.
Robert Smith?
May 9 (8 days ago)
to me, Andrea, José
Oh, this was my misunderstanding of the instructions, not a template problem.
It should have been SCSC. Fixed now.
Cheers,
- Robert Smith?
From: Jacob Barhak [jacob....@gmail.com]
Sent: Tuesday, 9 May 2017 2:46 AM
To: Robert Smith?
Cc: Andrea D'Ambrogio; José Luis Risco Martín
Subject: Re: Authors kit?
Andrea D'Ambrogio
May 9 (8 days ago)
to Robert, me, José
Correct, it is SummerSim (multiconference) and SCSC (conference withing
SummerSim).
All the best.
#########################
Jacob Barhak
I think the revised
paper is acceptable for publication pending one necessary
revision to the figures. I also have some additional comments that may improve
it further, but I don’t consider them blocking. I appreciate the authors’
efforts to improve the manuscript.
On our end, we appreciate the dialogue that means the manuscript is improved. We have done everything requested. Additions to this round are in blue.
Necessary revision:
The axes labels on the
figures 2-5 (and transition parameters in Figure 1) must
be larger so they can be read. Only
Figure 3A is fine as-is.
Done.
Other suggestions:
Some paragraphs in the
intro are not well-linked to the model and results, and
so it is difficult to see where the paper is going from its intro. While
interesting, I suggest removing the first 4 paragraphs of page 2 about
non-specific effects on mortality, poverty, and children, and streamlining the
rest of the intro to stay close to modeling goals. If the intro is intended to
provide general background to an audience unfamiliar with infectious disease,
it would help if the general background parts were tightened up, and if the
more relevant parts of the intro were called out as directly
relevant.
Deleted. You are right that this was interesting but not strictly relevant. Also, removing this saves space for one of the other reviewer’s comments, so this worked out nicely.
End of section 2,
“perverse outcomes” of vaccination: It would be useful to
have a sentence near the definition of perversity that gives an example of when
R_v might be higher than R_0 (such as behavior change as is described in the
discussion).
Done. (Page 7)
Thank you for the interesting read,
-----
I thank the authors for adding modeling related references to
the introduction,
and for their good answer regarding the omission of time delay effects. For the
latter, the comment explaining the role of "exposed" compartments in
eliminating the need for explicit delay figures, should be included in the
paper.
Done. (Page 5)
Sorry for keeping the paper as "borderline" a bit
longer, as I still have a few
additional concerns:
1) I like figure 1 that was added. The figure does not show the
paths that
involve "infectious dead body". Need to fix the figure.
The infectious dead bodies were already included, but we realise that the caption conflated both vaccinated and unvaccinated individuals, so we have made this more explicit. However, perhaps you are referring to drawing arrows that show which compartments infected which other compartments? For readability, these are not generally drawn in unless the mechanism is very straightforward (in our case, it would involve eight additional arrows), but we have added a note to the figure caption mentioning this. (Page 4)
2) It's nice that the authors explained the way some of the
"assumed"
parameters were chosen. I think that all "assumed" parameters must be
briefly
explained. In addition, the parameters that were taken from some references
should be checked to see whether these relate to real data (in these papers) or
to "assumed" parameters in these other papers.
This is a really nice idea! In fact, doing so threw up an error in the citation, which we have fixed (Hunter and Strickland 2000), so we are grateful for this suggestion. All assumed parameters have been explained, while those from secondary sources have been justified.
3) I still don't understand what figures 3-5 tell us exactly.
More explanations
are needed. Sorry.
Not a problem! We’re always happy to make the manuscript accessible to a general audience. We’ve expanded this significantly. (Pages 7–8) Happily, there was space to do so within the page limit once the introduction was cut down, due to the other reviewer’s suggestion.
-----
The paper did undergo a transformation. And at this point, I think it is acceptable. According to SCS rules I am aware of, there is a need for two accepting reviewers, so if the authors satisfy Mike's important change, the paper can be admitted. However, I do ask the authors to try to accommodate all reviewers - another pass always helps. It may be hard in the 12 page limit, so feel free to cut introduction and some references and shorten biographies if needed.
Done.
I especially like the request that Zvi is asking for to make the data traceable back to source as much as possible. The authors are already in good grounds with regards to traceability, yet if they can make this extra step, it may be even better. Traceability and reproducibility of models are important issues - if the authors excel there, it would be commendable.
Done. We agree that this is an excellent idea.
So Please send a revised version with at least Mike's request and hopefully other changes to accommodate all. I would also ask that a link to the reviews are provided in acknowledgments - the discussion the authors and reviewers had are as important as the paper and the reviewers spent their time and effort to improve the paper, I think they deserve acknowledgement.
Definitely! Done. (Pages 10 and 12)