SummerSim 2017: Using Sensitivity analysis to examine the effects of an Ebola vaccine - Reviewer Jacob Barhak
Jacob Barhak
This is a well written paper that explained well the following: 1) what the authors are typing to do – figure out how much vaccines affect Ebola 2) Mathematical model used – with great amounts of details to the point that a non infectious disease modeler can understand the basics 3) how they reached the conclusions.
I would recommend accepting the paper after some minor revisions that were pointed out by the other reviewers. I will therefore ask the authors to prepare another version for the paper that addresses all the concerns of the reviewers so far.
I read the other reviews and do find them helpful. For example, the figures are indeed hard to see as the last reviewer mentioned – perhaps making the points there bigger would help. It will also help adding the notations to the x axis.
And I would ask the authors to consider to create a table with a list of notations – there are many symbols floating around – I believe that the authors know what they are doing, yet it would help readers if such a table exists as reference – think about someone trying to reproduce this work. I did notice table 1, yet it comes near the end, it would help if it would be near the equation declarations and include other notations.
And since the second reviewer pointed a potential error in one of the equations, I suggest inspecting them all again to make sure that they are ok – we do want to prevent future readers from being stuck on an issue due to a typo in an equation. Incidentally, if the authors are releasing code publicly this would be wonderful as this will eliminate all sorts of issues and allow reproducibility – hopefully they can be persuaded.
And from a non expert observed point f view, can you add a few words on what is the waning effect of vaccines developed just to get an idea of ranges that can be expected – I know this is not definite, yet some orientation will help here for a novice reader.
Finally, the paper does not seem to follow the author kit format for SummerSim – this is a fix that has to be made to technically allow publication – better be made sooner than later.
Jacob Barhak
Jacob Barhak
| Reviewer: | Jacob Barhak | |
| Secondary Reviewer: | Mike Famulare, Zvi Roth | |
| Submission: | 22 | |
| Title: | Using Sensitivity analysis to examine the effects of an Ebola vaccine | |
| Authors: | Stephanie Wiafe, Emma Veilleux-Gravel and Robert Smith? |
Scored Evaluation
| Originality (1-5) | 5 |
| Relevance (1-5) | 5 |
| Reviewer's Confidence (1-5) | 5 |
| Nominate for Best Paper Award | No |
| Overall Recommendation | Accept |
Comments
Below are reviews from external reviewers ######## Review By Mike Famulare#Here is my review of “Using Sensitivity analysis to examine the effects of an Ebola vaccine” by Wiafe, Veilleux-Gravel, and Smith?. Summary and recommendation: Borderline (Note that while I am an experienced epidemiological modeler, this is paper is my first exposure to SCS, and so my comments on points of style and scholarship may not reflect the norms of the SCS community. Nonetheless, I look forward to attending my first SCS conference and appreciate the opportunity to review for your organization.) This paper highlights the dominant roles contact behavior and (a lack of) immunity play in ebolavirus outbreaks and studies them in the context of a bifurcation analysis around the endemic equilibrium of a deterministic ordinary differential equation (ODE) model. The results emphasize the importance of understanding the durability of vaccine-derived immunity on assessing the impact of an ebolavirus vaccine on endemic transmission. The introduction serves as a useful quick primer to non-specialist audiences on the outcome and impact of the recent ebolavirus outbreak. The methods are quickly but clearly explained. The mathematical results appear to be correctly derived from within the ODE model context. However, there are problems with the framing and analysis in the context of the recent ebolavirus outbreak. First, the authors discuss ebolavirus eradication. This is inappropriate as “eradication” has a specific meaning in public health policy (Dowdle 1998) and eradication is not possible for zoonoses unless the pathogen is also eliminated from animal reservoirs. A more appropriate way to frame this question is to ask “under what conditions is the elimination of an ebolavirus strain circulating within a human population likely?”. More importantly, analysis of eradication from an endemic equilibrium with a deterministic model is not appropriate for a disease that is dominated by stochastic zoonotic outbreaks with dynamics far from equilibrium. Their treatment ignores the transient immune dynamics during outbreaks that make post-outbreak elimination more likely in finite populations. The authors in the abstract and discussion note the relevance of stochastic effects, but do not quantify them. (A recent paper that the authors couldn’t have known about prior to submission examines this for ebolavirus.) Also, by assuming endemic equilibrium with Ro near 1, the authors implicitly assumed that the mean age of infection in an unvaccinated population is comparable to a typical human lifetime (exponentially-distributed in the model) (Keeling and Rohani p 33), and thus they also implicitly assumed their conclusion that immunity would need to persist for 50 years to guarantee elimination of an endemic strain. If this was a journal publication, I would recommend rejection as the analysis as presented isn’t appropriate to the disease as currently framed by the authors. However, as this is a conference proceeding, a lower bar for work in progress is reasonable, and so my recommendation is Borderline: acceptance at the discretion of the conference organizers. Language clarifying the meaning of “eradication” in the context of human ebolavirus outbreaks can be fixed, but a stochastic model for realistic finite population sizes (or at least an ODE in the post-outbreak transient regime) is more appropriate than an endemic ODE model for studying ebolavirus elimination. Minimally, I would recommend reframing the analysis as applying in contexts where an ebola-like pathogen escapes early control efforts prior to vaccine development. Major comments How was waning effect = 0.02 chosen to define “can be theoretically eradicated?” There are samples of Ro>1 at lower values of the waning effect in Figure 4. I don’t understand how there exists a disease-free equilibrium in this model with finite c_m (exposure to infected meat). Because c_m is very small (new zoonotic outbreaks are rare), it is reasonable to discuss the equilibrium in the c_m goes to zero limit (especially as the propagation of this exposure to a measurable outbreak is inherently stochastic such that the ODE equilibrium isn’t meaningful anyway). If this is the assumption being made, the paper needs to state it, and explain what (if any) role is played by c_m in the analysis because it is described as a relevant parameter in the methods. From the abstract and methods, I expected that this paper was addressing an interesting question that is often overlooked in setting vaccination policies: under what conditions can introducing a vaccine increase long-term disease burden over the unvaccinated state? However, this perverse behavior is only briefly mentioned in the discussion as not relevant for ebolavirus. If this point is important enough to remain in the paper, I would like to see the results for analyzing vaccinated contact behavior. How large an increase in contact rates would be required at what levels of vaccination coverage for perverse outcomes to occur? The figures need explanatory captions. Minor comments Top of page 5 and throughout: Replace “the vaccine may wane over time” with “immunity from vaccination may wane over time” and similar usages. Page 10: Numerical simulations: is it 1000 times per parameter or 1000 times for all parameters? Figure 4-6: bigger dots. The current symbols are distractingly small when viewed on my monitor. In general, I think the information in figures 3-6 would be better represented as two bivariate scatter plots with Ro as the color (see Fig 1 of McCarthy et al for example). This would show the correlations of waning, contact, and transmission as well as their effects on Ro in a manner more informative to the reader. Reference 18 is missing “HIV” from the title. I’m willing to engage further with you and the authors for any desired follow up. I declare no conflicts of interest. Thank you, --Mike Mike Famulare, Ph.D. Senior Research Scientist Institute for Disease Modeling ######## Review By Zvi RothSummerSim Paper Review / Dr. Zvi Roth (FAU) Paper 22: Stephanie Wiafe, Emma Veilleux-Gravel and Robert Smith, “Using Sensitivity Analysis to Examine the Effects of an Ebola Vaccine” (4/19/17) I enjoy going through the paper. It addresses an important subject – assessment of one of the more potent vaccines developed for the Ebola outbreak. The model developed by the authors is ambitious in its scope, both in terms of its number of compartments as well as in terms of the realistic way that the vaccine effects are integrated into the model. At this point in time I would recommend to deem the paper as “Borderline” until some of the issues that need to be clarified get resolved. Let me list my concerns in the order of appearance in the paper: 1) The Introduction section summarizes well the key features of the Ebola disease and the societal and economic misery caused by the last Ebola outbreak. It makes a strong case for the urgent need to develop vaccines. To that effect the Introduction points out to multiple important references. What the Introduction has not done so far, but it should, is to discuss issues that are relevant for the mathematical modeling done later in the paper: What are other references that model the last or earlier Ebola outbreaks? In what way is the new model attempts to improve on existing models? What is known about simpler models of Ebola – for instance, were the pairwise infection, recovery and death coefficients estimated for some of the models based on given infection and mortality figures? It seems from the Bibliography that the only reference to other modeling efforts is that by Salem and Smith (reference 14), and explanation is provided regarding the differences in the models. 2) On page 2 of the paper the following curious statement is made “…during the Ebola virus outbreak in West Africa, there was an observed increase in children dying from other vaccine-preventable diseases, such as measles,..”. I was a little disappointed discovering later on that the above effect was not captured by the authors’ model. 3) On page 3 of the paper there is a statement “Once infected, with an incubation period averaging 11 days, individuals begin to experience the onset of symptoms…”. It is indeed well known, even in the simplest SIR model, that proper time delay effects must be added to any spread of infectious diseases model. The model proposed by the authors has no delay effects. 4) Numbers should be added to each of the formulas. 5) For the model on page 5 all the variables and coefficients seem to be explained, with the exception of ?, associated with the amount of vaccines. 6) On page 7 the authors mention that “Using previous Ebola outbreaks and their corresponding populations at the time, an average contact rate was calculated.” Let me suggest that the authors explain it a little more. Obviously, earlier outbreak mathematical models involve less compartment due to the lack of any vaccines at the time. Contact rate coefficients are always estimated by multiple simulation runs, while least squares fitting simulation outputs (like simulated mortality figures) to the respective actual data (say actual mortality figures). In such parameter fitting the correct delay effects must be added. The authors further mention that several other contact rate parameters were simply taken from the paper by Salem and Smith. Again, it is important to understand how these were obtained. 7) For the formulas on page 8, involving non-obvious extensions to the “Basic Reproduction Ratio Numbers” some more detailed derivations must be provided, and references as well (possibly such as the paper by Dickermann et al, J. Math. Biology, 1990 and others). 8) The p variable in the dV/dt equation is a typo error. 9) Figures 4-6 are not visible. All I can see is a faint line at zero, which obviously does not explain at all the far reaching observation made on page 11 relating the eradication of the Ebola virus to the waning effect of the vaccine. I am looking forward to the continuation of the dialog. ######## Review By Jacob Barhak# |
This is a well written paper that explained well the following: 1) what the authors are typing to do – figure out how much vaccines affect Ebola 2) Mathematical model used – with great amounts of details to the point that a non infectious disease modeler can understand the basics 3) how they reached the conclusions. I would recommend accepting the paper after some minor revisions that were pointed out by the other reviewers. I will therefore ask the authors to prepare another version for the paper that addresses all the concerns of the reviewers so far. I read the other reviews and do find them helpful. For example, the figures are indeed hard to see as the last reviewer mentioned – perhaps making the points there bigger would help. It will also help adding the notations to the x axis. And I would ask the authors to consider to create a table with a list of notations – there are many symbols floating around – I believe that the authors know what they are doing, yet it would help readers if such a table exists as reference – think about someone trying to reproduce this work. I did notice table 1, yet it comes near the end, it would help if it would be near the equation declarations and include other notations. And since the second reviewer pointed a potential error in one of the equations, I suggest inspecting them all again to make sure that they are ok – we do want to prevent future readers from being stuck on an issue due to a typo in an equation. Incidentally, if the authors are releasing code publicly this would be wonderful as this will eliminate all sorts of issues and allow reproducibility – hopefully they can be persuaded. And from a non expert observed point f view, can you add a few words on what is the waning effect of vaccines developed just to get an idea of ranges that can be expected – I know this is not definite, yet some orientation will help here for a novice reader. Finally, the paper does not seem to follow the author kit format for SummerSim – this is a fix that has to be made to technically allow publication – better be made sooner than later. |
#Here is transcript of additional communications between the author and reviewers: On May 5, 2017 2:11 PM, "Robert Smith?" rsmi...@uottawa.ca wrote: Hi Jacob, I don't know how to respond to the reviewer directly, but I have a clarification question for Zvi Roth: 5) For the model on page 5 all the variables and coefficients seem to be explained, with the exception of ?, associated with the amount of vaccines. The symbol represented by ? did not transfer through google docs, so could you ask what symbol is being referred to? We don't have one that describes the amount of vaccines, so we're a bit confused. Thanks,
Hi Zvi, Robert Smith? Has a question about your review of his paper, see below. Please feel free to communicate directly. Please CC me to any communication you may have so I can post it later as part of the public review. So please press reply all rather than just reply to this message. Hopefully you can answer quickly. Zvi Roth May 5 (12 days ago) to me, Robert I think that it was a Greek capital lambda. It's on the first term of the dx/dt equation, denoting the amount of vaccinations. Zvi Sent from my iPhone #There was also communications regarding the authors kit that was explored within SCS - here it is below: Do you know when the author's kit for 2017 will be updated? Currently, the LaTeX kit is still for SpringSim 2016: http://scs.org/authorskit/ I can make an approximation, but I'm not sure whether it's best to adapt the SpringSim one for SummerSim or the SummerSim one from last year to 2017. Cheers, - Robert Smith? Jacob Barhak jacob....@gmail.com May 8 (9 days ago) to Andrea, José, Robert Hi Robert, Are you using this file? http://scs.org/wp-content/uploads/2016/10/SCS17LatexPaper.zip If so, I assume you just need to uncomment line #117 in scs17paper.tex and change the dates and conference location in the following lines: 127: July 9-12, 2017 130: Bellevue, WA, USA I am not a Latex user, yet is seems these are the actions to take. Yet you are right, perhaps it is the role of the organizers. I am CCing Jose and Andrea since I thin it would be better done by the organizers so everyone has the same template with the correct location. Jose, Andrea, please correct me if you have a better solution. Andrea D'Ambrogio May 9 (8 days ago) to me, Robert, José Hi Jacob, your solution is correct, there is a single template for all SCS conferences/events and it is stored in a single repository. Authors are thus requested to uncomment lines and modify text accordingly for a specific event (this is rather simple for LaTeX users, see %AUTHORS comments on the template). All the best. Robert Smith? May 9 (8 days ago) to Andrea, me, José Okay, got it. Although it was helpful in this exchange to learn that the conference was in Bellevue, rather than Seattle, as I had in my head :-) Also, is it SummerSim-ANSS? The template has that last bit, but the website doesn't, so I just wanted to check. Cheers, - Robert Smith? From: Andrea D'Ambrogio [dam...@uniroma2.it] Sent: Tuesday, 9 May 2017 1:12 AM To: Jacob Barhak Cc: Robert Smith?; José Luis Risco Martín Subject: Re: Authors kit? Jacob Barhak jacob....@gmail.com May 9 (8 days ago) to Robert, Andrea, José So Robert, You should just use SummerSim. The word template does not have ANSS after SpringSim in the footer - and it seems this will have to be changed in all papers after submission - I do not think anyone noticed so far. I know I didn't and the word template needs change as well. By the way we had similar issues last year with the template. I think that you should not bother too much. Once all the paper are submitted someone will make sure all accepted papers are standardized. Andrea, who is responsible for sending this message for the accepted paper authors? Hopefully next year this confusion can be avoided. Robert Smith? May 9 (8 days ago) to me, Andrea, José Oh, this was my misunderstanding of the instructions, not a template problem. It should have been SCSC. Fixed now. Cheers,
Andrea D'Ambrogio May 9 (8 days ago) to Robert, me, José Correct, it is SummerSim (multiconference) and SCSC (conference withing SummerSim). All the best. #Authors response can be found publicly online at: https://groups.google.com/forum/#!topic/public-scientific-reviews/KHREQiTyErc #Below is Reviewer response to Author response: #Second Review by Mike Famulare: Hi Jacob, I think the revised paper is acceptable for publication pending one necessary revision to the figures. I also have some additional comments that may improve it further, but I don’t consider them blocking. I appreciate the authors’ efforts to improve the manuscript. Necessary revision: The axes labels on the figures 2-5 (and transition parameters in Figure 1) must be larger so they can be read. Only Figure 3A is fine as-is. Other suggestions: Some paragraphs in the intro are not well-linked to the model and results, and so it is difficult to see where the paper is going from its intro. While interesting, I suggest removing the first 4 paragraphs of page 2 about non-specific effects on mortality, poverty, and children, and streamlining the rest of the intro to stay close to modeling goals. If the intro is intended to provide general background to an audience unfamiliar with infectious disease, it would help if the general background parts were tightened up, and if the more relevant parts of the intro were called out as directly relevant. End of section 2, “perverse outcomes” of vaccination: It would be useful to have a sentence near the definition of perversity that gives an example of when R_v might be higher than R_0 (such as behavior change as is described in the discussion). Thank you for the interesting read, --Mike Famulare Mike Famulare, Ph.D. Senior Research Scientist Institute for Disease Modeling #Second Review by Zvi Roth: Dear Jacob, First, I agree with Mike's latest suggestions. I thank the authors for adding modeling related references to the introduction, and for their good answer regarding the omission of time delay effects. For the latter, the comment explaining the role of "exposed" compartments in eliminating the need for explicit delay figures, should be included in the paper. Sorry for keeping the paper as "borderline" a bit longer, as I still have a few additional concerns: 1) I like figure 1 that was added. The figure does not show the paths that involve "infectious dead body". Need to fix the figure. 2) It's nice that the authors explained the way some of the "assumed" parameters were chosen. I think that all "assumed" parameters must be briefly explained. In addition, the parameters that were taken from some references should be checked to see whether these relate to real data (in these papers) or to "assumed" parameters in these other papers. 3) I still don't understand what figures 3-5 tell us exactly. More explanations are needed. Sorry. Regards, Zvi Dr. Zvi S. Roth Professor Department of Computer & Electrical Engineering & Computer Science Florida Atlantic University Engineering East Building, Room 519 777 Glades Road Boca Raton, FL 33431 561-297-3471 # |
Second Review by Jacob Barhak The paper did undergo a transformation. And at this point, I think it is acceptable. According to SCS rules I am aware of, there is a need for two accepting reviewers, so if the authors satisfy Mike's important change, the paper can be admitted. However, I do ask the authors to try to accommodate all reviewers - another pass always helps. It may be hard in the 12 page limit, so feel free to cut introduction and some references and shorten biographies if needed. I especially like the request that Zvi is asking for to make the data traceable back to source as much as possible. The authors are already in good grounds with regards to traceability, yet if they can make this extra step, it may be even better. Traceability and reproducibility of models are important issues - if the authors excel there, it would be commendable. So Please send a revised version with at least Mike's request and hopefully other changes to accommodate all. I would also ask that a link to the reviews are provided in acknowledgments - the discussion the authors and reviewers had are as important as the paper and the reviewers spent their time and effort to improve the paper, I think they deserve acknowledgement. |
#Third Review Round: Dear Jacob, Everything is satisfactory now. I deem the paper Acceptable. Regards, Zvi Dr. Zvi S. Roth Professor Department of Computer & Electrical Engineering & Computer Science Florida Atlantic University Engineering East Building, Room 519 777 Glades Road Boca Raton, FL 33431 561-297-3471 #Hi Jacob, I agree with Zvi. Also, I like this review process. Thank you for organizing, --Mike Mike Famulare, Ph.D. Senior Research Scientist Institute for Disease Modeling |