SummerSim 2016: Disease Model Sharing Formats - Response for Reviewers
Jacob Barhak
From: Jacob Barhak <jacob....@gmail.com>
Date: Fri, May 6, 2016 at 7:38 AM
Subject: SummerSim Paper 10 - Response to review
To: Robert Smith? <rsmi...@uottawa.ca>, Peppino Fazio <pfa...@dimes.unical.it>
Cc: JOSE LUIS RISCO MARTIN <jlr...@ucm.es>, Lucian Smith <luciano...@gmail.com>, Maciek Jacek Swat <macie...@gmail.com>
Jacob Barhak
In addition some new elements have been added to the paper due to recent advances.
General:
REPLY: Yes the reviewer is correct. This paper signifies the beginning of handling the topic. It is a summary of a discussion started in several mailing lists and if you follow this discussion, you will find out that the discussion reached beyond this paper – we truncated the paper to a certain scope that can be verified using existing techniques. This paper represents the first milestone handling this topic and it is ad hoc initial work that was done without funding using spare time, rather than comprehensive directed effort. There will probably be continuation of this work. In fact, the topic of disease model reproducibility addressed in the paper is a topic that is being considered to be dealt with in the next Mount Hood Diabetes Challenge planned for this September. This is according to recent communications I had with Philip Clarke. So it seems this direction will gain further attention by the disease modeling community beyond this paper. Nevertheless the points raised by the reviewer were considered and the paper was amended with major changes especially the discussion at the end of the paper. Hopefully the resulting revised version will be fully accepted by the reviewer.
^^^^^^^^^
Major comments:
“Yet those publications do not provide the model in reproducible format.” I’m not sure that’s the biggest issue here. Deterministic models are entirely reproducible, for example. They still don’t get shared very much, which is your key point.
REPLY:
The reviewer assumes an ideal world that all disease models are shared as code that can be reproduced. However, in some cases it does not happen. Many times models are documented on paper rather as code or at better cases they are documented as code that is hard to use. Moreover, even if equations are deterministic, due to random nature of Monte-Carlo Simulations, results reported are arguable and it is hard to figure out why. If all models were exposed in a traceable reproducible manner, it would have been great, yet many times those are not even shared – there is no real sharing open source culture, and the few who do share, use different methods for sharing making it hard to accumulate knowledge. So yes, there are several issues that need dealing with. This paper attempts to target the issue of a sharing mechanism by providing several options for practical sharing in hope that at least one will be adopted to improve the current situation. The paper text was modified to better reflect the goal in the introduction, yet again there are several issues that are related that need improvement.
^^^^^^^^^
“sharing models through scientific papers is not practical, and that sharing code is better, yet still not used widely.” Why can’t you have both? I’m not sure that call for the abandonment of scientific publishing (which is one way this could be read) is either practical or desirable. A more concrete goal might be to call for the inclusion of code with all papers (e.g. as supplementary material or on a website).
REPLY: The argument is not against scientific publication, on the contrary, the reviewer is correct, both code and documentation on model behavior should be made public. The text was changed to avoid conflict.
^^^^^^^^^
Figure 4: These aren’t readable in black and white (which was how I printed them). Also, six of the figures (in the first and third columns) are missing labels on both the x and y axes. The other three in the center column are missing units. The last line of the figure caption is missing a comma and an “and”.
REPLY: The figures are generated using three different systems and were not intended to be printed in black and white – contemporary technology allows display in color even in print with little cost. Initially the figures were generated using default values for the SBML tools, yet after the reviewer request some effort was made to improve the figures, even for black and white viewers who now can look at the leftmost column for a reference. Some code changes were made to match colors between all images to allow better comparison in color. Yet do note that we decided to keep smooth color lines on the right and center columns for various reasons, some esthetic, and some technical. We spend a few hours on fixing the figure, as the reviewer can see from our github repository history of the code, and decided it is now in better shape.
Note, however, that the figure should not be viewed as a comparison figure. It just shows that multiple implementations exist that can generate results. The different systems use different random generation methods and seeds, and simulations were not repeated sufficient number of times to show convergence to the same numbers. It is just an example to show current implementation capabilities of some sharing formats, not rigorous analysis of the models.
If the reviewer asks for more revisions, these can be done, yet hopefully the reviewer accepts these compromises we made considering the main goal of the paper is the sharing formats – not the models.
^^^^^^^^^
Minor comments:
Abstract:
“MIST, SBML, and PharmML” You switch back and forth in your use of the Oxford comma. I personally advise against it, but you should at least be consistent.
REPLY:
The reviewer is right. This was fixed and commas are not used before the “and” throughout the document.
^^^^^^^^^
Introduction:
You reference five papers for diabetes models but then only one apiece for the other applications. (I understand why you’re doing that, given what comes later, but it still reads oddly.) Also, Reference [5] isn’t really an HIV paper.
REPLY: Yes, the reviewer is correct, one of the authors works on diabetes models and therefore the bias towards diabetes model. And do note that diabetes modeling groups are somewhat more organized and even have regular challenges, so the bias can be justified. Yet it was important to show other disease models for reference. As for reference 5- it is indeed a general technique, yet the section 4 in that reference 4 brings the Swiss HIV cohort study as an example. So the reference is still relevant. There are plenty more infectious disease examples and HIV related work that can be referenced, yet since this reference works with Markov models, it is somewhat fitting this manuscript. We can drop the reference without affecting the paper, yet hopefully the reviewer will accept our decision to keep it since it is somewhat a relevant example.
^^^^^^^^^
“modelers participating in those meeting expressed interest” This should presumably be “meetings” (or possible “in that meeting”, depending on how many meetings there were).
REPLY: Reviewer is correct – this was fixed. There were many Mount Hood challenge meetings.
^^^^^^^^^
“has been providing c code” I believe “C” should be capitalised.
REPLY: Reviewer is correct – this was fixed.
^^^^^^^^^
“probably the best model sharing example” is missing a hyphen. It should be “probably the best model-sharing example”. When you have a noun-noun compound modifier acting as an adjective, it almost always takes a hyphen.
REPLY: Reviewer is correct – this was fixed.
^^^^^^^^^
“the model and data sharing working group” is missing hyphens. It should be “the model- and data-sharing working group”. This second hyphen is for the reasons above. The first is an open hyphen, because you’re displacing the second part of the compound modifier to later.
REPLY: Reviewer is grammatically correct – Yet in this case this was kept as originally written just to avoid confusion since the name of the group is dictated by the group organizers rather than the authors of this paper. Yet the organizers were notified.
^^^^^^^^^
“A discussion was started at the mailing list” This should be “on” the mailing list or “in” the mailing list, not “at” it. Unless you’re waving placards from outside the building and shouting obscenities at it :-)
REPLY: Reviewer is correct – this was fixed hopefully without any protestors shouting anything.
^^^^^^^^^
“to figure out how suitable is SBML for disease models “ should be “to figure out how suitable SBML is for disease models” (swap “SBML” and “is”)
REPLY: Reviewer is correct – this was fixed.
^^^^^^^^^
“Pharmacometrics Markup Language - PharmML” You’re using a hyphen here when it should be an em dash (one of these: — not one of these: - ). But actually, I think you want to put the abbreviation in parentheses, as this isn’t a strong interruption, which is what the dash signifies, but rather a definition.
REPLY: Reviewer is correct – this was fixed.
^^^^^^^^^
Section 2:
“The MIcro Simulation Tool - a dedicated” This shouldn’t be a hyphen, but it’s not a dash either; it’s a colon. You’re explaining what you just introduced, not interrupting what you just said.
REPLY: Reviewer is correct – this was fixed using the word “is”
^^^^^^^^^
“a dedicated Disease modeling framework” should be “a dedicated disease-modeling framework” Don’t capitalise “disease”, and insert a hyphen.
REPLY: Reviewer is correct – this was fixed. This is a negative side effect of an author used to writing too much code and capitalizing names of variables while writing variable names. The urge to capitalize remains when writing text and the eye or other readers do not always catch this. The reviewer was very observant in this case. The hyphen was added to several other occurrences of disease-modeling.
^^^^^^^^^
“MIST was initially designed in mind for disease models” Delete “in mind”.
REPLY: The reviewer suggestion is wise. This was fixed.
^^^^^^^^^
“that serializes the database MIST uses using python pickle, and zip” Several comments here: 1) Best not to have “uses” and “using” next to each other. 2) Delete the comma here. You don’t want a joining comma because what follows it isn’t an independent clause. 3) Either define “python pickle” and “zip” for those of us who don’t know what these are or provide references. Or preferably both. (Or possibly pictures, unless they’re street names for hard drugs, which is what they sound like, in which case... don’t :-) )
REPLY: The reviewer suggestion is wise. In this case the Jargon of some of the programming community does not pass well and requires explanation to outsiders. This sentence was rephrased. And frankly, some names used by the programming community sometimes seems to be somewhat humorous bordering inadequate language. So I see the reviewers point here. Pickle and zip are pretty common within python and do not really require reference, a Google search will bring those up first. Yet the sentence was restructured.
^^^^^^^^^
“in increasing complexity starting from a trivial model.” Insert comma after “complexity”, because “starting from a trivial model” is adding to your point, not explaining it.
REPLY: The reviewer is correct. This was fixed.
^^^^^^^^^
“typically fixed and discrete. Each time step the model provides the probability” Except this isn’t what you mean. Try “typically fixed and discrete. At each time step, the model provides the probability” (i.e., insert “At” and also a comma after “step” because “At each time step” is a weak interruption.
REPLY: The reviewer is correct. This was fixed.
^^^^^^^^^
“provides the probability of transition to” I think you want “provides the probability of a transition to” (insert “a”)
REPLY: The reviewer point is good, yet the correction word “the” seems more appropriate in this case.
^^^^^^^^^
“Output requested: Amount of people in each state for years 1-10.” Change “Amount” to “number” because “people” is a countable noun. Also, change the hyphen to an en dash for 1–10. (You use an en dash for ranges and relationships. Don’t confuse it with an em dash.)
REPLY: The reviewer is correct and a fine eye for small details – much more observant than the authors for sure – many thanks.
^^^^^^^^^
“Implementation of this model in MIST, SBML, PharmML are” The last comma should be an “and”.
REPLY: The reviewer is correct. This was fixed.
^^^^^^^^^
“The MIcro Simulation Tool (MIST) implementation of this model is provided with this paper for reference and reproducibility.” Where is this provided? Probably best to say, at this juncture.
REPLY: The reviewer is correct. The Github repository was mentioned a sentence before so this is an unnecessary repetition. The paragraph was changed.
^^^^^^^^^
“different pools of model elements at a particular rate, or with a” Delete comma, as what follows is not an independent clause.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“is not part of that model, but part of its execution instructions” Delete comma, as what follows is not an independent clause.
REPLY: This was fixed in the revised manuscript. The sentence was broken into two to give additional clarity.
^^^^^^^^^
“provided by both libraries. And note that other simulation types” Delete “And”. It’s best not to start a sentence with “And” unless you absolutely have to. Which you don’t here.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“finally modeling Steps where e.g. a simulation task is described” First, don’t capitalise “steps”. Second, if you replace “e.g.” with “example”, then this would real “where example a simulation task”, which makes no sense. Try “where, for example, a simulation task”
REPLY: This was fixed in the revised manuscript by dropping the “e.g.”. And again, unnecessary capitalization is a bad habit of programmers that transitions badly to human readable text. This review provides some remedy, yet this may be a chronic disease with no cure.
^^^^^^^^^
“Each of these implementations is XML format in a form” I can’t parse this sentence. What does “is XML format” mean? Do you want “is in XML format”? I suggest rewriting for clarity.
REPLY: The reviewer points to another disease of writing on smartphones. Typing on small touch keyboards with auto-correction tends to create incomprehensible sentences. Fortunately a good reviewer can catch this. The sentence was restructured to make sense.
^^^^^^^^^
“See figure 2.” Capitalise “Figure” because it’s followed by a number.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“Healthy to Dead 0.01” I would insert a colon here: “Healthy to Dead: 0.01” And likewise for all your probability definitions. This is good practice anyway but becomes crucial in a later instance. Also, finish the bulleted list with a period.
REPLY: This was fixed in the revised manuscript in all places.
^^^^^^^^^
“Output: Amount of people in each state for years 1-10.” Change hyphen to en dash.
REPLY: This was fixed in the revised manuscript in all places. Also Amount was changed to numbers as previous comment.
^^^^^^^^^
“accompanying this paper in [21]. Yet at this point it is” Delete “Yet” and add comma after “point”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“declaring the transition matrix therefore avoiding the complexity” Insert “and” after “matrix”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“When simulated such as through the code in [21]” I would rewrite this as “When simulated through the code such as that in [21]”
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“In Python, obviously everything runs sequentially” Delete “obviously” (for obvious reasons :-) ).
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
‘For SBML, “the model” is “statements about the way things are”.’ Change “is” to “consists of”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“In the original formulation of the model, above” Delete comma.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“have equilibrated?” (aka “reached the steady state”). And the answer in this case” Delete “And”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“the first 10 years of this process assuming everything happens deterministically” Insert comma after “process”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“The simulation can be seen in the Tellurium example python script in the line with ‘gillespie’ in it.” This doesn’t sound like it’s even in English! Both for undefined terms and also the sentence structure. Rewrite and define what you mean.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“There actually is another language” Delete “actually”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“as well as the next example that adds a small variation” Change “that” to “which” and insert comma after “example” (Because the bit about adding a small variation is a nonrestrictive clause, which always takes a comma. See what I did there?)
REPLY: This was fixed in the revised manuscript. And the example is helpful.
^^^^^^^^^
“So far the model counted” Insert comma after “far”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“in each state treating them the same” Insert comma after “treating”
REPLY: The comma was added after “state” and before ”treating”
^^^^^^^^^
“cohorts are not uniform and some modelers wish to” Insert comma after “uniform” because what follows is an independent clause.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“Figure 3 shows a stratified Markov model example.” Delete “example”
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“Healthy to Sick 0.2 for Male, 0.1 for Female” This is where the colons in the bulleted list are crucial. A colon after “Sick” solves all your problems. (Also, end the list with a period.)
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“The transition Matrix now depends on the cohort, Male or Female and can” What is your comma doing here? If you’re going for a weak interruption, why isn’t there a second one after “Female”? But it’s more of an aside, so I suggest deleting the comma altogether and using parentheses: “The transition Matrix now depends on the cohort (Male or Female) and can”
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“Output: How many Male, Female, are in each disease state for each of the first 10 years?” This just doesn’t work. I realise that you’re using “Male” and “Female” as parameters, but they’re also adjectives. Plus, this output would simply be better as English. So “Output: How many men and women are in each disease state for each of the first 10 years?” works infinitely better.
REPLY: This was revised reluctantly. In this case, it is somewhat intentional since parameter names represent something that has the same name. This about is as a transition between code and human English. The reviewer’s version is proper English, yet it is a mathematical model after all and tomorrow Male, Female can be HighBloodPresure or LowHDL.
^^^^^^^^^
“In MIST this can be written” Insert comma after “MIST”, as what precedes it is a weak interruption.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“we have to just list everything explicitly” Delete “just”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“HF, HM, SF, SM, DF, and DM” You’re mostly not using the Oxford comma, so delete its use here to be consistent.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“Then instead of” Insert comma because “instead of having 4 defined transitions” is a weak interruption, so takes commas on either side if in the middle of a sentence.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“having 4 defined transitions, you’d have 8.” Write out “four” and “eight” in full.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“This is obviously inefficient, and requires” Delete comma, as what follows is a dependent clause.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“with cohorts, and behind the scenes you then” First, what you have isn’t wrong, but I think it’d be better to split the sentence in two. Second, “behind the scenes” is a weak interruption. So “with cohorts. Behind the scenes, you then”
REPLY: The sentence was restructured to avoid multiple issues.
^^^^^^^^^
“called Multistate and Multi Component Species (Multi) [14] which deals with such models.” Change “which” to that” because you have a restrictive clause.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“that have several forms, but for which” Delete comma.
REPLY: This was fixed in the revised manuscript. And even improved further.
^^^^^^^^^
‘can have a phosphate bound”, and then define’ Delete comma.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
‘writing out, “prot_xxPx -> prot_xxxx; 3.2”, “prot_xxxP -> prot_xxxx; 3.2”, etc. etc.’ Just one “etc.” will be sufficient.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
Section 3:
“As can be seen, results are pretty similar” Delete “pretty”
REPLY: This was fixed in the revised manuscript. The word “somewhat” was used. Note that results are not exact due to Monte Carlo randomness. This was explained in more details.
^^^^^^^^^
“random nature of Monte-Carlo Simulation that was” Delete the hyphen and don’t capitalise “simulation”. Also, was there only one simulation or should this be a plural?
REPLY: This was fixed in the revised manuscript. In multiple places.
^^^^^^^^^
“The similarity amongst results demonstrate how” This should be “demonstrates”, because the subject is “similarity”, not “results”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
Section 4:
“The paper suggests” I think you mean “This paper suggests”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“At the same time it is still” Insert comma after “time”.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
“dedicated to healthcare related topics” Insert hyphen. I think you can guess where :-)
REPLY: The reviewer is a good teacher – I hope the revised version is correct.
^^^^^^^^^
“such as Pharmaceuticals, clinical trials” Don’t capitalise “pharmaceuticals”.
REPLY: This was fixed in the revised manuscript. And as mentioned before, improper capitalization seems to be a chronic disease for programmers.
^^^^^^^^^
“Therefore on the long run its interface” First, it should be “in the long run”, not on. Second, this is a weak interruption, so surround it with commas.
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
References:
[6] “Drug Disease Model Reesource Forum” You’ve misspelled “Resource”. Also, I clicked on this link and that wasn’t the title on the page. (It was “Resources” and no “Forum” to be found.) Make sure the titles are accurate.
REPLY: The title was corrected, the link it to a specific discussion form within the web site
^^^^^^^^^
[10] “IMAG - Interagency Modeling and Analysis Group - Model and Data Sharing Working Group.” These hyphens should be em dashes. Although I note that neither appear on the webpage itself.
REPLY: The dashes are to indicate sub titles. This is a page within a bigger web site. I used em dash as requested
^^^^^^^^^
[16] This reference is insufficient (and also capitalised incorrectly). “pharmml” tells me nothing about it.
REPLY: The reviewer is correct, yet this is the title of the web page as it exists now. Since PharmML is part of this. The title was changed and web page authors notified.
^^^^^^^^^
[18] “SBML Discuss - Google Groups: Can SBML be used for statistical disease models?” Again, the hyphen should be an em dash. Although, again when I clicked, it was “SBML-discuss > Can SBML be used for statistical disease models?” with a hyphenated name and no “Google Groups” in the title at all.
REPLY: This was fixed in the revised manuscript. However, the google groups is appropriate since this is the host – it is equivalent perhaps to publisher name. SO it was added at the back.
^^^^^^^^^
[19] “SBMLThe System Biology Markup Language” Missing a space. (And also a .org, according to the page itself.)
REPLY: This was fixed in the revised manuscript.
^^^^^^^^^
[21] This a) isn’t accurate with what’s on the page and b) could use some more information, especially as it’s the key reference in this manuscript.
REPLY: This is the correct link – the title of a repository in Github is funny since it composed by software and so this is why the reviewer may be confused. There is no perfect way of referencing this. Yet it is important to have this reference.
^^^^^^^^^
[25] Again the hyphen should be an em dash... and isn’t on the page itself.
REPLY: The reviewer suggestion was taken.
^^^^^^^^^
In general, I think you need to be more consistent about the references, more accurate with their titles and also include more information about them (e.g. author, date, etc.). APA style has examples of how to do this.
REPLY: The reviewer is observant about the references, web pages seem to have funny formats that challenge any convention and these change rapidly. Since this work refers to many web links, there may be little that can be done beyond what exists in the revised manuscripts. However, the world changes towards use of the web so compromises should be made. Hopefully the reviewer will accept the changes made.
Reply for Reviewer 2 – Peppino Fazio?
Thank you for the supportive review. The paper was revised accordingly. Below is a reply for each of the comments.
In addition some new elements have been added to the paper due to recent advances.
^^^^^^^^^
This paper deals with the existing models of diseases, of different nature. In particular the authors focused their attention on the possibilities of sharing the proposed models, although some issues exist for this aim. The number of disease models shared openly is small and disease models possess a number of issues such as inadequate transparency and failure to reproduce published results. The main aim of the paper is to summarize several examples corresponding to disease models. In addition, the paper shows some interesting models based on Markov chains, starting from a simple one and arriving to more complex matrix structures. The paper is well structured, but some remarks need to be made:
REPLY:
The reviewer summarizes the paper well. This is a correct description
^^^^^^^^^
- First of all, the English quality should be revised: there are many parts in the paper which are a little bit confusing;
REPLY: The paper was initially constructed from a free conversation on mailing lists dedicated to model sharing. It is understandable how this may cause confusing outside the list context. However, the paper was considerably revised since then, much due to the many detailed remarks made by the first reviewer Robert Smith?. Hopefully the reviewer finds the revised version in better shape now.
^^^^^^^^^
- There is a typo into the introduction section: the verb “posses” should be turned into “possess”;
REPLY: This is indeed a very accurate observation. The reviewer has a keen eye. This was corrected of course. And it was corrected later in one more location in the paper in page 4.
^^^^^^^^^
- Please explain better “Then the model results are compared and contrasted while models are treated as black boxes.”;
REPLY: The sentence was changed to “Participating modelers are expected to generate output using their models. Then the model results are compared and contrasted while models are treated as black boxes, i.e. modelers do not expose the inner working of their models, and only inputs and outputs are considered.” It is now somewhat a long sentence, yet it seems to still be in concert with the rest of the paragraph.
By the way, I think the term black box implies that the inside is so dark that you cannot see into the box.
^^^^^^^^^
- Section 2 needs a little introduction;
REPLY: A short introduction was added just before section 2.1 to allow readers to comprehend the sections afterwards.
^^^^^^^^^
- Figures need to be described adequately, not only few words in a very short section.
REPLY: The Markov models diagrams are straightforward, and need little introduction, yet I added a few words before figure 1 to explain the diagram. Note that the text following Figures 1,2,3 adds additional information needed to comprehend the figure. Therefore those few words before figure 1 should be sufficient to explain the state transition diagram figures. As for figure 4 a few sentences were added after the introduction of figure 4 to better explain the diagram. Also the discussion of the diagram results was prolonged.
^^^^^^^^^
For the reasons above, I can recommend the paper for publication, only after a huge round of major revisions.
REPLY: The new version contains many changes pointed by both reviewers. This reply document is about 20 pages long. Since all concerns have been addressed we are in hope this convinces the reviewer to fully accept the paper for publication. We do wish to publish this paper in SummerSim since this is a relevant timely publication that may start the process towards adoption of the ideas in the paper. This is timely since the Mount Hood Challenge where diabetes modelers meet is planned in September following SummerSim and is focused this year on issues of transparency of models. In fact one of the challenges is currently planned to be reproduction of a model. Note, however, that there may be continuation for this paper once new technologies show up. If the reviewer looks are our discussions on the SBML mailing list and the PharmML mailing list, the reviewer will find micro-simulation model examples that did not make it into this paper. We do not plan to include this in this revision since we wish tool and model sharing technology to advance before we publish more examples. So this paper is indeed work in progress, yet we wish to publish this version as a milestone and publish extensions regarding micro-simulation in another paper when time arrives. We hope the reviewer will see this importance and agree to accept the paper.
2nd Reply for Reviewer 1 – Robert Smith?
Below are responses for the reviewer at the 2nd review round.
^^^^^^^^^
Disease Model Sharing Formats review II
General
The authors have done an excellent job at taking into account both the scope and the detail of the changes I requested. Because much has changed, I approached the article from scratch, so have some further comments. These will improve the language in the manuscript, in order to make it the best version of the paper that it can be. I strongly recommend it for publication once these changes have been taken into account. I predict that this will become a highly cited paper; it’s certainly one that has had me thinking and will be something I will pass along to others in the future.
REPLY: The authors thank the reviewer for his confidence in this work and for the extra time and effort taken to review this paper.
^^^^^^^^^
Specific comments:
Title:
Your title is somewhat clunky. What’s the adjective here? I think it’s the triple compound “Disease Model Sharing” that is modifying “Formats”, in which case you should have “Disease-Model-Sharing Formats”. But that’s hideously ugly, so I recommend “Sharing Formats for Disease Models” as more elegant. Also, I think it brings the “sharing” part more to the fore, which is really your focus.
REPLY: The reviewer’s suggestion was implemented. It does sound better.
^^^^^^^^^
Introduction:
“analysis or for study outcome” would be better as “analysis or to study outcome” (change “for” to “to”)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“Many disease models exist for” would read better as “Models exist for”, to avoid repetition of “disease” in the same sentence.
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
I accept your reasoning about the multiple citations of diabetes references... but a) it just reads oddly and b) you have those same citations in the next column. So I would simply cite one or two papers for diabetes here.
REPLY: The reviewer’s suggestion was implemented. Only the UKPDS risk engine was used.
^^^^^^^^^
I accept your argument that [7] is an important paper... it just isn’t an important HIV paper. I would add in another HIV paper (one with HIV in the title!), either to replace [7] or in addition to it.
REPLY: The reviewer’s suggestion was implemented. A paper by Arielle Lasry who is now is the CDC was used as an example – it is now reference [15].
^^^^^^^^^
“shared openly is small and disease models possess” is missing a comma after “small”. The two phrases are independent, so the conjunction takes a joining comma.
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“positive impact is for the 'Mount Hood Challenge'” Delete “for”
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“are treated as black boxes, i.e. modelers do not” You have a comma splice here, which should be avoided. Change the comma to a semicolon. Also, insert a comma after “i.e.” This stands for “id est”, meaning “that is”... and if you were to say “that is, modelers do not” you would use a comma (because “that is” is a weak interruption), so the same applies to “i.e.” Hence: “are treated as black boxes; i.e., modelers do not”
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“issues with disease model sharing” should be “issues with disease-model sharing” (insert hyphen)
“there is no model sharing community” should be “there is no model-sharing community”(insert hyphen)
“deal with the latter issue assuming that” should be “deal with the latter issue, assuming that” (insert comma, because what follows it is a nonrestrictive clause)
REPLY: The reviewer’s suggestion was implemented for the entire paragraph.
^^^^^^^^^
Examples:
“In this section the purpose” should be “In this section, the purpose” (insert comma, because “In this section” is a weak interruption)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“as shown in sections 2.2-2.4, 2)” should be “as shown in Sections 2.2–2.4 and 2)” (capitalise “Sections”, since it is followed by a number; change the hyphen to an en dash; change the comma to “and”, since your second item is the last in your list)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“and model sharing formats” should be “and model-sharing formats” (insert hyphen)
REPLY: The reviewer’s suggestion was implemented. It was also changed throughout the document.
^^^^^^^^^
Example 1:
“At each time step the model provides” should be “At each time step, the model provides” (insert comma)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“In that diagram boxes with names” should be “In that diagram, boxes with names” (insert comma)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“provides a reproducible Monte Carlo based simulation capabilities on top or model encoding in MIST proprietary format.” Okay, several things have gone haywire with this. First, you have “a” single thing, but multiple “capabilities”, so don’t mix singular with plural. Second, what are the capabilities on top of? Third, should the “or” be “of”? (This would answer the second question.) Finally, you need an en dash for “Monte Carlo–based simulation” (because you have a compound modifier, but that compound consists of a proper noun with multiple parts; it would ordinarily be “Monte-Carlo-based simulation” with two hyphens, but because “Monte Carlo” is a singular, capitalised term, the en dash steps in to make it work. Complicated I know, but the English language has some elegant solutions to some pretty obscure problems. The en dash is basically the salad fork of punctuation: use it well and you’ll be revered in polite society :-) ) I think this is what you want: “provides reproducible Monte Carlo–based simulation capabilities on top of model encoding in MIST proprietary format.” Whew!
REPLY: The reviewer’s suggestion was implemented. The reviewers in depth understanding of English is amazing.
^^^^^^^^^
“reactions in mind, but can be used” should be “reactions in mind but can be used” (delete comma because what follows is not an independent sentence, so you don’t use a joining comma)
REPLY: The reviewer’s comment revealed a long convoluted sentence. It was split into two sentences. This was rephrased from the discussions on the SBML mailing list. The language was improved to make it more presentable.
^^^^^^^^^
“A model PharmML is separated” would be better as “A PharmML model is separated” (i.e., swap “model” and “PharmML”)
REPLY: The reviewer’s suggestion was implemented. The reviewer has done great effort for reading again through the revised paper. The authors did this several times and did not catch this word swap. Many thanks.
^^^^^^^^^
“This simplifies comprehension of the model and readers can actually read” should have a comma after “model” (because what follows is an independent sentence, so the joining comma applies). But please don’t inflict “so readers can read” upon your poor reader who has to read! :-)
Example 2:
“the following transition Matrix” should be “the following transition matrix” (don’t capitalise “matrix”)
“same as the mathematical Matrix multiplication” should be “same as the mathematical matrix multiplication” (don’t capitalise “matrix”)
REPLY: The reviewer’s suggestion was implemented in both cases. This is a nice catch and I take blame for the over capitalization. It is a nasty habit I formed by writing code. It is hard to get rid of it now and I do not see those issue when I read text since I am used to reading capitalized code.
^^^^^^^^^
“This triple quoted code” should be “This triple-quoted code” (insert hyphen)
“translated to SBML which is declarative format” should be “translated to SBML, which is a declarative format” (insert comma and insert “a”)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
‘the statement "Healthy to Dead 0.01” and the statement “Healthy to Sick 0.2” are’ should be ‘the statement "Healthy to Dead: 0.01” and the statement “Healthy to Sick: 0.2” are’ (Insert two colons to match the actual statements. Also, note that you’re mixing vertical and curled quotes; these should be unified throughout.)
REPLY: The reviewer’s suggestion was implemented. And the reviewers eye for detail is very keen.
^^^^^^^^^
‘ “assuming everything happens deterministically?”. Or, one can ask’ Delete the period; it gets overwritten by the question mark within the quotes.
REPLY: The reviewer’s suggestion was implemented. It shows the reviewer has many hours of English writing in their past.
^^^^^^^^^
“simulations go beyond the traditional Markov model as well as the next example” While I’m sure that’s true, I don’t think this is what you mean. I think you want “simulations go beyond the traditional Markov model, as shown in the next example” (change “as well as” to “as shown in” and insert comma)
REPLY: The reviewer is correct. This was changed.
^^^^^^^^^
“which is useful considering clinical trials” should be “which is useful when considering clinical trials” (insert “when”)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“In this example there are 3 disease states” should be “In this example, there are three disease states” (insert comma and write out cardinal numbers in full when they’re small)
REPLY: The reviewer’s suggestion was implemented. And a missing period was added in the next couple of sentences.
^^^^^^^^^
“How many men and women, are in each disease state” Delete comma.
REPLY: The reviewer’s suggestion was implemented. Thanks for catching this.
^^^^^^^^^
“This example can be solved as two separate problems for male and female and then recombined” should be “This example can be solved as two separate problems for men and women and then recombined” (change “male” to “men” and “female” to “women”)
REPLY: The reviewer’s suggestion was implemented. This is a remnant from a previous version where Male and Female were used as parameters. This change makes this more human readable although the link to code loosens. Yet this is all for the good.
^^^^^^^^^
“Yet we wish to explore how this resolved in different” should be “Yet we wish to explore how this is resolved in different” (insert “is”)
REPLY: Thanks for this correction. Small words are hard to spot and the reviewer has a good eye for details.
^^^^^^^^^
“HF, HM, SF, SM, DF and DM where A=Alive, D=Dead, H=Healthy, S=Sick, M=Male, F= Female” You can delete “A=Alive”, since it isn’t in the list.
REPLY: The reviewer is indeed observant – this was deleted.
^^^^^^^^^
“Then instead of, having four” should be “Then, instead of having four” (delete comma from after “of” but insert one after “Then”)
REPLY: The reviewer is correct again. This was corrected.
^^^^^^^^^
Results:
“in acronyms, i.e. A=Alive” should be “in acronyms; i.e., A=Alive” (change comma to semicolon and insert comma after “i.e.”; also, do you want “A=Alive” here, given that it wasn’t in the earlier list?)
“seen in all plots that number of dead” should be “seen in all plots that the number of dead” (insert “the”)
“while number of alive individuals” should be “while the number of alive individuals” (insert “the”)
“sick individuals fluctuates depending on several factors” should be “sick individuals fluctuates, depending on several factors” (insert comma)
“some fluctuations are due do Monte Carlo noise” should be “some fluctuations are due to Monte Carlo noise” (change “do” to “to”)
REPLY: The reviewer has good suggestions for the paragraph. And yes, A=Alive should be in this list since it is seen in the plots for the first example. And the last catch of “do” to “to” demonstrates the keen eye.
^^^^^^^^^
“demonstrates how the same model can be represented” would be better as “demonstrates how the same scenario can be represented” (change “model” to “scenario” to avoid repetition with “modeling” later in the same sentence)
REPLY: The reviewer’s suggestion was implemented. And it is indeed better phrasing.
^^^^^^^^^
“slight differences are seen since random generators” should be “slight differences are seen, since random generators” (insert comma)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“with the simulation tools which is not necessary” should be “with the simulation tools, which is not necessary” (insert comma)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“Therefore modelers have tool choices even in this point in time and can make their modeling work” would be better as “Therefore modelers have tool choices and can make their modeling work” (delete “even in this point in time”, which is both strange and unnecessary)
REPLY: The reviewer’s suggestion was implemented. The even in this point in time was intended to emphasize the modeling folk can start using those tools immediately. We did not have those tools a decade ago, so it is a good time to start using them to move the field forward. Yet the reviewer is correct, this emphasis complicates the language. And a hyphen was added at the end to have a week interruption. Hopefully the reviewers teachings have been absorbed well.
^^^^^^^^^
Figure 4:
“Example 3 (bottom row); with” should be “Example 3 (bottom row), with” (change semicolon to comma)
“different implementations using: Tellurium” (delete colon)
“All plots represent number of individuals” should be “All plots represent the number of individuals” (insert “the”)
REPLY: The reviewer’s suggestion was implemented for the figure. The reviewer again shows the contrast between human readable language and coding languages the previous punctuation would have been more appropriate for a computer program. The authors thank that reviewer for those corrections.
^^^^^^^^^
Discussion:
“This paper suggests MIST, SBML, or PharmML” Delete the last comma, as what follows is a dependent clause.
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“has other capabilities such as annotations” should be “has other capabilities, such as annotations”
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“used to provide meta data about parameters” should be “used to provide metadata about parameters” (make “metadata” one word)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“includes many tools and featured in model databases” should be “includes many tools and is featured in model databases” (insert “is”)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“it is still developing to include new capabilities” should be “it is still developing new capabilities” (delete “to include”; I know what you’re trying to say here, but it isn’t quite right and is much stronger with the revision)
REPLY: The reviewer’s suggestion was implemented. This is indeed much better.
^^^^^^^^^
“based on SBML giving a variety” should be “based on SBML, giving a variety” (insert comma)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“in the paper are still abstract since the” should be “in the paper are still abstract, since the” (insert comma)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“the exchange standards is still evolving” should be “the exchange standards are still evolving” (change “is” to “are”)
REPLY: The reviewer noticed a good point. However, the exchange standard here is singular – PharmML there fore the ‘s‘ should be deleted.
^^^^^^^^^
“Once disease models will be shared using common standards” should be “Once disease models are shared using common standards” (change “will be” to “are”)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“Sharing models will be easy as using code libraries” should be “Sharing models will be as easy as using code libraries” (insert “as” before “easy”)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“And the models will therefore be reproducible.” Delete “And”. Best not to start a sentence this way in formal writing.
REPLY: The reviewer’s suggestion was implemented. Actually, the word ‘the’ before models was also deleted.
^^^^^^^^^
“more complicated than the working example presented in this paper” should be “more complicated than the working examples presented in this paper” (change “example” to “examples”, since you have more than one)
REPLY: The reviewer’s suggestion was implemented. Thank you for the second reading and the good catch.
^^^^^^^^^
“2) Improve tools that implement the sharing standards” should be “2) Improving tools that implement the sharing standards” (change “Improve” to “Improving” to match the tense of the other two list items)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“hundreds of systems biology models” should be “hundreds of systems-biology models” (insert hyphen)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“the pharmacometrics lacked until very recently such resource” I don’t understand what you mean here. I think you might be missing a word, at the very least.
REPLY: The reviewer is observant. This was changed to: “the pharmacometrics field lacked such resource, until very recently”
^^^^^^^^^
“could develop to a long sought after community resource” should be “could develop into a long-sought-after community resource” (change “to” to “into” and insert two hyphens)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“perhaps the most difficult task since it requires” should be “perhaps the most difficult, since it requires” (delete “task” but insert a comma)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^
“This paper is just a start and hopefully more advances will follow” should be “This paper is just a start, and hopefully more advances will follow” (insert comma)
REPLY: The reviewer’s suggestion was implemented.
^^^^^^^^^