About half of the systems biology models could not be reproduced !!!

61 views

Skip to first unread message

Rahuman Sheriff

unread,

Aug 11, 2020, 7:46:06 AM8/11/20

to sbml-discuss, combine...@googlegroups.com

Hi all

Sometimes ago there was a long thread / discussion on reproducibility in the SBML-dicuss mailing list.

To continue on that topic, here is the contribution from BioModels.

Over the past 3 years we investigated >450 models published in 152 journals & identified the root cause for the lack of reproducibility.

About half of models fail to directly reproduce simulation results!

Read our #biorxiv_sysbio article https://doi.org/10.1101/2020.08.07.239855

Our tweet here

https://twitter.com/biomodels/status/1293148498565767168

Best regards

Sheriff

-----------------------------------------------------------------------
Rahuman Sheriff PhD
Project Leader - BioModels
European Bioinformatics Institute
European Molecular Biology Laboratory (EMBL-EBI)
Welcome Trust Genome Campus
Hinxton Cambridge CB10 1SD, UK
Email: she...@ebi.ac.uk
-----------------------------------------------------------------------
We invite you to try the recent release of new BioModels platform at http://www.ebi.ac.uk/biomodels/
We would be grateful if you can send your suggestions through the dedicated feedback form on the new platform or at biomodels-...@lists.sf.net

Sluka, James Patrick

unread,

Aug 11, 2020, 2:22:18 PM8/11/20

to sbml-d...@googlegroups.com

Sheriff (and the SBML community in general)-

I quickly read the manuscript, nice work! Is there a published list of the models examined and their scores? I suspect that the reducibility of published models that don’t use a model standard like SBML is considerably lower than ~50%.

Jim

James P. Sluka, PhD
Biocomplexity Institute

Intelligent Systems Engineering

School of Informatics, Computing and Engineering

Indiana University

Bloomington, IN USA

Office: 812-855-2441

Cell: 317-331-7465

JSl...@Indiana.edu

--
You received this message because you are subscribed to the Google Groups "sbml-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sbml-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/20200811114604.5475A62AC85_F3284FCB%40hh-mx3.ebi.ac.uk.

Jacob Barhak

unread,

Aug 11, 2020, 4:35:47 PM8/11/20

to sbml-discuss

Hi Sheriff,

Indeed you provide another example on why models should be reproducible. However, there are many obstacles on the way - some of those are not technical. Allow me to point to a recent open discussion that will reveal other issues:

https://forum.comses.net/t/issues-with-regard-to-call-for-transparency-of-covid-19-models/8433/8

You see, to allow transparency, one needs to create a proper environment. It is one thing pointing to the problem, another is trying to solve the problem. The solution should provide an incentive for others to use it. Currently there is no such good solution.

The SBML and COMBINE communities are trying to offer a solution, yet this solution is mostly technical , yet even the community solution they attempt to provide is far from sufficient since it does not provide an incentive. The inability to become a formal standard is one deterrent.

You referred to SBML as "standard SBML format" - this is a misconception. SBML is not a formal standard and is not issued by an SDO. SBML is a very nice specification driven by a community that is mostly government funded - I used it myself and think it has potential. However, why invest effort in a direction where its community includes objections to becoming a formal standard with all the legal elements?

This is one minor issue that needs mending to provide incentive to use SBML or other COMBINE standards. The link above describes other issues that need addressing, including proper licenses and use of government funding.

Your paper did a first step of identifying the problem. I urge you to take the next step and join a larger discussion on how to create incentives and a proper environment. Your call to provide source code in a computer language is great, yet there is more to do. Hopefully you have ideas to contribute.

Jacob

--
Jacob Barhak Ph.D.
https://sites.google.com/view/jacob-barhak/home

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/3394c111991a477ea6e7a23d45bc5a09%40BL-CCI-D2S05.ads.iu.edu.

Macklin, Paul

unread,

Aug 11, 2020, 5:15:20 PM8/11/20

to sbml-d...@googlegroups.com

An ISO or related gold star does nothing to incentivize using a standard.

If it’s useful (e.g., in giving access to a powerful ecosystem of compatible tools), it was already useful without the gold star.

Sorry for brevity of response – deeply immersed in covid19 work right now.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Paul Macklin, Ph.D.
Associate Professor

Director of Undergraduate Studies
Intelligent Systems Engineering
Indiana University

Founder and Lead of PhysiCell and MultiCellDS
mobile: 626-372-1203
email: mack...@iu.edu
email: Paul.M...@MathCancer.org
web: http://MathCancer.org
Twitter: @MathCancer

From: sbml-d...@googlegroups.com <sbml-d...@googlegroups.com> On Behalf Of Jacob Barhak
Sent: Tuesday, August 11, 2020 4:36 PM
To: sbml-discuss <sbml-d...@googlegroups.com>
Subject: [External] Re: [sbml-discuss] About half of the systems biology models could not be reproduced !!!

This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/CAM_y%2B3QBcAORygwSFeoVZKpvOEZtGjrQm%3DX5ZqL9itxmJwAnUA%40mail.gmail.com.

Jacob Barhak

unread,

Aug 11, 2020, 5:29:56 PM8/11/20

to sbml-discuss

Well Paul,

This is not about the star. It is about what this star means.

We had this discussion before and an extended group of SBML editors declined joining an SDO and adopting proper procedures.

I am not about to repeat the discussion - I also have little time and am also working on a newer version of my COVID-19 model as you can see from my link.

However, if someone is looking at this from the outside and has to decide if to adopt SBML, it doesn't give them an incentive - the group uses the terminology standard, yet does not invest the effort to handle the legal structure . However, to go back on track, even if SBML would become a formal standard, there should be more incentives for modelers to make things reproducible and some of those issues are discussed in the link I provided. It is a larger scope discussion that is less technical in nature.

When you find time, feel free to join that discussion and look at what suggested solutions make sense.

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/6a4779950cf04c48950e9e73035c1930%40BL-CCI-D1S02.ads.iu.edu.

Macklin, Paul

unread,

Aug 11, 2020, 6:16:53 PM8/11/20

to sbml-d...@googlegroups.com

Thanks Jacob.

I’d definitely agree that this as much or more a cultural / social issue than a technical issue. Now that the technical bottlenecks of “can a model be shared” have been solved in some key domains thanks to very hard work by the members here, there can be work on making sure it happens. Publishing, granting agencies, and promotion/tenure groups at large do not incentivize reproducible / repeatable work over non-reproducible / non-repeatable work. They currently incentivize novelty and first-to-publish, making reproducibility a dis-incentivized performance tax. These and other incentives / dis-incentives have been active topics of discussion by the scientific community for a long time.

Acceptance by a standards body does not change any of that, and there’s still disagreement on who gets to the gatekeepers on what is / isn’t a standard, what is “proper”, etc. I’d trust scientific experts in the domain over an external organization outside of the domain of expertise. If there is an SDO that’s relevant to this community, I’d be very interested to hear it.

I was a little puzzled by your point on lack of proper licenses. LGPL—the license of libSBML—is a proper, standardized license that has allowed very good interoperability of open source software. Do you perhaps you mean data reuse licenses?

Thanks for pointing out your covid19 work – looking forward to reading it!

And thank you for your discussion. While I personally don’t agree with some of your points, I agree with the goals of open transparent and reproducible science, and I’m grateful for the discussion.

Very best – Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/CAM_y%2B3Q9NzRjL-pocDeCkvfOjuWstTTK6pZKXafzYnEt1ETN3g%40mail.gmail.com.

Jacob Barhak

unread,

Aug 11, 2020, 7:12:06 PM8/11/20

to sbml-discuss

Yes Paul,

You raise some good points.

Technically SBML is perhaps the best technical specification I know for sharing models - despite not being a standard. and despite other shortcomings.

Introducing a tax on publication to incentivizing reproducibility through reforming the publication system makes sense. I am not sure how to approach it since the publication system is decentralized and there is no common standard it adheres to other than some laws that change from location to location. If you have ideas, those can be discussed, yet it is a hard problem - I am not sure I have good suggestions - perhaps someone else has. .

As for acceptance by a standard body - it is very important for dissemination. You see, the group dynamics that I witnessed when I suggested an SDO was such that reflects badly on the SBML community. Those bad examples represent control that is not property handled by a group of academics. However, this is not the discussion here. I rather see solutions and I believe that going through the legal process will fix things.

LGPL is better than other licenses, yet the copy left element is really problematic for the long term. In fact, GPL use has been declining. Also the legal Copyright mechanism that drives open source licenses has issues. You see, it gives control to the copyright holder than can be misused and disincentivize development and should not really appear in government funded work after funding goes away. There are better solutions these days that I mentioned in the discussion link I sent. If you want to discuss licenses, let us discuss those in the other link since I am not sure it will interest the SBML community in this thread and most of the discussion in there already.

And as for the COVID-19 model - feel free to contact me directly - it is not SBML related - SBML still cannot support all the mechanisms needed to implement it and funding to allow it has been denied. However, I will be happy to discuss it privately.

Hopefully separating discussions will help everyone focus.

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/b612edae77484c4582531b5ba40fb92a%40IN-CCI-D1S10.ads.iu.edu.

Macklin, Paul

unread,

Aug 11, 2020, 8:01:03 PM8/11/20

to sbml-d...@googlegroups.com

Hmm, wasn’t actually arguing for a tax – just that in the current landscape, doing the extra work for reproducibility itself is a tax compared to those who publish without.

Thanks for other points – Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/CAM_y%2B3QQsT4SaSW6jOcM9yahDc-gq1bH1oxt8mkRPnnu-udvww%40mail.gmail.com.

Rahuman Sheriff

unread,

Aug 13, 2020, 6:08:59 AM8/13/20

to sbml-discuss

Dear Jim and all,

Apologies for the late response. Due to the lockdown, I got childcare responsibilities and could’t respond in time.

On 11 Aug 2020, at 19:22, Sluka, James Patrick <jsl...@indiana.edu> wrote:

Sheriff (and the SBML community in general)-

I quickly read the manuscript, nice work! Is there a published list of the models examined and their scores? I suspect that the reducibility of published models that don’t use a model standard like SBML is considerably lower than ~50%.

We will submit the full list of examined (455) models for the journal publication. We have recommend the score card based on the experience we gathered during the exercise and the scores are not available for these models.

SBML offers a big advantage and it makes the curation process easier. Hence in our proposed scoring card, a SBML model will receive 1 point more than codes in non-standard format. But still when the necessary information is missing or incorrect in the SBML model, we cannot not reproduce it.

Out of 455 models we investigated, 66 were submitted to BioModels in SBML format and we could not reproduce 29 of them :(

Hence we proposed this 8-point score card.

Best

Sheriff

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/3394c111991a477ea6e7a23d45bc5a09%40BL-CCI-D2S05.ads.iu.edu.

Rahuman Sheriff

unread,

Aug 13, 2020, 8:03:00 AM8/13/20

to sbml-d...@googlegroups.com

Hi Jacob and Paul,

Thanks for the discussion.

@Jacob,

Thanks for highlighting the call for model transparency.

SBML is a community standard. It is widely accepted by the community and has a huge community support, which is more important than being a formal standard.

As I wrote earlier, SBML standard makes the curation process efficient, hence we use it BioModels. Hence, our newly proposed reproducibility score card will give an additional point to a SBML model.

Wrt incentive, our curated SBML models are dowloaded twice more than the non-curated ones. We have not done proper analysis, but what we know is curated models also get relatively more citation.

Encoding a model in SBML allows me to analyse, visulaize my model using >250 SBML supporting software and take advantage of their features. This is an incentive.

Doing extra work for reproducibility is not a tax, as it gives you incentive. Reproducible models when they are stamped as curated by repositories like BioModels, Physiome, JWS online, they are reused and cited more. For a scientist, a broad impact of his or her work is an incentive. It doesn’t matter, if the higher re-use of your model gives you personal satisfaction or a scientific mileage to your career, it is an incentive nevertheless.

Best

Sheriff

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/d87dae8b2f294fb69b418f2e415edfbd%40IN-CCI-D1S10.ads.iu.edu.

Jacob Barhak

unread,

Aug 13, 2020, 1:55:13 PM8/13/20

to sbml-discuss

So Sheriff,

The transparency COVID-19 paper is actually a bad scientific example. None of the original authors responded publicly to a discussion to expand on their paper in a forum selected by their corresponding author. This is an example of what not to do to fix a problem.

Yet you are publicly communicating, which is a good sign.

Do you suggest a score card to be given by an external entity post publication?

Are you suggesting that newer models will be scored by reviewers and editors prior to publication?

The former solution requires funding of the external body. The latter requires convincing a community to adopt a new approach.

Both solutions are difficult. Although may improve the situation if adopted.

The first solution is easier and can help if models reproduced this way are published again in a known location like Github. This is similar to what conda-forge did for python packages.

And I agree with you that if a model cannot be reproduced it cannot be reused. However, scientists are rewarded on publication, not on quality. This is the tax Paul was referring to if I understand correctly now. This is hard to fix.

Moreover, even reproducible models have difficulties in publication and reuse - regardless of a score card.

The problem is sociological rather than technical and may require many more tools than a score card.

I know Herbert Sauro has an Institute for reproducibility. He may have other solution ideas to contribute.

Hopefully he can join the discussion and discuss solutions.

I believe licensing and legal issues need to be a big part of the solution. Yet open discussion is a good start.

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/A92158B8-82F3-4DAD-A659-8B78E8A5BD90%40ebi.ac.uk.

Rahuman Sheriff

unread,

Aug 19, 2020, 8:31:56 PM8/19/20

to sbml-d...@googlegroups.com

Dear Jacob,

Apologies for missing your email.

On 13 Aug 2020, at 18:54, Jacob Barhak <jacob....@gmail.com> wrote:

So Sheriff,

The transparency COVID-19 paper is actually a bad scientific example. None of the original authors responded publicly to a discussion to expand on their paper in a forum selected by their corresponding author. This is an example of what not to do to fix a problem.

Yet you are publicly communicating, which is a good sign.

Do you suggest a score card to be given by an external entity post publication?

Or

Are you suggesting that newer models will be scored by reviewers and editors prior to publication?

We are suggesting all stake holders (authors / reviewers / journal editors / database curators) give the score.

The former solution requires funding of the external body. The latter requires convincing a community to adopt a new approach.

Both solutions are difficult. Although may improve the situation if adopted.

As for as BioModels is concerned, our model curators will be more than happy to adapt it.

The first solution is easier and can help if models reproduced this way are published again in a known location like Github. This is similar to what conda-forge did for python packages.

We recommend sharing of model via dedicated model databases such as BioModels, Physiome, JWSOnline which provides curation service and many other features including advanced search capability.

Our curation resource is limited and hence we would prefer to proactively curate modesl with higher score to increase the chance of our successful curation. A higher score means, a higher confidence on the model.

And I agree with you that if a model cannot be reproduced it cannot be reused. However, scientists are rewarded on publication, not on quality. This is the tax Paul was referring to if I understand correctly now. This is hard to fix.

Moreover, even reproducible models have difficulties in publication and reuse - regardless of a score card.

We also initiated modeleXchange, a consortium of modelling repositories where we aim to collaboratively build a central accession and search engine. We are using comicsDI https://www.omicsdi.org platform for this purpose. Among the features it provides is the comics score, which includes the count of downloads. As I wrote earlier, a curated reproducible model is downloaded at least twice more than the non-curated model. A scientist can certainly take advantage of this by mentioning it in the grant application and so on. But I also agree with you that the scientists should be rewarded more than this for the quality.

Best

Sheriff

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/CAM_y%2B3Rm5utS284R21SOJCZUnhM1Lr5OUzpDuZZCQ0LJq55YJA%40mail.gmail.com.

Jacob Barhak

unread,

Aug 20, 2020, 2:49:06 AM8/20/20

to sbml-discuss

Thanks Sheriff,

You are locally doing correct actions and trying to scale up. That is nice. I do believe that scaling up is not easy and you will eventually need to find better ways to standardize and make the material more genetically accessible, yet your intentions are good and it is more than ok to continue working in those directions.

One comment. Number of downloads is not really a good quality metric. Many times a model is published in multiple locations with different download metrics. For example the same software can appear in Github or in BitBucket with different downloads. Also, the number of downloads can be easily manipulated in multiple techniques. So providing this metric may play against your good intentions.

As an anecdote, I can show you a grant proposal where a reviewer was complaining about a project having 0 downloads. There was nothing to download from that project from that site and this was just a metric the web site reported generically regardless if there were any downloads. This grant proposal was supposed to fund SBML related activities. Yet this is anecdotal.

Perhaps adding links to projects that successfully reproduced a model may be more effective towards what you want. Although not perfect, this may be a better metric.

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/A90F243B-5471-4DF7-86CD-9B3AED9AF8EC%40ebi.ac.uk.

Rahuman Sheriff

unread,

Aug 20, 2020, 8:16:42 PM8/20/20

to sbml-d...@googlegroups.com

On 20 Aug 2020, at 07:48, Jacob Barhak <jacob....@gmail.com> wrote:

Thanks Sheriff,

You are locally doing correct actions and trying to scale up. That is nice. I do believe that scaling up is not easy and you will eventually need to find better ways to standardize and make the material more genetically accessible, yet your intentions are good and it is more than ok to continue working in those directions.

Thanks Jacob.

One comment. Number of downloads is not really a good quality metric. Many times a model is published in multiple locations with different download metrics. For example the same software can appear in Github or in BitBucket with different downloads. Also, the number of downloads can be easily manipulated in multiple techniques. So providing this metric may play against your good intentions.

I agree. I gave it as an example. OmicsDI also provides the number of reuse (reanalysis). During the curation process, if a model is fully or partly reused / extended to develop a new model, we annotate this information using ‘isDescribedBy’ qualifier in the new model. This allows us to track direct reuse of the model in addition to the canonical citation of the paper. Perhaps this could be helpful too.

As an anecdote, I can show you a grant proposal where a reviewer was complaining about a project having 0 downloads. There was nothing to download from that project from that site and this was just a metric the web site reported generically regardless if there were any downloads. This grant proposal was supposed to fund SBML related activities. Yet this is anecdotal.

Perhaps adding links to projects that successfully reproduced a model may be more effective towards what you want. Although not perfect, this may be a better metric.

Yes, adding the links and also the score when available will be helpful.

Best regards

Sheriff

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/CAM_y%2B3Qpbszk%3DFcWNvLcQ%3DOw8DsmpvX7ie%3D7jnAATwSk8J3jng%40mail.gmail.com.

Jacob Barhak

unread,

Aug 21, 2020, 1:16:42 AM8/21/20

to sbml-discuss

Yes Sheriff,

You should continue doing good work. We both agree there.

Yet the community needs to expand beyond the academic endeavour and work on becoming a proper standard. The current state where people refer to SBML as as standard while it is not formalized is not a good idea and has negative consequences. I suggest the term specification should be used in publication until a formal standard is issued.

Also in your scoring mechanism you should include a score for licensing. Not all licences are the same and some are more problematic than others for reuse.

You should give public-domain-like licences such as CC zero higher scores than other licences that may create issues. I have done some indemnification work in the past and had to score open software packages according to multiple criteria and I remember the difficulties.

When you start discussing adding license components to your score, do feel free to start the discussion online. I think I can contribute some ideas.

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/9950CCBD-982F-4A3A-A1DE-044D7CD24C81%40ebi.ac.uk.

Rahuman Sheriff

unread,

Aug 23, 2020, 11:57:50 AM8/23/20

to sbml-d...@googlegroups.com

Thanks Jacob,

All work we do in BioModels are under CC0. The models we curated are free to access and use under CC0.

Thanks for your suggestions

Best regards

Sheriff

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/CAM_y%2B3QTH-BXeEkqsRMU7674ar3hz%2BqoUZqJr-wzJFjD36vsZg%40mail.gmail.com.

Jacob Barhak

unread,

Aug 23, 2020, 1:10:13 PM8/23/20

to sbml-discuss

Smart Choice Sheriff,

This type of license is compatible with reuse in many scenarios including academia and industry.

Happy to learn you made a good choice. I will be sure to mention biomodels as an example of good reasoning about licences.

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/D97AF008-2EF7-4666-804B-BEC9C6352078%40ebi.ac.uk.

Pedro Mendes

unread,

Aug 24, 2020, 12:14:47 AM8/24/20

to sbml-d...@googlegroups.com

Jacob, you should probably also read this paper:

Minimum information requested in the annotation of biochemical models
(MIRIAM)
https://www.nature.com/articles/nbt1156

On 8/23/20 1:09 PM, Jacob Barhak wrote:
> Smart Choice Sheriff,
>
> This type of license is compatible with reuse in many scenarios
> including academia and industry.
>
> Happy to learn you made a good choice. I will be sure to mention
> biomodels as an example of good reasoning about licences.
>
> Jacob
>
>
> On Sun, Aug 23, 2020, 10:57 Rahuman Sheriff <she...@ebi.ac.uk
> <mailto:she...@ebi.ac.uk>> wrote:
>
> Thanks Jacob,
>
> All work we do in BioModels are under CC0. The models we curated are
> free to access and use under CC0.
>
> Thanks for your suggestions
>
> Best regards
> Sheriff
>
>
>> On 21 Aug 2020, at 06:16, Jacob Barhak <jacob....@gmail.com

>> <mailto:jacob....@gmail.com>> wrote:
>>
>> Yes Sheriff,
>>
>> You should continue doing good work. We both agree there.
>>
>>
>> Yet the community needs to expand beyond the academic endeavour
>> and work on becoming a proper standard. The current state where
>> people refer to SBML as as standard while it is not formalized is
>> not a good idea and has negative consequences. I suggest the term
>> specification should be used in publication until a formal
>> standard is issued.
>>
>> Also in your scoring mechanism you should include a score for
>> licensing. Not all licences are the same and some are more
>> problematic than others for reuse.
>>
>> You should give public-domain-like licences such as CC zero higher
>> scores than other licences that may create issues. I have done
>> some indemnification work in the past and had to score open
>> software packages according to multiple criteria and I remember
>> the difficulties.
>>
>> When you start discussing adding license components to your
>> score, do feel free to start the discussion online. I think I can
>> contribute some ideas.
>>
>> Jacob
>>
>>
>> On Thu, Aug 20, 2020, 19:16 Rahuman Sheriff <she...@ebi.ac.uk
>> <mailto:she...@ebi.ac.uk>> wrote:
>>
>>
>>
>>> On 20 Aug 2020, at 07:48, Jacob Barhak

>>>> <jacob....@gmail.com <mailto:jacob....@gmail.com>>

>>> <https://www.omicsdi.org/> platform for this purpose.

>>>>> <mack...@iu.edu <mailto:mack...@iu.edu>> wrote:
>>>>>
>>>>> Hmm, wasn’t actually arguing for a tax – just that
>>>>> in the current landscape, doing the extra work for
>>>>> reproducibility itself is a tax compared to those

>>>>> who publish without.____
>>>>> __ __
>>>>> Thanks for other points – Paul____
>>>>> __ __
>>>>>
>>>>>
>>>>> ____
>>>>> *Sorry for brevity of response – deeply immersed in
>>>>> covid19 work right now.____*
>>>>> ____
>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>> *Paul Macklin, Ph.D.
>>>>> *Associate Professor____
>>>>> Director of Undergraduate Studies
>>>>> *Intelligent Systems Engineering
>>>>> **Indiana University
>>>>> *

>>>>> Founder and Lead of PhysiCell

>>>>> <http://physicell.mathcancer.org/> and MultiCellDS
>>>>> <http://multicellds.org/>
>>>>> mobile: 626-372-1203 <tel:(626)%20372-1203>
>>>>> email: mack...@iu.edu <mailto:mack...@iu.edu>
>>>>> email: Paul.M...@MathCancer.org
>>>>> <mailto:Paul.M...@MathCancer.org>
>>>>> web: http://MathCancer.org <http://mathcancer.org/>
>>>>> Twitter: @MathCancer
>>>>> <http://www.twitter.com/MathCancer>
>>>>> ____
>>>>> __ __
>>>>> *From:*sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>
>>>>> <sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>>*On Behalf
>>>>> Of*Jacob Barhak
>>>>> *Sent:*Tuesday, August 11, 2020 7:12 PM
>>>>> *To:*sbml-discuss <sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>>
>>>>> *Subject:*Re: [External] Re: [sbml-discuss] About

>>>>> half of the systems biology models could not be

>>>>> reproduced !!!____
>>>>> __ __
>>>>> Yes Paul,____
>>>>> __ __
>>>>> You raise some good points. ____
>>>>> __ __

>>>>> Technically SBML is perhaps the best technical
>>>>> specification I know for sharing models -
>>>>> despite not being a standard. and despite other

>>>>> shortcomings.____
>>>>> __ __

>>>>> Introducing a tax on publication to incentivizing
>>>>> reproducibility through reforming the
>>>>> publication system makes sense. I am not sure how
>>>>> to approach it since the publication system is
>>>>> decentralized and there is no common standard it
>>>>> adheres to other than some laws that change from
>>>>> location to location. If you have ideas, those can
>>>>> be discussed, yet it is a hard problem - I am not
>>>>> sure I have good suggestions - perhaps someone else

>>>>> has. .____
>>>>> __ __

>>>>> As for acceptance by a standard body - it is very
>>>>> important for dissemination. You see, the group
>>>>> dynamics that I witnessed when I suggested an SDO
>>>>> was such that reflects badly on the SBML community.
>>>>> Those bad examples represent control that is not
>>>>> property handled by a group of academics. However,
>>>>> this is not the discussion here. I rather see
>>>>> solutions and I believe that going through the

>>>>> legal process will fix things. ____
>>>>> __ __

>>>>> LGPL is better than other licenses, yet the copy
>>>>> left element is really problematic for the long
>>>>> term. In fact, GPL use has been declining. Also the
>>>>> legal Copyright mechanism that drives open source
>>>>> licenses has issues. You see, it gives control to
>>>>> the copyright holder than can be misused and
>>>>> disincentivize development and should not really
>>>>> appear in government funded work after funding goes
>>>>> away. There are better solutions these days that I
>>>>> mentioned in the discussion link I sent. If you
>>>>> want to discuss licenses, let us discuss those in
>>>>> the other link since I am not sure it will interest
>>>>> the SBML community in this thread and most of the

>>>>> discussion in there already. ____
>>>>> __ __

>>>>> And as for the COVID-19 model - feel free to
>>>>> contact me directly - it is not SBML related - SBML
>>>>> still cannot support all the mechanisms needed to
>>>>> implement it and funding to allow it has been
>>>>> denied. However, I will be happy to discuss it

>>>>> privately. ____
>>>>> __ __

>>>>> Hopefully separating discussions will help everyone

>>>>> focus.____
>>>>> __ __
>>>>> Jacob____
>>>>> __ __
>>>>> __ __
>>>>> __ __

>>>>> On Tue, Aug 11, 2020 at 5:16 PM Macklin, Paul

>>>>> <mack...@iu.edu <mailto:mack...@iu.edu>> wrote:____
>>>>>
>>>>> Thanks Jacob.____
>>>>> ____

>>>>> I’d definitely agree that this as much or more
>>>>> a cultural / social issue than a technical
>>>>> issue. Now that the technical bottlenecks of
>>>>> “can a model be shared” have been solved in
>>>>> some key domains thanks to very hard work by
>>>>> the members here, there can be work on making
>>>>> sure it happens. Publishing, granting agencies,
>>>>> and promotion/tenure groups at large do not
>>>>> incentivize reproducible / repeatable work over
>>>>> non-reproducible / non-repeatable work. They
>>>>> currently incentivize novelty and
>>>>> first-to-publish, making reproducibility a
>>>>> dis-incentivized performance tax. These and
>>>>> other incentives / dis-incentives have been
>>>>> active topics of discussion by the scientific

>>>>> community for a long time.____
>>>>> ____

>>>>> Acceptance by a standards body does not change
>>>>> any of that, and there’s still disagreement on
>>>>> who gets to the gatekeepers on what is / isn’t
>>>>> a standard, what is “proper”, etc. I’d trust
>>>>> scientific experts in the domain over an
>>>>> external organization outside of the domain of
>>>>> expertise. If there is an SDO that’s relevant
>>>>> to this community, I’d be very interested to

>>>>> hear it.____
>>>>> ____

>>>>> I was a little puzzled by your point on lack of
>>>>> proper licenses. LGPL—the license of libSBML—is
>>>>> a proper, standardized license that has allowed
>>>>> very good interoperability of open source
>>>>> software. Do you perhaps you mean data reuse

>>>>> licenses?____
>>>>> ____

>>>>> Thanks for pointing out your covid19 work –

>>>>> looking forward to reading it!____
>>>>> ____

>>>>> And thank you for your discussion. While I
>>>>> personally don’t agree with some of your
>>>>> points, I agree with the goals of open
>>>>> transparent and reproducible science, and I’m

>>>>> grateful for the discussion.____
>>>>> ____
>>>>> Very best – Paul ____
>>>>>
>>>>> ____
>>>>>
>>>>> *Sorry for brevity of response – deeply
>>>>> immersed in covid19 work right now.*____
>>>>> ____
>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>> *Paul Macklin, Ph.D.
>>>>> *Associate Professor____
>>>>> Director of Undergraduate Studies
>>>>> *Intelligent Systems Engineering
>>>>> **Indiana University
>>>>> *

>>>>> Founder and Lead of PhysiCell

>>>>> <http://physicell.mathcancer.org/> and
>>>>> MultiCellDS <http://multicellds.org/>
>>>>> mobile: 626-372-1203 <tel:(626)%20372-1203>
>>>>> email: mack...@iu.edu <mailto:mack...@iu.edu>
>>>>> email: Paul.M...@MathCancer.org
>>>>> <mailto:Paul.M...@MathCancer.org>
>>>>> web: http://MathCancer.org <http://mathcancer.org/>
>>>>> Twitter: @MathCancer
>>>>> <http://www.twitter.com/MathCancer>
>>>>> ____
>>>>> ____
>>>>> *From:*sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com><sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>>*On
>>>>> Behalf Of*Jacob Barhak
>>>>> *Sent:*Tuesday, August 11, 2020 5:30 PM
>>>>> *To:*sbml-discuss
>>>>> <sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>>
>>>>> *Subject:*Re: [External] Re: [sbml-discuss]

>>>>> About half of the systems biology models could

>>>>> not be reproduced !!!____
>>>>> ____
>>>>> Well Paul,____
>>>>> ____

>>>>> This is not about the star. It is about what

>>>>> this star means. ____
>>>>> ____

>>>>> We had this discussion before and an extended
>>>>> group of SBML editors declined joining an SDO

>>>>> and adopting proper procedures. ____
>>>>> ____

>>>>> I am not about to repeat the discussion - I
>>>>> also have little time and am also working on a
>>>>> newer version of my COVID-19 model as you can

>>>>> see from my link.____
>>>>> ____

>>>>> However, if someone is looking at this from the
>>>>> outside and has to decide if to adopt SBML, it
>>>>> doesn't give them an incentive - the group uses
>>>>> the terminology standard, yet does not invest
>>>>> the effort to handle the legal structure .
>>>>> However, to go back on track, even if SBML
>>>>> would become a formal standard, there should be
>>>>> more incentives for modelers to make
>>>>> things reproducible and some of those issues
>>>>> are discussed in the link I provided. It is a
>>>>> larger scope discussion that is less technical

>>>>> in nature. ____
>>>>> ____

>>>>> When you find time, feel free to join that
>>>>> discussion and look at what suggested solutions

>>>>> make sense. ____
>>>>> ____
>>>>> Jacob____
>>>>> ____
>>>>> ____
>>>>> ____
>>>>> ____
>>>>> ____

>>>>> On Tue, Aug 11, 2020 at 4:15 PM Macklin, Paul

>>>>> <mack...@iu.edu <mailto:mack...@iu.edu>>
>>>>> wrote:____

>>>>>
>>>>> An ISO or related gold star does nothing to

>>>>> incentivize using a standard.____
>>>>> ____

>>>>> If it’s useful (e.g., in giving access to a
>>>>> powerful ecosystem of compatible tools), it

>>>>> was already useful without the gold star.____
>>>>> ____
>>>>> ____
>>>>> ____
>>>>>
>>>>> ____
>>>>>
>>>>> *Sorry for brevity of response – deeply
>>>>> immersed in covid19 work right now.*____
>>>>> ____
>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>> *Paul Macklin, Ph.D.
>>>>> *Associate Professor____
>>>>> Director of Undergraduate Studies
>>>>> *Intelligent Systems Engineering
>>>>> **Indiana University
>>>>> *

>>>>> Founder and Lead of PhysiCell

>>>>> <http://physicell.mathcancer.org/> and
>>>>> MultiCellDS <http://multicellds.org/>
>>>>> mobile: 626-372-1203 <tel:(626)%20372-1203>
>>>>> email: mack...@iu.edu <mailto:mack...@iu.edu>
>>>>> email: Paul.M...@MathCancer.org
>>>>> <mailto:Paul.M...@MathCancer.org>
>>>>> web: http://MathCancer.org
>>>>> <http://mathcancer.org/>
>>>>> Twitter: @MathCancer
>>>>> <http://www.twitter.com/MathCancer>
>>>>> ____
>>>>> ____
>>>>> *From:*sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com><sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>>*On
>>>>> Behalf Of*Jacob Barhak
>>>>> *Sent:*Tuesday, August 11, 2020 4:36 PM
>>>>> *To:*sbml-discuss
>>>>> <sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>>
>>>>> *Subject:*[External] Re: [sbml-discuss]

>>>>> About half of the systems biology models

>>>>> could not be reproduced !!!____
>>>>> ____

>>>>> This message was sent from a non-IU
>>>>> address. Please exercise caution when
>>>>> clicking links or opening attachments from

>>>>> external sources.____
>>>>> ____
>>>>> Hi Sheriff,____
>>>>> ____

>>>>> Indeed you provide another example on why
>>>>> models should be reproducible. However,
>>>>> there are many obstacles on the way - some
>>>>> of those are not technical. Allow me to
>>>>> point to a recent open discussion that will

>>>>> reveal other issues:____
>>>>> https://forum.comses.net/t/issues-with-regard-to-call-for-transparency-of-covid-19-models/8433/8
>>>>> ____
>>>>> ____

>>>>> You see, to allow transparency, one needs
>>>>> to create a proper environment. It is one
>>>>> thing pointing to the problem, another is
>>>>> trying to solve the problem. The solution
>>>>> should provide an incentive for others to
>>>>> use it. Currently there is no such good

>>>>> solution.____
>>>>> ____

>>>>> The SBML and COMBINE communities are trying
>>>>> to offer a solution, yet this solution is
>>>>> mostly technical , yet even the community
>>>>> solution they attempt to provide is far
>>>>> from sufficient since it does not provide
>>>>> an incentive. The inability to become a

>>>>> formal standard is one deterrent.____
>>>>> ____

>>>>> You referred to SBML as "standard SBML
>>>>> format" - this is a misconception. SBML is
>>>>> not a formal standard and is not issued by
>>>>> an SDO. SBML is a very nice specification
>>>>> driven by a community that is mostly
>>>>> government funded - I used it myself and
>>>>> think it has potential. However, why invest
>>>>> effort in a direction where its community
>>>>> includes objections to becoming a formal

>>>>> standard with all the legal elements?____
>>>>> ____

>>>>> This is one minor issue that needs mending
>>>>> to provide incentive to use SBML or other
>>>>> COMBINE standards. The link above describes
>>>>> other issues that need addressing,
>>>>> including proper licenses and use of

>>>>> government funding. ____
>>>>> ____

>>>>> Your paper did a first step of identifying
>>>>> the problem. I urge you to take the next
>>>>> step and join a larger discussion on how to
>>>>> create incentives and a proper environment.
>>>>> Your call to provide source code in a
>>>>> computer language is great, yet there is
>>>>> more to do. Hopefully you have ideas to

>>>>> contribute. ____
>>>>> ____
>>>>> Jacob____
>>>>> ____
>>>>>
>>>>> --
>>>>> Jacob Barhak Ph.D.
>>>>> https://sites.google.com/view/jacob-barhak/home____
>>>>>
>>>>> ____
>>>>> ____

>>>>> On Tue, Aug 11, 2020 at 1:22 PM Sluka,
>>>>> James Patrick <jsl...@indiana.edu

>>>>> <mailto:jsl...@indiana.edu>> wrote:____

>>>>>
>>>>> Sheriff (and the SBML community in

>>>>> general)-____
>>>>> ____

>>>>> I quickly read the manuscript, nice
>>>>> work! Is there a published list of the
>>>>> models examined and their scores? I
>>>>> suspect that the reducibility of
>>>>> published models that don’t use a model
>>>>> standard like SBML is considerably

>>>>> lower than ~50%.____
>>>>> ____
>>>>> Jim____
>>>>> ____
>>>>> James P. Sluka, PhD
>>>>> Biocomplexity Institute____
>>>>> Intelligent Systems Engineering____

>>>>> School of Informatics, Computing and

>>>>> Engineering____
>>>>> Indiana University____
>>>>> Bloomington, IN USA____
>>>>> Office: 812-855-2441____
>>>>> Cell: 317-331-7465____
>>>>> JSl...@Indiana.edu
>>>>> <mailto:JSl...@Indiana.edu>____
>>>>> ____
>>>>> ____
>>>>> ____
>>>>> *From:*sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>[mailto:sbml-d...@googlegroups.com
>>>>> <mailto:sbml-d...@googlegroups.com>]*On
>>>>> Behalf Of*Rahuman Sheriff
>>>>> *Sent:*Tuesday, August 11, 2020 7:46 AM
>>>>> *To:*sbml-discuss;combine...@googlegroups.com
>>>>> <mailto:combine...@googlegroups.com>
>>>>> *Subject:*[sbml-discuss] About half of

>>>>> the systems biology models could not be

>>>>> reproduced !!!____
>>>>> ____
>>>>> Hi all____
>>>>> ____

>>>>> Sometimes ago there was a long thread /

>>>>> discussion on*reproducibility*in the
>>>>> SBML-dicuss mailing list. ____

>>>>> To continue on that topic, here is the

>>>>> contribution from BioModels.____
>>>>> ____

>>>>> Over the past 3 years we investigated
>>>>> >450 models published in 152 journals &
>>>>> identified the root cause for the lack

>>>>> of reproducibility. ____

>>>>> About half of models fail to directly

>>>>> reproduce simulation results! ____
>>>>> Read our #biorxiv_sysbio article
>>>>> https://doi.org/10.1101/2020.08.07.239855____
>>>>> ____
>>>>> ____
>>>>> Our tweet here____
>>>>> https://twitter.com/biomodels/status/1293148498565767168____
>>>>> ____
>>>>> ____
>>>>> Best regards____
>>>>> ____
>>>>> Sheriff____
>>>>> ____
>>>>> ____

>>>>>
>>>>> -----------------------------------------------------------------------
>>>>> Rahuman Sheriff PhD
>>>>> Project Leader - BioModels
>>>>> European Bioinformatics Institute
>>>>> European Molecular Biology
>>>>> Laboratory (EMBL-EBI)
>>>>> Welcome Trust Genome Campus
>>>>> Hinxton Cambridge CB10 1SD, UK
>>>>> Email: she...@ebi.ac.uk

>>>>> <mailto:she...@ebi.ac.uk>

>>>>> -----------------------------------------------------------------------
>>>>> We invite you to try the recent release
>>>>> of new BioModels platform at
>>>>> http://www.ebi.ac.uk/biomodels/
>>>>> We would be grateful if you can
>>>>> send your suggestions through the
>>>>> dedicated feedback form on the new
>>>>> platform or at
>>>>> biomodels-...@lists.sf.net

>>>>> <mailto:biomodels-...@lists.sf.net>____
>>>>>
>>>>> ____

>>>>> --
>>>>> You received this message because you
>>>>> are subscribed to the Google Groups
>>>>> "sbml-discuss" group.
>>>>> To unsubscribe from this group and stop
>>>>> receiving emails from it, send an email

>>>>> tosbml-discus...@googlegroups.com
>>>>> <mailto:sbml-discuss...@googlegroups.com>.

>>>>> To view this discussion on the web
>>>>> visithttps://groups.google.com/d/msgid/sbml-discuss/20200811114604.5475A62AC85_F3284FCB%40hh-mx3.ebi.ac.uk

>>>>> <https://groups.google.com/d/msgid/sbml-discuss/20200811114604.5475A62AC85_F3284FCB%40hh-mx3.ebi.ac.uk?utm_medium=email&utm_source=footer>.____

>>>>> tosbml-discus...@googlegroups.com
>>>>> <mailto:sbml-discuss...@googlegroups.com>.