About half of the systems biology models could not be reproduced !!!

12 views
Skip to first unread message

Rahuman Sheriff

unread,
Aug 11, 2020, 7:46:05 AM8/11/20
to sbml-discuss, combine...@googlegroups.com
Hi all

Sometimes ago there was a long thread / discussion on reproducibility in the SBML-dicuss mailing list. 
To continue on that topic, here is the contribution from BioModels.

Over the past 3 years we investigated >450 models published in 152 journals & identified the root cause for the lack of reproducibility. 
About half of models fail to directly reproduce simulation results! 
Read our #biorxiv_sysbio article https://doi.org/10.1101/2020.08.07.239855


Our tweet here


Best regards

Sheriff


-----------------------------------------------------------------------
Rahuman Sheriff PhD
Project Leader - BioModels
European Bioinformatics Institute 
European Molecular Biology Laboratory (EMBL-EBI)
Welcome Trust Genome Campus
Hinxton Cambridge CB10 1SD, UK
Email: she...@ebi.ac.uk
-----------------------------------------------------------------------
We invite you to try the recent release of new BioModels platform at http://www.ebi.ac.uk/biomodels/ 
We would be grateful if you can send your suggestions through the dedicated feedback form on the new platform or at biomodels-...@lists.sf.net


Jacob Barhak

unread,
Sep 19, 2020, 9:35:04 AM9/19/20
to sbml-discuss, combine-discuss
Hi Sheriff,

Regarding your reproducibility paper. How did you address precision issues if at all?

I am mentioning this since once models are transported and executed on different systems, one loses precision. Sometime results are not repeatable even on the same hardware - check out this discussion about GPU repeatability: https://github.com/rapidsai/cuml/issues/2685

I was wondering if there is some acceptable threshold you used? 
Also, in case of models that use random numbers - how did you handle the seed?

These are just random thoughts and curiousity - I just could not locate this in the paper so I figured I ask.

Precision issues are minor, yet a fully reproducible result set that can me compared using a checksum if at all possible is a very nice to have feature and highest level of reproducibility possible. 

              Jacob



--
You received this message because you are subscribed to the Google Groups "sbml-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sbml-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/20200811114604.5475A62AC85_F3284FCB%40hh-mx3.ebi.ac.uk.

Rahuman Sheriff

unread,
Sep 23, 2020, 9:10:17 AM9/23/20
to sbml-discuss, Jacob Barhak, combine-discuss
Hi Jacob,

Thanks for sharing the GPU repeatability discussion.

We discussed about precision in the methods section. 

In our work we performed manual curation and we considered any minor deviation in the reproduced  simulation results acceptable if it does not affect the scientific conclusion of the research.

"The model was considered as reproducible when it reproduced at least one of the main simulation figures in the associated research article using a software different from the one used in the original manuscript. The reproduced simulation figure, such as time course plot with and without perturbation, phase-plane plot, etc., should precisely match the original figure, any minor deviation was still considered acceptable if it did not affect the scientific conclusion of the study...”

We only used deterministic ODE models in our work to keep things simple. 


Best regards
Sheriff






Jacob Barhak

unread,
Sep 23, 2020, 9:22:27 AM9/23/20
to Rahuman Sheriff, sbml-discuss, combine-discuss
Thanks Sheriff,

Yes, what you did is reasonable to do when humans do it. I also used the "look good by eye" method - most of us do - if we like to admit it or not. This is normal for humans. 

Yet think about a future when machines will need to do it. After all we are delegating more and more responsibilities to machines. I think we should think about those issues for the more automated future we are heading towards with the nice tools we develop. 

          Jacob

 

Rahuman Sheriff

unread,
Sep 25, 2020, 6:29:21 AM9/25/20
to Jacob Barhak, sbml-discuss, combine-discuss

On 23 Sep 2020, at 14:22, Jacob Barhak <jacob....@gmail.com> wrote:

Thanks Sheriff,

Yes, what you did is reasonable to do when humans do it. I also used the "look good by eye" method - most of us do - if we like to admit it or not. This is normal for humans. 


Yet think about a future when machines will need to do it. After all we are delegating more and more responsibilities to machines. I think we should think about those issues for the more automated future we are heading towards with the nice tools we develop. 

I agree with you Jacob, hence in our scorecard, the 8th point submission of simulation data. 

Best
Sheriff
Reply all
Reply to author
Forward
0 new messages