About half of the systems biology models could not be reproduced !!!

[Rahuman Sheriff]

Hi all

Sometimes ago there was a long thread / discussion on reproducibility in the SBML-dicuss mailing list. 

To continue on that topic, here is the contribution from BioModels.

Over the past 3 years we investigated >450 models published in 152 journals & identified the root cause for the lack of reproducibility. 

About half of models fail to directly reproduce simulation results! 

Read our #biorxiv_sysbio article https://doi.org/10.1101/2020.08.07.239855

Our tweet here

https://twitter.com/biomodels/status/1293148498565767168

Best regards

Sheriff

-----------------------------------------------------------------------
Rahuman Sheriff PhD
Project Leader - BioModels
European Bioinformatics Institute 
European Molecular Biology Laboratory (EMBL-EBI)
Welcome Trust Genome Campus
Hinxton Cambridge CB10 1SD, UK
Email: she...@ebi.ac.uk
-----------------------------------------------------------------------
We invite you to try the recent release of new BioModels platform at http://www.ebi.ac.uk/biomodels/ 
We would be grateful if you can send your suggestions through the dedicated feedback form on the new platform or at biomodels-...@lists.sf.net

[Jacob Barhak]

Hi Sheriff,

Regarding your reproducibility paper. How did you address precision issues if at all?

I am mentioning this since once models are transported and executed on different systems, one loses precision. Sometime results are not repeatable even on the same hardware - check out this discussion about GPU repeatability: https://github.com/rapidsai/cuml/issues/2685

I was wondering if there is some acceptable threshold you used? 

Also, in case of models that use random numbers - how did you handle the seed?

These are just random thoughts and curiousity - I just could not locate this in the paper so I figured I ask.

Precision issues are minor, yet a fully reproducible result set that can me compared using a checksum if at all possible is a very nice to have feature and highest level of reproducibility possible. 

              Jacob

--
You received this message because you are subscribed to the Google Groups "sbml-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sbml-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/20200811114604.5475A62AC85_F3284FCB%40hh-mx3.ebi.ac.uk.

[Rahuman Sheriff]

Hi Jacob,

Thanks for sharing the GPU repeatability discussion.

We discussed about precision in the methods section. 

In our work we performed manual curation and we considered any minor deviation in the reproduced  simulation results acceptable if it does not affect the scientific conclusion of the research.

"The model was considered as reproducible when it reproduced at least one of the main simulation figures in the associated research article using a software different from the one used in the original manuscript. The reproduced simulation figure, such as time course plot with and without perturbation, phase-plane plot, etc., should precisely match the original figure, any minor deviation was still considered acceptable if it did not affect the scientific conclusion of the study...”

We only used deterministic ODE models in our work to keep things simple. 

Best regards

Sheriff

To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/CAM_y%2B3SV%2BfACEtS0T99XJVVmS2x%2B0%2BF_%2BgLaVuRfBgh9rvFJJw%40mail.gmail.com.

[Jacob Barhak]

Thanks Sheriff,

Yes, what you did is reasonable to do when humans do it. I also used the "look good by eye" method - most of us do - if we like to admit it or not. This is normal for humans. 

Yet think about a future when machines will need to do it. After all we are delegating more and more responsibilities to machines. I think we should think about those issues for the more automated future we are heading towards with the nice tools we develop. 

          Jacob

 

[Rahuman Sheriff]

On 23 Sep 2020, at 14:22, Jacob Barhak <jacob....@gmail.com> wrote:

Thanks Sheriff,

Yes, what you did is reasonable to do when humans do it. I also used the "look good by eye" method - most of us do - if we like to admit it or not. This is normal for humans. 

Yet think about a future when machines will need to do it. After all we are delegating more and more responsibilities to machines. I think we should think about those issues for the more automated future we are heading towards with the nice tools we develop. 

I agree with you Jacob, hence in our scorecard, the 8th point submission of simulation data. 

Best

Sheriff