Adding Field/s to dataset_description.json describing known issues and Usability(usable/non-usable)

121 views
Skip to first unread message

Suyash

unread,
Sep 26, 2017, 6:27:50 PM9/26/17
to bids-discussion
hello All, 


During a discussion with Chris M, Oscar E, Ross B, we thought of proposing a  field that will share information about the dataset being usable or unusable somewhere along with metadata. 

We might very well think that why unusable dataset are published in public domain at first place however, there are/can be cases when they may not get filtered out. A simple use case is when one or more header information is either incomplete (eg: s_form, qform etc) or blank; usability of the dataset can get affected and such datasets are available publicly. Missing sform, qform translates to missing orientation issue limiting its usage. For the same, we can very well add a note in README, but from software development perspective wouldn't it be a good idea to have an optional field for this in dataset_description.json?
In addition to this how about adding an optional field for known issues?

Let us know what you call think?


Best, Suyash

Thomas Nichols

unread,
Sep 27, 2017, 4:29:47 AM9/27/17
to BIDS Discussion
I think the challenge here is that there are so many different ways in which data can be 'unusable'. Someone who has the patience can get around a missing sform/qform, so is it really "unusable" with no qualification?

But the use case that comes to mind is resting state fMRI, where you can have data that -- by many standards -- is crap, but is shared none-the-less (Cameron Craddock has spoken passionately about the importance of sharing all data, good bad & ugly).   

Do we have a JSON field for "QC: Pass, Fail"?  And then "QC comments"?

-Tom

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/2417a17f-cad0-43e6-b460-2cd358ca8eae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

__________________________________________________________
Thomas Nichols, PhD
Professor of Neuroimaging Statistics
Oxford Big Data Institute
Li Ka Shing Centre for Health Information and Discovery
Nuffield Department of Population Health
University of Oxford


Auer, Tibor

unread,
Sep 27, 2017, 4:35:50 AM9/27/17
to bids-di...@googlegroups.com

I like the idea of referring to QC – I feel almost tempting to make it mandatory. J

However we should also provide a field to specify what kind of QC (link to the tool/BIDS App) and metrics (e.g. SNR) has been used.

 

Vale,

Tibor

 

Auer, Tibor (Ph '99)

 

From: bids-di...@googlegroups.com [mailto:bids-di...@googlegroups.com] On Behalf Of Thomas Nichols
Sent: 27 September 2017 09:30
To: BIDS Discussion <bids-di...@googlegroups.com>
Subject: Re: [bids-discussion] Adding Field/s to dataset_description.json describing known issues and Usability(usable/non-usable)

 

I think the challenge here is that there are so many different ways in which data can be 'unusable'. Someone who has the patience can get around a missing sform/qform, so is it really "unusable" with no qualification?

 

But the use case that comes to mind is resting state fMRI, where you can have data that -- by many standards -- is crap, but is shared none-the-less (Cameron Craddock has spoken passionately about the importance of sharing all data, good bad & ugly).   

 

Do we have a JSON field for "QC: Pass, Fail"?  And then "QC comments"?

 

-Tom

On Tue, Sep 26, 2017 at 11:27 PM, Suyash <suyas...@gmail.com> wrote:

hello All, 

 

 

During a discussion with Chris M, Oscar E, Ross B, we thought of proposing a  field that will share information about the dataset being usable or unusable somewhere along with metadata. 

 

We might very well think that why unusable dataset are published in public domain at first place however, there are/can be cases when they may not get filtered out. A simple use case is when one or more header information is either incomplete (eg: s_form, qform etc) or blank; usability of the dataset can get affected and such datasets are available publicly. Missing sform, qform translates to missing orientation issue limiting its usage. For the same, we can very well add a note in README, but from software development perspective wouldn't it be a good idea to have an optional field for this in dataset_description.json?

In addition to this how about adding an optional field for known issues?

 

Let us know what you call think?

 

 

Best, Suyash

 

--

You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--

You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CAJoTcz4iWpJkzb%3D2B9PrFMT4FEsRoEqBo_qbC3cs74Bw3hV1Pg%40mail.gmail.com.

Harms, Michael

unread,
Sep 27, 2017, 9:35:31 AM9/27/17
to bids-di...@googlegroups.com

 

I think there are way too many ways to conduct QC (and no consensus) for a hard “Pass, Fail” metric to be appropriate.

 

-- 

Michael Harms, Ph.D.

-----------------------------------------------------------

Conte Center for the Neuroscience of Mental Disorders

Washington University School of Medicine

Department of Psychiatry, Box 8134

660 South Euclid Ave.                        Tel: 314-747-6173

St. Louis, MO  63110                                          Email: mha...@wustl.edu


For more options, visit https://groups.google.com/d/optout.

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

eric...@gmail.com

unread,
Sep 27, 2017, 9:37:31 AM9/27/17
to bids-discussion
I agree with both Thomas Nichols and tibor.auer.  If you are claiming a piece of data is "unusable" that seems a bit extreme.  You may just as well call it "needs_work" since it is undoubtedly valid or usable in some way, but may not pass QC (as mentioned) or perhaps is missing some metadata.  If missing metadata is the concern, then how about "missing_metadata"?  Either way, I disagree with the "unusable" proposal as is, though I do like the second idea more, "known_issues".  There could be "unknown_issues" causing someone else's BIDS App processing to not have enough information, but the nature of an unknown issue is it is unknown at the time of release.

~Eric

Satrajit Ghosh

unread,
Sep 27, 2017, 9:39:53 AM9/27/17
to bids-di...@googlegroups.com

I think there are way too many ways to conduct QC (and no consensus) for a hard “Pass, Fail” metric to be appropriate.


couldn't agree more. 

cheers,

satra

--

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/9DCBEA50-1274-4411-BDDC-FCA3486EF9EC%40wustl.edu.

Satrajit Ghosh

unread,
Sep 27, 2017, 9:47:37 AM9/27/17
to bids-di...@googlegroups.com
to follow up on michael's point a little more - there are many ways to conduct QC, this came up at the first bids sprint. we do not have any specific standards for QC. there are many approaches that people have adopted for various projects, but  these are often in relation to the target application/question of interest. 

QC involves:

1. a protocol involving human or algorithms to determine a quantitative (hopefully) or scale of quality
2. decision making criteria based on 1 and the target application/test of interest

we now have several measures that people extract (MRIQC, QAP, various FD/movement measures). but we haven't figured out 2

it seems having a set of measures (objective or subjective) that people filter for in determining whether a dataset is useful and should be included (include MRIQC derivatives on every dataset). however, filtering involves one knowing thresholds, ranges on measures that render something usable/unusable. i believe we have not reached that point in any aspect of imaging.

cheers,

satra

Chris Gorgolewski

unread,
Sep 27, 2017, 9:54:50 AM9/27/17
to bids-discussion
I think the case Suyash had in mind is a bit more extreme than "this subject moved way too much". It's a case of missing qform/sform which means that one cannot determine which side of the brain is left and which is right. 

Should metadata issues like this one be mentioned in any of the JSON files or is it sufficient to just mention this in the README file?

Best,
Chris

PS I admit this is a bit of an edge case - it only applies to some old data and I doubt we will see much of such cases in the future.

To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CA%2BA4wOmCmWvBFkX8gU%2BKcy4iFExqKTuyoTVSaKxdn5eKeiY4cQ%40mail.gmail.com.

Thomas Nichols

unread,
Sep 27, 2017, 9:55:26 AM9/27/17
to BIDS Discussion
I guess it's clear that there can be no *one* QC, but that it should be clear how to encode QC results in BIDS.  Even if it's like

QC: TomNicholsManualQC = 0     # up to 10
or 
QC: TomNicholsManualQC = 'Pass'   # or 'Fail'
or 
QC: TomNicholsManualQC = 'ItSux'   # free text

Always need to allow for a multitude of QC results, but need to say how to represent the most commonly used ones (i.e. quantitative scale; binary; free text).

-Tom


On Wed, Sep 27, 2017 at 2:47 PM, Satrajit Ghosh <sa...@mit.edu> wrote:
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CA%2BA4wOmCmWvBFkX8gU%2BKcy4iFExqKTuyoTVSaKxdn5eKeiY4cQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Vince Calhoun

unread,
Sep 27, 2017, 9:58:28 AM9/27/17
to bids-di...@googlegroups.com

+1

 

From: bids-di...@googlegroups.com [mailto:bids-di...@googlegroups.com] On Behalf Of Thomas Nichols
Sent: Wednesday, September 27, 2017 7:55 AM
To: BIDS Discussion <bids-di...@googlegroups.com>
Subject: Re: [bids-discussion] Adding Field/s to dataset_description.json describing known issues and Usability(usable/non-usable)

 

I guess it's clear that there can be no *one* QC, but that it should be clear how to encode QC results in BIDS.  Even if it's like

cheers,

satra

--

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.


For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--

You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CAJoTcz6ct19k8tqERRbaDo62gyjuG9bSifVu7Xp25wF16Z3ieg%40mail.gmail.com.

Stephen Strother

unread,
Sep 27, 2017, 10:25:54 AM9/27/17
to bids-di...@googlegroups.com
I would suggest this be dealt with using (pass, questionable, fail) + comments. This has worked quite well for a range of imaging projects we are working on and is supported within XNAT currently. Questionable often includes things like minor motion, statistical outliers that are not extreme, etc. It is then up to the user of the data to sort out what fail and questionable mean and how they deal with them.

For more options, visit https://groups.google.com/d/optout.


-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.

Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

Vince Calhoun

unread,
Sep 27, 2017, 10:42:08 AM9/27/17
to The Brain Imaging Data Structure (BIDS) discussion
This is good, but i still think we need to allow multiple qc calls as the approach, threshold, and tolerance of analysis methods varies a lot. 

--

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.


-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.

Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/84f8e7e0-0124-a580-426f-5e193ea86d68%40research.baycrest.org.

Satrajit Ghosh

unread,
Sep 27, 2017, 10:46:18 AM9/27/17
to bids-di...@googlegroups.com
tom: completely fine with tomnicholsQC - as long as tom's protocol for giving something a rating is well described somewhere :)

stephen: while xnat has used that scale, there is absolutely zero consistency across projects as to what pass/fail/questionable mean. if you can point me to a place that describes the protocol and some notion that this is used consistently across xnat projects, i would be very happy to retract my objection to that scale.

chris: in this particular edge case, let's bring up a few specific scenarios:

- investigator says i know this is left/right
- data was not converted from dicoms but from a format which has questionable left/right info (i won't name names)
- data converted from dicoms directly to nifti with version of conversion software noted
- data presumably converted from dicom to nifti then defaced with some software (so the chain has to preserve info properly)

i think it would be useful for any dataset to record how the dataset was derived, so that a user can make judgments as to the veracity of the orientation.

for retrospective datasets, where such information as sform/qform is missing, it seems that the particular dataset should be an invalid bids dataset in the first place. should it not simply be marked as an invalid bids dataset, since there is no way of knowing if the file is valid or not. i may still use something like this for projects where left/right do not matter, but i would think the most common use cases would entail knowing that information.

cheers,

satra

director, data models and integration | repronim: center for reproducible neuroimaging computation

--

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.
-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.

Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/84f8e7e0-0124-a580-426f-5e193ea86d68%40research.baycrest.org.

Auer, Tibor

unread,
Sep 27, 2017, 11:52:20 AM9/27/17
to bids-di...@googlegroups.com

I think any QC information is a good information as long as – as Satra described – it is well defined, transparent and replicable. So I am not against multiple QC descriptors. On the contrary.

However, I would rather like to see some ‘objective’ measure (say, tSNR with a description/specification of the protocol) rather than a ‘judgement’.

 

I also agree on that extreme cases should be perhaps identified (and invalidated) by the BIDS validator. It is already checking existence of metadata and sform/qform is a metadata.

 

Vale,

Tibor

 

Auer, Tibor (Ph '99)

 

From: bids-di...@googlegroups.com [mailto:bids-di...@googlegroups.com] On Behalf Of Satrajit Ghosh
Sent: 27 September 2017 15:46
To: bids-di...@googlegroups.com
Subject: Re: [bids-discussion] Adding Field/s to dataset_description.json describing known issues and Usability(usable/non-usable)

 

tom: completely fine with tomnicholsQC - as long as tom's protocol for giving something a rating is well described somewhere :)

--

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.


To view this discussion on the web visit

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.

-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.
 
Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.


For more options, visit https://groups.google.com/d/optout.

 

--

You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CA%2BA4wOkhs0Xg%2B2JTBr-q8xFy6c2iC5QyAJD2D4i8H4zg6VJd3w%40mail.gmail.com.

JB Poline

unread,
Sep 27, 2017, 12:11:06 PM9/27/17
to The Brain Imaging Data Structure (BIDS) discussion
Hi,

This discussion points to the need of a working group to gather and document QC standard procedures - then these can be reused and referred to. An INCF working group with people interested and knowledgeable on this would be great.

Cheers
JB

--

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.


To view this discussion on the web visit

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.
 
Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.


For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/00c001d337a8%24986c8ac0%24c945a040%24%40gmail.com.

Chris Gorgolewski

unread,
Sep 27, 2017, 12:30:22 PM9/27/17
to bids-discussion
On Wed, Sep 27, 2017 at 4:52 PM, Auer, Tibor <tibor...@gmail.com> wrote:

I also agree on that extreme cases should be perhaps identified (and invalidated) by the BIDS validator. It is already checking existence of metadata and sform/qform is a metadata.

This is already implemented. The validator raises an error for such data.

The question Suyash was trying to raise is how to annotate a dataset with incorrect sform/qform so it would not be misused. Putting this info in README might be enough.

@satra - this is a retrospective dataset, we don't know what lead to this mistake. 

Best,
Chris
 

--

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.


To view this discussion on the web visit

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.
 
Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.


For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/00c001d337a8%24986c8ac0%24c945a040%24%40gmail.com.

Oscar Esteban

unread,
Sep 27, 2017, 2:26:18 PM9/27/17
to bids-discussion
One important point I feel Suyash missed. We discussed two options that are not mutually exclusive:

1. As Suyash mentioned, having a field on the dataset_description.json or in the README as Chris suggested where issues can be enlisted. I'd rather vote for the dataset_description.json and give the following example: in OpenfMRI's DS030 we found that about 20% of images have a ghost artifact overlapping the temporal lobes, sometimes confusing FreeSurfer. An expert flagged this cases (this is less subjective, although still subjective, than the Pass/Fail example given above) so it would be great to have access from software to this list of subjects using the BIDS metadata (it could also be done at subject level, of course)

2. More importantly (and missing in Suyash's explanation), we argued the need for an extra file (like the README) to signal FATAL issues (like the alteration of the s/qform affines which compromises the L/R orientation). This would be a REVOKE file (or along these lines), which would contain a message about the reason for flagging the dataset. BIDS-Apps would yield at you the message in a big fat warning if you try to reuse that dataset. Basically, warn about the fact that trustworthiness of results is doubtful because something critical issue affects these data. The data can still be shared, but the idea is to make it really explicit that you are using compromised data.

Best,
Oscar


To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CAAQzouNzFhtOiVSMc9cFiV%3D895pPB1PmqsEzRSUJpSZYo1vWnA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
___________________________
Oscar Esteban, Ph.D.
Postdoctoral Fellow, Poldrack Lab
Stanford University

Stephen Strother

unread,
Sep 27, 2017, 4:37:10 PM9/27/17
to bids-di...@googlegroups.com
Satra - I agree that there is no consistently applied approach for any of the general categories of useable, questionable and failed. Effectively they capture some provenance information about the way the generator of the data chose to rate the items. This is the sense we have used them together with a description of our rating approach and associated software. Typically we recommend analysing the useable, and useable and questionable combined, and look for consensus or try to understand significant differences in the context of the questionable ratings.

Tibor - I also like quantitative measures but we are very far from even defining what they all might be in neuroimaging and even general neuroinformatics/biosttaistics (e.g., outlier detection approaches for eCRFs, genomics, etc) let alone agreeing on a small subset. My idea was to signal that there was such a set of measures generated, which in our case include visual inspection, fBIRN and QAPs pipelines + other measures, and that these were used to rate the quality. Better to know something was done and what it produced than simply ignore it completely because there is no clear standard available. I also think BIDS would be overwhelmed trying to capture all the current quantitative possibilities even if we could write them down and define them. But I don't think that is a reason for not capturing something with the details defined outside of BIDS.

JB - Agreed. Definitely a great role for an INCF working group. Let me know when we start.

Cheers, Stephen
-- You received this message because you are subscribed to the Google Groups "bids-discussion" group. To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com. To post to this group, send email to bids-di...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CALMaa835RmWnxZwUUrpK4oRDCUV%3DQg%3DtP7JqG916JuNi5YNFjA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.

Pierre Bellec

unread,
Sep 27, 2017, 8:38:09 PM9/27/17
to bids-di...@googlegroups.com, Yassine Ben Haj Ali
Dear all,

Satra, Stephen - I 100% agree with everything you said. Quantitative metrics are attractive. I know Cameron and others have worked hard towards generating such metrics and trying to turn them into actionable QC values.  But at this stage I would mostly feel uncomfortable excluding data on such basis; with the exception of motion/spiking metrics, and even those could be better validated. I believe there first need to be visual OK/maybe/failed QC procedures which are well documented and reproducible, before we automatize these QC procedures from QC metrics. Yassine and myself have worked on such a protocol for brain registration (which is for derivatives rather than raw data) and we were surprised by how hard it is to get an inter-rater kappa above 0.6. FYI protocol is described at the following link and we are looking for volunteers for validation https://www.zooniverse.org/projects/simexp/brain-match. 

JB - I think it would be fantastic to have a formal working group on this.

Chris and others - I very much like the idea of storing QC values, but my feeling is that those would be better treated as derivatives than stored in a raw BIDS dataset. They are dependent on a protocol/pipeline. 

Best,

Pierre

Pierre Bellec 
Assistant professor ("adjoint sous octroi"), Department of computer science and operational research (DIRO)
Member, Centre de recherche de l'institut Universitaire de gériatrie de Montréal (CRIUGM)
Université de Montréal, Montréal, Canada
Full coordinates on my home page

-- You received this message because you are subscribed to the Google Groups "bids-discussion" group. To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com. To post to this group, send email to bids-discussion@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CALMaa835RmWnxZwUUrpK4oRDCUV%3DQg%3DtP7JqG916JuNi5YNFjA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. 

-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.

Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to bids-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/32432096-eb8d-9fd5-ac2d-7eaba8d48a5c%40research.baycrest.org.

JB Poline

unread,
Sep 27, 2017, 11:08:59 PM9/27/17
to The Brain Imaging Data Structure (BIDS) discussion, Yassine Ben Haj Ali
Interesting and great tool : will try to rate !  And yes, I agree : a derivative folder sounds more appropriate, but would be good to have it standard.
cheers
JB

To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CADPHwxDY1o_Mf0%3DSCsQ4gPEXTZ5VhFpS-5t60uUA7De%2BM24Duw%40mail.gmail.com.

Auer, Tibor

unread,
Sep 28, 2017, 5:44:27 AM9/28/17
to bids-di...@googlegroups.com

Stephen – We are on the same page. I put “objective” in a quote, because I am also aware of they are not well defined. However, I assume they are comparable for a give QA pipeline. I am only saying that I would prefer a continuous value derived from a well-defined pipeline rather than a binary/trinary/olig-ary one derived from non-transparent decision.

 

Chris – I am not sure how much a ‘fatal’ dataset worth unless we assume that there might be some repair tools developed later, which might make these datasets usable again. My – no extremely strong – opinion is that we should rather remove those dataset from public databases, because the damage caused by ‘careless’ usage is – I think – greater than the (assumed) benefit.

 

Vale,

Tibor

 

Auer, Tibor (Ph '99)

 

From: bids-di...@googlegroups.com [mailto:bids-di...@googlegroups.com] On Behalf Of Stephen Strother
Sent: 27 September 2017 21:37
To: bids-di...@googlegroups.com
Subject: Re: [bids-discussion] Adding Field/s to dataset_description.json describing known issues and Usability(usable/non-usable)

 

Satra - I agree that there is no consistently applied approach for any of the general categories of useable, questionable and failed. Effectively they capture some provenance information about the way the generator of the data chose to rate the items. This is the sense we have used them together with a description of our rating approach and associated software. Typically we recommend analysing the useable, and useable and questionable combined, and look for consensus or try to understand significant differences in the context of the questionable ratings.

--

To unsubscribe from this group and stop receiving emails from it, send an email to


To post to this group, send email to bids-di...@googlegroups.com.



 

--

 

__________________________________________________________

Thomas Nichols, PhD

Professor of Neuroimaging Statistics

Oxford Big Data Institute

Li Ka Shing Centre for Health Information and Discovery

Nuffield Department of Population Health

University of Oxford

 

 

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to


To post to this group, send email to


To view this discussion on the web visit

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.

-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.
 
Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

-- You received this message because you are subscribed to the Google Groups "bids-discussion" group. To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com. To post to this group, send email to bids-di...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/84f8e7e0-0124-a580-426f-5e193ea86d68%40research.baycrest.org.

For more options, visit https://groups.google.com/d/optout.

 

-- You received this message because you are subscribed to the Google Groups "bids-discussion" group. To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com. To post to this group, send email to bids-di...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CA%2BA4wOkhs0Xg%2B2JTBr-q8xFy6c2iC5QyAJD2D4i8H4zg6VJd3w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.

-- You received this message because you are subscribed to the Google Groups "bids-discussion" group. To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com. To post to this group, send email to bids-di...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/00c001d337a8%24986c8ac0%24c945a040%24%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

-- You received this message because you are subscribed to the Google Groups "bids-discussion" group. To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com. To post to this group, send email to bids-di...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CALMaa835RmWnxZwUUrpK4oRDCUV%3DQg%3DtP7JqG916JuNi5YNFjA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. 

-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.
 
Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

--
You received this message because you are subscribed to the Google Groups "bids-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com.
To post to this group, send email to bids-di...@googlegroups.com.

Stephen Strother

unread,
Sep 28, 2017, 4:05:38 PM9/28/17
to bids-di...@googlegroups.com
Pierre - We would be interested in helping with this, but I cannot access your URL. I get:

"Project simexp/brain-match not found.

If you're sure the URL is correct, you might not have permission to view this project."

Cheers, Stephen

-- You received this message because you are subscribed to the Google Groups "bids-discussion" group. To unsubscribe from this group and stop receiving emails from it, send an email to bids-discussi...@googlegroups.com. To post to this group, send email to bids-di...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/bids-discussion/CALMaa83CBB8LUoMfAER310Fc9tqw%2BOrcGu8t5Km-S9BZp6yN8Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.

Oscar Esteban Sanz Dranguet

unread,
Oct 24, 2017, 6:16:52 PM10/24/17
to bids-discussion
Thanks a lot for all the comments. I'd like to reconvene the conversation to the initial points proposed:

- Known issues: define a way to easily read important issues related to the dataset from a BIDS-App or other interfaces (like OpenfMRI to show/link these easily). This cannot not be a field to describe QC (for all the reasons you all have noted above). Let me remind here that anyone interested in QC can contact JB about the working group he mentioned.

- Revoke file (or similar) to flag that a dataset that should not be used with "analysis" purposes. @Tibor: these flagged datasets are still very useful, so I would not stop sharing them. E.g. if you are training a skull-stripping algorithm that is insensitive to L/R flips, then you can perfectly use those data for training. Another example I though after this: make a new version of an existing dataset, where all images have been spatially decimated by a x4 factor for testing purposes. The original data has not been modified (only removed a big chunk of it). This dataset would be BIDS-compliant (because you want something compliant for testing your app) but needs a REVOKE file (or whatever name/metadata object) to clearly state "these data must not be used for research, only for testing purposes".
Reply all
Reply to author
Forward
0 new messages