Draft of the QC Survey

52 views
Skip to first unread message

Pradeep Reddy Raamana

unread,
Sep 13, 2018, 1:55:52 PM9/13/18
to NIQC
Hi Everyone,

Here is a first draft for the survey we discussed: https://goo.gl/forms/oBSrhbDbBtkmBVSU2

Let me know if I missed any relevant questions, or if I need to remove or reword some ("answers depend on the way we ask questions").

Questions for you:
  1. is it too long and onerous? We want widest possible consultation, I am okay with losing few details in the first pass to reach more researchers.
  2. In the invite, I'd like to give an estimate of 15 mins for survey, so they can plan and start it appropriately - does that sound right?
  3. Other than Twitter, institutional mailing lists of this groups members, mailing lists of OHBM, INCF and ISMRM, how else can we spread this survey? We do not yet have money to pay for some of this mailing lists, I'd like to reach out to these societies if they would do it in the name of science!
  4. Is there a way to make some "science" out of this survey?

Looking fwd to hear from all of you.

Thanks,
Pradeep

Pradeep Reddy Raamana

unread,
Sep 13, 2018, 2:04:15 PM9/13/18
to NIQC
Btw I created a Zotero group recently to collect all the relevant literature: https://www.zotero.org/groups/2221093/niqc

Please feel free to share what you must be considered in our discussion.

Remi Gau

unread,
Sep 13, 2018, 2:55:56 PM9/13/18
to NIQC
From the top of my head: we can add the mailing list of all the usual
suspects in term of neuroimaging softwares, the neurostars forum...

Pradeep Reddy Raamana

unread,
Sep 17, 2018, 4:20:52 PM9/17/18
to NIQC
I have also exported the survey to PDF for easy review - please take a look and let me know your comments. I'm okay with very different alternatives as well if they would provide us with more engagement.

Questions for you:
  1. is this too burdensome? We want widest possible consultation, I am okay with losing few details in the first pass to reach more researchers. We can do away with sections 2 and 3 if needed to make it shorter.
  1. Is there a way to make some "science" out of this survey?
  1. Other than Twitter, institutional mailing lists of this groups members, mailing lists of OHBM, INCF and ISMRM, how else can we spread this survey? We do not yet have money to pay for some of this mailing lists, I'd like to reach out to these societies if they would do it in the name of science.
Thanks Remi and Fidel for the input.

Pradeep
niQC community scan survey - Google Forms.pdf

Katie Bottenhorn

unread,
Sep 17, 2018, 5:02:41 PM9/17/18
to NIQC, Pradeep Reddy Raamana
Hi, all–

A few thoughts about the survey–
  1. It is a little long, especially for a first pass. In the interest of encouraging community involvement, it would be helpful to incorporate multiple-choice questions where possible 
    • E.g. starting out with “Modality?” “MRI”, “PET”, “NIRS”, “other:____”
    • "How long does the whole process take, approximately PER SUBJECT (or another unit appropriate for this task)? “ "<5 minutes”, “5-10 minutes”, etc. 
    • Currently, the bulk of the survey is free response, which makes completing the survey more time-consuming and assessing results (see 2) more challenging. Swapping out free response questions for multiple choice (always with an “other” field, so we don’t exclude any possibilities) would make things faster & easier on both fronts. I’m happy to help on this front, as I know a lot of work has been put into generating this survey.
  2. We can certainly use these results for “science”, especially if we incorporate more multiple-choice questions (where possible). Summarizing the proportions of respondents who use certain tools, the average length of different QC procedures, etc. could be interesting.
  3. We should share the survey in the Brainhack Slack group?

Katie
--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.
To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/4465f169-9522-4fb3-a429-a055b5f99235%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Simon Eickhoff

unread,
Sep 17, 2018, 5:09:17 PM9/17/18
to Katie Bottenhorn, ni...@googlegroups.com, Pradeep Ramana
Dear all

I absolutely concur with Katie. From my own experience, dealing with open questions is mess with the most likely outcome of there being no real outcome. This is even more true as some people answer with only a few words, others write long stories.

Hence, I would second and reinforce the suggestion to convert most, ideally all, questions to multiple conic or numeric entries. This would also reduce the time for completion and it makes a difference if you ask people for 5 or 15 minutes. Finally, it should be fairly easy to turn a quantitative survey into a paper.

Cheers
Simon


To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/f83f6995-9c75-4862-b557-b8437f36510f%40Spark.

For more options, visit https://groups.google.com/d/optout.


--
===================================
Univ.-Prof. Dr. med. Simon B. Eickhoff

Director, Institute of Systems Neuroscience
Heinrich Heine University Düsseldorf
40225 Düsseldorf, Germany 

and

Director, Institute of Neuroscience and Medicine (INM-7: Brain and Behaviour )  
Research Centre Jülich
52425 Jülich, Germany

Stephen Strother

unread,
Sep 17, 2018, 6:51:19 PM9/17/18
to Simon Eickhoff, Katie Bottenhorn, ni...@googlegroups.com, Pradeep Ramana
I also strongly agree. You will want limited multi-choice answers whenever possible to succeed.

Cheers, Stephen

Pradeep - this was the result of an earlier focus on resting state

For more options, visit https://groups.google.com/d/optout.


-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.

Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

Lei Wang

unread,
Sep 17, 2018, 7:26:06 PM9/17/18
to Simon Eickhoff, Katie Bottenhorn, ni...@googlegroups.com, Pradeep Ramana
I also agree with using choices rather than only free texts. However, many of the questions are open in nature in that the candidate choices are either too many or not yet clear, especially ones on actual QA approaches. I wonder if we should first agree whether the questions need re-posing (maybe that’s already done?) and then develop the choices. 

--Lei

--
Lei Wang, Ph.D.
-----------------------------------------------------------------
710 N. Lake Shore Drive, Abbott Hall 1322
Tel. office:            312-503-3983
Fax:              312-503-0527
Email:                 leiw...@northwestern.edu
URL:             http://niacal.northwestern.edu
-----------------------------------------------------------------

Sebastian Urchs

unread,
Sep 17, 2018, 7:45:28 PM9/17/18
to Stephen Strother, Simon Eickhoff, Katie Bottenhorn, ni...@googlegroups.com, Pradeep Ramana
I agree that we'll probably get more and better responses from a few multiple choice questions. In the spirit of our first meeting at the INCF hackathon I'd suggest we move the survey draft into a google doc that we can edit together. That will also help to make sure the multiple-choice questions cover our probably non-overlapping fields of expertise.

Based on the discussion so far I just hacked something together here and would invite you to edit and discuss in this document:

This is not a draft, but a suggestion to start from. I was thinking we bundle the questions in a few clear sections that represent the core things we want to find out and then give relevant answer options.

Best,
Seb

Pradeep Reddy Raamana

unread,
Sep 17, 2018, 7:46:49 PM9/17/18
to Lei Wang, Simon Eickhoff, Katie Bottenhorn, ni...@googlegroups.com
I agree with you all, that we should try converting to multiple choice questions when possible. Multiple choice questions are ideal when the question is targeted, with answers with a KNOWN range of choices (such as the modality a QC task related to). However, the PRIMARY purpose of this survey (if everyone agrees) is to seek info on "what is happening out there". As Lei put it well "many of the questions are open in nature in that the candidate choices are either too many or not yet clear".  Esp. about the process for a QC task/context we are not yet familiar with. So I'd prefer to take the unassuming approach for now. 

We can perhaps distill it better, and reduce the number of questions? I included the latter 2 sections to capture the often-ignored-yet-important aspects of this QC process, and I think we have an unique opportunity to capture them now. But in favour of reducing the time to less than 15 or 10 mins, I don't mind dropping them altogether.

Stephen, can you share the link to resting-state survey, and its results, if they are available?




On Mon, Sep 17, 2018 at 7:26 PM, Lei Wang <leiw...@northwestern.edu> wrote:
I also agree with using choices rather than only free texts. However, many of the questions are open in nature in that the candidate choices are either too many or not yet clear, especially ones on actual QA approaches. I wonder if we should first agree whether the questions need re-posing (maybe that’s already done?) and then develop the choices. 

--Lei

--
Lei Wang, Ph.D.
-----------------------------------------------------------------
710 N. Lake Shore Drive, Abbott Hall 1322
Tel. office:            312-503-3983
Fax:              312-503-0527
Email:                 leiwang1...@northwestern.edu
URL:             http://niacal.northwestern.edu
-----------------------------------------------------------------
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+unsubscribe@googlegroups.com.

To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/4465f169-9522-4fb3-a429-a055b5f99235%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+unsubscribe@googlegroups.com.

To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/f83f6995-9c75-4862-b557-b8437f36510f%40Spark.
For more options, visit https://groups.google.com/d/optout.


--
===================================
Univ.-Prof. Dr. med. Simon B. Eickhoff
Director, Institute of Systems Neuroscience
Heinrich Heine University Düsseldorf
40225 Düsseldorf, Germany 

and

Director, Institute of Neuroscience and Medicine (INM-7: Brain and Behaviour )  
Research Centre Jülich
52425 Jülich, Germany

--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+unsubscribe@googlegroups.com.

To post to this group, send email to ni...@googlegroups.com.



--
Postdoctoral fellow
Neuroinformatics,
3560 Bathurst Street, Toronto, ON
Canada, M6A 2E1.
Phone: +1 416-785-2500 x6408. 

Remi Gau

unread,
Sep 18, 2018, 6:18:34 AM9/18/18
to Pradeep Reddy Raamana, Lei Wang, Simon Eickhoff, Katie Bottenhorn, ni...@googlegroups.com

I agree with the general sentiment to have multiple choices. I think that having an almost systematic open-ended option like "other - please specify" might be useful to see if the choices we provide do not consistently miss something that we did no think of. In my spare time I am trying to convert the COBIDAS report in a checklist and that's option I am going for.

I know next to nothing about surveys but I guess that if we wanted to be thorough we would have to have 2 rounds of survey and the first one would only be there to validate our question set but I don't think that's what we want here.

On a different note I added to the "reading list" the preprocessing chapter of my fMRI data analysis "bible" (by Poldrack, Mumford and Nichols) because it has quite a few notes in QC (see fig below for example).

@+

Remi


To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.

To post to this group, send email to ni...@googlegroups.com.

Katja Heuer

unread,
Sep 18, 2018, 5:38:43 PM9/18/18
to Remi Gau, Pradeep Reddy Raamana, Lei Wang, Simon Eickhoff, Katie Bottenhorn, ni...@googlegroups.com
Hello,

this is such a great initiative! Thanks to all! 

I added thoughts and suggestions to the google doc – looking forward to it to grow :) 

I agree, it sounds better to me to put as much as possible into multiple choice format – always with the "other" option to add in free text form what everyone feels like adding. This is also a great way to make people aware of more / or alternative options to what they currently use...

You say that your principle current goal is to develop a standard pipeline for QC-ing functional MRI data – if you are happy, Roberto and I – and I guess Pradeep as well – could focus on the anatomy, segmentation and surface part, where our experience lies and where we have developed several tools for QC and data editing that all are setup for the most collaborative way of working (browser and peer-to-peer based).  

I think it is a great idea to maybe exchange our current practices on QCing data to refine the questions – thanks @Remi for adding this chapter and pasting your scheme. We don't have a nice scheme at hand, but if you think it would be helpful, we can make one (for anat/seg/surf). Let me know.


Have a lovely day.

sunny regards from Paris 
katja



--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+unsubscribe@googlegroups.com.
To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/cb5af33b-d8d2-fe95-4c43-64146579040a%40gmail.com.

Remi Gau

unread,
Sep 18, 2018, 5:50:08 PM9/18/18
to Katja Heuer, Pradeep Reddy Raamana, Lei Wang, Simon Eickhoff, Katie Bottenhorn, ni...@googlegroups.com

Eduard Klapwijk

unread,
Sep 21, 2018, 7:04:49 AM9/21/18
to remi...@gmail.com, Pradeep Reddy Raamana, leiw...@northwestern.edu, simon.b....@gmail.com, kbot...@fiu.edu, ni...@googlegroups.com
Yes totally agree with using multiple choice whenever possible with an added open-ended option would be the most user-friendly option. The survey draft looks great already!

Best,
Eduard Klapwijk
Leiden University, The Netherlands

PS: sorry for cross-posting, sent it from another mailadress so it did not make it to the Google list

On Tue, Sep 18, 2018 at 12:18 PM Remi Gau <remi...@gmail.com> wrote:

I agree with the general sentiment to have multiple choices. I think that having an almost systematic open-ended option like "other - please specify" might be useful to see if the choices we provide do not consistently miss something that we did no think of. In my spare time I am trying to convert the COBIDAS report in a checklist and that's option I am going for.

I know next to nothing about surveys but I guess that if we wanted to be thorough we would have to have 2 rounds of survey and the first one would only be there to validate our question set but I don't think that's what we want here.

On a different note I added to the "reading list" the preprocessing chapter of my fMRI data analysis "bible" (by Poldrack, Mumford and Nichols) because it has quite a few notes in QC (see fig below for example).

@+

Remi




On 18/09/2018 01:46, Pradeep Reddy Raamana wrote:
I agree with you all, that we should try converting to multiple choice questions when possible. Multiple choice questions are ideal when the question is targeted, with answers with a KNOWN range of choices (such as the modality a QC task related to). However, the PRIMARY purpose of this survey (if everyone agrees) is to seek info on "what is happening out there". As Lei put it well "many of the questions are open in nature in that the candidate choices are either too many or not yet clear".  Esp. about the process for a QC task/context we are not yet familiar with. So I'd prefer to take the unassuming approach for now. 

We can perhaps distill it better, and reduce the number of questions? I included the latter 2 sections to capture the often-ignored-yet-important aspects of this QC process, and I think we have an unique opportunity to capture them now. But in favour of reducing the time to less than 15 or 10 mins, I don't mind dropping them altogether.

Stephen, can you share the link to resting-state survey, and its results, if they are available?



On Mon, Sep 17, 2018 at 7:26 PM, Lei Wang <leiw...@northwestern.edu> wrote:
I also agree with using choices rather than only free texts. However, many of the questions are open in nature in that the candidate choices are either too many or not yet clear, especially ones on actual QA approaches. I wonder if we should first agree whether the questions need re-posing (maybe that’s already done?) and then develop the choices. 

--Lei

--
Lei Wang, Ph.D.
-----------------------------------------------------------------
710 N. Lake Shore Drive, Abbott Hall 1322
Tel. office:            312-503-3983
Fax:              312-503-0527
Email:                 leiw...@northwestern.edu
URL:             http://niacal.northwestern.edu
-----------------------------------------------------------------
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.
To post to this group, send email to ni...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.
To post to this group, send email to ni...@googlegroups.com.


--
===================================
Univ.-Prof. Dr. med. Simon B. Eickhoff
Director, Institute of Systems Neuroscience
Heinrich Heine University Düsseldorf
40225 Düsseldorf, Germany 

and

Director, Institute of Neuroscience and Medicine (INM-7: Brain and Behaviour )  
Research Centre Jülich
52425 Jülich, Germany
--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.
To post to this group, send email to ni...@googlegroups.com.



--
Postdoctoral fellow
Neuroinformatics,
3560 Bathurst Street, Toronto, ON
Canada, M6A 2E1.
Phone: +1 416-785-2500 x6408. 
--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.
To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/CAKYYBvoNPNwmSeZu%2BgFbnZa7PhmVfZctRuTsNJdm03gcZoxBug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.
To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/cb5af33b-d8d2-fe95-4c43-64146579040a%40gmail.com.

Pradeep Reddy Raamana

unread,
Oct 25, 2018, 3:22:17 PM10/25/18
to NIQC
Hi All,

Apologies for the delay in response, I was occupied with multiple other events and deadlines. 

Thank you all for providing input on the high-level survey as well as the more specific fMRI version - that was awesome. 

As it stands today, my thoughts are:
  • There is a clear need for the high-level survey, without specifics, simply to learn what is happening in the community. I believe this must happen before we put forth more specific drafts up for general consultation within the community. As most of you noted, it can't be too onerous and we just need to accept what length/depth would be useful without being onerous.
  • We can certainly start drafting more specific surveys (for different type of QC tasks now) - a slight danger in this exercise being these specific protocols may not sufficiently represent what happens out there. Given the larger goal of achieving maximum adoption for each of the protocols we would develop, it might be better to try catering to as wide a section of the community as possible (when we get started on the drafts). 
  • If and when we start working on the drafts, I suggest we let the sub-groups be open to all members of this group, so everyone interested will be welcome to contribute as much or as little as they like (in various aspects). And all the contributions will be recognized accordingly.
Let me know your thoughts.

To make the generic survey shorter and easier, I will give it another try now. Let me know if you want to help, happy to add you as a collaborator. AFAIK Google forms unfortunately does not allow inviting a group to collaborate or comment, and link sharing provides full editing access (no intermediate commenting). So sharing link here is not a great idea as this content is public and

Thanks,
Pradeep
Email:                 leiwang1...@northwestern.edu
URL:             http://niacal.northwestern.edu
-----------------------------------------------------------------
Reply all
Reply to author
Forward
0 new messages