Finalizing the High-level Open Survey

31 views
Skip to first unread message

Pradeep Reddy Raamana

unread,
Oct 27, 2018, 10:44:24 AM10/27/18
to NIQC

For the new group members and others who may not have looked at it yet, here is the link to the high-level survey :

I've now 1) improved the questions and added more info/examples, and 2) combined them all into a single page, so respondents can now skip to submission after answering mandatory questions.

To finalize this - make it easy and quick to respond to - I'd like to request you all to try respond to it using one of your regular QC tasks in the next 2 weeks or so. I request you to time them, so we get a sense of effort it takes, so we can consider which questions can be reworded or combined or removed altogether.

I am in favour of receiving fewer responses to some questions, than not asking them at all in the first place. I'd like to believe those who care about QC will respond, and I'm not a fan of worrying too much about others who are not willing to spend about 15-30 mins to respond to this survey.

Have a quality weekend first ;)

PS: Let me know if you want to edit the form, I can add you as a collaborator.

Thanks,
Pradeep

Pradeep Reddy Raamana

unread,
Oct 31, 2018, 11:14:51 AM10/31/18
to ni...@googlegroups.com
Thank you for those who responded this survey already - once you are done, let us know your experience in answering it, how long it took and how to improve it etc.. 

--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.
To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/45d5c076-b8fb-403b-97d4-53fac5478c9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Postdoctoral fellow
Neuroinformatics,
3560 Bathurst Street, Toronto, ON
Canada, M6A 2E1.
Phone: +1 416-785-2500 x6408

Pradeep Reddy Raamana

unread,
Nov 1, 2018, 11:13:57 PM11/1/18
to NIQC
Awesome to see more responses coming in - thank you! :).. At the risk of repeating myself,  let me mention that this is anonymous survey without built-in time-tracking, so I need your help to understand how long it took for you, and what you think would improve the survey to maximize its response rate (when distributed broadly to a much larger community). Thanks!


On Wednesday, October 31, 2018 at 11:14:51 AM UTC-4, Pradeep Reddy Raamana wrote:
Thank you for those who responded this survey already - once you are done, let us know your experience in answering it, how long it took and how to improve it etc.. 

On Sat, Oct 27, 2018 at 10:44 AM Pradeep Reddy Raamana <praa...@research.baycrest.org> wrote:

For the new group members and others who may not have looked at it yet, here is the link to the high-level survey :

I've now 1) improved the questions and added more info/examples, and 2) combined them all into a single page, so respondents can now skip to submission after answering mandatory questions.

To finalize this - make it easy and quick to respond to - I'd like to request you all to try respond to it using one of your regular QC tasks in the next 2 weeks or so. I request you to time them, so we get a sense of effort it takes, so we can consider which questions can be reworded or combined or removed altogether.

I am in favour of receiving fewer responses to some questions, than not asking them at all in the first place. I'd like to believe those who care about QC will respond, and I'm not a fan of worrying too much about others who are not willing to spend about 15-30 mins to respond to this survey.

Have a quality weekend first ;)

PS: Let me know if you want to edit the form, I can add you as a collaborator.

Thanks,
Pradeep

--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+unsubscribe@googlegroups.com.

To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/45d5c076-b8fb-403b-97d4-53fac5478c9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pierre Bellec

unread,
Nov 2, 2018, 12:22:04 AM11/2/18
to Pradeep Kumar Reddy Raamana, ni...@googlegroups.com
Hi Pradeep,

Just did it. Took me 9 mns.
I went very fast, so this is probably a low estimate.
Couple comments:

This sentence does not read well "How do you report how you performed this QC process and results?"

Some questions are really very vague and sometimes confusing, e.g.
As a researcher or reviewer, what errors or mistakes do you notice in the literature and community?
Would you be interested in crowdsourcing QC for public or your own datasets?

The survey is very long, I suggest cutting it down.
I would avoid the questions above, for example, as I really don't know what you can do with the answers. 
In its current form, I believe few people will take the time to go through the survey.

Also, as a general rule, I avoid to write words in all caps, even if I want people to pay attention.
I do shout in real life though, especially at my kids (sometimes), so I guess nobody's perfect.

Good luck with the survey,

Pierre

Pierre Bellec
 
Research associate professor | Professeur agrégé sous octroi, 
Psychology Department | Département de Psychologie 
Université de Montréal, Montréal, Canada
Director | Directeur, Unité de Neuroimagerie Fonctionnelle (UNF)
Researcher | Chercheur, Centre de recherche de l'institut Universitaire de gériatrie de Montréal (CRIUGM)
Phone | Téléphone +1 514 713 5596  
------

Please note that I am resetting my mail box on Sunday evening every week.
I do my best to answer all emails, but I am sometimes not able to.
If you have not heard back from me after 1 week, consider sending me a reminder.



On Thu, Nov 1, 2018 at 11:14 PM Pradeep Reddy Raamana <praa...@research.baycrest.org> wrote:
Awesome to see more responses coming in - thank you! :).. At the risk of repeating myself,  let me mention that this is anonymous survey without built-in time-tracking, so I need your help to understand how long it took for you, and what you think would improve the survey to maximize its response rate (when distributed broadly to a much larger community). Thanks!

On Wednesday, October 31, 2018 at 11:14:51 AM UTC-4, Pradeep Reddy Raamana wrote:
Thank you for those who responded this survey already - once you are done, let us know your experience in answering it, how long it took and how to improve it etc.. 

On Sat, Oct 27, 2018 at 10:44 AM Pradeep Reddy Raamana <praa...@research.baycrest.org> wrote:

For the new group members and others who may not have looked at it yet, here is the link to the high-level survey :

I've now 1) improved the questions and added more info/examples, and 2) combined them all into a single page, so respondents can now skip to submission after answering mandatory questions.

To finalize this - make it easy and quick to respond to - I'd like to request you all to try respond to it using one of your regular QC tasks in the next 2 weeks or so. I request you to time them, so we get a sense of effort it takes, so we can consider which questions can be reworded or combined or removed altogether.

I am in favour of receiving fewer responses to some questions, than not asking them at all in the first place. I'd like to believe those who care about QC will respond, and I'm not a fan of worrying too much about others who are not willing to spend about 15-30 mins to respond to this survey.

Have a quality weekend first ;)

PS: Let me know if you want to edit the form, I can add you as a collaborator.

Thanks,
Pradeep

--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.

To post to this group, send email to ni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/niqc/45d5c076-b8fb-403b-97d4-53fac5478c9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Postdoctoral fellow
Neuroinformatics,
3560 Bathurst Street, Toronto, ON
Canada, M6A 2E1.
Phone: +1 416-785-2500 x6408

--
You received this message because you are subscribed to the Google Groups "NIQC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to niqc+uns...@googlegroups.com.

To post to this group, send email to ni...@googlegroups.com.

Pradeep Reddy Raamana

unread,
Nov 2, 2018, 11:04:39 AM11/2/18
to Pierre Bellec, ni...@googlegroups.com
thanks Pierre - 9 mins doesn't sound too bad, although we should try not taking up any longer than we need to. As I am learning from others (16 mins ), this response time is primarily determined by their use case, how extensive their QC process is, how much detail they want to input it in, whether they have docs written already and how good are they keyboard etc.

I am thinking a good way to address this is to minimize the number of mandatory questions - which is currently at 7, so users are free to skip and save time if they choose to. The last 3 mandatory Qs are geared towards FAIR principles, and I personally consider them to be as important as the scientific protocol. But we can choose to focus on the science part in the first open survey atleast. What do you all think?

Thanks for comments on the questions - I will make them as direct, clear and specific as possible. 

yes, will try minimize use of capital letter.. Google forms does not let me emphasize text in any way (bold or underline or italicize), so using my CS/ASCII 101 training :)



For more options, visit https://groups.google.com/d/optout.

Pradeep Reddy Raamana

unread,
Nov 22, 2018, 9:53:51 AM11/22/18
to Pierre Bellec, ni...@googlegroups.com
Hi Everyone,

Thank you all for who have already provided their input in various forms, very helpful. I'd like finalize this by the end of next week, and start distributing it, so I'd appreciate if you could say "yes, this is in good form for distribution", or suggest modifications. As this will be broadcast rather widely, input from each one of you matters, both students and senior PIs (who may have previously involved in other standardization efforts etc).

here is the link to the high-level survey :

Thanks,
Pradeep

Thomas Nichols

unread,
Nov 24, 2018, 6:49:40 AM11/24/18
to Pradeep Raamana, Pierre Bellec, ni...@googlegroups.com
Hi Pradeep,

Here are some edits/comments on the survey... if on Mac be sure to use Acrobat reader and not Preview.

-Tom


For more options, visit https://groups.google.com/d/optout.


--
__________________________________________________________
Thomas Nichols, PhD
Professor of Neuroimaging Statistics
Nuffield Department of Population Health | University of Oxford
Big Data Institute | Li Ka Shing Centre for Health Information and Discovery
Old Road Campus | Headington | Oxford | OX3 7LF | United Kingdom
T: +44 1865 743590 | E: thomas....@bdi.ox.ac.uk
W: http://nisox.org | http://www.bdi.ox.ac.uk
Neuroimaging Quality Control Survey-TNcomments.pdf

Pradeep Reddy Raamana

unread,
Nov 27, 2018, 11:23:11 AM11/27/18
to Thomas Nichols, Pierre Bellec, ni...@googlegroups.com
Thanks a lot Tom, Katja, Eduard and others for your input. The survey is much improved now: https://goo.gl/forms/1TiNZGZJZrn04hTy2

As for:
"Do you really think anyone no doing QC will fill this out?  Or, that if people are not doing QC they'll have useful information?  Seems doubtful."

I see your point, but I think it doesn't hurt to hear them. The reason for asking that question is to learn what we don't know and reach beyond our "bubble", and learn why they haven't done it yet.

I've rephrased the note at the top to:
"If you do not engage in QC yet, it would be great if you could describe why you haven't engaged in yet e.g. it could due to some challenges involved, lack of clarity in how to perform it or need training or tools to do it etc. If you feel you might need to perform QC in your particular task or analysis, and not sure how to do it, just describe your observations and the context."

I've also attached the 10 responses I have received so far from this group, to give everyone an idea of what we can expect. Take a look, perhaps this can help us further sharpen and improve this survey.

Everyone, I'd like to finalize this survey by the end of this week. Please share your input, which would really help getting most useful or high-quality feedback from our community.

Thanks,
Pradeep

niQC community scan survey 27Nov2018.xlsx

JB Poline

unread,
Nov 27, 2018, 11:48:11 AM11/27/18
to Pradeep Kumar Reddy Raamana, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
Hi Pradeep

Sounds good - Quick question: do you need ethical approval for collecting these data ?

typos : "do you for" <- "do you use for..."
also: the simple question "what data are you performing your QC on" is a little burried in a couple of questions you have, may be you could refactor to make it obvious, although you do capture some of it already.

Cheers
JB



Pradeep Reddy Raamana

unread,
Nov 27, 2018, 12:25:30 PM11/27/18
to jbpo...@gmail.com, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
Ethics is a good question - I personally [with very limited experience] don’t see a need for it, but the elders can weigh in on this!

Thanks for the suggestions, will correct/improve them.

Martin Lindquist

unread,
Nov 27, 2018, 12:29:27 PM11/27/18
to praa...@research.baycrest.org, JB Poline, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
In the US any time you poll people you need IRB approval. I would recommend getting one before proceeding.

On Tue, Nov 27, 2018 at 12:26 PM Pradeep Reddy Raamana <praa...@research.baycrest.org> wrote:
Ethics is a good question - I personally [with very limited experience] don’t see a need for it, but the elders can weigh in on this!

Thanks for the suggestions, will correct/improve them.
On Tue, Nov 27, 2018 at 11:48 AM JB Poline <jbpo...@gmail.com> wrote:

For more options, visit https://groups.google.com/d/optout.


--
Martin Lindquist
Professor
Department of Biostatistics
Johns Hopkins University
http://www.biostat.jhsph.edu/~mlindqui/

Pradeep Reddy Raamana

unread,
Nov 27, 2018, 12:46:43 PM11/27/18
to Martin Lindquist, JB Poline, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
I see - I’lol look into this.

Also, this survey would be distributed globally (atleast 50 countries).. Perhaps there is an international organization for this?

Is the concern we might be making money off this survey, or use it harm humans or animals?

Anisha Keshavan

unread,
Nov 27, 2018, 1:21:39 PM11/27/18
to Pradeep Reddy Raamana, mlin...@jhsph.edu, JB Poline, Pierre Bellec, ni...@googlegroups.com, ten.p...@gmail.com
I think the concern is that if we are asking private questions on the survey, the IRB would want to make sure we have a plan to store the data securely, so that info doesn't hurt people if it gets released. And if we were asking private questions with the intent to release data, the IRB would want to make sure the subjects are properly informed of this. 

At UW this type of survey would be IRB exempt (e.g. doesn't go through the full review process) because there is no personal information we are collecting (other than email addresses, which we wouldn't release anyway). It took ~3 days for the braindr IRB to go through at UW, so its really not too bad.

-Anisha

Pradeep Reddy Raamana

unread,
Nov 27, 2018, 1:56:13 PM11/27/18
to Anisha Keshavan, Martin Lindquist, JB Poline, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
That's a very good point - I didn't think about the email addresses before, now that I do, I think we don't need to ask. We will have a website soon to send infrequent public announcements, and those interested can subscribe to an RSS or similar on the website (or even a separate google group niqc-announcements).

So, would adding a statement at the top to the effect of "we're collecting your input with the plan to release it, analyze trends and publishing summaries of it etc." be sufficient?

I will also check with my institute Ethics department and receive their approval.

Thanks,
Pradeep


For more options, visit https://groups.google.com/d/optout.

Anisha Keshavan

unread,
Nov 27, 2018, 2:15:19 PM11/27/18
to Pradeep Reddy Raamana, mlin...@jhsph.edu, JB Poline, Pierre Bellec, ni...@googlegroups.com, ten.p...@gmail.com
I will also check with my institute Ethics department and receive their approval.

yes, do this, and if they do require IRB for this type of survey, ask people in your department if they have a protocol + consent form that you can use as an example -- that way its easy to modify and submit. 

Also, another thing to consider is google analytics. For my IRB I had to clarify that we won't be collecting IP addresses or anything personal. Here's an article on google analytics and GDPR (for international web-stuff, gotta consider GDPR), the only relevant thing for us I think is that we turn on IP anonymization if we do use analytics. But its probably easiest not to use it :)

-anisha


Pradeep Reddy Raamana

unread,
Nov 27, 2018, 2:29:37 PM11/27/18
to Anisha Keshavan, Martin Lindquist, JB Poline, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
How do you think analytics will help us? Or how did you use them in your work?

Google forms does not track or offer any analytics by default. There are tricks to do this (host the survey on a website with embedded tracking) etc, I personally don't see any need for this (other than learning how many countries did we reach, how long different folks took to complete it etc, which are not really a big deal to me personally).

Thanks,
Pradeep

Pradeep Reddy Raamana

unread,
Nov 27, 2018, 2:41:28 PM11/27/18
to Anisha Keshavan, Mathew Abrams, Helena Ledmyr, Martin Lindquist, JB Poline, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
I am getting an approval from my institute, but as this survey would be distributed globally (many countries besides Canada), and we will be distributing it in the name of INCF SIG, perhaps we might an international organization to approve it? Not sure if INCF itself has some mechanism for this.

Cc-ing Matthew and Helena of INCF for input.

Anisha Keshavan

unread,
Nov 27, 2018, 2:41:32 PM11/27/18
to Pradeep Reddy Raamana, mlin...@jhsph.edu, JB Poline, Pierre Bellec, ni...@googlegroups.com, ten.p...@gmail.com
In braindr, i put it because i was curious if people were using phones or desktops mostly, and also wanted to see the countries.

I agree that we don't need it, but wanted to mention it in case anyone has strong opinions! 

Stephen Strother

unread,
Nov 27, 2018, 5:34:12 PM11/27/18
to Anisha Keshavan, Pradeep Reddy Raamana, mlin...@jhsph.edu, JB Poline, Pierre Bellec, ni...@googlegroups.com, ten.p...@gmail.com
Pradeep - I believe the ethics issues must be dealt with by INCF in Sweden since they are collecting and hosting the data, unless you plan to move it to Baycrest, and then besides ensuring GDPR requirements are satisfied I believe it will only be an issue for Rotman if there is personal health information, and it sounds as though there is none. Before bothering with anything formal check with Roshan Guna on the 8th floor. He handles all things ethics to do with the Rotman, and of course as you noted check with INCF.

To my knowledge there is no international organisation dealing with this although there are ongoing discussions in which INCF has been involved about if, how and when data may be transferred between Europe and the US. It may be a bit easier with Euroep to Canada but that seems to just be a guess at present as I know of no actual test case. The case study being managed by InTBIR (https://intbir.nih.gov/) with INCF involvement is looking a sharing data between CENTER TBI in Europe and TRACK TBI in the US, but no one I have talked to is clear on exactly what all the issues will be.

Cheers, Stephen

For more options, visit https://groups.google.com/d/optout.


-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.

Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org
Tel. Office: 416-785-2500 x2956
Fax: 426-785-2862

Pierre Bellec

unread,
Nov 27, 2018, 6:04:04 PM11/27/18
to Stephen Strother, Anisha Keshavan, Pradeep Kumar Reddy Raamana, mlin...@jhsph.edu, JB Poline, ni...@googlegroups.com, ten.p...@gmail.com
Hi Pradeep,

I went through a very similar process when designing feedback forms for brain hack.

To my utmost surprise my ethics committee told me not to submit anything!

There is a paragraph in the tri Council policy that explicit waives this type of data collection from ethics approval (and even more surprising to me, secondary analysis of public database as well, still sounds too good to be true).

I am on my phone now, but I will look for the document and forward it latter.

You may not have to do anything if you collect the data at Baycrest.

Pierre

Pradeep Reddy Raamana

unread,
Nov 27, 2018, 6:18:19 PM11/27/18
to Pierre Bellec, Stephen Strother, Anisha Keshavan, mlin...@jhsph.edu, JB Poline, ni...@googlegroups.com, Thomas Nichols, Mathew Abrams
Thanks Stephen and Pierre.

I was just going to reply to this, and mention "data" is collected via Baycrest email account (not INCF tools) and stored in Baycrest Google Drive, which I believe is physically in Canada only. Our Research Ethics Manager does not see this survey exercise as any Research Study at this point which otherwise would require an REB approval (e.g. if this is part of a larger research study).

I will try to get some sort of approval in the interest of publishing the results from this survey, as a backup, as it appears post-hoc approval would not be possible or is otherwise invalid. 

Hence it'd be ideal if INCF or OHBM (or a similar entity), with mandated oversight over such international efforts, provides an approval that is recognized by different neuroscience journals. 

Thanks,
Pradeep

PS CC-ing Matthew/INCF for his input.

Stephen Strother

unread,
Nov 27, 2018, 10:10:24 PM11/27/18
to Pradeep Reddy Raamana, Pierre Bellec, Anisha Keshavan, mlin...@jhsph.edu, JB Poline, ni...@googlegroups.com, Thomas Nichols, Mathew Abrams
Warning: This involves details of Canadian ethics rules regarding Pierre's observations and Pradeep's survey so read on at your own peril.

Thanks Pierre, I didn't know that. I got curious looked this up in the TCPS 2 (2014) docs.The tri-council's TCPS2 2014 states that (p. 15-16): "Article 2.2 Research that relies exclusively on publicly available information does not require REB review when: (a) the information is legally accessible to the public and appropriately protected by law; or (b) the information is publicly accessible and there is no reasonable expectation of privacy." Public accessibility seems to be the key here but not sure that applies in this case?

Reading through the 2 pages I came upon ..."Research that is non-intrusive, and does not involve direct interaction between the researcher and individuals through the Internet, also does not require REB review." Maybe this is the clause the provides an exemption but it seems to allow local REB wiggle room. Is providing an email and filling in a technical form non-intrusive? Probably depends on your REB, and read on if you want to do research on that data and publish the results.

Pradeep, I would be careful with the issue of "its a quality assurance/control study and not research". We have run into this in another setting with human scanner phantoms and are being currently told the data cannot be published because it was an unconsented QC study that involved a human phantom. This is covered by p. 18: "Article 2.5 Quality assurance and quality improvement studies, program evaluation activities, and performance reviews, or testing within normal educational requirements when used exclusively for assessment, management or improvement purposes, do not constitute research for the purposes of this Policy, and do not fall within the scope of REB review."  If you read through the following application paragraph it is clear that if it becomes research (i.e., publishable) REB review may be required. This leads us to (p. 76): Also on p. 76: "Article 5.5B Researchers shall seek REB review, but are not required to seek participant consent, for research that relies exclusively on the secondary use of non-identifiable information." I guess combined with 2.2 we get your secondary research of public databases without review, Pierre. But it is niQCs primary use so I don't think this applies. My guess is that you will need a quick REB approval for using the survey results for research, Pradeep, but Roshan is the gatekeeper wrt the REB.

And if you want to worry internationally I wonder if the GDPR applies in some way to Europeans filling in the survey? A good question for INCF to ponder.

Cheers, Stephen

Pierre Bellec

unread,
Nov 27, 2018, 11:42:58 PM11/27/18
to Stephen Strother, Pradeep Kumar Reddy Raamana, Anisha Keshavan, Martin Lindquist, JB Poline, ni...@googlegroups.com, Thomas Nichols, Mathew Abrams
Thanks Steve for looking into this that fast!
  And sorry everyone for the dry and Canado-Canadian discussion. 

Article 2.5 is exactly what I had in mind.

In my case, I did approach the REB and they did not require anything, even though I mentioned I wanted to publish with it.

FYI, based on this precedent, I made a unilateral decision that I did not need REB approval for a crowdsourced QC paper (that's the zooniverse one FYI).
  But now you are giving me second thoughts, the text is not as clear as I remembered. 
    I don't like to have to rely on the 2.5 -> 2.2 loophole.

Anyway, in your case Pradeep it seems safer to double check with your local REB indeed.

Best,

Pierre Bellec
 
Research associate professor | Professeur agrégé sous octroi, 
Psychology Department | Département de Psychologie 
Université de Montréal, Montréal, Canada
Director | Directeur, Courtois NeuroMod
Director | Directeur, Unité de Neuroimagerie Fonctionnelle (UNF)
Co-lead | Co-responsable, CCNA biomarkers team | CCNV équipe biomarqueurs
Researcher | Chercheur, Centre de recherche de l'institut Universitaire de gériatrie de Montréal (CRIUGM)
Phone | Téléphone +1 514 713 5596  

JB Poline

unread,
Nov 28, 2018, 12:10:32 AM11/28/18
to Stephen Strother, Pradeep Kumar Reddy Raamana, Pierre Bellec, Anisha Keshavan, Martin Lindquist, ni...@googlegroups.com, Thomas Nichols, Mathew Abrams
Hi Stephen

Those are indeed good questions to put to the secretariat, if my memory is correct they did get some first information session on the GDPR.

Cheers
JB

Malin Sandström

unread,
Nov 28, 2018, 9:11:58 AM11/28/18
to Pradeep Reddy Raamana, anishak...@gmail.com, Mathew Adrams, Helena Ledmyr, mlin...@jhsph.edu, JB Poline, pierre...@criugm.qc.ca, ni...@googlegroups.com, Thomas Nichols
Hello Pradeep,

Email addresses and names are personal data under GDPR, and if you have any Europeans in your survey dataset (which seems highly likely) you need to comply with GDPR. Where the data is stored doesn't affect that.

If you set up the survey in such a way that it can be done without referring to personal data – if it doesn't save email addresses or names or anything that jointly can be used to identify a person - GDPR does not apply.  So a completely anonymous survey would work. Have you seen the survey Tibor Auer set up recently (on Open Science, directed to OHBM and INCF communities, among others)? He has a consent form at the beginning, and only collects country, field of research, seniority level/position.


Best
Malin



For more options, visit https://groups.google.com/d/optout.


--
Malin Sandström, PhD
Community Engagement Officer
malin.s...@incf.org

International Neuroinformatics Coordinating Facility
Karolinska Institutet
Nobels väg 15 A
SE-171 77  Stockholm
Sweden
http://www.incf.org


Pradeep Reddy Raamana

unread,
Nov 28, 2018, 9:22:27 AM11/28/18
to Malin Sandström, anishak...@gmail.com, Mathew Adrams, Helena Ledmyr, mlin...@jhsph.edu, JB Poline, pierre...@criugm.qc.ca, ni...@googlegroups.com, Thomas Nichols
Thanks Malin - our survey is even more anonymous! No emails, country or seniority or anything direct. A remote proxy to identifiable info is in the papers they may link to in answering the survey - which in some cases they might be referring to their own work. Even then the author list is typically large enough to make identification difficult (one can also argue those typically responding to these type of survey are either the PI or the first author of the QC paper).

We already clearly note at the top we plan to release their input, and I can add a Yes/No consent question before proceeding to the survey.. would that sufficiently comply with GDPR?

Would INCF be able to provide an approval of some sort, to show it to journals later on if and when we try to publish this?

Thanks,
Pradeep

Malin Sandström

unread,
Nov 28, 2018, 9:34:24 AM11/28/18
to Pradeep Reddy Raamana, anishak...@gmail.com, Mathew Adrams, Helena Ledmyr, mlin...@jhsph.edu, JB Poline, pierre...@criugm.qc.ca, ni...@googlegroups.com, Thomas Nichols
Nearly. The part of the study where you give consent - presumably the first page - also needs to give a bit of information about the study and what will be done with the data. Also, 'yes' cannot be the default answer, you need respondents to indicate 'yes' explicitly for the consent to be valid.

Please specify what kind of approval you need. A sentence of text? "This study is run by the INCF Special Interest Group on niQC, and supported by INCF."?

/Malin


For more options, visit https://groups.google.com/d/optout.

Pradeep Reddy Raamana

unread,
Nov 28, 2018, 10:06:50 AM11/28/18
to Malin Sandström, Anisha Keshavan, Mathew Abrams, Helena Ledmyr, mlin...@jhsph.edu, JB Poline, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
Thanks - updated the survey now. Info on the analyses, explicit consent question, and users are taken to the actual survey only when they say Yes (which is not default). Take a look: https://docs.google.com/forms/d/e/1FAIpQLSeZNCcqVnKgfqVkRHciouHYyDQRNtAaUuJvgNsuh3tQiKT_7Q/viewform?usp=sf_link

I will get this reviewed with our Research Ethics Manager again, and get an approval.

Malin, I will get back to you on the form of approval needed.

Pradeep

Pradeep Reddy Raamana

unread,
Nov 29, 2018, 4:15:29 PM11/29/18
to Malin Sandström, Anisha Keshavan, Helena Ledmyr, Martin Lindquist, JB Poline, Pierre Bellec, ni...@googlegroups.com, Thomas Nichols
Sorry for the numerous reminders, but if you have any comments or reservations about the survey, please let me know by tomorrow, as I will be submitting this to REB approval from my institute (as soon I am able to fill the 23-page form, likely by the weekend) and we may not be able to make any major changes later on (minor changes in language etc would be okay).

Thanks,
Pradeep


Stephen Strother

unread,
Nov 29, 2018, 10:58:37 PM11/29/18
to ni...@googlegroups.com
Pradeep - My comments on the form are below. I recognise this was realtively easy to put together but as a result sorting out any useful inputs will be much, much  harder, if not close to impossible given that there is no attempt at even a small predefined vocabulary and limiting types of responses. Using a more structured form with drop down lists and defined options/categories, would of course take longer and be harder to put together but would be likely to yield much more useful data in my view. I know there has already been some discussion on this so I am not expecting you to necessarily adopt my view, but I am stating it as you asked. 

1. You probably should define what you mean by QC. My first question was do I include QA procedures like the fBIRN phantom, but in my set of definitions this is not QC, it is QA. If you want it all use QA/QC.

2. Are you asking for a new form for each context/pipeline/modality, etc, or a list of all of these on one form, which must then track through the form’s answers? I would prefer one form per context and then I would just choose a few primary contexts to fill in a few sets of forms. The whole exercise seems daunting otherwise where I would have to write a lot of text in some of the questions. Parsing and sorting this out across perhaps 100s of forms will be a very big, nightmarish task. Who is going to do it? Free text is a really bad way to manage primary survey inputs. I think a much better form would attempt a list of all the contexts you can think of that people can select from + an other line where they can still write whatever they want. This will be much easier to manage later, but much harder to put together. In my view you will end up doing even more total work with the current form structure.

3. Questions like below seem to assume one process/context per form.
Do you think it's possible to harmonize this process across labs/world?
You can use the Other option to note down your thoughts and provide more details.
Yes
No
Other:

4. My answer to below would probably be maybe with a set of caveats that would stand in the way of adoption. You don’t allow for this.
If a standardized protocol were to be developed, would you adopt it?
Yes
No

Cheers, Stephen

For more options, visit https://groups.google.com/d/optout.


-- 
----------------------------------------------------------------------------------------------------------------
To reduce e-mail overload follow the e-mail charter of Chris Anderson: http://emailcharter.org/.

Stephen Strother, PhD
Senior Scientist, Rotman Research Institute, Baycrest
Professor of Medical Biophysics, University of Toronto
E-mail: sstr...@research.baycrest.org

Pradeep Reddy Raamana

unread,
Nov 30, 2018, 11:23:56 AM11/30/18
to Stephen Strother, ni...@googlegroups.com
Thanks Stephen - see below for quoted responses:

On Thu, Nov 29, 2018 at 10:58 PM Stephen Strother <sstr...@research.baycrest.org> wrote:
Pradeep - My comments on the form are below. I recognise this was realtively easy to put together but as a result sorting out any useful inputs will be much, much  harder, if not close to impossible given that there is no attempt at even a small predefined vocabulary and limiting types of responses. Using a more structured form with drop down lists and defined options/categories, would of course take longer and be harder to put together but would be likely to yield much more useful data in my view. I know there has already been some discussion on this so I am not expecting you to necessarily adopt my view, but I am stating it as you asked. 

This was the first and most common feedback we received so far. You're right in observations, but the purpose of this first-of-more-to-come surveys is to simply scan and review the landscape, not to develop an implementable protocol. This survey would help us compile and categorize the usecases, and summarize the existing work (in various aspects). As this is an uncharted landscape (to the best of my knowledge at least), it is best to keep this "survey" not restricted to few categories we have already seen, and keep it open for discovery. Some of the niQC members have already proposed specialized drafts towards the usecases they are involved in, and we can pick up on them anytime.

Positive outcomes in the best case scenario : we all learn something new ( in terms of usecases, what is done, how it is justified etc), and would help develop a protocol that is building on what is already known in the community.

Worst cases scenario: no surprises and we do not learn anything new, and simply delayed us getting into the protocol work.

Perhaps I am naive, but I believe this would teach us all something new, and even otherwise nobody can blame us for not including their usecase or considering their work into the protocol development.

Soon after we compiled this survey (likely end of January or mid-February), we can get started on the specific protocols.
 

1. You probably should define what you mean by QC. My first question was do I include QA procedures like the fBIRN phantom, but in my set of definitions this is not QC, it is QA. If you want it all use QA/QC.

Good point - I will refer to them QC/QA everywhere. 
 

2. Are you asking for a new form for each context/pipeline/modality

Yes - this is our atomic "unit" for protocols.

 
, etc, or a list of all of these on one form, which must then track through the form’s answers?

Yes!
 
I would prefer one form per context and then I would just choose a few primary contexts to fill in a few sets of forms. The whole exercise seems daunting otherwise where I would have to write a lot of text in some of the questions.

Can you give me an example here?
 
Parsing and sorting this out across perhaps 100s of forms will be a very big, nightmarish task. Who is going to do it?

It might well be daunting, assuming we get 1000s of responses. I'd be happy to take primary responsibility, but I believe we have many niQC members willing to help ("divide and conquer"). My very limited experience tells me we won't even have a 1000 responses. I'd be quite happy if we get anything over 250 responses.
 
Free text is a really bad way to manage primary survey inputs. I think a much better form would attempt a list of all the contexts you can think of that people can select from + an other line where they can still write whatever they want.

I see where this is coming from, and I'd refer you back to my comments at the top. Repeating Lei's response from before "many of the questions are open in nature in that the candidate choices are either too many or not yet clear, especially ones on actual QC/QA approaches."
 
This will be much easier to manage later, but much harder to put together. In my view you will end up doing even more total work with the current form structure.

Probably. But I believe this needs to be done.
 

3. Questions like below seem to assume one process/context per form.

Yes.
 
Do you think it's possible to harmonize this process across labs/world?
You can use the Other option to note down your thoughts and provide more details.
Yes
No
Other:

4. My answer to below would probably be maybe with a set of caveats that would stand in the way of adoption. You don’t allow for this.

I've updated it to provide an Other option for custom text now.

Thanks,
Pradeep
 

For more options, visit https://groups.google.com/d/optout.

Stephen Strother

unread,
Nov 30, 2018, 2:42:34 PM11/30/18
to Pradeep Reddy Raamana, ni...@googlegroups.com
OK. I hope you find aggregating 100s of free text form responses easier than I believe it will be, but if the overall group is prepared to put significant time into aggregating the form responses that I agree it might be a valuable exercise. I am probably getting a bit cynical with age about what people will and wont do. For other responses see below.

On 2018-11-30 11:23 AM, Pradeep Reddy Raamana wrote:
Thanks Stephen - see below for quoted responses:

On Thu, Nov 29, 2018 at 10:58 PM Stephen Strother <sstr...@research.baycrest.org> wrote:
Pradeep - My comments on the form are below. I recognise this was realtively easy to put together but as a result sorting out any useful inputs will be much, much  harder, if not close to impossible given that there is no attempt at even a small predefined vocabulary and limiting types of responses. Using a more structured form with drop down lists and defined options/categories, would of course take longer and be harder to put together but would be likely to yield much more useful data in my view. I know there has already been some discussion on this so I am not expecting you to necessarily adopt my view, but I am stating it as you asked. 

This was the first and most common feedback we received so far. You're right in observations, but the purpose of this first-of-more-to-come surveys is to simply scan and review the landscape, not to develop an implementable protocol. This survey would help us compile and categorize the usecases, and summarize the existing work (in various aspects). As this is an uncharted landscape (to the best of my knowledge at least), it is best to keep this "survey" not restricted to few categories we have already seen, and keep it open for discovery. Some of the niQC members have already proposed specialized drafts towards the usecases they are involved in, and we can pick up on them anytime.

Positive outcomes in the best case scenario : we all learn something new ( in terms of usecases, what is done, how it is justified etc), and would help develop a protocol that is building on what is already known in the community.

Worst cases scenario: no surprises and we do not learn anything new, and simply delayed us getting into the protocol work.

Perhaps I am naive, but I believe this would teach us all something new, and even otherwise nobody can blame us for not including their usecase or considering their work into the protocol development.

Soon after we compiled this survey (likely end of January or mid-February), we can get started on the specific protocols.
 

1. You probably should define what you mean by QC. My first question was do I include QA procedures like the fBIRN phantom, but in my set of definitions this is not QC, it is QA. If you want it all use QA/QC.

Good point - I will refer to them QC/QA everywhere. 
 

2. Are you asking for a new form for each context/pipeline/modality

Yes - this is our atomic "unit" for protocols.

 
, etc, or a list of all of these on one form, which must then track through the form’s answers?

Yes!
 
I would prefer one form per context and then I would just choose a few primary contexts to fill in a few sets of forms. The whole exercise seems daunting otherwise where I would have to write a lot of text in some of the questions.

Can you give me an example here?
We are working with multiple resting state QA/QC pipelines, which are quite different in many aspects. If we try to shove them all into a single form it will be hard for both us and the reviewer/reader to track which response applies to which pipeline across questions.
Reply all
Reply to author
Forward
0 new messages