Issue running HCP data through HCP Pipeline using BIDs' Docker App

[Danielle Kurtin]

Hello, I hope this finds you well.

I am trying to run the unprocessed HCP Wu-Minn 1200 dataset using the HCP PreProcessing Pipeline Docker App. To ensure my data was in BIDS format, I found the BIDs validator website and ensured my data was in BIDs format. I then tried to run my data, and received a BIDS-related error stating a BIDs root doesn't exist (when the path it's referring to definitely exists). Help would be really appreciated!!

(base) dk00549@bigdata-master01:/vol/research/nemo/HCP/Scripts$ cat c524.p0.error
Traceback (most recent call last):
File "/run.py", line 246, in
layout = BIDSLayout(args.bids_dir, derivatives=False, absolute_paths=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 230, in init
self._validate_root()
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 460, in _validate_root
raise ValueError("BIDS root does not exist: %s" % self.root)
ValueError: BIDS root does not exist: /vol/research/nemo/HCP/UnprocessedHCPSubjects

[Glasser, Matthew]

Why are you rerunning the processing?

Matt.

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.
To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/01aedb55-f1cb-4079-9681-ee0730cb60aan%40humanconnectome.org.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Danielle Kurtin]

I want to be able to use and understand the preprocessing pipeline. I figured if I could get it to run on the HCP dataset, where it was meant to run, I could apply it to other datasets in the future. But I'm struggling to get it to work on the HCP dataset, as well as the UCLA Consortium dataset I was also trying to run it on: https://openneuro.org/datasets/ds000030/versions/1.0.0. I was playing around with that dataset since it already came in BIDs format, but I run into errors with it as well. 

You received this message because you are subscribed to a topic in the Google Groups "HCP-Users" group.
To unsubscribe from this topic, visit https://groups.google.com/a/humanconnectome.org/d/topic/hcp-users/aCCEZrDhAyE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hcp-users+...@humanconnectome.org.
To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/9DB05238-9374-4CF0-A4EB-3D6BA621D3F4%40wustl.edu.

--

__________________________________________

Danielle Kurtin, BSc MSc

Neuroscience | Psychology

University of Surrey | PhD Candidate

Imperial College London | Visiting Researcher

e: danielle...@imperial.ac.uk

web page: https://www.imperial.ac.uk/people/danielle.kurtin18
__________________________________________

[Glasser, Matthew]

Okay that is certainly reasonable.  Unfortunately I don’t know who maintains that Docker App and we don’t internally use BIDS.  I know there is a planned release of an official HCP containerized version of the pipelines that I believe is also compatible with BIDS called Qu|Nex.  Not sure how soon that will happen but perhaps Grega can comment.  Personally, I’ve never gotten into BIDS as it seems to be a substitute for understanding how the images and their meta data relate to running processing pipelines and I already have that understanding.

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CABOCz9drTpNW97zLX0eyzWDwUURN455yoaxdr6qPLv4aafwQ8Q%40mail.gmail.com.

[Kenley, Jeanette]

Just curious, did you mount the path in question using the  -v flag?

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

---

From: Glasser, Matthew <glas...@wustl.edu>
Sent: Friday, February 19, 2021 9:26 AM
To: hcp-...@humanconnectome.org <hcp-...@humanconnectome.org>
Cc: Grega Repovs <Grega....@ff.uni-lj.si>

Subject: Re: [hcp-users] Issue running HCP data through HCP Pipeline using BIDs' Docker App

 

* External Email - Caution *

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/7E248346-B71E-4743-AA9D-D658D877A33E%40wustl.edu.

[Elam, Jennifer]

Perhaps Roeland, Chris, or Zailyn can help.

Best,

Jenn

Jennifer Elam, Ph.D.
Scientific Outreach, Human Connectome Project
Washington University School of Medicine
Department of Neuroscience, Box 8108
660 South Euclid Avenue
St. Louis, MO 63110
314-362-9387
el...@wustl.edu
www.humanconnectome.org

---

From: Kenley, Jeanette <jkke...@wustl.edu>
Sent: Friday, February 19, 2021 9:29 AM

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CH2PR02MB666126A2DFF8E032E9EC76A3D8849%40CH2PR02MB6661.namprd02.prod.outlook.com.

[Tamayo, Zailyn]

Hello,

 

I am not familiar with this BIDS app they are trying to use (https://github.com/BIDS-Apps/HCPPipelines). We can refer them to use QuNex, which is BIDS compatible, as Matt stated. They should be able to register to get the container next week. They are welcome to reach out to me directly next week for an update on the container’s availability.

 

Thanks,

Z

[Glasser, Matthew]

I would also note that we will be coordinating with Qu|Nex to ensure that this container is kept up to date with HCP Pipeline releases and contains enables all released HCP Pipelines.  We have no relationship with this BIDS app, and so I don’t know if they are providing an up to date version of the HCP Pipelines (currently 4.3.0) or if they support all released HCP Pipelines. 

Matt.

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/MN2PR08MB6191A23C8C6E70D47ED8F72599849%40MN2PR08MB6191.namprd08.prod.outlook.com.

[Danielle Kurtin]

Hi Matthew, 

Thanks for the quick reply. I've asked questions on the BIDs page as well as raised (more than one) issue on their Git repo, and we'll see if I get a response. 

I switched to the BIDs app because setting up all the dependencies and libraries to use the Preprocessing Pipeline was taking a very long time, and (as with everyone in research) I was feeling the time crunch and thought using a Docker container would expedite the process. As for BIDs in general, I recognize the merits (standardization) and the flaws (of which you've voiced one). 

I checked out Qu|Nex and the website looks lovely; I may have a play around with it later :) Thank you for sharing that!

All the best,

Danielle

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/7E248346-B71E-4743-AA9D-D658D877A33E%40wustl.edu.

[Danielle Kurtin]

Hello Jeanette,

Thank you for your help- when you refer to mounting the path, do you mean if I've mounted the path reported in the error output using the -v command? And does this refer to mounting it within the submit_file, or just in general?

Many thanks,

Danielle

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CH2PR02MB666126A2DFF8E032E9EC76A3D8849%40CH2PR02MB6661.namprd02.prod.outlook.com.

[Glasser, Matthew]

I think containers definitely have their positives and negatives too.  On the one hand, it helps a lot to get the environment set up quickly and also to standardize what was actually run on a given dataset.  On the other, it can be harder to debug problems because one has introduced another layer between the inputs and the processing pipelines.  Overall, we are aiming to support both approaches (inside and outside a container) to processing with the HCP Pipelines. 

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CABOCz9fOgowANg7tsEUaj08VPJb%2BY5jWquY0z14PJWuioBNgbQ%40mail.gmail.com.

[Danielle Kurtin]

Hi Z,

Thank you for the heads up on the QuNex container! If I wanted to reach out to you about it, do you prefer I email you at zailyn...@yale.edu? When you refer to registering to get the container next week, do you mean it might be available for public use next week?

Many thanks for your time and help,

Danielle

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/MN2PR08MB6191A23C8C6E70D47ED8F72599849%40MN2PR08MB6191.namprd08.prod.outlook.com.

[Danielle Kurtin]

These are the pros and cons I try to weigh as well. I'd like to be adept at both methods, and I will continue to work on them even if I use the already preprocessed data to progress my project. The HCP has been an incredible resource largely because of the extensive, clear documentation and this forum. I deeply appreciate the support and the time and effort the HCP developers, maintainers, and community has spent creating and maintaining the database, documentation, code, and everything else!

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/4155822C-FB70-4355-B9F5-FBC34185290A%40wustl.edu.

[Harms, Michael]

 

Another point worth making is that the “simplification” that some wrappers attempt to provide, e.g., by hiding various parameters will certain assumed default values, is not necessarily a “good thing”, in that the assumed defaults may not be the correct ones for your data.  Last time I looked at the BIDS App for the HCPPipelines it (1) only covered some of the available HCPpipelines, and (2) had quite a bit of hidden simplication.  The QuNex container is better in this regard, but even there, you still need to be familiar with the HCPpipelines themselves, and whether you are calling with the correct arguments for your data.

 

Cheers,

-MH

 

-- 

Michael Harms, Ph.D.

-----------------------------------------------------------

Associate Professor of Psychiatry

Washington University School of Medicine

Department of Psychiatry, Box 8134

660 South Euclid Ave.                        Tel: 314-747-6173

St. Louis, MO  63110                          Email: mha...@wustl.edu

 

From: Danielle Kurtin <danielle...@gmail.com>
Reply-To: "hcp-...@humanconnectome.org" <hcp-...@humanconnectome.org>
Date: Friday, February 19, 2021 at 10:54 AM
To: "hcp-...@humanconnectome.org" <hcp-...@humanconnectome.org>
Subject: Re: [hcp-users] Issue running HCP data through HCP Pipeline using BIDs' Docker App

 

* External Email - Caution *

These are the pros and cons I try to weigh as well. I'd like to be adept at both methods, and I will continue to work on them even if I use the already preprocessed data to progress my project. The HCP has been an incredible resource largely because of the extensive, clear documentation and this forum. I deeply appreciate the support and the time and effort the HCP developers, maintainers, and community has spent creating and maintaining the database, documentation, code, and everything else!

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CABOCz9eck8HA2fxNf110vPcpDTMpJmB3REUR65TJY0FG1n0Gig%40mail.gmail.com.

[Kenley, Jeanette]

Danielle,

I am new to containers, so not an expert but one mistake I made recently is if I didn't mount the data on my server to where the docker expected to find it, then it fails.

I am also not using the containter you are working with, but it's just something I've messed up.

Thanks,

Jeanette

i.e.

docker run --rm --user 1191:10 -v /vol/research/nemo/HCP/UnprocessedHCPSubjects:/bids_input:ro   <----this line is saying I want you to make my folder look like the /bids_input folder  (this is what the docker I am using expects).

And *I believe* instead of having all of your input bids files pointing to /vol/research/nemo/HCP/UnprocessedHCPSubjects, you would want to use /bids_input, but in place of bids input you would put what your docker is expecting.

I've also used the mount flag with code and licenses as required by the docker I am using.

-v /usr/local/pkg/freesurfer6.0/license.txt:/license

-v /path/to/python/code/on/my/server/code.py:/path/on/docker/where/code/is/located/code.py

I hope this answers your question.

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

---

From: Danielle Kurtin <danielle...@gmail.com>
Sent: Friday, February 19, 2021 10:47 AM

To: hcp-...@humanconnectome.org <hcp-...@humanconnectome.org>
Subject: Re: [hcp-users] Issue running HCP data through HCP Pipeline using BIDs' Docker App

 

* External Email - Caution *

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CABOCz9dZw7nDVSKy2wEmbg9tezdpion4J1EQo-ULw_9KNo%2BMng%40mail.gmail.com.

[Kenley, Jeanette]

Danielle,

Unfortunately I have no experience with submit files.

But it looks like you are using a template already.  (I assume the template is from the HCP docker people and it looks like it is accounting for the mounting, I think.)

You have some parts towards the end of you file like this:

--skip_bids_validation  /user/HS228/dk00549/HCP/HCPSubjects /user/HS228/dk00549/HCP/HCPSubjects/output   

The first path being the input with the subject data inside and the second being where you want the output stored.    (I'm assuming there isn't an additional flag you can call like --docker-input   --docker-output?  That would be convenient.)

Based on your initial email, the first path exists and so I am assuming the second does too.  : )  One thing I run into with the docker I run is if I've tried to run the docker and it fails and the subject folder is in the output folder, I have to delete the subject folder in output directory before I can reprocess it.

The only other thing I can see on the submit file is that there is a line about environment, maybe that needs to be uncommented based on the paths above?  

Sorry I can't offer more than that.

Thanks,

Jeanette

The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email.

---

From: Danielle Kurtin <danielle...@gmail.com>
Sent: Saturday, February 20, 2021 6:58 AM

To: hcp-...@humanconnectome.org <hcp-...@humanconnectome.org>
Subject: Re: [hcp-users] Issue running HCP data through HCP Pipeline using BIDs' Docker App

* External Email - Caution *

Hi Michael and Jeanette,

Michael, I agree with you as well. I'm looking forward to learning more about QuNex!

Jeanette, thank you for your suggestions. Unfortunately I don't run Docker in that way, I use a .submit_file. I'm working on a HPC with HTCondor as the workflow management system. This is why I'm likely to agree with those who've highlighted the difficulty troubleshooting using Docker- I have very few lines to work with. I've attached my submit_file, this is what I submit to the queue, and (normally) would pull the container and run the HCPPipeline steps described in the submit_file.

Do you think before I submit, or perhaps somewhere in the .submit_file, I could add commands similar to the ones you've specified?

Thank you both for your help and suggestions! I appreciate your time, effort, and contributions immensely.

Cheers,

Danielle

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CH2PR02MB6661D0DD52848A749CF4948DD8849%40CH2PR02MB6661.namprd02.prod.outlook.com.

--

__________________________________________

Danielle Kurtin, BSc MSc

Neuroscience | Psychology

University of Surrey | PhD Candidate

Imperial College London | Visiting Researcher

e: danielle...@imperial.ac.uk

web page: https://www.imperial.ac.uk/people/danielle.kurtin18
__________________________________________

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CABOCz9dKRXSnK2SkOtd7_%2BtbYUgP%3Dk1WsEaELRvx3skAn-kmYQ%40mail.gmail.com.

[Danielle Kurtin]

Hi Jeanette,

Thank you for the suggestions! The submit_file template is from some tutorials for the HCP at my uni, and I worked with IT to figure out how to pull the container and specify the parameters. 

Regarding the input/output; there's no flags for it, as they're position dependent. However I did move the output folder to see if that helped, but it didn't change the error or log outputs. I also tried to uncomment the commented-out line specifying environment; I had commented it out since we're pulling a container remotely and not relying on a software environment or a manually configured environment. When I uncomment it, it throws the same error. 

We'll see if I can eventually figure it out; if I do I'll post the solution :) Thank you for your time, and I appreciate your help. Cheers,

Danielle

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/CH2PR02MB6661BA2EF7F01F2E8848C70FD8819%40CH2PR02MB6661.namprd02.prod.outlook.com.